As part of the desktop improvements project we spent time investing in the core code that powers skins. With support from volunteers (the majority of this support coming from the prolific @Ammarpad), we identified code patterns and made changes to the MediaWiki-Core-Skin-Architecture to retroactively define a data layer API for generating a skin.

Once this was in place, we updated the legacy MediaWiki skins Monobook, Modern, CologneBlue to use Mustache to bring them in line with how Vector and Minerva were built.

The rationale for doing this was as follows:

1. We wanted to centralize code into core, and standardize markup, to make it easier to roll out changes to all skins. Often developers found ourselves updating every skin every time we wanted to make a small change or forced to use specific classes to markup elements (e.g. T248137, T253938).
2. We wanted to move away from server-side technologies to client-side technologies to play better to the strengths of frontend engineers and designers who worked on skins.
3. Since many of these skins do not see active development, we wanted to support them better by reducing lines of code
4. Many of the skins didn't support certain extensions because they used different code (for example certain skins didn't run hooks that were used by certain features) e.g. 6ce3ce1acb68f0a3fdf1bd8824f6d0717bffa320 T259400
5. Stop supporting features in core that were never widely adopted e.g. T97892

This process reduced 106,078 lines of code to 85,310 lines of code - a 20% decrease.
Before the change around 45% of skin code was PHP. After the change PHP only accounted for 15% of the code.

It would be great to in the future migrate Timeless too, but Timeless using the legacy skin platform does help keep us accountable for ensuring we continue to support skins built on this platform.

Methodology for result

To measure code makeup we can run github-linguist before and after the change.

Monobook

Before:

46.53%  22713      Less
36.83%  17981      PHP
16.53%  8071       JavaScript
0.10%   50         CSS
Lines of code: 48815

After change (abe94aa4082dbc4f8b9060528a1b4fea2d0af0f1)

59.28%  22831      Less
20.96%  8071       JavaScript
11.67%  4496       Mustache
7.96%   3066       PHP
0.13%   50         CSS
Lines of code: 38514

Modern

Before:

52.25%  13752      CSS
40.99%  10790      PHP
4.16%   1094       Less
2.61%   686        JavaScript
Lines of code: 26322

After change (c74d67950b6de2bafd9e3b1e05e601caaa7d9452)

68.87%  13877      CSS
18.22%  3672       Mustache
5.43%   1094       Less
4.07%   821        PHP
3.40%   686        JavaScript
Lines of code: 20150

Cologne Blue

Before:

62.00%  19183      PHP
34.82%  10773      CSS
2.22%   686        JavaScript
0.97%   299        Less
Lines of code: 30941

After change (bf06742467f6c6c2bb42367f2e073eb26ed5d495)

40.40%  10765      CSS
31.87%  8491       PHP
24.04%  6405       Mustache
2.57%   686        JavaScript
1.12%   299        Less
Lines of code: 26646

PHP

The total number of lines of PHP before the change: 47954
After the change: 12378 lines of PHP
(This is a 74% decrease in lines of code)

Here I share some thoughts around the history of "responsive" MediaWiki skins and how we might want to think about it for Vector.

The buzzword "responsive" is thrown around a lot in Wikimedia-land, but essentially what we are talking about is whether to include a single tag in the page. The addition of a meta tag with name viewport, will tell the page how to adapt to a mobile device.

<meta name="viewport" content="width=device-width, initial-scale=1">

More information: https://css-tricks.com/snippets/html/responsive-meta-tag/

Since the viewport tag must be added, by default websites are not made mobile-friendly. Given the traditional Wikimedia skins were built before mobile sites and this tag existed, CologneBlue, Modern, Vector did not add this tag.

When viewing these skins on mobile the content will not adapt to the device and instead will appear zoomed out. One of the benefits of this is that the reader sees a design that is consistent with the design they see on desktop. The interface is familiar and easy enough to navigate as the user can pinch and zoom to parts of the UI. The downside is that reading is very difficult, and requires far more hand manipulation to move between sentences and paragraphs, and for this reason, many search engines will penalize traffic.

Enter Minerva

The Minerva skin (and MobileFrontend before it) were introduced to allow us to start adapting our content for mobile. This turned out to be a good decision as it avoided the SEO of our projects from being penalized. However, building Minerva showed that making content mobile-friendly was more than adding a meta tag. For example, many templates used HTML elements with fixed widths that were bigger than the available space. This was notably a problem with large tables. Minerva swept many of these issues under the rug with generic fixes (For example enforcing horizontal scrolling on tables). Minerva took a bottom-up approach where it added features only after they were mobile-friendly. The result of this was a minimal experience that was not popular with editors.

Timeless

Timeless was the 2nd responsive skin added to Wikimedia wikis. It was popular with editors as it took a different approach to Minerva, in that it took a top-down approach, adding features despite their shortcomings on a mobile screen. It ran into many of the same issues that Minerva had e.g. large tables and copied many of the solutions in Minerva.

MonoBook

During the building of Timeless, the Monobook skin was made responsive (T195625). Interestingly this led to a lot of backlash from users (particularly on German Wikipedia), revealing that many users did not want a skin that adapted to the screen (presumably because of the reasons I outlined earlier - while reading is harder, it's easier to get around a complex site. Because of this, a preference was added to allow editors to disable responsive mode (the viewport tag). This preference was later generalized to apply to all skins:

Responsive Vector

Around the same time, several attempts were made by volunteers to force Vector to work as a responsive skin. This was feature flagged given the backlash for MonoBook's responsive mode. The feature flag saw little development, presumably because many gadgets popped up that were providing the same service.

Vector 2022

The feature flag for responsive Vector was removed for legacy Vector in T242772 and efforts were redirected into making the new Vector responsive. Currently, the Vector skin can be resized comfortably down to 500px. It currently does not add a viewport tag, so does not adapt to a mobile screen.

However, during the building of the table of contents, many mobile users started complaining (T306910). The reason for this was that when you don't define a viewport tag the browser makes decisions for you. To avoid these kind of issues popping up it might make sense for us to define an explicit viewport to request content that appears scaled out at a width of our choosing. For example, we could explicitly set width 1200px with a zoom level of 0.25 and users would see:

If Vector was responsive, it would encourage people to think about mobile-friendly content as they edit on mobile. If editors insist on using the desktop skin on their mobile phones rather than Minerva, they have their reasons, but by not serving them a responsive skin, we are encouraging them to create content that does not work in Minerva and skins that adapt to the mobile device.

There is a little bit more work needed on our part to deal with content that cannot hit into 320px e.g. below 500px. Currently if the viewport tag is set, a horizontal scrollbar will be shown - for example the header does not adapt to that breakpoint:

Decisions to be made

1. Should we enable Vector 2022's responsive mode? The only downside of doing this is that some users may dislike it, and need to visit preferences to opt-out.
2. When a user doesn't want responsive mode, should we be more explicit about what we serve them? For example, should we tell a mobile device to render at a width of 1000px with a scale of 0.25 ( 1/4 of the normal size) ? This would avoid issues like T306910. Example code [1] demo
3. Should we apply the responsive mode to legacy Vector too? This would fix T291656 as it would mean the option applies to all skins.

[1]

<meta name="viewport" content="width=1400px, initial-scale=0.22">

For WMF staff's inspiration week, I decided to take a step back from my work building out a new skin architecture and a redesign of Vector and put myself into the shoes of a skin developer to see if the changes my team had made life easier. As a secondary objective, I was interested in how a MediaWiki skin could be written in Vue.js and what the challenges were to get there.

What I built

I decided to build a new skin called Alexandria named after the Great Library. The design was modeled on an open-source project and website I volunteer for called OpenLibrary.org.

I began by creating a skin that was JavaScript only.

I generated most of my boilerplate using the skins.wmflabs.org tool. I tweaked it so it gave me all the client-side tooling I needed.

Once I had that, I added a few more advanced features to my skin. In particular, I created a PHP class that extended the core SkinMustache class to allow me to extend the data given by core.

I wanted to be able to render my skin in JavaScript, so I needed to pass the data in PHP to the client. To do this, I added a template data value that represented a stringified JSON of the entire data that would be passed to the template like so:
https://github.com/jdlrobson/Alexandria/blob/master/SkinAlexandria.php#L29

The skin template rendered this JSON into a data attribute.

<div id="ol-app" data-json="{{data-json}}"></div>

This rendered a blank screen that was readable by JavaScript. While not a useful skin, from here, I was able to begin using Vue.js to parse that data attribute and pass it through Vue components to render the skin. https://github.com/jdlrobson/Alexandria/blob/master/resources/skin.js#L20

From here, I was up and running. I had a skin made from a Vue.js component that said hello world!

Building a skin with Vue.

This really was a breeze.

I made use of the [[ http://github.com/wikimedia/wvui | wvui library ]]for existing standard components, such as TypeaheadSearch and Button by including the wvui and vue libraries in my main skin module.

Keen to test out the work Roan and others did to request ES6 only modules, I decided to allow myself the luxury of writing code in ES6 and excluding older browsers.

https://github.com/jdlrobson/Alexandria/blob/master/skin.json#L86

When components didn’t exist in the wvui library, I made them and thinking in terms of components led to well scoped CSS. Aside from components inside the wvui library, I ended up creating components such as App.vue, AppArticle.vue, AppBanner.vue, AppFooter.vue, AppHeader.vue, DropdownMenu.vue, FooterMenu.vue, Portlet.vue, TypeaheadSearch.vue.

One thing that was frustrating as I created/renamed these components is I had to update these in the skin.json manifest. It was unintuitive and I often forgot to do it, which made development a little more tedious. I captured this in a Phabricator ticket, as I think it’s something that could be much better in the developer experience: https://phabricator.wikimedia.org/T283388.

While wvui and Vue.js were on npm, Since I was relying on some styles inside MediaWiki-core, I couldn’t use Vite or Parcel.js without lots of scaffolding, so I decided to develop without hot reloading. This slowed me down a lot as I was doing a lot of page refreshing.

Using the OpenLibrary project I copied across the CSS I needed. The resulting CSS was much better organized and scoped than the original project as I was constantly thinking about reuse.

Styling article content

To generate articles I used the MobileFrontend content provider ( https://www.mediawiki.org/wiki/Extension:MobileFrontend#Testing_with_articles_on_a_foreign_wiki_(live_data)) to generate articles to test on. I found myself running into a few issues with that and a few CSS rules that were in Minerva that should have been in MobileFrontend for toggling and ended up submitting patches to deal with that: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/693503, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/696644

Styles for thumbnails and table of contents are provided by core. Both of these styles didn’t fit in with the aesthetics of my design, so I ended up adding override CSS. I would have preferred to have not spent any time styling these elements so raised Phabricator tickets lest I forget to revisit our defaults https://phabricator.wikimedia.org/T283836 and https://phabricator.wikimedia.org/T283396 .

I wanted to do a lot with the content - such as move all images and infobox to the left, but after wrestling with inline styles, CSS grid and I gave up. It would be great if the parser marked up articles in a way that lent itself better to a grid system, but sadly it doesn’t. I didn’t know what tasks to raise here, as I didn’t think about it too deeply, but I want to acknowledge that this was a point of pain.

Server side rendering

I then turned my attention to server-side rendering. MediaWiki currently doesn’t support server-side rendering of Vue components. (https://phabricator.wikimedia.org/T272878)

To server-side render Vue components, a Node.js service is advised which PHP can request HTML from. Because I’m a little crazy and didn’t want to set up such a service, for now, I explored the differences between Mustache and Vue.js templates. I built a Node script that imported Vue components, found the template tag, and then traversed the DOM of that template rewriting it node by node recursively to a Mustache equivalent. Constraining myself to the minimal work possible I managed to create https://github.com/jdlrobson/Alexandria/blob/master/ssr.js

This mostly worked but of course, I ran into a few problems.

This couldn’t load the general components in the wvui library. For these, I had to define fallback templates such as https://github.com/jdlrobson/Alexandria/blob/master/resources/TypeaheadSearch.vue. For the search widget, I ended up rendering a form fallback that looked nothing like the JavaScript Vue version, but that was fine. I made sure the script threw an error so that I’d never load it accidentally in my JavaScript application.

I have a bad habit of giving Vue component props the same name as attributes. I had to stop this. A parameter id became menuId for example. This allowed me to avoid too much complexity in my Mustache template generator, to know when I was dealing with an attribute or a template property.

With for loops, it was easier to parse v-for=”a in list than it was to parse something like v-for=”a in data.list” so I made sure that was a constraint in the Vue templates I was writing.

I decided against computed properties as these involved JavaScript so those were not supported in my proof of concept.

Now I was generating a template via a build step, my skin was working with JavaScript loading, however, loading styles for the new experience became my next problem. I had included all my styles in a Vue template, so now needed them out. I extended my build script to generate a stylesheet as well, but later backtracked on that and pulled the styles out of the Vue components. I opened https://phabricator.wikimedia.org/T283882 to discuss best practices for that.

API-driven frontend!

Now I had a skin rendering in Vue via JavaScript. It would be silly not to play to its strengths and make it a single-page application, loading content from JS. Unfortunately, there was no Skin API, only APIs for generating content and various things can vary on a page such as JavaScript/CSS loaded, mw.config values and even items in menu.. eek!!

A while back I made https://github.com/jdlrobson/mediawiki-skins-skinjson to help with skin development. It allows you to see a JSON representation of the data that a skin template can render. I repurposed this to allow me to use it as an API in my app. I made use of this. Pages rendered via API calls and JavaScript with ease. Wire up was very small: https://github.com/jdlrobson/Alexandria/blob/master/resources/App.vue#L252 Maybe something for the #product-infrastructure-team-backlog ?

This allowed me to load articles, however, I ran into technical debt. Most MediaWiki extensions expect to be run on page load. Some work, because of the use of mw.hook. For event handlers bound to the body tag using a proxy pattern, things just worked. We clearly need to use that pattern more if we ever want to go down this route.

E.g

$(‘body’).on(‘click’, ‘.uls-button’, loadULS );

https://skins-demo.wmflabs.org/wiki/Alexandria?useskin=alexandria demonstrates article loading using the Parsoid API

Reflection

• Converting Vue templates to Mustache is possible with constraints
• There are a few kinks in ResourceLoader that need to be worked out
• API-driven skins are possible if we're willing to put in the effort across our codebases to build the APIs and rethink how our existing features load.

HEADER CAPTION: The head of the Statue of Liberty on exhibit at the Paris World's Fair, 1878. The statue was built in France ahead of time, shipped overseas in crates, and then assembled in New York. Image by Albert Fernique / public domain.

The process of mapping human-readable source code inputs to optimized, machine-readable outputs is called compiling or more generally, building. It's been a necessary part of software development since computers evolved past machine code. Even to serve the most abstract, high-level languages such as HTML and CSS, this build process is essential.

Just-in-time build steps

We build code all the time at Wikimedia. Every page request benefits from Less compilation, CSS and JavaScript minification, internationalization, URL mapping, and bundling build steps. All of this occurs at runtime through the ResourceLoader pipeline.

ResourceLoader's just-in-time build process is critical when key parameters vary on request. However, it has some notable limitations including:

• Every just-in-time build step must be extremely performant, so fast that it can run on-the-fly, or our pages will load slowly. Additionally, sequential steps cannot be appended ad infinitum.
• Effectively, ResourceLoader's just-in-time build steps can only use tools written in PHP. JavaScript execution is not possible.
• Just-in-time build steps are less secure. They execute on production servers and serve content directly to the user. This eliminates the separation between development and runtime-only dependency trees, which can dramatically increase the attack surface, sometimes by orders of magnitude. Additionally, build outputs are shipped directly to the user without any opportunity for security review. When it comes to security, a just-in-time build step always strives to be as secure as an ahead-of-time build step that produces static outputs.
• Just-in-time build steps are custom and complex. An ahead-of-time build step can easily be a one-liner that invokes standard tooling but the equivalent just-in-time build step, if one exists, is just as likely to be hundreds of lines of custom code. Historically, these custom steps have suffered from bus factor and received little attention beyond basic life support. Few engineers possess the abilities to write code of the caliber needed to add new build steps or change existing ones, which means the rest of Wikipedia and WMDE is blocked on their evolution. For example, we have been unable to keep pace with fundamental features like source map support (a formal request since 2013) or ES6 transpilation. In fact, there are laundry lists of missing features now standard elsewhere. The lack of standard functionality means that developing any code at Wikimedia is a completely different and far slower experience than the rest of the industry.
• Just-in-time build step outputs have worse caching. The most advanced build step executed at runtime endeavors to have the same caching that comes out-of-the-box with an ahead-of-time build step: a plain file on disk.

Solving problems too big for just-in-time

Some problems are only solvable by just-in-time build steps. However, many solutions cannot meet the constraints of just-in-time build steps, so only a subset of all problems can be solved. This is a more general limitation of just-in-time build steps, not the ResourceLoader implementation. In practice, this means that developers cannot add a build step to the pipeline but are still left with their problem unsolved.

There must be an alternative. Our options include:

1. Double down on building new features in ResourceLoader. This approach fails to address the fundamental limitations of all just-in-time build steps and may require reimplementing existing open-source solutions.
2. Ship extra tooling to every user's browser and let them process it. Besides significantly increased bandwidth and computation costs that go against our mission to serve everyone, this isn't very eco-friendly, fails to solve many problems, leads to the laggy browsing experiences users so loathe on JavaScript-heavy pages, and doesn't scale far past polyfills.
3. Replace ResourceLoader with industry standard tooling that has fewer constraints. This will require exploration, be expensive, and may have the same outcome as #1.
4. Enhance ResourceLoader by building what we can ahead-of-time.

The first two options don't work. The third option doesn't sound like a good first choice. The fourth is the most conventional and proven solution.

Ahead-of-time build steps

Ahead-of-time build steps are usually what people think of when they refer to "building code." Most build problems that remain to be solved in Wikimedia only fit in the ahead-of-time space. As you might expect, we're using these enhancements all over the place already and can't live without them. Some examples include:

• OOUI: Portions of this library are built with Grunt and a suite of packages from NPM for minification, uglification, and additional processing. The results are dozens of build products that are file-copied into Core manually.
• Page Previews: This gem of a codebase is fully compiled from the latest JavaScript with Webpack. It serves about two billion virtual pageviews a month.
• Wikibase : Ahead-of-time build tools are used by Wikibase including Webpack, TypeScript, and a plethora of other standards to serve the Wikidata communities.
• MultimediaViewer: Commits to MultimediaViewer use ahead-of-time build steps to replace any human readable source SVGs with optimized, machine-readable outputs.
• MediaWiki: Core uses a build step on every deployment. The process is called "a full scap." When the process fails, it's called "a full scapadapadoo."
• MobileFrontend: All JavaScript in MobileFrontend, the heart of the mobile site, is built by Webpack. That's over 50% of all pageviews benefiting from an ahead-of-time build step using industry standard tooling.
• Wikipedia for KaiOS: This Webpack-powered project uses a build step to serve a highly performant web app.
• ContentTranslation: The glittering new ContentTranslation app uses the Vue CLI and standard tooling to generate the next-generation interfaces essential to serving contributors around the world. Put plainly, this is the kind of modern experience that would be impossible to build without modern tooling that leverages ahead-of-time build steps.
• Wikipedia.org: Portals uses a build step to synchronize sister project statistics. I know someone who has a recurring task each week reminding him "it's build time." Although triggering the build step is person-powered, the outputs are what you would expect of an ahead-of-time build step: practical and project specific.
• VisualEditor: VE is a sophisticated application that requires a build step. I don't know what this does exactly but I would guess it's solving the same kinds of problems everyone else has ahead-of-time.
• And many more.

These ahead-of-time build steps are everywhere in Gruntfiles, Gulpfiles, Webpack configs, NPM package.json files, and shell scripts. Even if the Foundation mandated it today, we could never get rid of them.

Evolving the ResourceLoader pipeline with a new stage

Ahead-of-time build steps are the only solution for many problems, so it's fortunate they have such a proven track record of success both within and beyond the MediaWiki ecosystem. As everyone who is already using ahead-of-time build steps has discovered, they're the perfect complement to ResourceLoader's just-in-time build steps.

However, this is a problem at scale and it needs to be solved at scale. Informal developer builds work surprisingly well but aren't as efficient for developers as they could be. We need to extend the pipeline to include a pre-ResourceLoader stage. This stage is an ahead-of-time build step.

In conclusion:

• ResourceLoader provides useful just-in-time build steps.
• Many projects have requirements that cannot be solved at runtime. These real problems are only solvable by traditional ahead-of-time build steps.
• Just-in-time and ahead-of-time build steps are already in use by and are for everyone, and we can't change that.
• Ahead-of-time build steps often use standard tools but are highly project specific. These should not be centralized nor should they be constrained by artificial limitations. Per-project solution autonomy must be preserved.
• Adding a pre-ResourceLoader stage can integrate neatly with the current ResourceLoader system by extending the pipeline to include these existing ahead-of-time workflows.

Above all, a build step means freedom. The freedom to succeed and the freedom to use the tool that's right for the job, not the rare tool that fits into a runtime-only pipeline.

Thanks to Jan Drewniak, Santhosh Thottingal, Daniel Cipoletti, Joe Walsh, Bernd Sitzmann, and Mónica Pinedo Bajo for reviewing and providing detailed feedback.

This post is also available on the Wikimedia Tech Blog.

HEADER CAPTION: Screenshot from Wikimedia's famous Visual Editor. The typo "documenation" has a red squiggly line under it indicating the spell checker has automatically detected a spelling error by the author.

Tools for validating that JavaScript documentation is current and error-free have advanced significantly over the last several years. It is now possible to detect mismatches between a program's documentation and its source code automatically using a free and open-source, industry-standard type checker. This goes way beyond typos.

JavaScript typing is loose

JavaScript is an untyped language. Unlike a typed language, a JavaScript program is always generated regardless of whether the types in it are valid. Some consider JavaScript's fast-and-loose style a feature, not a bug. Notable proponents of that viewpoint include Douglas Crockford and Paul Graham.

There have been numerous articles written on the subject, but I suspect that most reading already understand the values of clear typing. For any nontrivial program with multiple authors and any longevity, especially those likely to be found among the sprawling wikis, strong typing is much more practical and sustainable than the alternative. With good typing, one can quickly grasp the structure of a program. That is, you can conceptualize and interface with any well-typed API whether you understand how it works internally or not. Refactors are a lot easier too and while not fearless, a typed codebase is far more malleable than an untyped one. Type checks are also a great way to verify your work, just like in grade school.

Many bugs could be caught before arriving in production if every patch had its typing validated—but don't take my word for it. Phan, the PHP type checker, is now a required validation test for any change to MediaWiki Core as well as many extensions. It's like a bunch of built-in unit tests specifically for types. Without automation, these tests can require thousands of lines of hand written code that are tedious and time consuming to author, read, and maintain (e.g., see the otherwise excellent Popups extension). In the worst cases, no tests are written at all.

Documentation should be correct

Good typing is just as important in documentation. For JavaScript, documentation is largely written in JSDoc (or its deprecated competitor, JSDuck). Wikipedians seem to agree that documentation is a very good idea. If documentation is a good idea, correct and up-to-date documentation is an even better one. There's a tool for that: it's called TypeScript.

If you haven't heard of TypeScript yet, it may be because it's not very common at Wikimedia except for the uber-amazing work by the WMDE and Wikidata communities (e.g., see wikibase-termbox which is over 80% TypeScript) as well as explorations several years back by Joaquín Oltra Hernández. However, it is now immensely popular globally and proven itself by capability to be far more than just a fashionable trend from 2012.

So what is TypeScript exactly? TypeScript is JavaScript with types. Whether you choose to use it for functional code like WMDE or not, TypeScript features the ability to lint plain JavaScript files for the type correctness of their JSDocs. You don't need Webpack and you don't need to make any functional changes to your code (unless it's incorrect and out-of-sync from the documentation—i.e., bug fixes). Your JavaScript is the same as it ever was but now, if your documentation and program don't match, TypeScript will report an error.

This isn't just better documentation, it's documentation as accurate as we can write in an automated way. Who doesn't want better documentation?

What changes are needed?

Typing at the seams. In practice, this usually means documenting function inputs and outputs, and user types using JSDoc syntax. E.g.:

/**
 * Template properties for a portlet.
 * @typedef {Object} PortletContext
 * @prop {string} portal-id Identifier for wrapper.
 * @prop {string} html-tooltip Tooltip message.
 * @prop {string} msg-label-id Aria identifier for label text.
 * @prop {string} [html-userlangattributes] Additional Element attributes.
 * @prop {string} msg-label Aria label text.
 * @prop {string} html-portal-content
 * @prop {string} [html-after-portal] HTML to inject after the portal.
 * @prop {string} [html-hook-vector-after-toolbox] Deprecated and used by the toolbox portal.
 */

/**
 * @param {PortletContext} data The properties to render the portlet template with.
 * @return {HTMLElement} The rendered portlet.
 */
function wrapPortlet( data ) {
  const node = document.createElement( 'div' );
  node.setAttribute( 'id', 'mw-panel' );
  node.innerHTML = mustache.render( portalTemplate, data );
  return node;
}

CAPTION: If this code was undocumented or the types inaccurate, would you always get the data properties right? Maybe you would, but what about everyone else?

Most programmers are already typing their JavaScript to some extent with JSDocs, so often only refinements are needed. In other cases, TypeScript's excellent type inference abilities can be leveraged so that no changes are required.

Type definitions are a useful supplement to JSDocs. Definitions are non-functional documentation that support type annotations inline. For example, the definition of the powerful but fantastically loose jQuery API could find marvelous utility in many Wikimedia codebases for at-your-fingertips documentation needs. Another very relevant example that ships with TypeScript itself is the DOM definition, which will alert you to misalignments such as attempting to access a classList on a Node instead of an Element. Thorough type checking is similar and the perfect complement to ESLint checks for ES5-only sources or more broadly ESLint's safety checks.

Type definitions are also a convenient way to describe globals and, more generally, share types. Definitions are either shipped with the NPM package itself or DefinitelyTyped (e.g., npm i -D @types/jquery) and are now standard practice for most noteworthy JavaScript libraries. Imagine if this degree of accuracy could be achieved in some of our most well-used codebases. Integrations between skins, extensions, Core, and peripheral libraries would be validated for alignment. It would be harder to break things and a much more welcoming experience for newcomers.

npm install jsdoc typescript → tsconfig.json → tsc → Document!

The actual project setup for adding documentation checks to an existing repository is minimal and requires no functional changes:

1. Add JSDoc and TypeScript as NPM development dependencies. Optionally: add any missing types for third-party libraries used.
2. Add a tsconfig.json to tell TypeScript to lint JavaScript documentation you wish to validate.
3. Add tsc to the project's NPM test script.

The real work is in fleshing out the missing documentation with JSDocs. However, TypeScript is quite flexible about how one chooses to opt in or out of documentation validation. If code isn't worth documenting, it's probably not worth keeping, but typing can be consciously deferred in a number of ways. The most straightforward is probably with a // @ts-ignore comment. Think of it as progressively enhanced code.

An example project setup for Vector is here which shows how typing and documentation can be retrofitted nicely even on codebases that predate TypeScript and make heavy use of sophisticated APIs like jQuery.

Editor support

It's unnecessary for setting up a project, but worth mentioning, that by ensuring that even a machine can model your documentation means that your code editor can understand it too. For most editors, this means you'll get accurate, split-second documentation lookup and documentation type checking. Visual Studio Code has a superb out-of-box experience for JavaScript including documentation awareness and code completion, but other editors are supported too.

You would see similar output from a continuous integration job or the command line:

And here are those excellent docs:

Conclusion

65 miles per hour is 104.60736 kilometers per hour. Language changes the way we think, and documentation is the encyclopedia of code. Tooling that improves our abilities to understand, reason, and express ourselves through language improves our ability to engineer.

In my own personal and professional development, I've found accurate documentation to be a great treasure that gives me confidence and efficiency in the code that I read and write. Maybe we should have the same hopes and expectations of our documentation that newcomers do. Maybe with better documentation—documentation that is as accurate as we can automate—some of Wikipedia's many JavaScript errors could be identified and eliminated as easily as changing units from mph to kph. Maybe with better documentation, we could write better software, faster. Software that users love using and developers enjoy writing. Let's get to work!

Thanks to Sam Smith, Joaquín Oltra Hernández, and Leszek Manicki for reviewing and providing feedback.

One of my longstanding pet peeves is that skin development for MediaWiki is so hard. I propose a radical change to how skins are installed and ask for feedback.

Having watched teenagers use Myspace.com and then tumblr.com and watching Wikimedians build all sorts of things using wikitext templating it's clear that skinning anything should be possible with a mixture of basic knowledge of web technology (HTML,CSS maybe JSON) and/or cargo cult programming. The MediaWiki skin ecosystem is pretty sparse and when skins are created they don't tend to be published for wider consumption or are lost in github repos that are never linked to. Some never even get built. After almost 10 years in this movement it's easy to see why.

At a recent offsite I got all my team to stand up in a room and asked them to sit down if they felt comfortable with HTML. A few sat down and I told them unfortunately they couldn't build a skin. When I asked them if they felt comfortable editing CSS, a few more sat down and I told them the same thing. Eventually everyone sat down. What was interesting was who sat down and when. The designers sat down at the mention of PHP (while comfortable with CSS and JS) as did many frontend engineers. Meanwhile backend engineers sat down at the mention of PHP.

Our skin code is pretty complicated. We currently encourage skin development by guiding users to the ExampleSkin. This extension is pretty scary to many developers not already in our ecosystem and many designers who are in it. There is an extreme amount of PHP and knowledge of folder structure and MediaWiki-concepts such as ResourceLoader is needed before someone can even start.

Currently to create a skin at minimum you must

• Download and setup MediaWiki
• Learn git and clone the ExampleSkin repo
• Understand ResourceLoader
• Understand our i18n system
• Understand how skin.json works
• Edit PHP to generate HTML
• Edit CSS

To encourage a healthy skin system we need to lower many of the barriers to implementing a skin. It should be as simple as:

• Clone a repo
• Edit some CSS and HTML
• Run some npm commands

During the implementation of MobileFrontend and MinervaNeue many changes were made to the skin system to help build the new skin whilst maintain the old skin. It also intentionally made some breaking changes from traditional skin - for example no longer were languages or categories part of the HTML of an article. JavaScript and CSS shipped by all skins was turned off in preference for its own versions. In my opinion this was the skin's right. A good skinning system allows you to create radically different skins and innovate. If our skin system was healthy we'd likely have skins of all sorts of shapes and sizes. A good skin system also makes maintenance easier. Right now because of class inheritance, it's very difficult to make changes to our existing skins or our core skin PHP without worrying about breaking something elsewhere. Similar changes and challenges happened with Timeless that I'm sure Isarra can attest to!

Exploring different approaches

I've been lamenting this situation for some time. A while back I created an extension called SimpleSkins that reduced the Minerva skin to 2 files with some ambitious and breaking changes to the PHP skin code that I dreamed of one day upstreaming.

At a recent hackathon, with this idea still in mind, I took a slightly different approach. Instead of trying to make a skin a folder of several files I thought - what if the skin folder was an output of a build step? Similar to the SimpleSkin approach I again focused on a folder of frontend friendly technologies and reduced Vector to 3 files - Mustache (template), JS (with require support), LESS(css) however the generation of skin.json, PHP was left to a build script. Remarkably this worked and was relatively straightforward. One interesting insight I had this time however was that no skin developer should require a MediaWiki install to build the skin - with templates a lot of data can be stubbed or come from an API endpoint. Not having to install a MediaWiki install is a big deal!

With a good architecture a lot of our skin system can be abstracted away. A skin without qqq codes is still useful provided en.json has been generated. ResourceLoader module config is much easier to auto-generate now we have packageFiles provided we enforce a common entry point e.g. index.js and index.css/less. The PHP skin class should and can do more. Instead of having skins that extend SkinTemplate e.g. SkinVector we should have a skin rendering class that renders folders that contain skin meta data....

How we do it.

Forgetting about existing technology choices and working with what we've got, I'd propose automating most of the skin registration process - to the point that PHP is irrelevant and JS and JSON files are optional.

I strongly believe the following is possible:

• An online skin builder that saves and exports skin folders to download folder or github similar to jsfiddle
• A valid skin is a folder with at minimum 2 files - index.mustache and index.(css/less)
• You should be able to copy and paste an existing skin and get a new skin without any modification except the folder name.

To achieve such a goal we would need a SkinRenderer class that locates a skin directory and renders the template inside it (Mustache is currently the template language we support in core). SkinRenderer when passed the skin key skinnew for example would find the folder skinnew in the skins folder and the files index.less, index.js and skin.mustache. It would pass skin.mustache data (which is subject to deprecation policy and well documented and it would register a ResourceLoader module using index.less and index.js and packageFiles. qqq.json and en.json if needed could live in the i18n folder as they currently do but their absence would not cause any problems.

A developer would fork a new version of the ExampleSkin which provides the basic file components, run npm install and then npm start. This would pull our core frontend technologies - mustache and LESS from npm and then pass the skin through a tool such as parceljs that allows live updating, the workflow of which is demonstrated in this video. However unlike in my hack experiment, installing the skin would be as simple as copying that folder into mediawiki's skin folder rather than running a build step :)

What do I do next?

Am I alone in thinking it should be possible to build skins without PHP? Do you have different opinions on what the skin system should be? Do you have concerns about how such a system would scale or whether such a system would get adoption? What do you think of my skin builder tool and should I work on it more? If so I'd love to hear more from you. Any feedback you can provide would be helpful to decide whether I should prepare and push and RFC.

Thank you for your time!

The lack of tooling or support for tooling has been causing problems in complicated code bases like the codebase for our mobile site, so we carved out a proposal to create a bridge from our existing codebase to a more modern one using Webpack. I'll talk about what we did and why.

The majority of Wikipedia's front-end assets are served by a system called ResourceLoader, that has been part of the MediaWiki software since 2010. Of particular interest to us is how it packs JavaScript assets, by concatenating and compressing them. This capability predates Webpack (2012) and similar tools so this should not be considered a case of Not invented here.

This tooling has allowed developers to build front-end code without tooling. Back in 2010, this was actually the norm. If you look at similar projects which have been around for as long as ours in the open source ecosystem, you'll find Makefiles concatenating JavaScript files - artifacts of this era.

As the JavaScript ecosystem has flourished, it's becoming impossible to build JavaScript without some reliance on tools. Needless to say most front-end developers are now accustomed to working with tooling. Being unable to use tooling to build our JavaScript has arguably handicapped us a little, especially as we turn our attention to more complicated ambitious projects such as the page previews feature.

Unlike many JavaScript module systems, the ResourceLoader system we use focuses on the collection and delivery of various loosely coupled assets which it discovers from a manifest rather than a file. Currently, when writing JavaScript we need to define a module inside a special file called extension.json which is then interpreted in PHP and sent to the user which looks like so:

"mobile.toc": {
        // define which environments to run this module
"targets": [
                "mobile",
                "desktop"
        ],
        // manifest of dependencies (think require/import)
"dependencies": [
                "mobile.startup",
                "mobile.toc.images"
        ],
// script files in order they need to be concatenated
        "scripts": [
                "resources/mobile.toc/TableOfContents.js"
        ],
        // styles that should be loaded via JavaScript
"styles": [
                "resources/mobile.toc/toc.less"
        ],
        // manifest of templates to load to support JavaScript
"templates": {
                "toc.hogan": "resources/mobile.toc/toc.hogan",
                "heading.hogan":  "resources/mobile.toc/tocHeading.hogan"
        },
        // message keys needed to load to support internationalization
"messages": [
                "toc"
        ]
},

Without a Node.js module system, our developers have had to work with a client-side library similar to Require.js and manual editing of what's essentially a manifest to discover JavaScript whilst keeping a mental picture of how it all fits together and the code is split. We had to manage the JavaScript dependency trees ourselves. As my work colleague, Stephen put so nicely:

we essentially had to fill out paperwork to create a file" and now "adding a new file is as easy as right click new file".

Using Webpack to handle our JavaScript rather than our own in-house ResourceLoader has achieved several things for us:

• Webpack manages complicated dependency trees for us which avoids loading errors (e.g. code loading in the wrong order)
• gives us more control over public and private interfaces to community gadgets
• allows us to expose interfaces for testing
• encourages separation of logic into reusable modules (files) without any mental strain
• allows delegation of problems such as code splitting to tooling rather than the human mind

Making code easier to work with

Previously, adding or even renaming any source file to our repository required not only creating the file but registering it by listing it in an array in a JSON file (see developer notes at the end of this article for more). We had 86 JavaScript files which we reduced to 19 files built via Webpack from 101 source files. The increase in source files reflects the teams ability to embrace Webpack and separate code responsibilities more effortlessly.

Now that we make use of Webpack, we only have to require it via a require statement. This might seem basic stuff, but it has made a big difference.

Similarly, we've not had to worry about polluting the global namespace. Previously all our files were wrapped in an IIFE but now we've been able to remove that wrapper and also the indenting associated with it.

A familiar stack

While we didn't measure it, and it could just because we hire awesome people, we've all noticed that our new hire got up and running with our code much quicker than previous hires.

We're using tooling that is well supported and has utilized new tooling to help us write better code. We're making use of bundlesize to track the size of our JavaScript assets in the code itself (and preventing unexpected increases with our continuous integration stack). We're also making use of nyc to track code coverage (see below).

Faster unit tests

Previously, our unit tests couldn't be run cheaply on the command line in Node.js. Any npm script wanting to run them would have to boot up a browser e.g. Phantomjs and have a working MediaWiki instance. This was slow, manual and error-prone as tests from other unrelated projects could break our own tests. It was pseudo-integration testing than unit testing. Now, with a few minor changes (mock libraries that emulate "MediaWiki"), we can run these tests from the command line in headless mode using the qunit node library. We use Webpack to build a file that can be run in the old method, to avoid confusing developers in other teams who are used to running tests this way. During our refactor, 44 QUnit tests were slowly migrated to run in Node.js.

This is obviously much faster as it doesn't require a MediaWiki install and doesn't require booting up a browser. As a result, there is more motivation for engineers to write tests in the first place, in fact we've already added 15 new test files.

Code coverage

Previously, our bespoke tooling made testing coverage very tricky and manual. As a result we didn't measure. Now that we are using Webpack and headless Node.js tests, we are able to work this out. After porting a file to the new module loading system, we made sure to document the code coverage of the file. It has shown us that roughly 50% of the JavaScript code we run is covered by tests. More alarmingly, 45 of our 81 files had 0% coverage. In particular it became clear that we were avoiding writing tests where it meant exposing private interfaces on a global JavaScript variable.

As we refactor with our remaining project time we aim closer to 100% test coverage, now that our tooling makes it easier to write tests and we are now able to enforce coverage in the repository via the nyc library meaning that code coverage can only get better.

More future-proof code

While we've been forced to look at the code, we've been noticing ways to improve it and prepare it for a modern future. We've been replacing calls to jQuery with calls to a wrapper for jQuery, meaning it's becoming clearer about what we use jQuery for, and when and how we might not need it. By keeping the definition of our project around problems rather than solutions, it's been easy to justify and prioritize this work.

Similarly, we've been migrating to use ES5 functions where possible instead of jQuery.

While we're not removing jQuery from our stack just yet, we've found inspiration in other efforts to do this such as Github to at least make this a real possibility.

Versioning

Our mobile front-end relies heavily on the Hogan template library. Previously, any vendor JavaScript had to copy and pasted into the repository itself. When we started we didn't actually know what version of Hogan we were using and had to diff it with several production versions! Now that we use Webpack, we can pull Hogan directly from the npm repository, so we know exactly what we are shipping and can upgrade easily if necessary. We are exploring leaning more on our tooling with ideas to use transpiling and include template source code in JavaScript files.

Webpack didn't solve everything

While Webpack has helped us organize our JavaScript files better, it doesn't seem to solve all our problems (yet). For example. since Wikipedia supports over 200 languages, we haven't found a way Webpack can ship message strings in the scaleable way that ResourceLoader does. I'm excited about the prospect of identifying these problems and filling in those blanks where necessary, but right now we have a nice balance of the best of ResourceLoader and Webpack in our codebase.

I use OSX. Vagrant has not been kind to me, but I'm hopeful that Docker will make development a lot easier for me in future.
Until then, I use MAMP which provides a pretty easy LAMP setup. I wanted to share it with other frontend engineers as this minimal setup works well for me - it's fast, it minimises the extensions I need to update and most importantly brings me closer to problems with frontend end-users are experiencing.

MAMP replicating Wikimedia paths

In MAMP I have the web server set to apache and use a symlink to point the wiki folder to a git clone of mediawiki/core.

I setup the wiki via the web installer - which if you've never tried yourself, I urge you to give it a go! It's not that complicated really, and that's quite impressive!

I have the following defined in LocalSettings.php to match production wikis.

# this is is important as otherwise things like page previews will not work correctly when using the proxying tips I tell you below
# With this setup articles can be viewed on the path `/wiki/Main_Page`
$wgArticlePath = "/wiki/$1";

Extensions and proxying content

I am lucky to work with extensions that require minimum setup - most are simply git clone, configure and play e.g.

wfLoadExtension( 'Popups' );
wfLoadExtension( 'MobileFrontend' );
wfLoadSkin( 'MinervaNeue' );

I have no working instance of Wikidata or VisualEditor - these are black boxes to me. As a general rule I only install what I need. When I do need to test integrations with them, I seek configurations that can point to production.

For instance, the following config allows me to load production content into VisualEditor without setting up a REST base server:

wfLoadExtension( 'VisualEditor' );
// Enable by default for everybody
$wgDefaultUserOptions['visualeditor-enable'] = 1;
$wgVirtualRestConfig['modules']['parsoid'] = array(
    // URL to the Parsoid instance
    // Use port 8142 if you use the Debian package
    'url' => 'https://en.wikipedia.org',
    // Parsoid "domain", see below (optional)
    'domain' => 'en.wikipedia.org',
);
$wgVisualEditorFullRestbaseURL = 'https://en.wikipedia.org/api/rest_';

Note, that this will trigger CORs problems on your localhost, but that MediaWiki's API supports CORs for localhost for readonly requests provided you pass a query string parameter "origin=*".

The following code when placed in the right place (I usually throw this line into core or above the api request I want to proxy) gives me
content to feed from production into VisualEditor:

$.ajaxSetup({ data: {  origin: '*'  }  } );

Whenever my team needs to work with any kind of service of API, I've always found it much more useful to proxy content from production as otherwise I miss bugs and I find replication difficult. Using Special:Import is slow and broken. In particular, if you import pages linked to Wikidata, you also need to clone the Wikidata page for that article to be replicated locally.

User generated content is particularly important when working with content on mobile. We added support to MobileFrontend to proxy content and it can easily be enabled with the following config in LocalSettings.php:

$wgMFContentProviderClass = 'MobileFrontend\ContentProviders\MwApiContentProvider';
$wgMFMwApiContentProviderBaseUri = "https://en.wikipedia.org/w/api.php";

With these two changes, any pages I view in mobile will be proxied from production. This is currently available in the beta cluster - check out https://en.m.wikipedia.beta.wmflabs.org/wiki/Singapore for example. The pages 404 to avoid indexing, but will be live copies of the production equivalent.

Proxying content for desktop too!

In Popups (page previews feature), this is pretty easy to avoid installing Popups dependencies PageImages and TextExtracts I use 2 lines of config:

$wgPopupsGateway = "restbaseHTML";
$wgPopupsRestGatewayEndpoint = 'https://en.m.wikipedia.org/api/rest_v1/page/summary/';

This configures my localhost to make use of the production REST endpoint to source summaries. If I create a page "Popups test page" and create red links on a page "Dog" and create a page "Dog" with 2 lines of text, previewing the link Dog on "Popups test page" will show me the production preview for the Dog article. However, this doesn't fully work as I need links to those pages for page previews to work. Since page previews runs on desktop I can use the same proxying I use for mobile....

// Enable MobileFrontend's "content provider" for desktop too!
$wgMFAlwaysUseContentProvider = true;

Sometimes I need to make articles, so I provide an override to allow me to edit and view local pages that don't live on my production wiki:

// This will ensure that any local pages are served instead of production copies where available.
$wgMFContentProviderTryLocalContentFirst = true;

Proxying read-only APIs in mobile

Sometimes, I want to proxy JavaScript as well, which MobileFrontend also allows me to do. If I'm testing workflows with read-only (ie. no POSTs or authentication) I can proxy APIs. This is useful for testing pages like Special:Nearby without having to create lots of articles with coordinates near your current location.

// Redirect API requests to English Wikipedia
$wgMFContentProviderScriptPath = "https://en.wikipedia.org/w";

Cache!

I use memcached to cache. Without this your wiki might be a little on the slow side:

$wgMainCacheType = CACHE_MEMCACHED;
$wgMemCachedServers = array( '127.0.0.1:11211' );
$wgParserCacheType = CACHE_MEMCACHED; # optional
$wgMessageCacheType = CACHE_MEMCACHED; # optional

Conclusions

That's it.. that's my setup. All the extensions I work on tend to mirror production config as closely as they possibly can so shouldn't require any additional setup, so I urge you if you haven't... try setting up a minimal MediaWiki and see what's the minimal you can get away with to be productive.

Let me know in comments if you run into any problems or I've converted you into a more effective engineer :-).

Table of contents

These changes and workflows have empowered the team to evolve and work effectively on the project, and we think they could benefit other projects that are heavy on client side UIs. Despite a few rough corners, and conversations to be had to streamline this kind of tooling into our ecosystem, we believe it is a win.

We are very happy to discuss, evolve the current setup, and help other teams or projects adopt the same kind of tooling. We would love creating opportunities for shared unified tools around these workflows for the ecosystem.

Please reach out in the comments, in phab tasks, or by email to reading-web-team@wikimedia.org.

If you want to read more about page previews in general, here are some of the recent blog posts:

• How we designed page previews for Wikipedia — and what could be done with them in the future
• Why it took a long time to build that tiny link preview on Wikipedia

Table of contents - Versions: HTML, Markdown

Intro

For testing frontend code, MediaWiki provides a browser based QUnit setup. For running the tests, you have to spin up MediaWiki, usually through vagrant, and load Special:JavaScriptTest in your browser, which will run all the QUnit tests for all the extensions and MediaWiki itself. From then on, you can filter by module or string, hide passed tests, and a few other things like select to load the assets in production mode or development mode (debug=true).

Like any testing strategy, this setup comes with tradeoffs. Specifically, there are a couple of big problems that we have had when working on the frontend code, which we set out to address when working in Extension:Popups:

1. Tests take a long time to run
2. It is very hard to write isolated unit tests

Tests usually take a long time to run

With this setup, tests have to go through to the MediaWiki server, ResourceLoader, and then run in the browser. This is especially noticeable in development mode, which we often enable to get readable stack traces and error messages on test failures, but makes the test run take a lot longer.

There are also big startup costs, for powering up vagrant and the whole system.

As a result of this, writing tests is costlier than it should be, which discourages developers to write and run tests, and over time ends up affecting the quality of our code and QUnit test suites. It also puts significant barriers to TDD style workflows, which rely on constantly running tests and require a fast feedback cycle with the developer.

• Example test run in local vagrant on a laptop (Warning: slow and long gif)

It is very hard to write isolated unit tests

In this environment, the real MediaWiki, global state, and the browser environment are available. As a result, tests are written often as integration tests, relying on implicit behavior, modules and state being available to perform the testing.

This is not a problem by itself, integration tests are important and have very valid use cases and a place in the testing process, but this environment itself makes it extremely complicated to write isolated unit tests, because of all the global state, variables, and existing loaded code.

As a result, tests written end up coupled to implicit behavior and state, which makes them very brittle or overly verbose because of the extensive mocking, and most of them are big integration tests, which makes them slower to run. All of this adds up to the previous point, making it an even slower moving setup for writing tests, with the same outcomes.

Requirements

Given that:

• Untested code is unmaintained
• Tests that run slow are never run
• Monolithic integration tests are slow to write, read, modify, debug, and execute; isolated unit tests are the opposite
• Code that is difficult to test in isolation may be indicative of functional concerns
• Efficient tests greatly contribute to efficient development

We need a way to write tests for our frontend JavaScript that:

• Encourage and enforce isolation

• Without global state or a full MediaWiki environment running
• Start up and run the tests very fast
• Re-run tests when our source files change automatically, without having to wait for the developer to go to the browser and run the tests
• Indicate clearly when a failure occurs and where it is

Additional considerations:

• We should rely on familiar tools, at least initially to ease transition and migration of existing tests to the new setup

Solution

We discussed options, and the solution we ended up on was:

• Running the test files in Node.js

• For speed, ease of setup and running it in CI, and the isolated environment
• With QUnit, jQuery, Sinon and jsdom

• To ease migration of existing unit tests to this setup
• Because of familiarity with the tools (like Special:JavascriptTest)

You can read some more details in the architecture design record 7. Prefer running QUnit tests in Node.js.

Results

We implemented a new npm script test:node that is run in CI as part of the script test on the npm job of the extension.

Tests were slowly migrated to tests/node-qunit from tests/qunit if it was possible to make them isolated. Integration tests were kept in tests/qunit as it made sense since they used the MediaWiki environment more heavily.

We created a small wrapper around QUnit -mw-node-qunit- that we've been using, which essentially gives us a CLI tool that sets up QUnit with jsdom, jQuery, and Sinon so that it was easier to migrate the QUnit tests in.

It was quite straight forward to migrate, especially since most of the Extension:Popups tests from the refactor had been written in a TDD fashion, and were already mostly isolated.

There was a bit of figuring out because eventually some pieces of code use mw.* functions or helpers, so we created a [[https://github.com/wikimedia/mediawiki-extensions-Popups/blob/2ddf8a96d8df27d6b5e8b4dd8ef33581951db9fe/tests/node-qunit/stubs.js|stubs.js]] where we created a few stub helpers for the tests.

We also kept a couple of tests as integration tests in tests/qunit, but eventually we did some work to refactor the code and made unit tests for the new code, so we got rid of the integration tests in MediaWiki entirely.

With this setup, tests run quite fast, and it is feasible (and we do) run them in watch mode while developing, giving you fast feedback and running your code on save, all from the terminal/editor.

The environment doesn't have any global state, or implicit knowledge of MediaWiki, which forces us to write properly isolated tests that don't rely on implicit behavior or state.

Finally, the move to a Node.js-based toolchain means we are easily able to leverage other great open source tools without much fuss, for example, for code coverage. We added another script -[[https://github.com/wikimedia/mediawiki-extensions-Popups/blob/2ddf8a96d8df27d6b5e8b4dd8ef33581951db9fe/package.json#L12|coverage]]- which just run the test command, but with the code coverage tool [[https://istanbul.js.org/|istanbul]] first, and just like that we got back coverage reports for the frontend code.

We recommend this approach for others wanting to improve how they test, and would be happy to help you figure out if this approach would work for you. For example, you can use this CLI runner, even if your JS sources just use globals instead of common.js or ES modules.

Problems

Overall, the move has gone great and we don't many issues to report.

When migrating existing tests, it is sometimes a bit tricky to figure out how to move them to the isolated environment, since most of the MediaWiki JS library is not available as an npm package, so in some occasions we had to restructure the code a bit to not implicitly assume as much of MediaWiki being available, and other times we had to set up some stubs for the tests to run well. This had the added benefit that the dependency on MediaWiki core libraries is explicit in the tests, so we should notice when adding new dependencies or changing them because of the failing tests, keeping the behavior and dependencies explicit.

Another extra thing that we have been doing has been maintaining [[https://github.com/joakin/mw-node-qunit/|mw-node-qunit]], which has taken a bit of additional time. Making sure our wrapper works well with qunitjs, and updating the dependency versions to not fall behind and leverage improvements on the libraries.

We will also be looking into moving the repository to the Wikimedia organization in GitHub if other teams or projects adopt this testing strategy.

Conclusions

This change has worked really well for us. We are able to run our tests really fast, even without vagrant running. The environment is isolated and really good for unit testing. The CLI wrapper had the specific helpers to ease migration from the existing tests, so it was fairly painless to switch.

Because of all of these, the extension has excellent code coverage, developers have an easier time contributing tests, and doing TDD is feasible. There is less uncertainty when refactoring and adding features, and the codebase is easy to work with. A big part of it is because of the unit testing story.

We're looking forward to adopting the same approach in other repositories and helping others do the same.

• Table of contents
• Previous post: Better minification for the frontend sources
• Next post: Conclusions

Table of contents - Versions: HTML, Markdown

Intro

MediaWiki via ResourceLoader uses JavaScriptMinifier to minimize JavaScript files so that their size is as small as possible.

Since the minification happens in the PHP server at runtime, even with a read through cache tradeoffs were made so that the server could minify in a performant way, giving up size gains for speed.

The JavaScript ecosystem has continued evolving minifiers based on Node.js tooling that can't be used on a PHP server easily.

There are gains to be had if we were to use node based minifiers and just give the smallest possible bundle to ResourceLoader for serving.

Requirements

Payload served can to users (specifically, JavaScript files), should be as small as possible.

Solution

Given we have introduced a build step previously, the solution seemed pretty straightforward.

We discussed we would introduce a minification step as part of the build process, and then the committed assets in /resources/dist would be minified and as small as possible.

We first did research and got numbers to check if there would actually be any gains and if it would be worth it. You can read about it here:

• Architecture design record 8. Enable minification of bundle with UglifyJS
• mediawiki.org: Extension:Popups/Minifying assets with uglifyjs

Then researched how to best introduce the minifier. As a standalone CLI we could run it in our now single index.js file, but if we leveraged other bundler features like multiple entry points or code splitting then the minification commands could become very complex.

As such, we chose to integrate the minifier as a plugin to the webpack bundler. See uglifyjs-webpack-plugin.

Results

We added UglifyJS minification via Webpack (config).

Minifying via UglifyJS brought the bundle size down from the previous version ~40%, gzipped ~25%.

Also, theoretically, with EcmaScript modules webpack with uglify can perform tree shaking of unused code. Webpack will mark unused exports which would then be removed in the production bundle by uglify. See the guide Tree shaking.

Problems

Initially we had to do some research to instruct ResourceLoader to not apply minification to this already minified files. What we wanted was to avoid it so that the source maps comment would be preserved, and then we would have source maps on the production version of the code.

In the end, we had to give up, and ended up removing the banner as it interfered with the minification of other modules in production, but we still have source maps in development.

Conclusions

This was a pretty straight forward addition that brought us benefits with little cost. It was enabled by the change to introduce a build step.

The gain in size is not super significant given how small the JS code base is, but if applied to bigger code bases we could get great improvements for free.

• Table of contents
• Previous post: Automatic JavaScript file bundling and library consumption
• Next post: Fast and isolated JS unit tests

Table of contents - Versions: HTML - Markdown

Intro

With MediaWiki ResourceLoader JavaScript files are run without a module system, so files have to export properties to and consume properties from the global scope, defined by other files.

This has caused issues in the development workflows related to the project's JavaScript sources, and the use of JavaScript libraries from the OSS ecosystem. Organizing the code, trying to keep it maintainable, and understanding how it is structured has a big cost on the project's quality, development and maintenance.

Application JavaScript sources

The order in which these files needs to be loaded so that all dependencies are set properly then needs to be [[https://github.com/wikimedia/mediawiki-extensions-Popups/blob/398ffb0e435f61133f6478f306ef266e147c9dea/extension.json#L75-L112|manually specified in extension.json]] by file name. As such, there are two sources of truth:

• The configuration that specifies some order of files,
• And the source code that implicitly uses files in some order.

The order in which parts your program is interpreted is defined outside of the program itself. At the file level, your file must be written with foreknowledge about which files have been loaded, which is defined elsewhere.

If you are doing anything mildly complex, you will end up with a big list of JavaScript files that depend on each other. Having to manually understand that dependency graph based on the list of dependencies on a JSON file and how the files use, define, or re-define global variables from or for other files, and managing a module's internal state, can be quite of a headache.

Changes to the source code, moving lines in the same file, refactoring code to other and new files, adding new code that uses other files, removing code, ...

All of these become really hard really quickly. In the end, organizing your code, trying to keep it maintainable, and understanding how it is structured have a very big cost on the project's quality, development and maintenance.

Figuring out if the files are specified in order properly in the configuration after and while you make changes has a high potential for obscure runtime bugs by forgetting or not seeing implicit dependencies in the manual order of files.

Libraries

In order to consume libraries for the front-end code, right now we rely on manually (or by script) pulling down npm dependencies or files from a website and including them in the repository. This is hard to verify, keep up to date, and cumbersome. If necessary, it should be possible to specify dependencies with the versions and have them be automatically included in the assets we serve from a trustworthy source. It should be easy to check if the libraries are outdated, and update them.

Requirements

For developing JavaScript files for the front-end:

• There must be a single source of truth for the dependency resolution
• Files should be authored in a standard module system instead of relying in the global namespace
• Moving code and files should be low cost and not trigger runtime errors in production if the dependencies are not properly specified
• If possible, it should be easier to tap into npm libraries for our front-end needs, and check for updates (see related discussion)

Solution

We considered our options, and after discussion among the engineers, we decided that:

• We would use a Node.js based file bundler (webpack), in order to:

• automatically bundle our JS sources with a single source of truth for intra-module dependencies and asset load order (the source code itself via import/export statements)
• use a standard module system (we started with common.js and afterwards migrated to ES modules when finalized)
• validate the dependency graph and asset building before going to production (webpack statically analyzes the asset tree and fails on build)
• We would introduce a build step to compile sources into production worthy assets
• We would use ResourceLoader as the way to serve the JS bundle(s) and to specify other dependencies (images, i18n messages, styles)

You can read some more details in the architecture design record 4. Use webpack that we wrote when we discussed this in the team.

Why webpack and not <my-favorite-tool>?

We looked at the ecosystem of bundlers at the time, and this is a summary of our evaluation:

• Browserify: Great tool, shrinking community and support in favor of others
• Rollup: Great tool, specialized in bundling code for producing libraries, not applications
• Webpack: Great tool, community, maintainers, financial support, mindshare and ecosystem of tools

We are not married to, or using any webpack specific features, so we can migrate from this tool in the future if it becomes a problem.

Results

• We introduced webpack (config) and build scripts
• The JS files are located in src/ and use EcmaScript modules (example)
• The production assets get built into /resources/dist to be served by ResourceLoader
• We added a CI step in the npm-test job that checks built assets have been committed and up to date (check-built-assets)
• Added a precommit hook that automatically runs the build step and adds the built assets (precommit)

These are some of the benefits we have seen after the introduction of these changes:

• ResourceModules configuration reduced for JS files, file order is automatic now, source code is the single truth for dependency order

• Before, and after (no manual ordering of files)
• Decreased cost and burden for splitting modules, reusing code, adding new code since file order is automatically sorted out
• Source maps in development that point to the original modules in src/ thanks to webpack
• Anecdotally, faster ResourceLoader performance by offloading file parsing and bundling to the Node.js CLI tool watcher (npm start)

• The simpler the dependency tree, the easier it is for the RL to decide if a module has been invalidated and less processing it needs to do on server and client
• We bundle redux and redux-thunk from npm, based on the pinned versions from package.json

• npm outdated will show if there are new versions of the libraries
• npm update redux will update the library, and be bundled in our sources when we npm run build
• The libraries passed security review, the hash is in package-lock.json and the build is verified independently in CI by a jenkins job from the npm version
• By using standard module systems, it enables us to use our sources in Node.js, which unlocks the possibility to run unit tests for the frontend code in Node.js for faster test runs and feedback loop for developers
• Easy introspection of the frontend code with tools like source-map-explorer and webpack-bundle-analyzer

Problems

The approach is a bit controversial since MediaWiki extensions usually don't have build steps, so there was no easy setup for CI and the deployment pipeline to run a build step before running jobs or deploying.

As such, we ended up committing the built sources to the repository, which is fine with the CI step and the pre-commit hook mentioned before, but has an annoying inconvenience: every time a patch is merged with the corresponding generated asset (resources/dist/*), any pending patches on Gerrit that also need to regenerate the asset (because they touch sources in src/), will now be on merge conflict with master.

We have discussed in phabricator and in wikitech-l:

• Wikitech-l: How does a build process look like for a mediawiki extension repository?
• T158980: Generate compiled assets from continuous integration

But sadly didn't get to any concrete steps. If you think you can help with this issue we would really appreciate your help, as we would like to help other projects, people and teams be able to use build steps in their extensions.

Right now, we sidestep the issues with a bot we have configured for the repository, that responds to the command rebase (parallel to the recheck command that jenkins-bot responds to. On comment, the bot will download the patch, rebase it with master, run the build step, and submit a new patch without the conflict.

As an interim solution it works and allows us to move along, but if we want to adopt this process for other projects we would really like to have a more streamlined solution.

Conclusions

This change has worked very well for us, decreasing the cognitive load and allowing us to work more effectively on our JS files. We recommend it if you have many JS files or ResourceLoader modules, and the order and dependencies are causing you headaches.

We hope to work together in standardizing some sort of CI + deploy process so that projects on the MediaWiki ecosystem can leverage build steps to improve their workflows and leverage powerful tools.

• Table of contents
• Next post: Better minification for the frontend sources

Extension:Popups is a MediaWiki extension that shows previews when hovering a link in a popup.

Extra requirements and a desire to find better ways to code for the frontend stack led to a series of interesting decisions in terms of tooling that we think could benefit other projects in the MediaWiki ecosystem.

In this series of posts we will explain the different technical decisions and choices in technology and tooling for the front-end part of the extension. We will provide reasoning, explanations, pros and cons, and our conclusions.

Table of contents:

1. Automatic JavaScript file bundling and library consumption (HTML, Markdown)
2. Better minification for frontend sources (HTML, Markdown)
3. Fast and isolated JS unit tests (HTML, Markdown)
4. Conclusions

The Popups MediaWiki extension previously used HTML UI templates inflated by the mustache.js template system. This provided good readability but added an 8.1 KiB dependency* for functionality that was only used in a few places. We replaced Mustache with ES6 syntax without changing existing device support or readability and now ship 7.8 KiB less of minified uncompressed assets to desktop views where Popups was the only consumer.

Background

Given that ES6 template literals provided similar readability** and are part of JavaScript itself, we considered this to be a favorable and sustainable alternative to Mustache templates. Additionally, although the usage of template strings require transpilation, adding support enabled other ES6 syntaxes to be used, such as let / const, arrow functions, and destructuring, all of which Extension:Popups now leverages in many areas.

We compared the sizes before and after transpiling templates and they proved favorable:

Where “index.js (gzip)” is the minified gzipped size of the resources/dist/index.js Webpack build product as reported by bundlesize, “index.js” is the minified uncompressed size of the same bundle as reported by source-map-explorer and Webpack performance hints, and the remaining columns are the sum of minified uncompressed assets for each relevant module as reported by [[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/resources/src/mediawiki/mediawiki.inspect.js | mw.loader.inspect() ]] with the last column being a total of these inspect() modules.

The conclusions to draw from this table are that transpiling templates does minimally increase the size of the Webpack bundle but that the overhead is less than that of the mustache.js dependency so the overall effect is a size improvement. Additionally, note that the transpiled bundle now encompasses the HTML templates which source-map-explorer reports as contributing a 2.53 KiB minified uncompressed portion of the 35.15 KiB bundle. (Previously, templates were part of ext.popups.main but only via ResourceLoader aggregation; now templates are part of index.js.) Allowing for rounding errors and inlining, this brings the approximate overhead of transpilation itself to nearly zero, 35.15 KiB - 32.88 KiB - 2.53 KiB ≈ 0, which suggests transpiling as a viable solution for improving code elsewhere that must be written in modern form without compromising on compatibility or performance.

We used the Babel transpiler with babel-preset-env to translate only the necessary JavaScript from ES6 to ES5 for grade A browsers. The overhead for this functionality may be nonzero in some cases but is expected to diminish in time and always be less than the size of the mustache.js dependency. Please note that while most ES6 syntaxes are supported, the transpiler does not provide polyfills for new APIs (e.g., Array.prototype.includes()) unless configured to do so via babel-polyfill. As polyfills add more overhead and are related but independent of syntax, API changes were not considered in this refactoring.

Manual HTML escaping of template parameters was a necessary part of this change. This functionality is built into the double-curly brace syntax of mustache.js but is now performed using [[ https://www.mediawiki.org/wiki/ResourceLoader/Core_modules#mediaWiki.html | mw.html.escape() ]]. These calls are a blemish on the code but appear only in the templates themselves and would be replaced transparently in a UI library with declarative rendering (such as Preact). We also anticipate that the template literal syntax would transition neatly to such a library. We don't know that Extension:Popups will ever want to use a UI library and accept these shortcomings may always exist.

*As reported by [[ https://www.mediawiki.org/wiki/ResourceLoader/Core_modules#mw.loader.inspect | mw.loader.inspect() ]] on March 22nd, 2018.
**The Mustache version of previews:

<div class="mwe-popups" role="tooltip" aria-hidden>
  <div class="mwe-popups-container">
    {{#hasThumbnail}}
    <a href="{{url}}" class="mwe-popups-discreet"></a>
    {{/hasThumbnail}}
    <a dir="{{languageDirection}}" lang="{{languageCode}}" class="mwe-popups-extract" href="{{url}}"></a>
    <footer>
      <a class="mwe-popups-settings-icon mw-ui-icon mw-ui-icon-element mw-ui-icon-popups-settings"></a>
    </footer>
  </div>
</div>

The ES6 version of the same template explicates dependencies but must manually escape them. The HTML snippet is quite similar but a call trim() is made so that parsing the result only creates a single text Node.

/**
 * @param {ext.popups.PreviewModel} model
 * @param {boolean} hasThumbnail
 * @return {string} HTML string.
 */
export function renderPagePreview(
	{ url, languageCode, languageDirection }, hasThumbnail
) {
	return `
		<div class='mwe-popups' role='tooltip' aria-hidden>
			<div class='mwe-popups-container'>
				${hasThumbnail ? `<a href='${url}' class='mwe-popups-discreet'></a>` : ''}
				<a dir='${languageDirection}' lang='${languageCode}' class='mwe-popups-extract' href='${url}'></a>
				<footer>
					<a class='mwe-popups-settings-icon mw-ui-icon mw-ui-icon-element mw-ui-icon-popups-settings'></a>
				</footer>
			</div>
		</div>
	`.trim();
}

The Reading Web team recently discovered a bug in Firefox wherein a load event is fired when Firefox loads certain pages from its Back-Forward Cache (BFCache). To JavaScript on those pages, this event is a second load event (the first having been fired before the user navigated away from the page). This proved to be problematic for the cornerstone of our instrumentation, the EventLogging extension and delayed the deployment of Page Previews for approximately three months.

Background

The Page Previews instrumentation revolves around the notion of a link interaction. Every time the user hovers over a link with their mouse or focuses on a link with their keyboard, a new interaction begins. Every link interaction has a unique identifier (herein, a “token”).

The token is a 64-bit integer in hexadecimal format that’s generated using crypto#getRandomValues(), which should use a “well-established cryptographic PRNG seeded with high-quality entropy”. If this is the case, then the probability of token collision should approach 50% when over 4 billion tokens are generated.

On Monday, 27th March 2017, @Tbayer reported an unusually high number of events with duplicate tokens ("duplicate events") being inserted into the EventLogging MySQL table. Naturally, we assumed that this was being caused by one or more bugs in the instrumentation. During the following two sprints we tracked down and fixed what we thought was all of them. We even went so far as to instrument the instrumentation so that we could be confident that the fixes that were deployed weren’t causing more duplicate events to be logged. However, while our instrumentation was assuring us that Page Previews wasn't generating duplicate events, the number of events with duplicate tokens wasn't affected.

While we were tracking down and fixing bugs, @Tbayer investigated further and reported that Firefox v51 and v52 were sending circa 98% of the duplicate events. He also noted that the distribution of OS's that were sending duplicate events didn't deviate from the general distribution for pageviews. The Readers Web engineers tried to consistently reproduce the issue in Firefox but to no avail.

The Discovery

@Jdlrobson's aha! moment was when he noticed that a lot of the duplicate events were being logged on pages with dense clusters of links, which the user might be clicking accidentally and immediately navigating back. Immediately after trying this, he saw duplicate events being logged by Firefox v54 and raised a thorough – and startling – bug report.

Under certain conditions, Firefox will serialize an entire page, including the state of the JavaScript VM, to memory in order to make navigating backward and forward between pages very fast. This feature is often referred to as the "Back-Forward Cache" (the BFCache). @Jdlrobson discovered that when Firefox loads a page from the BFCache, the load event fired. From the point of view of the JavaScript on that page, however, this is a second load event, the first being fired before the page was serialized prior to navigation.

The EventLogging protocol subscribes to the event.* topic after the document and its sub-resources have loaded so that logging events doesn't consume resources before or during rendering the page. So, when Firefox resumed a page from the BFCache the subscriber was re-subscribed due to the second load event, leading to logging of exact duplicate events.

The strict conditions that the BFCache requires and the user having to navigate backward or forward to a serialized page to trigger the issue very neatly explain the wild peaks and troughs in the daily % of Popups events with duplicate tokens that we were seeing.

Workaround

Both @Jdlrobson and I suggested a workaround.

While I was trying to reproduce and understand how the bug was affecting the EventLogging codebase, I noticed that the DOMContentLoaded event was behaving correctly and only firing once. I suggested that EventLogging register the subscriber when the latter event fires. However, the subscriber is meant to be registered as late as possible so as not to impact page load time on resource constrained devices.

@Jdlrobson's suggestion, on the other hand, was delightfully simple: make EventLogging register the subscriber exactly once. The workaround itself was a 1 character change, which, as they often do, required a much larger comment to provide much-needed context.

Just under an hour after the change was deployed, the number of duplicate events per day dropped from between 10-30% to roughly 0.08%. This new rate is consistent with the background noise levels of duplication in our other much simpler instrumentation, e.g. ReadingDepth and RelatedArticles.

What We Learned

This issue took us a little over three months to track down and fix so we had /a lot/ of time to reflect on what we could've done better.

How Are We Doing?

The Page Previews instrumentation is complex. The complexity of the instrumentation is proportional to the number of general questions we're asking about the feature. This complexity meant that implementing, testing, and QA'ing the instrumentation all took considerable time. Since our initial hypothesis was that the issue(s) were in the instrumentation itself, we also spent comparable amounts of time trying to re-verify that the instrumentation was working correctly.

Were we to answer only one or two questions at a time, i.e. collect less data, we may have saved ourselves some time as it would've been simpler to test our hypothesis. As software engineers, we regularly sacrifice velocity for confidence in our implementation; I don't see how this is any different.

QA Without A Test Plan

The Readers Web kanban board has a Needs QA column. For better or worse, technical tasks like implementing the instrumentation, tend to skip this column as QA tends to be done at the browser level. Moreover, the Readers Web engineers didn't set an expectation that QA would be done as part of code review and if it was, then there was no test plan created before or after merging the code.

This situation has since greatly improved as a response to a variety of problems. We've agreed that all tasks should move from the Needs Code Review column to Needs QA after all of the associated changes have been merged into the codebase – if a task is exceptional, then it must be documented. We've also agreed that before a task can be moved into the Needs QA column it must have a test plan in its description.

We've yet to talk about setting expectations around planning, executing, and documenting QA as part of code review as a team. When we do, I'm sure that we'll write about it.

Integration Testing

One of the goals of the Page Previews architecture and a driving force behind the design of new changes is testability. The extension is remarkably well covered by its unit tests.

The instrumentation is no exception and is 100% covered by unit tests. However, the focus of the unit tests was and still is the correctness of the properties of the events produced by the system, which is a result of misrepresenting events as POJO's and not value objects or types. So while the vanity metric of coverage is maximized, we lack proof that key invariants are holding.

Focussing on higher level integration tests, fuzzing, and mutation testing to prove the instrumentation correct would have allowed us to immediately decline our initial hypothesis that the bug was in the instrumentation. On the other hand, the suite of unit tests will give us confidence when refactoring the system to allow for these changes.

Notes

1. The title of this post comes from both the Beacon API, which is used to log events for the Page Previews instrumentation, and Cloudkicker’s Beacons album, which is pretty darn rad.
2. The Discovery, another Cloudkicker album, is also equally rad 🤘