At Fosdem 2025 I had the opportunity to arrange the Web Performance developer room together with Dave, Julien and Nazim from Mozilla.

FOSDEM is a completely free and open conference held annually in Brussels, Belgium, dedicated to promoting free and open-source software and knowledge sharing. One thing I particularly like about FOSDEM is that all speakers volunteer their time and expertise, making it truly community-driven. All sessions were recorded, so I'm excited to share the videos here for those who couldn't attend.

Our developer room was fully packed during several sessions, it was great to see that people still care about web performance.

Here are the talks from our Web Performance room this year:

• How browsers REALLY load Web pages - Robin Marx (video)
• Making Sense of the Long Animation Frames (LoAF) API - Andy Davies (video)
• Scheduling HTTP streams - Alexander Krizhanovsky (video)
• Chromium on Android: How we doubled Speedometer & developed the LoadLine benchmark - Eric Seckler, Gurj Bahia (video)
• Collaborate using the Firefox Profiler - Nazım Can Altınova (video)

Enjoy watching the talks!

In late 2023, an article claimed that Wikipedia is one of the fastest websites in the USA. Flattering, right? I've been measuring web performance for over a decade, I couldn't help but wonder: How did they measure that? How do you know that Wikipedia is one of the fastest websites? The article does not say anything on how they did measure it.

I went to the Web Performance Slack channel (yes, there's a dedicated place where web performance geeks hang out). I asked the question:

“Has anyone seen the data or the actual “study” done by DigitalSilk about the fastest loading US websites? https://www.technewsworld.com/story/craigslist-wikipedia-zillow-top-list-of-fastest-us-websites-178713.html - I can only find references to it and a screenshot, nothing else?”

Not providing references? That's not Wikipedia! We're all about citations and verifiable sources. No one on the Slack channel knew anything about how the test was run. But then, one of the channel members took action: Stoyan Stefanov emailed the journalist and actually got an answer!

Methodology
The most visited websites based on web traffic were ran through Google's PageSpeed Insights tool, to find out how long it takes for each site to load in full on average“

So, while it's flattering to see Wikipedia crowned as one of the fastest websites based on Google's PageSpeed Insights tool, I couldn't help but feel a tricked. They seemed to rely on the onload metric. That's a metric that, in the web performance world was regarded as old and not correlating to user experience since 2013.

Understanding the limitations using the onload metric, let’s shift our focus to modern metrics that better reflect real-world user experiences: Google Web Vitals

Google Web Vitals

Google Web Vitals is Google's initiative to focus on the metrics that matter to users and also affects Googles core ranking system. Unlike the old-school onload time, Google Web Vitals better measure aspects of real world user experience.

The core metrics at the moment are three metrics:

• Large Contentful Paint (LCP) - when in time is the largest element painted on the users screen. For Wikipedia that is very often a paragraph, but sometimes its an image element or a heading.
• Interaction To Next Paint (INP) - measure the responsiveness of page, meaning that a page responds quickly to user interactions. For Wikipedia the responsiveness can be slow depending on the amount of JavaScript we ship, event listeners or click events.
• Cumulative Layout Shift (CLS) - measure the visual stability of the page. That means it measures if content is moved around. In the Wikipedia case this means when campaigns runs and we move the content of Wikipedia.

Google also have two other web vitals, two metrics that are important for the user experience but not listed as core:

• Time To First Byte (TTFB) - measure the time between the request for a resource and when the first byte of a response begins to arrive. For Wikipedia TTFB depends on where the users are in the world and how far it is to the closest data center.
• First Contentful Paint (FCP) - measure the time from when the user first navigated to the page to when any part of the page's text/images painted on the screen. For Wikipedia this is often text.

Google pay special attention to the 75th percentile of those metrics. It was chosen because "First, the percentile should ensure that a majority of visits to a page or site experienced the target level of performance. Second, the value at the chosen percentile shouldn't be overly impacted by outliers.". But what does the 75 percentile mean for us at Wikipedia?

The 75th percentile at Wikipedia

Now let’s put the 75th percentile into perspective by applying it to Wikipedia’s vast global audience.

Imagine that there were 100 people visiting Wikipedia. Each person got a different user experience because of their device, the internet connection and how we build Wikipedia. For some users the experience will be really fast, for some it will be slower.

The 75th percentile focus on the worst experience of the best 75%. If you take all 100 users and then sort the experience from fastest to slowest, the 75th experience is where you draw the line. This means that 75% of the users had a better or equal experience and 25% had a worse one. So, how many users are in that 25% for us? We measure unique devices and not users so lets use that.

Well, for Wikipedia, those 100 users are actually 1,5 billion unique devices per month and 24 billion page views.

That means if we look at the 75 percentile and we see that a metric move we know that at least 6 billion page views per month ( 24 billion × 0.25) is affected. And 375 million unique devices (1.5 billion × 0.25).

That is many devices. Suppose we have a regression of just 100 milliseconds in the 75th percentile. That is at least 375 million devices are experiencing this delay. Collectively, those users are waiting an extra 434 days. Yes, over a year of extra wait time for the users with the worst experience because of a (tiny) 100 ms change.

Is the English Wikipedia the fastest website in the USA according to Google Web Vitals?

With the metrics Google collects from different web sites, you can compare different sites with each other! The metrics are available per domain (not user country), so we can not compare if the English Wikipedia is one of the fastest web sites in the USA, but we can compare the English Wikipedia against other web entities with users all around the world.

However before we do that, I want to point out that "Is the English Wikipedia fastest website in USA according to Google Web Vitals?" is a very exclusionary question to ask since:

• The English Wikipedia is used in more places than the USA
• There are many Wikipedias for other languages out there and we should not only focus on the English Wikipedia. We need to make sure that everyone independent of language has the same user experience.

Looking just at "Are we fast in the USA" we leave out a big part of the world. So today we gonna look at the English Wikipedia compared to other web sites and then also look at Wikipedias all around the world to see what kind of user experience all users have.

But first let's talk about how Google also categorises these experiences as good, needs improvement, or poor by setting specific limits for each metric. With Googles definitions we can see how many of our users have different kinds of experiences. In the data I will show, green means good, needs improvement yellow and red means bad/poor experience.

We collect all data that is available through the Chrome User Experience API and you can see that in our Chrome User Experience dashboard. There's a lot of metrics, so I will focus on just the Largest Contentful Paint today.

First let's look at the actual 75 percentile Largest Contentful Paint. We compare against a couple of other web sites. Lower numbers are better. Green is good. We will start to look at the numbers for mobile.

Mobile

This graph highlights that Wikipedia's mobile LCP performance is nearly as fast as Google's, which is quite remarkable!

We can also look at how many of our users have a slow/bad experience.

Wow we can see that we have less users in percentage with a bad experience than the rest of the sites. However the graph shows a small percentage of mobile users experiencing suboptimal LCP. For a website of Wikipedia's scale, this small percentage translates into millions of users, we need to be even better!

I wonder if it's the same for desktop users? Lets look at the 75 percentile again.

Desktop

Again we can see that Wikipedia is almost the fastest, outperforming many major websites! We seem to be fast on both mobile and desktop.

Yes we are really fast! Can we open the champagne and celebrate?

Are we the fastest site known to human kind?

Well I would take it a little easy here before we start to brag. Do you remember how we calculated how many users are left out when we use the 75 percentile? I would be careful with a web site with so many users. I would say that: "The English Wikipedia is really fast compared to other web pages looking at the Largest Contentful Paint at the 75% percentile for Chrome users that Google collects metrics from".

Another way of looking at the data we get from Google is to see how many users have a bad experience using Wikipedia. By taking the ones that need improvement and poor experience, we can see how many users in percentage we need to move to having a good experience.

First let's look at Largest Contentful Paint again for desktop users. This time we look at the number of users in percentage that have a non good experience per wiki.

And then we look at the same for mobile.

We can see that on desktop and mobile we have Wikipedias where we as developers have work to do to give more users a good experience.

As a last example I want to share the interaction to next paint data for mobile. This is interesting because here JavaScript comes into play and there are many things we can do on our side to give the user a better experience.

We see that for almost every Wikipedia, 5% of the users have a not so good user experience.

Summary

Wikipedia's performance story is one of scale and precision. By focusing on Google Web Vitals, we've seen how milliseconds of delay can impact millions of users. Metrics like Largest Contentful Paint (LCP) and Interaction to Next Paint (INP) can provide valuable insights into real-world user experiences, guiding us to optimize for both mobile and desktop users.

With billions of page views monthly, even the smallest regressions in performance ripple across the globe. Yet, Wikipedia stands as a benchmark of speed in the US, rivaling even the likes of Google. This achievement underscores the importance of continuous monitoring, fine-tuning, and maintaining a user-first perspective in web development.

As we celebrate, we also need to acknowledge the challenges. Moving the needle for those users with "non-good" experiences remains our mission. By using data and ongoing analysis, we can ensure that Wikipedia stays fast, accessible, and enjoyable for everyone, everywhere.

Prehistory

It's hard to believe it, but there were Selenium tests in Wikimedia repositories even before I joined the Wikimedia Foundation. (12 years ago, in 2012.) In MediaWiki Core, Selenium tests (in PHP) existed from 2010 until 2013.

When I started working for the Foundation, Selenium tests (in Ruby) existed in the qa/browsertests repository. The repository was active from 2012 until 2014. Tests from the qa/browsertests repository moved to the MediaWiki Core repository in 2014. They lived in tests/browser folder until 2017. (T139740)

History

In 2017 we decided to rewrite Selenium tests in Javascript.

Since then we had a few major framework updates:

• Port Selenium tests from Ruby to Node.js (2016 - 2023) (T139740)
• Evaluate WebdriverIO replacements for our browser automation framework (2020) (T247835)
• Upgrade WebdriverIO to v6 in all repositories (2020-2021) (T247844)
• Evaluate Microsoft Playwright as replacement for our browser automation (2020-2021) (T262682)
• Upgrade WebdriverIO to v7 in all repositories (2021-2023) (T274579)
• Switch to spec reporter (2021-2024) (T280331)
• Refactor WebdriverIO tests from sync to async mode (2020-2024) (T256626)

Async

The sync to async migration is just one of the major updates to the framework. But, it's an important one.

In 2021 WebdriverIO announced they are deprecating the sync framework. In 2020 I created an epic task (T256626) to track the migration in more than thirty repositories. (I'm not sure why I created the task before the announcement. Maybe I knew the change was coming in advance.) I had resolved the task in 2024. It took us a while.

The migration project started with an internship (2021-2022). During the internship we did the majority of the migration. The internship lasted only a few months. We tried to migrate as many repositories as possible. We managed to migrate repositories that didn't give us a lot of trouble. The few remaining repositories were hard to migrate.

This monster Gerrit search finds most commits related to this project. There are 92 commits. 58 merged, 34 abandoned.

We started strong in 2020, but we were not able to finish the migration for several repositories. The main problem was technical. We could not set up a working development environment for some repositories (T344682).

At the Wikimedia Hackathon 2024 I talked with several people from Wikimedia Deutschland about the async project. (At least with @Lucas_Werkmeister_WMDE and @WMDE-Fisch.) After looking at the tracking Phabricator task (T256626), they said their repositories are already migrated to async, but nobody closed the subtasks.

Documentation

After closing relevant subtasks, the only thing left to do was to update the documentation (T364268). All of our documentation was using sync syntax. After the hackathon I started updating the documentation.

After looking at the documentation I realized that there are several pages with a lot of code on them. All of the code was now broken. If we update the code now, it will become broken the next time we make a change to our framework. Nobody would know that the sample code on a wiki page was broken. That is, until somebody tried to use it.

To solve the problem of broken code on wiki pages, we have decided to move all sample code to MediaWiki Core (T368535). That way, we would be reminded (actually, forced) to update the sample code at the same time we update the framework. If we update the framework and not the sample code, our continuous integration would break.

I had a lot of help from people on my team. We have an office hours meeting almost every day. I was working on the documentation during the day and asking for feedback during the meeting. The process worked great.

When all the code was moved from the wiki pages to MediaWiki Core and updated to use async syntax, we copy/pasted the code back to the wiki pages. That's the last remaining step that is still manual. Nobody on my team knew how to embed a file in a wiki page. If you know how to do it, please let me know.

Updating the documentation finished the async migration project.

Languages and Tools

While doing some online archaeology for this blog post, I stumbled upon Selenium tests in Wikimedia projects written in PHP (search), Python (search) and Ruby (search).

Also, Selenium is not the only tool we use for end-to-end testing. There's Cypress (search), Puppeteer (search), Playwright (search) and Nightwatch (search).

Future

We're currently working on a few projects that will further improve our Selenium testing framework. There are plenty of future improvements we plan to do. See Selenium Framework Updates Timeline section at the end of the article for more information.

A notable project is T344682: Run Selenium tests targeting available local and remote testing environments. We need testing environments for more than forty repositories. Unfortunately, that's not a solved problem. There are many options, but I'm not sure that any of them would work with all repositories. A good solution would be to find at least one development environment where a particular repository would work. If that means we have to use multiple environments to get all repositories working locally, I'm fine with that. I'm not happy with that, but even that would be an improvement over the current situation. For some repositories we don't have a working local development environment at all.

Async Migration Timeline

• Core (2021-2022) (T293071)
• AbuseFilter (2022-2023) (T300790)
• AdvancedSearch (2022-2023) (T300791)
• CentralNotice (2022) (T300212)
• CheckUser (2022) (T321904)
• CirrusSearch (2022-2023) (T300793)
• Cite (2022-2023) (T300196)
• CodeMirror (2022-2023) (T300205)
• ContactPage (2023) (T334094)
• Echo (2021-2023) (T293073)
• ElectronPdfService (2022) (T299684)
• FileImporter (2022-2023) (T300535)
• GlobalWatchlist (2021-2023) (T293074)
• GrowthExperiments (2022) (T299102)
• Math (2021-2022) (T293076)
• Newsletter (2021-2022) (T293078)
• Popups (2021-2023) (T293072)
• ProofreadPage (2021-2023) (T293079)
• RelatedArticles (2022) (T300797)
• RevisionSlider (2022-2023) (T300798)
• TemplateWizard (2022-2023) (T300208)
• TwoColConflict (2022-2023) (T300800)
• VisualEditor (2022) (T299756)
• Wikibase (2021-) (T293080, T343886)
• WikibaseLexeme (2021-2024) (T293081)
• WikiLambda (2021-2022) (T293082)
• MinervaNeue (2021-2023) (T293084)
• Termbox (2022-2023) (T300807)
• Wikibase Query Service GUI (2022-2024) (T300808)
• wdio-wikibase (2022-2023) (T300811)

Selenium Framework Updates Timeline

Past

• Port Selenium tests from Ruby to Node.js (2016 - 2023) (T139740)
• Evaluate WebdriverIO replacements for our browser automation framework (2020) (T247835)
• Upgrade WebdriverIO to v6 in all repositories (2020-2021) (T247844)
• Evaluate Microsoft Playwright as replacement for our browser automation (2020-2021) (T262682)
• Upgrade WebdriverIO to v7 in all repositories (2021-2023) (T274579)
• Refactor WebdriverIO tests from sync to async mode (2020-2024) (T256626)
• Switch to spec reporter (2021-2024) (T280331)

Present

• Run Selenium tests targeting available local and remote testing environments (2023-) (T344682)
• Replace Node.js Assert library with WebdriverIO Expect library (2022-) (T325740)

Future

• Upgrade WebdriverIO to v8 in all repositories (2022-) (T324685)
  • Use import instead of require (2024-) (T373125)
• Upgrade WebdriverIO to v9 in all repositories (2024-) (T363704)
• Implement mediawiki core selenium test suite in popular alternatives (2022-) (T325552)
• Find flaky Selenium tests (2021-) (T291254)

Some time ago I did the final work to move out the web performance tooling out of Amazon Web Services (AWS). It's been a long journey. Let me tell you how it all started.

But first, what is the performance tooling? It's tools that automate the web browsers, where we navigate to Wikipedia and measure the performance of Wikipedia pages. When we are measuring the performance we want as stable server and environment as possible. The more stable server where we deploy our tool, the smaller performance regressions we can pick up.

The AWS Background

In 2015 I deployed our first own hosted version of the Open Source web performance tool WebPageTest (T109666). At that time, WebPageTest was Windows only and we didn't want to host Windows machines in our own hosting so we started to use AWS. Hosting outside of our data center also made sense since we wanted to simulate performance testing as our users. Our users do not live in our data center. We started out with a couple of Amazon Elastic Compute Cloud servers and used Amazon object storage for hosting the HTML reports from the tool.

A couple of years later (2018 in T189184), the tool evolved and moved to Linux. I did a lot of testing to make Firefox work in the new setup. We continued to use AWS because it was easy to just spin up a couple of Linux servers.

We also started to host our own time series database outside of our own hosting, to make it easy to add metrics from servers running outside of our own hosting. Have you guessed where it ended up? On AWS because it was easy.

We continued to work on our performance testing and @Gilles and I started to try out front end performance testing using replay proxies (T176361). A replay proxy records the web traffic and replay it locally. That way you measure the front end performance without the disturbance of the internet. That worked out great and we also tried out different host providers to see where we would get the most stable metrics and at that point it was on AWS. With the replay proxy tests, we added a couple of more servers.

We also tried running tests on bare metal servers, however in retrospect, at that time I didn't have the correct skills to configure the servers correctly and AWS gave us better results so we continued to use AWS.

The AWS Problem

Why did we want to move out of AWS? The numbers were good compared to other cloud providers. However we still had problems. One thing that the Netflix team talked about how even if you deploy the same EC2 type, the performance can vary between 10-20% of the instance. You can read more about the variance at Brendan Greggs blog post. We also did try to measure the variance. We could see that we had different performance on the same EC2 type. That meant we needed to make sure that a test ran on a specific EC2 instance. If it ran on another one, we would have a change in our performance metrics.

We also have seen that performance changes over time and that we get different performance at different times T289516. That makes it harder to find small performance regressions.

The New Setup

A couple of years ago I re-evaluated running tests on bare metal servers and it worked out great this time. We can get stable metrics on bare metal servers . We can get the same performance all the time by tuning and configuring the server.

I started to move all of our testing out of AWS to use bare metal (T311981 and T345595). The price for the servers was also cheaper, so the move was a win/win. The new provider re-use old servers from other customers. Instead of buying new hardware we can reuse older ones that were bought for other purposes. That's better in an ecological aspect.

A couple of weeks ago I took the last step and moved our time series database to another host T362020 and instead of Amazon object storage we store the performance result on a disk on a server.

The new setup run on bare metal servers. That is cheaper and gives us more reliable performance metrics. For services that do not need that stability we use Cloud VPS.

The Result

By moving the test servers to a cheaper provider I could run more performance tests at a lower price. We increased the number of test servers with a lower total price. With more test servers, we can do more tests and more runs.

I used extra test runs to implement the Mann Whitney U test (T351929). By running more tests and using Mann Whitney we know if a performance regression is of statistical significance. That way we can make sure that we only alert on real regressions. That decreases the number of false alerts and time spent investigating regressions. I'll write more about how we use Mann Whitney U in my next blog post.

This is a reminder that this Sunday we in QTE together with Mozilla's performance team host the web performance developer room at FOSDEM. You can find the schedule at https://fosdem.org/2024/schedule/track/web-performance/ and the talks will be live streamed on this link starting 09.00 CET on Sunday. There are some really interesting talks and I am really looking forward to them!

If you cannot watch it live, we will collect all the videos and make them available after the event.

I'm happy to announce that QTE together with the Mozilla performance team will host the web performance developer room at FOSDEM 2024! It will be half a day of web performance and will happen on the 4th of February 2024.

Wikimedias performance team has hosted the room twice and now the torch is passed to QTE :) The last time we had a physical event it was standing room only, I hope that it will be that popular again.

Please help us share the call for participation so we can get the same high quality talks as last times:
https://wikitech.wikimedia.org/wiki/Performance/FOSDEM_2024_Call_for_Participation

The developer room is a unique chance to talk about Open Source web performance. See you there!

Flashback to 2007: an engineer at Yahoo named Steve Souders studied users' time spent for different web sites and came to the conclusion that 80-90% of the load time of a web page is spent in the frontend (compared to the backend). He did a follow up 2012 and numbers were the same. He called it the web performance golden rule. What does that mean? Well, the conclusion was that to decrease the load time for users, start by optimising the frontend since there’s so much more time spent there.

I wonder if the same percentage applies to Wikipedia? We are serving content to the whole world and some of our users are far from our data centers. We have users with slow devices and slow internet connection. Also a lot has happened since 2007: traffic moved from desktop only to over 50% on mobile. Does that percentage apply? Let's find out!

How do we get the metrics?

We collect the metrics we need in our Navigation Timing Extension. The data is collected from real users. In the original post the time used for the calculations is the time to first byte and when the load event end happens (all responses are loaded) so let's use the same. We don’t do the calculation per user, instead we can use median or percentiles of those specific metrics. And we can slice and dice it per country and per desktop/mobile.

Users in the USA

Let's first look at desktop traffic from the USA, using the 50 percentile. That's the median and means that 50% of the users have that experience or better. Here we can see that 15% is spent in the backend and 85% is spent in the frontend. That matches Steve's findings.

But just to make sure, let's look at 95 percentile (the users in the USA that have the slower experience). We can see an increase in time spent in the backend 22% but the majority of the time is still spent in the frontend.

So what kind of numbers do you think our mobile users have? Let's again look at the 50 and 95 percentile.

Ah ok, we can see that more time here is spent in the backend. Something between 25-33%. Still the frontend is the major part.

Does this reflect in the rest of the world?

India, South Africa and Brazil

Here's a couple of graphs for mobile users in India, South Africa and users in Brazil. What we can see is that we have higher numbers than Steves' original study but users still spend 2/3 of the time in the frontend.

Summary

I think it's fair to say that our users spend most time in the frontend. You can check out the dashboard yourself and dig into our data.

But do the frontend/backend measurements give us the full story of our users' experience? Noooo it doesn't, it shows us where we should focus. In the next blog post I'll look deeper into the actual experience of our users.

Fuzheado, CC BY-SA 4.0, via Wikimedia Commons

Long time no see

Last month I was in beautiful Athens, Greece for the Wikimedia Hackathon 2023. The last time I saw anybody from the Wikimedia Foundation in person was in San Francisco in 2020. The last time I saw anybody from the Wikimedia movement was in Atlanta in 2019. The last in person hackathon was in Prague in 2019. So, I didn't see some people for 3 and some for 4 years. I have also seen some people for the first time, even if we collaborated for years.

Hackathon

It's really hard to summarize a hackathon. It's a crazy mixture of technical talks, coding, socializing, sightseeing and fun.

It started with project pitches. Everybody that wanted to, could get people interested in their project. But, since there were about 200 people there, everybody got only 30 seconds. Fun part was that if you went over your time, you were clapped off the stage.

One of the interesting sessions I went to was T332794: [Session] Using Pixel tool for visual regression by @Etonkovidova. Elena was very nervous before the session, but she did a great job. We had a great discussion during the session. @dom_walden showed us how he set up T337539: Add WikiLambda extension and tests to Pixel.

Another interesting session was how to T332881: [Session] Improve your technical writing by @kostajh. At that session I learned about LanguageTool and Google's Technical Writing Courses.

The last session I'm going to mention is the T337340: [session] Using Patch demo to test or demonstrate patches outside of your local dev environment (or without one!) by @matmarex. I'm very interested in development/testing environments, so I try to learn at least the basics of available ones.

My plan was to work on one of several projects at the hackathon. Either documentation (T256239: Run Selenium tests targeting MediaWiki-Docker), or a refactoring (T256626: Refactor WebdriverIO tests from sync to async mode), or an upgrade (T324685: Upgrade WebdriverIO to v8 in all repositories). That didn't happen. But that's how hackathons sometimes go.

I applied for the hackathon months ago. In the meantime, I didn't work for a couple of months because I was on parental leave. The hackathon was near the end of my leave, so I was pretty lost and confused at the event.

Misc

I managed to do some sightseeing. One day after the sessions a few of us walked to the Philopappos Monument. It's on the top of a hill with a beautiful view of Athens. Another day I visited Acropolis.

A few of my teammates from the Quality-and-Test-Engineering-Team were at the hackathon, so one day we met for lunch. We didn't see each other in person for at least 3 years. It was great to be able to socialize after such a long time.

I have also learned about a new saint, Saint Javelin.

I was disappointed that there was no organized sightseeing, considering we were in one of the most interesting cities in the world.

Chess

01tonythomas, CC BY-SA 4.0, via Wikimedia Commons

I forgot to bring a chess set, so I bought a cheap plastic set. I led a Chess Basics workshop during lunch one day and had a few games of chess during the hackathon. I was hoping for a few easy wins. I have played about 10 games. I think I managed to win just 2, and somehow draw 1 game. I lost the rest. I'm glad this was a hackathon and not a chess tournament. My rating would be seriously decreased. 😅

Links

• Wikimedia Hackathon 2023 at mediawiki.org
• 2023 Wikimedia Hackathon recap at legoktm.com
• Notes on the Wikimedia 2023 Athens Hackathon at kostaharlan.net
• Wikimedia Hackathon Athens 2023 at taavi.wtf
• A short account of the Wikimedia Hackathon 2023 at diff.wikimedia.org
• Wikimedia Hackathon 2023 Blog at github.io
• Wikimedia Hackathon Athens 2023 at commons.wikimedia.org

Three sentence summary

Prepare for a very hard read. You will not understand anything the first time you read the book. If you're like me, you will not like the book and you will not want to read it the second time.

Book Club

This was the sixth book in our T247665: QTE book club and the first one that wasn't on software testing. We were reading it for three months, from November 2022 to January 2023. The club meets once a month for an hour long discussion. We had three very interesting discussions. Some members of the club like the book, some don't. This is by far the oldest book we've read. It was published in 1975.

Random thoughts

It's one of those books that you have to read once just to get an idea of what the book is about. Then you have to read it at least one more time. I've read it once and I've had enough of it for now. I'm not sure I know what the book is about. Maybe I'll read it again in the future. Maybe.

According to my notes, in 2022 I've read over 100 books. Only 5 got a 2-star rating. 4 were kid's books I didn't like. The fifth 2-star book was this one.

This book was recommended in Lessons Learned in Software Testing (5-star rating from me), so I was expecting much more.

According to Wikipedia, this is one of the two of the author's most famous books. The second one is The Psychology of Computer Programming. I've read his Perfect Software: And Other Illusions about Testing (twice) and really liked it (4-star rating).

Goodreads and Amazon reviews say I'm not alone in not liking the book. But, at both sites, the book has a great average rating (3.96 and 4.2).

I usually sort books in fiction and non-fiction. Or, practical and non-practical. This book looked like a practical book, but I didn't get anything practical out of it. Maybe it's not the book. Maybe I just didn't put enough effort in the book. Maybe I've approached it wrongly, expecting it to be a practical book.

Usually, when I read a practical book, I summarize each chapter. That's what I did for this book. Well, I've tried. I've struggled with chapter summaries for the first three chapters, then gave up for chapters four to seven.

Quotes

All that said, there are good quotes in the book.

Chapter 1. The Problem

The first step to knowledge is the confession of ignorance. We know far, far less about our world than most of us care to confess. (p. 10)

Any field with the word “science” in its name is guaranteed not to be a science. (p. 32)

Chapter 2. The Approach

If you never say anything wrong, you never say anything. (p. 47)

Chapter 3. System and Illusion

What is a system? As any poet knows, a system is a way of looking at the world. (p. 55)

Chapter 4. Interpreting Observations

The Lump Law: If we want to learn anything, we mustn’t try to learn everything. (p. 105)

Chapter 5. Breaking Down Observations

The Axiom of Experience: The future will be like the past, because, in the past, the future was like the past. (p. 141)

Chapter 6. Describing Behavior

Count-to-Three Principle: If you cannot think of three ways of abusing a tool, you do not understand how to use it. (p. 191)

Chapter 7. Some Systems Questions

These, then, are the three great questions that govern general systems thinking, the Systems Triumvirate:

1. Why do I see what I see?
2. Why do things stay the same?
3. Why do things change? (p. 221)

Finally, after a few years, I've been to a conference. The last conference I went to was WebCamp Zagreb 2019. This was the first Testival conference that I have attended as a participant, not an organizer.

Unfortunately, I have been really busy in the last couple of months. I didn't have the time to write something about the conference until now. I took some photos and notes during the conference, so I think I have enough material for a blog post.

The organizing team did a great job. There were more than 130 people at the conference. When I was one of the organizers, we never had more than about 70. 😅

I really like the Testival conference open space format that the conference used for years. In addition to open space, Testival had two keynotes and lightning talks. I gave two lightning talks, one about the T247665: QTE book club I organized for my team at work and another one about chess. Since the lunch break was long, I organized a chess workshop during the break.

I have participated in several sessions including career development, the future of testing and self education.

After the conference I had dinner with Davor Banović (a published book author) and Karlo Šmid (a performing stand-up comedian). Davor has organized the first testing conference in Croatia, viaqa (in 2010). Karlo has organized the first testing meetup in Croatia, Software Testers Speak Up Meeting (in 2011). The three of us were the Testival core organizing team for about a decade, organizing monthly meetups and yearly conferences.

I woke up early before the conference, so I had the time for an easy and long run on the beautiful Drava river bank.

For more information and photos, read the official Testival blog post.

Three sentence summary

This book is about doing testing when your coworkers don't, won't and don't have to follow the rules (p. vii). Realistic test planning is dominated by the need to select a few test cases from a huge set of possibilities (p. 17). Testing is best conceived as a group that provides technical services and information. (p. 343)

Book Club

This is the fifth book that my team is reading for our T247665: QTE book club. We were reading this book from August to September 2022. We had a meeting every month to discuss the book. I like talking with people about good books as much as I like reading them.

Random thoughts

I've already started reading this book many years ago, and gave up. I remember thinking it was very hard to read. I didn't find it so hard to read now.

This is one of the rare books that I'm reading from a paper copy. I usually read e-books. This one is pretty old. It has the very familiar smell of an old book. The smell takes me back to my childhood when I was reading a lot of comics that smelled exactly like that.

I was planning to read the book while traveling the Croatian seaside this summer. So, the book visited three islands with me, Brač, Hvar and Čiovo. Unfortunately, I didn't have any time to read the book there.

Chapters

When I read a book, I like to write a short summary of each chapter. It's usually a quote from the chapter. For some chapters a have a few random thoughts or quotes.

Section 1 - Fundamentals

1 An example test series

Summary: This chapter introduces the book by illustrating how an experienced tester could approach the early testing of a simple program. (p. 1)

2 The objectives and limits of testing

Summary: Realistic test planning is dominated by the need to select a few test cases from a huge set of possibilities. No matter how hard you try, you will miss important tests. As we see it, testing is the process of searching for errors. Good test cases are more likely to find errors, or more likely to find serious errors, than poor test cases. (p. 17)

3 Test types and their place in the software development process

Summary: This chapter is a general overview of the field of testing. (p. 27)

Quote: Some readers have identified this as the most boring chapter of the book. People who stopped reading this book tell us they stopped here. (p. 28)

As far as I can remember, in my entire life, there are only two books that I've started reading but didn't finish. About a decade ago, I started reading this book but gave up. Probably in chapter 3. 😅 The other book is Crime and Punishment by Fyodor Dostoevsky. (It was required reading in high school.)

Quote: In bebugging a program, someone deliberately introduces bugs into it before the person who will test it starts testing. (p. 49)

I don't think I've ever seen this done.

4 Software errors

Summary: This brief chapter defines "quality" and "software error". Then (...) we describe thirteen categories of software errors. (p. 59)

5 Reporting and analyzing bugs

Summary: How well you report a bug directly affects how likely the programmer is to fix it. (p. 65)

Section 2 - Specific testing skills

6 The problem tracking system

Summary: Here we describe what happens to the Problem report after you report it. This chapter provides the basic design of a problem tracking database and puts it in perspective.

Random thoughts: This chapter might be the most obsolete part of the book. It assumes you have to create a task/bug tracking system. Many such systems exist today and it's very unlikely the company isn't already using one.

7 Test case design

Summary: This chapter is about creating good black box test cases. (p.123)

8 Testing printers (and other devices)

Summary: This chapter is about configuration testing in general and about printer testing in particular. (p. 143)

Random thoughts: I've never needed to test a printer. The vast majority of this chapter is about that. I guess an equally important device to test these days would be a phone.

9 Localization testing

Summary: Localization is even more important now than when the book was written. In the chapter they focus on West European languages and DOS. Today localization testing is much harder. There are three major desktop and two major mobile operating systems. Programs are translated to many more languages, some of them quite tricky (right-to-left, script...)

10 Testing user manuals

Summary: The product includes more than the program. For example, most products also include documentation, packaging, samples and service (such as technical support). (p. 179)

11 Testing tools

Summary: This is a basic introduction to black box testing tools: what you might want to accomplish with them, what types of things they can do, and what some of their limitations are. (p. 189)

Random thoughts: Fundamental tools have not changed in a couple of decades (since 1999). We still use a computer, text editors, spreadsheets, diff tools, file viewers, screen captures, inspectors... (pp. 189-190)

12 Test planning and test documentation

Summary: This chapter ... discusses the general strategy and objectives of test planning. We regard this chapter as the technical centerpiece of this book. (p. 203) Remember your primary task - finding bugs and getting them fixed - not designing or filling out forms. (p. 253)

Random thoughts: I found this chapter pretty boring to read.

Section 3 - Managing testing projects and groups

13 Tying it together

Summary: This chapter ties together the organizational and strategic issues. (p. 255)

Random thoughts: It's been a long time since I've tested a piece of software that went through all phases (idea, alpha, beta...). The book suggests that is the usual case. Teams start deploying very early these days, with incremental changes.

14 Legal consequences of defective software

Summary: QC should recognize product defects that are so serious that the company will be sued over them. QC should be able to use the law as ammunition, in an argument to get a serious defect fixed. (p. 303)

Random thoughts: This was a very boring chapter. It is also focused on US law, making it less useful for people outside the US.

15 Managing a testing group

Summary: Testing is best conceived as a group that provides technical services and information. (p. 343)

Random thoughts: I'm not a manager, but I find this chapter very interesting and useful. Especially a very short career growth section, a topic that I'm very interested in.

Quote: It seems to us that every company already has a proper group to set standards, evaluate and train staff, and generally monitor and work to improve every phase of product development. That group is called Management. Management is the real quality assurance group in every company. (p. 347)

I couldn't agree more.

Appendix: Common software errors

Summary: Check a comprehensive "standard" list to get further ideas for testing a program. (p. 363)

Random thoughts: I thought the appendix would be boring. It is actually an interesting read.

Three sentence summary

Tourism benefits from a mix of structure and freedom, and so does exploratory testing. (p. 42)
The industry’s approach to manual testing has been either to overprepare by writing scripts, scenarios, and plans in advance or underprepare by simply proceeding in an ad hoc manner. Exploratory testing with tours is a good middle ground. (pp. 121-122)

Book Club

This is the fourth book that my team at work is reading for our T247665: QTE book club. All books we've read so far were on software testing. We were reading this book from May to July 2022. We've had two book club meetings to discuss the book.

Random thoughts

As it usually goes 🙄 I've finished reading this book in June but I didn't have the time to write the review then. I was on vacation for a few weeks in August so writing the review got further delayed. I think it's way past the last responsible moment to write the review, but better now than even later.

After a very successful tech career, the author is now an investor/owner of a brewery.

I'm pretty sure the author's other popular book, How to Break Software is the first book on software testing that I've ever read. In 2004 I was interviewing for my first software testing job. Back then, I didn't even know software testing existed. The company gave me a couple of books to read. I'm not sure which was the other book, but I'm pretty sure How to Break Software was one of them. I remember that I liked the book and learned a lot from it. I plan to read it again one of these days.

The book is roughly separated into two parts. The book and the appendices. I was reading on Kindle. The book starts at 6%. The appendices start at 60% and end at 89%. So the book is 54%, the appendices are 29%.

I won't summarize all chapters, but a few of them stood out.

Chapter 6 (Exploratory Testing in Practice) is interesting because it provides viewpoints from different people, teams and products. But, it's also mediocre.

Chapter 7 (Touring and Testing’s Primary Pain Points) is one of the most interesting chapters of the book. I don't remember any other serious discussion about these serious problems that testing has.

Chapter 8 (The Future of Software Testing) offers very interesting thoughts on the future. I don't remember reading about the future anywhere else.

When I first scanned the book, I thought appendices were there just to make the book bigger. I was pleasantly surprised while reading them. There are three of them. The first one is Building a Successful Career in Testing and the second and the third one are a selection of author's blog posts. Both are worth reading. I have more notes from appendices than from the book.

Quotes

I have written down many many quotes from the book. Here are a few of them, with my comments.

“Any sufficiently advanced bug is indistinguishable from a feature.” Rich Kulawiec (p. 21).

A funny variation of: “Any sufficiently advanced technology is indistinguishable from magic.” Arthur C. Clarke

Imagine trying to predict the winner of the Super Bowl or Premier League before the season begins. (pp. 37-38)

A beautiful example of the impossibility of planning all testing in advance.

“One man’s crappy software is another man’s full time job.” Jessica Gaston (p. 113)

A funny variation of: "one man's meat is another man's poison".

These are some number of bugs that simply cannot be found until the house is lived in, and software is no different. (p. 119)

Oh so true.

Anyone who thinks they can learn testing in a single day is a fool who has no business testing software. (p. 152).

I guess it's the same for other things, chess for example. You can learn the basics in a day, but it will take you a lifetime to master it.

Certainly, I’ll agree that, like artists, software testers need to be creative, but art implies skill without training. Most virtuoso artists were born to the task, and those of us unlucky enough to have no artistic talent are unlikely to develop such skill despite a lifetime of practice. (p. 157)

I think this is completely wrong. Art requires a lot of training. I don't think any virtuoso was born with it. Some people have more talent than others, but if you don't train, your talent is wasted.

A discipline is a branch of knowledge or learning. Mastery of a discipline is achieved through training, not practice. (p. 158)

The above two quotes are from Software Testing as an Art, a Craft and a Discipline blog post. This is one of the most interesting chapters of the book. The difference between craft (learn on the job) and discipline (learn by deliberate training) is an eye opening insight.

Spam, phishing, and pharming...(p. 160)

TIL pharming is a cyberattack intended to redirect a website's traffic to another, fake site by installing a malicious program on the computer.

Smart people who dream big inspire me. Smart people who don’t understand testing and dream big scare the hell out of me. (p. 182).

I've been there. 😅

Software tester wanted. Position requires comparing an insanely complicated, poorly documented product to a nonexistent or woefully incomplete specification. Help from original developers will be minimal and given grudgingly. Product will be used in environments that vary widely with multiple users, multiple platforms, multiple languages, and other such impossibilities yet unknown but just as important. We’re not quite sure what it means, but security and privacy are paramount and post release failures are unacceptable and could cause us to go out of business. (p. 200)

This hilarious job post is from the Software Tester Wanted blog post.

Introduction

It was a very long time since the last in-person Testival meetup. Just before the COVID pandemic, we had meetup #58, in March 2020. Before that, we've been meeting regularly every month for years. We also had a conference every year. It feels really strange that there was no in-person meetup or a conference in over two years.

There were a few online-only meetups in 2020 and 2021. Even online meetups have stopped since Karlo and I have stepped down as organizers.

Niko Mađar has decided to organize the Testival conference this year. In addition to that, he decided to organize the Testival meetup. I wish him all the luck.

Testival Meetup #64

Finally, this month we had Testival Meetup #64. About 30 people came. A nice number considering the meetup was on a break for a couple of years.

The meetup used the usual format that I really like. Nike gave a short introduction. Then everybody shortly introduced themselves. I think this is such a critical part of the meetup. It's really important to get to know the people. It's really rare that somebody doesn't want to introduce themselves. During the introduction Iva Čičin-Šain said she was the one that came up with the name Testival. (I couldn't remember that when I wrote Stepping down as Testival Organizer.)

Talks

There were two talks. I have a few notes.

• Vesna Bilić - Excel in your QA career!
• Niko Mađar - How programs test programs

From Vesna's talk:

• You can manage your career proactive or reactive.
• To advance your career you have to step out of your comfort zone.
• Life is long. There is time.
• Besides technical knowledge, life experience is required.
• Book recommendation - Team Topologies: Organizing Business and Technology Teams for Fast Flow by Matthew Skelton and Manuel Pais.

From Niko's talk:

• Photomath Engineering blog.

There was interesting discussion during and after each talk.

The only complaint I have is that it was hard to hear the speakers. For the first talk I was sitting at the back and I was struggling to hear. I've moved to the first row during the break between the talks, so I could hear better. A sound system is required for such a big room.

Niko announced lightning talks at the beginning, but forgot about them at the end. I think it's perfectly normal to forget something the first time you're organizing a meetup.

After the Talks

After the talks the majority of the people stayed to chat. The reason I don't like online meetups is that this social component is missing. Online, you can't easily form small groups that split and merge chaotically.

Conclusion

It was great to see, after a very long time, Testival organizers and regular attendees. Niko said the plan is to continue with meetups every month. I'm looking forward to more meetups.

Three sentence summary

Testing = checking + exploring. You can plan some testing in advance, that's checking. You can't plan some testing in advance, that's exploring.

Random thoughts

@dom_walden suggested we should read Explore It for our book club at work. When I checked its ratings on Goodreads and Amazon, I was very impressed. I can't remember when I've seen such good ratings. Goodreads 4.30 (302 ratings, 29 reviews), Amazon 4.7 (77 ratings, 39 reviews). After reading the book, I completely agree with the ratings.

Book Club

Once a month, we have a book club meeting. We usually read a book for one or two months. So, we have one or two meetings about the book. This was the first time we were reading the book for three months. As with Blog Post: Lessons Learned in Software Testing: A Context-Driven Approach, I wanted to invite the author to the meeting.

After a bit of searching my archives, I've found that I've reported a bug on her website in 2005 and she sent me a journal and a couple of bracelets as a thank you note. I met her in 2013 while visiting San Francisco for business. In 2014 she gave a talk on exploratory testing at the Wikimedia Foundation office (video). So, I didn't feel like a complete fool when I asked her to come to the meeting.

To my big surprise, she agreed to come! 😅 My biggest takeaway from the meeting was the discussion about how hard it was to write the book and how much a good editor (the person, not the text editor) contributes to the end result.

Reviewers

I would like to thank @thcipriani for reviewing the article and a lot of advice on how to make it better.

Life is like riding a bicycle. To keep your balance you must keep moving. — Albert Einstien

Overview
Programs like Outreachy help an individual a lot when you are starting to work on open source. Over time, I have learned a lot throughout this program and have gotten the confidence to work in any system. Getting confidence, learning new technologies, and working with collaboration were a few aspects of this internship.

What were the most amazing and fearful aspects of this program?
When contributing to it for the first time, I was having a fear that I wouldn’t be having the required expertise but as I started to learn more about it, it became easier to resolve assigned tasks. During the internship journey, there were several challenges that I came across. One of the challenges was about “Falsy” and “Truthy” values were highly interesting. After a few experiments, I learned that these terms are not exactly the same “True” and “False”.

What technical skills did I acquire?
Getting to know more about Gerrit, how the system handles CI/CD, and upskilling my sync/async knowledge. This will be going to help me write code in an efficient manner in the future and develop resilient architecture. Moreover, working in an open-source environment would help me to contribute further to different technologies. My mentor used to help me whenever I used to get stuck on a bug. We used to have regular meetings, where we used to discuss issues and the latest methods. From time to time, he has provided me with valuable lessons and suggestions.

Did this program increase my confidence?
As mentioned earlier, the Outreachy program has provided me with all the right skills to contribute to more open-source areas, and hence, I feel more confident to contribute more to open-source in the future. Moreover, having conversations with individuals from different areas has improved my communication and presentation skills. Now I can deliver my message more clearly.

What are my future perspectives?
I was able to async and deploy one of the core repositories while continuing working on other sub repositories. Even though the Outreachy program has finished, I would still continue working on the remaining repositories and would try to fix the remaining issues.

Anyone who has never made a mistake has never tried anything new — Albert Einstein

When you are trying new things, you always need time to become an expert. It’s been quite a while since I have been working for Outreachy and it has taught me a lot of things. Well, these are the few things which I have come across.

The mystery behind “Truthy and Falsy"
From the definition, In JavaScript Truthy value is considered true when encountered in a Boolean context and vice versa for Falsy

So what’s the mystery behind this?

This is the concept behind it but why is it being utilized in my open source project?

Well, I was making all the use cases from sync to async and on the way, my mentor and I observed something strange. The values inside the assert were returning a “Promise” rather than an actual value. We didn’t figure this out earlier because tests were passing and we thought that there wasn’t an issue. In reality, what was actually happening was that the tests were passing because “assert” considers a Promise as a Truthy value rather than a Falsy, and for that when it used to receive a Promise, it used to mark it as a pass.

In simple terms, all the values besides these are considered Truthy.

• false
• 0
• -0
• “”
• null
• undefined
• NaN

So this was something new that I didn’t know before so definitely it was a learning experience for me.

WebdriverIO — An Encyclopedia of its own
The documentation of WebdriverIO is indeed intensive and that’s really helpful. Over the time of my career, I have come across a lot of documentation and for some of the time, I have written my own as well but I haven’t seen documentation as extensive as this one.

Zuul, Jenkins, and Gerrit — Three Musketeers
Zuul is considered to be an open-source CI that powers some of the largest Open Source development efforts whereas Jenkins itself is a leading open source automation server. Jenkins provides hundreds of plugins to support building, deploying, and automating any project. Gerrit is a free, web-based team code collaboration tool. Software developers in a team can review each other’s modifications to their source code using a Web browser and approve or reject those changes. If you are using these three in a project, developers in your team would be appreciating you a lot. My experience with these three tools has been great and it looks like when you are in a battle, this trio is always there to protect you.

Experience Uptill now
The experience has been really superb up till now and I have learned a lot. This experience would help me in my career to become even a better software engineer in the future who doesn't only know about software development but would know about software testing as well!

Mentoring

Every time I finish a mentorship, I read my blog posts about previous mentorships. Almost every time I finish a mentorship I write that I'll probably take a break from mentoring for a while, but then I don't. 😅

Introduction

For the last few months I have been mentoring @Osamaahmed17. He was an Outreachy Round 23 intern. He was working on T256626: Refactor WebdriverIO tests from sync to async mode. That was a very challenging task. Both of us didn't know much about async/await when the internship started.

As I was preparing for this blog post, I checked when was the first time I've met Osama. Looks like the first interaction we had was on Zulip in October 2021. He introduced himself and asked a question. We've had many conversations since and got to know each other pretty well.

Tasks

He created 23 tasks in Phabricator (our task tracking tool) and resolved 13 of them. Besides his main task, he also worked on several small side projects. An interesting one was T300386: Delete Invisible Unicode Characters. He also worked on T256239: Run Selenium tests targeting MediaWiki-Docker.

Commits

Osama created 45 commits in Gerrit, our code review tool. 16 of those commits are merged so far. A notable commit is 730546 for the MediaWiki Core repository that needed 69 updates before it was merged. We needed 3 more follow up commits to finish the task in that repository. 😬

Besides his main project, notable commits include releasing a new version of an npm package and fixing a dependency in one of our projects.

Communication

Osama has documented the internship both on his personal blog and on my team blog. We have used Zulip for chat and Google Meet for video meetings.

Conclusion

I really like mentoring. I learn something new with each intern I mentor. I hope they also learn something from me.

“Peace cannot be kept by force. It can only be achieved by understanding.” — Albert Einstein

When you are working on a project, sometimes things can become quite stressful but quotes like above are always motivating. The Outreachy program has taught me a few things related to technical fields which I believe would be beneficial in the long run in my career. As the program is halfway through and I have learned a lot, there are a few things that I like to mention.

Concept of Sync and Async
The project to which I was assigned relied heavily upon the concept of synchronous and asynchronous. As I have been working in the field of software engineering for a while, I was already aware of the concept. For people who don’t know about it, here you go.
In synchronous operations, tasks are executed one at a time, and only after one is finished is the next allowed. In other words, you must wait for one activity to complete before moving on to the next.

Asynchronous operations, on the other hand, allow you to switch to another activity before the preceding one completes. As a result of asynchronous programming, you may handle multiple requests at the same time, executing more tasks in a much shorter time.

This is the concept behind it but why is it being utilized in my open source project?

There is a package called WebdriverIO Sync and its support has been dropped by WebdriverIO due to breaking changes in fibers and V8. As this package was being used as a dependency for the selenium tests, it was required to move all selenium tests from sync to async.
To do so, the first challenge was to get aware of the documentation provided by WebdriverIO. The documentation provided me with an understanding of how the sync and async concept is being utilized and how it can be implemented in the MediaWiki Selenium tests. For instance, the following code contains a test and is mentioned in sync mode.

For us to make it into async, it is required that the test uses async and await. Plus, all WebdriverIO commands return a Promise and need to be awaited to get the result. Observe the following.

This is one area where I have been currently working and to be more familiar regarding sync and async usage would definitely recommend you to have a look at the getting started guide.

Gerrit and why do I like it?
I have been working with Gerrit for a while and personally, I like it. So before I can jump into more depth, let's see what Gerrit is.
As per Wikipedia, “it is a free, web-based team code collaboration tool. Software developers in a team can review each other’s modifications on their source code using a Web browser and approve or reject those changes. It integrates closely with Git, a distributed version control system”.

So why do I like Gerrit?

The reason is its ability to make multiple patches on a commit before merging it. You can easily make individual comments on each patch that can be reviewed by one or more individuals.

How will these things help me with my career?
So not to make this blog so long, in the end I would say that it's just a part of the things that I have learned uptill now and with a month still remaining, I am aiming to learn even more. Ultimately, working in an opensource platform will help me to get my dream job and I am definitely ensured that hard work will definitely help me to ace that future interview!

Lessons Learned in Software Testing signed by Bret Pettichord.

The Book in Five Sentences

Testing is a mental activity that requires skill and experience. Your job is to find information about the project, fast. You will not find all the problems. Writing well is as important as testing well. It's better to do a lot of kinds of testing well than to do just a few kinds of testing perfectly.

Introduction

I read 100-200 books a year. I rate books on a 1-5 scale. (5 is the best.) It's rare that a book gets a 5. This is one of those books. In the last two years, only about 2% of the books I've read got a 5. (In 2021 I've read over a 100 books, only 2 books got a 5. In 2020 I've read over 200 books, only 5 books got a 5.)

I read Lessons Learned for the first time a decade ago, in 2012. (I gave it a 4 then.) As with the previous book I wrote about, Blog Post: Perfect Software by Gerald M. Weinberg, I don't think I had enough experience to fully appreciate the book then. It took another decade of experience to understand how well written the book is, and how important the topics discussed in it are. In 2012 I thought the chapters on test automation and career development were the most important. Looking back, I agree that those two chapters were the most important then. At the time, I was working as a freelancer, mainly on small and short projects. Being good at test automation and taking care of my career were the most important things then, and in a lot of sense even now.

As I review the chapters now, I would like to add more chapters to the most-important list. I'm struggling not to put all 11 chapters on the list. If I had to pick only one more, I would pick the chapter on testing strategy. That's something I've seen not done at all, not done on purpose, or not done well, over and over.

Technical books get obsolete very fast. Lessons Learned was written in 2002. For a 20 year old book it's surprisingly up to date. There were a few mentions of obsolete technologies in it, but the vast majority of the book is as relevant now as it was when it was written.

Book Club

I was reading this book as part of a T247665: QTE book club at work. We pick a book and read it for a month or two, depending on how much it takes us to read it. At the end of each month we have a book club meeting discussing the book. We've read this book for a couple of months. For the second meeting I've invited one of the authors, Bret Pettichord, to join us. That was the best book club meeting we had so far. Bret shared a lot of stories with us. From how they wrote the book to some of his recent projects. We also got to ask a lot of questions.

“Simplicity is the ultimate sophistication” - Leonardo da Vinci

A human being is a complex organism because it is made up of a million individual systems. When you are ill, the doctor tries to find the root of the problem by often trying simple tests first. The same thing happens with software as well. As software often becomes complex, its debugging usually requires simple steps. That is one of the best things which I was able to learn from my journey up till now and I believe that over time, I will be further able to master my capability to simplify things. Simplification was one of the aspects which I was able to grasp but there were a few challenges along the way that helped me grasp other things as well.

Challenges
When you are working on code that has a really great impact on the software, you always have to be extra careful. I started my work on the core component of the software as most of my other work was highly dependent upon it. Making changes to that component and then merging that change to the main software was a challenging task. Changes or commits made used to go through Wikipedia CI/CD (Continuous Integration and Continuous Deployment) software called Zuul. As Wikipedia is an open source platform, dozens of people commit their code every day. Hence, there used to be a possibility that continuous integration would fail a few times so figuring out why it failed used to be the first challenge. It becomes even more challenging when other developers are working on the same library so when it used to fail, you need to figure out whether it’s your fault or someone else.

A complex platform is often dependent on multiple packages and from time to time, these packages get updated as well. There is also a possibility that the updated package would stop providing particular functionality and if your platform is using that functionality, you might get an error. It usually happens when you update the package in the platform as their support is no longer available. This was the second challenge that I faced during the debugging sessions and figuring out the right package version for the platform was a complicated task. However, after multiple tries and countless hours, I was able to figure out the package version and the code was able to pass all the tests in the Zuul.

Achievements and Learning
After facing multiple challenges in my journey up till now, I was able to learn a lot of things. Simplicity was one of the aspects which I was able to grasp and came to know about new software like Gerrit and Zuul. Even though I have worked with similar platforms and software, using it felt a bit different. Pairing programming, thinking out of the box, and debugging were some of the other aspects which I was able to learn. After working for multiple weeks and learning new things at the same time, I was able to get my first major commit merged for the MediaWiki Core!

Failures and setbacks are always part of life and sometimes it feels more like a roller coaster. You can either enjoy it or just spend the whole time screaming in fear. Those ups and downs in roller coasters are more similar to failures and success in life. If you think about it, it’s more like a journey where you have a starting point and a destination.

For myself, taking part in the Outreachy program was one of the steps in the journey and when you take part in a technical exam, it’s more like a rollercoaster where at one time your code is perfect and at other times, you are just giving up. I guess this is a story for everyone and from my perspective, how you conceive that journey, all it matters. If you learn from that journey, you are just adding power-up to yourself. But the secret ingredient to this power-up is that you need to trust your struggle altogether.

Journey Uptill Now!
So coming back to my struggles in the Outreachy program. Contributing to the project felt more like moving to a different country and learning a new language. Signs look different, language is different, rules are different … so many things.

So how do you counter this?

In the beginning, one of my struggles was to figure out Gerrit. I have used GitHub a lot but Gerrit felt different. In GitHub, you can push multiple commits at once and you can just log all your work into GitHub but in Gerrit, it feels more like an updated version of what I have been using for the past couple of years. In a single commit, you can have multiple “patchsets” and these patches can be then updated through updating the commit. Understanding this took me a while but once I learned it, it became a norm.

During the internship, there came a point when our code wasn’t working as per our requirements. We tried multiple things and spent hours understanding where we were going wrong. The struggle of debugging the code for the error was quite challenging but during this struggle, I learned an important art from my mentor which was “simplification”. Not everything needs to be made complicated. Sometimes, the answer lies just there and you are looking everywhere, making things complicated!

The Audience Perspective!
If you are learning throughout your journey and dealing with your struggles, you get noticed by people around you. Sometimes, people also have the same issues as you, and guiding them towards the right direction through your experience is something which I always value. It increases networking opportunities and empathy in society.

If you are dealing with any struggle, then few words of motivation!

“A journey of a thousand miles begins with a single step” - Lao Tzu

I have always been amazed by the latest technology and programming new software is a hobby of mine. From my perspective, the environment plays an important role in the way you develop software and whether you are working in a startup or corporate sector, programming style would be different. Hence, my feelings were different when I started to contribute to open source platforms. Joining the Outreachy program was my first step towards this journey.

What is an Outreachy Program?
As per Wikipedia, Outreachy is a program organized by the Software Freedom Conservancy that arranges three-month paid internships with free and open source software projects for people who are typically underrepresented in those projects.

About myself
I am Osama Ahmed Tahir, an explorer during the day and technology enthusiastic during the night. This would feel like someone who is just from DC comics but being a software engineer feels like you are doing something extraordinary every day. Maybe programming a delivery system for the latest startup or debugging errors of a new feature in an application that is gonna be used by millions of people. Every day feels like a different day when you are a fan of making the world a better place through technology. Every individual has core values which from my terms, are his superpowers! So I also do have some superpowers through which I am trying to make the world a better place. Here are a few of them.

Creativity
Everyone has a different term for the word “Creativity”. In nerd terms, it’s gonna be “Creativity is a phenomenon whereby something new and valuable is formed. The created item may be intangible or a physical object.” Doesn’t sound so creative, doesn’t it? Well in my view, if you are trying something new, you are then creative! I use this superpower a lot, like in everything! When developing software, I think from a different perspective how it can make the person sitting next to you say, wow!

Compassion
Whatever I love to do, I do with compassion. That’s something that has led me far ahead in the field of technology. Being a technology enthusiast from an early age, I started to do programming from an early age. Compassionate towards software development helped me to work on projects that were used by millions of people and were having a great impact on the environment around us.

Curiosity
The other superpower which has helped me to explore new stuff is being curious. It has helped me to learn new programming languages and explore the latest technologies. Combining this power with creativity and compassion has persuaded me to master the latest programming languages much quicker.

In the next blog post, I will discuss my experience regarding joining the Outreachy program so stay tuned!

The Book in Three Sentences

There is no perfect software. Testing provides information. Most people you work with will know little about testing.

Introduction

When I first read Perfect Software I didn't understand it. It took a decade of experience for the lessons to sink in.

Please keep in mind that he published the book when he was 75 years old. You can tell the author has many decades of experience. The book is full of real life examples. I could recognize myself in characters from the examples, for more examples that I would like to admit. 😅

Some parts of this book are pure gold. When reading those parts, I was almost writing down everything stated in the book. Some parts of the book are common sense, but it's still good to have it written down as a reminder and reference. Some parts of the book are more psychology than testing. Surprisingly, those parts are maybe the most important ones.

I really like the format of each chapter. First the main part, followed by a very short summary, then a lot of examples in the "common mistakes" section.

I really liked the chapters where the material is presented in the format of a story (9, 10, 12 and 13). I wish there was more of it. Even the entire novel that would teach you a topic (in this case software testing) by following different characters as the story develops. In the book, three fictional characters (a manager, a developer and a tester) are discussing a problem. First in the office, then over lunch. It's written very well. I've heard that The Phoenix Project is written that way. (I haven't read it yet, it's on my to-read list.)

I also really liked the last two chapters (17 and 18). They are about scams. I've seen so many of them in my career. I've written more about it in the chapter summary.

I'll try to summarize each chapter. For some chapters, I've included a quote that I particularly liked. Often the quote is the perfect summary of the chapter.

Chapters

1 Why do we bother testing?

We are testing all the time. If we were perfect, we would not need to test. Testing provides information. Testers should not decide if the project is ready, that's the manager's job.

We can predict anything except the future. (Page 8.)

2 What testing cannot do?

Don't test if you don't plan to use the information gathered.

3 Why not just test everything?

There are an infinite number of tests. You can't think and run them all. Make the best use of your resources and time by running the best sample you can.

Testing can be exhausting, but it can never be exhaustive. (Page 24.)

4 What's the difference between testing and debugging?

Many different tasks requiring many different skills are often lumped under the rubric of testing. (Page 36.)

5 Meta-Testing

You can greatly improve the efficacy of your testing, and lower your costs, if you learn to use meta-information - information about the quality of information. (Page 47.)

6 Information immunity

Information is neutral, but people's reactions to information are rarely neutral. To assess testing information, you must take into account people's emotional defenses. (Page 59.)

7 How to deal with defensive reactions

Take people's emotions into account, try to understand them and resolve the problems caused by them. (Page 66.)

8 What makes a good test?

You'll never know for sure whether your testing was done well, but there are many ways to know or estimate if it was done badly. (Page 72.)

9 Major fallacies about testing

Learning to recognize a handful of major fallacies about testing could eliminate half the gross mistakes project managers make. (Page 81.)

10 Testing is more than banging keys

To qualify as a test, an action has to seek information that will influence action, whether or not it involves banging on keys. (Page 90.)

11 Information intake

Intake is an active process. You are not a victim, having data put into you, but are at least potentially in control of what you take in. (Page 101.)

12 Making meaning

It is up to human beings to attach meaning to data they take in, and each person does so differently. It's best to keep in mind that there are many possible interpretations of the data. (Page 113.)

13 Determining significance

Our emotions carry information about how important things are. If we pay attention to emotions, listen and address important matters before unimportant matters, we'll be doing the best we can with the data we have. (Page 124.)

14 Making a response

If a project hasn't been managed well before testing, most good responses will no longer be available. In many cases, there is no response that will save the project - except starting over and doing it right from the beginning. (Page 134.)

15 Preventing software testing from growing more difficult

There are intrinsic system dynamics that make testing and fixing take longer as products grow larger and more complex. (Page 140.)

16 Testing without machinery

There are many ways of testing without involving computers, but no way of testing that doesn't involve using brains. Test early and often; use all the brains you can muster. (Page 151.)

17 Testing scams

I've used several testing tools. I've contributed to the development of one. I have many years of experience with tools. I am very suspicious of tools making promises. There are a lot of tools that are claiming they have some special super-power, magic or can read your mind. These days they frequently have AI/ML in their name or description.

Whenever you're desperate to clean up the bugs and ship a product, you're vulnerable to a variety of testing scams that promise quick, painless relief. (Page 160.)

18 Oblivious scams

When we want so much for everything to go well, it's all too easy to inject our fantasies into our data. (Page 168.)

Reviewers

I would like to thank @thcipriani for reviewing the article and a lot of advice on how to make it better.

I have been working remotely since 2005. Recently, an intern has asked me for advice about it. I've decided to write a blog post. I hope the advice will be useful to a wide range of people. From people new to remote work (possibly because of COVID) to people with years of experience.

Some of the advice I'll give in this post might apply to you, some of it might not. Most of the advice should be useful regardless of where you work from, from the office, from a co-working space or from home. You don't have to take all of the advice. Especially not all at once. I didn't.

Computer

You'll probably need one. I've used many desktops and laptops. Most computers today should work just fine. I prefer one big screen or two smaller ones. I have a 27 inch iMac from 2012, and it still works great. (I did upgrade RAM and SSD a few years ago.)

When traveling, I prefer a very small laptop. I've tried a few options over the years, even very small (and cheap) Chromebooks. I've settled on an 11 inch MacBook Air from 2014. It still works just fine.

Internet

I never had to pay much for the internet (even in 2005) and it worked well most of the time. I had some trouble with routers, but not much. Today, probably the cheapest wired internet you can buy would work just fine. If you work from a crowded apartment building, WiFi might now work all the time, for various reasons. It took me some effort to set it up because my desk was not so close to the router. Buying a very long ethernet cable and plugging it in the router and my desktop machine improved my internet speed, latency and stability significantly. I would recommend fast.com (by Netflix) for testing your connection.

Sometimes I work while traveling, and setting up my phone as a router works great.

Desk

"I'll work from my couch." Don't. You'll need your body long after you've retired. Don't break it while you're young. Our bodies are made for moving, not for sitting down for many hours, for many decades. I'm not an ergonomics expert, but I've learned a bit about it over the years. Working from the couch puts your body in a position that's not good for your back and your hands.

Any office desk should work just fine. Bonus points for desks with adjustable height, so you can alternate between sitting and standing. I know IKEA has a few of those.

Chair

"I'll work from the kitchen." Don't. It's probably better than the couch, but you'll be sitting down for many hours, for many years. Kitchen chairs are not made for that. Invest in a good chair. I have SpinaliS HACKER for a decade or so. I don't know how available they are globally.

Keyboard

I've used many keyboards. I really liked Apple's bluetooth keyboard, but after a few years of using it, it caused some hand pain. A few years ago, I switched to Kinesis Advantage2 and never had any problems since. It did take me a while to get used to it, but it was worth the effort.

Mouse

I've tried many mice, and I really like Apple's bluetooth mouse. Any mouse I've used with my right hand has caused some hand pain. I've tried switching to my left hand (I'm right-handed) and never had any trouble since. It does take some time to learn how to use the mouse with the left hand, but it's definitely worth the effort.

Headphones

You'll need headphones. And if you're sharing an office you'll probably want to block disturbing noises. Listening to music or white noise is a good way to do that. Headphones are also very important for video meetings, and it looks like we'll have more of those in the next few years. I've tried many, in-ear, over-the-ear, cheap and expensive, bluetooth and wired, and I mostly use the cheap Apple wired in-ear headphones.

Text Chat

I've used many chat software, but I don't have strong opinions. For many years, I've used IRC via IRCCloud. Lately I mostly use Slack. From a communication perspective, it's almost not important which tool you use. It's important that you can communicate both socially (non-work related topics) and professionally (work related topics) with the people you work with. When possible, I try to avoid Slack. I don't like their vendor lock-in model.

Audio/Video Chat

I've separated this from text chat, because it's different. Most of the time, e-mail, text chat and similar will work just fine, but sometimes, the best way to resolve problems is to talk face-to-face, or share a screen. As usual, I've used a lot of video chat apps and most of them work just fine.

Water-cooler Meeting

Water-cooler meeting is a special type of video call with your colleagues. It's a social time, where you can build social connections and learn more about people you work with. We're not robots. We are social creatures. To work well with other people, we need shared experiences and trust.

Time Tracking

I think I've started tracking time as soon as I've started working remotely. Remote work gives you great flexibility, most of the time. You can start or stop working whenever you want. You can split the day with a long lunch break. It's very easy to spend the day doing private stuff instead of working. To paraphrase Spider-Man: "with great flexibility comes great responsibility". I track my time in 5-minute intervals. I don't track things that take less than 5 minutes. I don't think any employer ever asked me to track my time, but I still do it.

Flexibility

I would encourage you to take advantage of the flexibility that remote work offers. You can do shopping, exercise at your favorite gym, go to movies (...) when most of the people are working. Just make sure you track the time you've worked.

Work Hours

There isn't a universal best time to work. It will mostly depend on your current situation. Most of the people I work with live in the US, while I live in Europe. For some of my colleagues, there is a nine-hour time difference. Some people choose to work at the time when most of the company works, however unusual that work hours are in their country. I have a family, so that doesn't work for me. I still have plenty of late meetings, sometimes until 7pm.

Travel

Before COVID, I used to travel for work a few times a year. Just enough for my taste to see the world, but not too much, since I like to travel. I've visited many countries and cities. Unfortunately, sometimes the only thing I saw was the hotel and the conference venue. Fortunately, trips like that were very rare. I usually manage to spend at least half a day sightseeing. Sometimes even for a few days.

Meeting face-to-face at least several times a year with people you work with is very important. It's much easier to resolve conflicts and problems with people you trust and have strong social connections with. That is currently very hard to do. The best we can do for now is to be social using technology.

Exercise

I originally named this chapter Serious Exercise. You will be sitting for many hours. For decades. Your body is not made for that.

I like running, so that's what I've decided to become serious about. It's almost not important which sport you'll pick. It's more important that you pick something you like doing, so you stick with it for years and decades.

If it's a group sport, join a club. If it's something you can do alone (like running), I would still recommend joining a club, or getting a coach. Especially if you haven't exercised in years, or decades. The most important thing is to create a habit of exercise and avoid injury. A club or a coach will help you start slow and improve over time.

To learn more, I would highly recommend The Healthy Programmer book. It is marketed towards programmers, but it offers good advice for any office job.

Reviewers

I would like to thank Hrvoje Šimić (Shime) for reviewing the article and a lot of advice on how to make it better.

Reading Club

A few years ago we had regular reading club meetings at work. I really liked it. We would pick an article or a book and read it. Every month we would have a meeting and discuss the article or the book. If it was a long book, we would split it in several months. We didn't have a reading club for a while. I've missed it.

I've started working from an office recently and I've found several interesting books there. I've read most of them, but a long time ago. I wanted to read them again. Reading with other people is always more fun, so I've decided to start the reading club again.

I've thought hard about the order in which to read the books. People on my team are on several continents, so I've tried finding books that are available in ebook format. That should make it easier for everybody to get the book. I've also checked book ratings on Goodreads and Amazon. T247665: QTE book club lists the books in the order in which we plan to read them. Book prices and ratings are also included.

Schools of Software Testing

I thought a shorter read would be a good warm up for the reading club. I've heard about schools of software testing many years ago. I liked the idea immediately. I find it a good tool to explain why the experts are talking about different things when they talk about testing. It's probably not the perfect model, but it's a useful one.

After a bit of searching, I've found several presentations, blog posts and videos about the topic. If you know about more good sources of information about the topic, please do let me know. This is the list.

• Four Schools of Software Testing by Bret Pettichord, slides (pdf), from 2003, ~ 5 minutes.
  • Schools: Analytical, Factory, Quality Assurance, Context-Driven.
• Schools of software testing by Cem Kaner, blog post, from 2006, ~ 20 minutes.
  • Schools: Factory, Control, Test-driven, Analytical, Context-driven.
• Five Schools of Software Testing by Bret Pettichord, slides (pdf), from 2007, ~ 5 minutes.
  • Schools: Analytic, Standard, Quality, Context-Driven, Agile.
• Are Testing "Schools" a good idea? by Doug Hoffman and James Bach, video, from 2012, ~ 70 minutes.
• Schools of Software Testing: Useful or Negative Influence? by Cem Kaner and Rex Black, blog post, from 2014, ~ 45 minutes.

School names change from author to author and with the time. These are the five schools, their names and short descriptions. (Descriptions are from Five Schools of Software Testing by Bret Pettichord.)

• Analytic / Analytical - sees testing as rigorous and technical with many proponents in academia.
• Standard / Factory - sees testing as a way to measure progress with emphasis on cost and repeatable standards.
• Quality / Quality Assurance / Control - emphasizes process, policing developers and acting as the gatekeeper.
• Context-Driven - emphasizes people, seeking bugs that stakeholders care about.
• Agile / Test-driven - uses testing to prove that development is complete; emphasizes automated testing.

Meetings

My team name is Quality and Test Engineering. There are about 10 people in it. We are part of a bigger team, Engineering Productivity. There are about 20 people in it. We had two meetings about the topic. First in the smaller team that's focused on testing. Then in the bigger team. There were about 5-10 people in each meeting. The meeting with the smaller team was in the form of a discussion. The meeting with the bigger team was mostly me talking about the schools. My guess is that people in the smaller team were more familiar with testing and had more opinions and experience with it.

Inspiration

Smokers pressure you to light one, drinkers pressure you to drink one, we pressure you to publish one. -- Hrvoje Šimić

In August 2020 I've read a very interesting article. Word rafting by Hrvoje Šimić. The idea is simple. Create a small club. Agree on the rules. Each member has to publish an article in a time frame. If you don't publish, you have to leave the club.

The Club

I liked the idea. I had a blog since 2005. Sometimes I would publish multiple posts in day. Sometimes there would be months between posts. I wanted to write more. I didn't think I could publish every week. Publishing every month sounded more realistic.

I've asked around at work and in October 2020 three of us started on the adventure. Elena and Soham joined me in this crazy experiment. In the first year, we've had members joining and leaving the club. The club never had less than 2 or more than 4 members.

Tyler joined Elena and me in June 2021, completing our current membership.

Rules

We've made some minor changes to the rules over the last year. These are the current rules.

• You have to publish 1 blog post a month. (The last day of the month is the deadline.)
• If you do not publish on time, you are kicked out of the club. (You can join the club after 1 month.)
• The club can have a maximum of 5 members at any time.
• All communication happens at invite-only private super-secret Slack group writing-club.

Results

In the last 12 months, I've published between 1 and 5 articles a month, 28 articles in total. That's a bit over 2 articles a month on average. Exactly what I was looking for. Positive peer pressure to keep me publishing every month.

I was hoping our internal chat will be more busy, with members reviewing each other draft posts. That didn't happen. At least not yet. I've also tried suggesting that every member leaves a public comment for other member's posts. That didn't happen too.

I'm hesitant to open the invitation to the club to anybody. I'm not sure how the club would work if we didn't all know each other. If I know you well, and you're interested, let me know. The club still is accepting 2 more members.

It has been my philosophy of life that difficulties vanish when faced boldly.
― Isaac Asimov

Since doing household chores is undoubtedly makes a house cleaner and better organized, so doing the team chores have the same effect on a team work. Creating a list of tasks for important team activities and going through that list regularly makes handling workload better, monitoring for issues better, and, ultimately, makes the overall quality of delivered products better. The Growth team has been practicing chores list duties for several months now and the results seem to be useful, helpful and even delightful.

It's all started with the idea of chores the Web team has implemented on Web team chores page. The page has a list of monitoring/triaging activities that are done every weekday by different team members. The Growth team has largely adopted the structure of the chores - Growth Team Chores ( thanks @kostajh!) . The Growth Team Chore list page explains details of doing Growth Team Chores and provides some guidance and instructions on how to do the chores efficiently. Both Growth team pages get regular evaluation on how well they reflect the current team's workflows and priorities, and whether the chores suit the needs of the current team projects.

Let's do a quick overview of the format for the chores list. The list of chores is a table that has three groups of chores - monitoring logstash (Logs) , triaging incoming phabricator tickets (Triage/Tidy), and looking for potential performance issues (Performance).

Two columns - Status and Last updated have important information on the state of those sections and when the info has been updated. The Status icons indicate whether something needs attention or not, and Last updated column should have a signature of a person who did the chores and a timestamp

Logs section has links to the Growth team dashboards. There are four separate dashboards for server-side errors, client-side errors, and for the event validation errors. The last link is for the Link recommendation service dashboard. Next section - Triage/Tidy refers to new phab tasks that are recently filed or have been moved into Needs discussion/analysis column on the Growth team workboard. And the last section, Performance, refers to grafana dashboards.

Going through the chores tasks turned out to be an insightful QA practice. Doing chores is useful . From the QA process point of view, the error reports in logstash might give a good picture on what issues are there and, more importantly, how they might be (or already are) affecting users. Ideally, each error report should be evaluated for its severity and frequency. Some errors are severe because they impact user experience or they may be severe because the number of reports is high.

The practice of going through the chores is also helpful - as a QA, I may see that some features need more testing, or that a different testing approach needs to be designed. Some errors might need to be investigated for user impact, or for how to reproduce it. So, there might be lots of valuable info for modifying testing methods and practices.

And, finally, it's really rewarding and even delightful to do such chores! The list of chores adds structure and reduces uncertainty. It increases the probability of catching bugs 🕷 sooner rather than later, thus, minimizing bugs' visibility and their impact. In addition, there might be bugs that haven't been caught during feature testing due to specific configuration or other environmental parameters that were difficult (or, sometime, even impossible) to have in a testing environment.

Introduction

In Blog Post: macOS vs Ubuntu Linux I've written about a fun experiment. I've dual booted an MacBook Air with macOS and Ubuntu. Then, I ran the same npm scripts on both operating systems. To make things more interesting, I've ran the scripts both on the host operating system and inside a Docker container. Running scripts on Ubuntu was so faster, that I wanted to dual boot my iMac too. At the time, I could not dual boot the iMac because my disk refused to be partitioned. In the last couple of weeks, I've formatted disks on both machines and did a clean install of both macOS and Ubuntu. It's time to run the experiment again!

Machines

For reference, here's the table with the two machines and the Fresh container, with some relevant data.

Docker

I've left Docker settings at default values. Increasing resources available to Docker might speed things up on MacOS, but that's a topic for another blog post.

For my reference, here are the default values in Docker Preferences > Resources on macOS.

Running the scripts

Similar to the previous blog post, I was running two npm scripts. Installing npm dependencies (npm ci) and running tests (npm run selenium-test). I've used MediaWiki Core repository (npm ci) and it's Selenium test suite (npm run selenium-test). The suite is targeting a local MediaWiki site. The site is running in a local Docker container via MediaWiki-Docker.

I was not only interested in how fast it is to run the scripts from the operating system (macOS or Linux). I also wanted to know how much time it takes to run the scripts from a Fresh container. (Fresh is *"a fast and ready-to-use Docker container with various developer tools pre-installed."*)

To make things as fair as possible, both machines were using the same wifi and were the same distance from the router. npm run selenium-test was running in headless mode.

Data

I've tried to keep the environments as simple as possible. When running the commands, only the Terminal application was open. I wasn't interested in very good data. I just wanted good enough data for my use case. For that reason, I've only run each command three times. That's just enough data to show any trends, but not so much that I'll get bored while waiting for them to finish. All results are in how much time (in minutes and seconds) each command took in different environments.

I made a mistake in the previous blog post on this topic by including all the data in the post. That's too much data. This time, I'll include only averages.

Chart by environment.

Chart by script.

Conclusion

As expected, running both scripts is much faster on Linux. Running the scripts is much slower on macOS+Fresh combination. The tiny laptop is running great on Linux!

Today is the last day of Google Summer of Code 2021. Vidhi Mody and Soham Parekh were mentoring Sahil Grewal on updating WebdriverIO to version 7 in Wikimedia repositories.

For me, this was the easiest internship that I was involved so far. Mostly because I was not an official mentor this time. I was just helping Vidhi and Soham.

To gather ideas for this post, I was reading my other posts about mentoring. Looks like even back in 2013 I've learned the lesson that there should be two mentors per intern. We've had situations where a mentor can't work for weeks because of health reasons. Also, the internship is over the summer and (at least in Europe) it's common for people to take a few weeks of vacation. So, two mentors is highly recommended.

This time I got the fun part of the internship. I didn't need to do any paperwork, like preparing for the internship and evaluating the student. I was mostly reviewing code and pair programming. I like both of those activities, so the internship was a lot of fun for me.

My plan was to take a break from mentoring for a while, but I have a good idea for an internship project and the next round of Outreachy internship is soon...

The phoenix is a long-lived, immortal bird associated with Greek mythology (with analogs in many cultures) that cyclically regenerates or is otherwise born again.
Phoenix (mythology) - Wikipedia

When I decided to read The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win ( The Phoenix Project ), I never thought that it'd be a quite reading journey. The book makes its impact, sure, but, at the same time, it marvelously manages to give you a whole range of completely opposite emotions: it motivates and discourages, it provides some valuable insights (and "aha!" moments) and, on a next page, makes you cringe from the triviality of solutions that described as magic.

Three aspects of the book definitely impressed me:

• unbeatable humor
• great illustration of what impact misplaced office politics and irrelevant management ambitions might have on overall motivation and effectiveness of work
• insightful overview of optimizing daily work processes (Four categories of work and Three Ways etc),

Let's start with the humor in the book which is really a great part of The Phoenix Project 's impact . The description of ever-changing CIOs, for example:

For the last decade, like clockwork, new CIOs would come and go every two years. They stay just long enough
to understand the acronyms, learn where the bathrooms are, implement a bunch of programs and initiatives to upset the apple cart, and then they’re gone.
CIO stands for “Career Is Over.” And VPs of IT Operations don’t last much longer

Or viewing developers as a threat to a would-be-otherwise-smooth work of an impeccable devops team:

The only thing more dangerous than a developer is a developer conspiring with Security. The two working together gives us means, motive, and opportunity.

Or how a devops team describes pesky disruptions:

They [Security] are always coming up with a million reasons why anything we do will create a security hole that alien space-hackers will exploit to pillage our entire organization and steal all our code, intellectual property, credit card numbers, and pictures of our loved ones.

The description of how releases are run is definitely sounds as a thriller:

One of the developers had actually walked in a couple of minutes ago and said, “Look, it’s running on my laptop. How hard can it be?” Wes started swearing, while two of our engineers and three of William’s engineers started poring through the developer’s laptop, trying to figure out what made it different from the test environment.

And this one as an unforgettable grand finale of a night when important deployment was happening:

In the corner, a developer is asleep under some chairs. It’s been a night of heroics.

In a sense, it's a dark humor. As the readers we observe the neverending chaos and the helplessness of otherwise capable and smart people. People who are dragged into a doomed whirlpool of work that has random deadlines and no structured processes, when no one is assigned a clear responsibility for the product and only could have a glimpse of understanding of how the results would be achieved.

What causes that stressful environment? The management? The curious thing about The Phoenix Project 's management in is that they obviously must have a great stamina to survive and cope with the everpresent hellish level of stress. Unsurprisingly, two (at least) main characters in the book - Steve Masters (the CEO) and Bill Palmer (the narrator and the VP of IT) both have that. They are former Marines. Quite often they reflect on their Marines experience to see if it's applicable to constantly unfolding crises around them (a spoiler: such experience was not relevant):

Each year, it gets harder. We have to do more with less, to simultaneously maintain competitiveness and reduce costs. Some days, I think that it can’t be done. Maybe I spent too much time as a sergeant in the Marines. You >learn that you argue your case as best as you can with your officer, but sometimes you have to say, “Yes, sir,” and then go take that hill.

Bill Palmer, described as "dependable, pragmatic, and willing to say what [he] really thinks", makes this logical conclusion:

I’ve figured out that the trick to a long career in IT Operations management is to get enough seniority to get good things done but to keep your head low enough to avoid the political battles that make you inherently vulnerable.

How such management views get translated into daily workflows, product and operations planning, into seeking feedback and getting everyone motivated? The following summarizes it all:

They [the management] are so freaked out about hitting their deadlines, they’re only now starting to think about how to test and deploy it.

One noticeable thing in the book - a QA team (and QA work) is mentioned only vaguely, like an unnecessary (and almost decorative) stage of a product development. Not recognizing the value of QA processes makes any product development unpredictable and stressful.

However, the most important part of The Phoenix Project is about finding ways to bring an order into the vicious circle of persistent crises and failures. As the phoenix who starts its new life after destruction, the teams in The Phoenix Project struggle to see what needs to be done to emerge from unpredictability and frustration with strength and clear vision. Here comes deus ex machina, Erik Reid, a board member, who has almost supernatural knowledge of how engineering processes should be organized. He guides Bill Palmer, by showing him the similarities between manufacturing plant processes and software development work and deriving from that analogy several valuable concepts. These concepts are presented with catchy titles - the Three Ways, the Four types of work, the Three Acts of Downward spiral, Seven major types of Non-value Adding items (aka "waste" in the manufacturing world). Even though it is a well-known trick to keep readers' attention, that gives a good structure and, hence, makes it easier to remember.

One of the main concepts - the Three Ways. The following description (taken from Book Summary: The Phoenix Project):

• The First Way — Continuously find and implement ways to improve delivery. This is synonymous with the - concept of Kaizen.
• The Second Way — Get fast feedback and work from strong failure signals to ever-weaker failure signals to get advance warning of quality issues.
• The Third Way — Use the efficiencies gained in The First Way and the safety enforced by The Second Way to introduce rapid experiments that help create qualitative gains.

Although those principles are hardly new, they would, indeed, have a tremendous impact on the productivity. Proposing "short and quick cycle times to integrate feedback", "reducing batch size, enabling fast feature flow" are key ingredients for success.

And the last important aspect of any efficient system - the allocating time for knowledge transfer. The Phoenix Project provides several examples of when the commitment to document procedures (and constantly updating it) would pay off:

[...] help convert individual expertise into artifacts that the rest of organization can use.

Was it worth reading the book? Undoubtedly, yes. The Phoenix Project shows what would happen when the processes are not in place, when they are disjointed or incorrectly implemented. The great tsunami of unplanned work (stalled deployments, bugs in production, not diligent maintenance etc) would encroach, inevitably destructing any islands of good work. To avoid the dangers of unplanned work, as one of my favorite quotes emphasizes, there should be a never ending search for improvement:

[it] is almost doesn't matter what you improve as long as you improve something. Why? Because if you are not improving, entropy guarantees that you are actually getting worse.

The Deployments page sets the working framework for QA workflow when it comes to deployment. Although the details and the level of QA work might be different from team to team depending on a specific team's workflow, specific projects and, even, specific phases of a project, the below is an outline of what seems to be, in a sense, typical QA activities for a weekly deploy cadence.

Beta cluster - the start

A patch's journey to production starts with beta cluster (see more details on what beta cluster is here. For an example of patch lifecycle (simplified, of course), let's take a look at the following patch - GrowthExperiments patch (and the corresponding phab task is T287636).
On the high level, the sequence of events that move any patch to be deployed to production is the following:
patch is not merged → patch is merged → patch is deployed to production

Some details on the steps:
(1) patch is not mergedThe first screenshot shows that there is no word "merged". It means that the patch cannot be tested in betalabs yet. The options to test a patch are to set up your local environment and checkout a patch or use patch demo.
(2) patch is merged Now the patch is merged - the word "merged" is present on a phab task section. At this point (or some time after) the related phab task may display the production version to which the patch would be deployed. The gerrit link shows that the patch was "included" (merged) to "master", i.e. a fix or a feature is now on the beta cluster and can be checked there.
(3) patch is deployed to production Finally, a patch is deployed to production - fixes or features can be checked on the production version (the tag shows the exact version). And for this specific example, it shows that the patch was "cherry-picked", i.e. it was deployed during one of the Backport windows

Betalabs	Betalabs	Production
(1) patch is not merged	(2) patch is merged	(3) patch is deployed to production

There might be times when it's useful to see all patches that would be (or were) deployed. Adding deployment and project tag(s) to Phabricator Advanced Search form would display all phab tasks that have patches associated with those tags.

Deployment cadence

The weekly deployment cadence, i.e. the weekly deployment schedule (see Deployments), presents detailed information on what and when would be deployed. The schedule lists the times for the deployment train events and backport windows:

Note: Monitor logstash Growth team has a workflow - Growth Team chores. If one of the deployment days is my chore day, then on that day I'll do more thorough monitoring of logstash (and other activities).

Monday - Checking betalabs

• check after a cutoff time (if needed).
• watch for regression and review new features/fixes that will be deployed to testwiki
• check that all features/fixes that are planned for deployment are included in the train (the phabricator tag is in place)

Tuesday - group 0. Checking testwiki
Why is it important to test testwiki?

• the obvious: it's the first point for production deployment; good to check for feature deployments and for any regression issues
• more user rights can be assigned to my test users than on usual wikis
• I can create/modify content on testwiki without impact on actual users.
• sometimes features get deployed to testwiki a week or more before being deployed to actual wikis to give more time for production-like testing or getting feedback from ambassadors|

Wednesday - group 1. Check specific wikis - Commons, Hebrew, Catalan
The testing on group 1 wikis is limited. I check:

• Commons - I work with Structured Data team and checking in production is a vital part of their team workflow.
• Catalan wikipedia - because it has Flow features.
• Hebrew wikipedia - for two reasons: 1) it is the first of RTL (right-to-left language) wikipedias to be deployed and 2) it has Growth Experiments features deployed on it.

Thursday - group 2. All remaining wikipedias are deployed. Check whatever needs to be checked.

Each team (and QA, of course) should evaluate the risks of deploying features and implement specific monitoring measures. The issues found in production fall into the category of "unplanned work", as it was dramatically defined in The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win (by Gene Kim etc) :

Like matter and antimatter, in the presence of unplanned work, all planned work ignites with incandescent fury, incinerating everything around it. [...]
Unlike the other categories of work, unplanned work is recovery work, which almost always takes you away from
your goals. That’s why it’s so important to know where your unplanned work is coming from.

Having good QA processes on planned work (feature testing, regression testing, and testing in production), the scorching effects of unplanned work, i.e. finding bugs late in production, will be kept to a minimum.

Happy testing!

There are two main purposes of the page object pattern.

• Move parts of the application that change a lot (user interface) to a central location. When the user interface changes, the test suite only needs to be updated in one place.
• Make tests easier to read and understand by moving complexity to page objects.

Let's describe the two purposes with code samples from MediaWiki Core (as of June 2021).

With page object pattern

The code is split into two files. user.js contains the test. createaccount.page.js contains the page object. Code is available at 702327.

Please notice how readable user.js is. You can immediately see that the point of User should be able to create account is to create an user account with username and password as arguments.

tests/selenium/specs/user.js

'use strict';

const assert = require( 'assert' );
const CreateAccountPage = require( '../pageobjects/createaccount.page' );
const Util = require( 'wdio-mediawiki/Util' );

describe( 'User', function () {
	it( 'should be able to create account', function () {
		const username = Util.getTestString( 'User-' );
		const password = Util.getTestString();

		CreateAccountPage.createAccount( username, password );

		assert.strictEqual( CreateAccountPage.heading.getText(), `Welcome, ${username}!` );
	} );
} );

Please notice that createaccount.page.js has two major sections. In the first section are elements on the page (username()...). The second section contains actions (createAccount()...).

tests/selenium/pageobjects/createaccount.page.js

'use strict';

const Page = require( 'wdio-mediawiki/Page' );

class CreateAccountPage extends Page {
	get username() { return $( '#wpName2' ); }
	get password() { return $( '#wpPassword2' ); }
	get confirmPassword() { return $( '#wpRetype' ); }
	get create() { return $( '#wpCreateaccount' ); }
	get heading() { return $( '#firstHeading' ); }

	open() {
		super.openTitle( 'Special:CreateAccount' );
	}

	createAccount( username, password ) {
		this.open();
		this.username.setValue( username );
		this.password.setValue( password );
		this.confirmPassword.setValue( password );
		this.create.click();
	}
}

module.exports = new CreateAccountPage();

Without page object pattern

Let's compare that to a solution without page object pattern. Code is available at 702341.

tests/selenium/specs/user.js

'use strict';

const assert = require( 'assert' );
const Util = require( 'wdio-mediawiki/Util' );

describe( 'User', function () {
	it( 'should be able to create account', function () {
		const username = Util.getTestString( 'User-' );
		const password = Util.getTestString();

		browser.url( '/wiki/Special:CreateAccount' );
		$( '#wpName2' ).setValue( username );
		$( '#wpPassword2' ).setValue( password );
		$( '#wpRetype' ).setValue( password );
		$( '#wpCreateaccount' ).click();

		assert.strictEqual( $( '#firstHeading' ).getText(), `Welcome, ${username}!` );
	} );
} );

Conclusion

With page object pattern, the framework is more complex, there are more files (two instead of one) and there is more overall code, but the tests are more readable and more maintainable.

Without page object pattern, the framework is simpler, there are less files (one instead of two) and there is less overall code, but the tests are less readable and less maintainable.

To keep this blog post short, I've selected a simple example. If the test suite creates accounts a lot, without page object pattern, code for creating account would have to be copy/pasted, leading to duplication and harder maintenance.

With the page object pattern, creating an account is just a call to CreateAccountPage.createAccount() function. If any of the elements for creating the account changes, only CreateAccountPage needs to be updated. If the logic of creating and account changes (for example, not requiring password re-entry) only CreateAccountPage.createAccount() needs to be updated.

More information

• PageObject by Martin Fowler (2013)
• PageObjects by Simon Stewart (Selenium, 2015)
• Page object models by Diego Molina (Selenium, 2019)
• Page Object Pattern by Christian Bromann (WebdriverIO, 2021)
• Selenium/Explanation/Stack
• Selenium/Reference/Stack

“But what am I going to see?
I don't know. In a certain sense, it depends on you.”
― Stanislaw Lem, Solaris

In Testing search in MediaSearch - Part 1 the top down analysis was used for identifying what needs to be tested in MediaSearch. The comparison analysis has been done to prove that MediaSearch, at a minimum, performs as well as Special:Search. All Special:Search functionality should be present in MediaSearch, while the search still works reliably and the users' workflows should not be disrupted. Testing the UI served the same purpose: reliable search, search features are easy to use, and the users have a better experience than with Special:Search.

But the most important question remains: how well the new search answers users' expectations? From the test outline diagram there are other aspects of search that need to be explored to answer that important question - Relevance, full text search vs phrase match search, and Searching in different languages .

[...] relevance is subjective, there is no way to return the perfect result set. There are, however, various approaches and tools that can be used to tune the result set for the most optimal results for your users.

"How scoring works in Elasticsearch" (2)

"Run some search queries and see if the results make sense" - this commonsense approach, probably, would work. I started with a search term that cannot fail - cat

Search term	Special:MediaSearch	Special:Search	MediaSearch full text search ES score (first result)
cat			98.84846 ( Link to scores: cat )

Only pictures of cats are returned by searches with Special:Search and with MediaSearch and Special:Search results match MediaSearch results. ES score (ElasticSearch score) for the first picture is high - the test is a success.

However, the next simple search - search term king - is a little more confusing:

Search term	Special:MediaSearch	Special:Search	MediaSearch full text search ES (first result)
king			82.770256 (Link to scores: king)

Again, there are no differences in the search results between Search and MediaSearch, which is great. But what does the first picture in the result - File:Polistes May 2013-2.jpg- have to do with the search term "king"? If we look closer, the surprising results are not that surprising. It turned out that the relevance scoring relies on the structured data and the structured data for File:Polistes May 2013-2.jpg has the statement "monarch" which is a wikidata item - Q116 that has the "king" label:

Another unexpected item in our search results is the picture of potatoes. The ES score is 58.06765, and it's rather difficult to see the relevance to the search term king. The file name for this picture of seemingly plain potatoes explains why it was included in the results - File:King Edward.jpg. Since the term "king" is in the file name, naturally, the picture will be returned when you search for "king". The structured data, files names, captions, and descriptions are what the search is based on.

Next to test - full text search and phrase matching search. When we search for Newton's cradle, the results are delightful: only results that explicitly portray Newton's cradle are returned. Not relevant results from separates searches Newton and cradle are not present.

search term - Newton	search term - cradle	search terms - Newton's cradle

What if I add the quotation marks and search for "Newton's cradle"? Will it limit the search even more (it should)? Yes, searching for "Newton's cradle" will bring only 50 results vs 56 results for Newton's cradle. And those 6 extra results are scored lower than the top results.
The next test is to check how phrase matching search will work for completely unrelated search terms - e.g. pika ballet. Would it find results for pika and ballet and return randomly (and ES scores will be low?). Nope, the search is smart enough to return nothing:

Searching in different languages is the last challenging item to test. How well the search performs for different languages? What should be the expected results? If I search for кот (Russian: cat), do I expect to see only the images that have кот in their structured data or description? Or the results should be the same as searching for cat since, thanks to wikidata, the labels for Cat (Q1022892) includes many languages. The search results for кот are very, very similar to the search results for cat:

For example

https://commons.wikimedia.org/w/index.php?>search=chien&title=Special:MediaSearch&go=Go&type=image&uselang=fr

contains a lot of images with 'chien' in the title before we get any images of dogs, despite the uselang=fr param. >statement matches are rated more highly than title matches, so that's unexpected

While testing searching in different languages, I viewed the results of suboptimal relevance as acceptable. But logically, when a user sets the UI language in User preferences or via uselang=, the search should give preferential ranking for results that relate to that language! So, searching for chien (French: dog) with uselang=fr should return more pictures with dogs in top results than for the the same search that performed with uselang=en. The issue was promptly fixed and the table below shows the improvement.

Before the fix - commons wmf.7	After the fix - commons wmf.9
search term "chien"	search term "chien"
French UI: https://commons.wikimedia.org/w/index.php?search=chien&title=Special:MediaSearch&go=Go&type=image&uselang=fr	The second image has dogs, that is a good improvement.

Testing search made me realize many aspects of how search results might be evaluated by our users and, based on testing results, to see whether MediaSearch needs to be improved. And it's incredibly rewarding to see improved search experience since having efficient and reliable search functionality adds a lot to user experience.

Special thanks to @matthiasmullie and @Cparle for guidance, insights, and specific tips on testing search functionality in MediaSearch.

1. Research:Measuring User Search Satisfaction https://meta.wikimedia.org/wiki/Research:Measuring_User_Search_Satisfaction retrieved on June 28/2021.
2. "How scoring works in Elasticsearch" https://www.compose.com/articles/how-scoring-works-in-elasticsearch/ retrieved on June 28/2021.

What makes a bug interesting? Any bug, by definition, has an element of a surprise (otherwise, it wouldn't be a bug). But not all bugs are equal in their ability to be surprising. Some bugs do stand out, presenting two essential elements of a surprise - unexpectedness and/or unusual manifestation. Based on my own classification criteria, I define a bug to be the most interesting if it is either (or both)
(1) difficult to find
(2) has a quite unexpected (mostly disruptive) impact

Let's take a look at some examples in those nominations. The first example of a bug that is "Difficult to find" - https://phabricator.wikimedia.org/T272103. The issue's has a long title "[regression - wmf.26] frwiki Homepage SE module has 'cirrussearch-query-too-long' for default filters". Even from the title two things look interesting - first, the issue is marked as a regression bug (indicating that previous testing did not find the issue) and second, it's very specific - mentions French wikipedia and a specific Cirrus error message.
To describe the issue briefly - a user visits Special:Homepage (to see the page you need to log in and to enable Homepage in User preferences), and with the default filters selection, the error would be displayed.

• the issue happens only on French wiki - which makes it classified as "Difficult to find"
• the issue happens for default selection (no user input) - therefore, the impact on user experience is big
• users are given false feedback that tasks are there (the tasks count is present) - again, the impact on user experience is big since users would see misleading info.

On the positive side, the error message is super specific, and points to the exact root cause of the issue:

Fetching task suggestions failed: cirrussearch-query-too-long cirrussearch-query-too-long

"Search request is longer than the maximum allowed length. (2122 > 2048)"

Although the issue possibly could be predicted, the chances that it would happen were low. And to answer the question "why it was difficult to find?" - the testing environment (the beta cluster) did not show the issue. Only when the Suggested edits module was deployed to seven wikis and post-deployment testing was done, the issue was found.

In the same nomination - "Difficult to find" - could be the issue for articles with the template {{Italic title}} (https://phabricator.wikimedia.org/T216044) that would break ContentTranslation workflows or https://phabricator.wikimedia.org/T274619 when enabling Homepage will remove "More" menu from User page, i.e.

User page with enabled Homepage	User page - Homepage disabled
More is not present	More is present

In fact, there is a not-so-small subgroup of "Difficult to find" bugs that could be titled "Regrettable regression bugs". The subgroup includes bugs that unxpectedly appear in places that should not be affected by new changes, or the bugs that could be found only in production.

The "Unexpected impact" group has issues, arguably, the most entertaining ones. The first example would be an issue ironically involving bug images (a pure coincidence!). The animated gif below shows that selecting a bug image (Special:MediaSearch with filters shows that there is only one of image returned) and clicking on it makes the image multiply:

Another example of a bug with "Unexpected impact"- https://phabricator.wikimedia.org/T280616. On an article with suggested Add links, a user clicks on 'Edit source'. The page starts reloading and keeps going with reloading, making a page disabled for any further interactions (click on the animated gif to see the action)

The list of examples of bugs in "Difficult to find" and "Unexpected impact" nominations doesn't gets new items too often. But each issue on the list gives a sort of a lesson in exploratory testing, and, in fact, making the testing more efficient.

Idea

I do most of my work on a 27-inch iMac from 2012. It's a good desktop machine.

When traveling, I use a 11-inch MacBook Air from 2014. It's a very small and convenient laptop. It's powerful enough, but barely. I have to be very careful with resources when I'm working on it.

For a while now I was thinking about dual booting the laptop with a Linux, to see how it would behave. I've finally dual booted it with Ubuntu this week. As soon as the machine was mostly functional under Ubuntu, I've decided to test it.

Machines

For reference, here's the table with the two machines, with some relevant data. For readability, I've omitted the data that's the same as in the above row.

Tests

Two things that I do the most are installing npm dependencies (npm ci) and running tests (npm run selenium-test). For this blog post, I'll use MediaWiki Core repository and it's Selenium test suite. The suite is targeting a local MediaWiki. It's running in a local Docker container via MediaWiki-Docker.

Both tests have two variants. One variant will run the command from the operating system (without a container). The other variant will run the commands from a Fresh container. Fresh is *"a fast and ready-to-use Docker container with various developer tools pre-installed."*

RAM

Since a blog post without an image is no fun, I took a couple of screenshots.

Air + Ubuntu + Fresh + npm run selenium-test

Air + macOS + Fresh + npm run selenium-test

The screenshots show that on Ubuntu the machine was using 1.5 GB RAM, while on macOS it was using 3.2 GB RAM. The difference is in the Linux virtual machine that Docker runs in the background on macOS.

Speed

I've tried to keep the environments as simple as possible. When running the commands, only the Terminal application was open. I was primarily interested in how much RAM will be used when running the tests in the Fresh container, since the laptop has only 4 GB. I've also measured how fast each command runs.

I wasn't interested in very good data. I just wanted good enough data for my use case. For that reason, I've only run each command three times. That's just enough data to show any trends, but not so much that I'll get bored while waiting for them to finish. All results are in how much time (in minutes) each command took in different environments.

iMac + macOS

iMac + macOS + Fresh

Air + macOS

Air + macOS + Fresh

Air + Ubuntu

Air + Ubuntu + Fresh

Conclusion

I was expecting to see less RAM used while running the commands on Ubuntu. No surprise there. How much time it takes to run the commands was a big surprise. It is so much faster on Ubuntu, it's amazing.

npm ci takes about half a minute to run on powerful desktop machine running macOS, outside of the Fresh container. It takes more than three minutes to run the same command inside the container.

It takes about quarter of a minute to run the command outside the Fresh container on a non-powerful laptop running Ubuntu. Running inside the Fresh container takes about half a minute. That's the same amount of time the desktop took to run the command outside the container!

npm run selenium-test tells the same story. Running it outside the container on desktop machine running macOS took about a minute. Running it inside the container took about two minutes.

On the laptop running Ubuntu, it took about half a minute outside and about a minute inside the container.

The speed improvement is so obvious, I'm seriously considering dual booting the desktop too.

INTRO
When running QA on emails, the most important things to check are the dynamic content populating correctly for different user types, and ensuring there are no layout issues. Especially in an increasingly mobile world, it's important to review how an email will be displayed on a large variety of devices at once.

OLD PROCESS
In the beginning of our email QA days, the process was simple and very manual, and only allowed for checking how emails were displayed on a few devices. The email marketing platform we use is called Acoustic, which allows us to populate emails with dynamic content and create a storyline of emails to be sent (for example: users that donate on the first email, will not receive any further emails). Inside each email template, we would select send out test email, and select a given test user to send it to. This was a cumbersome process that left a lot of space open for human error, and only resulted in us checking content manually on our own clients.

NEW PROCESS
Recent improvements to email QA software provide testers with a quicker way of reviewing email layout and dynamic content on a wider variety of devices. This is particularly useful when performing QA for email on mobile devices, where screen sizes vary much more than on desktop devices. A new tool we added to our email testing suite, called Email on Acid, allows us to do this in just one click in Acoustic. The Email on Acid tool provides an email, which can then be input in Acoustic, and after just a few minutes, it will display how the email looks on Desktop, Mobile and even Web clients. This is a powerful tool that helps catch even the most edge case errors that may appear on less common devices, which is especially important here at Wikimedia given the our platform is used by so many users around the world.

CONCLUSION
In an increasingly mobile world, it is becoming even more important to have an email QA system that allows for checking how emails are displayed on a wide variety of devices quickly. Email QA has always been a more niche area of QA with fewer specialized tools, but that is changing now and tools such as Email on Acid are a great step in that direction! I havent been able to find any open source alternatives yet but if anyone has any suggestions please add them to the comments!

Environments

This tip works if you're targeting:

• a local MediaWiki
  • MediaWiki-Docker
  • MediaWiki-Docker-Dev
  • MediaWiki-Vagrant
  • Quibble
• a remote wiki
  • Beta cluster
• continuous integration
  • Quibble
  • selenium-daily

logLevel

Change logLevel in tests/selenium/wdio.conf.js from the default error to info.

tests/selenium/wdio.conf.js

exports.config = {
	// Level of logging verbosity: trace | debug | info | warn | error | silent
	logLevel: 'info'
};

Run Selenium tests

npm run selenium-test

Output will contain a lot of useful debugging information.

[0-5] 2021-04-01T15:03:41.952Z INFO devtools: COMMAND deleteAllCookies()
[0-5] 2021-04-01T15:03:41.965Z INFO devtools: RESULT null

[0-5] 2021-04-01T15:03:42.841Z INFO devtools: COMMAND navigateTo("http://localhost:8080/w/index.php?title=Special%3AUserLogin")
[0-5] 2021-04-01T15:03:43.554Z INFO devtools: RESULT null

[0-5] 2021-04-01T15:03:43.566Z INFO devtools: COMMAND findElement("css selector", "#wpName1")
[0-5] 2021-04-01T15:03:43.571Z INFO devtools: RESULT { 'element-6066-11e4-a52e-4f735466cecf': 'ELEMENT-6' }

[0-5] 2021-04-01T15:03:43.577Z INFO devtools: COMMAND elementClear("ELEMENT-6")
[0-5] 2021-04-01T15:03:43.584Z INFO devtools: RESULT null

[0-5] 2021-04-01T15:03:43.612Z INFO devtools: COMMAND elementSendKeys("ELEMENT-6", "User-0.7731870706237638-Iñtërnâtiônàlizætiøn")
[0-5] 2021-04-01T15:03:43.730Z INFO devtools: RESULT null

...

[0-5] 2021-04-01T15:03:45.928Z INFO devtools: COMMAND takeScreenshot()
[0-5] 2021-04-01T15:03:46.224Z INFO devtools: RESULT iVBORw0KGgoAAAANSUhEUgAABLAAAAOECAYAAACxbcj6AAABKWlDQ1BTa2lhA...
[0-5] 
        Screenshot location: /Users/z/Documents/gerrit/mediawiki/core/tests/selenium/log/User-should-be-able-to-log-in-%40daily.png

More information

• MediaWiki documentation on running Selenium tests.
• WebdriverIO loglevel documentation.
• The code from this post is available as Gerrit patch 676397.

Since I was impressed with the depth of presentations at Breakpoint conference (at Browserstack) in July 2021 - see my post Breakpoint at BrowserStack conferences (2020-2021) - reflections and thoughts: Part 1 - Focus on Automation, it was really exciting to see that the next conference (March 2021) would be focusing on the intriguing topic of exploratory testing. The the presentations titles (here is the list of all presentations) looked equally exciting, for example

• "Manual testing is not dead...just the definition"
• "Whole team holistic quality"
• "Contemporary Exploratory Testing"
• "Building a testing culture"

One of the first presentation at the conference - Building a testing culture at Pinterest by Jennifer Uviña - provided details of their success story (and some mistakes that were made) improving the impact of QA team on delivering quality products. Jennifer described the whole experience as a process of going through three stages - Struggling- → Established- → Strong. At the Struggling stage Checklist for change was proposed and, as it turned out, it was extremely helpful. Their checklist had only three points:

• Why do we need a change?
• How our team would measure the success ( what impact on the product quality)?
• Distribute knowledge

Their QA team discovered some specific ways to evolve from the Struggling stage to the Strong stage and Jennifer reviewed what exactly was implemented for their Checklist for change

One of the specific pieces of advice - there don't need to be 10 mln tests - less is more in terms of tests, since every test has its cost in terms of maintenance.

Establishing a testing culture that is impactful on delivering a quality product is not only about team processes. In the end, the ultimate question always is - how exactly to do testing? And immediately the next question - "Do we need to have manual testing at all"? There were three presentations that addressed exactly that - Manual testing is not dead...just the definition, Whole team holistic quality, and Contemporary Exploratory Testing. Every of these presentation resulted in chat exploded - I did not see before that any of the other topics (either on July or March conferences) had so many comments, shared thoughts, and, well, just emotional responses. In a sense, reading the chat was as illuminating as listening to those three absolutely superb presentations.

Manual testing is not dead...just the definition by Erika Chestnut presentation started with describing an elephant in the room - "The rise of test automation has reduced the manual testers' footprint significantly. We have more and more AI and ML tools that are taking on things like visual regression and providing smart test automation. [...] So where does that leave manual testing? " Erika showed how manual testing, as an integrated engineering process, is still essential for the product quality. With a smart term "Humanual testing", Erika stated that "you cannot have an impactful testing without HUMAN. Manual testers speak language of the customers."

The comments from chat were in full agreement with the presentation topic and they were insightful on what value manual testing adds:

• QA becomes Business Analyst most of the time (Mercedes Zendejas)
• (answering to a comment about manual testing being slow and non-effective) [...] the process of testing is the bottleneck - not QA's; 75% of bugs come from bad requirements or conception but some people rely mainly on automation (later) to catch bugs... sorry, too late! (Pascal Brault)

So, when in the next presentation - Whole team holistic quality by Alexandra Schladebeck, the question "Do we still need testers?" had a Jean-Luc Picard facepalm picture, the chat applauded.

Two other presentations - Contemporary Exploratory Testing by Maaret Pyhäjärvi and Agile Tester Toolkit by Julia Pottinger - were focusing on redefining what exploratory testing includes and on having right tools for exploratory testing. As Maaret pointed out "Contemporary exploratory testing raises from the observation in teams that we too often do a bad job at both manual testing and test automation for the outcomes we seek in projects. What does it look like when we frame the two together, through an example of an application, of testing done and automation created?" When we look at the agile development process today, the previous QA practices, such as writing tons of tests, obviously are not applicable anymore. What is not fully realized, however, is that having tons of automated tests are not the answer either. Maaret listed several traps that usually were attributed to manual testing, but the automation testing, if the approach to testing is incorrect, would have exactly the same shortcomings as manual testing.

• Test cases trap
• Bug trap
• Algorithm trap

With a well-known saying "You can't automate well without exploring. You can't explore without automation", the presentation described how to scope exploratory testing correctly by differentiating at what level the exploratory testing is done - a user story level, a feature, and even a software architecture level.

Agile Tester Toolkit presentation by Julia Pottinger had detailed guidance on how to bring together the intelligence of exploratory testing and efficiency of automated testing by building a tester toolkit. As the first step, a simple visualization might help to see what skills a QA engineer has and what would be great to have:

The point #3 "Dedicate time to learn" sounds as one of the most important to me.

To conclude - it was great to see shared understanding of the role of exploratory testing in QA process, it was inspiring to get insightful tips of creating better QA practices, and it was fun to share in chat mutual enthusiasm for better testing. Happy testing, everyone!

“Improving daily work is even more important than doing daily work.”
― Gene Kim, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

I went to two Breakpoint (BrowserStack) conferences - the first one was in July 2020 and the second was in March 2021. The table below lists 6 out of 18 presentation that I watched at the Breakpoint conference in July. The choice which presentations to attend was mostly based on my time availability, so there could be some really interesting presentations that I missed.

Looking at the presentations' titles it's easy to figure out that the conference main theme was Automation Testing. With the motto "Delivering quality software at speed" the presenters shared their understanding of why testing automation is important and gave some insights into valuable lessons they learned from implementing automation testing in their organizations.

Although each presentation by itself probably could be a topic for a separate blog post, I'll focus here on the ideas that resonated with my thoughts on QA processes and on the role automation testing plays in delivering quality in software products.

Todd Eaton ("Transforming Test methodologies: from manual to robust strategies"), strongly proposed QA-Devops integration, stating that a QA engineer should be " a release boss", and "the one who pushes the deployment button". Such integration would help creating an optimal framework to increase test coverage which, in turn, would give an immediate feedback on what state the software is. Alan Richardson ("Automating to augment testing") fully supported that point of view. For him automation testing is a subset of more general automation workflows - basically, it's a part of dev tools for the build processes, pipelines and other scripting activities.

Re-thinking the place of QA automation Alan Richardson showed a classic Agile test pyramid but with "Manual testing" completely removed, replaced with a small cloud of "Exploratory testing" which crowns the pyramid that has all layers dedicated to automation testing.

What about a real-life success story of automation testing? Priyanka Halder( "Modern automation approaches") gave a detailed overview on how automation testing at GoodRx gradually evolved to play a vital role in ensuring the quality of software products. She mentioned 2,000 tests run for regression and about 600 tests that are run as someone pushes the commit to a pipeline. That variety of tests allowed QA engineers there to run A/B testing and campaign-based testing by quickly create test suites based on features or devices, or even on specific users audiences.

Given massive evidence of many successes for automation testing, the two questions remain

• what to automate (and how)?
• what communication processes should a QA team implement to be proactive in automation testing?

Surprisingly, the two questions above are closely related. Benjamin Bischoff ("How we reconstructed our releases at Trivago") described a disconnected triangle or, in his words, "a triangle of doom", when QA team functions separately from Developers and from Test Automation team. He advocated "not automation per se - it's about QA process and people involved into it".
Bischoff's practical advice - "Make your test automation user friendly, that would break the "triangle of doom"" finds a support in Todd Eaton presentation. Todd Eaton described an Automation architect who would define the priorities for developing automation testing tools.
As for the effective communication, Todd Eaton also emphasized that the Test Management with an efficient and user-friendly reporting system must be in place to give immediate and easily-digestible feedback to everyone - managers. devops, developers and, of course, QA engineers. He demonstrated a QA dashboard with a sweeping overview of all tests status and the ability to drill into group of tests or into just a single test to see the detailed info.

Alan Richardson added to the above a checklist with important questions that should be asked before starting automation testing:

• What we can automate?
• What processes do I do and would it help to automate some of that?
• What I am not doing?
• How to balance the time spent on automation with the risk you’re mitigating?

Each of the questions requires a good analysis to grasp what's needed most at the moment and what's needed in the long term.

The lessons learned? As I watched the presentations and was reflecting on what the most important to learn there, the two questions I wrote earlier are still valid but they may be re-prioritized (and rephrased) by putting the communication processes (including test management reporting system) first and adding why? (to automate) as the first question to ask:

• what communication processes should a QA team implement to be proactive in automation testing?
• why/what/how to automate

And I completely agree with what Alan Richardson said concluding his presentation: "Automating is driven by ideas. Don't let tools dictate what you automate.”

Outreachy Round 21 has finished! Vidhi Mody, Soham Parekh and I were mentoring Harriet Ayugi on project Evaluation of Microsoft Playwright.

Project management

In the previous internship (Google Summer of Code, June-August 2020, GSoC) we've used the usual Wikimedia tools. Phabricator for tasks, Gerrit for code hosting and review and Jenkins for continuos integration (CI). It worked well for Vidhi's project but not so well for Soham's project. Since this internship was really similar to Soham's project, we've decided to use GitHub for tasks, code hosting and review and GitHub Actions for CI.

Project

We have roughly split the project into three parts. The setup, the implementation and the report. We estimated each part to roughly 4 weeks. The internship lasted for 12 weeks.

The initial setup, mostly CI, took about 6 weeks. Luckily, implementing the code took about 2 weeks. The report took about 4 weeks. In the end, we had time for everything.

CI

CI setup took longer than expected. We've tried Travis CI first because Harriet was familiar with it, but it wasn't easy to store artifacts, like screenshots. Soham suggested GitHub Actions and it worked great. We've even managed to set up MediaWiki-Docker there.

Local development environment

I've noticed that Harriet could use some improvements to her local development environment, so gave her advice on how to set up VSCode, mostly by installing useful extensions.

• Code Spell Checker
• ESLint
• markdownlint

Mentors

Outreachy was much easier for me to mentor than GSoC. During GSoC, most of the time I was mentoring 2 interns. It was a lot of fun, but also a lot of work. During Outreachy, there were 3 mentors on 1 intern. Most of the mentoring was done by Vidhi and Soham, so things were much easier this time for me.

I think I did a decent job of letting Vidhi and Soham mentor Harriet, but I've sometimes made a mistake of mentoring her too much. I should have focused more on mentoring Vidhi and Soham and let them mentor Harriet. There's a chance we'll co-mentor again in the future. I'll make sure not to mentor the intern myself at all, so Vidhi and Soham could develop as mentors.

Communication

We've continued with the usual communication tools. We've used Google Meet for our almost daily video meeting and Zulip for chat.

Blogging

Harriet wrote a blog post every couple of weeks. She posted a few of those posts at my team blog. I don't think other mentors wrote anything about the internship. (I'll remind them to do so.) For my first article about the internship see Outreachy, September-November 2020.

Report

We've spent a lot of time writing the report. You can see all the discussion at GitHub. There were 199 comments and 25 commits from February 3 to March 1. The report is available at three places.

• Evaluation of Microsoft Playwright at GitHub
• Evaluation of Microsoft Playwright at DEV
• Evaluation of Microsoft Playwright at Phabricator

Table of Contents

• Introduction

• Project Background

• WebdriverIO
  • Sample WebdriverIO Code
  • Mocha]
  • Sample Mocha Code
  • WebdriverIO Advantages
  • WebdriverIO Disadvantages

• Playwright
  • Sample Playwright Code
  • Jest
  • Sample Jest Code
  • Playwright Advantages
  • Playwright Disadvantages

• Playwright Vs WebdriverIO
  • Statistics
  • MediaWiki Core Tests
  • Migrating From WebdriverIO
  • Advantages
  • Disadvantages

• What Next

• Summary

• Acknowledgement

• References

Introduction

Test automation is a way to automate a browser by simulating user actions like clicks. Applications become more complex as they are built on layers and entire networks of sub-systems, including UI and API layers, external databases, networks, and even third-party integrations. So, there is always a need for thorough testing to be done. This ranges from unit testing to end-to-end testing of applications. Test automation is one of the ways to ascertain the stability and the health of applications. That can lead to the success of the application when in production. The general stability of the application doesn't entirely depend on end-to-end testing but it surely helps to detect bugs in applications, among other benefits.

Below are some of the popular test automation frameworks being used;

• Selenium (WebdriverIO)
• Puppeteer
• Cypress
• Playwright

Playwright maybe the new kid on the block but it's gaining quite a trajectory among software developers and testers.

Project Background

The Playwright evaluation is a result of a need for to check if WebdriverIO is still a good test automation framework compared to some of the best non-Selenium modern test automation frameworks. MediaWiki is implemented in a large number of repositories. So, in order to ensure good code practices across all these repositories, an extensive amount of testing is performed. One of the tests performed is an end-to-end test. WebdriverIO is the current browser automation framework being used for implementing the end-to-end tests. However, with the recent increase in awareness about end-to-end testing, a number of equally competitive non-Selenium solutions have been introduced and one of them is Playwright.

At the time of choosing which automation framework to use in the implementation of the test automation, extensive research was done and an evaluation of WebdriverIO was carried out by Željko Filipin, who is a Senior Engineer in Test at Wikimedia Foundation and also one of my mentors. To verify if WebdriverIO is still a great testing framework for MediaWiki, a number of modern and non-Selenium automation frameworks have been evaluated against WebdriverIO. These evaluations were done by Soham Parekh who was Google Summer of Code 2020 intern and is also one of my mentors and Sim T.H. Tran. These evaluations can be found in the links below;

• Cypress Evaluation by Sim T.H. Tran.
• Puppeteer Evaluation by Soham Parekh.

So, this brings us to the current framework being evaluated against WebdriverIO, which is Playwright. The evaluations progress involved the following steps;

• Configuring and setting up of Wikimedia Core to run on continuous integration, in this case on Github Actions.
• Implementation of all MediaWiki Core tests in Playwright.
• Running MediaWiki Core tests in WebdriverIO.
• Analysis of both WebdriverIO and Playwright tests in terms of speed and stability.

WebdriverIO

WebdriverIO is a test automation framework that allows you to run tests based on the WebDriver protocol and Appium automation technology.

WebdriverIO provides the ability and options to run commands in both asynchronous and synchronous operations. For asynchronous operations, you can use the JavaScript usual async/await, however for synchronous, it can be done through node-fibers. Both of these options have their own benefits and issues which can be found on the WebdriverIO website.

Sample WebdriverIO Code

The code snippet below navigates to wikipedia.org in Chrome and checks if the correct title is being displayed.

const { remote } = require('webdriverio');

(async () => {
 const browser = await remote({
        capabilities: {
            browserName: 'chrome'
        }
  });
  await browser.url('https://www.wikipedia.org/');

  const title = await browser.getTitle();
  expect(title).toHaveTitle('The Free Encyclopedia');

  await browser.deleteSession();
})();

Mocha

The WebdriverIO WDIO runner currently supports Mocha, Jasmine, and Cucumber frameworks and WikiMedia Core test is currently using Mocha framework. Mocha is a feature-rich JavaScript test framework running on Node.js and in the browser, making asynchronous testing simple and fun.

WebdriverIO provides an adapter for Mocha which can easily be added as @wdio/mocha-framework.

Sample Mocha Code

The code snippet below navigates to wikipedia.org in Chrome and checks if the correct title is being displayed.

const { remote } = require('webdriverio');

describe('Wikipedia home page', async() => {
  const browser = await remote({
        capabilities: {
            browserName: 'chrome'
        }
  });

  it('should display correct page title', async() => {
    await browser.url('https://www.wikipedia.org/');

    expect(browser).toHaveTitle('The Free Encyclopedia');
    
    await browser.deleteSession();
  });
})();

WebdriverIO Advantages

WebdriverIO offers some great pros such as;

• Extendable, adding helper functions, or more complicated sets and combinations of existing commands is simple and really useful.
• It's compatibility nature enables it to run on the WebDriver Protocol for true cross-browser testing as well as Chrome DevTools Protocol for Chromium-based automation using Puppeteer.
• Rich feature with huge variety of community plugins that allows you to easily integrate and extend your setup to fulfill your requirements.
• Stable features.
• Synchronous implementation of asynchronous browser commands.
• Excellent API documentation.
• Support for modern web and mobile frameworks.

WebdriverIO Disadvantages

WebdriverIO also presents some cons such as;

• Additional effort to set up browser driver with selenium-standalone or ChromeDriver especially in v5 and below.
• Much slower compared to frameworks like Playwright and Puppeteer.

To explore more WebdriverIO capabilities, you can check out the WebdriverIO and MediaWiki documentation.

Playwright

Playwright is a Node.js library to automate Chromium, Firefox, and WebKit with a single API.

While working with Playwright for the past few months, it has come across as easy to install and setup. The package takes care of installing all the browsers (Chromium, Firefox, and WebKit). Capturing screenshots is an out-of-the-box experience. However, video recording requires separate installation of playwright-video and ffmpeg but they all blend in with Playwright seamlessly.

Sample Playwright Code

The code snippet below navigates to wikipedia.org in Chrome and checks if the correct title is being displayed.

const { chromium } = require('playwright');
const expect = require('expect-playwright');

(async () => {
  let browser, page;
  browser = await chromium.launch();
  page = await browser.newPage();
  await page.goto('https://www.wikipedia.org/');

  const title = await page.$('.localized-slogan');
  await expect(title).toEqualText('The Free Encyclopedia');

  await browser.close();
})();

Jest

Jest is a delightful JavaScript testing framework with a focus on simplicity. Playwright can be integrated into a project in two ways and one of them is by using the jest-playwright which has rich features like:

• Multi-browser and device (like iPhones with given screen sizes) support.
• jest-dev-server integration which can start your webserver like create-react-app before running the tests.
• expect-playwright integration which provides common expect helper functions.

jest-playwright is added to the jest configuration as a preset which makes all the features available to be used at your disposal. It was inspired by jest-puppeteer.

Sample Jest Code

The code snippet below navigates to wikipedia.org in Chrome and checks if the correct title is being displayed.

const { chromium } = require('playwright');

describe('Wikipedia home page', async() => {
  let browser, page;
  browser = await chromium.launch();
  page = await browser.newPage();

  it('should display correct page title', async() => {
  await page.goto('https://www.wikipedia.org/');

  const title = await page.title();
  await expect(title).toBe('The Free Encyclopedia');

  await browser.close();
})
})();

Playwright Advantages

Playwright offers some great pros such as;

• Test across all modern browsers with single API to automate Chromium, Firefox and WebKit.
• The API can be used in JavaScript & TypeScript, Python, C# and Java.
• It's simple to set up.
• Stable features.
• Bidirectional (events) – automating things like console logs is easy.
• Auto-wait for elements to be ready before executing actions (like click, fill).
• Intercept network activity for stubbing and mocking network requests.
• Seamless integration with Jest.

Playwright Disadvantages

Playwright also presents some cons such as;

• It is very new so the APIs are evolving.
• Has no support for IE11 or non-browser platforms.
• Documentations and community are not as good as the other framework yet.

To explore more on Playwright capability, you can check out the Playwright documentation and my MediaWiki Core fork.

Playwright Vs WebdriverIO

The below table is a comparison between Playwright and WebdriverIO in terms of;

• Performance
• Developer Experience
• Documentation
• Ecosystem/Community

Statistics

The above data was adopted from npm trends showing WebdriverIO and Playwright downloads. From the above charts, we see that WebdriverIO still trumps Playwright when it comes to popularity among the testing community.

MediaWiki Core Tests

The above visualized chart represents data that was collected by running MediaWiki Core tests in WebdriverIO and Playwright. A total of 10 tests were run using both frameworks on Github Actions platform.

The tests were run 40 times in both frameworks to ascertain stability and reliability. No flakiness and failures were encountered during the tests run. As shown above, there is consistency in time when the tests were run in WebdriverIO and Playwright.

The above chart further shows that Playwright is much faster than WebdriverIO in terms of speed.

Migrating From WebdriverIO

From the various evaluations done with Playwright, there seems to be great potential replacement in the future. Playwright seem to stand out in terms of both speed and stability. However, the question being asked is, is it worth migrating from WebdriverIO to another test automation framework? Let’s look at the advantages and disadvantages of a potential migration.

Advantages

• WebdriverIO is JavaScript based and is built over Node.js just like Playwright thus making it easier for developers to stick to the same programming language in the face of a potential switch to Playwright.
• The MediaWiki Core test implementation follows the Page Object Pattern, this would make it easy to write Playwright tests using the existing structure.
• Playwright has a number of useful features that come with it like the reset helper functions and single API to automate Chromium, Firefox and WebKit, among others. Some of these features are missing in WebdriverIO thus making Playwright a better option for a switch.

Disadvantages

• Investment in WebdriverIO tool. Most tests are already written in WebdriverIO. About 30 MediaWiki repositories use it, so migrating from it to another tool would require a great deal of time and manpower.
• Update of documentation. Most documentation is already written and has references to WebdriverIO. Updating the documentation would require some time.
• WebdriverIO incorporates Puppeteer as second automation driver tool allowing for the extra features which pretty much make up for some of the features that Playwright has, hence no much point in switching to Playwright.

What Next

• Refactor existing tests.
• Evaluate Puppeteer.

Summary

As of the above statistics and comparison, Playwright is a better alternative tool over WebdriverIO in terms of speed. However, WebdriverIO is a great tool given that it keeps evolving, new and modern features are being added to it and the community is still vibrant and supportive. With the recent release of WebdriverIO v7 which has some great feature update, and the fact that WebdriverIO incorporates a framework like Puppeteer as second automation driver tool makes WebdriverIO a powerful test automation framework. There is also a potential integration with Playwright in the near future. As of the time of the evaluation, there are no plans to switch to a different test automation framework but this could change in the future.

Acknowledgement

The completion of this project wouldn't have been a success without the support of my mentors, Željko Filipin, Vidhi Mody and Soham Parekh. I do appreciate the constant guidance, meetings, advice, code reviews, challenging, pair programmings, to mention but a few, that were offered during the project execution.

References

• Puppeteer, Selenium, Playwright, Cypress – how to choose? by Benjamin Gruenbaum.
• Using Jest with Playwright by Max Schmitt.
• State of JS 2020: Testing by Sacha Greif and Raphaël Benitte.
• Playwright Community.

“I felt in need of a great pilgrimage, so I sat still for three days.” Hafiz

In Testing instrumentation on Special:Homepage (QA perspective) the general, step-by-step testing approach was described and Special:Homepage was used as a specific example.

Special:Homepage with the SE module is enabled by default for most of the new accounts on participating wikis, e.g. testwiki, cswiki, svwiki, or arwiki. On betacluster Special:Homepage is enabled not only on the mentioned wikis but also on enwiki. For existing user accounts Special:Homepage can be enabled via Special:Preferences option Newcomer homepage in User profile tab settings.

This post adds more details on QA process for error validation. Let's start with a story.
On the Special:Homepage a banner module was added to inform users visiting Special:Homepage on new community events or campaigns.

However, when the banner module was deployed, the issue was reported - HomepageModule events with validation errors. Since the QA task was to verify whether the event - a user clicking on the banner - is correctly recorded, the question is - why validation errors were not caught during the testing?

There are several places where errors (or overlooks) might happen. To minimize failures and ensure that events are recorded as expected, the testing should happen in stages. The first step is to check the schema itself - the HomepageModuleschema description page to see if the banner module, indeed, has been added and the expected actions - including link-click - are listed:

Then, the testing should start with triggering users' action - visiting Special:Homepage and clicking on the banner and see if the event will be recorded. There are couple of extremely helpful javascript snippets that can be entered into the Console and the events can be viewed.
1. Showing all instrumentation schemas on a page (credits to @Catrope)

mw.loader.using( 'mediawiki.api' ).then( function () { new mw.Api().saveOption( 'eventlogging-display-web', '1' ); } );

Here how it looks - Special:Homepage has few schemas:

2. Showing HomepageModule schema events on a page (credits to @kostajh)

mw.trackSubscribe('event.HomepageModule', console.log);

The HomepageModule schema events will be visible in the Console.

Persistent log checkbox should be checked, otherwise the event of clicking on the banner won't be present in the Console.

The banner related event(s) can be filtered and reviewed for completeness.

The front-end instrumentation testing for the new banner module is finished. Does it mean that the event validation is all good too? The front-end testing can be done reliably in betalabs but the validation happens differently and quite often cannot be seen from the front end. Here is a twist - presently betalabs does not have a convenient way to check for the eventlogging validation errors. To see the validation errors for the schemas only production logstash can present the full pictures for the eventlogging validation.
.
What is logstash? From the wikitech page

Logstash is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

And from the following logstash introduction

Logstash is an open-source, centralized, events and logging manager. It is a part of the ELK (ElasticSearch, Logstash, Kibana) stack.

All the above sound really helpful for doing proper instrumentation QA, but the betalabs logstash presently is unable to provide any information on eventlogging validation errors. The only option to see what gets stored for betalabs eventlogging is to watch the following stream (and grep for getting the event info ) for the events that are being tested:

@deployment-eventlog05:~$ kafkacat -C -t eventlogging_HomepageModule -b deployment-kafka-jumbo-2.deployment-prep.eqiad.wmflabs:9092 -c 10 -o -10

As a result, the close monitoring the production logstash becomes really important. As soon as the deployment of updated schema is done to testwiki, the eventlogging testing should be rechecked to make sure that front-end actions are not resulted in validation errors. The logstash dashboard - eventgate-validation is a perfect reporting tool and should be checked during testing instrumentation on production.

Read more:

1. Event Platform/Instrumentation How To - specifically, how to check eventlogging on betacluster,
2. Diving into Wikipedia's ocean of errors - a developer's perspective on how to keep the level of reported errors in logstash manageable
3. Wikimedia Product/Better use of data

The secret of getting ahead is getting started. The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks, and then starting on the first one.
—Mark Twain

Prep work

Search is sine que non - a necessary component for making information accessible, so users and communities across wikimedia projects could effectively "share in the sum of all knowledge". Since Special:MediaSearch offers a new way of searching media (image, audio, video) and other files and pages, it is critical to have a thorough QA analysis of how well the new search answers users' expectations. However, the question of what makes the search efficient or whether the search works well at all is not an easy one. For example, the Measuring User Search Satisfaction research has found the quality of the search performance quite challenging to assess:

"On desktop alone, 64 million searches are made a day [...]. This makes measuring how well the system is doing of paramount concern, and we don't currently have a good way of doing that."

Well, what exactly could be tested then? What aspects of the search functionality are to be tested and how they should be tested?

Following the first part of Mark Twain's advice - The secret of getting ahead is getting started - we can start with understanding of what MediaSearch is supposed to do. The following description (see Commons:Structured data/Media search) gives a concise review of the context for MediaSearch and, more importantly, it gives specific guidelines for starting to map the goals for the high-level QA analysis:

Special:MediaSearch is an alternative, image-focused way to find media on Commons that is being developed. MediaSearch uses categories, structured data and wikitext from Commons, and Wikidata to find its results.

Looking closely at the description and breaking it into smaller chunks (following the next part of Mark Twain's advice -The secret of getting started is breaking your complex overwhelming tasks into small manageable tasks), the QA analysis for the future testing gets some specificity. The table below maps specific parts of the description to the high-level QA tasks.

Now the Special:MediaSearch page looks as a test map with the areas for testing:

Testing search

After breaking the big task (the task of testing Special:MediaSearch page functionality) into manageable tasks, we need to start on the first one - testing the search. The drawing below provides an overview of segmenting the search test cases. The segmentation has resulted from reading documentation - Help:Searching and Help:CirrusSearch and from previous experience with testing the search functionality for the wikipedia apps.

Two previous posts - Exempla Docent Part 1 and Part 2 outlined QA approach to testing functionality of the Suggested edits (SE) module on Special:Homepage. Part 1 explored testing ORES model-articletopic logic implementation and Part 2 described testing user workflows.

This post, as a Part 3 of Exempla Docent, presents another important aspect of QA work - the task of testing instrumentation. While analyzing data is a complex task which would be addressed by data analytics, there are still quite a few points that would be important to check before the instrumentation would be deployed to production. Let's look at a specific example - Schema:HomepageModule on Special:Homepage.

Note: Special:Homepage with the SE module is enabled by default for most of the new accounts on participating wikis, e.g. testwiki, cswiki, svwiki, or arwiki. On betacluster Special:Homepage is enabled not only on the mentioned wikis but also on enwiki. For existing user accounts Special:Homepage can be enabled via Special:Preferences option Newcomer homepage in User profile tab settings.

The Step zero should be, of course, getting a good understanding of the user workflows on the Special:Homepage. Understanding First Day - full user flow gives a good overview of that. EventLogging/TestingOnBetaCluster documentation page is also a good reference point for getting in-depth info on EventLogging on the betacluster.
Now, the Step one would be going through Schema:HomepageModule page. The page has full information on the actions that are included in the schema along with some description and other helpful info (e.g. impression: Module is shown to user or link-click: User clicks a link in the module). These descriptions are basically ready-to-go tests.

Schema	Action to check	Tests	Description
	impression	Test 1	when a page loads, the impression event should be recorded for each module
	link-click	Test 2	click on a link(s) present in the modules

Step two - using the test scenarios from Step one, perform actions on the Homepage. The browser dev tools console allows us to see the recorded events, e.g. testing Schema:HomepageModule includes the following steps:

• enable your browser's dev tools and enable "persistent log" in the Console

• use the following snippet (credit to @kostajh)

mw.trackSubscribe('event.HomepageModule', console.log);

• perform the actions that trigger eventlogging and compare with the info documented in the relevant schema description

As a result, the Console will display all recorded events with detailed information - e.g. see the info for the impression event:

Step four - quite often it makes sense to check the logs directly. For the server-side schemas, e.g. EditorJourney, it is, in fact, the only option for checking if the events are being recorded. The instructions include the following simple steps:

• log in and go to the directory with logs:

ssh deployment-eventlog05.eqiad.wmflabs

cd /srv/log/eventlogging

• to verify that events have been properly logged you may use grep to find specific records. For example, to see which editor_interface events were recorded, you may use:

grep '"schema": "HomepageModule"' all-events.log|grep 'impact'|grep 'link-click'|grep '"is_mobile": true'

What can be checked on the betacluster? The tests should verify, at the very basic, that

• there are no errors in the recorded events
• all actions and all info (according to a schema description) are present
• no missing/duplicate actions
• and, overall, the logic of tested user workflows is reflected accurately in the recorded events

Even with limitations of betalabs environment, checking instrumentation (the schemas) before deployment to production might give valuable insights and allows to catch bugs and errors or to modify, if it's necessary, the schema itself. In the context of Growth team projects the instrumentation plays a crucial role for answering important questions on how features are being used and to make decisions on how to improve the Special:Homepage, so newcomer editors would easily find editing tasks and successfully publish their first edits.

It has been more than 3 weeks into my Outreachy internship with Wikimedia foundations. The internship started well and the project that I'm working on is about evaluating Microsoft playwright as a possible replacement to the current automation testing framework being used. Week one was mostly about setting up the Wikimedia core by forking and cloning the Wikimedia core repository from Github. In order to simplify continuous integration, we are using Github as our code hosting platform to evaluate playwright instead of using Gerrit. The setup involved the following steps;

• Forking the Wikimedia repository
• Cloning the repository, setting up and running it on my local machine
• Connecting my forked with upstream
• Configuring CI.

Week two followed up with more configurations and installations of dependencies, configuring CI to work with docker, run test, etc. The configurations and setup continued into week 3, I continued with setting up and configuring Playwright, jest, etc. On the third week, my mentors and I realized that Travis CI might not be the best CI tool to use for continuous integration in evaluating playwright because of a couple of reasons;

1. Travis CI doesn't store artifacts out of the box, it requires additional setup and configurations such as using AWS, Github pages, or any other external services to store artifacts.
2. travis-ci.org is shutting down soon and will be migrating to travis-ci.com. So, to simplify the workflow and configurations, we decided to use Github actions as the CI pipeline for the project.

The change to using Github actions meant that I had to abandon Travis CI and move to Github actions whereby, I had to take a little bit of time to learn and get familiar with how Github actions work. Moving to Github actions has been a great idea because I'm seeing the tests and linter run much faster and we are now able to store artifacts by just adding minimal configurations. This has come with its own challenges like learning how to use GitHub actions and falling behind the project milestones and deadlines. I have struggled with a number of issues like;

• How to work with the forked repository, updating upstream with the forked repo. There was a time when I updated the upstream with my master and immediately rebased with a feature branch, this added all the commits from upstream and messed up all my commits, we had to abandon the pull request and created a new branch. I learned from this. It has been a bit of a challenge and learning when it comes to working with a forked repo but with the help of my mentors, I'm getting past this.
• I have had several issues with setting up and configuring jest with Microsoft playwright, but my mentors have been there to help unblock me.

All in all, I have learned so much so far within this short time and have been able to make progress with the project. I'm also putting in extra effort to ensure that I hit the project milestones in time. My mentors have been very helpful in pointing me in the right direction whenever I get stuck or have issues with the tasks I'm working on.

The year 2020 has been a year of massive change in the entire world, there are mixed feelings of loss, confusion among others, but all in all, there is always hope that keeps us moving forward. I must say that being accepted as an Outreachy intern has been that ray of light at the end of the tunnel that I needed to end the year and begin the new year. Outreachy is a paid, remote internship program with the goal to support people from groups underrepresented in tech. Starting my career in the field of software engineering has been a journey of hard work, persistence, and seizing every opportunity since where I come from such opportunities are rare and the support for women's engagement in technology is quite low.

My journey into being accepted as an outreachy intern started when I submitted my application about 3 months ago, the whole
Outreachy application takes approximately 3 months. During the application process, I got the opportunity to contribute to open-source projects, which is one of the requirements of the initial application process. Contributing to open-source projects is something new to me, I had never contributed to any open-source project before even though I have been coding for a while now. This is a completely new path for me in my career as a Software Engineer and it's one I'm truly excited to embark on, though a bit scary but I know that great things will come out of it!.

As part of the outreachy program, I got accepted as an intern at Wikimedia as a Software developer intern, which will last for 3 months, and December 1st marks my day as an official Wikimedia intern. Excited!!. Wikimedia is a global movement whose mission is to bring free educational content to the world. One of the reasons why I'm passionate about technology is the impact it makes in the lives of people and in the world at large, the kind of impact I see Wikimedia making by providing free educational content. I'm thrilled to be working with such a community of great and smart minded people and continue to make an impact by building and contributing to open source. I have the privilege to be mentored by Vidhi Mody, Željko Filipin, and Soham Parekh whom I look forward to learning from and collaborating with.

Keep an eye on this corner for more blog posts and adventure! I will be blogging about my experience with the new technology I will be using, Outreachy, internship experience at Wikimedia, Open source projects, to mention but a few.

This post was originally published on dev.to

In Part 1 of Exempla Docent for QA practices, some approaches to testing ORES model articletopic were explored. This post, as Part 2, will present an overview on testing Suggested edits module (SE) - the UI that presents the ORES articletopic logic to users (more info on Newcomers tasks on Special:Homepage).

Note: Special:Homepage with the SE module is enabled by default for new accounts on participating wikis, for example, testwiki, cswiki, svwiki, or arwiki. On betacluster Special:Homepage is enabled not only on the mentioned wikis but also on enwiki. For existing user accounts Special:Homepage can be enabled via Special:Preferences option Newcomer homepage in User profile tab settings.

From a practical point of view, there are two huge venues of testing:

• checking if all UX design specs are in place (including testing user workflows)
• checking the essentials - cross-browser testing, mobile testing, translation, accessibility

... and make sure that everything makes sense!

The scrutiny of exploratory QA testing starts with understanding of the problems that we need to solve as well as with the analysis of whether the implementation delivers the solution. What are the Suggested edits? The Suggested edits module provides guidance and support to newcomers from selecting a task (an article for edit) to publishing the edits. A newcomer workflow would include the following steps:

Intro/selection of topics and difficulty levels	Suggested edits	Edit Suggested edits	Post-edit dialog

Well, it's a lot to test. To illustrate QA analysis process, let's get the simplest example - I'll use the first screen of the intro tour for the SE module. This is what newcomers would see when they come to Special:Homepage - the intro tour gives newcomers some general information along with options to select topics of interests.

Overall view	QA analysis view

QA analysis view includes both testing venues mentioned above - checking UX design specs and checking the essentials, thus providing a sort of the mental map that helps to prioritize and structure the testing.

Let's proceed to the next level. There are quite a few ways for the user to interact, and all of them should be tested, of course. For example, testing the selection of the SE topics gives a whole cluster of possible tests illustrated in a table below:

Plus, there is a counter! The counter shows how many articles match users' selection. What if an article fits in two topics? Would it be counted twice? If it is not counted twice, would it be confusing for a user to see that the number of articles for several topics is lower than the number of articles for one topic?

Such questions are, in fact, the essence of the exploratory testing- "Unchallenged assumptions are dangerous". And the best QA practice is to remember that. Happy testing, everyone.

October 26-29 2020 was my team's second virtual offsite. We've had many offsites, but the first virtual one was in May 2020. The structure of this offsite was similar to the one in May. About four hours of sessions every day, from Monday to Thursday.

I've participated in several sessions. For example, Blog Post: Spin up a basic local Phabricator instance with Docker and T266622: Screenshot Tests, Automated Testing, and Data Driven Testing. I've lead a session on writing a blog post. The last day we've played a few games together, including Trivia Murder Party and Among Us.

In Blog Post: Google Summer of Code, June-August 2020 I've said:

I’ve decided to take a break from mentoring. (I might not take that advice. More about that soon.)

I didn't take my own advice. I will be mentoring @Harriet during the Outreachy internship. Her project is T266841: Outreachy 2020 - 2021 Proposal: Evaluate Microsoft Playwright as replacement for our browser automation. My co-mentors are previous Google-Summer-of-Code interns, @Vidhi-Mody and @Soham.

At one of our last meetings during Google-Summer-of-Code internship, I've suggested to @Vidhi-Mody and @Soham that mentoring could be a great next step in their careers. I didn't expect they would take my advice so seriously and act on it so quickly. They have noticed that Outreachy internship is soon and they have suggested two projects, T263221: Selenium Framework Cleanup and T262682: Evaluate Microsoft Playwright as replacement for our browser automation. We had 3 applicants for the first project, and 6 applicants for the second one. After a lot of thought and discussion we have decided to mentor @Harriet for the second project.

This time the application phase of the internship was not that busy for me (compared with the previous internship) because the vast majority of the work was done by @Vidhi-Mody and @Soham. This time I will mentor them, and they will mentor @Harriet. We've already met with her and discussed the internship in general and next steps.

If not before, I will write a review of the internship in March 2021, after it is over.

Links

• Playwright at playwright.dev
• Outreachy December 2020 to March 2021 Interns at outreachy.org
• Outreachy/Round 21 at mediawiki.org

ORES provides machine learning as a service for Wikimedia projects. The ORES model articletopic was used for the Growth team project - Suggested edits for newcomers on Special: Homepage.

Testing ORES models always presents several challenges

• understanding the background
• testing the model logic
• UI testing to see how the specific model logic is presented to users

Understanding background and terms

Special:RecentChanges and Special:Watchlist were the previous projects where ORES models damaging and goodfaith were tested. Another project - Special:NewPagesFeed (PageTriage) involved testing articlequality ORES model. Testing those models was insightful and quite helpful when articletopic model was used for the Homepage Suggested edits project.

The starting point for all ORES models testing is to understand the most basic terminology that machine learning tools are using - recall and precision.
The definitions include some deceptively easy looking formulas:

Recall = True Positive/(True Positive + False Negative)
Precision = True Positive/(True Positive + False Positive)

There are many good explanations (e.g. Wikipedia article - Precision and recall), but I was thinking of an example that would help me to fully grasp these important concepts.
My example:
There are two candidates for a QA position who need to perform a technical task. The task includes testing an application that has 10 intentional bugs for them to find.
Candidate #1 finds two bugs (True Positives) and reports them. Using the formulas above
Recall= True Positive/(True Positive + False Negative) = 2/(2+8) = 0.2 (20%)
The eight bugs that were not found by the Candidate #1, are, in fact, true positive outcomes that were rejected. 20% is the fraction of relevant instances that have been retrieved over total relevant instances. It doesn't look as an impressive result, right? However, when we look at the precision (also defined as the fraction of relevant instances among the retrieved instances), Candidate #1 looks much better:
Precision= True Positive/(True Positive + False Positive) = 2/(2+0) = 1 (100%)
Candidate #1 has excellent precision!
Now, Candidate #2 finds all 10 bugs (True Positives) but also reports 10 bugs that are not true bugs (False Positives).
Recall = 10/(10+0) = 1 (100%)
Precision = 10/(10+10) = 0.5 (50%)

And now, we are facing a tough choice - which one of the candidates is better? Candidate #1 who did not find many bugs but is very precise, or Candidate #2 who finds much more bugs but also submits many false bug reports? Well, it turns out to be a typical problem in machine learning. Finding the balance between recall and precision is almost always a tricky process of adjusting and re-calibrating machine learning models.

How to apply all of the above to practical purposes of testing ORES models? Let's review what ORES articletopic is about.

Testing the model logic

The major challenge the Homepage Suggested edits project faces is - how to suggest edits to newcomers who have not yet edited anything?

ORES articletopic recommendations should be based on (1) selecting articles that have maintenance templates (indicating that some work needs to be done) and (2) such articles' content should match specific topics - e.g. "Physics", "Africa", "Biography (women)" etc. As a non-ORES example, you may enter in the search box - morelikethis:Cat hastemplate:Copyedit - you'll see articles that supposedly relate (somehow to Cat) and have the Copyedit maintenance template. The result might be sometime quite surprising.

Here is the sample of data analysis (our knowledge of recall and precision comes quite handy here!). And just one more term to learn - a threshold. According to ORES/Thresholds, thresholds are trade-offs between the specificity (precision) and sensitivity (recall).

And - voila! - it's easy now to come with a practical test plan - QA testing should watch out for

• highest/lowest thresholds
• low precision
• low recall
• other anomalies

When it comes to an article (with the great help from User:EpochFail) the scores can be easily viewed:

So having good understanding of fundamental terms and having tools for looking inside scoring data, the task of testing ORES articletopic becomes quite enjoyable. And the testing results have helped to adjust machine learning model to optimize new editors experience by providing them with meaningful tasks that match their interests and areas of expertise.

If you want to experiment with Phabricator and/or the Phabricator APIs, it can be convenient to have a local instance to play with.

A relatively simple way to do this is with the following docker-compose.yml, which uses a Bitnami Phabricator Docker image:

( This isn't configured exactly the same as Wikimedia's Phabricator, but it may be close enough for experimentation purposes. A main objective for this example was to keep configuration to a minimum and the total time to roughly 10 minutes. )

Installation

1. Ensure you have Docker and Docker Compose installed
2. Create a directory called something like phabricator and place a docker-compose.yml in it with the text above ( for some reason 🤷‍♂️ copying the text above is easier if you click through to the paste file P13071 )
3. In the phabricator directory run the docker-compose up command
4. Wait a couple minutes for the Docker images specified in the docker-compose.yml file to be fetched and spun up

Loading

1. Apparently Phabricator really wants its URL to have a subdomain, so for this to work locally you can edit your /etc/hosts file (i.e. sudo atom /etc/hosts) adding this line: 127.0.0.1 phabricator.local.com
2. Now you should be able to load https://phabricator.local.com in a browser to see the login page of your local Phabricator instance
3. You'll also have to instruct your browser that it's ok we don't have a valid certificate for the local instance:

( tap image below for animation showing how this is done with Firefox )

Logging in

This Bitnami Phabricator image come with the following default account:
Username: user
Password: bitnami1

Making a Project, Workboard and Task

( tap image below for animation )

Generating an API Token

If you want to play with Phabricator APIs from the command line, you'll need a token:

( tap image below for animation )

Typically you'd want to keep such a token private, but since this one is specific to your local Phabricator instance it's not going to be of much use to anyone else.

Querying the API from a terminal window

In the Make first Project / Workboard / Task step above, the first task T1 was created. A fairly basic Phabricator API endpoint, phid.lookup, can be used to retrieve some basic information about this task, such as its phid, status, full name, etc.

It can be called from a new terminal window on the host system using:

• the url specified above ( http://phabricator.local.com )
• the endpoint ( /api/phid.lookup )
• the token created above ( api-yjnnbnmayjqf5z733qix2ebrri5k - yours will be different )
• the id of the ticket created above ( T1 )

For example, from a new terminal window:

curl http://phabricator.local.com/api/phid.lookup -d "api.token=YOUR_API_TOKEN_HERE" -d "names[0]=T1" | python -m json.tool

Note: | python -m json.tool simply formats the json returned by the endpoint

( tap image below for animation )

Querying the API from endpoint documentation pages

If you use your local instance's URL ( https://phabricator.local.com ) to load the API docs ( https://phabricator.local.com/conduit/ ), as long as you've logged in ( with the user / bitnami1 credentials mentioned above, for example ) you can experiment with calling API endpoints without having to mess with tokens. At the bottom of every API's documentation page there is a form which can be used to enter parameters and call the endpoint:

( tap image below for animation )

Enjoy!

Hopefully this is useful if you ever want to mess around with Phabricator in a worry-free manner :)

I've started using Vim in 2014. I've read a few books, wrote a few articles and gave a few talks on Vim. I really like editing text with Vim, but I was never happy with it's file management, extensions, things like that.

I've noticed that Visual Studio Code (VSCode) is used by slightly more than 50% of developers that participated in 2019 Stack Overflow survey. (I didn't find editors section in 2020 survey.)

I've learned about Neovim in Modern Vim book. I've started using Neovim in 2018 after I've started reading the book.

I've noticed more and more people using VSCode so I gave it a try earlier this year. The only record I have is a commit in my dotfiles from a few days ago, but I know I've been using it for a few months. I liked VSCode in general. It has all the features I need and a lot of extensions. But, after using Vim, editing text using anything else feels like walking backwards. (Editing text using Vim feels like running. Forwards.)

vscode-neovim VSCode extension was all I needed to connect the two. A real Vim implementation (not an emulator!) inside VSCode.

You need three things. VSCode, Neovim and vscode-neovim.

Installing VSCode is simple, download the installer from the site.

Installing Neovim should also be simple. I've installed it on a Mac using Homebrew and all I needed to do was:

brew install --HEAD neovim

--HEAD part is important. It will install version 0.5. If you don't use --HEAD you will get version 0.4. vscode-neovim needs version 0.5.

vscode-neovim is installed from VSCode Extensions. Search for Neo Vim by Alexey Svetliakov.

Links

• Visual Studio Code at visualstudio.com
• Neovim at neovim.io
• vscode-neovim at github.com

Originally published as Visual Studio Code + Neovim at Željko Filipin's blog.

Introduction

June and July were pretty busy. I was on vacation the majority of August. Interns and other mentors were busy even then. For more introduction, read my post Blog Post: Google Summer of Code, February-May 2020.

Things that went well

Code

We use Gerrit for hosting Git repositories and for code review.

@Vidhi-Mody project was updating about 30 repositories to WebdriverIO v6 (from v4 or v5). We are currently at 70 or 80% of all repositories at v6. (Depending on if you count repositories, or relevant lines of code.) She created about 60 commits. In the majority of cases, there were no significant problems with the update, so most of her contributions are already merged, about 40 commits.

@Soham evaluated two WebdriverIO alternatives, Puppeteer (from Google) and Cypress. He also took a quick look at Playwright (from Microsoft) but we decided not to evaluate it during the internship. (I'll probably mention Playwright in another post soon.) He created almost 100 commits. In the end, we decided not to adopt the tools, so we didn't merge the vast majority of his commits. Only 2 commits were merged.

They created a lot of small commits that were easy to review. Small commits are also easy to revert, if a problem is found later. Both students had to revert at least one commit. Vidhi had to revert 615682 and Soham 595932. I hope that experience made it clear why I was insisting on small commits from the start.

Mentors

I've managed to recruit a couple of mentors, @Jpita and @jbolorinos-ctr They were a lot of help, especially while I was on vacation in August. Pita's specialty during the internship was Cypress and John's bug reporting.

Tasks

We use Phabricator for task management.

It's easy to get overwhelmed or distracted, so good task management is important. We used Phabricator Workboards to organize work (Vidhi, Soham).

Blogging

Interns were writing blog posts every few weeks. Check out their blogs (Vidhi, Soham). Each of them even wrote a blog post for my team's blog, Wow. So wikimedia. Such quality. Many testing. Very team. Vidhi wrote Blog Post: GSoCpedia: The journey so far and Soham wrote Blog Post: Fanboying Cypress.

Communication

We had daily video meetings. When there was a problem to resolve, one of us would share the screen. That helped to resolve a lot of problems quickly.

We've used Zulip for text messages. That helped us stay in touch during the day, or on rare days when we didn't have a meeting.

Interns were invited to my team's IRC channel and weekly video meeting. They were also invited to our informal Telegram group and weekly watercooler video meeting. The group and the meeting are used for non-work related chat. I hope that helped them to see how a team works, and also to spend some social time with us.

They have participated in our virtual team offsite, workshops on test driven development, refactoring and bug reporting.

Documentation

Vidhi created the majority of the documentation on how use MediaWiki-Docker for various extensions (T256239: Run Selenium tests targeting MediaWiki-Docker). MediaWiki-Docker is the closest thing we have to the official local development environment.

Upstream

To help with debugging, Soham worked on recording videos of all test runs. We had a framework that works only on Linux. He found an NPM package that works on all platforms, wdio-video-reporter. When we discovered it supports only WebdriverIO v5 he contributed a patch to the package to make it work with v6.

Things that didn't go well

COVID-19

Both interns had hardware trouble. In normal times, buying a new laptop would not be a problem. During the pandemic it was pretty hard. Both of them managed to resolve the hardware problems somehow.

Local development environment

The biggest problem for Vidhi was that she needed to update about 30 different repositories, but there's is no official local development environment. If there were any problems with the update (for example, something breaks in our continuous integration), it's really hard to debug it, if you can't reproduce the problem locally. To fix the problem, she created the majority of the documentation on how use MediaWiki-Docker for the relevant repositories.

Continuous integration

The biggest problem for Soham was that our continuous integration is hard to debug, because it's hard to replicate locally. If a piece of code works fine on your machine, but fails in continuous integration, it's almost impossible to debug it.

Too many interns

I made a mistake of mentoring two interns. I never had enough time to resolve problems for both of them. If I was mentoring only one, I would have more time to focus on their project.

Mentors with limited experience

Both co-mentors were pretty new both to Wikimedia and to our Selenium/WebdriverIO framework, so most of the time they couldn't help much. I'm not saying mentors didn't do a good job. I think they did a great job. The failure is mine. I should have recruited more mentors with more experience.

Advice for interns

Vidhi and Soham

Both interns had good coding and communication skills. Vidhi's advantage was communication, Soham's coding. I'm not saying their other skills were not good, only that it was obvious each of them was clearly better in one skill.

After the internship, I've suggested that they take a break from coding. A week or two. Instead of coding, they should be thinking about the internship and writing about it.

Another piece of advice I had for them was to try mentoring. (You'll probably hear more about that soon.)

I guess one of the most important things they have learned during the internship is how to create small commits with good commit messages. They have also learned how to create a good bug report. Bug reporting is a skill that they have significantly improved in, but they still need to work on it.

General advice

A piece of advice for all interns: good communication skills are as important as good coding skills, if not slightly more important.

Advice for mentors

Myself

I've decided to take a break from mentoring. (I might not take that advice. More about that soon.) It was a lot of fun, but also a lot of work. Instead, like I've advised the interns, I would like to spend some time thinking about the internship and writing about it.

General advice

A piece of advice for all mentors: prepare to spend more time on mentoring than you've originally planned. I've spent about 10 hours a week per intern.

Originally published as Google Summer of Code, June-August 2020 at Željko Filipin's blog.

In February 2020 I've noticed an e-mail that Wikimedia is participating in Google-Summer-of-Code. Unfortunately, I've ignored the e-mail and soon forgot about it.

In March I've realized that I have a couple of projects that would be good internship projects, T247844: Upgrade WebdriverIO to v6 in all repositories and T247835: Evaluate WebdriverIO replacements for our browser automation framework. I've managed to apply as a mentor in the last responsible moment. I've also managed to recruit a couple of mentors, @Jpita and @jbolorinos-ctr.

I've expected we'll get maybe one student application. I really didn't expect more than that. We got six applications. One student was completely unresponsive, but five were very active. I was overwhelmed with the tasks and commits they have created, and questions they have asked. March and April were very busy.

In May the accepted students were announced. We will be mentoring @Vidhi-Mody and @Soham from June to August. We've spend May preparing for the internship. Both of them are already working on their projects. I've created a status page to have a quick overview of the projects.

Both of them already wrote about the internship. Vidhi wrote GSoCpedia: Chapter One and Soham wrote Road to Ninja ⚔️ and Virtues of a Programmer.

I'm really looking forward to the internship. I hope I will be a good mentor to the students. I hope I will be a good co-mentor to other mentors. I hope the students will learn many skills that will be useful for them in life. How to create small commits, how to write good commit messages, the importance of local development environments, how to write code so it's easy to review...

Originally published as Google Summer of Code, February-May 2020 at Željko Filipin's blog.

Software development is pretty agile in its nature in that things normally tend to move pretty quickly. However, the faster you move, the more things break. As a codebase grows in size, its pieces become more and more complex, with every line adding a potential bug. In Wikimedia Foundation, we keep a handle on this through rigorous amounts of testing. Manual testing requires a lot of effort especially when you have a large core repository with a plethora of plugins and extensions that need to be tested. One of the hot frameworks on the scene is Cypress, a complete end to end testing solution.

We have undoubtedly entered into the era of automated testing. Despite this, web end-to-end testing has always been a tricky beast. Good ol’ Selenium has been the main solution for quite some time now and has a huge history. Despite its impressive browser compatibility, having tests that are more consistent and less flakey can be a non-trivial matter mainly because Selenium was not designed for app testing. Enter Cypress, a modern automated testing tool, which not only promises to fix the old and broken ways of past framework but also make the entire process more developer centric. After spending over two months trying to migrate the tests written in WebdriverIO to Cypress, I wanted to share my journey with Cypress thus far.

What Cypress brings to the Party?

End-to-end testing is notoriously known for its fragmented experience. You need to bring a lot of your own tools, for example, you need:

• a test runner
• an assertion library
• some reporters
• mocks, etc

Cypress includes all of these good stuff out of the box, making setup and configuration dead simple. At the same time, Cypress gives developers an immense amount of freedom when it comes to plugging in things of your own preference. In short,

Cypress takes a batteries included but removable approach

Documentation

A good software should be accompanied by a great documentation for it to be complete.

Cypress has a documentation that is some of the best I have ever read. They have guides on everything that you are likely to encounter. They do a great job telling you how to use the product. In fact, they even have in-depth explanations on the architecture, flakey tests and best practices.

Prototyping

If you have the chance, before adopting anything of this scale, I always think it’s a good idea to test it on a small project first, just to get a feel. Before advocating for it, I added it to my personal blog, just to see how the experience was.

A very simple scenario:

Load an app -> Go to index page -> Click blog link -> Assert content shows up

This was really as simple as writing a few lines of Javascript for the test itself, the npm script in package.json and running it on the CI. I was blown away with how fast it took me, under an hour. I had toiled hard trying to set up video recording with Puppeteer and Jest. Not only did Cypress perform the assertions but also it was recording videos!

Page Object Model in Cypress

Page Object Model is an object design pattern in Selenium, where web pages are represented as classes, and the various elements on the page are defined as variables on the class. All possible user interactions can then be implemented as methods on the class.

Since well-named methods in classes are easy to read, this works as an elegant way to implement test routines that are both readable and easier to maintain or update in the future. Let us take an example of a page model for testing user account creation in WebdriverIO.

"use strict";

class CreateAccountPage {
    get username() {
        return $("#username");
    }

    get password() {
        return $("#password");
    }

    get confirmPassword() {
        return $("#retypePassword");
    }

    get create() {
        return $("#createBtn");
    }

    get heading() {
        return $("#firstHeading");
    }

    open() {
        browser.url("https://some_url.com/account/create");
    }

    createAccount(username, password) {
        this.open();
        this.username.setValue(username);
        this.password.setValue(password);
        this.confirmPassword.setValue(password);
        this.create.click();
    }
}

module.exports = new CreateAccountPage();

All you need to do is now call the CreateAccountPage.createAccount method with proper parameters in order to create a user account. Let us have a look at the Cypress version for this:

"use strict";

class CreateAccountPage {
    constructor() {
        this.username = "#username";
        this.password = "#password";
        this.confirmPassword = "#retypePassword";
        this.create = "#createBtn";
        this.heading = "#firstHeading";
    }

    open() {
        cy.visit("https://some_url.com/account/create");
    }

    createAccount(username, password) {
        this.open();
        cy.get(this.username).type(username);
        cy.get(this.password).type(password);
        cy.get(this.confirmPassword).type(password);
        cy.get(this.create).click();
    }
}

module.exports = new CreateAccountPage();

Writing Scenarios with Gherkin

If you have written end-to-end tests before, you may be familiar with Gherkin syntax, used by Cucumber. This is an expressive, English-like way to write test scenarios. It can help with documenting your features and non-developers can contribute to writing test cases. I found a way to integrate this file syntax into Cypress using a plugin.

Feature: Searching on Wikipedia
    As a programmer, I want to search Wikipedia, so that I can learn new things.

    Background:
        Given I am on Wikipedia

    Scenario: Simple Wikipedia search
        When the search phrase "Bayesian"  is entered
        Then results for "Bayesian" are shown

After writing these commands, the plugin will then go to Cypress to actually run the implementations:

import { Given } from 'cypress-cucumber-preprocessor/steps';

Given('I am on Wikipedia website', () => {
  cy.visit('https://en.wikipedia.com');
});

When('the search phrase "Bayesian" is entered', () => {
  ...
});

When('results for "Bayesian" are shown', () => {
  ...
});

Asserting Elements and Best Practices

End-to-end testing is all about making sure elements on the page have the right content. When writing Cypress tests, you will likely be spending 70% of your time selecting elements and peering inside them. Cypress has a standard get() command which exposes a jQuery-like selector to you. This should be familiar to those with some experience in Selenium.

However, a problem with this selector is that it can be used incorrectly and there is no way to enforce (with code) it’s usage. Welcome cypress-testing-library. This plugin exposes a myriad of commands prefixed with find which work similarly to how get() does in native Cypress. All of these commands make for selectors that are resilient to change. This can have a dramatic effect on how your tests stay consistent as your application progresses.

Time for Some Debugging

Debugging end-to-end tests with Selenium can be somewhat of a nightmare. With Cypress, this pain is at an all-time low. One of the biggest focuses of the core product is to be able to debug is one of the more pleasant experiences in your Cypress journey. Like for most things, they have a great guide to get you started.

Browser Support

Unfortunately, if your organization needs to have support for IE11, you are out of luck. The Cypress team has explicitly said they won’t be supporting it. There is an incredible thread on Github that I really hope you read through. It goes into why they are rolling this out slowly and didn’t choose WebDriver from the beginning and wrote their own custom driver. This is the only place where Cypress falls behind Selenium. However, it does support most of the Chromium-based browsers and is starting to roll-out support for Firefox too and this should serve the needs for most organisations.

I highly recommend everyone in the end-to-end testing community to give Cypress a shot. The quick setup, smooth documentation, fluid construction of tests and warm and supportive community will likely make you fall in love with it.

Originally published at https://www.sohamp.dev/blog/2020-06-29-fanboying-cypress

“Imagine a world in which every single human being can freely share in the sum of all knowledge.”
~ Wikimedia Foundation

Foreword

Google Summer of Code (GSoC) is a global program focused on introducing students to open source software development. Students work on a 3 month programming project with an open source organization during their break from university.

As a part of Google Summer of Code, student participants are paired with a mentor from the participating organizations, gaining exposure to real-world software development and techniques. Students have the opportunity to spend the break between their school semesters earning a stipend while working in areas related to their interests.

In turn, the participating organizations are able to identify and bring in new developers who implement new features and hopefully continue to contribute to open source even after the program is over. Most importantly, more code is created and released for the use and benefit of all.

Introduction

I believe that Open source projects are a great way of developing quality products by working together with other developers. I have always wanted to contribute to projects that I myself use to give back to the Community. Back in my school days, I remember using Wikipedia extensively for assignments and projects. While going through the GSoC organizations list, when I came across Wikimedia Foundation, I knew it was the organization I wanted to apply to.

For me, Google Summer of Code was an opportunity to meet the community, exchange ideas, learn new skills play a small role in the open source community, and contribute to the organisation that I love. I immediately started working on drafting my proposal as well as performed all the required micro-tasks to familiarize myself with the code-base.

Ad Astra Per Aspera 🎊

May 4 2020, I got an email with the subject line “GSoC 2020: Congratulations, your proposal with Wikimedia Foundation has been accepted!" 🎉
My joy knew no bounds. I was super excited to get started and was looking forward to a great summer!

The project I will be working on is Upgrade WebdriverIO to the latest version for all repositories. My Google Summer of Code Proposal can be found here.

The Journey so far

A huge shoutout to my mentor, Željko for giving me the best on-boarding experience ever! In such a short duration, the community has really grown on me. One of the major reasons for this is that the organization offers so many opportunities to get involved in the community.

Test Stratergy by RiskStorming with TestSphere 💡

This was a part of the EngProd virtual offsite: micro hackathon. In such hackathons, the main motive is to pick small bugs/tasks and work with people you don’t normally collaborate with.

Testsphere is a card game for testers! It was developed by Beren Van Daele and is sold through Ministry of Testing. RiskStorming is a way to expose the biggest and most important risks to a product or project. It’s a way to discuss and discover things that are often quite scary to talk about. No one really wants to dwell on ways their product could catastrophically fail. The cards provide a nice, fun, colourful vehicle to discuss these.

The projects that we tested were :

1. The Death Star (I wish I was a Star Wars fan to actually understand more of what was going on 😅)
✔ Quality aspects picked: Functionality, Testability, Operations, Resource Management, Disaster Control, Security and Permissions, Performance

2. Wikipedia Donation Banners
✔ Quality aspects picked: Accessibility, Internationalization, User-friendliness, Performance, Security and Permissions, Concurrency

For more details check here.

The WaterCooler 🚰

I always wondered how networking was possible if you are working remotely. But there’s a solution for every problem right? So what if normal workplace scenarios like, heading out to get a glass of water and running into a co-worker at that point of time is not possible. You can always set up a meet to catch up with each others lives. The WaterCooler meet aims to do just that. I personally feel it is a great initiative to learn more about each other and take some time off work.

I would like to share one of the stories I got to hear on the WaterCooler by James. Early in his career he worked in an aviation company in the UK. The project was related to the network connectivity of the airplane during it’s entire journey. The code was written. The code was tested by flying the plane through UK Everything seemed to be working fine. Now the interesting part, during an international journey the network connection was working just fine uptil it suddenly disconnected completely. When the flight landed, the development team was called to figure out what went wrong. Turns out the flight passed through the equator (lattitude: 0) leading to the divide by zero error! 😱

For someone like me, who is just getting started as a developer, I loved interacting with the highly experienced members of the community. (Not kidding, few of them have years of experience greater than what my age is!)

The TDD Workshop 🛠️

This workshop was by far one of the best experiences I’ve had so far. The workshop was about learning basics of Test Driven Development - its rules and Red-Green-Refactor loop. The workshop was very interactive and all the participants worked together while writing code as well as refactoring it. (You can refer to the code for this workshop on GitHub)

Carpe diem ✨

As far as the internship is concerned, I’m quite on track. Currently, the second work period has begun. I still remember the day I had my first commits merged. It was and always will be special, for I had made my first ever contribution to the organization. Since the community bonding period, not only have my coding skills improve, I have even learnt the importance of good commit messages, bug reports and small patches. I have also gained a new sense of professionalism and a clearer view of what it meant to be working as a community! You can view my progress so far here.

Afterword 🔚

The organization has been very helpful and welcoming since the day I began my application process. Whenever I was stuck with something, I have always received the guidance I need however silly my questions might have been. I see myself continuing to contribute to Wikimedia Foundation as a Volunteer even after the program ends. 💜

Originally published at https://www.vidhimody.me/blog/2020-07-04-gsocpedia-the-journey-so-far/