Summary

• Install debugging packages: apt-get -y install php7.4-common-dbgsym php7.4-cli-dbgsym
• curl -o ~/php-gdbinit https://raw.githubusercontent.com/php/php-src/php-7.4.30/.gdbinit
• gdb <your php command>
• Enter run then once the command has failed: bt and zbacktrace

The Beta-Cluster-Infrastructure is a farm of wikis we use for experimentation and integration testing. It is updated continuously: new code is every ten minutes and the databases every hour by running MediaWiki maintenance/update.php. The scheduling and running are driven by Jenkins jobs which statuses can be seen on the Beta view:

On top of that, Jenkins will emit notification messages to IRC as long as one of the update job fails. One of them started failing on July 25th and this is how I was seeing it the alarm (times are for France, UTC+2):

(wmf-insecte is the Jenkins bot, insecte is french for bug (animals), and the wmf- prefix identifies it as a Wikimedia Foundation robot).

Clicking on the link gives the output of the update script which eventually fails with:

+ /usr/local/bin/mwscript update.php --wiki=wikifunctionswiki --quick --skip-config-validation
20:31:09 ...wikilambda_zlanguages table already exists.
20:31:09 ...have wlzl_label_primary field in wikilambda_zobject_labels table.
20:31:09 ...have wlzl_return_type field in wikilambda_zobject_labels table.
20:31:09 /usr/local/bin/mwscript: line 27:  1822 Segmentation fault      sudo -u "$MEDIAWIKI_WEB_USER" $PHP "$MEDIAWIKI_DEPLOYMENT_DIR_DIR_USE/multiversion/MWScript.php" "$@"

The important bit is Segmentation fault which indicates the program (php) had a fatal fault and it got rightfully killed by the Linux Kernel. Looking at the instance Linux Kernel messages via dmesg -T:

[Mon Jul 24 23:33:55 2023] php[28392]: segfault at 7ffe374f5db8 ip 00007f8dc59fc807 sp 00007ffe374f5da0 error 6 in libpcre2-8.so.0.7.1[7f8dc59b9000+5d000]
[Mon Jul 24 23:33:55 2023] Code: ff ff 31 ed e9 74 fb ff ff 66 2e 0f 1f 84 00 00 00 00 00 41 57 41 56 41 55 41 54 55 48 89 d5 53 44 89 c3 48 81 ec 98 52 00 00 <48> 89 7c 24 18 4c 8b a4 24 d0 52 00 00 48 89 74 24 10 48 89 4c 24
[Mon Jul 24 23:33:55 2023] Core dump to |/usr/lib/systemd/systemd-coredump 28392 33 33 11 1690242166 0 php pipe failed

With those data, I had enough to the most urgent step: file a task (T342769) which can be used as an audit trail and reference for the future. It is the single most important step I am doing whenever I am debugging an issue, since if I have to stop due to time constraint or lack of technical abilities, others can step in and continue. It also provides an historical record that can be looked up in the future, and indeed this specific problem already got investigated and fully documented a couple years ago. Having a task is the most important thing one must do whenever debugging, it is invaluable. For PHP segmentation fault, we even have a dedicated project php-segfault

With the task filed, I have continued the investigation. The previous successful build had:

19:30:18 ...have wlzl_label_primary field in wikilambda_zobject_labels table.
19:30:18 ...have wlzl_return_type field in wikilambda_zobject_labels table.
19:30:18 	❌ Unable to make a page for Z7138: The provided content's label clashes with Object 'Z10138' for the label in 'Z1002'.
19:30:18 	❌ Unable to make a page for Z7139: The provided content's label clashes with Object 'Z10139' for the label in 'Z1002'.
19:30:18 	❌ Unable to make a page for Z7140: The provided content's label clashes with Object 'Z10140' for the label in 'Z1002'.
19:30:18 ...site_stats is populated...done.

The successful build started at 19:20 UTC and the failing one finished at 20:30 UTC which gives us a short time window to investigate. Since the failure seems to happen after updating the WikiLambda MediaWiki extension, I went to inspect the few commits that got merged at that time. I took advantage of Gerrit adding review actions as git notes, notably the exact time a change got submitted and subsequently merged. The process:

Clone the suspect repository:

git clone https://gerrit.wikimedia.org/r/extensions/WikiLambda
cd WikiLambda

Fetch the Gerrit review notes:

git fetch origin refs/notes/review:refs/notes/review

The review notes can be shown below the commit by passing --notes=review to git log or git show, an example for the current HEAD of the repository:

$ git show -q --notes=review
commit c7f8071647a1aeb2cef6b9310ccbf3a87af2755b (HEAD -> master, origin/master, origin/HEAD)
Author: Genoveva Galarza <ggalarzaheredero@wikimedia.org>
Date:   Thu Jul 27 00:34:03 2023 +0200

    Initialize blank function when redirecting to FunctionEditor from DefaultView
    
    Bug: T342802
    Change-Id: I09d3400db21983ac3176a0bc325dcfe2ddf23238

Notes (review):
    Verified+1: SonarQube Bot <kharlan+sonarqubebot@wikimedia.org>
    Verified+2: jenkins-bot
    Code-Review+2: Jforrester <jforrester@wikimedia.org>
    Submitted-by: jenkins-bot
    Submitted-at: Wed, 26 Jul 2023 22:47:59 +0000
    Reviewed-on: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/942026
    Project: mediawiki/extensions/WikiLambda
    Branch: refs/heads/master

Which shows this change has been approved by Jforrester and entered the repository on Wed, 26 Jul 2023 22:47:59 UTC. Then to find the commits in that range, I ask git log to list:

• anything that has a commit date for the day (it is not necessarily correct but in this case it is a good enough approximation)
• from oldest to newest
• sorted by topology order (aka in the order the commit entered the repository rather than based on the commit date)
• show the review notes to get the Submitted-at field

I can then scroll to the commits having a Submitted-at in the time window of 19:20 UTC - 20:30 UTC. I have amended the below output to remove most of the review notes except for the first commit:

$ git log --oneline --since=2023/07/25 --reverse --notes=review --no-merges --topo-order
<scroll>
653ea81a Handle oldid url param to view a particular revision
Notes (review):
    Verified+1: SonarQube Bot <kharlan+sonarqubebot@wikimedia.org>
    Verified+2: jenkins-bot
    Code-Review+2: Jforrester <jforrester@wikimedia.org>
    Submitted-by: jenkins-bot
    Submitted-at: Tue, 25 Jul 2023 19:26:53 +0000
    Reviewed-on: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/941482
    Project: mediawiki/extensions/WikiLambda
    Branch: refs/heads/master

fe4b0446 AUTHORS: Update for July 2023
Notes (review):
    Submitted-at: Tue, 25 Jul 2023 19:49:43 +0000
    Reviewed-on: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/941507

73fcb4a4 Update function-schemata sub-module to HEAD (1c01f22)
Notes (review):
    Submitted-at: Tue, 25 Jul 2023 19:59:23 +0000
    Reviewed-on: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/941384

598f5fcc PageRenderingHandler: Don't make 'read' selected if we're on the edit tab
Notes (review):
    Submitted-at: Tue, 25 Jul 2023 20:16:05 +0000
    Reviewed-on: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/941456

Or in a Phabricator task and human friendly way:

• 19:26 Handle oldid url param to view a particular revision
• 19:43 AUTHORS: Update for July 2023
• 19:59 Update function-schemata sub-module to HEAD (1c01f22) for T335583
• 20:16 PageRenderingHandler: Don't make 'read' selected if we're on the edit tab

The Update function-schemata sub-module to HEAD (1c01f22) has a short log of changes it introduces:

• New changes:
• abc4aa6 definitions: Add Z1908/bug-bugi and Z1909/bug-lant ZNaturalLanguages
• 0f1941e definitions: Add Z1910/piu ZNaturalLanguage
• 1c01f22 definitions: Re-label all objects to drop the 'Z' per Amin

Since the update script fail on WikiLambda I have reached out to its developers so they can investigate their code and maybe find what can trigger the issue.

On the PHP side we need a trace. That can be done by configuring the Linux Kernel to take a dump of the program before terminating it and having it stored on disk, it did not quite work due to a configuration issue on the machine and in the first attempt we forgot to run the command by asking bash to allow the dump generation (ulimit -c unlimited). From a past debugging session, I went to run the command directly under the GNU debugger: gdb.

There are a few preliminary step to debug the PHP program, at first one needs to install the debug symbols which lets the debugger map the binary entries to lines of the original source code. Since error mentions libpcre2 I also installed its debugging symbols:

$ sudo apt-get -y install php7.4-common-dbgsym php7.4-cli-dbgsym libpcre2-dbg

I then used gdb to start a debugging session:

sudo  -s -u www-data gdb --args /usr/bin/php /srv/mediawiki-staging/multiversion/MWScript.php update.php --wiki=wikifunctionswiki --quick --skip-config-validation
gdb>

Then ask gdb to start the program by entering in the input prompt: run ⏎. After several minutes, it caught the segmentation fault:

gdb> run
<output>
<output freeze for several minutes while update.php is doing something>

Thread 1 "php" received signal SIGSEGV, Segmentation fault.
0x00007ffff789e807 in pcre2_match_8 (code=0x555555ce1fb0, 
    subject=subject@entry=0x7fffcb410a98 "Z1002", length=length@entry=5, 
    start_offset=start_offset@entry=0, options=0, 
    match_data=match_data@entry=0x555555b023e0, mcontext=0x555555ad5870)
    at src/pcre2_match.c:6001
6001	src/pcre2_match.c: No such file or directory.

I could not find a debugging symbol package containing src/pcre2_match.c but that was not needed afterall.

To retrieve the stacktrace enter to the gdb prompt bt ⏎:

gdb> bt
#0  0x00007ffff789e807 in pcre2_match_8 (code=0x555555ce1fb0, 
    subject=subject@entry=0x7fffcb410a98 "Z1002", length=length@entry=5, 
    start_offset=start_offset@entry=0, options=0, 
    match_data=match_data@entry=0x555555b023e0, mcontext=0x555555ad5870)
    at src/pcre2_match.c:6001
#1  0x00005555556a3b24 in php_pcre_match_impl (pce=0x7fffe83685a0, 
    subject_str=0x7fffcb410a80, return_value=0x7fffcb44b220, subpats=0x0, global=0, 
    use_flags=<optimized out>, flags=0, start_offset=0) at ./ext/pcre/php_pcre.c:1300
#2  0x00005555556a493b in php_do_pcre_match (execute_data=0x7fffcb44b710, 
    return_value=0x7fffcb44b220, global=0) at ./ext/pcre/php_pcre.c:1149
#3  0x00007ffff216a3cb in tideways_xhprof_execute_internal ()
   from /usr/lib/php/20190902/tideways_xhprof.so
#4  0x000055555587ddee in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at ./Zend/zend_vm_execute.h:1732
#5  execute_ex (ex=0x555555ce1fb0) at ./Zend/zend_vm_execute.h:53539
#6  0x00007ffff2169c89 in tideways_xhprof_execute_ex ()
   from /usr/lib/php/20190902/tideways_xhprof.so
#7  0x000055555587de4b in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at ./Zend/zend_vm_execute.h:1714
#8  execute_ex (ex=0x555555ce1fb0) at ./Zend/zend_vm_execute.h:53539
#9  0x00007ffff2169c89 in tideways_xhprof_execute_ex ()
   from /usr/lib/php/20190902/tideways_xhprof.so
#10 0x000055555587de4b in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at ./Zend/zend_vm_execute.h:1714
#11 execute_ex (ex=0x555555ce1fb0) at ./Zend/zend_vm_execute.h:53539
#12 0x00007ffff2169c89 in tideways_xhprof_execute_ex ()
   from /usr/lib/php/20190902/tideways_xhprof.so
#13 0x000055555587de4b in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at ./Zend/zend_vm_execute.h:1714
#14 execute_ex (ex=0x555555ce1fb0) at ./Zend/zend_vm_execute.h:53539
#15 0x00007ffff2169c89 in tideways_xhprof_execute_ex ()
   from /usr/lib/php/20190902/tideways_xhprof.so
#16 0x000055555587c63c in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER ()
    at ./Zend/zend_vm_execute.h:1602
#17 execute_ex (ex=0x555555ce1fb0) at ./Zend/zend_vm_execute.h:53535
#18 0x00007ffff2169c89 in tideways_xhprof_execute_ex ()
   from /usr/lib/php/20190902/tideways_xhprof.so
#19 0x000055555587de4b in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at ./Zend/zend_vm_execute.h:1714
#20 execute_ex (ex=0x555555ce1fb0) at ./Zend/zend_vm_execute.h:53539
#21 0x00007ffff2169c89 in tideways_xhprof_execute_ex ()
   from /usr/lib/php/20190902/tideways_xhprof.so
#22 0x000055555587de4b in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at ./Zend/zend_vm_execute.h:1714
#23 execute_ex (ex=0x555555ce1fb0) at ./Zend/zend_vm_execute.h:53539
#24 0x00007ffff2169c89 in tideways_xhprof_execute_ex ()
   from /usr/lib/php/20190902/tideways_xhprof.so
#25 0x000055555587de4b in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
 at ./Zend/zend_vm_execute.Quit
CONTINUING

Which is not that helpful. Thankfully the PHP project provides a set of macro for gdb which lets one map the low level C code to the PHP code that was expected. It is provided in their source repository /.gdbinit and one should use the version from the PHP branch being debugged, since we use php 7.4 I went to use the version from the latest 7.4 series (7.4.30 at the time of this writing): https://raw.githubusercontent.com/php/php-src/php-7.4.30/.gdbinit

Download the file to your home directory (ex: /home/hashar/gdbinit) and ask gdb to import it with, for example, source /home/hashar/gdbinit ⏎:

(gdb) source /home/hashar/gdbinit

This provides a few new commands to show PHP Zend values and to generate a very helpfull stacktrace (zbacktrace):

(gdb) zbacktrace
[0x7fffcb44b710] preg_match("\7^Z[1-9]\d*$\7u", "Z1002") [internal function]
[0x7fffcb44aba0] Opis\JsonSchema\Validator->validateString(reference, reference, array(0)[0x7fffcb44ac10], array(7)[0x7fffcb44ac20], object[0x7fffcb44ac30], object[0x7fffcb44ac40], object[0x7fffcb44ac50]) /srv/mediawiki-staging/php-master/vendor/opis/json-schema/src/Validator.php:1219 
[0x7fffcb44a760] Opis\JsonSchema\Validator->validateProperties(reference, reference, array(0)[0x7fffcb44a7d0], array(7)[0x7fffcb44a7e0], object[0x7fffcb44a7f0], object[0x7fffcb44a800], object[0x7fffcb44a810], NULL) /srv/mediawiki-staging/php-master/vendor/opis/json-schema/src/Validator.php:943 
[0x7fffcb44a4c0] Opis\JsonSchema\Validator->validateKeywords(reference, reference, array(0)[0x7fffcb44a530], array(7)[0x7fffcb44a540], object[0x7fffcb44a550], object[0x7fffcb44a560], object[0x7fffcb44a570]) /srv/mediawiki-staging/php-master/vendor/opis/json-schema/src/Validator.php:519 
[0x7fffcb44a310] Opis\JsonSchema\Validator->validateSchema(reference, reference, array(0)[0x7fffcb44a380], array(7)[0x7fffcb44a390], object[0x7fffcb44a3a0], object[0x7fffcb44a3b0], object[0x7fffcb44a3c0]) /srv/mediawiki-staging/php-master/vendor/opis/json-schema/src/Validator.php:332 
[0x7fffcb449350] Opis\JsonSchema\Validator->validateConditionals(reference, reference, array(0)[0x7fffcb4493c0], array(7)[0x7fffcb4493d0], object[0x7fffcb4493e0], object[0x7fffcb4493f0], object[0x7fffcb449400]) /srv/mediawiki-staging/php-master/vendor/opis/json-schema/src/Validator.php:703 
[0x7fffcb4490b0] Opis\JsonSchema\Validator->validateKeywords(reference, reference, array(0)[0x7fffcb449120], array(7)[0x7fffcb449130], object[0x7fffcb449140], object[0x7fffcb449150], object[0x7fffcb449160]) /srv/mediawiki-staging/php-master/vendor/opis/json-schema/src/Validator.php:523 
[0x7fffcb448f00] Opis\JsonSchema\Validator->validateSchema(reference, reference, array(0)[0x7fffcb448f70], array(7)[0x7fffcb448f80], object[0x7fffcb448f90], object[0x7fffcb448fa0], object[0x7fffcb448fb0]) /srv/mediawiki-staging/php-master/vendor/opis/json-schema/src/Validator.php:332 
<loop>

The stacktrace shows the code entered an infinite loop while validating a Json schema up to a point it is being stopped.

The arguments can be further inspected by using printz and giving it as argument an object reference. For the line:

For [0x7fffcb44aba0] Opis\JsonSchema\Validator->validateString(reference, reference, array(0)[0x7fffcb44ac10], array(7)[0x7fffcb44ac20], object[0x7fffcb44ac30], object[0x7fffcb44ac40], object[0x7fffcb44ac50]) /srv/mediawiki-staging/php-master/vendor/opis/json-schema/src/Validator.php:1219

(gdb) printzv 0x7fffcb44ac10
[0x7fffcb44ac10] (refcount=2) array:     Hash(0)[0x5555559d7f00]: {
}

(gdb) printzv 0x7fffcb44ac20
[0x7fffcb44ac20] (refcount=21) array:     Packed(7)[0x7fffcb486118]: {
      [0] 0 => [0x7fffcb445748] (refcount=17) string: Z2K2
      [1] 1 => [0x7fffcb445768] (refcount=18) string: Z4K2
      [2] 2 => [0x7fffcb445788] long: 1
      [3] 3 => [0x7fffcb4457a8] (refcount=15) string: Z3K3
      [4] 4 => [0x7fffcb4457c8] (refcount=10) string: Z12K1
      [5] 5 => [0x7fffcb4457e8] long: 1
      [6] 6 => [0x7fffcb445808] (refcount=6) string: Z11K1
}

(gdb) printzv 0x7fffcb44ac30
[0x7fffcb44ac30] (refcount=22) object(Opis\JsonSchema\Schema) #485450 {
id => [0x7fffcb40f508] (refcount=3) string: /Z6#
draft => [0x7fffcb40f518] (refcount=1) string: 07
internal => [0x7fffcb40f528] (refcount=1) reference: [0x7fffcb6704e8] (refcount=1) array:     Hash(1)[0x7fffcb4110e0]: {
      [0] "/Z6#" => [0x7fffcb71d280] (refcount=1) object(stdClass) #480576
}

(gdb) printzv 0x7fffcb44ac40
[0x7fffcb44ac40] (refcount=5) object(stdClass) #483827
Properties     Hash(1)[0x7fffcb6aa2a0]: {
      [0] "pattern" => [0x7fffcb67e3c0] (refcount=1) string: ^Z[1-9]\d*$
}

(gdb) printzv 0x7fffcb44ac50
[0x7fffcb44ac50] (refcount=5) object(Opis\JsonSchema\ValidationResult) #486348 {
maxErrors => [0x7fffcb4393e8] long: 1
errors => [0x7fffcb4393f8] (refcount=2) array:     Hash(0)[0x5555559d7f00]: {
}

Extracting the parameters was enough for WikiLambda developers to find the immediate root cause, they have removed some definitions which triggered the infinite loop and manually ran a script to reload the data in the Database. Eventually the Jenkins job managed to update the wiki database:

16:30:26 <wmf-insecte> Project beta-update-databases-eqiad build #69029: FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69029/

One problem solved!

References:

• T342769: Beta update.php fails due to PHP segfault in libpcre2-8.so.0.7.1
• A similar debugging session I had to do for update.php back in 2021 {T296539#7531235}
• 2006 notes by @tstarling from whom I have learned about gdb and the PHP gdb config: https://wikitech.wikimedia.org/wiki/GDB_with_PHP

If you've submitted patches for MediaWiki core, skins or extensions, you've seen this output in Gerrit:

That is a list of links to each job's console output for a patch that failed verification.

You can see a job that failed at 1m 54s. But jenkins-bot does not post a comment on the patch until all jobs have completed. That means you won't get email/IRC notifications for test failures on your patch until the longest running job completes, in this case, after 14m 57s.[0] ⏳⏱️

With all due respect to xkcd/303... wouldn't it be nice to get notified as soon as a failure occurs, so you can fix your patch earlier to avoid context switching, or losing time during a backport window?

IMHO, yes, and, now it's possible!

⚙️ Get started

Commit a quibble.yaml file (documentation, example patch) to your MediaWiki project[1]:

earlywarning:
    should_vote: 1
    should_comment: 1

The next time that there is a test failure[2] in your repository, you will see a comment from the Early warning bot and a Verified: -1 vote.

Here's an example of how that might look in practice:

So, the bot announces 2 minutes after the patch is updated that there's a problem, with the output of the failed command. The full report from jenkins-bot arrives 14 minutes later.

📚 Further reading

For details on how this works, please see the documentation for the Early warning bot. Your feedback and contributions are very welcome on T323750: Provide early feedback when a patch has job failures (feel free to tag T323750 with patches adding quibble.yaml to your project.)

🙌🏻 Thank you

• @hashar for code & architecture review, pushing through Jenkins configuration updates, and solving some low-level plumbing for T331061: Capture output from failed command and transmit to earlywarningbot
• @Urbanecm for fixing a bug where the bot reported on non-essential pipelines (T331236)

Cheers,
Kosta

[0] An alternative for getting real time progress is to watch Zuul TV

Our code review system Gerrit has several caches, the largest ones being backed up on disk. The disk caches offload memory usage and persist the data between restarts. As a Java application, the caches are stored in H2 database files and I recently had to find how to connect to them in order to inspect their content and reduce their size.

In short: java -Dh2.maxCompactTime=15000 ... would cause the H2 driver to compact the database upon disconnection.

context

During an upgrade, the Gerrit installation filed up the system root partition entirely (incident report for Gerrit 3.5 upgrade). The reason is two caches occupying 9G and 11G out of a the 40G system partition. Those caches hold differences to files made by patchsets and are stored in two files:

An easy fix would have been to stop the service, delete all caches, restart the service and let the application refile the cold caches. It is a short term solution, long term what if it is an issue in the application and we have to do the same all over again in the next few weeks? The large discrepancy also triggered my curiosity and I had to know the exact root cause to find a definitive fix to it. There started my journey of debugging.

They are all empty?

When looking at the cache through the application shows caches are way smaller at around 150MBytes:

  Name                          |Entries              |  AvgGet |Hit Ratio|
                                |   Mem   Disk   Space|         |Mem  Disk|
--------------------------------+---------------------+---------+---------+
D gerrit_file_diff              | 24562 150654 157.36m|  14.9ms | 72%  44%|
D git_file_diff                 | 12998 143329 158.06m|  14.8ms |  3%  14%|
                                               ^^^^^^^

One could assume some overhead but there is no reason for metadata to occupy hundred times more space than the actual data they are describing. Specially given each cached item is a file diff which is more than a few bytes. To retrieve the files locally I compressed them with gzip and they shrunk to a mere 32 MBytes! It is a strong indication those files are filled mostly with empty data which suggests the database layer never reclaims no more used blocks. Reclaiming is known as compacting in H2 database or vacuuming in Sqlite.

Connecting

Once I retrieved the files, I have tried to connect to them using the H2 database jar and kept doing mistakes after mistakes due to my completely lack of knowledge on that front:

Version matters

At first I tried with the latest version h2-2.1.214.jar and it did not find any data. I eventually found out the underlying storage system has been changed compared to version 1.3.176 used by Gerrit.I thus had to use an older version which can be retrieved from the Gerrit.war package.

File parameter which is not a file

I then wanted to a SQL dump of the database to inspect it using the Script java class: java -cp h2-1.3.176.jar org.h2.tools.Script, it requires a -url option which is a jdbc URI containing the database name. Intuitively I gave the full file name:

java -cp h2-1.3.176.jar org.h2.tools.Script -url jdbc:h2:git_file_diff.h2.db'

It returns instantly and generate the dump:

CREATE USER IF NOT EXISTS "" SALT '' HASH '' ADMIN;

Essentially an empty file. Looking at file on disk it created a git_file_diff.h2.db.h2.db file which is 24kbytes. Lesson learned, the h2.db suffix must be removed from the URI. I was then able to create the dump using:

java -cp h2-1.3.176.jar org.h2.tools.Script -url jdbc:h2:git_file_diff'

Which resulted in a properly sized backup.sql.

Web based admin

I have altered the SQL to make it fit Sqlite in order to load it in SqliteBrowser (a graphical interface which is very convenient to inspect those databases). Then I found invoking the jar directly starts a background process attached to the database and open my web browser to a web UI: java -jar h2-1.3.176.jar -url jdbc:h2:git_file_diff:

That is very convenient to inspect the file. The caches are are key value storages with a column keeping track of the size of each record. Summing them is how gerrit show-caches finds out the size of the caches (roughly 150Mbytes for the two diff caches).

Compacting solutions

The H2 Database feature page mentions empty space is to be re-used which is not the case as seen above. The document states when the database connection is closed, it compact it for up to 200 milliseconds. Gerrit establish the connection on start up and keep it up until it is shutdown at which point the compaction occurs. It is not frequent enough, and the small delay is apparently not sufficient to compact our huge databases. To run a full compaction several methods are possible:

SHUTDOWN COMPACT: this request an explicit compaction and terminates the connection. The documentation implies it is not subject to the time limit. That would have required a change in the Gerrit Java code to issue the command.

org.h2.samples.Compact script: H2 has a org.h2.samples.Compact to manually compact a given database, it would need some instrumentation to trigger it against each file after Gerrit is shutdown, possibly as a systemd.service ExecStopPost and iterating through each files.

jdbc URL parameter MAX_COMPACT_TIME: the 200 milliseconds can be bumped by adding the parameter to the JDBC connection URL (separated by a semi column ;). Again it would require a change in Gerrit Java code to modify the way it connects.

The beauty of open source is I could access the database source code. It is hosted in https://github.com/h2database/h2database in the version-1.3 tag which holds a subdirectory for each sub version. When looking at a setting, the database driver uses the following piece of code (code licensed under Mozilla Public License Version 2.0 or Eclipse Public License 1.0):

60     /**
61      * Get the setting for the given key.
62      *
63      * @param key the key
64      * @param defaultValue the default value
65      * @return the setting
66      */
67     protected String get(String key, String defaultValue) {
68         StringBuilder buff = new StringBuilder("h2.");
69         boolean nextUpper = false;
70         for (char c : key.toCharArray()) {
71             if (c == '_') {
72                 nextUpper = true;
73             } else {
74                 // Character.toUpperCase / toLowerCase ignores the locale
75                 buff.append(nextUpper ? Character.toUpperCase(c) : Character.toLowerCase(c));
76                 nextUpper = false;
77             }
78         }
79         String sysProperty = buff.toString();
80         String v = settings.get(key);
81         if (v == null) {
82             v = Utils.getProperty(sysProperty, defaultValue);
83             settings.put(key, v);
84         }
85         return v;
86     }

When retrieving the setting MAX_COMPACT_TIME it forges a camel case version of the setting name prefixed by h2. which gives h2.maxCompactTime then look it up in the JVM properties an if set pick its value.

Raising the compact time limit to 15 seconds is thus all about passing to java: -Dh2.maxCompactTime=15000.

Applying and resolution

7f6215e039 in our Puppet applies the fix and summarize the above. Once I applied, I restart Gerrit once to have the setting taken in account and restarted it a second time to have it disconnect from the databases with the setting applied. The results are without appeal. Here are the largest gains:

The gerrit_file_diff and git_file_diff went from respectively 12GB and 8.2G to 0.5G which addresses the issue.

Conclusion

Setting the Java property -Dh2.maxCompactTime=15000 was a straightforward fix which does not require any change to the application code. It also guarantee the database will keep being compacted each time Gerrit is restarted and the issue that has lead to a longer maintenance window than expect would not reappear.

Happy end of year 2022!

References:

• T323754: Investigate Gerrit h2 cache being way too large
• 2022-11-17 incident report: Gerrit 3.5 upgrade
• H2 Database Features: Compacting a Database
• SettingsBase.java for version 1.3.176
• operations/puppet.git change 865023

Mediawiki developers, have you ever thought, “I wish I could deploy my own code for Mediawiki”? Now you can! More deploys! More fun!

Next time you want to get some code deployed, why not try scap backport?

One Command To Deploy

scap backport is one command that will +2 your patch, deploy to mwdebug and wait for your approval, and finally sync to all servers. You only need to provide the change number or gerrit url of your change.

You can run scap backport on patches that have already merged, or re-run scap backport if you decided to cancel in the middle of a run. scap backport can also handle multiple patches at a time. After all the patches have been merged, they’ll be deployed all together. scap backport will confirm that your patches are deployable before merging, and double check no extra patches have sneaked into your deployment.

One Command To Revert

And if your code didn’t work out, don’t worry, there’s scap backport —revert, which will create a revert patch, send it to Gerrit, and run all steps of scap backport to revert your work. You’re offered the choice to give a reason for revert, which will show up in the commit message. Just be aware that you'll need to wait for tests to run and your code to merge before it gets synced, so in an emergency this might not be the best option.

Extra Information

You can also list available backports or reverts using the —list flag!

If you'd like some guidance on deploying backports, please sign up here to join us for backport training, which happens once a week on Thursday during the UTC late backport window!

Scap Backport In Action

Compare to Manual Steps

For comparison, the previous way to backport would require the user to enter the following commands on the deployment host:

cd /srv/mediawiki-staging/php-<version>
git status
git fetch
git log -p HEAD..@{u}
git rebase

Then, if there were changes to an extension: git submodule update [extensions|skins]/<name>
Then, log in to mwdebug and run scap pull
Then, back on the deployment host: scap sync-file php-<version>/<path to file> 'Backport: [[gerrit:<change no>|<subject> (<bug no>)]]' for each changed file

Example Usage

List backports
scap backport --list

Backport change(s)
scap backport 1234
scap backport https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1234
scap backport 1234 5678

Merge but do not sync
scap backport --stop-before-sync 1234

List revertable changes
scap backport --revert --list

Revert change(s)
scap backport --revert 1234
scap backport --revert 1234 5678

That's all for now, and happy backporting!

How are we doing in our strive for operational excellence? Read on to find out!

Incidents

7 documented incidents in July, and 4 in August (Incident graphs). Read more about past incidents at Incident status on Wikitech.

2022-07-03 shellbox
Impact: For 16 minutes, edits and previews for pages with Score musical notes were slow or unavailable.

2022-07-10 thumbor
Impact: For several days, Thumbor p75 service response times gradually regressed by several seconds.

2022-07-11 FrontendUnavailable cache text
Impact: For 5 minutes, the MediaWiki API cluster in eqiad responded with higher latencies or errors.

2022-07-11 Shellbox and parsoid saturation
Impact: For 13 minutes, the mobileapps service was serving HTTP 503 errors to clients.

2022-07-12 codfw A5 power cycle
Impact: No observed public-facing impact. Internal clean up took some work, e.g. for Ganeti VMs.

2022-07-13 eqsin bandwidth
Impact: For 20 minutes, there was a small increase in error responses for thumbnails served from the Eqsin data center (Singapore).

2022-07-20 eqiad network
Impact: For 10-15 minutes, a portion of wiki traffic from Eqiad-served regions was lost (about 1M uncached requests). For ~30 minutes, Phabricator was unable to access its database.

2022-08-10 cassandra disk space
Impact: During planned downtime, other hosts ran out of space due to accumulating logs. No external impact.

2022-08-10 confd all hosts
Impact: No external impact.

2022-08-16 Beta Cluster 502
Impact: For 7 hours, all Beta Cluster sites were unavailable.

2022-08-16 x2 database replication
Impact: For 36 minutes, errors were noticeable for some editors. Saving edits was unaffected.

Incident follow-up

Recently completed incident follow-up:

Replace certificate on elastic09 in Beta Cluster
Brian (@bking, WMF Search) noticed during an incident review that an internal server used an expired cert and renewed it in accordance with a documented process.

Localisation cache must be purged after train deploy
@Tchanders (WMF AHT) filed this in 2020 after a recurring issue with stale interface labels. Work led by Ahmon (@dancy, WMF RelEng).

Remember to review and schedule Incident Follow-up work in Phabricator! These are preventive measures and tech debt mitigations written down after an incident is concluded.

Highlight from the "Oldest incident follow-up" query:

• T83729 Fix monitoring of poolcounter service.

Trends

The month of July saw 22 new production errors of which 9 are still open today. In August we encountered 29 new production errors of which 10 remain open today and have carried over to September.

Take a look at the Wikimedia-production-error workboard and look for tasks that could use your help.

💡 Did you know?

To zoom in and find your team's error reports, use the appropriate "Filter" link in the sidebar of the workboard.

For the month-over-month numbers, refer to the spreadsheet data.

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

How are we doing in our strive for operational excellence? Read on to find out!

Incidents

There were 6 incidents in June this year. That's double the median of three per month, over the past two years (Incident graphs).

2022-06-01 cloudelastic
Impact: For 41 days, Cloudelastic was missing search results about files from commons.wikimedia.org.

2022-06-10 overload varnish haproxy
Impact: For 3 minutes, wiki traffic was disrupted in multiple regions for cached and logged-in responses.

2022-06-12 appserver latency
Impact: For 30 minutes, wiki backends were intermittently slow or unresponsive, affecting a portion of logged-in requests and uncached page views.

2022-06-16 MariaDB password
Impact: For 2 hours, a current production database password was publicly known. Other measures ensured that no data could be compromised (e.g. firewalls and selective IP grants).

2022-06-21 asw-a2-codfw power
Impact: For 11 minutes, one of the Codfw server racks lost network connectivity. Among the affected servers was an LVS host. Another LVS host in Codfw automatically took over its load balancing responsibility for wiki traffic. During the transition, there was a brief increase in latency for regions served by Codfw (Mexico, and parts of US/Canada).

2022-06-30 asw-a4-codfw power
Impact: For 18 minutes, servers in the A4-codfw rack lost network connectivity. Little to no external impact.

Incident follow-up

Recently completed incident follow-up:

Audit database usage of GlobalBlocking extension
Filed by Amir (@Ladsgroup) in May following an outage due to db load from GlobalBlocking. Amir reduced the extensions' DB load by 10%, through avoiding checks for edit traffic from WMCS and Toolforge. And he implemented stats for monitoring GlobalBlocking DB queries going forward.

Reduce Lilypond shellouts from VisualEditor
Filed by Reuven (@RLazarus) and Kunal (@Legoktm) after a shellbox incident. Ed (@Esanders) and Sammy (@TheresNoTime) improved the Score extension's VisualEditor plugin to increase its debounce duration.

Remember to review and schedule Incident Follow-up work in Phabricator! These are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status on Wikitech.

Trends

In June and July (which is almost over), we reported 27 new production errors and 25 production errors respectively. Of these 52 new issues, 27 were closed in weeks since then, and 25 remain unresolved and will carry over to August.

We also addressed 25 stagnant problems that we carried over from previous months, thus the workboard overall remains at exactly 299 unresolved production errors.

Take a look at the Wikimedia-production-error workboard and look for tasks that could use your help.

💡 Did you know?

To zoom in and find your team's error reports, use the appropriate "Filter" link in the sidebar of the workboard .

For the month-over-month numbers, refer to the spreadsheet data.

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

"Mr. Vice President. No numbers, no bubbles."
— 🔴🟠🟡🟢🔵🟣

Release Engineering's "GitLab-a-thon" sprint for May 10th-24th (roughly) focused on the mechanics of migrating a Wikimedia service to GitLab, setting up a CI pipeline, building container images from that service, and publishing images to the Wikimedia registry. We selected the Blubber project as a good candidate for experimentation:

• Workboard: Release-Engineering-Team (GitLab-a-thon 🦊)
• T301168: Migrate Blubber project to GitLab
• T307536: Build and Run Blubber test variant on GitLab untrusted runners

We evaluated build mechanisms including GitLab's suggested docker-in-docker, Kaniko, Podman, and BuildKit:

• T307599: Investigate alternatives to docker-in-docker for container image creation in GitLab
• T308213: Investigate Kaniko as an option to build CI images
• T307810: Investigate buildkitd instances as image builders for GitLab

We ultimately landed on BuildKit as the least constraining for future options, and the most in line with features we'd like to offer.

We explored a range of options for building and publishing, including variations on:

• Building on runners provisioned on a DigitalOcean Kubernetes cluster and importing to the production registry from some trusted location (contint, for example) by way of a shim.
• Building on trusted runners and publishing to the GitLab Container Registry, then importing to the production registry by way of a shim.
• Building on trusted runners and publishing directly from there to the prod registry, authenticated against GitLab by way of JWT.

We eventually landed on this latter, and work is well underway on implementation: T308501: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token

Other work included implementing CI for Blubber on GitLab (T307534), improvements to user-facing documentation (T307535, T307538), enforcing the allowlist for container images in GitLab CI (T291978), experimentation with the GitLab Container Registry (T307537), and extensive discussions with ServiceOps on GitLab infrastructure.

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

By golly, we've had quite the month! 10 documented incidents, which is more than three times the two-year median of 3. The last time we experienced ten or more incidents in one month, was June 2019 when we had eleven (Incident graphs, Excellence monthly of June 2019).

I'd like to draw your attention to something positive. As you read the below, take note of incidents that did not impact public services, and did not have lasting impact or data loss. For example, the Apache incident benefited from PyBal's automatic health-based depooling. The deployment server incident recovered without loss thanks to Bacula. The Etcd incident impact was limited by serving stale data. And, the Hadoop incident recovered by resuming from Kafka right where it left off.

2022-05-01 etcd
Impact: For 2 hours, Conftool could not sync Etcd data between our core data centers. Puppet and some other internal services were unavailable or out of sync. The issue was isolated, with no impact on public services.

2022-05-02 deployment server
Impact: For 4 hours, we could not update or deploy MediaWiki and other services, due to corruption on the active deployment server. No impact on public services.

2022-05-05 site outage
Impact: For 20 minutes, all wikis were unreachable for logged-in users and non-cached pages. This was due to a GlobalBlocks schema change causing significant slowdown in a frequent database query.

2022-05-09 Codfw confctl
Impact: For 5 minutes, all web traffic routed to Codfw received error responses. This affected central USA and South America (local time after midnight). The cause was human error and lack of CLI parameter validation.

2022-05-09 exim-bdat-errors
Impact: During five days, about 14,000 incoming emails from Gmail users to wikimedia.org were rejected and returned to sender.

2022-05-21 varnish cache busting
Impact: For 2 minutes, all wikis and services behind our CDN were unavailable to all users.

2022-05-24 failed Apache restart
Impact: For 35 minutes, numerous internal services that use Apache on the backend were down. This included Kibana (logstash) and Matomo (piwik). For 20 of those minutes, there was also reduced MediaWiki server capacity, but no measurable end-user impact for wiki traffic.

2022-05-25 de.wikipedia.org
Impact: For 6 minutes, a portion of logged-in users and non-cached pages experienced a slower response or an error. This was due to increased load on one of the databases.

2022-05-26 m1 database hardware
Impact: For 12 minutes, internal services hosted on the m1 database (e.g. Etherpad) were unavailable or at reduced capacity.

2022-05-31 Analytics Hadoop failure
Impact: For 1 hour, all HDFS writes and reads were failing. After recovery, ingestion from Kafka resumed and caught up. No data loss or other lasting impact on the Data Lake.

Incident follow-up

Recently completed incident follow-up:

Invalid confctl selector should either error out or select nothing
Filed by Amir (@Ladsgroup) after the confctl incident this past month. Giuseppe (@Joe) implemented CLI parameter validation to prevent human error from causing a similar outage in the future.

Backup opensearch dashboards data
Filed back in 2019 by Filippo (@fgiunchedi). The OpenSearch homepage dashboard (at logstash.wikimedia.org) was accidentally deleted last month. Bryan (@bd808) tracked down its content and re-created it. Cole (@colewhite) and Jaime (@jcrespo) worked out a strategy and set up automated backups going forward.

Remember to review and schedule Incident Follow-up work in Phabricator! These are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status on Wikitech.

💡Did you know?: The form on the Incident status page now includes a date, to more easily create backdated reports.

Trends

In May we discovered 28 new production errors, of which 20 remain unresolved and have come with us to June.

Last month the workboard totalled 292 tasks still open from prior months. Since the last edition, we completed 11 tasks from previous months, gained 11 additional errors from May (some of May was counted in last month), and have 7 fresh errors in the current month of June. As of today, the workboard houses 299 open production error tasks (spreadsheet, phab report).

Take a look at the workboard and look for tasks that could use your help.
View Workboard

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

Last month we experienced 2 (public) incidents. This is below the three-year median of 3 incidents a month (Incident graphs).

2022-04-06 esams network
Impact: For 30 minutes, wikis were slow or unreachable for a portion of clients to the Esams data center. Esams is one of two DCs primarily serving Europe, Middle East, and Africa.

2022-04-26 cr2-eqord down
Impact: No external impact. Internally, for 2 hours we were unable to access our Eqord routers by any means. This was due to a fiber cut on a redundant link to Eqiad, which then coincided with planned vendor maintenance on the links to Ulsfo and Eqiad. See also Network design.

Incident follow-up

Remember to review and schedule Incident Follow-up work in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status on Wikitech.

Recently resolved incident follow-up:

Reduce mysql grants for wikiadmin scripts
Filed in 2020 after the wikidata drop-table incident (details). Carried out over the last six months by Amir @Ladsgroup (SRE Data Persistence).

Improve reliability of Toolforge k8s cron jobs and Re-enable CronJobControllerV2
Filed earlier this week after a Toolforge incident and carried out by Taavi @Majavah.

Trends

During the month of April we reported 27 new production errors. Of these new errors, we resolved 14, and the remaining 13 are still open and have carried over to May.

Last month, the workboard totalled 298 unresolved error reports. Of these older reports that carried over from previous months, 16 were resolved. Most of these were reports from before 2019.

The new total, including some tasks for the current month of May, is 292. A slight decrease! (spreadsheet).

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

No Tales:

In a fair fight, I'd kill you!
— Well, that's not much incentive for me to fight fair then, is it?

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

We've had quite the month, with 8 documented incidents. That's more than double the two-year median of three a month (Incident graphs).

2022-03-01 ulsfo network
Impact: For 20 minutes, clients normally routed to Ulsfo were unable to reach our projects. This includes New Zealand, parts of Canada, and the United States west coast.

2022-03-04 esams availability banner sampling
Impact: For 1.5 hours, all wikis were largely unreachable from Europe (via Esams), with more limited impact across the globe via other data centers as well.

2022-03-06 wdqs-categories
Impact: For 1.5 hours, some requests to the public Wikidata Query Service API were sporadically blocked.

2022-03-10 site availability
Impact: For 12 min, all wikis were unreachable to logged-in users, and to unregistered users trying to access uncached content.

2022-03-27 api
Impact: For ~4 hours, in three segments of 1-2 hours each over two days, there were higher levels of failed or slow MediaWiki API requests.

2022-03-27 wdqs outage
Impact: For 30 minutes, all WDQS queries failed due to an internal deadlock.

2022-03-29 network
Impact: For approximately 5 minutes, Wikipedia and other Wikimedia sites were slow or inaccessible for many users, mostly in Europe/Africa/Asia. (Details not public at this time.)

2022-03-31 api errors
Impact: For 22 minutes, API server and app server availability were slightly decreased (~0.1% errors, all for s7-hosted wikis such as Spanish Wikipedia), and the latency of API servers was elevated as well.

Incident follow-up

Remember to review and schedule Incident Follow-up (Sustainability) in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status on Wikitech. Some recently completed sustainability work:

Add linecard diversity to router-to-router interconnect at Codfw
Filed by Chris @CDanis (SRE Infra) in 2020 after an incident where all hosts in the Codfw data center lost connectivity at once. Completed by Arzhel @ayounsi and Cathal cmooney (SRE Infra), and @Papaul (DC Ops); including in Esams where the same issue existed.

Expand parser tests to cover language conversation variants in table-of-contents output
Suggested and carried out by @cscott (Parsoid) after reviewing an incident in November. The TOC on wikis that rely on the LanguageConverter service (such as Chinese Wikipedia) were no longer localized

Fix unquoted URL parameters in Icgina health checks
Suggested by Riccardo @Volans (SRE Infra) in response to an early warning signal for TLS certificate expiry. He realized that automated checks for a related cluster were still claiming to be in good health, when they in fact should have been firing a similar warning. Carried out by Filippo @fgiunchedi and Daniel @Dzahn.

Provide automation to quickly show replication status when primary is down
Filed in April by Jaime (SRE Data Persistence), carried out by John @jbond and Amir @Ladsgroup.

Trends

Since the last edition, we resolved 24 of the 301 unresolved errors that carried over from previous months.

In March, we created 54 new production errors. That's quite high compared to the twenty-odd reports we find most months. Of these, 17 remain open today a month later.

In the month of April, so far, we reported 20 new errors of which also 17 remain open today.

The production error workboard once again adds up to exactly 298 open tasks (spreadsheet).

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

♥️ Grape:

Becky: What do you want?
Gilbert: I want Momma to take aerobics classes. I want Ellen to grow up. I want a new brain for Arnie.
Becky: — What do you want for you? Just for you?
Gilbert: I want to be a good person.

Developers should own the process of putting their code into production. They should decide when to deploy, monitor their deployment, and make decisions about rollback.

But that’s not how we work at Wikimedia today, and we on Release Engineering aren’t sure how to get there, so we’ve decided to experiment.

Typically a deployment takes us a full week to complete—the week of March 21st, 2022, we deployed MediaWiki four times.

We called that week 🚂🧪Trainsperiment Week.

📻 Deployment frequency

MediaWiki's mainline branch is changing constantly, but we deploy MediaWiki weekly (kind of). We keep stats that measure how far our main branch is from production.

The trainsperiment changed our deployment frequency, which affected all the other metrics, too. Faster deployment means smaller batch size, and shorter change lead time.

📦 Change lead time

The number that we knew would change during trainsperiment week was change lead time—the time from merge to deploy. If I merge a change, then a minute later I deploy it, that change’s lead time is one minute.

This chart shows the average lead time of all patches in a given train:

The chart below compares a typical week (1.38.0-wmf.1) to trainsperiment week (1.38.0-wmf.2, wmf.3, and wmf.4). Each dot is a change in a particular version—fewer dots mean fewer changes.

During trainsperiment week, we deployed faster. Each deployment was smaller, and the lead time of each patch in a release was shorter.

Here’s the same data on a logarithmic scale. During trainsperiment week there were only a few hours between trains, so the lead time could be measured in hours, not days!

📝 Survey Feedback

At the end of the week, we asked for feedback via the Wikitech-l mailing list. We collected comments from the mediawiki.org talk page and the summaries of candid conversations.

👍 Satisfaction

A small number of people took the time to respond to the survey—20 people answered our questions.

Almost everyone who took the survey seemed satisfied with communication. Most were satisfied with the experiment overall.

There were concerns on the talk page and in the survey responses about testing. Testing felt time-crunched, and everyone was worried about the time pressure on our Quality and Test Engineering Team (QTE).

🌚 Impact

Less than half of our respondents felt that the Trainsperiment positively impacted their work, with one respondent strongly disagreeing that there was a positive impact.

Most people were neutral about the impact of this experiment on their work.

The person who felt that there was a negative impact was concerned about the lack of time allotted for testing—they urged us to rethink testing if we wanted to try this again.

💌 Comments

The survey contained free-form prompts for feedback. Below is a smattering of representative responses. Most of the comments below are amalgamations and simplifications, but the reactions in quotes are verbatim.

What should RelEng have done differently

• Automated alerts: emails whenever there’s a deploy or the train is blocked

What would you need to change if we did this every week?

• No time to find and fix regressions means the QA process would need to change somehow
• More transparency around when train rolls out and a clearer blocking process
• Translations
• “my mental model.”

Other Feedback

• “With less time between groups, breakage will reach all wikis very quickly”
• “Often Tuesdays are currently used to deploy bug fixes that are hard to test locally […] we would need to revisit many of our workflows”
• “This, at least on paper, will help devs”
• “This was a pure win, IMO.”

🗣️ Conversations

We talked individually to people who had concerns about the experiment on Slack and IRC, in meetings, in the survey feedback, and on the talk page.

People were concerned about shortening the time for review. This is understandable given that we shortened a 168-hour process to a 12-hour process. 

Our QA process takes time. Our overburdened principal engineers take time to review code going live on a weekly basis. Due to some esoteric details, even our CI system gives us more confidence given more time—it was possible that MediaWiki could have broken compatibility with an extension without alerting anyone.

We have come to rely on the weekly cadence to make a careful release, and a faster process would mean rethinking our process pipeline to production.

🎀 Release Engineering's Feedback

The weekly train hides a lot of technical debt—it’s a giant feature flag and the missing testing environment rolled into one. It goes out every week (mostly), and Release Engineering spends about 20% of its time monitoring the release.

During trainsperiment week, we spent 100% of our time deploying—that’s not sustainable for our team.

We surfaced process pain points with this experiment, which was a success. We added to the already overlarge burdens of our principal engineers and quality engineers, which was a failure.

But this isn’t the end of the experiments. We endeavor to bring developers and production closer together—preferably with us standing back a healthy distance. If you’d like to help us get there—get in touch.

Thanks to @kchapman, @brennen, and @Krinkle for reading earlier drafts of this post and offering their feedback.

Over here in the Release-Engineering-Team, Train Deployment is usually a rotating duty. We've written about it before, so I won't go into the exact process, but I want to tell you something new about it.

It's awful, incredibly stressful, and a bit lonely.

And last week we ran an experiment where we endeavored to perform the full train cycle four times in a single week... What is wrong with us? (Okay. I need to own this. It was technically my idea.) So what is wrong with me? Why did I wish this on my team? Why did everyone agree to it?

First I think it's important to portray (and perhaps with a little more color) how terrible running the train can be.

How it usually feels to run a Train Deployment and why

Here's a little chugga-choo with a captain and a crew. Would the llama like a ride? Llama Llama tries to hide.

―Llama Llama, Llama Llama Misses Mama

At the outset of many a week I have wondered why, when the kids are safely in childcare and I'm finally in a quiet house well fed and preparing a nice hot shower to not frantically use but actually enjoy, my shoulder is cramping and there's a strange buzzing ballooning in my abdomen.

Am I getting sick? Did I forget something? This should be nice. Why can't I have nice things? Why... Oh. Yes. Right. I'm on train this week.

Train begins in the body before it terrorizes the mind, and I'm not the only one who feels that way.

A week of periodic drudgery which at any moment threatens to tip into the realm of waking nightmare.

―Stoic yet Hapless Conductor

Aptly put. The nightmare is anything from a tiny visual regression to taking some of the largest sites on the Internet down completely.

Giving a presentation but you have no idea what the slides are.

―Bravely Befuddled Conductor

Yes. There's no visibility into what we are deploying. It's a week's worth of changes, other teams' changes, changes from teams with different workflows and development cycles, all touching hundreds of different codebases. The changes have gone through review, they've been hammered by automated tests, and yet we are still too far removed from them to understand what might happen when they're exposed to real world conditions.

It's like throwing a penny into a well, a well of snakes, bureaucratic snakes that hate pennies, and they start shouting at you to fill out oddly specific sounding forms of which you have none.

―Lost Soul been 'round these parts

Kafkaesque.

When under the stress and threat of the aforementioned nightmare, it's difficult to think straight. But we have to. We have to parse and investigate intricate stack traces, run git blames on the deployment server, navigate our bug reporting forms and try to recall which teams are responsible for which parts of the aggregate MediaWiki codebase we've put together which itself is highly specific to WMF's production installation and really only becomes that long after changes merge to main branches of the constituent codebases.

We have to exercise clear judgement and make decisive calls of whether to rollback partially (previous group) or completely (all groups to previous version). We may have to halt everything and start hollering in IRC, Slack channels, mailing lists, to get the signal to the right folks (wonderful and gracious folks) that no more code changes will be deployed until what we're seeing is dealt with. We have to play the bad guys and gals to get the train back on track.

Trainsperiments Week and what was different about it

Study after study shows that having a good support network constitutes the single most powerful protection against becoming traumatized. Safety and terror are incompatible. When we are terrified, nothing calms us down like a reassuring voice or the firm embrace of someone we trust.

―Bessel Van Der Kolk, M.D., The Body Keeps the Score

Four trains in a single week and everyone in Release Engineering is onboard. What could possibly be better about that?

Well there is a safety in numbers as they say, and not in some Darwinistic way where most of us will be picked off by the train demons and the others will somehow take solace in their incidental fitness, but in a way where we are mutually trusting, supportive, and feeling collectively resourced enough to do the needful with aplomb.

So we set up video meetings for all scheduled deployment windows, had synchronous hand offs between our European colleagues and our North American ones. We welcomed folks from other teams into our deployments to show them the good, the bad, and the ugly of how their code gets its final send off 'round the bend and into the setting hot fusion reaction that is production. We found and fixed longstanding and mysterious bugs in our tooling. We deployed four full trains in a single week.

And it felt markedly different.

One of those barn raising projects you read about where everybody pushes the walls up en masse.

―Our Stoic Now Softened but Still Sardonic Conductor

Yes! Lonely and unwitnessed work is de facto drudgery. Toiling safely together we have a greater chance at staving off the stress and really feeling the accomplishment.

Giving a presentation with your friends and everyone contributes one slide.

―Our No Longer Befuddled but Simply Brave Conductor

Many hands make light work!

It was like throwing a handful of pennies into a well, a well of snakes, still bureaucratic and shouty, oh hey but my friends are here and they remind me these are just stack traces, words on a screen, and my friends happen to be great at filling out forms.

―Our Once Lost Now Found Conductor

When no one person is overwhelmed or unsafe, we all think and act more clearly.

The hidden takeaways of Trainsperiment Week

So how should what we've learned during our Trainsperiment Week inform our future deployment strategies and process. How should train deployments change?

The known hypothesis we wanted to test by performing this experiment was in essence:

1. More frequent deployments will result in fewer changes being deployed each time.
2. Fewer changes on average means the deployment is less likely to fail. The deployment is safer.
3. A safer deployment can be performed more frequently. (Positive feedback loop to #1.)
4. Overall we will: move faster; break less.

I don't know if we've proved that yet but we got an inkling that yes, the smaller subsequent deployments of the week did seem to go more smoothly. One week, however, even a week of four deployment cycles is not a large enough sample to say definitively whether doing train more frequently will for sure result in safer, more frequent deployments with fewer failures.

What was not apparent until we did our retrospective, however, is that it simply felt easier to do deployments together. It was still a kind of drudgery, but it was not abjectly terrible.

My personal takeaway is that a conductor who feels resourced and safe is the basis for all other improvements to the deployment process, and I want conductors to not only have tooling that works reliably with actionable logging at their disposal, but to feel a sense of community there with them when they're pushing the buttons. I want them to feel that the hard calls of whether or not to halt everything and rollback are not just their calls but shared in the moment among numerous people with intimate knowledge of the overall MediaWiki software ecosystem.

Better tooling—particularly around error reporting and escalation—is a barrier to entry for sure. Once we've made sufficient improvements there we need to get that tooling into other people's hands and show them that this process does not have to be so terrifying. And I think we're on the right track here with increased frequency and smaller sets of changes, but we can't lose sight of the human/social element and foundational basis of safety.

More than anything else, I want wider participation in the train deployment process by engineers in the entire organization along with volunteers.

Thanks to @thcipriani for reading my drafts and unblocking me from myself a number of times. Thanks to @jeena and @brennen for the inspirational analogies.

I'll start with a bit of general administrivia. First, our migration of Wikimedia code review & CI to GitLab continues, and we're mindful that people could use regular updates on progress. Second, I need to think through some stuff about the project, and doing that in writing is helpful for all involved. I'm going to try writing occasional blog entries here for both purposes.

Now on to the main topic of this post: Access control for groups and projects on the Wikimedia GitLab instance.

The tl;dr: We've been modeling access to things on GitLab by using groups under /people to contain individual users and then granting those groups access to things under /repos. This has been tricky to explain and doesn't work as well at a technical level as we'd hoped, so we're mostly scrapping the distinction, and moving control of project access to individual memberships in groups under /repos. This should be easier to think about, simpler to manage, and seems like it will suit our needs better. Read on for the nitty-gritty detail.

(Thanks to @Dzahn, @Majavah, @bd808, @AntiCompositeNumber, and @thcipriani for helping me think through the issues underlying this post.)

Background

During the GitLab consultation, when we were working on building up a model of how we'd use GitLab for Wikimedia projects, we wrote up a draft policy for managing users and their access to projects.

GitLab supports Groups. GitLab groups are similar to GitHub's concept of organizations, although the specifics differ. Groups can contain:

• Other, nested groups
• Individual projects (repositories & metadata)
• Users as members; members of other groups can be invited to a group
  • A user who is a member of a top-level group is also a member of every group it contains

We've since changed the original draft policy in some small ways - in particular, we decided to move most projects into a top-level /repos group in order to offer shared CI runners (see T292094). You can read the policy we landed on at the latest revision of GitLab/Policy on mediawiki.org.

The basic idea was that we would separate groups out into:

1. Sub-groups of /repos: Namespaces for projects, split up by functional area of code
2. Sub-groups of /people: Namespaces for individual users, split up by organizational units like:
   • Volunteer group
   • Teams at organizations such as the WMF, WMDE, etc.

Groups in /people could then be given access to projects under /repos.

Our hope was that this would let us decouple the management of groups of humans from the individual projects they work on, and ease onboarding for new contributors. A new member of the WMF Release Engineering team, for example, could be added to a single group and then have access to all the things they need to do their job.

We intended for most /people groups to be owned by their members, who would in turn have ownership-level access to their projects under /repos, allowing for contributors to a project to manage access and invite new contributors.

As a concrete example:

• https://gitlab.wikimedia.org/repos/cloud contains various cloud services projects
• The Wikimedia Foundation Cloud Services team and volunteer cloud administrators are modeled by membership in:
  • https://gitlab.wikimedia.org/people/wmf-team-cloud-services
  • https://gitlab.wikimedia.org/people/volunteer-group-cloud-admin

Problems with this scheme

I've been proceeding under this plan as people request the creation of GitLab project groups, but there turn out to be some problems.

First, it doesn't seem like permission inheritance for nested groups with other groups as members works the way you'd expect & hope: See T300939 - "GitLab group permissions are not inherited by sub-groups for groups of users invited to the parent repo".

Second, users have concerns about equity of access and tight coupling of things like employment with a specific organization to project access. We didn't have any intention of modeling any group of users as second-class citizens within this scheme, but it seems to create the impression of one all the same. It's also striking that the set of projects people work on just isn't that cleanly mapped to any particular organizational structure. Once you've been a technical contributor for a while, you've almost certainly collected responsibilities that no org chart reflects accurately.

Finally, and maybe most importantly, this is a complicated way to do things. People have a hard time thinking about it, and it requires a lot of explanation. That seems bad for an abstraction that we'd like to be basically self-serve for most users.

Proposed solution

Mostly, my plan is to use groups closer to how they seem to be designed:

1. Sub-groups of /repos will contain both individual contributor memberships and projects
2. Except in occasional one-off cases, access should be granted at the level of a containing group rather than at the level of individual projects, so as to avoid micromanaging access to many projects.
3. We'll keep /people in mind as a potential solution for some problems (for example, it might be a good tool for synchronizing groups of users from LDAP and granting access to certain projects on that basis), but not rely on it for anything at the moment.

There are some unanswered questions here, but I plan to redraft the policy doc, move existing project layouts to this scheme, and start creating new project groups on this basis in the coming week or so.

My main philosophical takeaway here is that I work with a bunch of anarchists, and it's always best to plan accordingly.

Originally, one of our goals for this migration was avoiding a repeat of the weird, nested morass that is our current set of Gerrit permissions. While it would be a good idea to keep the structure of things on GitLab flatter and easier to think about, I'm no longer that worried about it. Some of the complexity is inherent to any large set of projects and contributors; some of it just reflects a long-lived technical culture that's emergent and largely self-governing, tendencies that nearly always resist well-intentioned efforts to rationalize and map structure to things like official organizational layout.

If you’ve ever experienced the pride of seeing your name on MediaWiki's contributor list, you've been involved in our deployment process (whether you knew it or not).

The Wikimedia deployment process — 🚂🌈 The Train — pushed over 13,000 developer changes to production in 2021 . That's more than a change per hour for every single hour of the year—24 hours per day, seven days per week!

As you deploy more software to production, you may begin to wonder: is anything I've been working on going to be deployed this week? What's the status of production? Where can I find data about any of this?

🤔 Current train info

Bryan Davis (@bd808) created the versions toolforge tool in 2017. The versions tool is a dashboard showing the current status of Wikimedia's more than 900 wikis.

Other places to find info about the current deployment:

• Deployment calendar – shows upcoming deployments happening this week and next
• Current deployment task – an indispensable tool created by @Majavah that always points to this week's train

📈 Past train data

There's an aphorism in management: you can't manage what you can't measure. For years the train chugged along steadily, but it's only recently that we've begun to collect data on its chuggings.

The train stats project started in early 2021 and contains train data going back to March 2016.

Now we're able to talk about our deployments informed by the data. Release-Engineering-Team partnered with Research late last year to explore the data we have.

We're able to see metrics like Lead time and Cycle time

We measured product delivery lead time as the time it takes to go from code committed to code running in production.

– Accelerate (pg. 14, 15)

Our lead time — the time to go from commit in mainline to production — is always less than a week. In the scatter plots above, we can see some evidence of work-life balance: not many patches land two days before deployment — that's the weekend!

For the software delivery process, the most important global metric is cycle time. This is the time between deciding that a feature needs to be implemented and having that feature released to users.

– Continuous Delivery (pg 138)

Our cycle time — the time between a patch requesting code review and its deployment — varies. Some trains have massive outliers. In the chart above, for example, you can see one train that had a patch that was five years old!

It is now possible to see what we on Release Engineering had long suspected: the number of patches for each train has slowly been ticking up over time:

Also shown above: as the number of patches continues to rise, the number of comments per patch — that is, code-review comments per patch — has dropped.

The data also show that the average number of lines of code per patch is slightly going up:

🔥 Train derailment

The train-stats repo has data on blockers and delays. Most trains have a small number of blockers and deploy without fanfare. Other trains are plagued by problems that explode into an endless number of blockers — cascading into a series of psychological torments, haunting deployers like the train-equivalent of ringwraiths. Trainwraiths, let’s say.

The shape of the histogram of this data shows that blockers per train follows a power law — most trains have a few blockers:

Surprisingly, most of our blockers happen before we even start a train. Bugs from the previous week that we couldn't justify halting everything to fix, but need to be fixed before we lay down more code on top.

The data also let us correlate train characteristics with failure signals. Here we see that the number of patches (“patches”) per train (trending ↑) positively correlates with blockers, and lines of code review (“loc_per_train_bug”) per patch (trending ↓) negatively correlates with blockers — more patches and less code review are both correlated with more blockers:

Contrast this with Facebook's view of train risk. In a 2016 paper entitled "Development and Deployment at Facebook," Facebook's researchers documented how their Release Engineering team quantified deployment risk:

Inputs affecting the amount of oversight exercised over new code are the size of the change and the amount of discussion about it during code reviews; higher levels for either of these indicate higher risk.
– Development and Deployment at Facebook (emphasis added)

In other words, to Facebook, more code, and more discussion about code, means riskier code. Our preliminary data seem to only partially support this: more code is riskier, but more discussion seems to lessen our risk.

🧭 Explore on your own

This train data is open for anyone to explore. You can download the sqlite database that contains all train data from our gitlab repo, or play with it live on our datasette install.

There are a few Jupyter notebooks that explore the data:

• Miriam Redi's excellent exploration (where most of the charts in this post came from), and a presentation of her analysis.
• My ham-fisted attempts at data science

An audacious dream for the future of this data is to build a model to quantify exactly how risky a patchset is. We keep data on everything from bugs to rollbacks. Perhaps in future a model will help us roll out code faster and safer.

Thanks to @Miriam, @bd808, and @brennen for reading early drafts of this post: it'd be wronger without their input 💖.

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

3 documented incidents last month.

2022-02-01 ulsfo network
Impact: For 3 minutes, clients served by the ulsfo POP were not able to contribute or display un-cached pages.

2022-02-22 wdqs updater codfw
Impact: For 2 hours, WDQS updates failed to be processed. Most bots and tools were unable to edit Wikidata during this time.

2022-02-22 vrts
Impact: For 12 hours, incoming emails to a specific recently created VRTS queue were not processed with senders receiving a bounce with an SMTP 550 Error.

Figure from Incident graphs.

Incident follow-up

Remember to review and schedule Incident Follow-up work in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded. Read about past incidents at Incident status on Wikitech.

Recently conducted incident follow-up:

Create a dashboard for Prometheus metrics about health of Prometheus itself.
Pitched by CDanis after an April 2019 incident, carried out by Filippo (@fgiunchedi).

Improve wording around AbuseFilter messages about throttling functionality.
Originally filed in 2018. This came up last month during an incident where the wording may've led to a misunderstanding. Now resolved by @Daimona.

Exclude restart procedure from automated Elasticsearch provisioning.
There can be too much automation! Filed after an incident last September. Fixed by @RKemper.

Outstanding errors

Take a look at the workboard and look for tasks that could use your help.

View Workboard

I skip breakdowns most months as each breakdown has its flaws. However, I hear people find them useful, so I'll try to do them from time to time with my noted caveats. The last breakdown was in the December edition, which focussed on throughput during a typical month. Important to recognise is that neither high nor low throughput is per-se good or bad. It's good when issues are detected, reported, and triaged correctly. It's also good if a team's components are stable and don't produce any errors. A report may be found to be invalid or a duplicate, which is sometimes only determined a few weeks later.

The below "after six months" breakdown takes more of that into consideration by looking at what's on the table after six months (tasks upto Sept 2021). This may be considered "fairer" in some sense, although has the drawback of suffering from hindsight bias, and possibly not highlighting current or most urgent areas.

WMF Product:

• Anti Harassment Tools (3): 1 MW Blocks, 2 SecurePoll.
• Community Tech (0).
• Design Systems (1): 1 WVUI.
• Editing Team (15): 14 VisualEditor, 1 OOUI.
• Growth Team (13): 11 Flow, 1 GrowthExperiments, 1 MW Recent changes.
• Language Team (6): 4 ContentTranslation, 1 CX-server, 1 Translate extension.
• Parsoid Team (9): 8 Parsoid, 1 ParserFunctions extension .
• Product Infrastructure: 2 JsonConfig, 1 Kartographer, 1 WikimediaEvents.
• Reading Web (0).
• Structured Data (4): 2 MW Uploading, 1 WikibaseMediaInfo, 1 3D extension.

WMF Tech:

• Data Engineering: 1 EventLogging.
• Fundraising Tech: 1 CentralNotice.
• Performance: 1 Rdbms.
• Platform MediaWiki Team (19): 4 MW-Page-data, 1 MW-REST-API, 1 MW-Action-API, 1 MW-Snapshots, 1 MW-ContentHandler, 1 MW-JobQueue, 1 MW-libs-RequestTimeout, 9 Other.
• Search Platform: 1 MW-Seach.
• SRE Service Operations: 1 Other.

WMDE:

• WMDE-Wikidata (7): 5 Wikibase, 2 Lexeme.
• WMDE-TechWish: 1 FileImporter.

Other:

• Missing steward (7): 2 Graph, 2 LiquidThreads, 2 TimedMediaHandler, 1 MW Special-Contributions-page.
• Individually maintained (2): 1 WikimediaIncubator, 1 Score extension.

Trends

In February, we reported 25 new production errors. Of those, 13 have since been resolved, and 12 remain open as of today (two weeks into the following month). We also resolved 22 errors that remained open from previous months. The overall workboard has grown slightly to a total of 301 outstanding error reports.

For the month-over-month numbers, refer to the spreadsheet data.

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

There were no incidents this January. Pfew! Remember to review and schedule Incident Follow-up work in Phabricator. These are preventive measures and tech debt mitigations written down after an incident is concluded. Read about past incidents at Incident status on Wikitech.

Trends

During 2021, I compared us to the median of 4 incidents per month, as measured over the two years prior (2019-2020).

I'm glad to announce our median has lowered to 3 per month over the past two years (2020-2021). For more plots and numbers about our incident documentation, refer to Incident stats.

Since the previous edition, we resolved 17 tasks from previous months. In January, there were 45 new error reports of which 28 have been resolved within the same month, the remaining 17 have carried over to February.

With precisely 17 tasks both closed and added, the workboard remains at the exact total of 298 open tasks, for the third month in a row. That's quite the coincidence.

Take a look at the workboard and look for tasks that could use your help.

View Workboard

For the month-over-month numbers, refer to the spreadsheet data.

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

🎸 Doc Brown:

It could mean that that point in time contains some cosmic significance.., as if it were the temporal junction point of the entire space-time continuum… Or it could just be an amazing coincidence.

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

One documented incident last month:

2021-12-03 mx
Impact: A portion of outgoing email from wikimedia.org was delivered with a delay of upto 24 hours. This affected staff Gmail, and Znuny/Phabricator notifications. No mail was lost, it was eventually delivered.

Image from Incident graphs.

Incident follow-up

Remember to review and schedule Incident Follow-up work in Phabricator. These are preventive measures and tech debt mitigations written down after an incident. Read about past incidents at Incident status on Wikitech.

Recently resolved incident follow-up:

Create paging alert for high MX queues.
Filed in December after the mail delivery incident, resolved later that month by Keith (Herron).

Limit db execution time of expensive MW special pages.
Filed in December after various incidents due to high DB/appserver load, carried out by Amir (Ladsgroup).

Trends

In December we reported 22 new errors in December, of which 5 have since been resolved, and 17 remain open and have carried over to January. From the 298 issues previously carried over, we also resolved 17, thus the workboard still adds up to 298 in total.

In previous editions, we sometimes looked at the breakdown of tasks that remained unresolved. This time, I'd like to draw attention to the throughput and distribution of tasks that did get resolved.

Production errors resolved in the month of December, by team and component (query):

• Community-Tech (2): GlobalPreferences (1), CodeMirror (1).
• DBA: DjVuHandler (1).
• Editing-team: DiscussionTools (1).
• Fundraising Tech: CentralNotice (1).
• Growth-Team (8): GrowthExperiments (6), Image-Suggestions (1), StructuredDiscussions (1).
• Language-Team: UniversalLanguageSelector (1).
• Parsoid (1).
• Product-Infrastructure: TemplateStyles (1).
• Readers-Web (2).
• Structured-Data (2).
• Wikidata team: Wikidata-Page-Banner (1).
• Missing steward (1): MediaWiki-Logevents (T289806: Thanks @Umherirrender!).

For the month-over-month numbers, refer to the spreadsheet data.

Outstanding errors

Take a look at the workboard and look for tasks that could use your help.
View Workboard

Oldest unresolved errors:

• (June 2020) WikibaseClient: RuntimeException in wblistentityusage API. T254334
• (June 2020) WikibaseClient: Deadlock in EntityUsageTable::addUsages method. T255706

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

💡 Did you know:

To find your team's error reports, use the appropriate "Filter" link in the sidebar of the workboard.

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

6 documented incidents last month. That's above the two-year and five-year median of 4 per month (per Incident graphs).

2021-11-04 large file upload timeouts
Impact: For 9 months, editors were unable to upload large files (e.g. to Commons). Editors would receive generic error messages, typically after a timeout. In retrospect, a dozen different distinct production errors had been reported and regularly observed that were related and provided different clues, however most of these remained untriaged and uninvestigated for months. This may be related to the affected components having no active code steward.

2021-11-05 TOC language converter
Impact: For 6 hours, wikis experienced a blank or missing table of contents on many pages. For up to 3 days prior, wikis that have multiple language variants (such as Chinese Wikipedia) displayed the table of contents in an incorrect or inconsistent language variant (which are not understandable to some readers).

2021-11-10 cirrussearch commonsfile outage
Impact: For ~2.5 hours, the Search results page was unavailable on many wikis (except English Wikipedia). On Wikimedia Commons the search suggestions feature was unresponsive as well.

2021-11-18 codfw ipv6 network
Impact: For 8 minutes, the Codfw cluster experienced partial loss of IPv6 connectivity for upload.wikimedia.org. This did not affect availability of the service because the "Happy Eyeballs" algorithm ensures browsers (and other clients) automatically fallback to IPv4. The Codfw cluster generally serves Mexico and parts of the US and Canada. The upload.wikimedia.org service serves photos and other media/document files, such as displayed in Wikipedia articles.

2021-11-23 core network routing
Impact: For about 12 minutes, Eqiad was unable to reach hosts in other data centers via public IP addresses. This was due to a BGP routing error. There was no impact on end-user traffic, and impact on internal traffic was limited (only Icinga alerts themselves) because internal traffic generally uses local IP subnets which we currently route with OSPF instead of BGP.

2021-11-25 eventgate-main outage
Impact: For about 3 minutes, eventgate-main was down. This resulted in 25,000 MediaWiki backend errors due to inability to queue new jobs. About 1000 user-facing web requests failed (HTTP 500 Error). Event production briefly dropped from ~3000 per second to 0 per second.

Incident follow-up

Remember to review and schedule Incident Follow-up work in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded. Read more about past incidents at Incident status on Wikitech.

Recently resolved incident follow-up:

Disable DPL on wikis that aren't using it.
Filed after a July 2021 incident, done by Amir (Ladsgroup) and Kunal (Legoktm).

Create easy access to MySQL ports for faster incident response and maintenance.
Filed in Sep 2021, and carried out by Stevie (Kormat).

Create paging alert for primary DB hosts.
Filed after a Sept 2019 incident, done by Stevie (Kormat).

Trends

November saw 27 new production error reports of which 14 were resolved, and 13 remain open and carry over to the next month.

Of the 301 errors still open from previous months, 16 were resolved. Together with the 13 carried over from November that brings the workboard to 298 unresolved tasks.

For the month-over-month numbers, refer to the spreadsheet data.

Outstanding errors

💡 Did you know:

To find your team's error reports, use the appropriate "Filter" link in the sidebar of the workboard.

View Workboard

Issues carried over from recent months:

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

There were 4 documented incidents last month. This is currently on average, compared to the past five years (per Incident graphs).

2021-10-08 network provider
Impact: For upto an hour, some regions experienced a partial connectivity outage. This primarily affected the US East Coast for ~13 minutes, and Russia for 1 hour. It was caused by a routing problem with one of several redundant network providers.

2021-10-22 eqiad networking
Impact: For ~40 minutes clients that are normally geographically routed to Eqiad experienced connection or timeout errors. We lost about 7K req/s during this time. After initial recovery, Eqiad was ready and repooled in ~10 minutes.

2021-10-25 s3 db replica
Impact: For ~30min MediaWiki backends were slower than usual. For ~12 hours, many wiki replicas were stale for Wikimedia Cloud Services such as Toolforge.

2021-10-29 graphite
Impact: During a server upgrade, historical data was lost for a subset of Graphite metrics. Some were recovered via the redundant server, but others were lost as the redundant was also upgraded since then and lost some in a similar fashion.

Remember to review and schedule Incident Follow-up work in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded. Read about past incidents at Incident status on Wikitech.

Trends

Norwegian blue 🐦

298 bugs were up on the board.
We solved 20 of those over the past thirty days.

How many might now be left unexplored?
We also added new bugs to our database.

Half those bugs are pining for their fjord.
The other 23 carry on, with their dossiers.

All in all, 301 bugs up on the board.

In October, 49 new tasks were reported as production errors. Of these, we resolved 26, and 23 remain unresolved and carry forward to the next month.

Previously, the production error workboard held an accumulated total of 298 still-open error reports. We resolved 20 of those. Together with the 23 new errors carried over from October, this brings us to 301 unresolved errors on the board.

For the month-over-month numbers, refer to the spreadsheet data.

Outstanding errors

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Issues carried over from recent months:

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

We've had quite an eventful month, with 8 documented incidents in September. That's the highest since last year (Feb 2020) and one of the three worst months of the last five years.

• 2021-09-01 partial Parsoid outage
  • Impact: For 9 hours, 10% of Parsoid requests to parse/save pages were failing on all wikis. Little to no end-user impact apart from delays during RESTBase retries.
• 2021-09-04 appserver latency
  • Impact: For 37 minutes, MW backends were slow with 2% of requests receiving errors. This affected all wikis through logged-in users, bots/API queries, and some page views from unregistered users (e.g. pages that were recently edited or expired from CDN cache).
• 2021-09-06 Wikifeeds
  • Impact: For 3 days, the Wikifeeds API failed ~1% of requests (e.g. 5 of 500 req/s).
• 2021-09-12 Esams upload
  • Impact: For 20 minutes, images were unavailable for people in Europe, affecting all wikis.
• 2021-09-13 CirrusSearch restart
  • Impact: For ~2 hours, search was unavailable on Wikipedia from all regions. Search suggestions were missing or slow, and the search results page errored with "Try again later".
• 2021-09-18 appserver latency
  • Impact: For ~10 minutes, MW backends were slow or unavailable for all wikis.
• 2021-09-26 appserver latency
  • Impact: For ~15 minutes, MW backends were slow or unavailable for all wikis.
• 2021-09-29 eqiad kubernetes
  • Impact: For 2 minutes, MW backends were affected by a Kubernetes issue (via Kask sessionstore). 1500 edit attempts failed (8% of POSTs), and logged-in pageviews were slowed down, often taking several seconds.

Remember to review and schedule Incident Follow-up work in Phabricator, which are preventive measures and tech debt mitigations written down after an incident is concluded.

Image from Incident graphs.

Trends

The month of September saw 24 new production error reports of which 11 have since been resolved, and today, three to six weeks later, 13 remain open and have thus carried over to the next month. This is about average, although it makes it no less sad that we continue to introduce (and carry over) more errors than we rectify in the same time frame.

On the other hand, last month we did have a healthy focus on some of the older reports. The workboard stood at 301 unresolved errors last month. Of those, 16 were resolved. With the 13 new errors from September, this reduces the total slightly, to 298 open tasks.

For the month-over-month numbers, refer to the spreadsheet data.

Did you know

• 💡 The default "system error" page now includes a request ID. T291192

• 💡 To zoom in and find your team's error reports, use the appropriate "Filter" link in the sidebar of the workboard.

Outstanding errors

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Summary over recent months:

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

This post gives a quick introduction to a benchmarking tool, phpbench, ready for you to experiment with in core and skins/extensions.[1]

What is phpbench?

From their documentation:

PHPBench is a benchmark runner for PHP analagous to PHPUnit but for performance rather than correctness.

In other words, while a PHPUnit test will tell you if your code behaves a certain way given a certain set of inputs, a PHPBench benchmark only cares how long that same piece of code takes to execute.

The tooling and boilerplate will be familiar to you if you've used PHPUnit. There's a command-line runner at vendor/bin/phpbench, benchmarks are discoverable by default in tests/Benchmark, a configuration file (benchmark.json) allows for setting defaults across all benchmarks, and the benchmark tests classes and tests look pretty similar to PHPUnit tests.

Here's an example test for the Html::openElement() function:

namespace MediaWiki\Tests\Benchmark;

class HtmlBench {

	/**
	 * @Assert("mode(variant.time.avg) < 85 microseconds +/- 10%")
	 */
	public function benchHtmlOpenElement() {
		\Html::openElement( 'a', [ 'class' => 'foo' ] );
	}
}

So, taking it line by line:

• class HtmlBench (placed in tests/Benchmark/includes/HtmlBench.php) – the class where you can define the benchmarks for methods in a class. It would make sense to create a single benchmark class for a single class under test, just like with PHPUnit.
• public function benchHtmlOpenElement() {} – method names that begin with bench will be executed by phpbench; other methods can be used for set-up / teardown work. The contents of the method are benchmarked, so any set-up / teardown work should be done elsewhere.
• @Assert("mode(variant.time.avg) < 85 microseconds +/- 10%") – we define a phpbench assertion that the average execution time will be less than 85 microseconds, with a tolerance of +/- 10%.

If we run the test with composer phpbench, we will see that the test passes. One thing to be careful with, though, is adding assertions that are too strict – you would not want a patch to fail CI because the assertion for execution was not flexible enough (more on this later on).

Measuring performance while developing

One neat feature in PHPBench is the ability to tag current results and compare with another run. Looking at the HTMLBench benchmark test from above, for example, we can compare the work done in rMW5deb6a2a4546: Html::openElement() micro-optimisations to get before and after comparisons of the performance changes.

Here's a benchmark of e82c5e52d50a9afd67045f984dc3fb84e2daef44, the commit before the performance improvements added to Html::openElement() in rMW5deb6a2a4546: Html::openElement() micro-optimisations

❯ git checkout -b html-before-optimizations e82c5e52d50a9afd67045f984dc3fb84e2daef44 # get the old HTML::openElement code before optimizations
❯ git review -x 727429 # get the core patch which introduces phpbench support
❯ composer phpbench -- tests/Benchmark/includes/HtmlBench.php --tag=original

And the output [2]:

Note that we've used --tag=original to store the results. Now we can check out the newer code, and use --ref=original to compare with the baseline:

❯ git checkout -b html-after-optimizations 5deb6a2a4546318d1fa94ad8c3fa54e9eb8fc67c # get the new HTML::openElement code with optimizations
❯ git review -x 727429 # get the core patch which introduces phpbench support
❯ composer phpbench -- tests/Benchmark/includes/HtmlBench.php --ref=original --report=aggregate

And the output [3]:

We can see that the execution time roughly halved, from 18 microseconds to 8 microseconds. (For understanding the other columns in the report, it's best to read through the Quick Start guide for phpbench.) PHPBench can also provide an error exit code if the performance decreased. One way that PHPBench might fit into our testing stack would be to have a job similar to Fresnel, where a non-voting comment on a patch alerts developers whether the PHPBench performance decreased in the patch.

Testing with extensions

A slightly more complex example is available in GrowthExperiments (patch). That patch makes use of setUp/tearDown methods to prepopulate the database entries needed for the code being benchmarked:

/**
 * @BeforeMethods ("setUpLinkRecommendation")
 * @AfterMethods ("tearDownLinkRecommendation")
 * @Assert("mode(variant.time.avg) < 20000 microseconds +/- 10%")
 */
public function benchFilter() {
	$this->linkRecommendationFilter->filter( $this->tasks );
}

The setUpLinkRecommendation and tearDownLinkRecommendation methods have access to MediaWikiServices, and generally you can do similar things you'd do in an integration test to setup and teardown the environment. This test is towards the opposite end of the spectrum from the core test discussed above which looks at Html::openElement(); here, the goal is to look at a higher level function that involves database queries and interacting with MediaWiki services.

What's next

You can experiment with the tooling and see if it is useful to you. Some open questions:

• do we want to use phpbench? or are the scripts in maintenance/benchmarks already sufficient for our benchmarking needs?
• we already have a benchmarking tools in maintenance/benchmarks that extend a Benchmarker class; would it make sense to convert these to use phpbench?
• what are sensible defaults for "revs" and "iterations" as well as retry thresholds?
• do we want to run phpbench assertions in CI?
  • if yes, do we want assertions about using absolute times (e.g. "this function should take less than 20 ms") or relative assertions ("patch code is within 10% +/- of old code)
  • if yes, do we want to aggregate reports over time, so we can see trends for the code we benchmark?
  • should we disable phpbench as part of the standard set of tests run by Quibble, and only have it run as a non-voting job like Fresnel?

Looking forward to your feedback! [4]

[1] thank you, @hashar, for working with me to include this in Quibble and roll out to CI to help with evaluation!

[2]

> phpbench run --config=tests/Benchmark/phpbench.json --report=aggregate 'tests/Benchmark/includes/HtmlBench.php' '--tag=original'
PHPBench (1.1.2) running benchmarks...
with configuration file: /Users/kostajh/src/mediawiki/w/tests/Benchmark/phpbench.json
with PHP version 7.4.24, xdebug ✔, opcache ❌

\MediaWiki\Tests\Benchmark\HtmlBench

    benchHtmlOpenElement....................R1 I1 ✔ Mo18.514μs (±1.94%)

Subjects: 1, Assertions: 1, Failures: 0, Errors: 0
Storing results ... OK
Run: 1346543289c75373e513cc3b11fbf5215d8fb6d0
+-----------+----------------------+-----+------+-----+----------+----------+--------+
| benchmark | subject              | set | revs | its | mem_peak | mode     | rstdev |
+-----------+----------------------+-----+------+-----+----------+----------+--------+
| HtmlBench | benchHtmlOpenElement |     | 50   | 5   | 2.782mb  | 18.514μs | ±1.94% |
+-----------+----------------------+-----+------+-----+----------+----------+--------+

[3]

> phpbench run --config=tests/Benchmark/phpbench.json --report=aggregate 'tests/Benchmark/includes/HtmlBench.php' '--ref=original' '--report=aggregate'
PHPBench (1.1.2) running benchmarks...
with configuration file: /Users/kostajh/src/mediawiki/w/tests/Benchmark/phpbench.json
with PHP version 7.4.24, xdebug ✔, opcache ❌
comparing [actual vs. original]

\MediaWiki\Tests\Benchmark\HtmlBench

    benchHtmlOpenElement....................R5 I4 ✔ [Mo8.194μs vs. Mo18.514μs] -55.74% (±0.50%)

Subjects: 1, Assertions: 1, Failures: 0, Errors: 0
+-----------+----------------------+-----+------+-----+---------------+-----------------+----------------+
| benchmark | subject              | set | revs | its | mem_peak      | mode            | rstdev         |
+-----------+----------------------+-----+------+-----+---------------+-----------------+----------------+
| HtmlBench | benchHtmlOpenElement |     | 50   | 5   | 2.782mb 0.00% | 8.194μs -55.74% | ±0.50% -74.03% |
+-----------+----------------------+-----+------+-----+---------------+-----------------+----------------+

[4] Thanks to @zeljkofilipin for reviewing a draft of this post.

Last week I spoke to a few of my Wikimedia Foundation (WMF) colleagues about how we deploy code—I completely botched it. I got too complex too fast. It only hit me later—to explain deployments, I need to start with a lie.

M. Jagadesh Kumar explains:

Every day, I am faced with the dilemma of explaining some complex phenomena [...] To realize my goal, I tell "lies to students."

This idea comes from Terry Pratchett's "lies-to-children" — a false statement that leads to a more accurate explanation. Asymptotically approaching truth via approximation.

Every section of this post is a subtle lie, but approximately correct.

Release Train

The first lie I need to tell is that we deploy code once a week.

Every Thursday, Release-Engineering-Team deploys a MediaWiki release to all 978 wikis. The "release branch" is 198 different branches—one branch each for mediawiki/core, mediawiki/vendor, 188 MediaWiki extensions, and eight skins—that get bundled up via git submodule.

Progressive rollout

The next lie gets a bit closer to the truth: we don't deploy on Thursday; we deploy Tuesday through Thursday.

The cleverly named TrainBranchBot creates a weekly train branch at 2 am UTC every Tuesday.

• Tuesday
  • Deploy to Group0—132 wikis, including Test Wikipedia, mediawiki.org, and Office wiki (our internal WMF MediaWiki)
• Wednesday
  • Deploy to Group1—528 wikis, including Commons and Wikidata. Most non-Wikipedia wikis (plus Catalan Wikipedia, Italian Wikipedia, and Hebrew Wikipedia)
• Thursday
  • Deploy to remaining 320 wikis, including our largest wiki: English Wikipedia

Progressive rollouts give users time to spot bugs. We have an experienced user-base—as Risker attested on the Wikitech-l mailing list:

It's not always possible for even the best developer and the best testing systems to catch an issue that will be spotted by a hands-on user, several of whom are much more familiar with the purpose, expected outcomes and change impact on extensions than the people who have written them or QA'd them.

Bugs

Now I'm nearing the complete truth: we deploy every day except for Fridays.

Brace yourself: we don't write perfect software. When we find serious bugs, they block the release train — we will not progress from Group1 to Group2 (for example) until we fix the blocking issue. We fix the blocking issue by backporting a patch to the release branch. If there's a bug in this release, we patch that bug in our mainline branch, then git cherry-pick that patch onto our release branch and deploy that code.

We deploy backports three times a day during backport deployment windows.  In addition to backports, developers may opt to deploy new configuration or enable/disable features in the backport deployment windows.

Release engineers train others to deploy backports twice a week.

Emergencies

We deploy on Fridays when there are major issues. Examples of major issues are:

• Security issues
• Data loss or corruption
• Availability of service
• Preventing abuse
• Major loss of functionality/visible breakage

We avoid deploying on Fridays because we have a small team of people to respond to incidents. We want those people to be away from computers on the weekends (if they want to be), not responding to emergencies.

Non-MediaWiki code

There are 42 microservices on Kubernetes deployed via helm. And there are 64 microservices running on bare metal. The service owners deploy those microservices outside of the train process.

We coordinate deployments on our deployment calendar wiki page.

The whole truth

We progressively deploy a large bundle of MediaWiki patches (between 150 and 950) every week. There are 12 backport windows a week where developers can add new features, fix bugs, or deploy new configurations. There are microservices deployed by developers at their own pace.

Important Resources:

• Deployment calendar
• Deployment training
• Current release train status

More resources:

• Trains vs Backports
• Long-term deployment calendar
• Train blocker task
• Emergencies

Thanks to @brennen, @greg, @KSiebert, @Risker, and @VPuffetMichel for reading early drafts of this post. The feedback was very helpful. Stay tuned for "How we deploy code: Part II."

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

Zero documented incidents last month. Isn't that something!

Learn about past incidents at Incident status on Wikitech. Remember to review and schedule Incident Follow-up in Phabricator, which are preventive measures and other action items to learn from.

Image from Incident graphs.

Trends

In August we resolved 18 of the 156 reports that carried over from previous months, and reported 46 new failures in production. Of the new ones, 17 remain unresolved as of writing and will carry over to next month.

The number of new errors reports in August was fairly high at 46, compared to 31 reports in July, and 26 reports in June.

The backlog of "Old" issues saw no progress this past month and remained constant at 146 open error reports.

Unified graph:

💡 Did you know:

You can zoom in to your team's error reports by using the appropriate "Filter" link in the sidebar of our shared workboard.

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Progress

Last few months in review:

Tally:

For more month-over-month numbers refer to the spreadsheet.

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

3 documented incidents last month. That's at the median for the past twelve months, and slightly below the median of 4 over the past five years (Incident stats).

• 2021-07-14 eventgate latency spike
  • Impact: For ~ 10min MediaWiki API clients experienced request failures.
• 2021-07-16 codfw-a2 network
  • Impact: For ~ 1 hour Restbase clients received errors, affecting mobile apps and ContentTranslation.
• 2021-07-26 ruwikinews DynamicPageList
  • Impact: For 30min, 15% of requests from contributors on all wikis failed. There were also brief moments during which no readers could load recently modified or uncached pages.

Learn about past incidents at Incident status on Wikitech. Remember to review and schedule Incident Follow-up in Phabricator, which are preventive measures and other action items filed after an incident.

Trends

Last month the workboard held 154 non-old unresolved error reports. Over the past thirty days, the collective efforts of our volunteers and engineering teams have closed 14 of those.

In the month of July we've also introduced or discovered thirty-one new error reports (that's an average of one production regression every day!). Of those new error reports, fifteen were resolved and 16 remain unresolved. The workboard now tallies up to 156 tasks.

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Over on the backlog, we're continuing to ploddingly present progress on production problems from phantoms of christmases past.

For more month-over-month numbers refer to the spreadsheet data.

Outstanding errors

Below are various older issues that may have fallen by the wayside, taken from somewhat-random stab-in-the-dark queries.

Oldest unresolved errors that are still reproducible (Phab query):

• Reported in 2015: Unable to view history of protected Flow board (StructuredDiscussions, Growth team), T118502.
• Reported in 2016: Error when deleting a heading next to a table (VisualEditor, Editing team), T140871.

Stalled error reports (Phab query):

• Stalled Mar 2021: Constraints check for Q142 France times out (Wikidata, WMDE), T212282.

Oldest error with a patch for review (Phab query):

• Reported in 2016: Maps broken during 2nd live preview (Maps, Product Infra), T151524.
• Reported in 2018: Corrupt connection for cross-wiki db query (Platform team), T193565.

Tally

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

The release engineering team triages tasks flagged Release-Engineering-Team on a weekly basis. It is an all hands on deck one hour meeting in which we pick tasks one by one and find out what to do with them. We have started with more than a hundred of them and are now down to just a dozen or so, most filed since the last meeting.

I have been doing those routine triages for the projects I closely manage, often on Friday afternoon. I have recently started being a bit more serious about it and even allocated a couple weeks entirely dedicated to act on the backlog. This post summarizes some of my discoveries, will hopefully inspire the reader to tackle their own backlogs, technical debt and hopefully in the end we will have improved our ecosystem.

Finding tasks

Tasks you have filed

I keep filing tasks rather than taking notes or writing emails, I find Phabricator interface convenient since it lets me flag a task with labels however I want (Technical-Debt , Documentation, #mediawiki), subscribe individuals or even a whole team. It is great. With time those tasks pill up and it is easy to forget old ones, they have to be revisited from time to time. It as easy I searching for any open tasks I have filed and order them by creation date:

https://phabricator.wikimedia.org/maniphest/query/Wws2E0C7IaFd/#R

The first bug in the list is the oldest you have created and most probably deserve to be acted on. From there pick the tasks one by one.

Some will surely be obsolete since they have been acted on or the underlying infrastructure entirely changed. An example of a 6 years old task I declined is T100099, it followed a meeting to deploy MediaWiki services to Beta-Cluster-Infrastructure . The task has been partially achieved for a few services (notably Parsoid) and was left open since we never moved all services to the same system. Nowadays developers deploy a Docker image and restart the Docker container. The notes are obsolete and the task has thus no purpose anymore.

T149924 came from deploying static web assets using git directly to /srv. However the partition also hosted dynamically generated content such as all the content from https://doc.wikimedia.org/ , https://integration.wikimedia.org/ or state from a CI daemon. The issue is problematic when we reimage the server, specially during OS upgrades which we do every two years, and the task history reflect that:

• Filed in 2016 after an OS upgrade
• The part affecting https://integration.wikimedia.org/ is partially addressed in 2018 as part of an OS upgrade
• In 2020 we had yet another OS upgrade and this time I decided to complete the task

I completed it because that task showed up in my list of oldest bugs, it thus kept showing up whenever I did the triage and that was an incentive to get it gone. We are in a much better shape, the services have been decoupled on different machines, the static assets are deployed using our deployment tool: Scap.

Check your projects

Beside your team projects, you surely have side pet projects or legacy tags you might want to revisit. They can be found in search for your projects you are a member of (assuming you made yourself a member): https://phabricator.wikimedia.org/project/query/JS0zmX.yalpI/#R

I for example introduced Doxygen to generate the MediaWiki PHP documentation, git-review to assist interactions with Gerrit for which bugs are tracked in a column of the Gerrit project, and I am probably the one one actively acting on this task.

You can again list tasks filed against each project sorted by creation dates, and since you are a member of the project you will most probably be able to act on those old tasks.

One of the oldest tasks I had was T48148, which is to hide CI or robot comments from Gerrit change. The task has been filed in 2013, I found the upstream proposed solution back in 2019 and well *cough* forgot about it. Since I encountered the task during a triage, I went to tackle it and in short the required code boils down to add a single line in the CI configuration:

 gerrit:
   verified: 2
+  tag: autogenerated:ci

That took almost 9 months, since I was not actively triaging old tasks.

Technical debt

Just like we have the generic Documentation tag for any tasks relating to documentation, we have Technical-Debt to mark a task as requiring an extra effort and bring us to modernity. When triaging your own or your projects tasks, you can flag them as technical debt to easily find them later on.

Some tasks can immediately be filed as being a technical debt, that was the case of T141324 which is to send logging of the Gerrit code review system to logstash and thus make them easier to dig through or discover. Sounds simple? Well not that much.

The story is a bit complicated, but in short Gerrit is a java application and our team does not necessarily have much experience with it, the state of Java logging is a bit unclear (Gerrit uses log4j). Luckily we had some support from actual Java developers and managed to do some injecting, though the fields were not properly formatted, it was a progress.

After I got assigned as the primary maintainer of our Gerrit setup, I definitely needed proper logging. When we upgraded Gerrit to 3.2, the library we used to format the logs to Json was no longer provided by upstream, forcing us to maintain a fork of Gerrit just for that purpose.

Luckily upstream has made improvements and I found out it supports json logging out of the box while our logging infrastructure learned to ingest json logs. We even got as far as supporting Elastic Common Schema to use predefined field names.

That task has been a technical debt for 5 years, but since I kept seeing it I kept remembering about it and managed to address it.

Some tasks can not be acted on cause they depend on an upstream change that might be delayed for some reasons. A massive issue we have encountered since at least 2015 was slowness when doing a git fetch from our busiest repository. I previously blogged about it Blog Post: Faster source code fetches thanks to git protocol version 2 and Google addressed it by proposing a version 2 of the git protocol. It was one of the incentives for us to upgrade Gerrit, and as soon as we upgraded I made a point to test the fix and make it well known to our developers (do use protocol.version=2 in your .gitconfig).

Grooming pleasure

When processing old tasks, you can find it hard to tackle ones that need to focus for a few days if not weeks as in the example above. But there are also a bunch of little annoying tasks that are surprisingly very easy to solve and give immediate reward. The positive feedback loop would get you in the mood of finding more easy tasks and thus reducing your backlog. A few more examples:

T221510, filed in 2019 and addressed two years later, was requesting to expose a machine readable test coverage report. The file was there (clover.xml) it was simply not exposed in the web page, a simple <a href="clover.xml">clover.xml</a> is the only thing that was required.

My favorite tasks are obviously the ones that already have been solved and are just pending the paperwork to mark them resolved. T138653 was for a user unable to login to Gerrit due to a duplicate account, 3 years after it had been filed the user reported he was able to login properly and I marked it resolved one hour later. I guess that user was grooming their old tasks as well.

And finally, some old tasks might not be worth fixing. We are probably too kind with those and should probably be more strict in declining very old tasks. An example is T63733, the MediaWiki source code is deployed to the Wikimedia production cluster under a directory named php-<version>. Surely the php- prefix does not offer any meaningful information. However, since it is hardcoded in various places and would require moving files around on the whole fleet of servers, it might be a bit challenging and would definitely be a risky change. Should we drop that useless prefix? For sure. Is it worth facing outage and possibly multiple degraded services? Definitely not and I have thus just declined it.

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

3 documented incidents. That's lower than June in the previous five years where the month saw 5-9 incidents. I've added a new panel ⭐️ to the Incident statistics tool. This one plots monthly statistics on top of previous years, to more easily compare them:

Learn more from the Incident documents on Wikitech, and remember to review and schedule Incident Follow-up in Phabricator, which are preventive measures and other action items filed after an incident.

Trends

In June, work on production errors appears to have stagnated a bit. Or more precisely, the work only resulted in relatively few tasks being resolved. 15 of the 26 new tasks are still open as of writing.

Of the tasks from previous months, only 11 were resolved, leaving most columns unchanged. See the table further down for a more detailed breakdown and links to Phabricator queries for the tasks in question.

With the 15 remaining new tasks, and the 11 tasks resolved from our backlog, this raises the chart from 150 to 154 tasks.

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Month-over-month plots based on spreadsheet data.

Outstanding errors

Summary over recent months:

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

🕳 O'Neill: We've done this!
Dr Jackson: We do this every day.
O'Neill: I'm not talking about briefings in general, Daniel, I'm talking about this briefing; I'm talking about this day.
Teal'c: Col. O'Neill is correct. Events do appear to be repeating themselves.

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

Zero incidents recorded in the past month. Yay! That's only five months after November 2020, the last month without documented incidents (Incident stats).

Remember to review Preventive measures in Phabricator, which are action items filed after an incident.

Trends

In May, we unfortunately saw a repeat of the worrying pattern we saw in April, but with higher numbers. We found 54 new errors. This is the most new errors in a single month, since the Excellence monthly began three years ago in 2018. About half of these (29 of 54) remain unresolved as of writing, two weeks into the following month.

Month-over-month plots based on spreadsheet data.

New errors in May

Below is a snapshot of just the 54 new issues found last month, listed by their code steward.

Be mindful that the reporting of errors is not itself a negative point per-se. I think it should be celebrated when teams have good telemetry, detect their issues early, and address them within their development cycle. It might be more worrisome when teams lack telemetry or time to find such issues, or can't keep up with the pace at which issues are found.

Outstanding errors

Take a look at the workboard and look for tasks that could use your help.

View Workboard

Summary over recent months:

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:
Incident status, Wikitech.
Wikimedia incident stats by Krinkle, CodePen.
Production error data (spreadsheet and plots).
Phabricator report charts for Wikimedia-production-error project.

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

6 documented incidents. That's above the historical average of 3–4 per month.

Learn about recent incidents at Incident status on Wikitech, or Preventive measures in Phabricator.

Trends

In April, we saw a continuation of the healthy trend that started this January — a trend where the back of the line is moving forward at least as quickly as the front of the line. We did take a little breather in March where we almost broke even, but otherwise the trend is going well.

Last month we bade farewell to the production errors we found in July 2019. This month we cleared out the column for October 2019.

One point of concern is that we did encounter a high number of new production errors — errors that we failed to catch during development, code review, continuous integration, beta testing, or pre-deployment checks. Where we used to discover about a dozen of those a month, we found 42 during this month. As of writing, 17 of the 42 April-discovered errors have been resolved.

The "Old" column (generally tracking pre-2019 tasks) grew for the first time in six months. This increase can largely be attributed to improved telemetry of client-side errors uncovering issues in under-resourced products, such as the old Kaltura video player.

Month-over-month plots based on spreadsheet data.

Outstanding errors

View Workboard

Summary over recent months, per spreadsheet:

Take a look at the workboard and look for tasks that could use your help:

View Workboard

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in production!

Until next time,

– Timo Tijhof

🎥 McMurphy: That nurse, man... she, uh, she ain't honest.
Doctor: Ah now, look. Miss Ratched is one of the finest nurses we've got in this institution.
McMurphy: Ha! Well […] She likes a rigged game, know what I mean?

One of the critical pieces of our infrastructure is Gerrit. It hosts most of our git repositories and is the primary code review interface. Gerrit is written in the Java programming language which runs in the Java Virtual Machine (JVM). For a couple years we have been struggling with memory issues which eventually led to an unresponsive service and unattended restarts. The symptoms were the usual ones: the application responses being slower and degrading until server side errors render the service unusable. Eventually the JVM terminates with:

java.lang.OutOfMemoryError: Java heap space

This post is my journey toward identifying the root cause and having it fixed up by the upstream developers. Given I barely knew anything about Java and much less about its ecosystem and tooling, I have learned more than a few things on the road and felt like it was worth sharing.

Prior work

The first meaningful task was in June 2019 (T225166) which over several months has led us to:

• replace aging underlying hardware
• tuning the memory garbage collector and switching to the G1 garbage collector
• raising the amount of memory allocated to the JVM (the heap)
• upgraded the Debian operating system by two major release (Jessie → Stretch → Buster)
• conduct a major upgrade of Gerrit (June 2020, Gerrit 2.15 → 3.2)
  • see @QChris report Some insights from successful Gerrit 2.15 -> 3.2 upgrade at Wikimedia
• bots crawling the repositories get moved to a replica
• fixing lack of cache in a MediaWiki extension querying Gerrit more than it should have

All of those were sane operations that are part of any application life-cycle, some were meant to address other issues. Raising the maximum heap size (20G to 32G) definitely reduced the frequency of crashes.

Still, we had memory filing over and over. The graph below shows the memory usage from September 2019 to September 2020. The increase of maximum heap usage in October 2020 is the JVM heap being raised from 20G to 32G. Each of the "little green hills" correspond to memory filing up until we either restarted Gerrit or the JVM unattended crash:

Zooming on a week, it is clearly seen the memory was almost entirely filled until we had to restart:

This had to stop. Complaints about Gerrit being unresponsive, SRE having to respond to java.lang.OutOfMemoryError: Java heap space or us having to "proactively" restart before a week-end. They were not good practices. Back and fresh from vacations, I filed a new task T263008 in September 2020 and started to tackle the problem on my spare time. Would I be able to find my way in an ecosystem totally unknown to me?

Challenge accepted!

stuff learned

• Routine maintenance are definitely a need
• Don't expect things to magically solve but commit to thoroughly identify the root cause instead of hoping.

Looking at memory

Since the JVM runs out of memory, lets look at memory allocation. The JDK provides several utilities to interact with a running JVM. Be it to attach a debugger, writing a copy of the whole heap or sending admin commands to the JVM.

jmap lets one take a full capture of the memory used by a Java virtual machine. It has to run as the same user as the application (we use Unix username gerrit2) and when having multiple JDKs installed, one has to make sure to invoke the jmap that is provided by the Java version running the targeted JVM.

Dumping the memory is then a magic:

sudo -u gerrit2 /usr/lib/jvm/java-8-openjdk-amd64/bin/jmap \
  -dump:live,format=b,file=/var/lib/gerrit-202009170755.hprof <pid of java process here>

It takes a few minutes depending on the number of objects. The resulting .hprof file is a binary format, which can be interpreted by various tools.

jhat, a Java heap analyzer, is provided by the JDK along jmap. I ran it disabling tracking of of object allocations (-stack false) as well as references to object (|-refs false) since even with 64G of RAM and 32 core it took a few hours and eventually crashed. That is due to the insane amount of live objects. On the server I thus ran:

/usr/lib/jvm/java-8-openjdk-amd64/bin/jhat -stack false -refs false gerrit-202009170755.hprof

It spawns a web service which I can reach from my machine over ssh using some port redirection and open a web browser for it:

ssh  -C -L 8080:ip6-localhost:7000 gerrit1001.wikimedia.org &
xdg-open http://ip6-localhost:8080/

Instance Counts for All Classes (excluding native types)

2237744 instances of class org.eclipse.jgit.lib.ObjectId
2128766 instances of class org.eclipse.jgit.lib.ObjectIdRef$PeeledNonTag
735294 instances of class org.eclipse.jetty.util.thread.Locker
735294 instances of class org.eclipse.jetty.util.thread.Locker$Lock
735283 instances of class org.eclipse.jetty.server.session.Session
...

And an other view shows 3.5G of byte arrays.

I got pointed to https://heaphero.io/ however the file is too large to upload and it contains sensitive information (credentials, users personal information) which we can not share with a third party.

Nothing really conclusive at this point, the heap dump has been taken shortly after a restart and Gerrit was not in trouble.

Eventually I found Javamelody has a view providing the exact same information without all the trouble of figuring out jmap, jhat and ssh proper set of parameters. Just browse to the monitoring page and:

stuff learned

• jmap to issue commands to the jvm including taking a heap dump
• jhat to run analysis with some options required to make it workable
• Use JavaMelody instead

JVM handling of out of memory error

An idea was to take a heap dump whenever the JVM encounters an out of memory error. That can be turned on by passing the extended option HeapDumpOnOutOfMemoryError to the JVM and specifying where the dump will be written to with HeapDumpPath:

java \
  -XX:+HeapDumpOnOutOfMemoryError \
  -XX:HeapDumpPath=/srv/gerrit \
  -jar gerrit.war ...

And surely next time it ran out of memory:

Nov 07 13:43:35 gerrit2001 java[30197]: java.lang.OutOfMemoryError: Java heap space
Nov 07 13:43:35 gerrit2001 java[30197]: Dumping heap to /srv/gerrit/java_pid30197.hprof ...
Nov 07 13:47:02 gerrit2001 java[30197]: Heap dump file created [35616147146 bytes in 206.962 secs]

Which results in a 34GB dump file which was not convenient for a full analysis. Even with 16G of heap for the analyze and a couple hours of CPU churning it was not any helpful

And at this point the JVM is still around, the java process is still there and thus systemd does not restart the service for us even though we have instructed it to do so:

[Service]
ExecStart=java -jar gerrit.war
Restart=always
RestartSec=2s

That lead to our Gerrit replica being down for a whole weekend with no alarm whatsoever (T267517). I imagine the reason for the JVM not exiting on an OutOfMemoryError is to let one investigate the reason. Just like heap dump, the behavior can be configured via the ExitOnOutOfMemoryError extended option:

java -XX:+ExitOnOutOfMemoryError

Next time the JVM will exit causing systemd to notice the service went away and so it will happily restart it again.

stuff learned

• automatic heap dumping with the JVM for future analysis
• Be sure to have the JVM exit when running out of memory so systemd will restart the service
• Process can be up while still not serving its purpose

Side track to jgit cache

When I filed the task, I suspected enabling git protocol version 2 (J199) on CI might have been the root cause. That eventually lead me to look at how Gerrit caches git operations. Being a Java application it does not use the regular git command but a pure Java implementation jgit, a project started by the same author as Gerrit (Shawn Pearce).

To speed up operations, jgit keeps git objects in memory with various tuning settings. You can read more about it at T263008#6601490 , but in the end it was of no use for the problem. @thcipriani would later point out that jgit cache does not overgrow past its limit:

The investigation was not a good lead, but surely it prompted us to have a better view as to what is going on in the jgit cache. But to do so we would need to expose historical metrics of the status of the cache.

Stuff learned

• Jgit has in memory caches to hold frequently accessed repositories / objects in the JVM memory speeding up access to them.

Metrics collection

We always had trouble determining whether our jgit cache was properly sized and tuned it randomly with little information. Eventually I found out that Gerrit does have a wide range of metrics available which are described at https://gerrit.wikimedia.org/r/Documentation/metrics.html . I always wondered how we could access them without having to write a plugin.

The first step was to add the metrics-reporter-jmx plugin. It registers all the metrics with JMX, a Java system to manage resources. That is then exposed by JavaMelody and at least let us browse the metrics:

I long had a task to get those metrics exposed (T184086) but never had a strong enough incentive to work it. The idea was to expose those metrics to the Prometheus monitoring system which would scrape them and make them available in Grafana. They can be exposed using the metrics-reporter-prometheus plugin. There is some configuration required to create an authentication token that lets Prometheus scrape the metrics and it is then all set and collected.

In Grafana, discovering which metrics are of interest might be daunting. Surely for the jgit cache it is only a few metrics we are interested in and crafting a basic dashboard for it is simple enough. But since we now collect all those metrics, surely we should have dashboards for anything that could be of interest to us.

While browsing the Gerrit upstream repositories, I found an unadvertised repository: gerrit/gerrit-monitoring. The project aims at deploying to Kubernetes a monitoring stack for Gerrit composed of Grafana, Loki, Prometheus and Promtail. While browsing the code, I found out they already had a Grafana template which I could import to our Grafana instance with some little modifications.

During the Gerrit Virtual Summit I raised that as a potentially interesting project for the whole community and surely a few days later:

• The existing dashboard got adjusted to be easily imported outside of the gerrit-monitoring context ( https://gerrit-review.googlesource.com/c/gerrit-monitoring/+/289222/ )
• The authors have sent a wide announcement to highlight the project: Monitoring dashboards for Gerrit (An update about the gerrit-monitoring project)

In the end we have a few useful Grafana dashboards, the ones imported from the gerrit-monitoring repo are suffixed with (upstream): https://grafana.wikimedia.org/dashboards/f/5AnaHr2Mk/gerrit

And I crafted one dedicated to jgit cache: https://grafana.wikimedia.org/d/8YPId9hGz/jgit-block-cache

Stuff learned

• Prometheus scraping system with auth token
• Querying Prometheus metrics in Grafana and its vector selection mechanism
• Other Gerrit administrators already created Vizualization
• Raising our reuse prompted upstream to further advertise their solution which hopefully has led to more adoption of their solution.

Despair

After a couple months, there was no good lead. The issue has been around for a while, in a programming language I don't know with assisting tooling completely alien to me. I even found jcmd to issue commands to the JVM, such as dumping a class histogram, the same view provided by JavaMelody:

$ sudo -u gerrit2 jcmd 2347 GC.class_histogram
num     #instances         #bytes  class name
3	----------------------------------------------
4	   5:      10042773     1205132760  org.eclipse.jetty.server.session.SessionData
5	   8:      10042773      883764024  org.eclipse.jetty.server.session.Session
6	  11:      10042773      482053104  org.eclipse.jetty.server.session.Session$SessionInactivityTimer$1
7	  13:      10042779      321368928  org.eclipse.jetty.util.thread.Locker
8	  14:      10042773      321368736  org.eclipse.jetty.server.session.Session$SessionInactivityTimer
9	  17:      10042779      241026696  org.eclipse.jetty.util.thread.Locker$Lock

That is quite handy when already in a terminal, saves a few click to switch to a browser, head to JavaMelody and find the link.

But it is the last week of work of the year.

Christmas is in two days.

Kids are messing up all around the home office since we are under lockdown.

Despair.

Out of rage I just stall the task shamelessly hoping for Java 11 and Gerrit 3.3 upgrades to solve this. Much like we hoped the system would be fixed by upgrading.

Wait..

1 million?

ONE MILLION ??

TEN TO THE POWER OF SIX ???

WHY IS THERE A MILLION HTTP SESSIONS HELD IN GERRIT !!!!!!?11??!!??

10042773  org.eclipse.jetty.server.session.SessionData

There. Right there. It was there since the start. In plain sight. And surely 19 hours later Gerrit had created 500k sessions for 56 MBytes of memory. It is slowly but surely leaking memory.

stuff learned

• Everything clears up once one has found the root cause

When upstream saves you

At this point it was just an intuition, albeit a strong one. I know not much about Java or Gerrit internals and went to invoke upstream developers for further assistance. But first, I had to reproduce the issue and investigate a bit more to give as many details as possible when filing a bug report.

Reproduction

I copied a small heap dump I took just a few minutes after Gerrit got restarted, it had a manageable size making it easier to investigate. Since I am not that familiar with the Java debugging tools, I went with what I call a clickodrome interface, a UI that lets you interact solely with mouse clicks: https://visualvm.github.io/

Once the heap dump is loaded, I could easily access objects. Notably the org.eclipse.jetty.server.session.Session objects had a property expiry=0, often an indication of no expiry at all. Expired sessions are cleared by Jetty via a HouseKeeper thread which inspects sessions and deletes expired ones. I have confirmed it does run every 600 seconds, but since sessions are set to not expire, they pile up leading to the memory leak.

On December 24th, a day before Christmas, I filed a private security issue to upstream (now public): https://bugs.chromium.org/p/gerrit/issues/detail?id=13858

After the Christmas and weekend break upstream acknowledged and I did more investigating to pinpoint the source of the issue. The sessions are created by a SessionHandler and debug logs show: dftMaxIdleSec=-1 or Default maximum idle seconds set to -1, which means that by default the sessions are created without any expiry. The Jetty debug log then gave a bit more insight:

DEBUG org.eclipse.jetty.server.session : Session xxxx is immortal && no inactivity eviction

It is immortal and is thus never picked up by the session cleaner:

DEBUG org.eclipse.jetty.server.session : org.eclipse.jetty.server.session.SessionHandler
==dftMaxIdleSec=-1 scavenging session ids []
                                          ^^^ --- empty array

Our Gerrit instance has several plugins and the leak can potentially come from one of them. I then booted a dummy Gerrit on my machine (java -jar gerrit-3.3.war) cloned the built-in All-Projects.git repository repeatedly and observed objects with VisualVM. Jetty sessions with no expiry were created, which rules out plugins and point at Gerrit itself. Upstream developer Luca Milanesio pointed out that Gerrit creates a Jetty session which is intended for plugins. I have also narrowed down the leak to only be triggered by git operations made over HTTP. Eventually, by commenting out a single line of Gerrit code, I eliminated the memory leak and upstream pointed at a change released a few versions ago that may have been the cause.

Upstream then went on to reproduce on their side, took some measurement before and after commenting out and confirmed the leak (750 bytes for each git request made over HTTP). Given the amount of traffic we received from humans, systems or bots, it is not surprising we ended up hitting the JVM memory limit rather quickly.

Eventually the fix got released and new Gerrit versions were released. We upgraded to the new release and haven't restarted Gerrit since then. Problem solved!

Stuff learned

• Even with no knowledge about a programming language, if you can build and run it, you can still debug using print or the universal optimization operator: //.
• Quickly acknowledge upstream hints, ideas and recommendations. Even if it is to dismiss one of their leads.
• Write a report, this blog.

Thank you upstream developers Luca Milanesio and David Ostrovsky for fixing the issue!

Thank you @dancy for the added clarifications as well as typos and grammar fixes.

References

• T225166 - 2019 issue referencing out of heap
• T263008 - 2020 fresh new task that eventually has lead to resolution
• Upstream task https://bugs.chromium.org/p/gerrit/issues/detail?id=13858
• The root cause, code adding auditing of GIT over http commands: https://gerrit-review.googlesource.com/c/gerrit/+/203611
• Fix: https://gerrit.googlesource.com/gerrit/+/4b2b2d831b39f57720923d5b0c23f1b6d94be0e9
• CVE-2021-22553 - Google Inc. vulnerability report

How’d we do in our strive for operational excellence last month? Read on to find out!

Incidents

2 documented incidents. That's average for this time of year, when we usually had 1-4 incidents.

Learn about recent incidents at Incident status on Wikitech, or Preventive measures in Phabricator.

Trends

In March we made significant progress on the outstanding errors of previous months. Several of the 2020 months are finally starting to empty out. But with over 30 new tasks from March itself remaining, we did not break even, and ended up slightly higher than last month. This could be reversing two positive trends, but I hope not.

Firstly, there was a steep increase in the number of new production errors that were not resolved within the same month. This is counter the positive trend we started in November. The past four months typically saw 10-20 errors outlive their month of discovery, and this past month saw 34 of its 48 new errors remain unresolved.

Secondly, we saw the overall number of unresolved errors increase again. This January began a downward trend for the first time in thirteen months, which continued nicely through February. But, this past month we broke even and even pushed upward by one task. I hope this is just a breather and we can continue our way downward.

Month-over-month plots based on spreadsheet data.

Outstanding errors

Take a look at the workboard and look for tasks that could use your help:

View Workboard

Summary over recent months, per spreadsheet:

Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:

Incident status, Wikitech.
Wikimedia incident stats by Krinkle, CodePen.
Production Excellence: Month-over-month spreadsheet and plot.
Report charts for Wikimedia-production-error project, Phabricator.

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Incidents

3 documented incidents last month, [1] which is average for the time of year. [2]

Learn about these incidents at Incident status on Wikitech, and their Preventive measures in Phabricator.

For those with NDA-restricted access, there may be additional private incident reports 🔒 available.

💡 Did you know: Our Incident reports have switched to using the ISO date format in their titles and listings, for improved readability and edit-ability (esp. when publishing on a later date). So long 202010221, and hello 2021-02-21!

📊 Trends

In February we saw a continuation of the new downward trend that began this January, which came after twelve months of continued rising. Let's make sure this trend sticks with us as we work our way through the debt, whilst also learning to have a healthy week-to-week iteration where we monitor and follow-up on any new developments such that they don't introduce lasting regressions.

The recent tally (issues filed since we started reporting in March 2019) is down to 138 unresolved errors, from 152 last month. The old backlog (pre-2019 issues) also continued its 5-month streak and is down to 148, from 160 last month. If this progress continues we'll soon have fewer "Old" issues than "Recent" issues, and possibly by the start of 2022 we may be able to report and focus only on our rotation through recent issues as hopefully we are then balancing our work such that issues reported this month are addressed mostly in the same month or otherwise later that quarter within 2-3 months. Visually that would manifest as the colored chunks having a short life on the chart with each drawn at a sharp downwards angle – instead of dragged out where it was building up an ever-taller shortcake. I do like cake, but I prefer the kind I can eat. 🍰

Month-over-month plots based on spreadsheet data. [3] [4]

📖 Outstanding errors

Summary over recent months:

• ⚠️ July 2019 (2 of 18 issues left): no change.
• ⚠️ August 2019 (1 of 14 issues): no change.
• ⚠️ October 2019 (4 of 12 issues): no change.
• ⚠️ November 2019 (1 of 5 issues): no change.
• ⚠️ December 2019 (1 of 9 issues): One task resolved (-1).
• ⚠️ January 2020 (2 of 7 issues): no change.
• ⚠️ February 2020 (1 of 7 issues): no change.
• ⚠️ March 2020 (2 of 2 issues): no change.
• April 2020 (9 of 14 issues left): no change.
• May 2020 (6 of 14 issues left): no change.
• June 2020 (7 of 14 issues left): no change.
• July 2020 (9 of 24 new issues): no change.
• August 2020 (20 of 53 new issues): Two tasks resolved (-2).
• September 2020 (9 of 33 new issues): Five tasks resolved (-5).
• October 2020 (26 of 69 new issues): Five tasks resolved (-5).
• November 2020 (11 of 38 new issues): Three tasks resolved (-3).
• December 2020 (12 of 33 new issues): Seven tasks resolved (-7).
• January 2021 (5 of 50 new issues): Two tasks resolved (-2).
• February 2021: 11 of 20 new issues survived the month and remained unresolved (+20; -9)

For the on-going month of March 2021, we've got 12 new issues so far.

Take a look at the workboard and look for tasks that could use your help!

View Workboard

🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:

[1] Incident status Wikitech.
[2] Wikimedia incident stats by Krinkle, CodePen.
[3] Month-over-month, Production Excellence spreadsheet.
[4] Open tasks, Wikimedia-prod-error, Phabricator.

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Incidents

1 documented incident last month. That's the third month in a row that we are at or near zero major incidents – not bad! [1] [2]

Learn about recent incidents at Incident status on Wikitech, or Preventive measures in Phabricator.

💡 Did you know: Our Incident status page provides a green-yellow status reflection over the past ten days, with a link to the most recent incident doc if there was any during that time.

📊 Trends

This January saw a small recovery in our otherwise negative upward trend. For the first time in twelve month more reports were closed than new reports having outlived the previous month without resolution. What happened twelve months ago? In January 2020, we also saw a small recovery during the otherwise upward trend before and after it.

Perhaps it's something about the post-December holidays that temporarily improves the quality and/or reduces the quantity — of code changes. Only time will tell if this is the start of a new positive trend, or merely a post-holiday break. [3]

While our month-to-month trend might not (yet) be improving, we do see persistent improvements in our overall backlog of pre-2019 reports. This is in part because we generally don't file new reports there, so it makes sense that it doesn't go back up, but it's still good to see downward progress every month, unlike with reports from more recent months which often see no change month-to-month (see "Outstanding errors" below, for example).

This positive trend on our "Old" backlog started in October 2020 and has consistently progressed every month since then (refer to the "Old" numbers in red on the below chart, or the same column in the spreadsheet). [3][4]

📖 Outstanding errors

Summary over recent months:

• ⚠️ July 2019 (2 of 18 issues left): no change.
• ⚠️ August 2019 (1 of 14 issues): no change.
• ✅ September 2019 (0 of 12 issues): Last two tasks were resolved (-2).
• ⚠️ October 2019 (4 of 12 issues): One task resolved (-1).
• ⚠️ November 2019 (1 of 5 issues): no change.
• ⚠️ December 2019 (2 of 9 issues), Two tasks resolved (-2).
• ⚠️ January 2020 (2 of 7 issues), no change.
• ⚠️ February 2020 (1 of 7 issues left), One task resolved (-1).
• March 2020 (2 of 2 issues left), no change.
• April 2020 (9 of 14 issues left): no change.
• May 2020 (6 of 14 issues left): One task resolved (-1).
• June 2020 (7 of 14 issues left): no change.
• July 2020 (9 of 24 new issues): no change.
• August 2020 (22 of 53 new issues): One task resolved (-1).
• September 2020 (13 of 33 new issues): One task resolved (-1).
• October 2020 (31 of 69 new issues): Four tasks fixed (-4).
• November 2020 (14 of 38 new issues): no change.
• December 2020 (19 of 33 new issues) Three tasks resolved (-3)
• January 2021: 7 of 50 new issues survived the month and remained unresolved (+50; -43)

January saw +50 new production errors reported in a single month, which is an unfortunate all-time high. However, we've also done remarkably well on addressing 43 of them within a month, when the potential root cause and diagnostics data were still fresh in our minds. Well done!

For the on-going month of February, there have been 16 new issues reported so far.

Take a look at the workboard and look for tasks that could use your help!

View Workboard

🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:

[1] Incident status Wikitech.
[2] Wikimedia incident stats by Krinkle, CodePen.
[3] Month-over-month, Production Excellence spreadsheet.
[4] Open tasks, Wikimedia-prod-error, Phabricator.

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Incidents

1 documented incident in December. [1] In previous years, December typically had 4 or fewer documented incidents. [3]

Learn about recent incidents at Incident documentation on Wikitech, or Preventive measures in Phabricator.

📊 Trends

Month-over-month plots based on spreadsheet data. [4] [2]

📖 Outstanding errors

Take a look at the workboard and look for tasks that could use your help.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

• ⚠️ July 2019 (2 of 18 issues left): no change.
• ⚠️ August 2019 (1 of 14 issues): no change.
• ⚠️ September 2019 (2 of 12 issues): One task resolved (-1).
• ⚠️ October 2019 (5 of 12 issues): no change.
• ⚠️ November 2019 (1 of 5 issues): no change.
• ⚠️ December 2019 (4 of 9 issues), no change.
• ⚠️ January 2020 (2 of 7 issues), no change.
• February 2020 (2 of 7 issues left), no change.
• March 2020 (2 of 2 issues left), no change.
• April 2020 (9 of 14 issues left): no change.
• May 2020 (7 of 14 issues left): no change.
• June 2020 (7 of 14 issues left): no change.
• July 2020 (9 of 24 new issues): no change.
• August 2020 (23 of 53 new issues): no change.
• September 2020 (13 of 33 new issues): One task resolved (-1).
• October 2020 (35 of 69 new issues): Four issues fixed (-4).
• November 2020 (14 of 38 new issues): Five issues fixed (-5).
• December 2020: 22 of 33 new issues survived the month and remained unresolved (+33; -22)

🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:

[1] Incident documentation 2020, Wikitech.
[2] Open tasks, Wikimedia-prod-error, Phabricator.
[3] Wikimedia incident stats by Krinkle, CodePen.
[4] Month-over-month, Production Excellence spreadsheet.

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Incidents

Zero documented incidents in November. [1] That's the only month this year without any (publicly documented) incidents. In 2019, November was also the only such month. [3]

Learn about recent incidents at Incident documentation on Wikitech, or Preventive measures in Phabricator.

📊 Trends

The overall increase in errors was relatively low this past month, similar to the November-December period last year.

What's new is that we can start to see a positive trend emerging in the backlogs where we've shrunk issue count three months in a row, from the 233 high in October, down to the 181 we have in the ol' backlog today.

Month-over-month plots based on spreadsheet data. [4]

📖 Outstanding errors

Take a look at the workboard and look for tasks that could use your help.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

• ⚠️ July 2019 (2 of 18 tasks): One task closed (-1).
• ⚠️ August 2019 (1 of 14 tasks): no change.
• ⚠️ September 2019 (3 of 12 tasks): no change.
• ⚠️ October 2019 (5 of 12 tasks): no change.
• ⚠️ November 2019 (1 of 5 tasks): no change.
• ⚠️ December 2019 (3 of 9 tasks left), no change.
• January 2020 (3 of 7 tasks left), One task closed (-1).
• February (2 of 7 tasks left), no change.
• March (2 of 2 tasks left), no change.
• April (9 of 14 tasks left): no change.
• May (7 of 14 tasks left): no change.
• June (7 of 14 tasks left): no change.
• July 2020 (9 of 24 new tasks): no change.
• August 2020 (23 of 53 new tasks): Three tasks closed (-3).
• September 2020 (14 of 33 new tasks): One task closed (-1).
• October 2020 (39 of 69 new tasks): Six tasks closed (-6).
• November 2020: 19 of 38 new tasks survived the month and remain open today (+38; -19)

The on-going month of December, has 19 unresolved tasks so far.

🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

👨🏻 Monsieur Gustave H.:

❝   The plot "thickens" as they say. Why, by the way? Is it a soup metaphor? ❞

Footnotes:

[1] Incident documentation 2020, Wikitech.
[2] Open tasks, Wikimedia-prod-error, Phabricator.
[3] Wikimedia incident stats, Krinkle, CodePen.
[4] Month-over-month, Production Excellence (spreadsheet).

Recently there has been a small effort on the Release-Engineering-Team to encode some of our institutional knowledge as runbooks linked from a page in the team's wiki space.

What are runbooks, you might ask? This is how they are described on the aforementioned wiki page:

This is a list of runbooks for the Wikimedia Release Engineering Team, covering step-by-step lists of what to do when things need doing, especially when things go wrong.

So runbooks are each essentially a sequence of commands, intended to be pasted into a shell by a human. Step by step instructions that are intended to help the reader accomplish an anticipated task or resolve a previously-encountered issue.

Presumably runbooks are created when someone encounters an issue, and, recognizing that it might happen again, helpfully documents the steps that were used to resolve said issue.

This all seems pretty sensible at first glance. This type of documentation can be really valuable when you're in an unexpected situation or trying to accomplish a task that you've never attempted before and just about anyone reading this probably has some experience running shell commands pasted from some online tutorials, setup instructions for a program, etc.

Despite the obvious value runbooks can provide, I've come to harbor a fairly strong aversion to the idea of encoding what are essentially shell scripts as individual commands on a wiki page. As someone who's job involves a lot of automation, I would usually much prefer a shell script, a python program, or even a "maintenance script" over a runbook.

After a lot of contemplation, I've identified a few reasons that I don't like runbooks on wiki pages:

• Runbooks are tedious and prone to human errors.
  • It's easy to lose track of where you are in the process.
  • It's easy to accidentally skip a step.
  • It's easy to make typos.
• A script can be code reviewed and version controlled in git.
• A script can validate it's arguments which helps to catch typos.
• I think that command line terminal input is more like code than it is prose. I am more comfortable editing code in my usual text editor as apposed to editing in a web browser. The wikitext editor is sufficient for basic text editing, and visual editor is quite nice for rich text editing, but neither is ideal for editing code.

I do realize that mediawiki does version control. I also realize that sometimes you just can't be bothered to write and debug a robust shell script to address some rare circumstances. The cost is high and it's uncertain whether the script will be worth such an effort. In those situations a runbook might be the perfect way to contribute to collective knowledge without investing a lot of time into perfecting a script.

My favorite web comic, xkcd, has a lot few things to say about this subject:

Potential Solutions

I've been pondering a solution to these issues for a long time. Mostly motivated by the pain I have experienced (and the mistakes I've made) while executing the biggest runbook of all on a regular basis.

Over the past couple of years I've come across some promising ideas which I think can help the problems I've identified with runbooks. I think that one of the most interesting is Do-nothing scripting. Dan Slimmon identifies some of the same problems that I've detailed here. He uses the term *slog* to refer to long and tedious procedures like the Wikimedia Train Deploys. The proposed solution comes in the form of a do-nothing script. You should go read that article, it's not very long. Here are a few relevant quotes:

Almost any slog can be turned into a do-nothing script. A do-nothing script is a script that encodes the instructions of a slog, encapsulating each step in a function.

...

At first glance, it might not be obvious that this script provides value. Maybe it looks like all we’ve done is make the instructions harder to read. But the value of a do-nothing script is immense:

• It’s now much less likely that you’ll lose your place and skip a step. This makes it easier to maintain focus and power through the slog.
• Each step of the procedure is now encapsulated in a function, which makes it possible to replace the text in any given step with code that performs the action automatically.
• Over time, you’ll develop a library of useful steps, which will make future automation tasks more efficient.

A do-nothing script doesn’t save your team any manual effort. It lowers the activation energy for automating tasks, which allows the team to eliminate toil over time.

I was inspired by this and I think it's a fairly clever solution to the problems identified. What if we combined the best aspects of gradual automation with the best aspects of a wiki-based runbook? Others were inspired by this as well, resulting in tools like braintree/runbook, codedown and the one I'm most interested in, rundoc.

Runnable Runbooks

My ideal tool would combine code and instructions in a free-form "literate programming" style. By following some simple conventions in our runbooks we can use a tool to parse and execute the embedded code blocks in a controlled manner. With a little bit of tooling we can gain many benefits:

• The tooling will keep track of the steps to execute, ensuring that no steps are missed.
• Ensure that errors aren't missed by carefully checking / logging the result of each step.
• We could also provide a mechanism for inputting the values of any variables / arguments and validate the format of user input.
• With flexible control flow management we can even allow resuming from anywhere in the middle of a runbook after an aborted run.
• Manual steps can just consist of a block of prose that gets displayed to the operator. With embedded markup we can format the instructions nicely and render them in the terminal using [Rich][7]. Once the operator confirms that the step is complete then the workflow moves on to the next step.

Prior Art

I've found a few projects that already implement many of these ideas. Here are a few of the most relevant:

• braintree/runbook
• codedown
• rundoc

The one I'm most interested in is Rundoc. It's almost exactly the tool that I would have created. In fact, I started writing code before discovering rundoc but once I realized how closely this matched my ideal solution, I decided to abandon my effort. Instead I will add a couple of missing features to Rundoc in order to get everything that I want and hopefully I can contribute my enhancements back upstream for the benefit of others.

Demo: https://asciinema.org/a/MKyiFbsGzzizqsGgpI4Jkvxmx
Source: https://github.com/20after4/rundoc

References

[1]: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Runbooks "runbooks"
[2]: https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys "Train deploys"
[3]: https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-the-key-to-gradual-automation/ "Do-nothing scripting: the key to gradual automation by Dan Slimmon"
[4]: https://github.com/braintree/runbook "runbook by braintree"
[5]: https://github.com/earldouglas/codedown "codedown by earldouglas"
[6]: https://github.com/eclecticiq/rundoc "rundoc by eclecticiq"
[7]: https://rich.readthedocs.io/en/latest/ "Rich python library"

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Incidents

2 documented incidents in October. [1] Historically, that's just below the median of 3 for this time of year. [3]

Learn about recent incidents at Incident documentation on Wikitech, or Preventive measures in Phabricator.

📊 Trends

Month-over-month plots based on spreadsheet data. [5]

📖 Outstanding errors

Take a look at the workboard and look for tasks that could use your help.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

• ⚠️ July 2019 (3 of 18 tasks): One task closed.
• ⚠️ August 2019 (1 of 14 tasks): no change.
• ⚠️ September 2019 (3 of 12 tasks): no change.
• ⚠️ October 2019 (5 of 12 tasks): One task closed.
• ⚠️ November 2019 (1 of 5 tasks): Two tasks closed.
• December (3 of 9 tasks left), no change.
• January 2020 (4 of 7 tasks left), no change.
• February (2 of 7 tasks left), no change.
• March (2 of 2 tasks left), no change.
• April (9 of 14 tasks left): One task closed.
• May (7 of 14 tasks left): no change.
• June (7 of 14 tasks left): no change.
• July 2020 (9 of 24 new tasks): One task closed.
• August 2020 (26 of 53 new tasks): Five tasks closed.
• September 2020 (15 of 33 new tasks): Two tasks closed.
• October 2020: 45 of 69 new tasks survived the month of October and remain open today.

For the on-going month of November, there are 25 new tasks so far.

🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

 👤  Howard Salomon:

❝   Problem is when they arrest you, you get put on the justice train, and the train has no brain. ❞  

Footnotes:

[1] Incident documentation 2020, Wikitech
[2] Open tasks in Wikimedia-prod-error, Phabricator
[3] Wikimedia incident stats by Krinkle, CodePen
[4] Month-over-month, Production Excellence (spreadsheet)

If you're making changes to a service that is deployed to Kubernetes, it sure is annoying to have to update the helm deployment-chart values with the newest image version before you deploy. At least, that's how I felt when developing on our dockerfile-generating service, blubber.

Over the last two months we've added

• A script to update the image versions (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/update_version/)
• A new 'promote' step to PipelineLib (https://wikitech.wikimedia.org/wiki/PipelineLib) to run the script and create the patchset on merge
• Various infrastructure pieces to ensure necessary packages are available

And I'm excited to say that CI can now handle updating image versions for you (after your change has merged), in the form of a change to deployment-charts that you'll need to +2 in Gerrit. Here's what you need to do to get this working in your repo:

Add the following to your .pipeline/config.yaml file's publish stage:

promote: true

The above assumes the defaults, which are the same as if you had added:

promote:
  - chart: "${setup.projectShortName}"  # The project name
    environments: []                    # All environments
    version: '${.imageTag}'             # The image published in this stage

You can specify any of these values, and you can promote to multiple charts, for example:

promote:
  - chart: "echostore"
    environments: ["staging", "codfw"]
  - chart: "sessionstore"

The above values would promote the production image published after merging to all environments for the sessionstore service, and only the staging and codfw environments for the echostore service. You can see more examples at https://wikitech.wikimedia.org/wiki/PipelineLib/Reference#Promote

If your containerized service doesn't yet have a .pipeline/config.yaml, now is a great time to migrate it! This tutorial can help you with the basics: https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial#Publishing_Docker_Images

This is just one step closer to achieving continuous delivery of our containerized services! I'm looking forward to continuing to make improvements in that area.

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Incidents

5 documented incidents. [1] Historically, that's right on average for the time of year. [3]

For more about recent incidents see Incident documentation on Wikitech, or Preventive measures in Phabricator.

📊 Trends

Month-over-month plots based on spreadsheet data. [5]

📖 Outstanding errors

Take a look at the workboard and look for tasks that could use your help.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

• ⚠️ July 2019 (4 of 18 tasks left): no change.
• ⚠️ August 2019 (1 of 14 tasks left): no change.
• ⚠️ September 2019 (3 of 12 tasks left): no change.
• ⚠️ October 2019 (6 of 12 tasks left), no change.
• November (3 of 5 tasks left): no change.
• December (3 of 9 tasks left), no change.
• January 2020 (4 of 7 tasks left), One task closed.
• February (2 of 7 tasks left), no change.
• March (2 of 2 tasks left), no change.
• April (10 of 14 tasks left): no change.
• May (7 of 14 tasks left): no change.
• June (7 of 14 tasks left): Three tasks closed.
• July 2020 (10 of 24 new tasks): Three tasks closed.
• August 2020 (31 of 53 new tasks): Six tasks closed.
• September 2020: 17 of 33 new tasks survived the month of September and remain open today.

Previously, we had 106 unresolved production errors from the recent months up to August. Since then, 13 of those were closed. But, the 18 errors surviving September raise our recent tally to 110.

The workboard overall (including errors from 2019 and earlier) holds 343 open tasks in total, an increase of +47 compared to the 296 total on Sept 23rd.

🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

🕵️‍♀️ Holmes: “So, she pulled five bullets out of you?”

     Shinwell: ”That's right.”
     Holmes: “I too have been shot five times. But, uh..., separate occasions.”
     Shinwell: “That's... great.”

Footnotes:

[1] Incidents. – wikitech.wikimedia.org/wiki/Incident_documentation
[2] Open tasks. – phabricator.wikimedia.org/maniphest/query…
[3] Wikimedia incident stats. – codepen.io/Krinkle/full/wbYMZK
[4] Month-over-month plots. – docs.google.com/spreadsheets/d/1tRC…

How’d we do in our strive for operational excellence last month? Read on to find out!

📈   Incidents

4 documented incidents in July, and 2 documented incidents in August. [1] Historically, that's on average for this time of year. [5]

For more about recent incidents see Incident documentation on Wikitech, or Preventive measures in Phabricator.

📊   Trends

Take a look at the workboard and look for tasks that could use your help.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

• ⚠️ July 2019 (4 of 18 tasks left): One task closed.
• ⚠️ August 2019 (1 of 14 tasks left): no change.
• ⚠️ September 2019 (3 of 12 tasks left): Two tasks closed.
• October (6 of 12 tasks left), no change.
• November (3 of 5 tasks left): no change.
• December (3 of 9 tasks left), Two tasks closed.
• January 2020 (5 of 7 tasks lef), no change.
• February (2 of 7 tasks left), Two tasks closed.
• March (2 of 2 tasks left), no change.
• April (10 of 14 tasks left): One task closed.
• May (7 of 14 tasks left): Four tasks closed.
• June (10 of 14 tasks left): Four tasks closed.
• July 2020: 13 of 24 new tasks survived the month of July and remain open today.
• August 2020: 37 of 53 new tasks survived the month of August and remain open today.

Previously, we had 72 open production errors over the recent months up to June. Since then, 16 of those were closed. But, the 13 and 37 errors surviving July and August raise our recent tally to 106.

The workboard overall (including tasks from 2019 and earlier) held 192 open production errors on July 23rd. As of writing, the workboard holds 296 open tasks in total. [4] This +104 increase is largely due to the merged backlog of JavaScript client errors, which were previously untracked. Note that we backdated the majority of these JS errors under “Old”, and thus are not amongst the elevated numbers of July and August.

🎉   Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

👊🍺 Tyler: “You know man, it could be worse! […]” Narrator: “[but] I was close... to being complete.”

Tyler: “Martha's polishing the brass on the Titanic. It's all going down, man. […] Evolve! Let the chips fall where they may.”
Narrator: “What!?” Tyler: “The things you own..., they end up owning you.”

Footnotes:
[1] Incidents. – https://wikitech.wikimedia.org/wiki/Incident_documentation
[2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query…
[3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query…
[4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query…
[5] Wikimedia incident stats. – https://codepen.io/Krinkle/full/wbYMZK

How’d we do in our strive for operational excellence last month? Read on to find out!

📈 Month in review

• 4 documented incidents in June. [1]
• 37 new production errors were filed and 27 were closed. [2] [3]
• 72 recent production errors still open (up from 68).
• 203 total Wikimedia-prod-error tasks currently open (up from 192). [4]

For more about recent incidents see Incident documentation, on Wikitech or Preventive measures in Phabricator.

📖 Outstanding errors

Breakdown of new errors reported in June that are still open today:

1. (Needs owner) / Newsletter extension: Unexpected locking SELECT query. T253926
2. (Needs owner) / FlaggedRevs extension: Unable to submit review of page due to bad fr_page_id record. T256296
3. Editing team / MassMessage extension: Delivery fails due to system user conflict. T171003
4. Parsing team / Parsoid: Pagebundle data unavailable due to a bad UTF-8 string. T236866
5. Growth team / Recent changes: Update for ActiveUsers data failing due to deadlock. T255059
6. Growth team / GrowthExperiments: Issue with question display on personal homepage. T255616
7. Language team / Translate extension: Update jobs fail due to invalid function call. T255669
8. Language team / ContentTranslation: Save action fails due to duplicate insert query. T256230
9. Core Platform team / Content handling: Incompatible content type during content merge/stash. T255700
10. Core Platform team / Monolog: API usage logs and error logs sometimes missing due to socket failure. T255578
11. Search Platform team / WikibaseCirrus: Elevated error levels from EntitySearchElastic warnings. T255658
12. Wikidata / API: Generator query fails due to invalid API result format. T254334
13. Wikidata / API: EntityData query emits warning about bad RDF. T255054
14. Wikidata / Repo: Entity relation update jobs fail due to deadlock. T255706

📊 Trends

Take a look at the workboard and look for tasks that could use your help.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

• July 2019 (5 of 18 tasks left): Two tasks closed.
• August (1 of 14 tasks left): Another task closed, only one remaining! 🚀
• September (5 of 12 tasks left): Two tasks closed.
• October (6 of 12 tasks left), no change.
• November (3 of 5 tasks left): Another task closed.
• December (5 of 9 tasks left), no change.
• January 2020 (5 of 7 tasks lef), no change.
• February (4 of 7 tasks left), no change.
• March (2 of 2 tasks left), no change.
• April (11 of 14 tasks left): Three tasks closed.
• May (11 tasks left): Three tasks closed.
• June: 14 new tasks survived the month of June. ⚠️

At the end of May the number of open production errors over recent months was 68. Of those, 10 got closed, but with 14 new tasks from June still open, the total has grown further to 72.

The workboard had 192 open tasks last month, which saw another increase, to now 203 open tasks (this includes tasks from 2019 and earlier).

🎉 Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

⛰ ATC: “Do you want to report a UFO?” Pilot: “Negative. We don't want to report.”
   ATC: “Do you wish to file a report of any kind to us?” Pilot: “I wouldn't know what kind of report to file.”
  ATC: “Me neither…”

Footnotes:
[1] Incidents. – https://wikitech.wikimedia.org/wiki/Incident_documentation#2020
[2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query/VTpmvaJLYVL1/#R
[3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query/qn5yeURqyl3D/#R
[4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query/Fw3RdXt1Sdxp/#R

In 2015 I noticed git fetches from our most active repositories to be unreasonably slow, sometimes up to a minute which hindered fast development and collaboration. You can read some of the debugging details I have conducted at the time on T103990. Gerrit upstream was aware of the issue and a workaround was presented though we never went to implement it.

When fetching source code from a git repository, the client and server conduct a negotiation to discover which objects have to be sent. The server sends an advertisement that lists every single reference it knows about. For a very active repository in Gerrit it means sending references for each patchset and each change ever made to the repository, or almost 200,000 references for mediawiki/core. That is a noticeable amount of data resulting in a slow fetch, especially on a slow internet connection.

Gerrit originated at Google and has full time maintainers. In 2017 a team at Google went to tackle the problem and proposed a new protocol to address the issue, and they closely worked with git maintainers while doing so. The new protocol makes git smarter during the advertisement phase, notably to filter out references the client is not interested in. You can read Google introduction post at https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html

Since June 28th 2020, our Gerrit has been upgraded and now supports git protocol version 2. But to benefit from faster fetches, your client also needs to know about the newer protocol and have it explicitly enabled. For git, you will want version 2.18 or later. Enable the new protocol by setting git configuration protocol.version to 2.

It can be done either on an on demand basis:

git -c protocol.version=2 fetch

Or enabled in your user configuration file:

[protocol]
    version = 2

On my internet connection, fetching for mediawiki/core.git went from ~15 seconds to just 3 seconds. A noticeable difference in my day to day activity.

If you encounter any issue with the new protocol, you can file a task in our Phabricator and tag it with git-protocol-v2.

How’d we do in our strive for operational excellence last month? Read on to find out!

📊   Month in numbers

• 5 documented incidents in May. [1]
• 28 new production error tasks filed in May. [2] [3]
• 68 recent production errors currently open (up from 61).
• 193 currently open Wikimedia-prod-error tasks (up from 178). [4]

For more about recent incidents see Incident documentation on Wikitech, or Preventive measures in Phabricator.

📉   Outstanding reports

Take a look at the workboard and look for tasks that could use your help.
→  https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Breakdown of recent months:

• July 2019: One task closed, 7 of 18 tasks left. ⚠️
• August: 2 of 14 tasks left (unchanged).
• September: 7 of 12 tasks left (unchanged).
• October: 4 of 12 tasks left (unchanged).
• November: 4 of 5 tasks left (unchanged).
• December: 4 of 9 tasks left (unchanged).
• January 2020: 5 of 7 tasks left (unchanged).
• February: Two tasks closed, 4 of 7 tasks left. ⚠️
• March: 2 of 2 tasks left (unchanged).
• April: 14 of 14 tasks left (unchanged).
• May: 14 new tasks survived the month of May.

At the end of April the total of open production errors over recent months was 61. Of those, 7 got closed, but with 14 new tasks from May still open, the total has grown to 68.

The workboard had 178 open tasks in April, which saw a steep increase to now 192 open tasks (this includes June 2020 so far, and pre-2019 tasks).

🎉   Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:
[1] Incidents. – https://wikitech.wikimedia.org/wiki/Incident_documentation#2020
[2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query/7Z4Us2BS02Uo/#R
[3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query/FoIFMu5UO8pw/#R
[4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query/Fw3RdXt1Sdxp/#R

Earlier today, the 600,000th commit was pushed to Wikimedia's Gerrit server. We thought we'd take this moment to reflect on the developer services we offer and our community of developers, be they Wikimedia staff, third party workers, or volunteers.

At Wikimedia, we currently use a self-hosted installation of Gerrit to provide code review workflow management, and code hosting and browsing. We adopted this in 2011–12, replacing Apache Subversion.

Within Gerrit, we host several thousand repositories of code (2,441 as of today). This includes MediaWiki itself, plus all the many hundreds of extensions and skins people have created for use with MediaWiki. Approximately 90% of the MediaWiki extensions we host are not used by Wikimedia, only by third parties. We also host key Wikimedia server configuration repositories like puppet or site config, build artefacts like vetted docker images for production services or local .deb build repos for software we use like etherpad-lite, ancillary software like our special database exporting orchestration tool for dumps.wikimedia.org, and dozens of other uses.

Gerrit is not just (or even primarily) a code hosting service, but a code review workflow tool. Per the Wikimedia code review policy, all MediaWiki code heading to production should go through separate development and code review for security, performance, quality, and community reasons. Reviewers are required to use their "good judgement and careful action", which is a heavy burden, because "[m]erging a change to the MediaWiki core or an extension deployed by Wikimedia is a big deal". Gerrit helps them do this, providing clear views of what is changing, supporting itemised, character-level, file-level, or commit-level feedback and revision, and allowing series of complex changes to be chained together across multiple repositories, and ensuring that forthcoming and merged changes are visible to product owners, development teams, and other interested parties.

Across all of repositories, we average over 200 human commits a day, though activity levels vary widely. Some repositories have dozens of patches a week (MediaWiki itself gets almost 20 patches a day; puppet gets nearly 30), whereas others get a patch every few years. There are over 8,000 accounts registered with Gerrit, although activity is not distributed uniformly throughout that cohort.

To focus engineer time where it's needed, a fair amount of low-risk development work is automated. This happens in both creating patches and also, in some cases, merging them.

For example, for many years we have partnered with TranslateWiki.net's volunteer community to translate and maintain MediaWiki interfaces in hundreds of languages. Exports of translators' updates are pushed and merged automatically by one of the TWN team each day, helping our users keep a fresh, usable system whatever their preferred language.

Another key area is LibraryUpgrader, a custom tool to automatically upgrade the libraries we use for continuous integration across hundreds of repositories, allowing us to make improvements and increase standards without a single central breaking change. Indeed, the 600,000th commit was one of these automatic commits, upgrading the version of the mediawiki-codesniffer tool in the GroupsSidebar extension to the latest version, ensuring it is written following the latest Wikimedia coding conventions for PHP.

Right now, we're working on upgrading our installation of Gerrit, moving from our old version based on the 2.x branch through 2.16 to 3.1, which will mean a new user interface and other user-facing changes, as well as improvements behind the scenes. More on those changes will be coming in later posts.

Header image: A vehicle used to transport miners to and from the mine face by 'undergrounddarkride', used under CC-BY-2.0.

How are we doing on that strive for operational excellence during these unprecedented times?

📊  Numbers for March and April

• 3 documented incidents. [1]
• 60 new Wikimedia-prod-error reports. [2]
• 58 Wikimedia-prod-error reports closed. [3]
• 178 currently open Wikimedia-prod-error reports in total. [4]

For more about recent incidents and pending actionables see Wikitech and Phabricator.

📉  Outstanding reports

Take a look at the workboard and look for tasks that could use your help.

→  https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Breakdown of recent months:

• April 2019: Two reports closed, 2 of 14 left.
• May: (All clear!)
• June: 4 of 11 left (unchanged). ⚠️
• July: 8 of 18 left (unchanged).
• August: 2 of 14 reports left (unchanged).
• September: 7 of 12 left (unchanged).
• October: Two reports closed, 4 of 12 left.
• November: One report closed, 4 of 5 left.
• December: Two reports closed, 4 of 9 left.
• January 2020: One report closed, 5 of 7 reports left.
• February: One report closed, 6 of 7 reports left.
• March: 2 new reports survived the month of March.
• April: 13 new reports survived the month of April.

At the end of February the total of open reports over recent months was 58. Of those, 12 got closed, but with 15 new reports from March/April still open, the total is now up at 61 open reports.

The workboard overall (which includes pre-2019 tasks) has 178 tasks open. This is actually down by a bit for the first time since October with December at 196, January at 198, and February at 199, and now April at 178. This was largely due to the Release Engineering and Core Platform teams closing out forgotten reports that have since been resolved or otherwise obsoleted.

💡 Tip: Verifying existing tasks is a good way to (re)familiarise yourself with Kibana. For example: Does the error still occur in the last 30 days? Does it only happen on a certain wiki? What do the URLs or stack traces have in common?

🎉  Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:
[1] Incidents. – https://wikitech.wikimedia.org/wiki/Incident_documentation
[2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query/HjopcKClxTfw/#R
[3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query/ts62HKYPBxod/#R
[4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query/Fw3RdXt1Sdxp/#R

How’d we do in our strive for operational excellence last month? Read on to find out!

📊  Month in numbers

• 8 documented incidents. [1]
• 27 new Wikimedia-prod-error reports. [2]
• 26 Wikimedia-prod-error reports closed. [3]
• 199 currently open Wikimedia-prod-error reports in total. [4]

With a median of 4–5 documented incidents per month (over the last three years), there were a fairly large number of them this past month.

To read more about these incidents and pending actionables; check Incident documentation § 2020, or Explore Wikimedia incident stats (interactive).

📖  Unset vs array splice

Our error monitor (Logstash) received numerous reports about an “Undefined offset” error from the OATHAuth extension. This extension powers the Two-factor auth (2FA) login interface on Wikipedia.

@ItSpiderman and @Reedy investigated the problem. The error message:

PHP Notice: Undefined offset: 8
at /srv/mediawiki/extensions/OATHAuth/src/Key/TOTPKey.php:188

This error means that the code was accessing item number 8 from a list (an array), but the item does not exist. Normally, when a “2FA scratch token” is used, we remove it from a list, and save the remaining list for next time.

The code used the count() function to compute the length of the list, and used a for-loop to iterate through the list. When the code found the user’s token, it used the unset( $list[$num] ) operation to remove token $num from the list, and then save $list for next time.

The problem with removing a list item in this way is that it leaves a “gap”. Imagine a list with 4 items, like [ 1: …, 2: …, 3: … , 4: … ]. If we unset item 2, then the remaining list will be [ 1: …, 3: …, 4: … ]. The next time we check this list, the length of the list is now 3 (so far so good!), but the for-loop will access the items as 1-2-3. The code would not know that 3 comes after 1, causing an error because item 2 does not exist. And, the code would not even look at item 4!

When a user used their first ever scratch token, everything worked fine. But from their second token onwards, the tokens could be rejected as “wrong” because the code was not able to find them.

To avoid this bug, we changed the code to use array_splice( $list, $num, 1 ) instead of unset( $list[$num] ). The important thing about array_splice is that it renumbers the items in the list, leaving no gaps.

– T244308 / https://gerrit.wikimedia.org/r/570253

📉  Outstanding reports

Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Breakdown of recent months:

• March: 3 of 10 reports left (unchanged). ⚠️
• April: 4 of 14 left (unchanged).
• May: (All clear!)
• June: 4 of 11 left (unchanged).
• July: 8 of 18 left (unchanged).
• August: Two reports closed! 2 of 14 reports left.
• September: One report closed, 7 of 12 left.
• October: Two reports closed, 6 of 12 left.
• November: 5 of 5 left (unchanged).
• December: 6 of 9 left (unchanged).
• January: One report closed, 6 of 7 reports left.
• February: 7 new reports survived the month of February.

Last month’s total over recent months was 57 open reports. Of those, 6 got closed, but with 7 new reports from February still open, the total is now up at 58 open reports.

🎉  Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production.

Together, we’re getting there!

Until next time,

– Timo Tijhof

Footnotes:
[1] Incidents. – wikitech.wikimedia.org/wiki/Incident_documentation#2020
[2] Tasks created. – phabricator.wikimedia.org/maniphest/query…
[3] Tasks closed. – phabricator.wikimedia.org/maniphest/query…
[4] Open tasks. – phabricator.wikimedia.org/maniphest/query…

How’d we do in our strive for operational excellence last month? Read on to find out!

📊  Month in numbers

• 3 documented incidents. [1]
• 26 new Wikimedia-prod-error reports. [2]
• 26 Wikimedia-prod-error reports closed. [3]
• 198 currently open Wikimedia-prod-error reports in total. [4]

To read more about these incidents and pending actionables; check Incident documentation § 2020, or Explore Wikimedia incident stats (interactive).

📖  Paradoxical array key

Wikimedia encountered several Zend engine bugs that could corrupt a PHP program at run-time, during the upgrade from HHVM to PHP 7.2. (Some of these bugs are still being worked on.) One of the bugs we fixed last month was particularly mysterious. Investigation led by @hashar and @tstarling.

MediaWiki would create an array in PHP and add a key-value pair to it. We could iterate this array, and see that our key was there. Moments later, if we tried to retrieve the key from that same array, sometimes the key would no longer exist!

After many ad-hoc debug logs, core dumps, and GDB sessions, the problem was tracked down to the string interning system of Zend PHP. String interning is a memory reduction technique. It means we only store one copy of a character sequence in RAM, even if many parts of the code use the same character sequence. For example, the words “user” and “edit” are frequently used in the MediaWiki codebase. One of those sequences is the empty string (“”), which is also used a lot in our code. This is the string we found disappearing most often from our PHP arrays. This bug affected several components, including Wikibase, the wikimedia/rdbms library, and ResourceLoader.

Tim used a hardware watchpoint in GDB, and traced the root cause to the Memcached client for PHP. The php-memcached client would “free” a string directly from the internal memory manager after doing some work. It did this even for “interned” strings that other parts of the program may still be depending on.

@jijiki and @Joe backported the upstream fix to our php-memcached package and deployed it to production. Thanks! — T232613

📉  Outstanding reports

Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Breakdown of recent months (past two weeks not included):

• March: 3 of 10 reports left (unchanged). ⚠️
• April: Two reports closed, 4 of 14 left.
• May: (All clear!)
• June: Two reports closed. 4 of 11 left.
• July: Four reports closed, 8 of 18 left.
• August: 4 of 14 reports left (unchanged).
• September: One report closed, 8 of 12 left.
• October: 8 of 12 left (unchanged).
• November: 5 of 5 left (unchanged).
• December: Three reports closed, 6 of 9 left.
• January: 7 new reports survived the month of January.

There are a total of 57 reports filed in recent months that remain open. This is down from 62 last month.

🎉  Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Footnotes:
[1] Incidents. – wikitech.wikimedia.org/wiki/Incident_documentation#2019
[2] Tasks created. – phabricator.wikimedia.org/maniphest/query…
[3] Tasks closed. – phabricator.wikimedia.org/maniphest/query…
[4] Open tasks. – phabricator.wikimedia.org/maniphest/query…

How’d we do in our strive for operational excellence in November and December? Read on to find out!

📊 Month in numbers

• 0 documented incidents in November, 5 incidents in December. [1]
• 17 new Wikimedia-prod-error reports. [2]
• 23 Wikimedia-prod-error reports closed. [3]
• 190 currently open Wikimedia-prod-error reports in total. [4]

November had zero reported incidents. Prior to this, the last month with no documented incidents was December 2017. To read about past incidents and unresolved actionables; check Incident documentation § 2019.

Explore Wikimedia incident graphs (interactive)

📖 Many dots, do not a query make!

@dcausse investigated a flood of exceptions from SpecialSearch, which reported “Cannot consume query at offset 0 (need to go to 7296)”. This exception served as a safeguard in the parser for search queries. The code path was not meant to be reached. The root cause was narrowed down to the following regex:

/\G(?<negated>[-!](?=[\w]))?(?<word>(?:\\\\.|[!-](?!")|[^"!\pZ\pC-])+)/u

This regex looks complex, but it can actually be simplified to:

/(?:ab|c)+/

This regex still triggers the problematic behavior in PHP. It fails with a PREG_JIT_STACKLIMIT_ERROR, when given a long string. Below is a reduced test case:

$ret = preg_match( '/(?:ab|c)+/', str_repeat( 'c', 8192 ) );
if ( $ret === false ) {
    print( "failed with: " . preg_last_error() );
}

• Fails when given 1365 contiguous c on PHP 7.0.
• Fails with 2731 characters on PHP 7.2, PHP 7.1, and PHP 7.0.13.
• Fails with 8192 characters on PHP 7.3. (Might be due to php-src@bb2f1a6).

In the end, the fix we applied was to split the regex into two separate ones, and remove the non-capturing group with a quantifier, and loop through at the PHP level (Gerrit change 546209).

The lesson learned here is that the code did not properly check the return value of preg_match, this is even more important as the size allowed for the JIT stack changes between PHP versions.

For future reference, @dcausse concluded: The regex could be optimized to support more chars (~3 times more) by using atomic groups, like so /(?>ab|c)+/. — T236419

📉 Outstanding reports

Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Or help someone that’s already started with their patch:

→ Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

• March: 3 of 10 reports left. (unchanged). ⚠️
• April: Three reports closed, 6 of 14 left.
• May: (All clear!)
• June: Three reports closed. 6 of 11 left (unchanged). ⚠️
• July: One report closed, 12 of 18 left.
• August: Two reports closed, 4 of 14 left.
• September: One report closed, with 9 of 12 left.
• October: Four reports closed, 8 of 12 left.
• November: 5 new reports survived the month of November.
• December: 9 new reports survived the month of December.

🎉 Thanks!

Thank you to everyone who helped by reporting, investigating, or resolving problems in Wikimedia production.

Until next time,

– Timo Tijhof

Footnotes:
[1] Incidents. – wikitech.wikimedia.org/wiki/Incident_documentation#2019
[2] Tasks created. – phabricator.wikimedia.org/maniphest/query…
[3] Tasks closed. – phabricator.wikimedia.org/maniphest/query…
[4] Open tasks. – phabricator.wikimedia.org/maniphest/query…

How’d we do in our strive for operational excellence last month? Read on to find out!

📊 Month in numbers

• 3 documented incidents. [1]
• 33 new Wikimedia-prod-error reports. [2]
• 30 Wikimedia-prod-error reports closed. [3]
• 207 currently open Wikimedia-prod-error reports in total. [4]

There were three recorded incidents last month, which is slightly below our median of the past two years (Explore this data). To read more about these incidents, their investigations, and pending actionables; check Incident documentation § 2019.

📖 To Log or not To Log

MediaWiki uses the PSR-3 compliant Monolog library to send messages to Logstash (via rsyslog and Kafka). These messages are used to automatically detect (by quantity) when the production cluster is in an unstable state. For example, due to an increase in application errors when deploying code, or if a backend system is failing. Two distinct issues hampered the storing of these messages this month, and both affected us simultaneously.

Elasticsearch mapping limit

The Elasticsearch storage behind Logstash optimises responses to Logstash queries with an index. This index has an upper limit to how many distinct fields (or columns) it can have. When reached, messages with fields not yet in the index are discarded. Our Logstash indexes are sharded by date and source (one for “mediawiki”, one for “syslog”, and one for everthing else).

This meant that error messages were only stored if they only contained fields used before, by other errors stored that day. Which in turn would only succeed if that day’s columns weren’t already fully taken. A seemingly random subset of error messages was then rejected for a full day. Each day it got a new chance at reserving its columns, so long as the specific kind of error is triggered early enough.

To unblock deployment automation and monitoring of MediaWiki, an interim solution was devised. The subset of messages from “mediawiki” that deal with application errors now have their own index shard. These error reports follow a consistent structure, and contain no free-form context fields. As such, this index (hopefully) can’t reach its mapping limit or suffer message loss.

The general index mapping limit was also raised from 1000 to 2000. For now that means we’re not dropping any non-critical/debug messages. More information about the incident at T234564. The general issue with accommodating debug messages in Logstash long-term, is tracked at T180051. Thanks @matmarex, @hashar, and @herron.

Crash handling

Wikimedia’s PHP configuration has a “crash handler” that kicks in if everything else fails. For example, when the memory limit or execution timeout is reached, or if some crucial part of MediaWiki fails very early on. In that case our crash handler renders a Wikimedia-branded system error page (separate from MediaWiki and its skins). It also increments a counter metric for monitoring purposes, and sends a detailed report to Logstash. In migrating the crash handler from HHVM to PHP7, one part of the puzzle was forgotten. Namely the Logstash configuration that forwards these reports from php-fpm’s syslog channel to the one for mediawiki.

As such, our deployment automation and several Logstash dashboards were blind to a subset of potential fatal errors for a few days. Regressions during that week were instead found by manually digging through the raw feed of the php-fpm channel instead. As a temporary measure, Scap was updated to consider the php-fpm’s channel as well in its automation that decides whether a deployment is “green”.

We’ve created new Logstash configurations that forward PHP7 crashes in a similar way as we did for HHVM in the past. Bookmarked MW dashboards/queries you have for Logstash now provide a complete picture once again. Thanks @jijiki and @colewhite! – T234283

📉 Outstanding reports

Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Or help someone that’s already started with their patch:
→ Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

• March: 1 report fixed. (3 of 10 reports left).
• April: 8 of 14 reports left (unchanged). ⚠️
• May: (All clear!)
• June: 9 of 11 reports left (unchanged). ⚠️
• July: 13 of 18 reports left (unchanged).
• August: 2 reports were fixed! (6 of 14 reports left).
• September: 2 reports were fixed! (10 of 12 new reports left).
• October: 12 new reports survived the month of October.

🎉 Thanks!

Thank you, to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

🌴“Gotta love crab. In time, too. I couldn't take much more of those coconuts. Coconut milk is a natural laxative. That's something Gilligan never told us.”

Footnotes:
[1] Incidents. – wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident…
[2] Tasks created. – phabricator.wikimedia.org/maniphest/query…
[3] Tasks closed. – phabricator.wikimedia.org/maniphest/query…
[4] Open tasks. – phabricator.wikimedia.org/maniphest/query…

Introduction

This past week marks the release of a little tool that I've been working on for a while. In fact, it's something I've wanted to build for more than a year. But before I tell you about the solution, I need to describe the problem that I set out to solve.

Problem

Production errors are tracked with the tag Wikimedia-production-error. As a member of the Release-Engineering-Team, I've spent a significant amount of time copying details from Kibana log entries and pasting into the Production Error Report form here in Phabricator. There are several of us who do this on a regular basis, including most of my team and several others as well. I don't know precisely how much time is spent on error reporting but at least a handful of people are going through this process several times each week.

This is what lead to the idea for rPHAT Phatality: I recognized immediately that if I could streamline the process and save even a few seconds each time, the aggregate time savings could really add up quickly.

Solution

So after considering a few ways in which the process could be automated or otherwise streamlined, I finally focused on what seemed like the most practical: build a Kibana plugin that will format the log details and send them over to Phabricator, eliminating the tedious series of copy/paste operations.

Phatality has a couple of other tricks up it's sleeve but the essence of it is just that: capture all of the pertinent details from a single log message in Kibana and send it to Phabricator all at once with the click of a button in Kibana.

Clicking the [Submit] button, as seen in the above screenshot, will take you to the phabricator Production Error form with all of the details pre-filled and ready to submit:

Conclusion

Now that Phatality is deployed to production and a few of us have had a chance to use it to submit error reports, I can say that I definitely think it was a worthwhile effort. The Kibana plugin wasn't terribly difficult to write, and thanks to @fgiunchedi's help, the deployment went fairly smoothly. Phatality definitely streamlines the reporting process, saving several clicks each time and ensuring accuracy in the details that get sent to Phabricator. In a future version of the tool I plan to add more features such as duplicate detection to help avoid duplicate submissions.

If you use Wikimedia's Kibana to report errors in Phabricator then I encourage you to look for the Phatality tab in the log details section and save some clicks!

What other repetitive tasks are ripe for automation? I'd love to hear suggestions and ideas in the comments.

In Changes and improvements to PHPUnit testing in MediaWiki, I wrote about efforts to help speed up PHPUnit code coverage generation for local development.[0] While this improves code coverage generation time for local development, it could be better.

As the Manual:PHP unit testing/Code coverage page advises, adjusting the whitelist in the PHPUnit XML configuration can speed things up dramatically. The problem is, adjusting that file is a manual process and a little cumbersome, so I usually didn't do it. And then because code coverage generation reports were slow locally[1], I ended up not running them while working on a patch. True, you will get feedback on code coverage metrics from CI, but it would be nicer if you could quickly get this information in your local environment first.

This was the motivation to add a Composer script in MediaWiki core that will help you adjust the PHPUnit coverage whitelist quickly while you're working on a patch for an extension or skin.

You can run it with composer phpunit:coverage-edit -- extensions/$EXT_NAME, e.g. composer phpunit:coverage-edit -- extensions/GrowthExperiments.

The ComposerPhpunitXmlCoverageEdit.php script copies the phpunit.xml.dist file to phpunit.xml (not version controlled), and modifies the whitelist to add directories for that extension/skin. vendor/bin/phpunit then reads phpunit.xml instead of the phpunit.xml.dist file. Tip: Make sure "Edit configurations" in your IDE (PhpStorm in my case) is using vendor/bin/phpunit and phpunit.xml, not phpunit.xml.dist, when executing the tests.

When you want to reset your configuration, you can rm phpunit.xml and vendor/bin/phpunit will read from phpunit.xml.dist again.

Further improvements to the script could include:

• Reading the extension.json file to determine which directories to add to the whitelist, rather than using a hardcoded list (T235029)
• Allow passing arbitrary directories/filenames, e.g. for working with subsections of core or of a larger extension (T235030)
• Adding a flag for flipping the addUncoveredFilesFromWhitelist property, so that phpunit-suite-edit.py in the integration/config repo could be removed in favor of the Composer script (T235031)

Thanks to @Mainframe98 and @Krinkle for review of the patch and to @AnneT for reviewing this post. Happy hacking!

[0] [[ https://gerrit.wikimedia.org/r/c/mediawiki/core/+/520459 | One patch changed <whitelist addUncoveredFilesFromWhitelist="true"> to false ]] to help speed up PHPUnit code coverage generation, the [[ https://gerrit.wikimedia.org/r/c/integration/config/+/521190 | second patch flipped the flag back to true in CI ]] for generating complete coverage reports.
[1] For GrowthExperiments, generating coverage reports without a customized whitelist takes ~17 seconds. With a custom whitelist, it takes ~1 second. While 17 seconds is arguably not a lot of time, the near-instant feedback with a customized whitelist means one is less likely to face interruptions to their flow or concentration while working on a patch.

How’d we do in our strive for operational excellence last month? Read on to find out!

📊 Month in numbers

• 5 documented incidents. [1]
• 22 new errors reported. [2]
• 31 error reports closed. [3]
• 213 currently open Wikimedia-prod-error reports in total. [4]

There were five recorded incidents last month, equal to the median for this and last year. – Explore this data.

To read more about these incidents, their investigations, and pending actionables; check Incident documentation § 2019.

*️⃣ A Tale of Three Great Upgrades

This month saw three major upgrades across the MediaWiki stack.

Migrate from HHVM to PHP 7.2

The client-side switch to toggle between HHVM and PHP 7.2 saw its final push — from the 50% it was at previously, to 100% of page view sessions on 17 September. The switch further solidified on 24 September when static MediaWiki traffic followed suit (e.g. API and ResourceLoader). Thanks @jijiki and @Joe for the final push. – More details at T219150 and T176370.

Drop support for IE6 and IE7

The RFC to discontinue basic compatibility for the IE6 and IE7 browsers entered Last Call on 18 September. It was approved on 2 Oct (T232563). Thanks to @Volker_E for leading the sprint to optimise our CSS payloads by removing now-redundant style rules for IE6-7 compat. – More at T234582.

Transition from PHPUnit 4/6 to PHPUnit 8

With HHVM behind us, our Composer configuration no longer needs to be compatible with a “PHP 5.6 like” run-time. Support for the real PHP 5.6 was dropped over 2 years ago, and the HHVM engine supports PHP 7 features. But, the HHVM engine identifies as “PHP 5.6.999-hhvm”. As such, Composer refused to install PHPUnit 6 (which requires PHP 7.0+). Instead, Composer could only install PHPUnit 4 under HHVM (as for PHP 5.6). Our unit tests have had to remain compatible with both PHPUnit 4 and PHPUnit 6 simultaneously.

Now that we’re fully on PHP 7.2+, our Composer configuration effectively drops PHP 5.6, 7.0 and 7.1 all at once. This means that we no longer run PHPUnit tests on multiple PHPUnit versions (PHPUnit 6 only). The upgrade to PHPUnit 8 (PHP 7.2+) is also unlocked! Thanks @MaxSem, @Jdforrester-WMF and @Daimona for leading this transition. – T192167

📉 Outstanding reports

Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Or help someone that’s already started with their patch:
→ Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

• February: 1 report was closed. (1 / 5 reports left).
• March: 4 / 10 reports left (unchanged).
• April: 8 / 14 reports left (unchanged). ⚠️
• May: The last 4 reports were resolved. Done! ❇️
• June: 9 of 11 reports left (unchanged). ⚠️
• July: 4 reports were fixed! (13 / 18 reports left).
• August: 6 reports were fixed! (8 / 14 reports left).
• September: 12 new reports survived the month of September.

🎉 Thanks!

Thank you, to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

📖“I'm not crazy about reality, but it's still the only place to get a decent meal.”

Footnotes:

[1] Incidents. –
wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident…

[2] Tasks created. –
phabricator.wikimedia.org/maniphest/query…

[3] Tasks closed. –
phabricator.wikimedia.org/maniphest/query…

[4] Open tasks. –
phabricator.wikimedia.org/maniphest/query…

How’d we do in our strive for operational excellence in August? Read on to find out!

📊 Month in numbers

• 3 documented incidents. [1]
• 42 new Wikimedia-prod-error reports. [2]
• 31 Wikimedia-prod-error reports closed. [3]
• 210 currently open Wikimedia-prod-error reports in total. [4]

The number of recorded incidents in August, at three, was below average for the year so far. However, in previous years (2017-2018), August also has 2-3 incidents. – Explore this data.

To read more about these incidents, their investigations, and pending actionables; check Incident documentation § 2019.

*️⃣ When you have eliminated the impossible...

Reports from Logstash indicated that some user requests were aborted by a fatal PHP error from the MessageCache class. The user would be shown a generic system error page. The affected requests didn’t seem to have anything obvious in common, however. This made it difficult to diagnose.

MessageCache is responsible for fetching interface messages, such as the localised word “Edit” on the edit button. It calls a “load()” function and then tries to access the loaded information. However, sometimes the load function would claimed to have finished its work, but yet the information was not there.

When the load function initialises all the messages for a particular language, it keeps track of this, so as to not do the same a second time. From any one angle I could look at this code, no obvious mistakes stood out. A deeper investigation revealed that two unrelated changes (more than a year apart), each broke 1 assumption that was safe to break. But, put together, and this seemingly impossible problem emerges. Check out T208897#5373846 for the details of the investigation.

📉 Outstanding reports

Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Or help someone that’s already started with their patch:
→ Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

• January: 1 report left (unchanged). ⚠️
• February: 2 reports left (unchanged). ⚠️
• March: 4 reports left (unchanged). ⚠️
• April: 2 reports got fixed! (8 of 14 reports left). ❇️
• May: 4 of 10 reports left (unchanged).
• June: 1 report got fixed! (8 of 11 reports left). ❇️
• July: 2 reports got fixed (17 of 18 reports left).
• August: 14 new reports remain unsolved.
• September: 11 new reports remain unsolved.

🎉 Thanks!

Thank you to @aaron, @Catrope, @Daimona, @dbarratt, @Jdforrester-WMF, @kostajh, @pmiazga, @Tarrow, @zeljkofilipin, and everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

🎭“I think you should call it Seb's because no one will come to a place called Chicken on a Stick.”

Footnotes:

[1] Incidents. – wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident…

[2] Tasks created. – phabricator.wikimedia.org/maniphest/query…

[3] Tasks closed. – phabricator.wikimedia.org/maniphest/query…

[4] Open tasks. – phabricator.wikimedia.org/maniphest/query…

Building off the work done at the Prague Hackathon (T216260), we're happy to announce some significant changes and improvements to the PHP testing tools included with MediaWiki.

PHP unit tests can now be run statically, without installing MediaWiki

You can now download MediaWiki, run composer install, and then composer phpunit:unit to run core's unit test suite (T89432).

The standard PHPUnit entrypoint can be used, instead of the PHPUnit Maintenance class

You can now use the plain PHPUnit entrypoint at vendor/bin/phpunit instead of the MediaWiki maintenance class which wraps PHPUnit (tests/phpunit/phpunit.php).

Both the unit tests and integration tests can be executed with the standard phpunit entrypoint (vendor/bin/phpunit) or if you prefer, with the composer scripts defined in composer.json (e.g. composer phpunit:unit). We accomplished this by writing a new bootstrap.php file (the old one which the maintenance class uses was moved to tests/phpunit/bootstrap.maintenance.php) which executes the minimal amount of code necessary to make core, extension and skin classes discoverable by test classes.

Tests should be placed in tests/phpunit/{integration,unit}

Integration tests should be placed in tests/phpunit/integration while unit tests go in tests/phpunit/unit, these are discoverable by the new test suites (T87781). It sounds obvious now to write this, but a nice side effect is that by organizing tests into these directories it's immediately clear to authors and reviewers what type of test one is looking at.

Introducing MediaWikiUnitTestCase

A new base test case, MediaWikiUnitTestCase has been introduced with a minimal amount of boilerplate (@covers validator, ensuring the globals are disabled, and that the tests are in the proper directory, the default PHPUnit 4 and 6 compatibility layer). The MediaWikiTestCase has been renamed to MediaWikiIntegrationTestCase for clarity.

Please migrate tests to be unit tests where appropriate

A significant portion of core's unit tests have been ported to use MediaWikiUnitTestCase, approximately 50% of the total. We have also worked on porting extension tests to the unit/integration directories. @Ladsgroup wrote a helpful script to assist with automating the identification and moving of unit tests, see P8702. Migrating tests from MediaWikiIntegrationTestCase to MediaWikiUnitTestCase makes them faster.

Note that unit tests in CI are still run with the PHPUnit maintenance class (tests/phpunit/phpunit.php), so when reviewing unit test patches please execute them locally with vendor/bin/phpunit /path/to/tests/phpunit/unit or composer phpunit -- /path/to/tests/phpunit/unit.

Generating code coverage is now faster

The PHPUnit configuration file now resides at the root of the repository, and is called phpunit.xml.dist. (As an aside, you can copy this to phpunit.xml and make local changes, as that file is git-ignored, although you should not need to do that.) We made a modification (T192078) to the PHPUnit configuration inside MediaWiki to speed up code coverage generation. This makes it feasible to have a split window in your IDE (e.g. PhpStorm), run "Debug with coverage", and see the results in your editor fairly quickly after running the tests.

What is next?

Things we are working on:

• Porting core tests to integration/unit
• Porting extension tests to integration/unit.
• Removing legacy testsuites or ensuring they can be run in a different way (passing the directory name for example).
• Switching CI to use new entrypoint for unit tests, then for unit and integration tests

Help is wanted in all areas of the above! We can be found in the #wikimedia-codehealth channel and via the phab issues linked in this post.

Credits

The above work has been done and supported by Máté (@TK-999), Amir (@Ladsgroup), Kosta (@kostajh), James (@Jdforrester-WMF), Timo (@Krinkle), Leszek (@WMDE-leszek), Kunal (@Legoktm), Daniel (@daniel), Michael Große (@Michael), Adam (@awight), Antoine (@hashar), JR (@Jrbranaa) and Greg (@greg) along with several others. Thank you!

thanks for reading, and happy testing!

Amir, Kosta, & Máté

How’re we doing on that strive for operational excellence? Read this first anniversary edition to find out!

📊 Month in numbers

• 5 documented incidents. [1]
• 53 new Wikimedia-prod-error reports. [2]
• 44 closed Wikimedia-prod-error reports. [3]
• 218 currently open Wikimedia-prod-error reports in total. [4]

The number of recorded incidents over the past month, at five, is equal to the median number of incidents per month (2016-2019). – Explore this data.

To read more about these incidents, their investigations, and pending actionables; check Incident documentation § 2019.

📖 One year of Excellent adventures!

Exactly one year ago this periodical started to provide regular insights on production stability. The idea was to shorten the feedback cycle between deployment of code that leads to fatal errors and the discovery of those errors. This allows more people to find reports earlier, which (hopefully) prevents them from sneaking into a growing pile of “normal” errors.

576 reports were created between 15 July 2018 and 31 July 2019 (tagged Wikimedia-prod-error).
425 reports got closed over that same time period.

Read the first issue in story format, or the initial e-mail.

📉 Outstanding reports

Take a look at the workboard and look for tasks that might need your help. The workboard lists error reports, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Or help someone who already started with their patch:
→ Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

• November: 1 report left (unchanged). ⚠️
• December: 3 reports left (unchanged). ⚠️
• January: 1 report left (unchanged). ⚠️
• February: 2 reports left (unchanged). ⚠️
• March: 4 reports left (unchanged). ⚠️
• April: 10 of 14 reports left (unchanged). ⚠️
• May: 2 reports got fixed! (4 of 10 reports left). ❇️
• June: 2 reports got fixed! (9 of 11 reports left). ❇️
• July: 18 new reports from last month remain unsolved.

🎉 Thanks!

Thank you to @aaron, @Anomie, @ArielGlenn, @Catrope, @cscott, @Daimona, @dbarratt, @dcausse, @EBernhardson, @Jdforrester-WMF, @jeena, @MarcoAurelio, @SBisson, @Tchanders, @Tgr, @tstarling, @Urbanecm; and everyone else who helped by finding, investigating, or resolving error reports in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

Quote: 🎙 “Unlike money, hope is for all: for the rich as well as for the poor.”

Footnotes:

[1] Incidents. – wikitech.wikimedia.org/wiki/Special:PrefixIndex?prefix=Incident…

[2] Tasks created. – phabricator.wikimedia.org/maniphest/query…

[3] Tasks closed. – phabricator.wikimedia.org/maniphest/query…

[4] Open tasks. – phabricator.wikimedia.org/maniphest/query…

How’d we do in our strive for operational excellence last month? Read on to find out!

📊 Month in numbers

• 11 documented incidents. ⚠️ [1]
• 39 new Wikimedia-prod-error reports. [2]
• 25 Wikimedia-prod-error reports closed. [3]

The number of incidents in June was high compared to previous years. At 11 incidents, this is higher than this year’s median (5), the 2018 median (4), and the 2017 median (5). It is also higher than any month of June in the last 4 years. – More data at CodePen.

To read more about these incidents, their investigations, and pending actionables; check Incident documentation § 2019.

There are currently 204 open Wikimedia-prod-error reports (up from 186 in April, and 201 in May). [4]

📖 [Op-ed] Integrated maintenance cost

Hereby a shoutout to the Wikidata and Core Platform teams, at WMDE and WMF respectively. They both recently established a rotating subteam that focuses on incidental work. Such as maintenance, and other work that might otherwise hinder feature development.

I expect this to improve efficiency by avoiding context switches between feature and incidental work. The rotational aspect should distribute the work more evenly among team members (avoiding burnout). And, it may increase exposure to other teams, and lesser-known areas of our code; which provide opportunities for personal growth and to retain institutional knowledge.

📉 Current problems

Take a look at the workboard and look for tasks that might need your help. The workboard lists known issues, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error

Or help someone who already started with their patch:
→ Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

• November: 1 issue got fixed! (1 issue left).
• December: 3 issues left (unchanged). ⚠️
• January: 1 issue left (unchanged). ⚠️
• February: 2 issues left (unchanged). ⚠️
• March: 4 issues left (unchanged). ⚠️
• April: 2 issues got fixed! (10 of 14 issues, that survived April, remain open). ❇️
• May: 4 issues got fixed! (6 of 10 issues, that survived May, are left). ❇️
• June: 11 new issues from last month remain unresolved.

By steward and software component, the unresolved issues that survived June:

• CPT / MW Auth (PHP fatal): T228717
• CPT / MW Actor (DB contention): T227739
• CPT or Multimedia / Thumb handler (MultiCurl error): T225197
• Multimedia / File metadata (PHP error): T226751
• Wikidata / Commons page view (PHP fatal): T227360
• Wikidata / Jobrunner (PHP memory fatal): T227450
• Wikidata / Jobrunner (Trx error): T225098
• Product-Infra / ReadingList API (PHP fatal): T226593
• (Unknown?) / Special:ConfirmEmail (PHP fatal): T226337
• (Unknown?) / Page renaming (DB timeout): T226898
• (Unknown?) / Page renaming (Bad revision fatal): T225366

💡Ideas: To suggest something to investigate or highlight in a future edition, contact me by e-mail or private IRC message.

🎉 Thanks!

Thank you to everyone who has helped by reporting, investigating, or resolving problems in Wikimedia production. Including: @Anomie, @brion, @Catrope, @cscott, @daniel, @dcausse, @DerFussi, @Ebe123, @fgiunchedi, @Jdforrester-WMF, @kostajh, @Legoktm, @Lucas_Werkmeister_WMDE, @matmarex, @matthiasmullie, @Michael, @Nikerabbit, @SBisson, @Smalyshev, @Tchanders, @Tgr, @Tpt, @Umherirrender, and @Urbanecm.

Thanks!

Until next time,

– Timo Tijhof

🔮“These are his marbles...” “Ha! He really did lose his marbles, didn't he?” “Yeah, he lost them good.”

Footnotes:

1. Incidents. – wikitech.wikimedia.org/wiki/Special:PrefixIndex…
2. Tasks created. – phabricator.wikimedia.org/maniphest/query…
3. Tasks closed. – phabricator.wikimedia.org/maniphest/query…
4. Open tasks. – phabricator.wikimedia.org/maniphest/query…

How’d we do in our strive for operational excellence last month? Read on to find out!

📊 Month in numbers

• 6 documented incidents. [1]
• 41 new Wikimedia-prod-error tasks created. [2]
• 36 Wikimedia-prod-error tasks closed. [3]

The number of incidents in May of this year was comparable to previous years (6 in May 2019, 2 in May 2018, 5 in May 2017), and previous months (6 in May, 8 in April, 8 in March) – comparisons at CodePen.

To read more about these incidents, their investigations, and pending actionables; check wikitech.wikimedia.org/wiki/Incident_documentation#2019.

As of writing, there are 201 open Wikimedia-prod-error tasks (up from 186 last month). [4]

📉 Current problems

Take a look at the workboard and look for tasks that might need your help. The workboard lists known issues, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error

Or help someone that’s already started with their patch:
→ Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

• November: 2 issues left (unchanged).
• December: 1 issue got fixed. 3 issues left (down from 4).
• January: 1 issue left (unchanged).
• February: 2 issues left (unchanged).
• March: 1 issue got fixed. 4 issues remaining (down from 5).
• April: 2 issues got fixed. 12 issues remain unresolved (down from 14).
• May: 10 new issues found last month survived the month of May, and remain unresolved.

By steward and software component, unresolved issues from April and May:

• Wikidata / Lexeme (API query fatal): T223995
• Wikidata / WikibaseRepo (API Fatal hasSlot): T225104
• Wikidata / WikibaseRepo (Diff link fatal): T224270
• Wikidata / WikibaseRepo (Edit undo fatal): T224030
• Growth / Echo (Notification storage): T217079
• Growth / Flow (Topic link fatal): T224098
• Growth / Page deletion (File pages): T222691
• Multimedia or CPT / API (Image info fatal): T221812
• CPT / PHP7 refactoring (File descriptions): T223728
• CPT / Title refactor (Block log fatal): T224811
• CPT / Title refactor (Pageview fatals): T224814
• (Unstewarded) Page renaming: T223175, T205675