PSA for Civilization V aficionados: the Windows and Mac versions are no longer compatible for online multiplayer.

It seems the game’s online state management is probably based on passing raw game structures, and some types differ between 32-bit and 64-bit versions: the Windows version of the game is 32-bit, but the Mac version was updated to 64-bit last year to allow it to run on recent versions of macOS that dropped 32-bit support.

It’s unclear whether updating the Windows version to 64 bit would resolve the incompatibility, as macOS and Windows have some different types at 64-bit as well.

Sigh.

This was an avoidable problem, by either using device independent serializations or device independent core representations. And it was exacerbated by Apple dropping 32-bit compatibility and forcing developers to make rushed decisions about supporting or abandoning legacy products.

This fall, we’re celebrating the 10th anniversary of the Wikipedia Education Program with a series of blog posts telling the story of the program in the United States and Canada.

Since its inception in 2010, the program now known as the Wikipedia Student Program has added 66 million words to 90,000 articles. In 2018 the program hit the milestone of having added as many words as the final print issue of Encyclopedia Britannica — a total of 44 million words. Two years later, we’re at 1.5 Britannicas. In a decade in which the pool of active contributors to Wikipedia declined sharply before leveling off, the efforts by student editors to contribute content to Wikipedia have become an increasingly important part of the project to write the world’s largest and most inclusive encyclopedia.

But the simple ability of students to add content to Wikipedia is neither positive nor negative. Bad content, after all, imposes a cost to the project by putting demands on volunteer time (Wikipedia’s most limited resource) to clean things up and remove bad work. So it’s not only important that students add content, it’s important that they add good content.

Across an extremely wide swath of topics, student editors have been able to make substantial improvements to the quality of content on Wikipedia. Sometimes, like with the Euglossa imperialis article shown in the image at right, the article created by the student tends to stay pretty much as is. Although this article was created in 2015, 91% of the article text remains the work of the student editor who created it; most subsequent edits have been WikiGnoming of one sort or another. In other cases though, student edits act as a catalyst for further development of the article.

At the start of 2016 the article about Kalief Browder — an African American boy who was imprisoned in Riker’s Island for three years without trial for the crime of allegedly stealing a backpack — was about 380 words long. In spring term that year, a student editor in Fabian Neuner’s Black Lives and Deaths class worked on Browder’s biography and expanded it to more than 5,500 words. The difference in this case is that the article has continued to attract edits since the class ended. In the remainder of 2016, after the class ended, the article received 135 edits, almost precisely the same number of edits (133) as it had received in the previous two years. Not only that, the article has continued to develop as other editors have tweaked and modified the article. At the end of the class, 82% of the article was the work of one student editor; today it’s only 53%. The article about the short life and tragic death of Kalief Browder hasn’t gotten much longer, but its overall quality has continued to improve.

In 2016, a student in George Waldbusser’s Biogeochemical Earth class created the Boring Billion article about the roughly 1 billion-year long period in Earth’s history in which not a whole lot seems to have happened. By the time the class was done working on that article, they had turned a redirect into a well-written, well-referenced, well-illustrated article. Three years later, another Wikipedian, User:Dunkleosteus77, improved the article and had it promoted to Good Article status. Although Dunkleosteus77 restructured and rewrote most of the article (to the point where they are now responsible for 66% of the article text), they had something that was fairly substantial to work from.

The increased editing could be attributed to a number of causes. Kalief Browder’s biography received may have attracted edits due to the increased awareness of his imprisonment and suicide after his release set against the slow rise in attention paid to the Black Lives Matter movement between 2016 and early 2020. It’s possible that User:Dunkleosteus77 would have created the Boring Billion article themselves if it hadn’t been created by students. But it’s also possible that these articles attracted efforts to improve them because they existed and because they already had a substantial amount of good content.

It’s very difficult to generalize from individual examples like this. It seems likely, but each individual story may just be an outlier, an unusual case. But a recent study by Kai Zhu and colleagues showed that articles improved by student editors in Wiki Education-supported classes attracted more page views and edits than did a paired set of similar articles that students hadn’t edited. They were able to show that in the six months following the expansion of articles by students in our program, other Wikipedians continued to edit these articles at higher rates than they did before the student editors expanded them. This may reflect the fact that student editors play an outsized role in the early development of articles related to academic topic areas.

It’s important to recognize that the process of content creation over the past ten years has been a partnership between Wiki Education and a growing group of dedicated instructors who bring their students back to Wikipedia year after year. For the instructors that do this, there’s a possibility for their students to make a major impact on the way an area of knowledge is portrayed on Wikipedia — or whether it’s present at all. In 2013, Erik Herzog brought the Wikipedia assignment to his chronobiology class for the first time, and they’re returned every other Spring. When three chronobiologists shared the Nobel Prize for Physiology or Medicine in 2017, the world rushed to Wikipedia to discover who these people were. And they read biographies that had been written by students in different iterations of the class. When instructors bring their students back to the same general area, year after year, as some have, a true corpus of knowledge is added to Wikipedia; beginning in 2012, Joan Strassmann’s behavioral ecology classes have added almost 1.5 million words to Wikipedia in articles about bees, wasps, and flies.

When student editors contribute in areas that are outside of the subjects that Wikipedia’s volunteer community tends to favor, they aren’t just adding content. They’re also encouraging other Wikipedians to also edit in these topic areas, expanding the coverage of underrepresented topics even more than their individual editing can. When you combine the cumulative impact of a decade of student contributions to Wikipedia with the contributions of other Wikipedians who built off their contributions, the cumulative impacts to open knowledge are immense.

We awoke this morning to the exciting news that the Nobel Prize for Chemistry was awarded today to Emmanuelle Charpentier and Jennifer Doudna for their work on CRISPR. It’s exciting to see this amazing discovery recognized, and it’s great to see the prize go to women, but I was especially pleased that a participant in one of our courses had made such a major impact on the Wikipedia biography of one of these Nobel laureates.

Two years ago, Dr. Laura Hoopes, as a participant in one of our Wiki Scholars classes, improved Dr. Doudna’s biography on Wikipedia. Before Dr. Hoopes started working on the article, Dr. Doudna’s work on CRISPR was discussed in a four-sentence paragraph which described it as “a new discovery that would reduce the time and work needed to edit genomic DNA”. Thanks to Dr. Hoopes’ additions, the article explains the revolutionary potential of CRISPR and the role played by Dr. Doudna and her team. Had she not added this content, readers who arrived at Dr. Doudna’s Wikipedia biography to understand why she was awarded this year’s Nobel Prize might have left none the wiser.

There’s more to this story too. As I wrote in a blog post in 2018:

If you read Dr. Doudna’s biography at the start of August, you would you have read about her career as Jack W. Szostak‘s student at Harvard Medical School, as Thomas Cech‘s postdoc at the University of Colorado Boulder, and the way her graduate student (and now, husband) helped her elucidate the three-dimensional folding structure of the Tetrahymena Group I ribozyme. And, below that, four sentences on her contribution to gene editing, including the all-important TED talk. If you read Dr. Doudna’s biography as it stood at the start of August, you would read the biography of a remarkable woman, told in relation to the major men in her life.

Far too often, the women’s achievements are told in relation to men. It’s almost like they are interpreted for “normal readers” through the context of the men in their lives. This is so commonplace, so normal, that it almost becomes invisible. By reframing these sentences just a little, to bring the focus to Dr. Doudna (who is, after all, the central character in her life’s story) Dr. Hoopes was able to eliminate some of the unconscious bias that pervades the way we write about, and see, women’s achievements.

Three years ago, when the Nobel Prize for Physiology or Medicine was announced, participants in Wiki Education’s Student Program were major contributors to the biographies of the three winners. Two years ago when Dr. Donna Strickland won the Nobel Prize for Physics, her absence from Wikipedia was a major embarrassment. Because most Wikipedia articles are written by volunteers who write about what interests them, what’s covered depends on who shows up to write.

If you can, please find a way to show up. To see our current course offerings, visit learn.wikiedu.org.

Image of Dr. Jennifer Doudna: Jussi Puikkonen/KNAW / CC BY

By Nuria Ruiz, Joseph Allemandou, Leila Zia, and MusikAnimal, Wikimedia Analytics and Research

Wikipedia, one of the largest websites in the world, gets a lot of traffic. At peak times, Wikipedia might serve as much as 10,000 pageviews per second, and as you would expect, most of these pageviews come from people reading articles to find information. Many other pageviews (about 25% of total) are from entities that we refer to as “Bots.” 

“Bot” is a very overloaded term nowadays. It is used to talk about fraudulent accounts on social networks, the crawler technology that Google uses to “read” and index websites, and in the wiki world, the “bot” term is used frequently to describe automated scripts with different levels of ability that “watch” and edit Wikipedia pages. The work of these scripts is needed to fight vandalism by reverting edits that are harmful in intent, like edits that might remove most of the content of a page.

These “helpful scripts” are what we refer to as “good bots” (the word “good” here just means that they actually do meaningful work). The bots we are going to talk about in this article are —you guessed it— not so helpful, and while they are not sophisticated, they can cause some damage.

For a very innocuous example of the “damage” one of these bots can cause, take a look at the top pageview list for English Wikipedia for March 2020 right when the COVID pandemic is expanding in Europe.

See anything strange on that list? There are more than 100 million pageviews for the “United States Senate” page, a glaring anomaly in a list that is mostly about COVID. So, what’s going on here? Well, someone is spamming Wikipedia requesting the “United States Senate” page over and over; that much is obvious. But, why? The answer is that we have no idea. 

As it is often the case with big data, we can tell what is happening but not why. One possibility is that it could be someone testing their botnet by pulling a Wikipedia page. Our overall request rate metrics are public, and it might be very tempting to try to move those for some bad actors. Botnets are cheap to run nowadays and an experiment like requesting the “United States Senate” page a hundred million times might only cost a very modest amount. 

Another possibility is that these are “keep-alive” requests: an application requesting some URL over and over to make sure it is online.

Other examples of pages that get quite a lot of traffic at times are the planet  Venus and the List of Australian stand-up comedians. The latter has hundreds of millions of pageviews. 

We call these types of actors “bot spammers.” They are actors that, while not totally legitimate, cause little harm. 

The second category of bot traffic that we worry about is not so harmless.

Here’s a  list of the top pageview pages from Hungarian Wikipedia in October 2019. While the list is in Hungarian it is probably easy for an English speaker to identify what stands out:

See it? There are quite a few sex and drug-related terms for starters. Not only that; the number of pageviews for those ‘questionable’ terms is very close, hovering around 200,000 for all of them. These are what we call “bot vandals.” The intent is pretty clear: vandalize the list of the top articles of a small Wikipedia. Vandals do try to do the same with larger Wikipedias like Spanish or English, but the number of requests required to get in the top 20 is a lot larger. 

We have been working this past year to better identify and tag the “bot spam” traffic so we can produce top pageview lists that (mostly) do not require manual curation. 

Up to April 2020, our classification for pageviews included only two types of actors: “users” and “spiders” (self-identified crawlers like Googlebot). Since all traffic not self-identified as “bot” was identified as “user,” quite a significant percentage of “bot spam” traffic was tagged as “user” traffic. This simple classification distorted our pageview numbers, as it made it seem that there was more user traffic than there really was. For example, we estimated that in 2019 between 5 to 8% of desktop pageviews in English Wikipedia tagged as “user” traffic was actually coming from “bot spammers/vandals.” That number could be as high as 800 pageviews a second. We continue processing and cataloging these requests, and from April 2020 onwards the traffic we identify as coming from spammers is tagged with the label “automated.” You can see here that it represents about 5% of total traffic. 

It is interesting to notice that the effect of “bot spam” in the traffic data is not distributed equally across sites, wikis like Japanese Wikipedia seem hardly affected by spammers.

The tagging of automatic traffic is far from perfect. Some automated traffic is still being labeled as “user” pageviews and a very small percentage of user traffic might be labeled as “automated.” 

This happens because classifying traffic as automated is a less trivial problem to solve than it might seem at first instance—for two reasons: data volume and semantics. The volume of data you need to comb through to “weed out” the bad requests are huge; it could be in the terabytes range for just 1 day. Also, from our example above, it is by no means obvious to a machine that the “United States Senate” page should not be in a list that mostly contains pandemic related pages as in the regular (non-pandemic) times, the Top 20 list for Wikipedia contains pages about many different topics, so outliers might not be so apparent. We decided that for our first take at the problem we would focus on computing “content agnostic features,” like ratios of requests and number of pages requested per actor per “session” to see if, with this simpler set of dimensions, we could detect the bulk of the bot spam traffic.

The community has been hard at work finding the characteristics of spammy traffic by hand for a while. For example,  a key factor in distinguishing “automated” versus “real user traffic” was assessing whether pages had similar pageview numbers in the desktop and mobile versions. If a page had millions of “only desktop” pageviews it was not included in any of the top lists that were curated by hand. After a few tries, we concluded that dimensions that did not consider the nature of the page and just focused on request ratios (a.k.a “content agnostic”) were sufficient for the first iteration of the system we rolled it out in April this year. We did a careful study of how the bot spam traffic affects the most popular wikis and if you are interested, there are many plots and details in our public technical docs. For questions and comments, you can drop us a line to analytics@lists.wikimedia.org.

About this post

Featured image credit: Unknown Chinese Maker Tin Wind Up Radar Robot Front, D J Shin, CC BY-SA 3.0

This fall, we’re celebrating the 10th anniversary of the Wikipedia Education Program with a series of blog posts telling the story of the program in the United States and Canada.

In 2010, I joined a team of free knowledge enthusiasts to launch a program that is today known as Wiki Education’s Student Program. Our goal was simple: enable higher education students to add content to the English Wikipedia as a class assignment. For students, it’s an opportunity to gain key 21st century skills and have their academic work be meaningful outside the classroom; for Wikipedia, it was an opportunity to gain more content — and, if we could scale the program up, help solve Wikipedia’s new editor crisis.

Ten years later, I’m proud that we’re well on our way to success: Our students have added more content to the English Wikipedia than the last print edition of the Encyclopædia Britannica had — and they represent 19% of all new active editors joining the English Wikipedia.

Both are impressive markers of success. But I am particularly proud of the work my colleagues and I have done to scale Wiki Education’s Student Program to the point where we bring 20% of all the new active editors to English Wikipedia because of what it means for the future of Wikipedia — and the equity of open knowledge.

Online communities require new members

Our program started in 2010, as a response to the Wikimedia movement’s then-new strategic plan. In particular, we wanted to both increase participation and improve quality. Participation was particularly important because the number of active editors of English Wikipedia had skyrocketed in 2005–06, but had started decreasing in 2007 — and the retention of new editors was simultaneously tanking. This retention vs active editors chart’s trend lines boded poorly for Wikipedia’s future.

Bringing new editors in is critical to the future of the encyclopedia because of the natural churn in online communities. Over time, the people who serve as Wikipedia’s community of editors have typical life changes. Jobs change, family situations change, and life priorities change — and in that, it’s natural for some people to have less time for volunteer roles or hobbies like editing Wikipedia. Online communities need new participants joining at the same rate as people are naturally leaving the community in order to continue thriving.

A thriving community is critical for the existence of Wikipedia. Wikipedia is the force it is today because it’s the place people go for access to information. An average of 859 million devices access the English Wikipedia each month. But if content weren’t up to date, if people couldn’t find what they were looking for when they looked something up on Wikipedia, Wikipedia would cease to be a top-10 website in terms of page views. The Wikipedia we know and rely on today wouldn’t exist without the corps of volunteer content contributors who make up the active editor base.

Knowledge equity

Another area in which new editors are important is content diversity, or as we in the Wikimedia movement call it, “knowledge equity”. English Wikipedia’s editing community is 86% male, and this gender gap results in content gaps as well, where we see extremely detailed coverage of military history and video gaming topics and gaps in coverage of the arts and many other academic subject areas. Bringing in new contributors from backgrounds that are more diverse than the existing editing population is critical to adding content perspectives missing from Wikipedia.

In our movement, “knowledge equity” also encompasses challenges of coverage in other language Wikipedias. While I firmly believe it’s important to improve the content available in the 300+ language Wikipedias, it’s also critically important to improve English Wikipedia’s equity gaps as well. Over the last year, English Wikipedia got 126 billion page views — and the combined total page views of all language Wikimedia projects is 285 billion, meaning English Wikipedia gets 44% of all the page views that go to Wikimedia projects.

I believe in Wikimedia’s vision to collect the sum of all knowledge for all humans in their own language. But we are far from achieving this vision. It’s critical to invest in both growing Wikipedia coverage in other languages and also ensuring that the Wikipedia read by 44% of our user base addresses its own systemic bias issues. Addressing content gaps on English Wikipedia is even more important when you consider a significant portion of the content development in emerging communities is coming from translation from English and other major language Wikipedias. Translating English Wikipedia’s biases into emerging language Wikipedias just exacerbates our content equity problems.

Put simply, English Wikipedia needs new active editors — and those from diverse communities — to achieve knowledge equity in the Wikimedia movement.

Wiki Education’s role

When we started the program in 2010, the collaboration between Wikipedia and academia was already happening. Faculty who were Wikipedians were already assigning students to edit — and many of the existing Wikipedia editors were students themselves. We weren’t crazy to think we could engage students to edit Wikipedia at scale — we just needed the time to build a program that worked.

Two years ago, the Wikimedia Foundation’s Audiences team released a slide deck that included the highlight: “In January, global new editor retention hit its highest level since 2007”, with the sub-bullet: “Apparent cause is the trend toward higher and higher January and September spikes on the English Wikipedia”. This point immediately struck me: As our Student Program is tied to the academic calendar, we bring more and more new editors at the beginning of each term in the U.S. and Canadian calendars, so every January and September. We reached out to the Audiences team, provided additional information on our program participants, and in early 2019, their next slide deck included the data that 19% of all English Wikipedia’s new active editors come from Wiki Education. One year later, we are holding steady: according to a Wikimedia Foundation analysis, Wiki Education remains responsible for 19% of all English Wikipedia’s new active editors.

What does it mean that we’re bringing in 19% of all of English Wikipedia’s new active editors? Consider that 59% of the 16,000 students we bring to the English Wikipedia every year identify as women (plus an additional 2% who identify as non-binary or other), and 42% identify as non-white, meaning we’re significantly increasing diversity in English Wikipedia’s editor population. We’re also specifically reaching out to higher education faculty in underrepresented topic areas, such as race, gender, and sexuality. We’re improving Wikipedia’s knowledge equity — and at such a scale that our program can be seen in the global English Wikipedia contributor numbers. Our program participants represent 3% of all active editors on the English Wikipedia.

This is an incredible accomplishment. I couldn’t be prouder of the hard work of all the people who have worked on the program over the last decade — the 1,800+ faculty, the 80,000 students, the hundreds of Wikipedians who have helped support our students in their volunteer roles, the funders who have believed in our work enough to support us financially, and the small but mighty staff of our program who’ve build structures and processes to grow the program to this milestone. Thank you for all your efforts!

Between 1560 and 1689, there were some 1,100 parishes in Scotland, each one served by a minister who, ideally, preached sermons once or twice a week, offered communion at least once a year, visited the sick and dying, oversaw the behaviour and morality of his neighbours, alongside a host of other responsibilities. As the Protestant Reformation in Scotland progressed, these men (and their families) became increasingly central to how religious change was experienced at a local level. Despite this prominence, we know remarkably little about them.

The project Mapping the Scottish Reformation is an attempt to track all of Scotland’s ministers between the Reformation Parliament in 1560 and the Glorious Revolution in 1689. And by ‘track’, I mean that the project seeks to trace their movements: the big milestones in clerical careers, where and when they lived, and where they moved, where they were educated, who they married, and what happened to their children. The end result will be a weather map of sorts, that will allow us to observe ministers’ movements and the process of religious change.

How do we find out about these people? We rely on surviving documents that were made at the time, usually by Church courts known as presbyteries. These documents are all housed in the National Records of Scotland in Edinburgh, must be searched by most users on site, and must be transcribed by each visitor. There are no real indexes to help. Here’s an example of a two-page spread, typical of the manuscripts we use:

To date, as part of our first case study, we have parsed around 3,500 pages of manuscript material like this, extracting twenty-two different data points: including ministers’ names, the parishes they served, the dates when they were appointed, the dates they transferred to other parishes, where they were educated, the locations to which they moved, when they were disciplined, when they died, etc.

Most of the individuals we have found are obviously relatively obscure and have no item entry in Wikidata. Some were captured as part of the recent project the Survey of Scottish Witchcraft, but most remained unknown to Wikidata. As such, we have created around one thousand new items over the last two months, all of which serve as a framework to record the key parameters of a cleric’s career.

To ensure our data entry was as quick and reliable as possible, it was essential that we structured each new cleric on Wikidata by using existing properties. We followed the structure for most ‘human’ (Q5) entries to Wikidata, but then added ‘occupation’ and ‘residence’ properties. We added ‘start time’ and ‘end time’ qualifiers to ‘residence’ so we could track precisely when a minister arrived and left his parish.

To capture the complexity of clerical careers in early modern Scotland, we made use of the ‘Significant Event’ property (P793). We had to get a little creative here, because some of the specialist terminology relating to the Church did not exist in Wikidata’s reams of items. As such, we created the following items that can be classified as ‘significant events’:

• deposed (Q89273000)
• suspension (Q89273310)

In some rather more fruity cases, we made use of existing items such as ‘illness’, ‘repentance’, and even ‘execution’, as other ways to capture significant events. Coupled with the ‘point in time’ qualifier, we were able to record this information in quite a detailed way.

One of the essential parts of this project is that we need to meticulously record where we obtained our information. As such, we had to devise a method to record manuscript information in Wikidata’s referencing system. Again, after lengthy consideration, we made use of existing properties already on Wikidata to capture the information we required. Incredibly, when used together, the properties ‘collection’, ‘inventory number’, and ‘folio(s)’ allowed us to recreate an academically robust system to reference manuscripts in their complexity. Here is an example:

In summary, Wikidata’s existing menu of properties and items provides significant amounts of flexibility in designing schema that can capture the messy and complex world of the early modern cleric. Indeed, the scope of Wikidata’s flexibility could accommodate a range of projects that gather data from manuscript sources. During the several months we have been working with Wikidata, we have successfully ported over thousands of data points from manuscripts kept under lock and key at the National Records of Scotland, traced the careers of over 900 clerics and started, tentatively to manipulate this data through the Wikidata Query Service. Keep watching our website mappingthescottishreformation.org and our Twitter account, @MappingScotsRef to see how our project, and our integration with Wikidata, develops.

Chris R. Langley is Senior Lecturer in Early Modern History at Newman University, Birmingham. Mapping the Scottish Reformation is funded by the National Endowment for the Humanities and the Strathmartine Trust.

You can now use media files from your wiki inside of Wikibase.

We created a new data type for Wikibase that enables working with media files defined on your wiki. Previously Wikibase only supported working with media files from Wikimedia Commons. The new data type, called "Media file" works exactly like "Commons media file", but then for your local media files. The "Media file" data type is available via our new extension Wikibase Local Media.

Wikibase Local Media was developed for, and funded by, Rhizome. It is available as free open source software and can be installed on any Wikibase wiki that runs Wikibase version 1.34 or 1.35. A huge thanks to Rhizome for funding this key functionality.

How’d we do in our strive for operational excellence last month? Read on to find out!

📊 Month in numbers

• 6 documented incidents. [1]
• 41 new Wikimedia-prod-error tasks created. [2]
• 36 Wikimedia-prod-error tasks closed. [3]

The number of incidents in May of this year was comparable to previous years (6 in May 2019, 2 in May 2018, 5 in May 2017), and previous months (6 in May, 8 in April, 8 in March) – comparisons at CodePen.

To read more about these incidents, their investigations, and pending actionables; check wikitech.wikimedia.org/wiki/Incident_documentation#2019.

As of writing, there are 201 open Wikimedia-prod-error tasks (up from 186 last month). [4]

📉 Current problems

Take a look at the workboard and look for tasks that might need your help. The workboard lists known issues, grouped by the month in which they were first observed.

→ https://phabricator.wikimedia.org/tag/wikimedia-production-error

Or help someone that’s already started with their patch:
→ Open prod-error tasks with a Patch-For-Review

Breakdown of recent months (past two weeks not included):

• November: 2 issues left (unchanged).
• December: 1 issue got fixed. 3 issues left (down from 4).
• January: 1 issue left (unchanged).
• February: 2 issues left (unchanged).
• March: 1 issue got fixed. 4 issues remaining (down from 5).
• April: 2 issues got fixed. 12 issues remain unresolved (down from 14).
• May: 10 new issues found last month survived the month of May, and remain unresolved.

By steward and software component, unresolved issues from April and May:

• Wikidata / Lexeme (API query fatal): T223995
• Wikidata / WikibaseRepo (API Fatal hasSlot): T225104
• Wikidata / WikibaseRepo (Diff link fatal): T224270
• Wikidata / WikibaseRepo (Edit undo fatal): T224030
• Growth / Echo (Notification storage): T217079
• Growth / Flow (Topic link fatal): T224098
• Growth / Page deletion (File pages): T222691
• Multimedia or CPT / API (Image info fatal): T221812
• CPT / PHP7 refactoring (File descriptions): T223728
• CPT / Title refactor (Block log fatal): T224811
• CPT / Title refactor (Pageview fatals): T224814
• (Unstewarded) Page renaming: T223175, T205675

💡Ideas: To suggest an investigation to write about in a future edition, contact me by e-mail, or private message on IRC.

🎉 Thanks!

Thank you to everyone who has helped by reporting, investigating, or resolving problems in Wikimedia production.

Until next time,

– Timo Tijhof

🎙“It’s not too shabby is it?”

Footnotes:

[1] Incidents. –
wikitech.wikimedia.org/wiki/Special:PrefixIndex…

[2] Tasks created. –
phabricator.wikimedia.org/maniphest/query…

[3] Tasks closed. –
phabricator.wikimedia.org/maniphest/query…

[4] Open tasks. –
phabricator.wikimedia.org/maniphest/query…

22/09/2020-28/09/2020

About us

• The number of local chapters recognised by the OSM Foundation is increasing. For this reason from issue #533, weeklyOSM will have a new category ‘Local Chapter News’. Here every local chapter will have the opportunity to post articles using our guest mode. The local chapter can also decide in which language(s) the article should be published.

Mapping

• OSM contributor bkil proposed a scheme for tagging auditory signals, which are often used at pedestrian lights.
• User Vollis proposed the tag amenity=chapel_of_rest for a room or building where families and friends can come and view someone who has died before their funeral.

Community

• Mateusz Konieczny recommended the following links as good instructions for 100% beginners in JOSM: https://wiki.openstreetmap.org/wiki/JOSM,
  https://wiki.openstreetmap.org/wiki/JOSM/Guide and https://learnosm.org/en/josm/. Mikel Maron also suggested the instructions on the Mapbox website.
• Lukasz Kruk laments the lack of mapping quality in some regions and explains his view on causes and possible remedies.
• The idea to create a French newsletter (we reported) is discussed > on loomio.
• Map maintenance with StreetComplete is the first of the 12 projects funded by the first OSMF microgrant to submit a final report.
• OpenStreetMap US published the 11th issue (September 2020) of its newsletter.
• A (draft) wiki page concerning limitations on mapping private information has been set up. It states where mapping efforts should stop, for example not adding the names of inhabitants of dwellings or limiting the mapping of personal possessions (like TV sets, numbers of livestock or washing machines) to only those for communal use.

Imports

• Pieter (user: pcmill) would like to start a project to import around 160,000 trees in the city of Utrecht, The Netherlands, and asks for comments.
• The wiki page ‘Friesland Hydranten Import’ describes > a procedure to import an official and complete list of fire_hydrants=* (only two were mapped before in OSM); user osmcux asked > for support to avoid duplication in a more complex scenario.

OpenStreetMap Foundation

• The OSMF Board approved two new local chapters during their September monthly meeting: USA and Kosovo.
• The Board is considering forming a personnel committee to have a structured way to support people working with the OSMF nearly full time. Mikel Maron is now asking for input to the discussion draft.

Humanitarian OSM

• The Forecast-based Financing Programme blogged on the work of the HOT-OSM community and ‘Missing Maps’ of the American Red Cross in the interest of cooperation through worldwide mapping to combat the COVID-19 epidemic.
• HOT reported on the progress of mapping in Liberia in connection with financial support through a microgrant from the COVID-19 Rapid Response programme.
• Can Unen reports that BİMTAŞ A.Ş. and HOT partner Yer Çizenler are working together to put the Istanbul Planning Agency campus on OpenStreetMap.

Maps

• Michael Schultz reports on a contiguous high resolution OSM landuse map of the European Union made by combining Copernicus data and OpenStreetMap.
• Robert Goedl, Head of Linux Bibel Österreich > , in the past has presented > Marble as a Linux alternative to Google Earth; now he describes > how to use Marble for navigation, based on OpenStreetMap data.

switch2OSM

• [1] The Ministry of Ecology and Renewable Natural Resources of the Province of Misiones presented the ‘Interactive Map for Environmental Emergency Management‘ , based on OpenStreetMap.

Licences

• A company using OSM data passed on complaints > about outdated OSM data in Mexico and asked for help.
• Adam Franco found his contributions to OSM on maps being used by a college in the USA and asked if he should follow the path described in the wiki for missing attribution.

Software

• MRVDH launched BuDongSang, a real estate idle game based on OpenStreetMap data. It is still in early development on GitHub; feel free to leave feedback and suggestions.
• Jochen Topf reported the launch of the new osm2pgsql.org website, the first visible result of the support by the OSMF for osm2pgsql development.
• Sven Geggus, maintainer of the German Style OSM Carto fork, explained why the development of the fork of the OSM Carto Style is currently stuck.

Releases

• HeiGIT presents version 6.3.0 of openrouteservice, with an improved technique to speed up the calculation of isochrones (a map area that can be reached from a starting point within a given time or distance limit). By taking advantage of the topological properties of a road network, network cells connected to other cells by very few roads can be identified; with this technique, called partitioning, it is possible to drastically reduce the computational time required to find all roads that can be reached within a given time limit.

Did you know …

• … that the JOSM plugin Gridify will generate a grid of ways (as blocks or lines) from any four nodes? It may be useful for creating repeating features, such as individual parking spaces.
• … Mapping with OpenStreetMap? This website, hosted by MapBox, provides multiple resources to help learn how to map, from novice level to power user.
• … that OpenStreetMap India has a YouTube channel?
• … the ‘Europe map with countries and their capitals‘? The English menu navigation has some unexpected Cyrillic submenu entries. There is still some room for improvement.

Other “geo” things

• Matthias Schwindt reviewed > the new Garmin ‘TOPO Switzerland V2 PRO’ map, which brings together government and commercial data sources. His conclusion was: ‘It would only be worth the price to me if I were touring the Swiss mountains very often. Otherwise, the free OSM maps are also very suitable for mountain biking’.
• SuperMap, a China based GIS platform developer, displayed at the GIS Technology at 2020 GIS Software Technology Conference.
• Heise reported > about projects that rethink unethical ‘Smart City’ concepts for the common good. Among the linked projects are works against surveillance, projects that benefit citizens, and a conference held with an interesting schedule.
• Garmin launched their first GPS devices with multi-band GNSS receivers. Previously, accuracy has been improved by better antennas and multiple GNSS satellite systems (GPS, GLONASS, Galileo). Now Garmin has added a new GNSS receiver chip that can evaluate both frequency bands.
• On 23 September 2020, Norway’s Ministry of Climate and Environment entered into a contract with Kongsberg Satellite Services (KSAT) and its partners, to provide universal access to high-resolution satellite monitoring of the tropics in order to support efforts to stop the destruction of the world’s rainforests.
• The mathematics learning website Mathigon uses different map projections to sum up their lesson about spheres, cones, and cylinders. The interactive map projection animation is worth a try – not only for kids.
• Kate Behncken, Vice President and Lead of Microsoft Philanthropies, reported on their planned expansion of programmes to support underserved refugee camps. Over the past two years, the company has worked with non-profit and humanitarian organisations to help address global humanitarian issues, such as disaster relief by improving open mapping of vulnerable areas with Bing and the Humanitarian OpenStreetMap Team (HOT).
• Beginning this week, Google Maps will show, after an update, how prevalent COVID-19 is in a geographic area (we reported earlier). For an area, seven-day averages of daily new COVID-19 cases per 100,000 people will be shown using colours together with shadings, and labels will show whether cases are increasing or decreasing.
• Understanding the risks associated with COVID-19 outbreaks in British universities requires navigating many datasets, and a deep understanding of the arcane administrative geographies of the United Kingdom and their inter-relationships. David Kernohan provides a useful guide in a post on the WonkHE blog.
• Michael Prodger shared his insights about maps and landscape painting in the 16th century.

Upcoming Events

Where	What	When	Country
Taipei	OSM x Wikidata #21	2020-10-05
London	Missing Maps London Mapathon	2020-10-06
Montrouge	Rencontre des contributeurs de Montrouge et alentours	2020-10-07
Stuttgart	Stuttgarter Stammtisch	2020-10-07
Berlin	148. Berlin-Brandenburg Stammtisch (Online)	2020-10-09
Michigan	Michigan Online Meetup	2020-10-12
Cologne Bonn Airport	132. Bonner OSM-Stammtisch (Online)	2020-10-13
Munich	Münchner Stammtisch	2020-10-13
Salt Lake City / Virtual	OpenStreetMap Utah Map Night	2020-10-13
Cobb Virtual Academy	OSMUK AGM plus presentations	2020-10-17
Lüneburg	Lüneburger Mappertreffen	2020-10-20
Berlin	OSM-Verkehrswende #16 (Online)	2020-10-20
Nottingham	Nottingham pub meetup	2020-10-20
Online	2020 Pista ng Mapa	2020-11-13-2020-11-27
Online	FOSS4G SotM Oceania 2020	2020-11-20

Note: If you like to see your event here, please put it into the calendar. Only data which is there, will appear in weeklyOSM. Please check your event in our public calendar preview and correct it, where appropriate.

This weeklyOSM was produced by Elizabete, Climate_Ben, Lejun, MatthiasMatthias, MichaelFS, Nordpfeil, PierZen, Polyglot, Rogehm, TheSwavu, richter_fn.

By Marcel Forns, Joseph Allemandou, Dan Andreescu and Nuria Ruiz, Wikimedia Analytics Team

Wikimedia projects are an integral part of how people build Free and Open knowledge on the Internet.  In May 2020, for example, hundreds of thousands of active editors created 4.5 million new articles.  They added and changed 60 gigabytes of content in 22 million edits.  High-level numbers are fun to explore, but they don’t say a lot about the day to day experience of editing.  What is it like to collaborate on these projects?  What kind of work does an average editor do?

We can use the Mediawiki History Dataset to explore this together!  

Let’s begin with some questions.  Personally, I’ve been a little scared to jump into the fray of editing Wikipedia.  Maybe it would help if I knew more about the culture?

Q: How often does a typical editor edit?

Q: How much work goes into each edit?

As we look for answers, we run into some intricacies of editing:

• Automated bots make a lot of edits.  People program them to do amazing things, but right now we want to focus on humans.
• Undoing an edit can take only a second, but can change a lot of content.  Counting that as “work” might skew our answer.
• Pages surrounding the articles: there is a lot of editing on talk pages, people’s own user pages, and so on.  We want to focus just on the articles, but that can be tricky as pages move around.

Clearing these hurdles was quite hard before we built the Mediawiki History dataset.  The operational database behind Mediawiki does a great job of serving the UI but does not organize data to answer questions like the ones we’re asking here.  These are analytics-style questions that depend a lot on the time dimension.  To illustrate this, let’s visualize the timeline of a page, alongside the timeline of an editor:

This page started out in the Draft namespace and moved to the Main namespace.  The user turned on their “Bot” flag to indicate that an automatic script was making edits on their behalf, then turned it off.

If we look at the database, it tells us what’s true right now, at the end of those timelines.  According to this latest state, the user is not a bot and the page is in the Main namespace.  If we count “edits by users that are not bots in the Main namespace,” we would count all three of the edits above.  What we need is a way to remember the context at each point in history.

Enter the logging table.  We used historical information about page moves, user renames, user group changes, and more, to historify each edit.  We enrich edits with context about the page being edited and the user doing the editing.  We also add fields that would be computationally expensive to get out of the database, like whether an edit was reverted in the future and how many seconds passed until that revert.  Fields that end with _historical describe the state at the time of the edit.  For example, whether the user was a bot when they made the edit.  You can find details on each field in the schema documentation.  The logic that creates this data from the Mediawiki databases is written in Scala and leverages SPARK to distribute the computation.

So let’s answer the question “How often typical users edit?”  

We’ll assume you load up this data in an SQL database with columns as described in the schema.  The field event_user_seconds_since_previous_revision might be a mouthful, but it measures exactly what we need: how often, in seconds, a particular user edits.  

Let’s take the median (50th percentile) seconds between edits, per user:

select event_user_text,
        percentile(event_user_seconds_since_previous_revision, 0.50) as median

   from mediawiki_history

  where event_entity = ‘revision’
    and event_type = ‘create’
  group by event_user_text

Next, we filter out bots.  If the editor was in the “bot” group, we add “group” to the event_user_is_bot_by_historical array.  If they had a name that contained “Bot” at the time, we add “name.”  By the way, we check the name because the group wasn’t used consistently until around 2012.  To exclude bots, we just check that the array is empty:

size(event_user_is_bot_by_historical) = 0

A few fields tell us whether this edit is reverted later or is a revert itself.  Let’s say we don’t count reverts, as editing and filter them out:

not revision_is_identity_revert

And finally, we make sure the page was considered “content,” as opposed to a supporting page, at the time of the edit:

page_namespace_is_content_historical

Once we put all that together, we have the median time between edits for each user.  As a rough answer to our question, we could average this out, which gets us:

with median_per_user as (
 select event_user_text,
        percentile(event_user_seconds_since_previous_revision, 0.50) as median

   from wmf.mediawiki_history

  where snapshot = '2020-06'
    and event_entity = 'revision'
    and event_type = 'create'
    and size(event_user_is_bot_by_historical) = 0
    and not revision_is_identity_revert
    and page_namespace_is_content_historical

  group by event_user_text
)

 select avg(median)
   from median_per_user

>>> 2186838.3628780665

2186838.3628780665 seconds is about 25 days.

To figure out the amount of content changed, we just select revision_text_bytes_diff, which is the number of bytes added or removed from the article by this edit.  Looking at the absolute value of this, because it’s negative when removing content, the average median is 468 bytes, or about half of this paragraph.

While running that, we can also average the count(*), to find the average user has made 18 edits since they signed up.  So, while some editors make the news with tens of thousands of articles created and the amazing work that they do, it seems that maybe a lot of the work is small incremental progress done by a lot of people.  Digging into the data more will reveal distributions look like power laws, so maybe average medians are not the best representation.

An important caveat is that the logging table does not have perfect information.  There are missing records, missing page and user ids, and other inconsistencies.  For example, we often found pages that didn’t exist but were being deleted in a log entry.  We do our best to reconcile these problems, and we will continue to improve the quality by going back and looking at the Wikitext content.  But we have vetted that the data is accurate to 99.9% of the metrics we want it to compute for now.  That ~0.1% can look big if it happens to be the subject of your analysis, so if you find anything like that, do let us know.

Hopefully, this exploration gives a feel for the Mediawiki History dataset, an idea of the kinds of questions it’s designed to answer.  The most important thing is that this dataset is for everyone.  If a new field would be useful, we are on Phabricator, let us know!  We update and publish this data monthly, but are working on faster incremental updates this quarter.  The Analytics mailing list is the place to get news and updates.

This dataset is the heart of Wikistats, our community-focused stats site, and is already being used by researchers to build better ways to examine and improve our movement.  How will you use it?

About this post

Feature image credit: Emna Mizouni ‘I edit Wikipedia’ heart t-shirt, VGrigas (WMF), CC BY-SA 3.0

This fall, we’re celebrating the 10th anniversary of the Wikipedia Education Program. Started as an experimental pilot project by the Wikimedia Foundation in 2010, it quickly grew into a global program run by different organizations across more than 80 countries. Over the past decade, Wiki Education – an independent spin-off of the Wikimedia Foundation – has been supporting instructors at more than 500 universities in the United States and Canada. Today’s post by our Executive Director Frank Schulenburg, who also founded the Education Program back in 2010, will be the first in a series of blog posts that tell the story of the program in the United States and in Canada, while celebrating some of its achievements.

The idea

When the Wikimedia Foundation hired me as “Head of Public Outreach” in July 2008, Wikipedia had already gained some steam. During the first seven years of its existence, the online encyclopedia “that everybody can edit” had attracted an extremely dedicated group of volunteers who spent an incredible amount of their free time writing articles, uploading images, and curating content. As a result, Wikipedia’s readership had multiplied year over year and the site was clearly on its way to become a mainstream staple of the internet. Given that an increasing number of people around the globe relied on Wikipedia’s content to be accurate and trustworthy, attracting new contributors became a goal of paramount importance. But how can you “broaden participation” and “encourage qualified authors to contribute”, as my job description then mandated? Would you run an ad campaign asking people to become “encyclopedists”? How could you even incentivize participation if a reputation gain – as one of the enticements knowledgeable people are traditionally after – wasn’t attainable due to the fact that Wikipedia wouldn’t publicly list the authors’ names the same way as other encyclopedias did?

The answer to these questions became more obvious when I noticed an emerging trend in higher education: an increasing number of university instructors had started using Wikipedia as a teaching tool. Instead of writing a traditional term paper, students wrote Wikipedia articles. That way, they engaged in a “real-world assignment”, gained critical thinking skills, learned how to collaborate online, and how to convey facts to a non-expert audience. From an instructors perspective, students were far more motivated and had a much deeper learning experience. All seemed to line up.

The first years

So, in early 2010, we decided to boost the existing trend of “Using Wikipedia as a teaching tool” driven by the grassroots efforts of highly innovative instructors in U.S. higher education. In order to learn from these pioneers’ experiences, we invited some of them to Wikimedia’s office in downtown San Francisco and spent two days on listening to what they had to say. Then, after some preliminary research – including a tour to select colleges in the United States – we hired a small team and tasked them with a feasibility study that lasted 18 months. Our main goal was twofold: in order to support a larger number of instructors, we first needed to build a broad range of support mechanisms so that “Teaching with Wikipedia” would be easy to replicate. Secondly, we had to create awareness of the program and its benefits, so that instructors at a wide range of institutions of higher education would be eager to join the program.

Over the course of 18 months, a small project team of five worked on what became to be one of the most successful outreach projects ever undertaken by the Wikimedia Foundation. The project team supported professors at 24 universities who taught 47 classes. More than 800 students contributed the equivalent of 5,800 printed pages of high-quality content to Wikipedia. The students had a much deeper learning experience and they gained critical 21st-century skills. Most importantly, the pilot project successfully demonstrated that the underlying model worked. That’s when we decided to move some of the initial members of the team to permanent positions in order to scale our efforts up.

Between 2011 and 2014, the program grew moderately in numbers of participants. It was clear there was interest among academia to scale the program up beyond the 75-ish courses that were teaching with Wikipedia each term. But the Wikimedia Foundation’s strategy suggested it wasn’t the right home for a U.S.-based program, as they were focusing on global efforts. Wikimedia supported a year-long volunteer effort to create a spinoff organization: Wiki Education, the missing link between the world of academia and Wikipedia.

Scaling up

At Wiki Education, we focused our attention on scaling the program. We made several significant changes in the 2014–16 time frame that have brought us to where we are today. First, we created an online course management platform. Next, we started establishing formal partnerships with academic associations. These two elements were absolutely critical for our ability to scale.

The partnerships with academic associations resulted in significant growth in the number of people teaching with Wikipedia, as these institutions helped recruit new instructors through their vast networks of members. At the same time, this new way of building partnerships allowed us to target Wikipedia’s content in specific fields.

In order to provide a rapidly growing number of instructors and students with appropriate support, we shifted from in-person help (through email, phone, and Wikipedia Ambassadors) to our online course management platform. The new system has since been helping instructors with planning their assignments as well as with monitoring and assessing their students’ work. For both instructors and students, the system offers a variety of different online trainings. On our end, the system enables us to have a clear understanding of how our program is performing and where we need to intervene.

The learnings we made over the course of the first two years of Wiki Education enabled us to start the “Wikipedia Year of Science” in 2016, a large-scale campaign involving U.S. and Canadian education institutions in increasing Wikipedia science content quality while improving students’ science communication skills. As a result, we provided more than 300 million Wikipedia readers around the globe with free access to high-quality science information in 2016 alone.

In the Fall of 2016, we partnered with Zach McDowell who surveyed 1,627 students and 97 instructors about the benefits of a Wikipedia writing assignment. The resulting study found that Wikipedia-based assignments enhance students’ digital literacy and critical research skills, foster their ability to write for a public audience, promote collaboration, and motivate them more than traditional assignments. The results of this study enabled us to make an even better point about the benefits of our programmatic work when talking to potential partners and to individual instructors.

In April 2018 we reached a significant milestone: after eight years of Teaching with Wikipedia, students enrolled in our program had added more than 44 million words to Wikipedia, eclipsing the amount of information available in the last print edition of Encyclopædia Britannica.

10 years later

As we enter the fall 2020 term, it’s been ten years since our program started, and I couldn’t be prouder of where we are today. We have successfully improved the efficiency of our work to the point where we can run a high-impact program with a relatively small team. Each term anew, participants in our Student Program improve Wikipedia’s content to the benefit of millions of readers around the world. Generations of students have gained critical skills while engaging in real-world assignments. Higher education has embraced the concept of “Teaching with Wikipedia” at a scale that we could not have imagined ten years ago.

The impact we’ve had along the way has been astounding. Keep an eye on our blog over the next few weeks as we feature different ways of looking at the changes the Student Program has brought.

We would be remiss without acknowledging the hard work from former colleagues involved in various aspects of this program over the years, including: At Wikimedia, Rod Dunican, Pete Forsyth, Mishelle Gonzales, Annie Lin, and Ashlen Roth; at Wiki Education, TJ Bliss, Paul Carroll, Sara Crouse, Rob Fernandez, Tanya Garcia, Bill Gong, Ozge Gundogdu, Victoria Hinshaw, Adam Hyland, Renee LeVesque, Zach McDowell, Ryan McGrady, Tom Porter, Wes Reid, Eryk Salvaggio, Kevin Shiroo, Shalor Toncray, Cassidy Villeneuve, Samantha Weald, and Elysia Webb. And, of course, all the volunteers, instructors, students, librarians, and others who have supported the program!

Application review

Sage and I are always discussing ways to make the reviewing process better and this round they introduced a small tool to help us communicate which themes we’ve identified in essays. A few of them also are also useful to warn a reviewer of sensitive content (for instance, mentions of violence). Note that having more themes in an essay doesn’t necessarily mean it is a more compelling essay — this tool was implemented to support decision-making, but by no means it replaces the more complex process of understanding someone’s situation.

Communication and planning

Synchronous and asynchronous communication have been key to the success of this round — we’ve been using Etherpad to keep track of the progress of communications with all communities and to discuss ideas to improve the application review process, GitLab for more permanent internal documentation (guidelines, templates), Zulip for occasional questions and updates and meetings to get on the same page when needed.

Promotion

I participated in a livestream with one organizer and of participant of Big Open Source Sister in a show they call Talk like a BOSS. They interviewed me about Outreachy both from a former intern and current organizer perspective, technical writing and strategies to create a better environment for global collaboration in open source projects. I’m always excited to accept opportunities that allow me to promote Outreachy in Brazilian Portuguese, but this one was special because it’s an initiative created by a former intern of ours, Clarissa Borges.

I also participated in a private, informal chat with members of Projeto ADAs, a group of women in technology created by students and professors of my university. We mainly focused on my experience as an intern, talking about expectations around language proficiency and working remotely.

Pinging communities

I’ve been building a closer relationship with communities this round. I was responsible for a good amount of emails related to the participation in the December round, in special reminders of important dates and deadlines. I was also in an onboarding meeting with Mixxx DJ Software to understand better what usually brings communities to Outreachy, their expectations, struggles and success stories.

By Srishti Sethi, Senior Developer Advocate

Overview

Small Wiki Toolkits is an initiative that focuses on building technical capacity in smaller language wikis and a global community of practice for discussing and resolving the challenges of these wikis. 

The Wikimedia projects are powered by the MediaWiki software and extensions. Community-built tools, bots, gadgets and templates enhance the software experience and support a variety of workflows. A visible example in the user interface is an infobox, which is generated through templates. Tools, bots, templates and gadgets play an essential role for the curation, maintenance and growth of content. Technical contributors across wikis perform a wide variety of tasks, such as site configuration requests (e.g. adding a custom namespace, changing a logo), importing gadgets and templates, software development, documentation, reporting bugs and translating technical news.

About 1/3 of the total edits to the Wikimedia projects come from tools and bots that help wikis with site maintenance, content moderation, editing, etc. Most of them are using the Wikimedia Cloud Services (WMCS) infrastructure. In the screenshot below, you can see that 7.5% of Punjabi wiki’s edits came from the tools and bots that are hosted via WMCS in March of 2020. To develop and use impactful tools and workflows, communities need technical contributors. Hence the initiative–Small Wiki Toolkits seeks to support smaller wikis that cannot address unmet technical needs by developing toolkits and recommendations on technical topics and organizing workshops to equip people with the necessary skills to make their work on wikis easier and more effective.

This initiative was kicked off at Wikimania last year. Since its inception, over 200 people have participated in 16 informational sessions, technical workshops, and group discussions through 6 Wikimedia events (local, regional, and international) online and offline. Through these events, the initiative engaged people from many smaller wikis from all over the world. 

Activities and highlights

Let’s take a look at the different formats that the initiative experimented with in its first year:

Brainstorming sessions to understand the technical challenges smaller wikis face.

In a typical brainstorming session, participants would discuss the situation in their wiki community, problems they would like to solve as a technical contributor and missing gaps in their workflows (tools, processes, skills, etc.). Participants shared ideas for how to enable more people to do technical work and get better at working together across wikis. Five such group discussions helped gather challenges and ideas for possible solutions from around 30 smaller language wikis including Breton, Macedonian, Basque in the European region, Indic, Arabic and indigenous language wikis from the North American region.

In a nutshell, challenges reported were:

• Lack of awareness and knowledge to use tools and technologies (e.g., anti-vandalism tools, bots, Phabricator for bug reporting, etc.)
• Lack of specific technical skills (e.g., connecting infoboxes to Wikidata, using Lua modules, using templates, etc.)
• It’s hard to keep up with new changes that break the wikis
• It’s difficult to handle cumbersome wiki workflows (e.g., importing templates or gadgets from one wiki to another)
• Lack of technical resources in different languages
• Localization/ translation issues in many areas

In these conversations, people from a few wikis that are just getting started and working to grow content also participated. To address the needs of such wikis currently feels out of the scope of this initiative. Small Wiki Toolkits is focusing on smaller wikis that have reached a tipping point in their content development and are now looking for technical help to expand further. In these sessions, connections were made between participants from different wikis to collaborate on longstanding issues. Also, some sessions surfaced success stories; for example, immediately after one such session, a community member felt encouraged to apply for global interface rights and received it. 

Technical workshops and informational sessions

The Small Wiki Toolkit’s initiative’s members developed and curated a few teaching and learning resources on a wide variety of technical topics. These were documented on the Small wiki toolkits’ landing page on meta-wiki. There were also guidelines developed for individuals interested in developing the toolkits, keeping in mind that potential trainers and learners might utilize them and should have a lot of notes in the slides.

Ten workshops were conducted on eight technical topics: developing user scripts and gadgets, working with Wikimedia APIs, writing templates in Lua, generating Wikidata infoboxes, leveraging Wikimedia cloud services, translating pages via the Translate extension, writing Wikidata queries, using Phabricator, etc. Out of these ten workshops, seven were part of the 2019’s Wikimania and remaining of a workshop series designed specifically for the Indic community. Some topics were selected based on the needs of community members gathered through a survey.

One of the participants at Wikimania live-tweeted a workshop:

These workshops showed promising results. Attendees seemed to be engaged and gained mastery in a topic. As per the Indic workshop series report, participants’ skills and awareness in technical areas increased by 30% on average.

Two participants provided feedback: 

The session was useful and moreover it was in Hinglish (A mix of Hindi and English), and I feel it was more adaptive for me to understand. The workshop started with the basics, which really helped me. Workshop was so interesting that some of the attendees were eager to learn more even after workshop time limits. I think it should not [be limited] to one session.

Overall experience of [the] workshop was excellent, and I would say that such [a] kind of workshop should be organised frequently.

Starter kit for smaller language wikis

One of the solutions proposed in response to the technical challenges during a brainstorming session was to develop a set of resources, tools, recommendations in technical areas relevant to smaller wikis that are just getting started. With continuous feedback from experts from many different wikis, the Small Wiki Toolkit initiative developed a starter kit for smaller language wikis. After receiving final feedback from participants of the Celtic Knot Conference 2020, it was released and announced on the mailing lists. Overall, the starter kit received a good response, and since it was published, it has been viewed 1,234 pageviews, with a daily average of 41.

Looking ahead

The Small Wiki Toolkit initiative identified a vast technical skill gap in small wiki communities. These communities need to rely on outside experts to address technical issues. A post-workshop series survey reflected the participants’ continued interest in attending workshops.

For the second year, the initiative is planning monthly technical skill-sharing workshops for the smaller wikis, beginning with the South Asian region, in which there are eight countries. This project will be a collaboration between the Indic-TechCom and the Wikimedia Foundation’s Developer Advocacy team. These workshops will allow participants to exchange challenges in the brainstorming session format discussed above, learn new skills, and connect with members from other communities. Exploring use cases and needs for automation, and a program around developing and running bots are further topics under consideration.

Another experimental idea is to allow community members who are interested in attending the technical workshop to take a prerequisite course via an external course provider and obtain a certificate for it. For example, for writing user scripts and gadgets, one must be familiar with JavaScript & JQuery fundamentals. Without prior knowledge of these topics, for a novice programmer, the learning curve to jump straight into the user script topic could be very high. And, this idea would help reduce it.

If you are interested in joining and supporting the Small Wiki Toolkits initiative, please add your name to the members list. Ideas and feedback are welcome on the talkpage of the initiative.

About this post

Featured image credit: Participants at WikiArabia 2019 in Marakesh, Noureddine Akabli, CC BY-SA 4.0

How we went from an error counter to fixing our code

By Jon Robson, Staff Software Engineer, The Wikimedia Foundation

A while back I wrote about debugging an issue on mobile armed only with an error counter and referrer information. Thankfully I’m pleased to report that we are now tracking and fixing client-side JavaScript errors for Wikimedia wikis which are providing more error-free experiences to our end users. In this blog post, I wanted to document what led us to prioritize it; how we went about implementing a solution; and what we learned from the experience. If you are planning to log client errors in your own project, you may find the summarizing section at the end of the article useful.

Preparing for the voyage

You may have recently heard about how Wikipedia has made a recommendation to adopt Vue.js or how we are redesigning the desktop site. As you can imagine migrating an entire front-end stack comes with certain challenges. We were diving into a large technical project with no knowledge of the existing landscape in which we were working. In fact, much of our code hadn’t been touched in some time. In addition to this our mobile error counting had also jumped from 10k a minute to 40k a minute with no obvious cause¹ signally we had introduced a bug somewhere. It was clear to the developers tasked with this work, that sophisticated error handling was a requirement for this project.

Luckily this dovetailed nicely with the scaling up of our Event Platform client and a global pandemic where our focus has shifted to site stability. Existing solutions such as Raven and Sentry were considered, but given the size of the client libraries comparatively to the rest of the software and out of a desire to minimize additional tools for developers, a decision was made to roll our own client code and send unmodified errors and stacktraces to our existing Kafka-Logstash logging infrastructure².

We had little idea what we would uncover through our explorations.

Preparing for launch was just the start

Before I even got involved in the work, there was an amazing collaboration between the security, performance, operations, analytics, and product infrastructure to get everything in place. This effort is tracked on our public task management system for those that are interested in the details.

Thanks to the collaboration and planning of these teams, the code for catching errors and sending them to the server ended up being relatively small. This effort culminated in the launch on Hawaiian Wikipedia, one of our less often visited wikis. The challenge now became how to do this at scale.

Diving into the unknown

We maintain multiple projects and our biggest Wikipedia alone has 293 languages. It’s an understatement to say that our ecosystem is complex. You’ve likely read about how bots are widely used in Wikipedia, but many of our users all rely on bespoke tooling to manage a variety of editing activities that are provided by browser extensions and user gadgets (special wiki pages that allow users to write and run their own JavaScript).³

To roll out further, we had to ensure that bugs were getting fixed, the information we didn’t care about from user gadgets and scripts did not drown the signals we got from more important errors, and most importantly the traffic to the end point was low such that it wouldn’t bring down our services.

We started small on a small wiki for the software that runs our sites: mediawiki.org. From a product perspective, even though the audience was low, it gave us a hint about what to expect as we rolled out further. The errors logged were a good way to capture very prominent errors e.g. errors occurring on every page view. Many of these errors had very actionable stack traces meaning we could file bugs and fix them, which we did, however others were more cryptic. “Script error.” was one of the most prominent and the most unclear to fix.

The reason we were getting “Script error” was due to the fact that code was being loaded from across our websites. Many of our users were loading gadgets from other wikis. So while this could be explained, it was not very actionable without a stack trace. However, it did provide the file which caused the error which could be linked to the associated wiki page. Similar to my bug hunting adventures in my previous blog post, it could be used to manually reproduce those errors in exploratory testing.

However, the errors we were seeing on mediawiki.org represented a small drop in a large ocean of errors for our projects. Despite the rate of errors for these smaller wikis being sufficiently low, we knew that there were more errors out there waiting to be explored, the kind that only come from really stretching the use our software — for example editing a wiki page with complicated markup or copying and pasting a table with thousands of rows! We needed to roll out to larger wikis to help prioritize bug fixing in our product workflow by identifying where in our complex code ecosystem our users were commonly hitting problems in our code. But where to begin?

We looked at traffic first to guide us. We wanted wikis bigger than our current wikis, but not too big. Thanks to the error counter we introduced in our mobile site we were able to get guesstimates of the actual error rates on mobile for possible wikis by looking at referrer traffic. Combining this information with the actual error rates being ingested for Hawaiian Wikipedia and mediawiki.org for all our traffic and the traffic for all our wikis, we were able to predict the volume of errors for our candidate wikis on desktop too. We choose a wiki that would inform us more about the errors our users were actually facing without overwhelming our event platform. We decided to enable on our Catalan language Wikipedia.

Lesson one: Not all user errors are equal

Running the client-side error tracking on Catalan Wikipedia was really helpful for prioritizing roll out. Due to the larger volume of traffic, we went from around 2,000 errors a day to 40,000, however, this taught us a lot. Certain errors we were seeing in the same software on our smaller websites were now occurring more frequently which helped us prioritize them.

What became apparent however was certain errors were being repeated by a single IP. We hadn’t been able to notice that with such low traffic. At one point 38,830 of 48,175 errors in a 12 hr period logged to Catalan Wikipedia came from a single IP address who was running some badly maintained gadgets that they had included almost a decade ago.

In another interesting development, from the stack traces, we identified a bug related to our map feature which was only occurring on slow connections. When the bug occurred unfortunately it got executed in an interval timer so appeared in high volumes. A patch was provided and that bug was squashed. It turned out this accounted for 50% of our errors on mobile and our error counting graph adjusted accordingly.

As for other bugs, many of them came from faulty and forgotten scripts that users were running. After reaching out to those users, we managed to clean those up.

These two incidents once fixed got the error rate down to a very manageable and respectable rate of 735 errors ever 3hrs. A lesson was learned and we began to limit the number of bugs we logged from the same user session to five. With that obstacle out of the way, we felt confident to roll out further.

Lesson Two: Not all page JavaScript is equal

With errors limited, the list of errors was much more manageable, however, it was clear that many of the stack traces were not useful. In certain cases, the code was clearly coming from places outside of our control. Interestingly we also saw a lot of bugs from browser extensions and non-Wikimedia domains — Google Translate for example and browser extensions like GreaseMonkey that allow the running of locally written JavaScript. Some of these errors did not have any information about stack traces or the source of the error. The noise of this unfortunately made it difficult to identify real errors, so we decided to exclude any errors without information on their origin. If a bug falls and nobody knows what file it came from, is it really a bug?

We however decided to defer the filtering of browser extensions and non-Wikimedia sites as this could be taken care of using logstash filtering if needed and it was useful to know, however, a bug has been filed about reconsidering that in the future, potentially sending them to a different channel.

Lesson Three: Some errors are the same

The end-users for the mediawiki.org and Catalan Wikipedia were not fully representative of all our users so a logic next step was to enable bug tracking on one of our larger language projects written in a right to left text script. Hebrew was an obvious choice, as the community there had previously volunteered for earlier deployments to help us catch errors before they reach production meaning if we saw client errors there we would have time to block the bug before it could impact other projects.

When we enabled the error tracking on Hebrew Wikipedia, the bump in errors was not as significant as it might have been. We didn’t learn much from this deployment, other than we were getting closer to the finish line. Sometimes a deployment just gives you the validation you need to continue.

After this, we expanded coverage further to Wikimedia Commons, our site for uploading images. As we’ve rolled out further we’ve benefited more from our scale, we noticed problems in our post-processing normalization. Some errors are prefixed with “Uncaught” and some are not. For example, the error TypeError: $submitButton.submitOnEnter is not a function is the same as Uncaught TypeError: $submitButton.submitOnEnter is not a function . This meant that similar errors were not being grouped.

After a month of roll out a significant milestone was reached when our infrastructure surfaced our first deployment blocker not reported by an end-user which was swiftly fixed. Hurray!

Future voyages

The majority of the issues that we are now surfacing relate to user gadgets. User gadgets are a historic artifact of the wiki ecosystem that predates browser extensions which have allowed many editors to self-serve and many of our editors depend on them. Some of these user-written gadgets are defaulted to run for all users, including anonymous users, so we have to be careful that as we roll out further we address or communicate with our editors to get those fixes.

We’ll continue this approach with further wikis, but this will take time. At the time of writing, we are seeing about 60,000 errors a day from 10 of the ~1000 sites we maintain.

While hopefully obvious, one thing that’s important is that now we have the ability to identify client-side errors that we use this information to fix bugs and to block deployments where necessary. Just adding the tool is not enough. This requires a lot of socializing the change, which is partly why I am writing this blog post. I have also started attending a weekly triage meeting and including consideration of these errors in our existing triaging processes. My hope is that all teams working on software for our projects will have a workflow for triaging and addressing such bugs without my assistance.

Before you jump in on your own project

I’ve linked to the associated tasks for these takeaways so you can see further discussion and implementation details if you are interested.

• If you are not sure of the number of errors your site will generate, allow yourself an educated guess by counting errors first to get an idea. You could also use page views to restrict the error ingesting to only certain pages.
• Limit the number of errors you track on a single IP. One faulty script from one power user could mislead you and overwhelming your data collection— particularly if it’s in run regularly e.g. using setInterval or a scroll event handler!
• Make sure you only track errors from domains you care about and consider checking the stack trace for hints that the script originated from Firefox and Chrome extensions or user scripts while logging or post-processing.
• Remember when grouping and filtering errors that some errors may be prefixed with “Uncaught.” Normalize your messages before grouping and displaying them in a developer tool.
• Some user scripts will have no associated file URI or stack trace. In my opinion, these are not worth caring about so avoid logging them.
• If you load code from external domains these will be anonymized with “Script error”. Try to replicate those issues through exploratory testing or if possible run local mirrors. It may even make sense to filter out those errors.
• Make sure your engineers have workflows for triaging new errors during and after deployments and ensure fixing and slowing down is part of your workflow.

How volunteers can help

• If you are a Wikimedia staff member or volunteer and have sufficient permissions you can use our JavaScript dashboard to help identify and fix bugs caused by gadgets or your own code.
• If you are an editor, you can help make sure errors get the attention they need by doing your part. Please review your user scripts that live on the wikis you frequent. Please blank them if you are not using them, use gadgets if you can, and ensure they are not throwing errors on every page you view.

Footnotes

¹ for those interested, we eventually diagnosed the increase to relate to a bug in some incorrectly configured code using the mapbox library which triggered on slow connections leading errors to be thrown inside a setInterval function in high quantities. The full bug report can be read on https://phabricator.wikimedia.org/T257872
² The meeting notes can be found on the Wikimedia etherpad for those interested: https://etherpad.wikimedia.org/p/clients-error-logging
³ An example of a gadget that allows users to place comments onto images shown on file description pages can be viewed at https://commons.wikimedia.org/wiki/Help:Gadget-ImageAnnotator. The code lives at https://commons.wikimedia.org/wiki/MediaWiki:Gadget-ImageAnnotator.js

About this post

This post also appeared on https://jdlrobson.com/posts/

Featured image credit: Christoph Wolff, son of Dr. Michael Wolff, scubadiving at Crystal Bat, Nusa Penida, Indonesia at the age of 12, HenryHiggens, CC BY-SA 3.0

15/09/2020-21/09/2020

Mapping

• Alex started a discussion on what to consider when mapping a street’s width and the different variants currently used. In the discussion, Tobias Zwick, from StreetComplete, referred to a helpful app for measuring with your phone and the help of augmented reality.
• After the approval of amenity=funeral_hall, user Vollis is proposing the subtag funeral_hall=*, for a building or room used for funeral ceremonies ancillary to a funeral director’s shop or a crematorium.
• Andrew Wiseman, from Apple, has updated > the challenges on MapRoulette for Bolivia using new data. The challenges can be found here.
• Michael Montani, from the UN Department of Operational Support, invited > OSM contributors, on various African local discussion lists, to help support UN missions. Mappers can do this by participating in the Tasking Manager jobs they have created to map highways, waterways, land cover, land use, places, and any other relevant points of interest.
• User stragu warned users of OsmAnd Maps for iOS about a bug that shifts POIs without notice, due to incorrect rounding of coordinates. The issue affects versions 3.10 to 3.14, but was promptly addressed and is solved in version 3.80.

Community

• After a long break, the Belgian community continues its choice of the ‘Mapper of the Month’. This month it is Jacques Fondaire, aka jfonda, from Belgium.
• [1] The widely-used tool ‘How did you contribute to OpenStreetMap’ celebrated its 10th anniversary – we congratulate and say thank you, Pascal Neis, for this and also for all the other nice stats you have added over the years!
• The 35th edition of the Geomob Podcast features an interview with Sarah Hoffmann who, as a software developer, is responsible for the maintenance of Nominatim, the most important software for geocoding used by OpenStreetMap. Sarah explains the technical aspects and challenges of managing a technically complex and publicly visible open source project.
• Maning Sambale notes that diversity and inclusion are fundamental for OSM’s spread and growth, but seldom make it to centre stage. Plenary sessions such as State of the Country at SotM Asia 2016 and GeoLadies-PH plenary at Pista ng Mapa 2019 aim at increasing the audience listening to community stories. The second Pista ng Mapa (November 2020, online) hopes to feature more stories, maps and posters from the community.
• OSM contributor hocu wrote about a changeset discussion where he (once more) realised that, thanks to OSM contributor Alikam, the OSM data is more up to date than the official sources, IETT (the governing body of Istanbul’s transport network) in this case.

OpenStreetMap Foundation

• Christoph Hormann took a look at OSMF’s strategic reorientation and changed way of working and communicating over the past year and expresses concern and frustration about the, in his view, insufficient attention to the long-term consequences and risks of this.
• Rory McCann (᚛ᚏᚒᚐᚔᚏᚔᚋ᚜ ) wrote in his blog about his activities in August 2020 and reports, among other things, about his activities on the OSMF Board of Directors and the Communication Working Group. mmd complained in a comment that apparently no formal candidate selection process (for the new paid positions on the OSMF) has taken place. Instead, he suspects that this was ‘all done behind the scenes’. Rory confirmed, in his response, that OSMF ‘had a preferred candidate in mind, rather than an open interview process’.

Events

• Last weekend members from the Italian OSM community gathered > in Limone, Piemonte for a weekend dedicated to OSM. On Saturday, they gave talks and shared their OSM experiences and useful tools for mapping. Sunday was dedicated to mapping excursions around the area using OSM-based phone apps.
• On Monday 21 September, the New Caledonian chapter of the cycling association ‘Droit au Vélo’ hosted a mapping party in its office in Nouméa. The event focused on mapping cycling infrastructure and welcoming new mappers.
• The Hungarian colleagues have moved their meetings and ‘regulars’ table’ to the Internet .
• A virtual mapathon will be held > on 30 September by the Italian OSM community. During the mapathon participants will draw the maps of the 78 municipalities of Benevento based on satellite images using OpenStreetMap. The event is a collaboration between Valerio De Luca (Map For Future Roma > ), Luciano Amodio (ThinkSannio ), and Nicola De Innocentis (GeoPillole), with the support of Wikimedia Italia > .

Humanitarian OSM

• The European Youth Humanitarian OpenStreetMap is planning a two-day online workshop on 15 and 16 October, aimed at students and teachers. The participants will learn to use JOSM and will map the streets and houses on the island of Terceira (Azores), as pre-disaster mapping.
• The app MapSwipe is being extended to be able to compare the completeness of OSM data with airborne or satellite imagery. A first prototype was developed and tested in the LOKI project at Heidelberg University. The use case is building footprints, as they are important inputs to earthquake risk exposure models.
• HOT detailed their strategy for achieving equal pay and organisational gender equality in a news post on International Equal Pay Day.
• HOT is currently in a hiring push, with many jobs available, to scale up their staff to carry out the Audacious Project to map an area across 94 countries, home to one billion people.

Education

• Daniel Feldmeyer et al. have published an article with the title ‘Using OpenStreetMap Data and Machine Learning to Generate Socio-Economic Indicators’ in the International Journal of Geoinformation. A remarkable sentence from the abstract: ‘OSM provides an unparalleled standardised global database with a high spatio-temporal resolution. Surprisingly, the potential of OSM seems largely unexplored in this context’. In their study, they used machine learning to predict four exemplary socio-economic indicators for communities based on OSM data.

Maps

• Camille Scheffler announced > on Talk-fr that magOSM > , a project of services linked to OSM thematic data from the Magellium company, offers four new thematic layers at the level of France :
  1. Train Itineraries
  2. Social structures
  3. Construction Proposals
  4. Constructions in progress
• Jean-Louis Zimmermann mentioned >
  on Talk-fr that OSM-fr publishes a demo map of thematic layers known only to the initiated. Further announcements are planned, and comments are welcome. The different layers, such as territories, schools, and transportation, are selected
  simply on the map from a list.
• Google Maps will soon show COVID-19 risk areas in mobile versions so that users can avoid them. This does not mean automated routing.

Open Data

• The Digital Elevation Model (DEM) of Italy at 10 m resolution is available > for the whole country as open data and downloadable here.
• The Baden-Württemberg mobility data platform MobiData BW started > operations on 14 September 2020. On the new website > , the available data and interfaces are clearly described and are available for further use.

Software

• A new WordPress plugin called ‘Out of the Block’ has been published. Benefiting from WordPress’s Gutenberg editor, the plugin tries a different approach to adding locations on a map and other user actions compared to existing plugins.
• Google is a heavy user of open source tools. They are using iD for the historical map project Kartta Labs. Kartta Labs uses a customised version of iD, which lets everyone draw the historical landscape of a city based on georeferenced old paper maps.

Releases

• An update of OsmAnd has been released. For iOS, version 3.80 introduces application profiles with independent settings, the ability to import and export profile settings, the capability to download online maps to cache, and fixes a crash that appeared while starting navigation. On Android, version 3.8 brings an updated plan a route function, a new appearance menu for tracks, and improved search algorithms.

Did you know …

• … you can download > geological maps for all of Germany to your smartphone, with this detailed guide, and use them offline with OsmAnd?
• … Taginfo, the site which allows you to explore the tags and values used in OSM? Several local versions are available for you to explore at a country level.
• … there is a step-by-step guide > on how to display parking spaces reserved for people with disabilities in OsmAnd?

OSM in the media

• Just van den Broecke informed > us that the live streaming web show, which ran April to June 2020, will return, running on the first Thursday each month (19:00-21:00 CEST), starting on 1 October.
• Rina Chandran reported about HOT’s activities in Indonesia, where handwashing stations were mapped to help fight COVID-19.

Other “geo” things

• Geospatial Media and Communications will present the Geospatial World Awards on 6 October virtually for the first time due to the COVID-19 pandemic . Since its launch in 2007, the awards have been presented to more than 200 individuals and organisations for remarkable innovations and ideas in global geodesy.
• After a successful trial showed enormous potential, Ramblers have embarked on the Mapping Scotland’s Paths project.

Upcoming Events

Where	What	When	Country
Montrouge	Soirée de fin de projet “Ça reste ouvert”	2020-09-24
Bratislava	Missing Maps Mapathon Bratislava #9	2020-09-24
Munich	TUM – Mapping Party	2020-09-24
online	HOT Working Groups 101 Community Webinar	2020-09-25
Düsseldorf	Düsseldorfer OSM-Stammtisch	2020-09-25
Helsinki	State of the Map Suomi 2020	2020-09-26
online	FOSSGIS OSM Communitytreffen	2020-09-27
Salt Lake City / Virtual	OpenStreetMap Utah Map Night	2020-09-29
Zurich	Missing Maps Mapathon Zürich	2020-09-30
Ulm + virtuell	Covid-19-Mapathon	2020-10-01
San José	Civic Hack & Map Night	2020-10-01
Taipei	OSM x Wikidata #21	2020-10-05
London	Missing Maps London Mapathon	2020-10-06
Stuttgart	Stuttgarter Stammtisch	2020-10-07
Berlin	148. Berlin-Brandenburg Stammtisch	2020-10-09
Michigan	Michigan Online Meetup	2020-10-12	USA
Cologne Bonn Airport	132. Bonner OSM-Stammtisch (Online)	2020-10-13
Munich	Münchner Stammtisch	2020-10-13
online	2020 Pista ng Mapa	2020-11-13-2020-11-27
online	FOSS4G SotM Oceania 2020	2020-11-20

Note: If you like to see your event here, please put it into the calendar. Only data which is there, will appear in weeklyOSM. Please check your event in our public calendar preview and correct it, where appropriate.

This weeklyOSM was produced by AnisKoutsi, Anne Ghisla, Climate_Ben, MatthiasMatthias, MichaelFS, Nordpfeil, NunoMASAzevedo, PierZen, Polyglot, Rogehm, Supaplex, TheSwavu, YoViajo, alesarrett, derFred, richter_fn.

What’s the deal with these string methods, and how are they different?

substr()

str.substr(start[, length])

This method takes a start index, and optionally a number of characters to read from that start index with the default being to read until the end of the string.

'foobar'.substr(2, 3); // "oba"

The start parameter may be a negative number, for starting relative from the end.

Note that only the first parameter of substr() supports negative numbers. This in contrast to most methods you may be familiar with that support negative offsets, such as String#slice() or Array#slice(). The second parameter may not be negative. In fact, it isn’t an end index at all. Instead, it is the (maximum) number of characters to return.

But, in Internet Explorer 8 (and earlier IE versions), the substr() method deviates from the ECMAScript spec. Its start parameter doesn’t support negative numbers. Instead, these are silently ignored and treated as zero. (I noticed this in 2014, shortly before Wikimedia disabled JavaScript for IE 8.)

IE 8:

'faux'.substr( -1 ); // "faux"

Standard behaviour:

'faux'.substr( -1 ); // "x"

And, the name and signature of substr() are deceptively similar to those of the substring() method.

String substring()

str.substring(start[, end])

This method takes a start index, and optionally an end index. At glance, a very simple and low-level method. No relative lengths, negative offsets, or any other trickery. Right?

Behold! The two parameters automatically swap if start is larger than end. ^[1]

'foobar'.substring(1, 4); // "oob"
'foobar'.substring(4, 1); // "oob", also!

Unexpected values such as null, undefined, or NaN are silently treated as zero. For substring() this also applies to negative numbers.

And, of course, the name and signature of substring() are deceptively similar to substr().

String slice()

str.slice(start[, end])

This method takes a start index, and optionally an end index that defaults to the end of the string. Either parameter may be a negative number, which is interpreted as a relative offset from the end of the string.

I found no defects in browsers or JavaScript engines implementing this method. And it has been around since the beginning of time.

Its only weakness is also its greatest strength — full support for negative numbers.

One might think this can be ignored for cases where you only intend to work with positive numbers. You’d be right, until you write code like the following:

start = something.indexOf(needle); // returns -1 if needle not found.
remainder = str.slice(start); // oops, -1 means something else here!

The notion of negative offsets was confusing to me when I first learned it. But, over the years, I’ve come to appreciate it and it actually became second nature to think about offsets in this way. If you’re unfamiliar, see the examples below.

Conclusion

First, let us compare these methods once more:

str = 'foobarb…z';

// Strip start "foo" > "barb…z"
str.slice(3);
str.substring(3);
str.substr(3);

// Strip end "z" > "foobarb…"
str.slice(0, -1);
str.substring(0, str.length - 1);
str.substr(0, str.length - 1);

// Strip "foo" and "z" > "barb…"
str.slice(3, -1);
str.substring(3, str.length - 1);
str.substr(3, str.length - 3 - 1); // 👀

// Extract start > "foo"
str.slice(0, 3);
str.substring(0, 3);
str.substr(0, 3);

// Extract end > "z"
str.slice(-1);
str.substring(str.length - 1);
str.substr(str.length - 1); // Compat
str.substr(-1); // Modern

// Extract 4 chars at [3] > "barb"
str.slice(3, 3 + 4);
str.substring(3, 3 + 4);
str.substr(3, 4); // 👀

None of these seem unreasonable, in isolation. It’s nice that slice() allows negative offsets. It’s nice that substring() may limit the damage of accidentally negative offsets. It’s nice that substr() allows extracting a specific number of characters without needing to add to the start index.

But having all three? That can incur a very real cost on development in the form of doubt, confusion, and inevitably mistakes. I don’t think any of these is worth that cost over some minute localised benefit.

I find substr() or substring() cast doubt on surrounding code. I need to second-guess the author’s intentions when reviewing or debugging such code. Which is wasteful even, or especially, when they (or I) used one correctly.

But what about unit tests? Well, there’s sufficient overlap between the three that a couple of good tests may very well pass. It’s easy to forget exercising every possible value for a parameter, especially one that is passed through to a built-in. The question isn’t whether the built-in works. The question is – did we use the right one?

This ubiquitous signature of slice() is well-understood. It is a de facto standard in technology, seen in virtually all programming languages. It is applies to strings, arrays, and sequences of all sorts. As such, that’s the one I tend to prefer.

But more important than which one you choose, I think, is the act of choosing itself. Eliminating the others from your work environment reduces cognitive overhead in development, with one less worry whilst reading code, and one less decision when writing it. ^[2]

1. This “argument swapping” behaviour in substring() has existed since the original JavaScript 1.0 as implemented in Netscape 2 (1996), and reverse-engineered by Microsoft in IE 3. The behaviour was briefly removed by Netscape 4 with JavaScript 1.2 in June 1997, but that same month the misfeature finished its fast-tracked standardisation as part of ECMAScript 1. Thus, the misfeature returned in 1998 with the release of Netscape 4.5 and JavaScript 1.3, which aligned itself with the new specification. ↩︎
2. In 2014, I wrote a lengthy code review about the string methods which, after much delay, I used as the basis for this article. ↩︎

For the third year in a row, Amazon has renewed its commitment to the Wikimedia Endowment, a permanent fund dedicated to ensuring the long-term future of Wikipedia and the other Wikimedia free knowledge projects. With this gift, Amazon reinforces the importance of ensuring people everywhere can create and share free knowledge.  

With millions of students learning from home, teachers adapting their curricula to online education, and people looking for up-to-date information about the spread of COVID-19, the world has turned to Wikipedia as a critical source of knowledge during the pandemic. The need to conserve and expand the world’s largest free knowledge ecosystem has become even more acute. 

Corporate donations, like those from Amazon today, allow the Wikimedia Foundation, the nonprofit that operates Wikipedia, to prepare for uncertain times and ensure Wikipedia and the other Wikimedia free knowledge projects remain available for everyone to use, even during urgent crises. Companies regularly use Wikimedia projects and our freely-licensed content to power their work and further advance their goals. Commensurate support from these organizations helps sustain our projects and mission of delivering free knowledge to the world.

“Wikipedia is one of the largest and most beloved websites in the world, with hundreds of millions of people seeking information from its articles every month. The Wikimedia Foundation is committed to keeping Wikipedia and the other Wikimedia projects free and available to all, and gifts like this latest contribution from Amazon help us do just that,” said Amy Parker, Endowment Director at the Wikimedia Foundation. 

The Foundation has raised $64 million for the Endowment since its inception, thanks to gifts from generous individuals, corporations, and foundations donors.To learn more about how you can support the Wikimedia Endowment, visit www.wikimediaendowment.org or email endowment@wikimedia.org.

If you're making changes to a service that is deployed to Kubernetes, it sure is annoying to have to update the helm deployment-chart values with the newest image version before you deploy. At least, that's how I felt when developing on our dockerfile-generating service, blubber.

Over the last two months we've added

• A script to update the image versions (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/update_version/)
• A new 'promote' step to PipelineLib (https://wikitech.wikimedia.org/wiki/PipelineLib) to run the script and create the patchset on merge
• Various infrastructure pieces to ensure necessary packages are available

And I'm excited to say that CI can now handle updating image versions for you (after your change has merged), in the form of a change to deployment-charts that you'll need to +2 in Gerrit. Here's what you need to do to get this working in your repo:

Add the following to your .pipeline/config.yaml file's publish stage:

promote: true

The above assumes the defaults, which are the same as if you had added:

promote:
  - chart: "${setup.project}"           # The project name
    environments: []                    # All environments
    version: '${.imageTag}'             # The image published in this stage

You can specify any of these values, and you can promote to multiple charts, for example:

promote:
  - chart: "echostore"
    environments: ["staging", "codfw"]
  - chart: "sessionstore"

The above values would promote the production image published after merging to all environments for the sessionstore service, and only the staging and codfw environments for the echostore service. You can see more examples at https://wikitech.wikimedia.org/wiki/PipelineLib/Reference#Promote

If your containerized service doesn't yet have a .pipeline/config.yaml, now is a great time to migrate it! This tutorial can help you with the basics: https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial#Publishing_Docker_Images

This is just one step closer to achieving continuous delivery of our containerized services! I'm looking forward to continuing to make improvements in that area.

By Andrew Otto, Staff Site Reliability Engineer

In our previous post in this series, we described how Wikimedia manages and distributes event schemas. Now that we’ve got that nailed down, let’s look into how we use them to ensure events conform to their schemas.

Kafka has client implementations in many languages, but those implementations don’t do any schema validation. If we were using Avro with Confluent’s Schema Registry, we’d need custom Serializers (e.g. KafkaAvroSerializer or this Python AvroProducer) that know how to contact a specific Schema Registry service before they can produce data. Fortunately for us, we don’t need to do any special serialization, as we don’t need a schema to serialize JSON data. However, we’d like to enforce that event data in Kafka topics always contain events that validate with a specific schema. 

Even though Kafka clients exist in many languages, the quality and freshness of those client implementations vary. One of Wikimedia’s main Kafka use cases is producing event data from MediaWiki, which is written in PHP. Our MediaWiki application servers handle around 15000 requests per second. Kafka is pretty efficient at producing messages, but the bootstrap and connection time for a new Kafka producer isn’t quite as fast. Being PHP, MediaWiki would have to create and connect a new Kafka producer client for each of these requests, which would add serious overhead and latency to the application servers handling these requests.

We also need to intake events from external sources, like browsers and mobile apps, which don’t have access to internal Kafka clusters.

Sometimes you need to produce data to Kafka without a Kafka client. Confluent has a Kafka-HTTP REST Proxy to handle situations like these. Wikimedia has EventGate.

EventGate

EventGate is a NodeJS library and HTTP service that receives event data in HTTP POST requests, validates the event, and then produces it. The validate and produce logic are pluggable, but the original intention was for EventGate to validate events against JSONSchemas looked up from $schema URIs, and produce them to Kafka.

When used as an HTTP service, the API is very simple:

POST /v1/events

The POST body should be a JSON serialized array of events (or just a single event object) to validate and produce. It is expected that each event contains information that the provided validate and produce logic can use to do their jobs. The provided EventValidator implementation of validate expects that events have a schema URI field (default $schema) that can be used to lookup the event’s schema for validation. The provided Kafka produce functionality expects that a configured field in the event contains the destination ‘stream’ name, which will be used to determine the Kafka topic that the event should be produced. 

Note that the API itself does not include information about what topic an event should be produced to, or what schema the body of events should be validated with. This is intentional. Even though EventGate was built to use with Kafka, we wanted to keep the main functionality generic and pluggable, and so the API had to be generic too. We also wanted to be able to batch events of multiple types and destinations in a single HTTP request, so an API parameter that specified e.g. a single schema would not help if the POSTed events could all have different schemas.

The provided implementations in the main EventGate repository work well. They solve the problem of finding event schemas at runtime and using them to validate events, and they can produce to Kafka. But they are missing a crucial requirement: the ability to restrict the types of events that make it into a Kafka topic.

Stream configuration

Confluent’s REST Proxy doesn’t do this either. Instead, it relies on you setting specific topic configurations that specify the name of a ‘strategy’ implementation that (by default) maps from either the topic name to the name (subject) of a schema in the schema registry. Your producer implementation must know how to look up topic configurations and interact with the schema registry. This is all fine if you are using Java, as Confluent expects you to. But what if you are using a different language? You’ll now not only need a good Kafka client, but also an implementation of e.g. io.confluent.kafka.serializers.subject.TopicNameStrategy in your language.

Wikimedia solves this by creating what we call ‘stream configuration’, and using that to map from stream names (which are just an abstraction of Kafka topic names) to configuration about those streams, including the title of the JSONSchema that is allowed. JSONSchemas have a title, and we expect that all schema versions of the same lineage have the same title. The configuration for a stream has a schema_title setting. When our EventGate validate function gets an event to validate, it extracts both the $schema URI and the stream name from the event. The stream name is used to lookup stream configuration, from which the schema_title setting is compared with the schema’s title. If they match, and if the event validates against its JSONSchema, it is allowed to be produced.

We also use stream configuration for other things, including arbitrary configuration of remote producer clients. This allows product engineers and analysts to change certain behaviors of data producers, like the sampling rate, via configuration rather than code deployment.

Wikimedia’s stream configuration is useful for us, but its implementation would have been difficult to make generic for outside users, so we did not include it in the main EventGate library. We also had the need for some other Wikimedia specific customizations of EventGate. Since EventGate is pluggable, our implementation is in a separate repository that uses EventGate as a dependency. 

Pros and cons

I’ve talked a lot about how Confluent does things, and why we’ve done things differently. However, I’d like to acknowledge that Confluent is building software for use by many different organizations, which is not easy to do. The Schema Registry and all of the custom code needed to use it does have one huge advantage: you can use a Kafka client to produce messages directly to Kafka, and still get all the automated schema validation done for you in your application code. Kafka clients have some amazing features (exactly once, transactions, etc.) that are not available if you are using a HTTP proxy. The downside is that you need Confluent specific code available in your language of choice.

Wikimedia’s polyglot and distributed nature makes doing things the Confluent way difficult. EventGate and our decentralized schema model work for us, and we think there are other organizations out there that might find our approach useful. See the EventGate README for more on how to use EventGate.

EventGate, jsonschema-tools, and everything else that Wikimedia creates is always 100% Free Open Source software. We’d love any thoughts and contributions you might have, so please reach out with questions or issues or pull requests.

Acknowledgements

Wikimedia’s Event Data Platform would not be possible without the numerous Open Source projects it is built on. I’d like to take a moment to thank a few of the critical ones.

• Thanks to Confluent for all their work in the world of Event Driven Architectures. Their blog is amazing.
• Thanks to Evgeny Poberezkin and all the other contributors who work on AJV.
• Thanks to James Messinger and collaborators for json-schema-ref-parser as well as Martin Hansen for json-schema-merge-allof, both critical components of jsonschema-tools.
• Many thanks to all the incredible people working on Apache Kafka. Open Source is amazing.

About this post

This is the final part of a 3 part series Read part 1; read part 2.

Featured image credit: A stile at the bottom of Arreton Down, Editor5807, CC BY 3.0

Viann N. Nguyen-Feng, PhD, MPH, is an assistant professor in the Department of Psychology at the University of Minnesota, Duluth. She serves as core faculty in the counseling/clinical track and directs the Mind-Body Trauma Care Lab.

In late Fall 2019, I was doing my usual perusal of the Association for Psychological Science’s (APS) e-newsletter with its Weekly Wiki feature: “Wikipedia says: ‘In psychology and other social sciences, the contact hypothesis suggests that intergroup contact under appropriate conditions can effectively reduce prejudice between majority and minority group members.’ Did they get it right? Read on.”

Although the APS Wikipedia Initiative has been active since 2015 or so, that day, I decided that, yes, I wanted to read on. Many of the previously highlighted terms had appeared more social or cognitive focused than my area of work in counseling psychology (pop quiz: what does Wikipedia say about executive function? Anchoring? Self-categorization theory? Spatial memory?). But I then realized that my students and I can also find spaces to contribute — anyone can help make psychological science on Wikipedia as complete and accurate as possible.

Selecting psychological statistics for Wiki Education

The question then shifted to determining which Spring 2020 class I could incorporate a semester-long Wiki Education assignment. Two of my courses involved too many direct-service hours (Internship, Assessment II) to add another assignment without kindling overwhelming student stress. My remaining course option was Advanced Statistics II.

Involving a dozen early graduate students in compiling information for a Wikipedia article felt daunting. Statistics in psychology has traditionally — and understandably — been taught with a focus on methodology without intensive writing components. Further, the Psychology WikiProject, a categorization of psychology-related articles targeted for improvement, did not appear to have a psychological statistics topic. 

With curiosity about statistics as an option, I reached out to Dr. Helaine Blumenthal (Wikipedia Student Program Manager) and Dr. Ian Ramjohn (Senior Wikipedia Expert), who provided guidance on completing Wiki Educations’s instructor orientation. The orientation was only 30 minutes long yet enabled me to see that Wikipedia “isn’t just for words.” There was space for multimedia, photography, and illustration courses and, most relevantly, for courses in which students visualize data and could create original graphical figures. Wiki Education even provided guidance on designing a media contribution assignment. And thus, Wiki Education entered my Advanced Statistics II syllabus.

Making it happen

My first Wiki Education statistics semester took off with feedback and advice from Helaine, Ian, and my assigned psychology Wiki Education mentor, Dr. Patricia J. Brooks. I landed on having students complete three manuscript-style written reports for which they needed to provide their own data. Reports included interpretation of data analyses performed in “lab” and required at least one table or figure. In turn, students were assigned to provide three Wiki Education media contributions related to each report topic discussed during the semester. Among my class of nine students enrolled in Spring 2020 (an unprecedented semester, as you all know), 32 original files were uploaded to Wikimedia Commons.

I did not realize that psychological statistics was an uncommon Wiki Education course until that particularity was mentioned during an end-user testing meeting of new Dashboard features. The listed courses on the APS Wiki Education Campaign indeed seem to suggest a paucity of statistics classes — perhaps Advanced Statistics II at the University of Minnesota, Duluth was it. However, I hope psychological statistics instructors will come to recognize that Wiki Education is certainly a place for you and your students. 

Visualization of statistical concepts, particularly in areas quite relevant to the social sciences (e.g., mediation, moderation), appear quite needed. The vast majority of my students contributed graphical statistical models that used general or specific examples, yet color-coded equations and a table were also in the assortment. All students learned how to write captions that communicated concise, accurate, and understandable information to the general public. Currently, more than four months after the conclusion of the course, 31 files remain in Wikipedia articles. These files comprise 26% of all APS Wiki Education Campaign files currently retained in articles and demonstrate a 97% overall retention rate for the course (vs. a typical 53% of all files, exclusive of this statistics course). 

As the idiom goes, a picture is worth a thousand words — or, in the case of psychological statistics, a thousand numbers.

Getting student feedback

By incorporating Wiki Education into Advanced Statistics II, my students saw that statistics could be applied outside of the classroom, as they indicated on midterm and end-of-term evaluations. Learning statistical theories and procedures were not simply intellectual exercises, but a means to understanding other domains and an opportunity to share seemingly difficult concepts in a meaningful way to others:

“I think I will find myself using some of these teaching methods in the future! For example, I am a big advocate for being able to teach a concept/skill to someone else as a demonstration that you really understand that skill.”

“I really enjoyed learning the material and have been able to apply it in my work and daily life.”

“She really helps us engage with the information. I feel like I can take what I’ve learned in this class and apply it outside of the course.”

Because the media contributions were cumulative assignments of in-lab material, Wiki Education seemed well integrated into the course rather than a random side assignment. Approaching statistics as an applied laboratory and public engagement concept was initially challenging, yet grew on students over time:

“I really enjoyed how the course was set up once we knew how to handle labs.”

“I also like the way the weekly labs were set up as it required applying the information that we had just learned.”

“The design of the course has been different than what many of us are traditionally used to, but I think that the design has applied learning built into it with immediate feedback on how we did. It’s still hard to get used to the fact… but the labs seem to be a good check on our knowledge in a more applied way.”

By creating a space for students to share their knowledge, the students also appeared to increase their confidence in statistics:

“Personally, I think that I was able to grow in confidence in my statistical knowledge. When the first labs were completed, I felt like I knew nothing, and now when I perform a lab I feel like I know what I turned in is great!”

“This semester I have learned so much about statistics. I’ve gained a newfound interest and appreciation for it. In the past, I had previous teachers who pretty much made me lose my interest in statistics and made me feel like I would never be able to get it. But by taking this class with you I’ve learned so much and I’ve been able to apply statistics in my own projects, and am now even able to help other students understand it! …I’m really happy I took this class!!!”

China yesterday blocked the Wikimedia Foundation’s application for observer status at the World Intellectual Property Organization (WIPO), the United Nations (UN) organization that develops international treaties on copyright, IP, trademarks, patents and related issues. As a result of the block, the Foundation’s application for observer status has been suspended and will be reconsidered at a future WIPO meeting in 2021.

China was the only country to raise objections to the accreditation of the Wikimedia Foundation as an official observer. Their last-minute objections claimed Wikimedia’s application was incomplete, and suggested that the Wikimedia Foundation was carrying out political activities via the volunteer-led Wikimedia Taiwan chapter. The United Kingdom and the United States voiced support for the Foundation’s application.

WIPO’s work, which shapes international laws and policies that affect the sharing of free knowledge, impacts Wikipedia’s ability to provide hundreds of millions of people with information in their own languages. The Wikimedia Foundation’s absence from these meetings further separates those people from global events that shape their access to knowledge.

“The Wikimedia Foundation operates Wikipedia, one of the most popular sources of information for people around the world. Our organization can provide insights into global issues surrounding intellectual property, copyright law, and treaties addressed by WIPO that ensure access to free knowledge and information,” said Amanda Keton, General Counsel of the Wikimedia Foundation. “The objection by the Chinese delegation limits Wikimedia’s ability to engage with WIPO and interferes with the Foundation’s mission to strengthen access to free knowledge everywhere. We urge WIPO members, including China, to withdraw their objection and approve our application.”

A wide range of international and non-profit organizations as well as private companies are official observers of WIPO proceedings and debates. These outside groups offer technical expertise, on-the-ground experience, and diversity of opinions to help WIPO with its global mandate.

“The Wikimedia Foundation calls on the member states of WIPO to reconsider our application for observer status and encourages other UN member states to voice their support for civil society inclusion and international cooperation,” said Keton.

The Wikimedia Foundation provides the essential infrastructure for free knowledge and advocates for a world in which every single human being can freely share in the sum of all knowledge.

How’d we do in our strive for operational excellence last month? Read on to find out!

📈   Incidents

4 documented incidents in July, and 2 documented incidents in August. [1] Historically, that's on average for this time of year. [5]

For more about recent incidents see Incident documentation on Wikitech, or Preventive measures in Phabricator.

📊   Trends

Take a look at the workboard and look for tasks that could use your help.
→ https://phabricator.wikimedia.org/tag/wikimedia-production-error/

Summary over recent months:

• ⚠️ July 2019 (4 of 18 tasks left): One task closed.
• ⚠️ August 2019 (1 of 14 tasks left): no change.
• ⚠️ September 2019 (3 of 12 tasks left): Two tasks closed.
• October (6 of 12 tasks left), no change.
• November (3 of 5 tasks left): no change.
• December (3 of 9 tasks left), Two tasks closed.
• January 2020 (5 of 7 tasks lef), no change.
• February (2 of 7 tasks left), Two tasks closed.
• March (2 of 2 tasks left), no change.
• April (10 of 14 tasks left): One task closed.
• May (7 of 14 tasks left): Four tasks closed.
• June (10 of 14 tasks left): Four tasks closed.
• July 2020: 13 of 24 new tasks survived the month of July and remain open today.
• August 2020: 37 of 53 new tasks survived the month of August and remain open today.

Previously, we had 72 open production errors over the recent months up to June. Since then, 16 of those were closed. But, the 13 and 37 errors surviving July and August raise our recent tally to 106.

The workboard overall (including tasks from 2019 and earlier) held 192 open production errors on July 23rd. As of writing, the workboard holds 296 open tasks in total. [4] This +104 increase is largely due to the merged backlog of JavaScript client errors, which were previously untracked. Note that we backdated the majority of these JS errors under “Old”, and thus are not amongst the elevated numbers of July and August.

🎉   Thanks!

Thank you to everyone else who helped by reporting, investigating, or resolving problems in Wikimedia production. Thanks!

Until next time,

– Timo Tijhof

👊🍺 Tyler: “You know man, it could be worse! […]” Narrator: “[but] I was close... to being complete.”

Tyler: “Martha's polishing the brass on the Titanic. It's all going down, man. […] Evolve! Let the chips fall where they may.”
Narrator: “What!?” Tyler: “The things you own..., they end up owning you.”

Footnotes:
[1] Incidents. – https://wikitech.wikimedia.org/wiki/Incident_documentation
[2] Tasks created. – https://phabricator.wikimedia.org/maniphest/query…
[3] Tasks closed. – https://phabricator.wikimedia.org/maniphest/query…
[4] Open tasks. – https://phabricator.wikimedia.org/maniphest/query…
[5] Wikimedia incident stats. – https://codepen.io/Krinkle/full/wbYMZK

Heather Olins is an Assistant Professor of the Practice in the Biology Department and Environmental Studies Program at Boston College. Here she reflects on her student’s experience with Wikipedia editing in an upper level elective course, and adjustments she is making to the project in its second iteration.

I teach a biology elective course called Deep Sea Biology. My primary goal in this course of mostly senior bio majors is to increase awareness of, and enthusiasm for, the deep sea. I don’t particularly care if 5 years after my course students still remember what depth corresponds to the ocean’s abyssopelagic zone, but I hope that students are still thinking about how little we know about the ocean’s depths, that they remember the ways in which their own lives are indirectly affected by Earth’s oceans, and that they consider the impacts on the marine realm that their daily actions have. 

Because of these course goals, I have chosen to forego a traditional final exam, and instead have students take on a final project. I’ve done this in a number of ways, but in the Spring of 2020, I added Wikipedia editing (through Wiki Education) to the course for a few reasons. I wanted something that was an authentic contribution to the world beyond our classroom. When students know that I am the only person reading their paper they feel a different level of engagement and ownership of the project than something that may, ultimately, be publicly accessible. One of the themes I stress over and over again is how poorly understood the deep sea is. We spend a lot of time reading the scientific literature for the course, which is accessible to us through our institutional journal subscriptions, but much of this information is not accessible to the public. Translating and summarizing some of that information for a general audience seemed like a valuable academic exercise, and sharing it on Wikipedia felt like a valuable contribution. I could tell my students that they were going to play a role in making our knowledge about the deep sea more accessible to the general public. I could tell that this responsibility and challenge was something that appealed almost instantly to many of my students. They all use Wikipedia regularly, and have heard mixed messages about whether or not it is a “good” source, so in addition to the contributions they would make, this project would help them better understand how Wikipedia works.

Now that I’ve gone through the Wikipedia editing process once, I’ve learned a few things that will hopefully make things smoother. I want to separate out the research from the editing more than I did in the past. Trying to do both things simultaneously was frustrating for some students. The Wiki Education platform has lots of great tutorials, but having students do these too early (before they’ve selected a topic, for example) left some students feeling like the Wiki Education platform was a lot of busy work… I’m hoping that condensing these tutorials and assigning them right when students need that information will help.

This semester (Fall 2020) I am formatting the project in 3 parts:

1. Background research: Students choose their topic, read primary source documents, and create an annotated bibliography.

2. Wikipedia editing: Students summarize and translate their research into content that can be added to Wikipedia. For some topics this might just be adding some references and citations, for others it may be a page reorganization, and for some it could involve adding substantial text to their topic’s page (or even creating a new page). 

3. Get creative: Students propose a creative way to share what they’ve learned about their topic, that they will submit digitally for peer review before final submission. This semester especially, I wanted students to be excited and energized by their projects so I’m leaving this pretty open. I might end up with children’s books, music videos, art projects… who knows!

The following are some quotes from my students last semester about the Wikipedia editing project.

• With the Wiki Education project, I thought it was a cool way to combine research and teamwork while making an impact by providing updated information to be people browsing Wikipedia. I enjoyed being in groups for the project and would not want to complete the task alone. Some of the training felt tedious but it was also helpful to understand how to actually make edits and utilize the software.
•  I did enjoy the Wikipedia assignment, it was cool to be behind the scenes like that and get to make a contribution!
• I really enjoyed the Wikipedia project. It was interesting to study just one deep sea organism, and although we didn’t get to finish, it helped a lot with scientific writing and research.
• I liked doing the research and editing the article myself, and it made me feel like I was contributing to something bigger than had I just done the research and written a paper just for a professor to read and then nothing came from it.
• I think Wikipedia was a good way to expose those who do not know a lot about the deep sea to the greater community. It showed that people search this ad look through this information.
• I think I’ll probably remember the Wikipedia project — I now have a lot of oddly specific knowledge of an obscure genus of octopus, which should be fun at parties.
• I think the Wikipedia project was actually fun and it was a good experience in finding proper sources and adding to articles.
• I did enjoy the Wikipedia assignment, it was cool to be behind the scenes like that and get to make a contribution!
• I also really enjoyed the Wikipedia project, the team, and that it was something I could feel good about because others benefitted from our efforts.

By Tyler Cipriani, Manager, Release Engineering

There is a lot of Wikimedia code canonically hosted by the Wikimedia Gerrit install. Gerrit is a web-based git repository collaboration tool that allows users to submit, comment on, update, and merge code into its hosted repositories. 

Gerrit’s workflow and user experience are unique when compared to other popular code review systems like GitHub, Bitbucket, and GitLab. Gerrit’s method of integration is focused on continuous integration of stacked patchsets that may be rearranged and merged independently. In Gerrit there is no concept of feature branches where all work on a feature is completed before it’s merged to a mainline branch—the only branch developers need to worry about is the mainline branch. The consequence of this is that each commit is a distinct unit of change that may be merged with the mainline branch at any time. 

The primary unit of change for GitHub and other review systems is the pull request. Thanks to the proliferation of GitHub, pull requests (synonymous with “merge requests”) have become the defacto standard for integration. The type of continuous integration used by Gerrit can allow for more rapid iteration of closely aligned teams but might be hostile to new contributors.

Following an announcement in 2011, in early 2012 Wikimedia moved from Subversion to Git and chose Gerrit as the code review platform. The following summer a consultation resulted in affirming that Wikimedia development was staying on Gerrit “for the time being”. Since 2012, new Open Source tools for git-based code review have continued to evolve. Integrated self-service continuous integration, easy repository creation and browsing, and pull requests are used for development in large Open Source projects and help define user expectations about what a code review should do.

Gerrit’s user interface has improved — particularly with the upgrade from version 2 to version 3 — but Gerrit is still lacking some of the friendly features of many of the modern code review tools like easy feature branch creation, first-class self-service continuous integration, and first-class repository navigation. Meanwhile, the best parts of Gerrit’s code review system — draft comments, approvals, and explicit approvers — have made their way into other review systems. Gerrit’s unique patchset workflow has a lot of advantages over the pull request model, but, maybe, that alone is not a compelling enough reason to avoid alternatives.

Enter GitLab

Earlier this year, as part of the evaluation of continuous integration tooling, the Wikimedia Foundation’s Release Engineering team reviewed GitLab’s MIT-licensed community edition (CE) offering and found that it met many of the needs for our continuous integration system—things like support for self-service pre- and post-merge testing, a useful ACL system for reviewers, multiple CI executors supporting physical hosts and Kubernetes clusters, support for our existing git repositories, and more.

GitLab has been adopted by comparable Open Source entities like Debian, FreeDesktop.org, KDE, Inkscape, Fedora, and the GNOME project.

GitLab is a modern code review system that seems capable of handling our advanced CI workflows. A move to GitLab could provide our contributors with a friendly and open code review experience that respects the principles of freedom and open source.

Feedback

As shepherds of the code review system, the Release Engineering team reached the stage of evaluations where we need to gather feedback on the proposal to move from Gerrit to GitLab. The Wikimedia Gerrit install is used in diverse ways by over 2,500 projects. To reach an equitable decision about whether or not GitLab is the future for our code hosting, we need the feedback of the technical community.

On 2 September 2020, we announced the beginning of the GitLab consultation period. We invite all technical contributors with opinions about code review to speak their mind on the consultation talk page.

From now until the end of September 2020 a working group composed of individuals from across our technical communities will be collecting and responding on the consultation talk page. Following this consultation period, the working group will review the feedback it has received, and it will produce a summary, recommendation, and supporting deliverables.

It’s difficult to make decisions collaboratively, but those decisions are stronger for their efforts. Please take the time to add a topic or add to the discussion — our decision can only be as strong as our participation.

About this post

Featured image credit: Vulpes vulpes Mallnitz 01, Uoaei1, CC BY-SA 4.0

As more and more people enter to the Malayalam digital world, the issue of not having any formal training for Malayalam computing is more observed. People sometimes just search for input methods in web, ask friends or use whatever coming with the devices they have. Since I myself is an author of two input methods, sometimes people ask me too. This essay is about the characteristics of a good input method to help people make the right choice.