The Cloud Services team is in the process of updating and standardizing the use of DNS names throughout Cloud VPS projects and infrastructure, including the Toolforge project. A lot of this has to do with reducing our reliance on the badly-overloaded term 'Labs' in favor of the 'Cloud' naming scheme. The whole story can be found on this Wikitech proposal page. These changes will be trickling out over the coming weeks or months, but one change you might notice already.

New private domain for VPS instances

For several years virtual machines have been created with two internal DNS entries: <hostname>.eqiad.wmflabs and <hostname>.<project>.eqiad.wmflabs. As of today, hosts can also be found in a third place: <hostname>.<project>.eqiad1.wikimedia.cloud. There's no current timeline to phase out the old names, but the new names are now the preferred/official internal names. Reverse DNS lookups on a instance's IP address will return the new name, and many other internal cloud services (for example Puppet) will start using the new names for newly-created VMs.

Eventually the non-standard .wmflabs top level domain will be phased out, so you should start retraining your fingers and updating your .ssh/config today.

Last week, we completed a piece of long-neglected work relating to Puppet, the tool that manages the configuration of every virtual machine in our cloud. Historically, each VM has received its configuration from a physical, production server (the 'puppetmaster'). This meant that there was a constant chatter of traffic back and forth between each VM and unrelated networks and hardware sitting in Wikimedia production. Now, the puppetmasters are located on VMs, so all of that chatter is internal to Cloud Services.

Generally, we like to think of the cloud as an isolated sandbox, a safe place for volunteering and experimentation. Any tight links between the cloud and production require extra vigilance; as we sever those links we can worry a bit less about issues (security and otherwise) bleeding back and forth between the cloud and the public wikis.

A notable thing about this move is that nearly all the work was done by a technical volunteer, @Krenair. Krenair updated the code that runs the in-cloud puppetmasters, built out the server cluster, and designed the migration flow that transferred control over from the old controllers. It was his hard work (done on top of his unrelated day job) that moved this task from long-neglected to a box with a check mark.

Quite a few Cloud Services projects are partially (or, in some cases, completely) maintained and managed by technical volunteers. Not only does this allow us to run infrastructure well beyond the capacity of our small team, it's also a clear success in the mission of the Technical Engagement team (of which Cloud Services is a part). We work to build technical capacity in the community, and when volunteers start doing our job for us, we know we've succeeded. Almost all levels of access are available to trusted volunteers, and getting permission to hack on the WMCS infrastructure is not as hard as you might think. Come and join us![4]

A couple of week ago we finally moved the last lingering VMs in our OpenStack platform from the nova-network region to the Neutron region (Blog Post: Neutron is here!). The bulk of the work had been done a month earlier, so the final nails in nova-network's coffin felt a bit anticlimactic -- nevertheless, this is a big step that represents a huge amount of work on the part of both staff and volunteers.

VPS and Toolforge users have been extraordinarily patient with all of the downtime, deprecations, and hand-tuning that this change required. The lack of grumbles did not go unnoticed -- it's great to work with such a supportive group of users. Thank you!

Our reliance on the long-obsolete nova-network service was blocking many, many things. Now that it's gone, we can...

• Officially discontinue support for Ubuntu Trusty throughout the WMF's infrastructure. (some of the last Trusty servers were supporting nova-network)
• Shut down and decommission half a dozen long-obsolete servers.
• Rip out hundreds of lines of special-case puppet code that supported the old nova-network OpenStack region.
• Upgrade our OpenStack Nova services past version Mitaka. (Mitaka was the last release that supported nova-network; there have been SIX upstream releases since Mitaka)
• Upgrade Horizon.wikimedia.org to run newer versions with fancier panels. (Previously we were stuck running the last version that supported nova-network)
• Switch to a less buggy Designate/DNS model that should result in fewer VM creation failures and fewer DNS record leaks.
• Possibly spend some time on other WMCS features that aren't 100% dedicated to repaying technical debt.

Most of those changes will be invisible to the end-users, but will make for much happier operations staff!

Ubuntu Trusty was released in April 2014, and support for it (including security updates) will cease in April 2019. We need to shut down all Trusty hosts before the end of support date to ensure that Toolforge remains a secure platform. This migration will take several months because many people still use the Trusty hosts and our users are working on tools in their spare time.

Initial timeline

Subject to change, see Wikitech for living timeline.

• 2019-01-11: Availability of Debian Stretch grid announced to community
• Week of 2019-02-04: Weekly reminders via email to tool maintainers for tools still running on Trusty
• Week of 2019-03-04:
  • Daily reminders via email to tool maintainers for tools still running on Trusty
  • Switch login.tools.wmflabs.org to point to Stretch bastion
• Week of 2019-03-18: Evaluate migration status and formulate plan for final shutdown of Trusty grid
• Week of 2019-03-25: Shutdown Trusty grid

What is changing?

• New job grid running Son of Grid Engine on Debian Stretch instances
• New limits on concurrent job execution and job submission by a single tool
• New bastion hosts running Debian Stretch with connectivity to the new job grid
• New versions of PHP, Python2, Python3, and other language runtimes
• New versions of various support libraries

What should I do?

Some of you will remember the Ubuntu Precise deprecation from 2016-2017. This time the process is similar, but slightly different. We were unable to build a single grid engine cluster that mixed both the old Trusty hosts and the new Debian Stretch hosts. That means that moving your jobs from one grid to the the other is a bit more complicated than it was the last time.

The cloud-services-team has created the News/Toolforge Trusty deprecation page on wikitech.wikimedia.org to document basic steps needed to move webservices, cron jobs, and continuous jobs from the old Trusty grid to the new Stretch grid. That page also provides more details on the language runtime and library version changes and will provide answers to common problems people encounter as we find them. If the answer to your problem isn't on the wiki, ask for help in the #wikimedia-cloud IRC channel or file a bug in Phabricator.

See also

• News/Toolforge Trusty deprecation on Wikitech for full details including links to tools that will help us monitor the migration of jobs to the new grid and help with common problems

Starting 2019-01-03, GET and HEAD requests to http://tools.wmflabs.org will receive a 301 redirect to https://tools.wmflabs.org. This change should be transparent to most visitors. Some webservices may need to be updated to use explicit https:// or protocol relative URLs for stylesheets, images, JavaScript, and other content that is rendered as part of the pages they serve to their visitors.

Three and a half years ago @yuvipanda created T102367: Migrate tools.wmflabs.org to https only (and set HSTS) about making this change. Fifteen months ago a change was made to the 'admin' tool that serves the landing page for tools.wmflabs.org so that it performs an http to https redirect and sets a Strict-Transport-Security: max-age:86400 header in its response. This header instructs modern web browsers to remember to use https instead of http when talking to tools.wmflabs.org for the next 24 hours. Since that change there have been no known reports of tools breaking.

The new step we are taking now is to make this same redirect and set the same header for all visits to tools.wmflabs.org where it is safe to redirect the visitor. As mentioned in the lead paragraph, there may be some tools that this will break due to the use of hard coded http://... URLs in the pages they serve. Because of the HSTS header covering tools.wmflabs.org, this breakage should be limited to resources that are loaded from external domains.

Fixing tools should be relatively simple. Hardcoded URLs can be updated to be either protocol relative (http://example.org ➜ //example.org) or explicitly use the https protocol (http://example.org ➜ https://example.org). The proxy server also sends an X-Forwarded-Proto: https header to the tool's webservice which can be detected and used to switch to generating https links. Many common web application frameworks have support for this already:

• Django's SECURE_PROXY_SSL_HEADER setting
• Flask's ProxyFix middleware
• Symfony's Request::setTrustedProxies

If you need some help figuring out how to fix your own tool's output, or to report a tool that needs to be updated, join us in the #wikimedia-cloud IRC channel.

As promised in an earlier post (Blog Post: Neutron is (finally) coming), we've started moving a few projects on our Cloud-VPS service into a new OpenStack region that is using Neutron for its software-defined networking layer. It's going pretty well! The new region, 'eqiad1', is currently very small, and growth is currently blocked by hardware issues (see T199125 for details) but we hope to resolve that issue soon.

Once we have some more hardware allocated to the eqiad1 region we will start migrating projects in earnest. Here's what that will look like for each project as it is migrated:

1. A warning email about impending migrations will be sent to the cloud-announce mailing list at least 7 days before migration.
2. On the day of the migration: Instance creation for each migrating project will be disabled in the legacy 'eqiad' region. This means that Horizon will still show instances in eqiad, but creation of new instances will be disabled there.
3. The current project quotas will be copied over from eqiad to eqiad1.
4. Security groups will be copied from eqiad to eqiad1, and some rules (those that refer to 10.0.0.0/8 or 'all VMs everywhere') will be duplicated to include the new IP range in eqiad1.
5. Then, the following will happen to each instance:
   1. The instance will be shut down
   2. A new shadow instance will be created in eqiad1 with the same name but a new IP address or addresses.
   3. The contents of the eqiad instance will be copied into the new instance. This step could take several hours, depending on the size of the instance.
   4. Any DNS records or proxies that pointed to the old instance will be updated to point at the new instance.
   5. The new instance will be started up, and then rebooted once for good measure.
   6. Once the new instance is confirmed up and reachable, the old instance will be deleted.
6. You will want to check some things afterwards. In particular:
   1. Verify that any external-facing services supported by your project are still working. If they need to be started, start them. If something drastic is happening, notify WMCS staff on IRC (#wikimedia-cloud)
   2. In some cases you may need to restart services if they're unable to restart themselves after a system reboot. For example, Wikimedia-Vagrant seems to usually have this problem.

If you would like an early jump on migration, we have space to move a few projects now. In particular, if you would like access to the eqiad1 region so that you can start building out new servers there, please open a quota request here: https://phabricator.wikimedia.org/project/profile/2880/

The migration process for Toolforge will be utterly different -- in the meantime people who only use Toolforge can disregard all of this for the time being.

The History

When Wikimedia Labs (the umbrella-project now known as 'Cloud VPS') first opened to the public in 2012 it was built around OpenStack Nova version 'Diablo'.[1] Nova included a simple network component ("nova-network") which works pretty well -- it assigns addresses to new VMs, creates network bridges so that they can talk to the outside internet, and manages dynamic firewalls that control which VMs can talk to each other and how.

Just as we were settling into nova-network (along with other early OpenStack adopters), the core developers were already moving on. A new project (originally named 'Quantum' but eventually renamed 'Neutron') would provide stand-alone APIs, independent from the Nova APIs, to construct all manners of software-defined networks. With every release Neutron became more elaborate and more reliable, and became the standard for networking in new OpenStack clouds.

For early adopters like us, there was a problem. The long-promised migration path for existing nova-network users never materialized, and nova-network got stuck in a kind of support limbo: in version after version it was announced that it would be deprecated in the next release, but nova-network users always pushed back to delay the deprecation until an upgrade path was ready. Finally, in late 2016 nova-network was finally dropped from support, but still with no well-agreed-on upgrade path.

So, after years of foot-dragging, we need to migrate (T167293) our network layer to Neutron. It's going to be painful!

The Plan

Since there is not an in-place upgrade path, Chase and Arturo have built a new, parallel nova region using Neutron that is named 'eqiad1-r'. It shares the same identity, image, and DNS service as the existing region, but instances in the eqiad1-r region live on different hosts and are in a different VLAN with different IPs. I (Andrew) will be pulling projects, one at a time, out of the existing 'eqiad' region and copying everything into 'eqiad1-r'. Each instance will be shut down in eqiad, copied to eqiad1-r, and started up again. The main disruption here is that once moved, the new VMs will have a new IP address and will probably be unable to communicate with VMs in the old region; for this reason, project migration will mean substantial, multi-hour downtime for the entire VPS project.

Here are a few things that will be disrupted by IP reassignment:

• Internal instance DNS (e.g. <instance>.<project>.eqiad.wmflabs)

• External floating-IP DNS (e.g. <website>.<project>.wmflabs.org)

• Dynamic web proxies (e.g. http://<website>.wmflabs.org>)

• Nova security group rules

• Anything at all internal to a project that refers to another instance by IP address

I'm in the process of writing scripted transformations for almost all of the above. Ideally when a VM moves, the DNS and proxy entries will be updated automatically so that all user-facing services will resume as before the migration. The one thing I cannot fix is literal IP references within a project; if you have any of those, now would be a good time to replace those with DNS lookups, or at the very least, brace yourself for a lot of hurried clean-up.

Once we've run through a few trial migrations, I'll start scheduling upgrade windows and coordinating with project admins. We'll probably migrate Toolforge last -- there are even more issues involved with that move which I won't describe here.

This is another technical-debt/cleanup exercise that doesn't really get us anything new in the short-run. Moving to Neutron clears the path for an eventual adoption of IPv6, and Neutron has the potential to support new, isolated testing environments with custom network setups. Most importantly however, this will get us back on track with OpenStack upgrades so that we can keep getting upstream security fixes and new features. Expect to hear more about those upgrades once the dust settles from this migration.

The Timeline

Honestly, I don't know what the timeline is yet. There are several projects that are wholly managed by Cloud Services or WMF staff, and those projects will be used as the initial test subjects. Once we have an idea of how well this works and how long it takes, we'll start scheduling other projects for migration in batches. Keep an eye on the cloud-announce mailing list for related announcements.

How you can help

• Fix any literal IP references within your project(s). Replace them with DNS lookups. If they can't be replaced with lookups, make a list of everywhere that they appear and get ready to edit all those places come migration day

• Delete VMs that you aren't using. Release floating IPs that you aren't using. Delete Proxies that aren't doing anything. The fewer things there are to migrate, the easier this will be.

• Subscribe to https://lists.wikimedia.org/mailman/listinfo/cloud-announce and actually read the postings. Project admins that don't read their emails may find their projects migrated by surprise.

Thank you!

[1] OpenStack releases are alphabetical, two per year. The current development version is Rocky (released late 2018); WMCS is currently running version Mitaka (released early 2016) and Neutron was first released as part of version Folsom (late 2012). So this has been a long time coming.

Between 2017-11-20 and 2017-12-01, the Wikimedia Foundation ran a direct response user survey of registered Toolforge users. 141 email recipients participated in the survey which represents 11% of those who were contacted.

Demographic questions

Based on responses to demographic questions, the average [1] respondent:

• Has used Toolforge for 1-3 years
• Developed 1-3 tools & actively maintains 1-2 tools
• Spends an hour or less a week maintaining their tools
• Programs using Python and/or PHP
• Does 80% or more of their development work locally
• Uses source control
• Was not a developer or maintainer on Toolserver

[1]: "Average" here means a range of responses covering 50% or more of responses to the question. This summarization is coarse, but useful as a broad generalization. Detailed demographic response data is available on wiki.

Qualitative questions

• 90% agree that services have high reliability (up time). Up from 87% last year.
• 78% agree that it is easy to write code and have it running on Toolforge. Up from 71% last year.
• 59% agree that they feel they are supported by the Toolforge team when they contact them via cloud mailing list, #wikimedia-cloud IRC channel, or Phabricator. This is down dramatically from 71% last year, but interestingly this question was left unanswered by 36% of respondents.
• 59% agree that they receive useful information via cloud-announce / cloud mailing lists. Up from 46% last year.
• 52% agree that documentation is easy to find. This is up from 46% last year and the first time crossing the 50% point. We still have a long way to go here though!
• 96% find the support they receive when using Toolforge as good or better than the support they received when using Toolserver. Up from 89% last year.
• 50% agree that Toolforge documentation is comprehensive. No change from last year.
• 53% agree that Toolforge documentation is clear. Up from 48% last year.

Free form responses

The survey included several free form response sections. Survey participants were told that we would only publicly share their responses or survey results in aggregate or anonymized form. The free form responses include comments broadly falling into these categories:

• Documentation (58 comments)
• Platform (48 comments)
• Workflow (48 comments)
• Community (17 comments)
• Support (6 comments)

Documentation

Comments on documentation included both positive recognition of work that has been done to improve our docs and areas that are still in need of additional work. Areas with multiple mentions include need for increased discoverability of current information, better getting started information, and more in depth coverage of topics such as wiki replica usage, Kubernetes, and job grid usage.

There were also comments asking for a self-service version control system and pre-installed pywikibot software. Both of these are offered currently in Toolforge, so these comments were classified as missing or difficult to find documentation.

Platform

Comments about the Toolforge platform have been subcategorized as follows:

• Software (26 comments)
  • The majority of software comments were related to a desire for newer language runtime versions (PHP, Java, nodejs, Python) and more flexibility in the Kubernetes environment.
• Database (10 comments)
  • Database comments include praise for the new Wiki Replica servers and multiple requests for a return of user managed tables colocated with the replica databases.
• Reliability (10 comments)
  • Reliability comments included praise for good uptime, complaints of poor uptime, and requests to improve limits on shared bastion systems.
• Hardware (2 comments)
  • (sample too small to summarize)

Workflow

• Deploy (12 comments)
  • The major theme here was automation for software deployment including requests for full continuous delivery pipelines.
• Debugging (10 comments)
  • People asked for better debugging tools and a way to create a more full featured local development environment.
• Monitoring (10 comments)
  • Monitoring comments included a desire for alerting based on tracked metrics and tracking of (more) metrics for each tool.
• Setup (10 comments)
  • Comments included praise for https://toolsadmin.wikimedia.org/ improvements and other work done in last year.
• Files (6 comments)
  • Improved workflows for remote editing and file transfer are desired.

Community

Comments classified as community related broadly called for more collaboration between tool maintainers and better adherence to practices that make accessing source code and reporting bugs easier.

Support

Support related comments praised current efforts, but also pointed to confusion about where to ask questions (irc, email, phabricator).

The Foundation fiscal quarter running from January 2018 through March 2018 was a busy one for the Cloud Services team:

@Harej joined us as an Associate Product Manager on a six month contract. James is working on defining a product roadmap for "Toolhub". This project is an effort inspired by T115650: Create an authoritative and well promoted catalog of Wikimedia tools. You can follow his progress via the Toolhub Phabricator project and the project's pages on Meta.

@Bstorm was hired as an Operations Engineer. She joined us just before the Foundation's annual all hands event and got a crash course in the names and faces of about 300 co-workers. Brooke has a lot of prior experience in both systems administration and software development, and is coming up to speed quickly with the Cloud Services environment. Her recent projects include improvements to the Toolforge CDNJS mirror and enhancements for the automation tools we use to update the Wiki Replicas indexes and views.

Our third new team member in January was @Chicocvenancio. A long time Wikimedian and Toolforge user, Chico is working as a Technical Support contractor. If you stop by the #wikimedia-cloud irc channel and ask for !help Chico is likely to be one of the folks who tries to help you out.

@srodlund also officially became part of the team as a technical writer. Sarah had been working with us on an ad hoc basis for months, but in January we came to an agreement for her to spend 50% of her paid Foundation time working on technical writing projects. Sarah has many years of experience as both a technical writer and as a writing instructor. We are excited to have her leading our efforts to create a community of technical writing contributors for the many Wikimedia projects.

The team was also busy working on the 'normal' projects which, when things go well, are seldom noticed. Wikitech, Horizon, and the Toolforge admin console have been moved to new physical servers thanks to @Andrew. You can read more about the details in Running red-queen-style. @aborrero has been working on making software security updates easier to manage. @chasemp is progressing on our OpenStack Neutron migration project. The public facing parts of Dumps have been moved to new servers thanks to a collaboration by @madhuvishy and @ArielGlenn.

In Toolforge news, long time volunteer @zhuyifei1999 was granted Toolforge administrator rights. YiFei has been providing great technical support advice to our community and code contributions for Toolforge and related services for many months. The adminship gives him greater abilities to troubleshoot and correct problems for Toolforge tool maintainers.

I've spent the last few months building new web servers to support some of the basic WMCS web services: Wikitech, Horizon, and Toolsadmin. The new Wikitech service is already up and running; on Wednesday I hope to flip the last switch and move all public Horizon and Toolsadmin traffic to the new servers as well.

If everything goes as planned, users will barely notice this change at all.

This is a lot of what our team does -- running as fast as we can just to stay in place. Software doesn't last forever -- it takes a lot of effort just to hold things together. Here are some of the problems that this rebuild is solving:

• T186288: Operating System obsolescence. Years ago, the Wikimedia Foundation Operations team resolved to move all of our infrastructure from Ubuntu to Debian Linux. Ubuntu Trusty will stop receiving security upgrades in about a year, so we have to stop using it by then. All three services (Wikitech, Horizon, Toolsadmin) were running on Ubuntu servers; Wikitech was the last of the Foundation's MediaWiki hosts to run on Ubuntu, so its upgrade should allow for all kinds of special cases to be ignored in the future.

• T98813: Keeping up with PHP and HHVM. In addition to being the last wiki on Trusty, Wikitech was also the last wiki on PHP 5. Every other wiki is using HHVM and, with the death of the old Wikitech, we can finally stop supporting PHP 5 internally. Better yet, this plays a part in unblocking the entire MediaWiki ecosystem (T172165) as newer versions of MediaWiki standardize on HHVM or PHP 7.

• T168559: Escaping failing hardware. The old Wikitech site was hosted on a machine named 'Silver'. Hardware wears out, and Silver is pretty old. The last few times I've rebooted it, it's required a bit of nudging to bring it back up. If it powered down today, it would probably come back, but it might not. As of today's switchover, that scenario won't result in weeks of Wikitech downtime.

• T169099: Tracking OpenStack upgrades. OpenStack (the software project that includes Horizon and most of our virtual machine infrastructure) releases a new version every six months. Ubuntu packages up every version with all of its dependencies, and provides a clear upgrade path between versions. Debian, for the most part, does not. The new release of Horizon is no longer deployed through an upstream package at all, but instead is a pure Python deploy starting with the raw Horizon source and requirements list, rolled into Wheels and deployed into an isolated virtual environment. It's unclear exactly how we'll transition our other OpenStack components away from Ubuntu, but this Horizon deploy provides a potential model for deploying any OpenStack project, any version, on any OS. Having done this I'm much less worried about our reliance on often-fickle upstream packagers.

• T187506: High availability. The old versions of these web services were hosted on single servers. Any maintenance or hardware downtime meant that the websites were gone for the duration. Now we have a pair of servers with a shared cache, behind a load-balancer. If either of the servers dies (or, more likely, we need to reboot one for kernel updates) the website will remain up and responsive.

Of course, having just moved wikitech to HHVM, the main Wikimedia cluster is being upgraded from HHVM to PHP 7, and Wikitech will soon follow suit. The websites look the same, but the race never ends.

Long ago, the Wikimedia Operations team made the decision to phase out use of Ubuntu servers in favor of Debian. It's a long, slow process that is still ongoing, but in production Trusty is running on an ever-shrinking minority of our servers.

As Trusty becomes more of an odd duck in production, it grows harder to support in Cloud Services as well. Right now we have no planned timeline for phasing out Trusty instances (there are 238 of them!) but in anticipation of that phase-out we've now disabled creation of new Trusty VMs.

This is an extremely minor technical change (the base image is still there, just marked as 'private' in OpenStack Glance). Existing Trusty VMs are unaffected by this change, as are present ToolForge workflows.

Even though any new Trusty images represent additional technical debt, The WMCS team anticipates that there will still be occasional, niche requirements for Trusty (for example when testing behavior of those few remaining production Trusty instances, or to support software that's not yet packaged on Debian). These requests will be handled via phabricator requests and a bit of commandline magic.

One of our quarterly goals was "Define a metric to track OpenStack system availability". Despite the weak phrasing, we elected to not only pick something to measure but also to actually measure it.

I originally proposed this goal based on the notion that VPS creation seems to break pretty often, but that I have no idea how often, or for how long. The good news is that several months ago Chase wrote a 'fullstack' testing tool that creates a VM, checks to see if comes up, makes sure that DNS and puppet work, and finally deletes the new VM. That tool is now running in an (ideally) uninterrupted loop, reporting successes and failures to graphite so that we can gather up long-term statistics about when things are working.

In addition to the fullstack test, I wrote some Prometheus tests that check whether or not individual public OpenStack APIs are responding to requests. When these services go down the fullstack test is also likely to break, but other things are affected as well: Horizon, the openstack-browser, and potentially various internal Cloud Services things like DNS updates.

All of these new stats can now be viewed on the WMCS API uptimes dashboard. The information there isn't very detailed but should be useful to the WMCS staff as we work to improve stability, and should be useful to our users when they want to answer the question "Is this broken for everyone or just for me?"

24% of Wikipedia edits over a three month period in 2016 were completed by software hosted in Cloud Services projects. In the same time period, 3.8 billion Action API requests were made from Cloud Services. We are the newly formed Cloud Services team at the Foundation, which maintains a stable and efficient public cloud hosting platform for technical projects relevant to the Wikimedia movement. -- https://blog.wikimedia.org/2017/09/11/introducing-wikimedia-cloud-services/

With a lot of help from @MelodyKramer and the Diff-blog team, we have published a blog post on the main Wikimedia blog. The post talks a bit about why we formed the Wikimedia Cloud Services team and what the purpose of the product rebranding we have been working on is. It also gives a shout out to a very small number of the Toolforge tools and Cloud VPS projects that the Wikimedia technical community make. I wish I could have named them all, but there are just too many!

TL;DR

• Change your tools and scripts to use:
  • *.web.db.svc.eqiad.wmflabs (real-time response needed)
  • *.analytics.db.svc.eqiad.wmflabs (batch jobs; long queries)
• Replace * with either a shard name (e.g. s1) or a wikidb name (e.g. enwiki).
• The new servers do not support user created databases/tables because replication can't be guaranteed. See T156869 and below for more information. tools.db.svc.eqiad.wmflabs (also known as tools.labsdb) will continue to support user created databases and tables.
• Report any bugs you find with these servers in Phabricator using the Data-Services tag.

Wiki Replicas

The Wiki Replicas are one of the unique services that Wikimedia Cloud Services helps make available to our communities. Wiki Replicas are real-time replicas of the production Wikimedia MediaWiki wiki databases with privacy-sensitive data removed. These databases hold copies of all the metadata about content and interactions on the wikis. You can read more about these databases on Wikitech if you are unfamiliar with their details.

The current physical servers for the <wiki>_p Wiki Replica databases are at the end of their useful life. Work started over a year ago on a project involving the DBA team and cloud-services-team to replace these aging servers (T140788). Besides being five years old, the current servers have other issues that the DBA team took this opportunity to fix:

• Data drift from production (T138967)
• No way to give different levels of service for realtime applications vs analytics queries
• No automatic failover to another server when one failed

Bigger, faster, more available

Each of the three new servers is much larger and faster than the servers they are replacing. Five years is a very long time in the world of computer hardware:

• 2 x 4 core processors (16 virtual cores total)
• 512 GB of RAM (2.5 times the old amount)
• 12 TB of SSD disks (vs 5 TB of fixed disks previously)

We have also upgraded the database software itself. The new servers are running MariaDB version 10.1. Among other improvements, this newer database software allows us to use a permissions system that is simpler and more secure for managing the large number of individual tools that are granted access to the databases.

The new servers use InnoDB tables rather than the previous TokuDB storage. TokuDB was used on the old servers as a space-saving measure, but it has also had bugs in the past that caused delays to replication. InnoDB is used widely in the Wikimedia production databases without these problems.

The new cluster is configured with automatic load balancing and failover using HAProxy. All three hosts have identical data. Currently, two of the hosts are actively accepting connections and processing queries. The third is a ready replacement for either of the others in case of unexpected failure or when we need to do maintenance on the servers themselves. As we learn more about usage and utilization on these new hosts we can change things to better support the workloads that are actually being generated. This may include setting up different query duration limits or pooling the third server to support some of the load. The main point is that the new system provides us with the ability to make these types of changes which were not possible previously.

Improved replication

The work of scrubbing private data is done on a set of servers that we call "sanitarium" hosts. The sanitarium servers receive data from the production primary servers. They then in turn act as the primary servers which are replicated to the Wiki Replica cluster. The two sanitarium servers for the new Wiki Replica cluster use row-based replication (RBR). @Marostegui explains the importance of this change and its relationship to T138967: Labs database replica drift:

[W]e are ensuring that whatever comes to those hosts (which are, in some cases, normal production slaves) is exactly what is being replicated to the [Cloud] servers. Preventing us from data drifts, as any data drift on row based replication would break replication on the [Cloud] servers. Which is bad, because they get replication broken, but at the same time is good, because it is a heads up that the data isn't exactly as we have it in core. Which allows us to maintain a sanitized and healthy dataset, avoiding all the issues we have had in the past.

The data replicated to the new servers has been completely rebuilt from scratch using the RBR method. This has fixed many replication drift problems that exist on the older servers (T138967). If your tool performs tasks where data accuracy is important (counting edits, checking if a page has been deleted, etc), you should switch to using the new servers as soon as possible.

New service names

Populating the new sanitarium servers with data was a long process (T153743), but now that it is done our three new Wiki Replica servers are ready for use. With the old setup, we asked people to use a unique hostname with each database they connected to (e.g. enwiki.labsdb). The new cluster extends this by adding using service names to separate usage by the type of queries that are being run:

• Use *.web.db.svc.eqiad.wmflabs for webservices and other tools that need to make small queries and get responses quickly.
• Use *.analytics.db.svc.eqiad.wmflabs for longer running queries that can be slower.

If you were using enwiki.labsdb you should switch to either enwiki.analytics.db.svc.eqiad.wmflabs or enwiki.web.db.svc.eqiad.wmflabs. The choice of "analytics" or "web" depends on what your tool is doing, but a good rule of thumb is that any query that routinely takes more than 10 seconds to run should probably use the "analytics" service.

Right now there is no actual difference between connecting to the "web" or "analytics" service names. As these servers get more usage and we understand the limitations they have this may change. Having a way for a user to explicitly choose between real-time responses and slower responses for more complicated queries will provide more flexibility in tuning the systems. We expect to be able to allow queries to run for a much longer time on the new analytics service names than we can on the current servers. This in turn should help people who have been struggling to gather the data needed for complex reports within the current per-request timeout limits.

A breaking change

These new servers will not allow users to create their own databases/tables co-located with the replicated content. This was a feature of the older database servers that some tools used to improve performance by making intermediate tables that could then be JOINed to other tables to produce certain results.

We looked for solutions that would allow us to replicate user created data across the three servers, but we could not come up with a solution that would guarantee success. The user created tables on the current servers are not backed up or replicated and have always carried the disclaimer that these tables may disappear at any time. With the improvements in our ability to fail over and rebalance traffic under load, it is more likely on the new cluster that these tables would randomly appear and disappear from the point of view of a given user. This kind of disruption will break tools if we allow it. It seems a safer solution for everyone to disallow the former functionality.

User created databases and tables are still supported on the tools.db.svc.eqiad.wmflabs server (also known as tools.labsdb). If you are using tables co-located on the current c1.labsdb or c3.labsdb hosts we are recommending that your tool/scripts be updated to instead keep all user managed data on tools.db.svc.eqiad.wmflabs and perform any joining of replica data and user created data in application space rather than with cross-database joins.

There will be further announcements before the old servers are completely taken offline, but tools maintainers are urged to make changes as soon as they can. The hardware for the older servers is very old and may fail in a non-recoverable way unexpectedly (T126942).

Curated datasets

There are some datasets produced by ORES, the Analytics team, or volunteers that really do need to be co-located with the wiki tables to be useful. We are looking for a solution for these datasets that will allow them to be replicated properly and available everywhere. See T173511: Implement technical details and process for "datasets_p" on wikireplica hosts for further discussion of providing some method for 'curated' datasets to be added to the new cluster.

Quarry will be switched over to use *.analytics.db.svc.eqiad.wmflabs soon. As noted previously, using the analytics service names should allow more complex queries to complete which will be a big benefit for Quarry's users who are doing analytics work. This change may however temporarily interrupt usage of some datasets that are blocked by T173511. Follow that task for more information if your work is affected.

You can help test the new servers

Before we make the new servers the default for everyone, we would like some early adopters to use them and help us find issues like:

• missing permissions
• missing views
• missing wikidb service names

To use them, change your tool to connect to:

• *.web.db.svc.eqiad.wmflabs (real-time response needed)
• *.analytics.db.svc.eqiad.wmflabs (batch jobs; long queries)

Replace the * with either a shard name (e.g. s1) or a wikidb name (e.g. enwiki).

For interactive queries, use one of:

• sql --cluster analytics <database_name>
• mysql --defaults-file=$HOME/replica.my.cnf -h <wikidb>.analytics.db.svc.eqiad.wmflabs <database_name>

Report any bugs you find with these servers and their data in Phabricator using the Data-Services tag.

Thanks

The cloud-services-team would like to especially thank @jcrespo and @Marostegui for the work that they put into designing and implementing this new cluster. Without their technical experience and time this project would never have been successful.

Learn more

• Overview of LabsDBs by @jcrespo at 2017 Wikimedia Developer Summit (video, slides)
• Data Services portal on Wikitech
• T140788: Labs databases rearchitecture (tracking)

Toolsadmin.wikimedia.org is a management interface for Toolforge users. On 2017-08-24, a new major update to the application was deployed which added support for creating new tool accounts and managing metadata associated with all tool accounts.

Under the older Wikitech based tool creation process, a tool maintainer sees this interface:

As @yuvipanda noted in T128158, this interface is rather confusing. What is a "service group?" I thought I just clicked a link that said "Create a new Tool." What are the constrains of this name and where will it be used?

With the new process on toolsadmin, the initial form includes more explanation and collects additional data:

The form labels are more consistent. Some explanation is given for how the tool's name will be used and a link is provided to additional documentation on wikitech. More information is also collected that will be used to help others understand the purpose of the tool. This information is displayed on the tool's public description page in toolsadmin:

After a tool has been created, additional information can also be supplied. This information is a superset of the data needed for the toolinfo.json standard used by Hay's Directory. All tools documented using toolsadmin are automatically published to Hay's Directory. Some of this information can also be edited collaboratively by others. A tool can also have multiple toolinfo.json entries to support tools where a suite of functionality is published under a single tool account.

The Striker project tracks bugs and feature ideas for toolsadmin. The application is written in Python3 using the Django framework. Like all Wikimedia software projects, Striker is FLOSS software and community contributions are welcome. See the project's page on wikitech for more information about contributing to the project.

Back in year zero of Wikimedia Labs, shockingly many services were confined to a single box. A server named 'virt0' hosted the Wikitech website, Keystone, Glance, Ldap, Rabbitmq, ran a puppetmaster, and did a bunch of other things.

Even after the move from the Tampa data center to Ashburn, the model remained much the same, with a whole lot of different services crowded onto a single, overworked box. Since then we've been gradually splitting out important services onto their own systems -- it takes up a bit more rack space but has made debugging and management much more straightforward.

Today I've put the final finishing touches on one of the biggest break-away services to date: The puppetmaster that manages most cloud instances is no longer running on 'labcontrol1001'; instead the puppetmaster has its own two-server cluster which does puppet and nothing else. VMs have been using the new puppetmasters for a few weeks, but I've just now finally shut down the old service on labcontrol1001 and cleaned things up.

With luck, this new setup will gain us some or all of the following advantages:

• fewer bad interactions between puppet and other cloud services: In particular, RabbitMQ (which manages most communication between openstack services) runs on labcontrol1001 and is very hungry for resources -- we're hoping it will be happier not competing with the puppetmaster for RAM.

• improved puppetmaster scalability: The new puppetmaster has a simple load-balancer that allows puppet compilations to be farmed out to additional backends when needed.

• less custom code: The new puppetmasters are managed with the same puppet classes that are used elsewhere in Wikimedia production.

Of course, many instances weren't using the puppetmaster on labcontrol1001 anyway; they use separate custom puppetmasters that run directly on cloud instances. In many ways this is better -- certainly the security model is simpler. It's likely that at some point we'll move ALL puppet hosting off of metal servers and into the cloud, at which point there will be yet another giant puppet migration. This last one went pretty well, though, so I'm much less worried about that move than I was before; and in the meantime we have a nice stable setup to keep things going.

Tool owners want to create accessible and pleasing tools. The choice of fonts has previously been difficult, because directly accessing Google's large collection of open source and freely licensed fonts required sharing personally identifiable information (PII) such as IPs, referrer headers, etc with a third-party (Google). Embedding external resources (fonts, css, javascript, images, etc) from any third-party into webpages hosted on Toolforge or other Cloud VPS projects causes a potential conflict with the Wikimedia Privacy Policy. Web browsers will attempt to load the resources automatically and this will in turn expose the user's IP address, User-Agent, and other information that is by default included in an HTTP request to the third-party. This sharing of data with a third-party is a violation of the default Privacy Policy. With explict consent Toolforge and Cloud VPS projects can collect and share some information, but it is difficult to secure that consent with respect to embedded resources.

One way to avoid embedding third-party resources is for each Tool or Cloud VPS project to store a local copy of the resource and serve it directly to the visiting user. This works well from a technical point of view, but can be a maintenance burden for the application developer. It also defeats some of the benefits of using a content distribution network (CDN) like Google fonts where commonly used resources from many applications can share a single locally cached resource in the local web browser.

Since April 2015, Toolforge has provided a mirror of the popular cdnjs library collection to help Toolforge and Cloud VPS developers avoid embedding javascript resources. We did not have a similar solution for the popular Google Fonts CDN however. To resolve this, we first checked if the font files are available via bulk download anywhere, sort of like cdnjs, but they were not. Instead, @zhuyifei1999 and @bd808 have created a reverse proxy and forked a font-searching interface to simplify finding the altered font CSS URLs. You can use these features to find and use over 800 font families.

You can use these assets in your tools now!

fontcdn

• https://tools.wmflabs.org/fontcdn/ - provides the web search interface for over 800 font families, from Open Sans to Ubuntu Mono

cdnjs

• https://tools.wmflabs.org/cdnjs/ - the full listing of over 3,000 libraries, from Angular to React

Onwiki docs

• https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#External_assets

Please give us feedback on how these features could be further improved, submit patches, or just show us how you are using them!

The shared Elasticsearch cluster hosted in Toolforge was upgraded from 2.3.5 to 5.3.2 today (T164842). This upgrade comes with a lot of breaking API changes for clients and indexes, and should have been announced in advance. @bd808 apologizes for that oversight.

The stashbot, sal, and bash tools have been fixed to work with the new version. They all mostly needed client library upgrades and minor API usage changes due to the library changes. If you are one of the few other users of this cluster and your tool is broken due to the change and you need help fixing it, open a task or better yet come to the #wikimedia-cloud Freenode channel and ask @bd808 for help.

(reposted with minor edits from https://lists.wikimedia.org/pipermail/labs-l/2017-July/005036.html)

TL;DR

• Tool Labs is being renamed to Toolforge
• The name for our OpenStack cluster is changing from Labs to Cloud VPS
• The prefered term for projects such as Toolforge and Beta-Cluster-Infrastructure running on Cloud-VPS is VPS projects
• Data Services is a new collective name for the databases, dumps, and other curated data sets managed by the cloud-services-team
• Wiki replicas is the new name for the private-information-redacted copies of Wikimedia's production wiki databases
• No domain name changes are scheduled at this time, but we control wikimediacloud.org, wmcloud.org, and toolforge.org
• The Cloud Services logo will still be the unicorn rampant on a green field surrounded by the red & blue bars of the Wikimedia Community logo
• Toolforge and Cloud VPS will have distinct images to represent them on wikitech and in other web contexts

In February when the formation of the Cloud Services team was announced there was a foreshadowing of more branding changes to come:

This new team will soon begin working on rebranding efforts intended to reduce confusion about the products they maintain. This refocus and re-branding will take time to execute, but the team is looking forward to the challenge.

In May we announced a consultation period on a straw dog proposal for the rebranding efforts. Discussion that followed both on and off wiki was used to refine the initial proposal. During the hackathon in Vienna the team started to make changes on Wikitech reflecting both the new naming and the new way that we are trying to think about the large suite of services that are offered. Starting this month, the changes that are planned (T168480) are becoming more visible in Phabricator and other locations.

It may come as a surprise to many of you on this list, but many people, even very active movement participants, do not know what Labs and Tool Labs are and how they work. The fact that the Wikimedia Foundation and volunteers collaborate to offer a public cloud computing service that is available for use by anyone who can show a reasonable benefit to the movement is a surprise to many. When we made the internal pitch at the Foundation to form the Cloud Services team, the core of our arguments were the "Labs labs labs" problem and this larger lack of awareness for our Labs OpenStack cluster and the Tool Labs shared hosting/platform as a service product.

The use of the term 'labs' in regards to multiple related-but-distinct products, and the natural tendency to shorten often used names, leads to ambiguity and confusion. Additionally the term 'labs' itself commonly refers to 'experimental projects' when applied to software; the OpenStack cloud and the tools hosting environments maintained by WMCS have been viable customer facing projects for a long time. Both environments host projects with varying levels of maturity, but the collective group of projects should not be considered experimental or inconsequential.

Debian Stretch was officially released on Saturday[1], and I've built a new Stretch base image for VPS use in the WMF cloud. All projects should now see an image type of 'debian-9.0-stretch' available when creating new instances.

Puppet will set up new Stretch instances just fine, and we've tested and tuned up several of the most frequently-used optional puppet classes so that they apply properly on Stretch. Stretch is /fairly/ similar to Jessie, so I'd expect most puppet classes that apply properly on Jessie to work on Stretch as well, but I'm always interested in the exceptions -- If you find one, please open a phabricator ticket.

The WMF and the Cloud team is committed to long-term support of this distribution. If you are starting a new project or rebuilding a VM you should start with Stretch to ensure the longest possible life for your work.

Back in the dark ages of Labs, all instance puppet configuration was handled using the puppet ldap backend. Each instance had a big record in ldap that handled DNS, puppet classes, puppet variables, etc. It was a bit clunky, but this monolithic setup allowed @yuvipanda to throw together a simple but very useful tool, 'watroles'. Watroles answered two questions:

1. What puppet roles and classes are applied to a given instance?
2. What instances use a given puppet class or role?

#2 turned out to be especially important -- basically any time an Op merged a patch changing a puppet role, they could look at watroles to get a quick list of all the instances that were going to break. Watroles was an essential tool for keeping VMs properly puppetized during code refactors and other updates.

Alas, the puppet ldap backend fell into disrepair. Puppetlabs stopped maintaining it, and Labs VMs were left out of more and more fancy puppet features because those features were left out of ldap. So... we switched to a custom API-based puppet backend, one that talks to Horizon and generally makes VM puppet config more structured and easier to handle (as well as supporting project-wide and prefix-wide puppet config for large-scale projects.)

That change broke Watroles, and the tool became increasingly inaccurate as instances migrated off of ldap, and eventually it was turned off entirely. A dark age followed, in which puppet code changes required as much faith as skill.

Today, at last, we have a replacement. I added a bunch of additional general-purpose queries to our puppet configuration API, and we've added pages to the OpenStack Browser to display those queries and answer both of our previous questions, with bonus information as well:

1. What puppet roles and classes are applied to a given instance?
2. What instances, prefixes, or projects use a given puppet class or role?
3. Which puppet classes are currently in use on Cloud VMs?

The data on those pages is cached and updated every 20 minutes, so won't update instantly when a config is changed, but should nonetheless provide all the information needed for proper testing of new code changes.

The first very visible step in the plan to rename things away from the term 'labs' happened around 2017-06-05 15:00Z when IRC admins made the #wikimedia-labs irc channel on Freenode invite-only and setup an automatic redirect to the new #wikimedia-cloud channel.

If you were running a bot in the old channel or using an IRC bouncer like ZNC with "sticky" channel subscriptions you may need to make some manual adjustments to your configuration.

See also

• T166420: Change Cloud Services irc channels for rebranding

The v0.37 build of rOSTW operations-software-tools-webservice has been deployed to Toolforge hosts and Tools-Kubernetes Docker images.

This release contains the following changes:

• rOSTWe653167c0cfd: Always cleanup manifest on stop for T163355: webservice stop says service not running but service.manifest not cleared
• rOSTW7e8da902d4ac: Sort backend provided types when displaying help

Kubernetes webservices will need to be restarted to pick up the new Docker images with the updated package installed. The new Docker images also contain the latest packages from the upstream apt repositories which may provide some minor bug fixes. We are not currently tracking the exact versions of all installed packages, so we cannot provide a detailed list of the changes.

When @Ryan_Lane first built OpenStackManager and Wikitech, one of the first features he added was an interface to setup project-wide sudo policies via ldap.

I've basically never thought about it, and assumed that no one was using it. A few months ago various Labs people were discussing sudo policies and it turned out that we all totally misunderstood how they worked, thinking that they derived from Keystone roles rather than from a custom per-project setup. I immediately declared "No one is using this, we should just rip out all that code" and then ran a report to prove my point... and I turned out to be WRONG. There are a whole lot of different custom sudo policies set up in a whole lot of different projects.

So... rather than ripping out the code, I've implemented a new sudo interface that runs in Horizon. [T162097] It is a bit slow, and only slightly easier to use than the old OpenStackManager interface, but it gets us one step closer to moving all PVS user interfaces to Horizon. [T161553]

For the moment, users can edit the same policies either on Horizon or on Wikitech. If I don't get complaints then I'll remove the UI from wikitech in a few weeks.

For nearly a year, Horizon has supported instance management. It is altogether a better tool than the Special:NovaInstance page on Wikitech -- Horizon provides more useful status information for VMs, and has much better configuration management (for example changing security groups for already-running instances.)

So... I've just now removed the sidebar 'Manage Instances' link on Wikitech, and will shortly be disabling the Special:NovaInstance page as well. This is one more (small) step down the road to standardizing on Horizon as the one-stop OpenStack management tool.

I've just installed a new public base image, ' debian-9.0-stretch (experimental)' and made it available for all projects. It should appear in the standard 'Source' UI in Horizon any time you create a new VM.

Please take heed of the word 'experimental' in the title. Stretch is not yet an official Debian release, and may still contain unexpected bugs and security issues. Additionally, the operations puppet repo has only begun to be Stretch-aware, so many roles and classes will likely fail or engage in otherwise undefined behavior.

So...

• Please DO use this image to test and update puppet classes, and to get a preview of things to come.

• Please DO NOT use this image to build any instances that you plan to expose to the public, or that you want to keep around for more than a few months.

When Stretch is officially released and we've ironed out some more puppet issues I will delete this base image and purge those VMs that were built from it in order to avoid the debt of having to support 'experimental' VMs for eternity.

Andrew will be upgrading our Openstack install from version 'Kilo' to version 'Liberty' on Tuesday the 2nd. The upgrade is scheduled to take up to three hours. Here's what to expect:

• Tools services and existing Labs uses should be unaffected apart from brief ( < 1 minute) interruptions in network service.

• Continuous Integration tests (Jenkins, Zuul, etc.) will be disabled for most of the upgrade window.

• Creation/Deletion of new instances will be disabled for most of the window.

• Wikitech and Horizon may error out occasionally and/or display inconsistent information. Users may need to refresh their web logins after the upgrade.

Apart from fixing a few seldom-seen bugs, this upgrade shouldn't result in noticeable changes for Labs users. It will lay the groundwork for an upcoming Horizon upgrade, but that will be announced in future posts/emails.

If you are exclusively a user of tool labs, you can ignore this post. If you use or administer another labs project, this REQUIRES ACTION ON YOUR PART.

We are reclaiming unused resources due to an ongoing shortage.

Visit this page and add a signature under projects you know to be active:

VM Purge 2016

Associated wmflabs.org domains are included to identify projects by offered services.

We are not investigating why projects are needed at this time. If one person votes to preserve then we will do so in this round of cleanup.

In a month, projects and associated instances not claimed will be suspended or shutdown. A month later if no one complains these projects will be deleted.

The Kubernetes ('k8s') backend for Tool Labs webservices is open to
beta testers from the community as a replacment for Grid Engine
(qsub/jsub).

We have focused on providing support for PHP webservices with other
job types to follow. No action is required for users who do not want
to change at this time.

Advantages:

• Debian Jessie
• PHP (5.6)
• Better isolation between tools
• Testing the future platform and helping Tool Labs mature

How to switch an existing webervice to the Kubernetes backend:

webservice --backend=gridengine stop
webservice --backend=kubernetes start

How to switch an existing webservice back to Grid Engine:

webservice --backend=kubernetes stop
webservice --backend=gridengine start

If you encounter issues, please let us know at
https://phabricator.wikimedia.org/T139107 or on irc at
Cloud-Services.

For instructions and more information visit
https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Web/Kubernetes

The Wikimedia Legal team is interested in revising, updating, and clarifying the existing Labs Terms of Use governing developers and their projects on labs.

We have opened up Round 1 of our community consultation to hear feedback on the Labs Terms of Use. We will try to respond the best we can, but the main purpose of this round is to hear all your thoughts. After the feedback round, we will prepare a draft revision of the Terms based on that feedback and other minor revisions to clarify statements in existing the Terms. We will then engage in a community discussion about the revised Terms.

We plan to leave the discussion open until June 9, 2016.

horizon (OpenStack Dashboard) is the canonical implementation of OpenStack’s Dashboard, which provides a web based user interface to OpenStack services.

Horizon requires 2fa to manage project resources. 2fa (which has long been required of all wikitech admins) can be setup on your special preferences page.

tools-login.wmflabs.org is on a new bastion host with twice
the RAM and CPU of the old one. This should hopefully provide a better
bandaid against it getting overloaded up. More discussion about a
longer term solution at https://phabricator.wikimedia.org/T131541

You can find new fingerprints at
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/tools-login.wmflabs.org

Apologies for the interruption! The old bastion will continue to be available for now at tools-bastion-05.eqiad.wmflabs.

On Tuesday, 2016-04-05, we'll be upgrading Kubernetes to 1.2 and using
a different deployment method as well. While this should have no user
facing impact (ideally!) the following things might be flaky for a
period of time on that day:

1. The Gerrit IRC bot
2. NAGF (tools.wmflabs.org/nagf)
3. PAWS (paws.wmflabs.org, for those using it)

You can follow the upgrade at
https://phabricator.wikimedia.org/T130972 if you're interested.