Memcached
This page is about the Memcached infrastructure (clusters, pools and servers) that backs MediaWiki and Thumbor in production.
For how MediaWiki itself uses this cluster (WANObjectCache, ParserCache, MicroStash), see Memcached for MediaWiki. For the proxy that MediaWiki talks to in order to reach these servers, see mw-mcrouter.
This page is not about other Memcached clusters in production, such as, Wikimedia Cloud Services, and Swift.
Servers
There are three logical pools of memcached servers for MediaWiki:
- Main: 18 shards per DC, running on the
mc10XXhosts (in Eqiad) andmc20XXhosts (in Codfw). - Gutter: 3 shards per DC, hosted on
mc-gp100xandmc-gp200xhosts. This is a fallback pool for the Main pool, and should not generally receive traffic. See mcrouter routes for how failover to the Gutterpool works. - mc-wf (Wikifunctions): a separate Memcached cluster dedicated to the Wikifunctions service. Unlike the Main/Gutter pools above, mw-wikifunctions does not go through the shared mw-mcrouter daemonset, but rather talks to its own in-pod mcrouter container.
Cross-DC traffic between the MediaWiki and Main pools (e.g. WANCache purges relayed by mcrouter's WAN route) is encrypted using TLS.
Configuration
The list of Memcached servers for the Main and Gutter pools is defined in Puppet, under profile::mediawiki::mcrouter_wancache::shards. This list is passed to mw-mcrouter after being parsed by some wonderful code.
After a change to this list has been merged and propagated to the deployment server, mw-mcrouter needs to be redeployed to pick up the change.
For how to replace a host in a pool, see mw-mcrouter.
Using memkeys and cachedump
See Performance/Runbooks/Analyze memcached for how to use memkeys or cachedump to inspect what a memcached instance is storing, for example to find hot keys or an unexpectedly large key.
Rebooting & Restarting
Runbooks
- Memcached server failure
- Performance/Runbooks/Analyze memcached (how to use memkeys or cachedump)
