More Related Content
Similar to Fallacies of distributed computing with Kubernetes on AWS
Fallacies of distributed computing with Kubernetes on AWS
- 1. FALLACIES OF DISTRIBUTED COMPUTING WITH KUBERNETESON AWS Raffaele Di Fazio 05.10.2017
- 2.
- 3. 3 ZALANDO 15 markets 6 fulfillmentcenters 21 million active customers 3.6 billion € net sales 2016 200 million visits per month 13,000 employees in Europe
- 4. 4 ZALANDO TECHNOLOGY HOME-BREWED, CUTTING-EDGE & SCALABLE technologysolutions >1,800 employees from tech locations + HQs in Berlin6 77 nations
- 5. HISTORIA MAGISTRA VITAE Photo byJace Grandinetti on Unsplash
- 6. 6 Node KUBERNETES ARCHITECTURE kubectl Master Node API Server Scheduler Controller Manager SkipperKube2IAM Kubelet etcd Logging agent ….
- 7. WHAT HAPPENS WHEN THEAPI SERVER IS NOT RUNNING? Photo by Thomas Kvistholt on Unsplash
- 8. 8 “A FEW MORE”404s THAN USUAL...
- 9. 9 Thanks to AshleyMcNamara for the picture
- 10. WHY? Photo by RicardoGomez Angel on Unsplash
- 11. 11 “A FEW” MOREPODS…
- 12. 12 KILLING KUBERNETES’ APISERVER Too much memory API server OOMKilledLots of pods
- 13. 13 SYSTEM VIEW -TRAFFIC FLOW https://github.com/zalando/skipper ALB Node Skipper Node Skipper MyApp MyApp MyApp Service Service K8S network EC2 network TLS HTTP
- 14.
- 15. 15 WHAT REALLY HAPPENED •All routes removed: • No routes to the applications deployed inside the cluster • Healthcheck “unhealthy” because of no connection to API server • => All nodes were unhealthy in the ELBv2
- 16. WHAT HAPPENS WHEN ALLTHE TARGETS IN AN ELBv2 ARE UNHEALTHY? Photo by Sandro Katalina on Unsplash
- 17. 17 WHAT ABOUT THEELBv2?
- 18. 18 WHAT ABOUT THEELBv2? If no Availability Zone contains a healthy target, the load balancer nodes route requests to all targets.
- 19. KUBERNETES API SERVER AVAILABILITY ANDCONTROL LOOPS Photo by chuttersnap on Unsplash
- 20.
- 21.
- 22. 22 WE ARE NOTALONE • … a test that simulated the failure of a single apiserver node disrupted the cluster in a way that negatively impacted the availability of running workloads • ... helped us identify that the disruption was likely related to an interaction between the various clients that connect to the Kubernetes apiserver (like calico-agent, kubelet, kube-proxy, and kube-controller-manager) and our internal load balancer’s behavior during an apiserver node failure. • Source: Kubernetes at GitHub
- 23.
- 24. 24 HOW WE FIXEDIT • Do not change the healthcheck in case of API server failures • Do not drop the routes in case of API server failures • => Delete when you are really sure you want to delete!
- 25. FALLACIES OF DISTRIBUTED COMPUTING Photo bychuttersnap on Unsplash
- 26. 26 8 FALLACIES OFDISTRIBUTED COMPUTING • The network is reliable. • Latency is zero. • Bandwidth is infinite. • The network is secure. • Topology doesn't change. • There is one administrator. • Transport cost is zero. • The network is homogeneous.
- 27.
- 28. 28 THE FALLACIES OFCLOUD COMPUTING • The API call you will make will succeed. • The next API call you will make will succeed. • Deleting resources is the same as adding new. • Your cloud provider will have no outages. • The dependencies between your services are clear.
- 29. MAKING YOUR SYSTEM RESILIENT Photoby Aaron Barnaby on Unsplash
- 30. 30 WHEN MAKING APICALLS • Every API call can fail • Retry (with backoff) • Circuit breakers • Fallbacks • Don’t scale down / delete resources fast! • Deal with rate limiting • Deal with “weird” values due to a broken cloud provider feature
- 31. 31 TEST ALL THETHINGS • Continuous integration tests • Continuous deployment of cluster updates • Load tests • Chaos tests
- 32. 32 CONTINUOUS INTEGRATION TESTS •Test the interactions between components • For every configuration change we run extensive e2e tests
- 33.
- 34.
- 35. 35 LOAD TESTING • Lotsof request to the API server • Lots of pods running • Write/reads to the data storage (etcd) • => what matters: observe the impact on running applications
- 36. 36 CHAOS TESTING • Randomshutdown of Kubernetes components • https://github.com/linki/chaoskube • http://chaostoolkit.org/ • https://github.com/asobti/kube-monkey • http://principlesofchaos.org • Random shutdown of nodes (EC2 Instances) • https://github.com/Netflix/chaosmonkey
- 37. 37 MORE ON CHAOSTESTING • Netflix’s principles of Chaos Engineering • http://principlesofchaos.org • Chaos Engineering free ebook ->
- 38. 38 THAT’S NOT ALL •You think Kubernetes the hard way is hard • The hard part was never only the setup • Sometimes you will have to break things to learn • …setup a healthy post mortem culture and learn from mistakes!
- 39. THAT’S IT Photo byDhruva Reddy on Unsplash
- 40.
