Containers
Running Airflow Workflow Jobs on Amazon EKS with EC2 Spot Instances
Apache Airflow is an open-source distributed workflow management platform for authoring, scheduling, and monitoring multi-stage workflows. It is designed to be extensible, and it’s compatible with several services like Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Container Service (Amazon ECS), and Amazon EC2. Many AWS customers choose to run Airflow on containerized environments with […]
Read MoreAutomated software delivery using Docker Compose and Amazon ECS
In November 2020, Docker Compose for Amazon ECS became generally available. It is now even easier for a developer to take a containerized microservices-based application from their workstation and deploy it straight to the AWS Cloud. Developers can now run docker compose up and deploy their existing Docker Compose files straight to Amazon ECS, as […]
Read MoreUsing Dex & dex-k8s-authenticator to authenticate to Amazon EKS
Introduction In an earlier post, Paavan Mistry introduced us to the OIDC identity provider (IdP) authentication for Amazon Elastic Kubernetes Service (Amazon EKS), a feature that allows you to use an OIDC identity provider with new or existing clusters. Before launching this feature, IAM principles and service account bearer tokens were the only authentication methods […]
Read MoreBuilding container images on Amazon ECS on AWS Fargate
Building container images is the process of packaging an application’s code, libraries, and dependencies into reusable file systems. Developers create a Dockerfile alongside their code that contains all the commands to assemble a container image. This Dockerfile is then used to produce a container image using a container image builder tool, such as the one […]
Read MoreManaging compute for Amazon ECS clusters with capacity providers.
Customers running containers are often challenged with having to manage and understand how to run and scale the compute for their clusters. For customers taking advantage of Amazon Elastic Container Service (Amazon ECS) on AWS Fargate, the burden is lifted as the underlying compute layer is fully managed by AWS, enabling the customer to focus […]
Read MorePolicy-based countermeasures for Kubernetes – Part 2
Choosing the Right Policy-As-Code Solution In Part 1 of this series, we introduced the concept of policy-as-code (PaC), and discussed the following solutions: OPA, OPA/Gatekeeper and MagTape. In this post (Part 2) we will review the Kyverno and k-rail PaC solutions. For Kubernetes, there are several PaC solutions available in the open-source software (OSS) community. […]
Read MorePolicy-based countermeasures for Kubernetes – Part 1
Choosing the right policy-as-code solution This is Part 1 in a two part series where we discuss policy-as-code solutions. As more organizations adopt containerization as a delivery strategy, the need for automated security, compliance, and privacy controls that detect, prevent, reduce, and counteract known and unknown threats, has increased. Out of this increased need for […]
Read MorePreparing for Kubernetes API deprecations when going from 1.15 to 1.16
The way that Kubernetes evolves and introduces new features is via its APIs. These APIs are versioned and features go through three phases: alpha, beta, and general availability (also known as stable). For users of Kubernetes this usually takes the form of an apiVersion version you are putting on the top of your YAML spec […]
Read MoreHow Vanguard uses AWS X-Ray and Amazon CloudWatch to improve observability for Amazon ECS cloud applications
This post was contributed by Jeffrey Emberger, Technical Lead, The Vanguard Group and John Formento, Solutions Architect, AWS. Cloud applications are changing the speed at which companies can deliver new capabilities for their customers. With increased speed comes the need to more quickly, reliably, and inexpensively observe cloud application health. Observability is no longer an […]
Read MoreUtilizing NVIDIA Multi-Instance GPU (MIG) in Amazon EC2 P4d Instances on Amazon Elastic Kubernetes Service (EKS)
In November 2020, AWS released the Amazon EC2 P4d instances. The Amazon EC2 P4d instances deliver the highest performance for machine learning (ML) training and high performance computing (HPC) applications in the cloud. This instance comes with the following characteristics: Eight NVIDIA A100 Tensor core GPUs 96 vCPUs 1 TB of RAM 400 Gbps Elastic […]
Read More







