Litmus 2.0.0-Beta3

Major Updates

• Support for policy-based control of event tracker where users can define their own policy using JMESPath query and based on that event-tracker will react to the application changes.
• Enhanced UI for workflow Scheduling, gives users the ability to tune annotations, target application details like application namespace, labels, and kind, and probe data using User Interface.
• New UI for workflow visualization for showing information about workflow and nodes in a better way.
• We made the onboarding process for users and easier to use through the new UI.
• Enhanced the homepage to show information like Recent workflow runs, Agent details, and Project details.
• Shifting project switching from using Redux-based technique to URL-based technique to avoid caching problems.
• Migrated CircleCI to GitHub workflow and enhanced the continuous integration of the project.
• Enhanced the analytics module in terms of UI and computation
• Enhanced the browse workflows table to show resilience score and the total number of experiments passed for the listed workflows.* Support role-based access control in the backend for handling authorization for all requests.
• Support for storing scheduled workflow templates and adding some new podtato-head predefined workflow templates

Minor Updates

• Increment in the Better Code Hub(BCH) score
• Optimized the frontend by shifting the resiliency score calculation to the backend.
• Restructured the directory structure for settings in the frontend to modularise the code.
• Support for a reinstall of litmus agents by moving the litmus-portal-config configmap independent of the subscriber.
• Support for Ingress and Load balancer network type for connecting external agents with Litmus Portal. Based on the server service type, it will generate the endpoint for the external agent.

Major Updates

• Support for in-built analytics, where users can connect their data sources and generate dashboard panels.
• Support for Git as a single source of truth for workflow artifacts. This enables users to have their workflows synced between the portal and Git source.
• Introduces the event-tracker microservice to trigger chaos workflows automatically upon change to application images. This feature works in tandem with GitOps frameworks that rollout changes to applications upon manual changes in the Git source or upon image push to registries.
• Support for re-running of existing chaos workflow from the litmus portal.
• Adding a command-line tool called litmusctl to manage litmus portal services. The key role of litmusctl is to connect the external cluster with the litmus server and install the external agents.
• Redesigning the teaming user interface and adding some significant features such as leave project, decline invitation.
• Recreating litmus docs for litmus 2.0.x. For more information, visit https://litmusdocs-beta.netlify.app/
• Integration of Litmus-UI with litmus portal components
• Major directory restructuring of litmus portal’s server for database handlers

Minor updates

• Changing MongoDB kind from deployment to statefulsets
• Adding chaos-exporter as default external cluster agents for litmusportal
• Refactoring authentication server to accommodate new teaming integration
• Removing some unnecessary inputs from the welcome modal and predefined chaos workflow

New Features & Enhancements

• For updates on 2.0.0-Beta releases, refer the notes for Litmus 2.0.0-Beta3.

• Enhances the EC2 termination experiments to filter targets by tags (apart from IDs), along with support for list and percentage-based selection of instances, serial and parallel failure modes

• Supports collection of chaos metrics for all ChaosEngine resources by default instead of selective monitoring controlled via spec attributes

• Supports the definition of ‘context’ (metadata) for an experiment via a Kubernetes label on the ChaosEngine that translates to a metric label value on Prometheus. This can be used to group experiment results via context/reason or derive useful insights from metrics.

• Introduces a new chaos metric litmuschaos_experiment_verdict that provides an instance-specific run result (instead of cumulative result stats) that can be used alongside the litmuschaos_awaited_experiments to obtain improved chaos interleaved dashboards.

• Adds documentation around supported chaos metrics and their utility.

• Allows users to specify the terminationGracePeriodSeconds for the chaos experiment and helper pods to allow abort routines to go through (useful in clusters with high API traffic or under group chaos execution on multiple apps at once)

• Provides new environment variables (translating to stress-ng flags) for node resource chaos experiments to ensure the granular definition of the load/stress profile.

• Adds abort routines for infra/node and autoscaler experiments and optimizes the same for pod experiments in which they are already defined.

• Introduces a randomness factor in the pod-delete experiment to ensure that the delete operations occur at random intervals (the random periods being picked within a time range defined by lower-upper bounds).

• Enhances the pumba chaoslib for stress experiments by providing an additional ENV var for defining the stress image (that is pulled at runtime on the target pod’s node to inject the stressor). This is useful for folks running experiments with images from their private registries.

• Introduces a tech-preview of a DNS-chaos experiment (available in litmuschaos/go-runner:ci image) that can cause dns errors/failure in target containers

• Updates the Chaos Github Actions used in the PR/commit-based e2e suite on the litmus-go repository.

• Improves the e2e dashboard to represent the experiment e2e coverage in a clearer way.

• Begins the migration process of specific e2e pipelines to GitHub Actions from Gitlab to aid definition of multiple component/feature-based workflows from within a single branch

• Adds a new utility (nsutil) to execute commands on the target containers namespace, with potential usage in multiple pod-level chaos experiments

Major Bug Fixes

• Fixes repeated scheduling of experiment pods upon helper failure/ungraceful exits (error state )- the pods will now enter the completed state upon first error.

• Appends missing CRD validation schema for image pull policy for experiments

• Upgrades all litmus artifacts containing CRD spec to use version v1 from v1beta1 to support newer Kubernetes platforms

• Adds checks to validate the definition of app labels when annotation checks are set to false on the ChaosEngine (and fail fast with appropriate error).

• Fixes the behavior where multiple “downstream” probes defined in the same phase (pre/post/on chaos) fail if the first probe evaluates to failure.

• Fixes an issue that is seen when running chaos on multiple application replicas/targets at once, where chaos injection against the last replica/target alone is considered for the success of the experiment.

• Adds retries to factor in the pending status of helper pods in populated/dense clusters where it takes time for the pod to be scheduled.

• Adds logs to the Kafka liveness/load pod launched during the Kafka broker failure experiments to verify successful service discovery & topic creation success/failure.

Major Known Issues & Limitations

Issue:

The pod-cpu-hog & pod-memory-hog experiments that run in the pod-exec mode (which is typically used when the users don’t want to mount runtime’s socket files on their pods) using the default lib can tend to fail - in spite of chaos being injected successfully - due to the unavailability of certain default utils in the target’s image that is used for detecting the chaos process and killing them/reverting chaos at the end of the chaos duration.

Workaround:

Users can identify the necessary commands to identify and kill the chaos processes and pass them to the experiment via env variable CHAOS_KILL_COMMAND
Alternatively, then can make use of the pumba chaoslib that uses external containers with SYS_ADMIN docker capability to inject/revert the chaos, while mounting the runtime socket file. Note that this is supported only on docker at this point.

Fix:

This is being actively worked on (native litmus chaoslib that can inject stress processes w/o exec requirement for docker/containerd/crio) and should be available in a subsequent release.

Installation

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v1.13.3.yaml

Verify your installation

• Verify if the chaos operator is running
  kubectl get pods -n litmus

• Verify if chaos CRDs are installed
  kubectl get crds | grep chaos

For more details refer to the documentation at Docs

New Features & Enhancements

• Litmus Portal progress will be tracked in the fortnightly 2.0.0-beta releases henceforth as we build towards Litmus 2.0 GA. Check out the release notes for Litmus 2.0.0-Beta0 & Litmus 2.0.0-Beta1

• Enhances the comparator logic for cmdProbe & promProbe to include more operations (OneOf, Range, etc.,) against a list of int, float & string values.

• Streamlines the application health status check process by considering only the pods of annotated parent workloads (when annotationCheck is true). Also provides support to verify the readiness for a specific container (specified by ChaosExperiment/ChaosEngine env var TARGET_CONTAINER) within a pod.

• Simplifies the disk-fill generic chaos experiment to work on applications that don’t have ephemeral storage limits defined explicitly, with env var EPHEMERAL_STORAGE_MEBIBYTES.

• Improves the chaoslib & health-checks used in the AWS ec2-terminate experiment to work with managed node groups (ex: KOPS, EKS)

• Extends support for termination signal type SIGKILL (as env var) in the container-kill experiment for Containerd, CRI-O runtimes.

• Removes the need to change permissions of the container runtime socket-files on the node (mounted into the experiment helpers and permissions updated via init-containers) when using Litmus LIB for container-kill & network-chaos experiments

• Extends OnChaos mode of probe operation to all native Litmus experiments

• Improves the K8sProbe schema to be more meaningful, by replacing the “command” field with GVR (group-version-resource) fields in the probe inputs. Also adds K8s CRUD operation as a valid probe input.

• Speeds up the abort routine (impact observed at scale) of network chaos experiments by shifting the chaosresult and event generation (kube-api calls) steps from helper pods to experiment pod.

• Provides support for defining ResponseTimeouts for API calls in the httpProbe.

• Adds flexibility in the definition of applabel in ChaosEngine (recommended kubectl patterns such as =, ==, !=, in, notin, exists) with annotationCheck enabled/disabled.

• Introduces a new chaos category for Azure Cloud with the Azure VM instance kill experiment (available in tech preview mode with image: litmuschaos/go-runner: azure)

• Adds additional labels (ChaosEngine name) in the Chaos Exporter metrics for improved tracking purposes in monitoring systems.

• Adds new e2e tests (for validation of chaos execution in serial/parallel mode when pod_affected_percentage > 0) for PRs on chaos-operator

• Increased unit-test coverage (+35%) on chaos-runner component

• Includes the litmus-portal pipeline coverage details & execution results to the litmus-e2e dashboard.

Major Bug Fixes

• Removes the force flag (terminationGracePeriodSeconds: 0 ) from the abort function to facilitate the experiment & helper pods to successfully execute the chaos rever/rollback and notification (via event) routine. The need is for the chaos rollback to occur instantaneously and in a guaranteed manner rather than immediate removal of the chaos pods/resources.

• Ensures guaranteed rollback/revert of chaos process (tc rule) on target pods upon abort when the chaos injection occurs in parallel at scale (100-150 replicas). This is enabled via a change in the ordering of tasks in the abort routine and preventing further execution in the inject routine once SIGTERM is received.

• Skips the AUT (Application-Under-Test) status checks in the chaos namespace when the .spec.appInfo is not specified within the ChaosEngine (with portal driven execution, the chaos namespace may contain completed experiment/argo pods that are not in “Running” status).

• Handles invalid DESTINATION_HOSTS better in the network chaos experiments. The inability to resolve specified hostnames to valid IPs is logged, with only valid hosts injected with chaos instead of applying total egress chaos on the interface.

• Fixes the OpenAPI schema validation error for httpProbe that resulted in a failed evaluation of the probe.

• Fixes the result notification error for failure cases in the Kafka Broker Pod Failure experiment and makes the partition leader identification process more resilient.

• Includes the missing step to propagate the ImagePullPolicy of experiment images to the helper pod (via ENV vars)

• Includes the missing step to propagate the imagePullSecrets to the helper pods (via spec attribute)

Major Known Issues & Limitations

Issue:

The pod-cpu-hog & pod-memory-hog experiments that run in the pod-exec mode (which is typically used when the users don’t want to mount runtime’s socket files on their pods) using the default lib can tend to fail - in spite of chaos being injected successfully - due to the unavailability of certain default utils in the target’s image that is used for detecting the chaos process and killing them/reverting chaos at the end of the chaos duration.

Workaround:

Users can identify the necessary commands to identify and kill the chaos processes and pass them to the experiment via env variable CHAOS_KILL_COMMAND
Alternatively, then can make use of the pumba chaoslib that uses external containers with SYS_ADMIN docker capability to inject/revert the chaos, while mounting the runtime socket file. Note that this is supported only on docker at this point.

Fix:

This is being actively worked on (native litmus chaoslib that can inject stress processes w/o exec requirement for docker/containerd/crio) and should be available in a subsequent release.

Installation

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v1.13.2.yaml

Verify your installation

• Verify if the chaos operator is running
  kubectl get pods -n litmus

• Verify if chaos CRDs are installed
  kubectl get crds | grep chaos

For more details refer to the documentation at Docs

New Features & Enhancements

• Moves the Litmus Portal to beta-2 phase with the following improvements:

  • Ability to disable workflow schedules
  • Support for configuration of private Git repositories as a source for experiments & predefined workflows (private MyHub)
  • Allows the full set of CRUD operations on the embedded ChaosHub/MyHub
  • Improves the chaos visualization via horizontal/vertical workflow views and proper formatting of logs for the workflow nodes.

• Enhances the ChaosExperiment CRD to take HostPath Volume Type input.

• Removes the limitation that only a single workload (amongst those sharing the labels) can be annotated for chaos.

• Enhances the httpProbe to perform POST operations with payload described in the ChaosEngine or via a file mounted as a configmap.

• Simplifies node resource chaos experiments to accept resources in units (mebibytes) along with relative percentage inputs.

• Makes the termination mode configurable for the container-kill experiment (defaults to SIGKILL)

• Adds more details to experiment logs around annotated workloads & filtered pod targets

• Improves the disk-fill chaos experiment to use the helper pod approach for injection instead of running a dummy pod with a sleep command into which multiple exec operations occur.

• Additional unit tests in the chaos-operator & chaos-runner repos.

• Improves e2e tests (PRs/Commits) (pod chaos with combinations of pods_affected_perc & sequence env, annotation on multiple workloads etc.,) in the litmus-go repo

• Updates the litmus-sdk based on recent changes to experiment templates

Major Bug Fixes

• Ensures that different helper pods within an experiment instance are labeled with unique values (for fixed keys) in order to query them for status. Without this, these helper pods were being filtered by common labels resulting in incorrect validation. This is more so when multiple instances of the same experiment are executed in parallel.

• Reflects the correct verdict of the experiment upon failure and abort, along with improved events in the Kafka & Cassandra chaos experiments.

• Ensures smooth re-run of network chaos on a target with residual tc rule from the previous instance of chaos injection (RTNETLINK answers: File exists)

• Fixes the console spamming log messages on chaos-exporter which were seen until the ChaosResult/Engine resources were created.

Major Known Issues & Limitations

Issue:

Forced removal of the experiment helper pods (where applicable: notably network chaos experiments) either manually or due to Kubernetes eviction can render the chaos revert operation at the end of the chaos duration a failure/ a non-event. This will cause the application under test (AUT) to continue being subjected to chaos unless manually recovered.

Workaround:

With experiment pod logs it can be deciphered that the helper operations have failed. In which case, the AUT pod(s) can be deleted so they can be rescheduled again (this is applicable only to those applications deployed as a higher-level controller such as deployment/statefulset/daemonset, etc.,) with a new network namespace.

Fix:

This is being actively worked on (retry mechanism for chaos revert initiated in case of failed/missing helper pods) and should be available in a subsequent release.

Issue:

The pod-cpu-hog & pod-memory-hog experiments that run in the pod-exec mode (which is typically used when the users don’t want to mount runtime’s socket files on their pods) using the default lib can tend to fail - in spite of chaos being injected successfully - due to the unavailability of certain default utils in the target’s image that is used for detecting the chaos process and killing them/reverting chaos at the end of the chaos duration.

Workaround:

Users can identify the necessary commands to identify and kill the chaos processes and pass them to the experiment via env variable CHAOS_KILL_COMMAND
Alternatively, then can make use of the pumba chaoslib that uses external containers with SYS_ADMIN docker capability to inject/revert the chaos, while mounting the runtime socket file. Note that this is supported only on docker at this point.

Fix:

This is being actively worked on (native litmus chaoslib that can inject stress processes w/o exec requirement for docker/containerd/crio) and should be available in a subsequent release.

Installation

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v1.13.0.yaml

Verify your installation

• Verify if the chaos operator is running
  kubectl get pods -n litmus

• Verify if chaos CRDs are installed
  kubectl get crds | grep chaos

For more details refer to the documentation at Docs

New Features & Enhancements

• Moves the Litmus Portal to beta-1 phase with the following improvements:

  • Supports edit of (cron) schedule in chaos workflows
  • Ability to suspend/disable schedules
  • Improved chaos workflow diagrams with appropriate log representation for different stages/steps
  • Increased (K8s) validation in the workflow construction wizard
  • Adds the infra changes necessary to support private repositories for MyHub (UI support to come in 1.13.0)

• Introduces a revamped chaos-exporter that removes the current dependency on the heptio event-router for the experiment execution state, which was being used to build chaos-interleaved application dashboards. The chaos exporter now pushes an increased set of metrics on chaos start/end times, status, success percentage per run, experiment specific cumulative pass/fail counts, etc., and has options to operate in both cluster-wide as well as the namespaced modes.

• Enhances the httpProbe with options to skip certificate checks via the insecureSkipVerify flag in the ChaosEngine schema

• Enhances the pod-autoscaler experiment with the ability to scale multiple applications (type: deployments, statefulsets) based on an APP_AFFECTED_PERC environment variable, with the apps being filtered via label selectors. Also adds support for OnChaos probes for the experiment.

• Supports random selection of EC2 instances/Kubernetes nodes for the ec2-terminate experiment in cases where the target instance is not explicitly specified.

• Improves error handling logic in the node-drain experiment and also adds a timeout (equal to the chaos duration period) flag to the drain operation to prevent indefinite execution (ex: to honor pod disruption budgets, stuck evictions)

• Extends the ImagePullPolicy configuration to external probe pods (in cases where the cmdProbe is configured to run on “source” images other than the litmus go-runner).

• Homogenizes the experiment pod logs for target pod information prior to chaos injection

• Promotes the non-root go-runner from tech-preview to a release image. Accompanied by changes to experiments where applicable (commands, paths & file permissions)

• Introduces a tech-preview of enhanced chaos rollback/revert logic (used initially for network chaos experiments executed in “serial” sequence ) to achieve guaranteed chaos rollback/revert under failure conditions (helper pod eviction, unexpected chaos process termination, deletion/removal, etc.,) (litmuschaos/go-runner:1.12.0-revert)

• Enhances the ChaosResult schema to hold cumulative success/failure count information of the different run instances for a given experiment.

• Introduces a new scaffolded chaoslib template in the litmus SDK that allows injection and revert of chaos via the CHAOS_INJECT_COMMAND & CHAOS_KILL_COMMAND environment variables, thereby giving users flexibility in creating preview experiments.

• Releases the v0.3.1 of the chaos-ci-lib with fixes and enhancements to the chaos BDD library, and updates the e2e suites to use it.

• Migration to GitHub Actions (with parallel workflows for lint, security scan, e2e & build/push operations) from TravisCI (where applicable) in lieu of reduced support for OSS projects on the latter.

• Enhances the litmus-e2e suite with new tests for verification of annotation-enabled & disabled chaos execution, ec2-terminate experiment & pumba-based chaoslib functionality. Adds the feature coverage tracker with an initial set of testcases for litmus-portal e2e pipelines

• Enhances the litmus-helm chart testing workflows as per the latest K8s/Helm standards

• Improves the node-restart & adds node-poweroff experiment documentation with steps to obtain the ssh-keys & setup the secrets for execution.

• Simplifies the experiment pages UX on the ChaosHub with explanation/steps to use the chaos artifacts

Major Bug Fixes

• Fixes spurious events received on ChaosEngines installed with engineState set to stop (for deferred execution purposes). Also ensures that the ChaosInitialization is recorded once finalizers have been applied on the CR

• Prevents a false positive with probe execution (in cases where probes were defined without the RunProperties specification) by mandating the latter using CRD validation.

• Fixes failed/timed-out helper pod checks in the node-restart and node-poweroff experiments with an enhanced status check logic that looks for variadic/desired pod states (such as Succeeded, Running, etc..,) instead of just “Running”

• Fixes the failure to kill target docker containers using the “litmus” LIB due to the missing “host” flag pointing to the correct daemon socket path

• Fixes a regression on the pod-cpu-hog experiment that caused only a single md5sum process to be launched on the target pods irrespective of the CPU_CORES (number of cores) input to the experiment.

• Fixes a regression (panic) on the chaos-runner caused upon secret volumes definition in the ChaosExperiment/ChaosEngine

• Synchronizes event messages (from the experiment pod as well as chaos-runner pod sources) with the latest experiment status/verdict in case of repeated execution (caused by frequent abort/restart operations) instead of holding stale info.

• Replaces hardcoded socket paths in experiment helper configurations with values derived from the SOCKET_PATH environment variable

• Fixes failed application status checks on infra-chaos experiments where the .spec.appinfo.applabel is not specified/skipped. In this case, the health of all pods in the chaos namespace is verified.

• Fixes the documentation with the correct kubectl command to patch the ChaosEngine for abort/restart.

Major Known Issues & Limitations

Issue:

Forced removal of the experiment helper pods (where applicable: notably network chaos experiments) either manually or due to Kubernetes eviction can render the chaos revert operation at the end of the chaos duration a failure/ a non-event. This will cause the application under test (AUT) to continue being subjected to chaos unless manually recovered.

Workaround:

With experiment pod logs it can be deciphered that the helper operations have failed. In which case, the AUT pod(s) can be deleted so they can be rescheduled again (this is applicable only to those applications deployed as a higher-level controller such as deployment/statefulset/daemonset, etc.,) with a new network namespace.

Fix:

This is being actively worked on (retry mechanism for chaos revert initiated in case of failed/missing helper pods) and should be available in a subsequent release.

Issue:

The pod-cpu-hog & pod-memory-hog experiments that run in the pod-exec mode (which is typically used when the users don’t want to mount runtime’s socket files on their pods) using the default lib can tend to fail - in spite of chaos being injected successfully - due to the unavailability of certain default utils in the target’s image that is used for detecting the chaos process and killing them/reverting chaos at the end of the chaos duration.

Workaround:

Users can identify the necessary commands to identify and kill the chaos processes and pass them to the experiment via env variable CHAOS_KILL_COMMAND
Alternatively, then can make use of the pumba chaoslib that uses external containers with SYS_ADMIN docker capability to inject/revert the chaos, while mounting the runtime socket file. Note that this is supported only on docker at this point.

Fix:

This is being actively worked on (native litmus chaoslib that can inject stress processes w/o exec requirement for docker/containerd/crio) and should be available in a subsequent release.

Installation

kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v1.12.0.yaml

Verify your installation

• Verify if the chaos operator is running
  kubectl get pods -n litmus

• Verify if chaos CRDs are installed
  kubectl get crds | grep chaos

For more details refer to the documentation at Docs