The Wayback Machine - https://web.archive.org/web/20211021064543/https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html
Upgrading OpenSearch and Elasticsearch - Amazon OpenSearch Service (successor to Amazon Elasticsearch Service)
Troubleshooting an upgradeStarting an upgradeUsing a snapshot to migrate data

Upgrading OpenSearch and Elasticsearch

Note

OpenSearch and Elasticsearch version upgrades differ from service software updates. For information on updating the service software for your OpenSearch Service domain, see Service software updates in Amazon OpenSearch Service.

Amazon OpenSearch Service offers in-place upgrades for domains that run OpenSearch 1.0 or later, or Elasticsearch 5.1 or later. If you use services like Amazon Kinesis Data Firehose or Amazon CloudWatch Logs to stream data to OpenSearch Service, check that these services support the newer version of OpenSearch before migrating.

Currently, OpenSearch Service supports the following upgrade paths:

From version To version
Elasticsearch 7.x

Elasticsearch 7.x or OpenSearch 1.x

Important

OpenSearch 1.x introduces numerous breaking changes. For details, see Amazon OpenSearch Service - Summary of changes.

Elasticsearch 7.10 introduces a breaking change with regard to dynamic templates. For more information, see Mapper parsing exception while indexing.

Elasticsearch 6.8

Elasticsearch 7.x or OpenSearch 1.x

Important

Elasticsearch 7.0 and OpenSearch 1.0 include numerous breaking changes. Before initiating an in-place upgrade, we recommend taking a manual snapshot of the 6.x domain, restoring it on a test 7.x or OpenSearch 1.x domain, and using that test domain to identify potential upgrade issues. For breaking changes in OpenSearch 1.0, see Amazon OpenSearch Service - Summary of changes.

Like Elasticsearch 6.x, indices can only contain one mapping type, but that type must now be named _doc. As a result, certain APIs no longer require a mapping type in the request body (such as the _bulk API).

For new indices, self-hosted Elasticsearch 7.x and OpenSearch 1.x have a default shard count of one. OpenSearch Service domains on Elasticsearch 7.x and later retain the previous default of five.

Elasticsearch 6.x

Elasticsearch 6.x
Elasticsearch 5.6

Elasticsearch 6.x

Important

Indices created in version 6.x no longer support multiple mapping types. Indices created in version 5.x still support multiple mapping types when restored into a 6.x cluster. Check that your client code creates only a single mapping type per index.

To minimize downtime during the upgrade from Elasticsearch 5.6 to 6.x, OpenSearch Service reindexes the .kibana index to .kibana-6, deletes .kibana, creates an alias named .kibana, and maps the new index to the new alias.

Elasticsearch 5.x Elasticsearch 5.6

The upgrade process consists of three steps:

  1. Pre-upgrade checks – OpenSearch Service performs a series of checks for issues that can block an upgrade and doesn't proceed to the next step unless these checks succeed.

  2. Snapshot – OpenSearch Service takes a snapshot of the OpenSearch or Elasticsearch cluster and doesn't proceed to the next step unless the snapshot succeeds. If the upgrade fails, OpenSearch Service uses this snapshot to restore the cluster to its original state. For more information about this snapshot, see Can't downgrade after upgrade.

  3. Upgrade – OpenSearch Service starts the upgrade, which can take from 15 minutes to several hours to complete. OpenSearch Dashboards might be unavailable during some or all of the upgrade.

Troubleshooting an upgrade

In-place upgrades require healthy domains. Your domain might be ineligible for an upgrade or fail to upgrade for a wide variety of reasons. The following table shows the most common issues.

Issue Description
Too many shards per node OpenSearch, as well as 7.x versions of Elasticsearch, have a default setting of no more than 1,000 shards per node. If a node in your current cluster exceeds this setting, OpenSearch Service doesn't allow you to upgrade. See Exceeded maximum shard limit for troubleshooting options.
Domain in processing The domain is in the middle of a configuration change. Check upgrade eligibility after the operation completes.
Red cluster status One or more indices in the cluster is red. For troubleshooting steps, see Red cluster status.
High error rate The cluster is returning a large number of 5xx errors when attempting to process requests. This problem is usually the result of too many simultaneous read or write requests. Consider reducing traffic to the cluster or scaling your domain.
Split brain Split brain means that your cluster has more than one master node and has split into two clusters that never will rejoin on their own. You can avoid split brain by using the recommended number of dedicated master nodes. For help recovering from split brain, contact AWS Support.
Master node not found OpenSearch Service can't find the cluster's master node. If your domain uses multi-AZ, an Availability Zone failure might have caused the cluster to lose quorum and be unable to elect a new master node. If the issue does not self-resolve, contact AWS Support.
Too many pending tasks The master node is under heavy load and has many pending tasks. Consider reducing traffic to the cluster or scaling your domain.
Impaired storage volume The disk volume of one or more nodes isn't functioning properly. This issue often occurs alongside other issues, like a high error rate or too many pending tasks. If it occurs in isolation and doesn't self-resolve, contact AWS Support.
KMS key issue The KMS key that is used to encrypt the domain is either inaccessible or missing. For more information, see Monitoring domains that encrypt data at rest.
Snapshot in progress The domain is currently taking a snapshot. Check upgrade eligibility after the snapshot finishes. Also check that you can list manual snapshot repositories, list snapshots within those repositories, and take manual snapshots. If OpenSearch Service is unable to check whether a snapshot is in progress, upgrades can fail.
Snapshot timeout or failure The pre-upgrade snapshot took too long to complete or failed. Check cluster health, and try again. If the problem persists, contact AWS Support.
Incompatible indices One or more indices is incompatible with the target version. This problem can occur if you migrated the indices from an older version of OpenSearch or Elasticsearch. Reindex the indices and try again.
High disk usage Disk usage for the cluster is above 90%. Delete data or scale the domain, and try again.
High JVM usage JVM memory pressure is above 75%. Reduce traffic to the cluster or scale the domain, and try again.
OpenSearch Dashboards alias problem .kibana is already configured as an alias and maps to an incompatible index, likely one from an earlier version of OpenSearch Dashboards. Reindex, and try again.
Red Dashboards status OpenSearch Dashboards status is red. Try using Dashboards when the upgrade completes. If the red status persists, resolve it manually, and try again.
Cross-cluster compatibility You can only upgrade if cross-cluster compatibility is maintained between the source and destination domains post the upgrade. During the upgrade process, any incompatible connections are identified. To proceed with the upgrade, delete the incompatible connections.
Other OpenSearch Service service issue Issues with OpenSearch Service itself might cause your domain to display as ineligible for an upgrade. If none of the preceding conditions apply to your domain and the problem persists for more than a day, contact AWS Support.

Starting an upgrade

The upgrade process is irreversible and can't be paused or canceled. During an upgrade, you can't make configuration changes to the domain. Before starting an upgrade, double-check that you want to proceed. You can use these same steps to perform the pre-upgrade check without actually starting an upgrade.

If the cluster has dedicated master nodes, upgrades complete without downtime. Otherwise, the cluster might be unresponsive for several seconds post-upgrade while it elects a master node.

To upgrade a domain to a later version of OpenSearch or Elasticsearch (console)

  1. Take a manual snapshot of your domain. This snapshot serves as a backup that you can restore on a new domain if you want to return to using the prior OpenSearch version.

  2. Go to https://aws.amazon.com and choose Sign In to the Console.

  3. Under Analytics, choose Amazon OpenSearch Service.

  4. In the navigation pane, under Domains, choose the domain you want to upgrade.

  5. Choose Actions and Upgrade.

  6. Choose the version to upgrade to. If you're upgrading from an Elasticsearch OSS version to an OpenSearch version, the Enable compatibility mode option appears. If you enable this setting, OpenSearch reports its version as 7.10 to allow Elasticsearch OSS clients and plugins like Logstash to continue working with Amazon OpenSearch Service.

  7. Choose Upgrade.

  8. Check the Status on the domain dashboard to monitor the status of the upgrade.

To upgrade a domain to a later version of OpenSearch or Elasticsearch (AWS CLI and SDK)

You can use the following operations to identify the correct version of OpenSearch or Elasticsearch for your domain, start an in-place upgrade, perform the pre-upgrade check, and view progress:

  • get-compatible-versions (GetCompatibleVersions)

  • upgrade-domain (UpgradeDomain)

  • get-upgrade-status (GetUpgradeStatus)

  • get-upgrade-history (GetUpgradeHistory)

For more information, see the AWS CLI command reference and Configuration API reference for Amazon OpenSearch Service.

Using a snapshot to migrate data

In-place upgrades are the easier, faster, and more reliable way to upgrade a domain to a later OpenSearch or Elasticsearch version. Snapshots are a good option if you need to migrate from a pre-5.1 version of Elasticsearch or want to migrate to an entirely new cluster.

The following table shows how to use snapshots to migrate data to a domain that uses a different OpenSearch or Elasticsearch version. For more information about taking and restoring snapshots, see Creating index snapshots in Amazon OpenSearch Service.

From version To version Migration process
Elasticsearch 6.x or 7.x OpenSearch 1.x

  1. Review breaking changes for OpenSearch 1.0 to see if you need to make adjustments to your indices or applications. For other considerations, see the table in Upgrading OpenSearch and Elasticsearch.

  2. Create a manual snapshot of the Elasticsearch 7.x or 6.x domain.

  3. Create an OpenSearch 1.x domain.

  4. Restore the snapshot from the Elasticsearch domain to the OpenSearch domain. During the operation, you might need to restore the .kibana index under a new name: 

    POST _snapshot/<repository-name>/<snapshot-name>/_restore
{
  "indices": "*",
  "ignore_unavailable": true,
  "rename_pattern": ".kibana",
  "rename_replacement": ".backup-kibana"
}

    Then you can reindex .backup-kibana on the new domain and alias it to .kibana.

  5. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 6.x Elasticsearch 7.x

  1. Review breaking changes for 7.0 to see if you need to make adjustments to your indices or applications. For other considerations, see the table in Upgrading OpenSearch and Elasticsearch.

  2. Create a manual snapshot of the 6.x domain.

  3. Create a 7.x domain.

  4. Restore the snapshot from the original domain to the 7.x domain. During the operation, you likely need to restore the .kibana index under a new name: 

    POST _snapshot/<repository-name>/<snapshot-name>/_restore
{
  "indices": "*",
  "ignore_unavailable": true,
  "rename_pattern": ".kibana",
  "rename_replacement": ".backup-kibana"
}

    Then you can reindex .backup-kibana on the new domain and alias it to .kibana.

  5. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 6.x Elasticsearch 6.8

  1. Create a manual snapshot of the 6.x domain.

  2. Create a 6.8 domain.

  3. Restore the snapshot from the original domain to the 6.8 domain.

  4. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 5.x Elasticsearch 6.x

  1. Review breaking changes for 6.0 to see if you need to make adjustments to your indices or applications. For other considerations, see the table in Upgrading OpenSearch and Elasticsearch.

  2. Create a manual snapshot of the 5.x domain.

  3. Create a 6.x domain.

  4. Restore the snapshot from the original domain to the 6.x domain.

  5. If you no longer need your 5.x domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 5.x Elasticsearch 5.6

  1. Create a manual snapshot of the 5.x domain.

  2. Create a 5.6 domain.

  3. Restore the snapshot from the original domain to the 5.6 domain.

  4. If you no longer need your original domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 2.3 Elasticsearch 6.x

Elasticsearch 2.3 snapshots are not compatible with 6.x. To migrate your data directly from 2.3 to 6.x, you must manually recreate your indices in the new domain.

Alternately, you can follow the 2.3 to 5.x steps in this table, perform _reindex operations in the new 5.x domain to convert your 2.3 indices to 5.x indices, and then follow the 5.x to 6.x steps.

Elasticsearch 2.3 Elasticsearch 5.x

  1. Review breaking changes for 5.0 to see if you need to make adjustments to your indices or applications.

  2. Create a manual snapshot of the 2.3 domain.

  3. Create a 5.x domain.

  4. Restore the snapshot from the 2.3 domain to the 5.x domain.

  5. If you no longer need your 2.3 domain, delete it. Otherwise, you continue to incur charges for the domain.

Elasticsearch 1.5 Elasticsearch 5.x

Elasticsearch 1.5 snapshots are not compatible with 5.x. To migrate your data from 1.5 to 5.x, you must manually recreate your indices in the new domain.

Important

1.5 snapshots are compatible with 2.3, but OpenSearch Service 2.3 domains do not support the _reindex operation. Because you cannot reindex them, indices that originated in a 1.5 domain still fail to restore from 2.3 snapshots to 5.x domains.

Elasticsearch 1.5 Elasticsearch 2.3

  1. Use the migration plugin to find out if you can directly upgrade to version 2.3. You might need to make changes to your data before migration.

    1. In a web browser, open http://domain-endpoint/_plugin/migration/.

    2. Choose Run checks now.

    3. Review the results and, if needed, follow the instructions to make changes to your data.

  2. Create a manual snapshot of the 1.5 domain.

  3. Create a 2.3 domain.

  4. Restore the snapshot from the 1.5 domain to the 2.3 domain.

  5. If you no longer need your 1.5 domain, delete it. Otherwise, you continue to incur charges for the domain.