[TSDB] Error: the document timestamp X is outside of ranges of currently writable indices

[constanca-m]

Enabling TSDB on some data streams can lead to this error:

... (status=400): {"type":"illegal_argument_exception","reason":"the document timestamp [2023-08-07T00:00:00.000Z] is outside of ranges of currently writable indices [[2023-08-07T08:55:38.000Z,2023-08-07T12:55:38.000Z]]"}, dropping event!

Having a look at how the time window works for a TSDB index, we can see that:

A TSDS is designed to ingest current metrics data. When the TSDS is first created the initial backing index has:

-     an index.time_series.start_time value set to now - index.look_ahead_time
-     an index.time_series.end_time value set to now + index.look_ahead_time

Only data that falls inside that range can be indexed.

But what happens if we receive a document with some delay, and the document timestamp no longer fits the time window of the current writable index? The document ends up being rejected.

To solve this issue, we have on the table this options:

1. Define the look_ahead for each data stream.
   Cons of this approach:
   • When does the customization stop for a time window?
     How to do this:
   1. Go to the manifest.yml file.
   2. Add the following:

   elasticsearch:
     index_mode: "time_series"
     index_template:
       settings:
         index.look_ahead_time: "10h"
   

   Note: Updating the package with this does not cause an automatic rollover on the data stream. You have to do that manually.

2. Update the timestamp of the document being rejected.
   Cons of this approach:
   • Updating the document timestamp might cause us to loose important data.
3. Find a fix to receive the document without a delay.

[lalit-satapathy]

There are few options we have:

• See if we can customise index.look_ahead_time from package with existing configurations. Is this exposed via Fleet? If it is supported, we could expand the time to more than 3 hours.
• Work with elasticsearch to support loading older timestamp document, but this may not be a priority now.

We have certain data streams, where the count of document ingested is too few (For example: 1 docs per day by default in AWS billing?), I would like to question the advantage of enabling TSDB in such data streams. It's OK, to temporarily disable TSDB on such data streams, if there are such issues.

If the first solution takes time, We should disable time series on this data stream (AWS billing), where documents are getting dropped because of old time stamp, so that immediately there is no impact on users of this package.

CC: @felixbarny @ruflin @andresrc

[muthu-mps]

Similarly, Azure billing metrics integrations have a delay in fetching the billing metrics. In this case, we use two different APIs to collect metrics. The Usage details API has a delay of -24 hours and the consumption API has a delay of -48 hours.

[lalit-satapathy]

Similarly, Azure billing metrics integrations have a delay in fetching the billing metrics. In this case, we use two different APIs to collect metrics. The Usage details API has a delay of -24 hours and the consumption API has a delay of -48 hours.

How many documents are approximately generated by Azure Billing metrics per day?

[muthu-mps]

Similarly, Azure billing metrics integrations have a delay in fetching the billing metrics. In this case, we use two different APIs to collect metrics. The Usage details API has a delay of -24 hours and the consumption API has a delay of -48 hours.

How many documents are approximately generated by Azure Billing metrics per day?

The documents are grouped by product. Single document is generated for each product every day with the usage cost information. The number of documents depends on the number of products and the resource groups etc...

[lalit-satapathy]

@constanca-m Please confirm if these below are feasible, short term?

See if we can customise index.look_ahead_time from package with existing configurations. Is this exposed via Fleet? If it is supported, we could expand the time to more than 3 hours.

1. Let's confirm if we are able to configure index.look_ahead_time in the package?

Update the timestamp of the document being rejected

2. We could also explore to use the current timestamp for the ingested document? Is that a feasibility?

3. If these two options are ruled out, we can disable TSDB on this data stream.

[constanca-m]

Let's confirm if we are able to configure index.look_ahead_time in the package?

I will check this, but I don't think this is the best solution. We don't know the delay of the document, so adjusting the time window with a new custom look ahead could be just by chance.

We could also explore to use the current timestamp for the ingested document? Is that a feasibility?

For which packages exactly @lalit-satapathy ? I don't know enough about AWS to say if that solution will cause any problems.

[lalit-satapathy]

Summary of short-term solution in TSDB integration sync-up:

This is a known issue, where TSDB only indexes recent metrics data in a given time window as defined by: index.look_ahead_time and late arrival is not accepted.

• Disable TSDB on such data stream now (short-term)
• Extending the time window to 24 hours by configuring index.look_ahead_time may have some side impact. @martijnvg to track a separate issue in elasticsearch to discuss what is the best way to handle it. There was also a suggestion that look back time should be configured differently from look ahead time.

[lalit-satapathy]

Let's confirm if we are able to configure index.look_ahead_time in the package?

@constanca-m,
any updates on this above, if it working?

Can we also have PR ready to TSDB on the AWS billing package?

[constanca-m]

Let's confirm if we are able to configure index.look_ahead_time in the package?

@constanca-m, any updates on this above, if it working?

I don't know how to add the parameter in the main manifest, but you can change the settings on the template and that works.

Can we also have PR ready to TSDB on the AWS billing package?

Yes, it is linked on this issue.

[lalit-satapathy]

I don't know how to add the parameter in the main manifest, but you can change the settings on the template and that works.

Once we confirm that it works manually directly, we can check if Fleet/package spec supports it next.

[constanca-m]

I don't know how to add the parameter in the main manifest, but you can change the settings on the template and that works.

Once we confirm that it works manually directly, we can check if Fleet/package spec supports it next.

It works, it is in Elastic documentation, also linked in the description of this issue. More precisely, you can find the example here.

[constanca-m]

Let's confirm if we are able to configure index.look_ahead_time in the package?

@lalit-satapathy I updated the description of this issue to include how to do that.