Formed in 2009, the Archive Team (not to be confused with the archive.org Archive-It Team) is a rogue archivist collective dedicated to saving copies of rapidly dying or deleted websites for the sake of history and digital heritage. The group is 100% composed of volunteers and interested parties, and has expanded into a large amount of related projects for saving online and digital history.
Scalyr has moved to reduce the cost of observability by enabling DevOps teams to retain log and event data in S3-compatible object storage systems than use a pay-per-TB-scanned pricing model to interrogate that data.
Company CEO Christine Heckart said the Scalyr Hindsight service will make it possible for IT teams to more cost-effectively store historical event data if they so choose. This option comes at a time when they are under pressure to reduce the total cost of IT as a result of the economic downturn brought on by the COVID-19 pandemic.
Scalyr Hindsight complements an existing real-time log analytics platform that Scalyr provides via a software-as-a-service (SaaS) platform. However, IT teams no longer are required to store data in the Scalyr cloud platform itself.
Heckart said collectively Scalyr Hindsight and the company’s core log analytics platform provide an alternative to custom log analytics platforms based on open source Elasticsearch, Logstash and Kibana (ELK) software that can be difficult to maintain and scale. The core Scalyr log analytics cloud service, built on top of a columnar database, is now capable of ingesting more than 200TBs of data per customer per day in real-time at a price that can be lower than $5 per GB. The Scalyr Hindsight platform only charges fees when data stored is actually scanned.
With the rise of microservices, analytics that can be applied broadly at reasonable price points has become a major issue. Previously, IT teams might have limited their usage of analytics tools to their most important monolithic applications. Because of the way many of those platforms priced their software, IT teams found it necessary to also limit the amount of data that might be stored in these platforms. Given the complexity of microservices-based applications that tend to have lots of dependencies, an observability platform that provides an affordable approach to gaining insights into modern application environments is now required.
Heckart said Scalyr also offloads the management of the observability platform from the internal IT team. Given the massive amounts of data that needs to be stored and managed, Heckert noted it’s not uncommon for data lakes that IT teams manage to become swamps simply because there are no dedicated resources to manage them. It’s not clear how many IT teams might be willing to give up managing their own observability platforms in favor of only needing to access analytics tools, but the one thing that is clear is the cost of support open source platforms tends to climb over time. IT teams may find an ELK-based approach initially sufficient only to discover the platform needs to be replaced as the number of applications that need to managed starts to increase.
Regardless of who manages the underlying log analytics platform, it’s no longer feasible for most internal IT teams to manage modern application environments without the aid of an advanced set of analytics applications.
