Over the years there has been a steady flow of technologies developed by web-scale companies that have benefited the larger IT community by becoming open source projects. The latest two examples of this trend are Fossor, a tool for investigating broken hosts and services, and Ascii Etch, a Python library designed to complement Fossor that takes streams of numbers and turns them into visual graphs using ascii characters. Both were developed by LinkedIn.
Steven Callister, senior site reliability engineer for LinkedIn, said Fossor, which is Latin for grave digger, was developed to provide a mechanism to mine logs to more easily discover unknown issues such a memory fragmentation. Typically there are 10 commands that DevOps teams employ to investigate whether certain issues might be occurring. Fossor eliminates the repetitive tasks associated with issuing the commands manually by enabling them to be automated then executed in parallel. This approach allows DevOps teams to continuously execute tests to discover any potential unknown issues that might arise at any time, he said.
Callister noted LinkedIn decided to open-source Fossor to give back to the IT community in a way that is extensible. Fossor has two basic components: a core engine and a mechanism for creating plugins that DevOps teams can use to add tests specific to their site architecture. Linkedin is hoping that an entire library of plugins will emerge around a Fossor open source community.
Of course, no organization is obligated to contribute the plugin they create to the open source community. There may be information in that plugin an organization considers too proprietary to share. But Callister said there are plenty of use cases where a plugin developed for one type of site might apply to another.
One of the single biggest issues most IT organizations struggle with in managing IT at scale is a lack of engineers on staff. The average IT organization tends to rely on administrators that don’t have much in the way of coding skills. Organizations that manage IT environments that operate at web-scale typically have hired software engineers capable of developing their own tools. Open source projects such as Fossor provide a means through which software engineers working at web-scale companies can share their skills and expertise with the rest of the IT community.
It’s not clear to what degree open source tools will replace commercial software tools. It’s not unusual for an IT vendor to leverage an open source project to create a commercial offering around which they provide extensions and professional support. But many DevOps teams are becoming more self-reliant out of necessity, either technical or budgetary. The challenge now becomes sorting through all the open source code available to determine which of the many tools available today solves a problem well enough to not only rely on every day, but also to make ongoing contributions.
