As we close out 2018, we at DevOps.com wanted to highlight the five most popular articles of the year. Following is the second in our weeklong series of the Best of 2018.
Security information and event management (SIEM) is the cornerstone of IT security. All other network solutions are merely data flows that feed into an organization’s SIEM. Not all SIEMs are created equal, and their capabilities can vary wildly. Choosing the right one for your needs can mean the difference between detecting a security weakness and becoming just another statistic.
A SIEM solution is a combination of a security event management (SEM) system and a security information management (SIM) system. SEMs monitor servers and networks in real time, while SIMs store the data.
Both SEMs and SIMs provide analysis. SEMs focus on real-time event correlation, alerting and those fancy network operations center (NOC) “something has gone wrong” dashboards that one may come across in the movies. SIMs focus on bulk data computational analysis (BDCA) of large quantities of logs.
In other words, a SEM is designed to tell you when something is going down as it happens, and SIMs are designed to spot the subtle attacks that SEMs don’t catch. Merging the two into a SIEM solution seems like a natural fit.
Related Content:
The Value of Commercial Open Source
How Developers Can Take a More Proactive Approach to Security
SIEM stacks usually consist of at least three components for data collection, data storage and data analysis. The data in question is frequently log files, but can also be netflow traffic or other real-time data feeds.
There are a number of SIEM tools on the market, both open source and commercial. With the rise of DevOps, containers and other modern application development methods, the open source solutions are seeing a resurgence of interest. Let’s take a look at some of the top open source SIEM tools out there.
OSSEC
OSSEC is a popular host-based intrusion detection system (IDS) that works with Linux, Windows, MacOS and Solaris, as well as OpenBSD and FreeBSD. OSSEC is composed of two components: the host agent (responsible for collecting the logs) and the main OSSEC application (responsible for processing the logs.)
There is also a now-deprecated GUI, but since other open source solutions do a better job of data visualization, the OSSEC project recommends using those solutions instead. Popular open-source data visualization tools include Kibana and Grafana.
OSSEC directly monitors a number of parameters on a host. This includes log files, file integrity, rootkit detection and Windows registry monitoring. OSSEC also can perform log analysis from other network services, including most of the popular open source FTP, mail, DNS, database, web, firewall and network-based IDS solutions. OSSEC can also analyze logs from a number of commercial network services and security solutions.
OSSEC has a number of alerting options and can be used as part of automated intrusion detection or active response solutions. OSSEC has a primitive log storage engine. By default, log messages from host agents are not retained. Once analyzed, OSSEC deletes these logs unless the <logall> option is included in the OSSEC manager’s ossec.conf file. If this option is enabled, OSSEC stores the incoming logs from agents in a text file that is rotated daily.
Whether OSSEC counts as a “full” or “proper” SIEM is the source of numerous internet debates. OSSEC does the hard work of a SIEM: It collects data and analyzes it. OSSEC is at the core of a number of other SIEM solutions, and part of any number of application stacks that pair OSSEC with a more advanced long-term log retention system and advanced visualization capabilities.
Snort
Snort is a popular network-based IDS. Network-based IDS systems differ from host-based IDS systems in exactly the way one might expect—while a host-based IDS lives on an individual server or endpoint, network-based IDS systems live further out on the network, and scan all traffic that they can see.
Snort’s purpose is to sniff, log and perform real-time analysis on network flows. It can display a real-time stream of packets to a console, dump packets to a log file or perform analysis.
Snort relies on output plugins to determine how and where it will store the information it generates. Snort can store this information in a flat text file, output to a database, a Unix Socket, a pager or cell phone and any number of other destinations.
Snort has a number of filtering and pattern matching options. This means that snort doesn’t have to display, log or analyze all packets traversing a network link. It can restrict itself to information pertaining to a specific host or matching specific patterns—something that is of increasing utility as network throughput of 100Gbps on individual network links becomes more common.
Configuring Snort requires careful planning. Dumping 100Gbps worth of packets to a flat text file isn’t the best plan, and most organizations aren’t exactly going to be able to afford to ship that kind of data off to a public cloud-based log repository, either. Judicious use of Snort’s filtering and analysis capabilities combined with output plugins are required to make Snort practicably useful.
With a narrowly configured set of alerts, Snort’s ability to send alerts directly to a cell phone becomes meaningful. A broader set of filters could be used to pass data to a long-term storage repository for more in depth analysis by another tool.
Like OSSEC, Snort relies on third-party solutions in order to perform visualizations. As one of the earliest network-based intrusion detection solutions available, there are a number of solutions (both open source and commercial) that exist only to visualize Snort data, making the lack of a built-in visualization system somewhat irrelevant.
Like OSSEC, Snort’s qualification as a SIEM solution is somewhat debatable. Snort collects data and analyses it, and is a core component to more complete SIEM solutions. Snort is also part of any number of application stacks which add log retention and advanced visualization capabilities.
It is not unusual to see Snort and OSSEC working together to each fill different niches in the grand scheme of data center SIEM needs. It is also perfectly normal to see them deployed independently, especially as part of cloud-native applications, where monitoring needs can be narrower.
Suricata is a competing open source network-based IDS that is frequently used in place of Snort.
ELK
The ELK stack is arguably the most popular open source SIEM tool available, though, like OSSEC and Snort, there is room for debate about whether the ELK stack even qualifies as a SIEM on its own.
The ELK stack consists of the open source products Elasticsearch, Logstash and Kibana. Logstash is a receiver for log data from virtually any source. It can filter, process, correlate and generally enhance any log data that it encounters.
Elasticsearch is the storage engine, and is one of the best solutions in its field currently available. Kibana is the visualization portion of the equation, and it is hands down one of the best visualization system the open source community has yet produced.
Logstash, a part of the ELK stack, uses input plugins to collect logs; however, it also can accept input from more purpose-built solutions such as OSSEC or Snort. Combined, the ELK Stack’s log processing, storage and visualization capabilities are functionally unmatched. The solution is powerful, but that power comes at the price of complexity and a high TCO. Cloud offerings such such as Logz.io offer a hosted ELK alternative.
The ELK stack forms the core of a growing number of commercial SIEM offerings. These solutions bring expertise in standing up, configuring and managing the applications in the stack. They also may offer canned or easier-to-design filters for Logstash than are available in the native open source solution. Both approaches offer significant added value.
Prelude
Prelude is a SIEM framework that unifies various other open source tools. It is the open source version of the commercial tool by the same name. Prelude aims to fill the roles that tools such as OSSEC and Snort leave out.
Prelude accepts logs and events from multiple sources and stores them all in a single location using the Intrusion Detection Message Exchange Format (IDMEF). It provides filtering, correlation, alerting, analysis and visualization capabilities. The open source version of Prelude is significantly limited when compared to the commercial offering in all of these capabilities.
OSSIM
If ELK isn’t the most popular SIEM, then OSSIM likely wins the crown. The open source version of Alien Vault’s Unified Security Management offering, OSSIM is a framework like Prelude. OSSIM combines its native log storage and correlation capabilities with numerous open source projects to build a complete SIEM.
The open source projects included in OSSIM include FProbe, Munin, Nagios, NFSen/NFDump, OpenVAS, OSSEC, PRADS, Snort, Suricata and TCPTrack. The inclusion of OpenVAS is of particular interest, as OpenVAS is used both for vulnerability assessment by correlating IDS logs with vulnerability scanner results.
Like Prelude, the open source OSSIM is not as feature-rich as its commercial sibling. Both solutions work fine for small deployments, but experience significant performance issues at scale, ultimately driving organizations towards the commercial offerings. Log management capabilities in the open source version of OSSIM, for example, are virtually nonexistent.
Conclusion
A complete SIEM solution includes the ability to collect information from various data sources, store the data long term, perform analysis on that data and visualize it in useful ways. There are no clear open source winners that deliver all capabilities.
When selecting an open source SIEM you are either going to have to use feature and performance limited solution designed to drive organizations into the arms of a commercial solution, or you are going to have to combine multiple open source projects to achieve your goals.
Consider the ELK stack as an example. On its own, the ELK Stack can be a high-quality SIEM solution. Its input filters include public cloud infrastructure monitoring capabilities the other open source solutions can’t match.
That said, the more dedicated collection and analysis tools, such as OSSEC and Snort, can be far more powerful when dealing with on-premises solutions. These solutions can be combined. OSSEC and Snort can feed the ELK Stack, creating a more complete solution that meets the needs of the majority of organizations.
Open source SIEM solutions—in whole or in part—are versatile and powerful. They require experience, expertise, and above all time to deploy properly. It is for this reason that commercial offerings dominate the SIEM landscape, even when open source tools lie at the core of those commercial offerings.
Having 80 percent of your SIEM solution handled for you is better than having to do all of it yourself. Commercial solutions handle installation, basic configuration and provide filters, correlation configurations and visualisation designs for the most common use cases. Don’t underestimate the value of these commercial features: There are a seemingly unlimited number of things to monitor in today’s data centers, and none of us have time to manually configure applications to watch them all.
