Log Management and SIEM: The Network’s Trusty Watchdogs

Managing a data center is more complicated than ever with the growing sophistication and interconnectedness of enterprise applications, networks and now private and public cloud resources. The challenges created by these interwoven infrastructures are difficult to overcome without the right tools.

One important weapon in the data center manager’s arsenal is Log management/security information event management (SIEM), which can provide deep and wide visibility into a data center’s many moving parts and serve as both an early warning system and a forensic tool for finding and fixing root causes of problems. A complete Log Management/SIEM solution should collect and analyze all log data to provide meaningful information on operations events as well as audit-related activities, such as privileged user activity — not just security and/or compliance events.

Logs Capture Security Incidents, but Need to be Mined

Not too long ago, traditional SIEMs focused almost exclusively on security-related events. One drawback was the fact that most operated within the confines of a subset of log data, oftentimes leaving out key data points that may not appear to be important on their own. However, when trying to piece together complex problems or threats that span time and different IT resources, these missing pieces of data become much more significant.

According to the most recent Verizon Data Breach Investigations Report, 86 percent of the time, companies had evidence of the breach within their log files.

Let’s take a rather routine example: brute force password attacks. Detecting the most basic version of this attack is well-understood and implemented within traditional SIEMs. However, what if the attack is more sophisticated? What if the attack was successful? Would you know and could you respond in time?

This is where most traditional SIEMs fall down because they have generally only been interested in exception-based logs (“events”). Successful logins are often not considered exceptions. So while detecting multiple failed logins on the same host is important, what about flagging the successful login that occurred a day after the attack began? Only when we can see across all the data at the log layer do we get the required visibility to detect both successful and attempted brute force attacks.

This visibility also provides us with a means of understanding immediately and completely what other systems the user accessed after compromising the initial host. Visibility into the log layer for authentication logs can also help us detect accounts that may have previously been compromised or acquired through fraud and extortion.

Logging the Insider Threat

Generally speaking, significant resources and capabilities have been brought to bear in thwarting the external threat. However, if a user with malicious intent has found his or her way in, or comes in through the front door every morning, it is usually game over. Most concerning are privileged users that have the proverbial “keys to the kingdom.u201d What if one of these users has gone rogue? What if their account was compromised and is in the hands of an outsider? What if they are able to hide their tracks? Next-generation SIEM provides powerful capabilities for not only monitoring the activities of privileged users but also detecting compromised accounts and rogue users.

Simply ensuring that all log messages pertaining to user activity are collected and safeguarded is a great start — after all, the first thing an insider threat will do is delete the audit log. By adding intelligence and contextualization to each log message, the picture of activity and intent becomes much clearer.

For example, the ability to easily differentiate between external, internal and outbound logins reveals where users are connecting to and from. The ability to understand the physical location from which a login originated provides critical context. This context can enable the detection of accounts currently compromised by automatically detecting simultaneous logins using the same credentials from two different locations. By having visibility to all access logs, suspicious users can be detected based on file access and network transmission patterns. Knowing that a user downloaded 1,000 files from a file server and subsequently transferred 10 MB of data to a foreign country is a good example. Only SIEMs rooted in log management can provide this level of automated analysis and intelligence.

Context Brings Latent Activity to the Surface

Deploying a next-generation SIEM is like switching from watching a black and white movie on a circa 1950s TV to watching it in 3D Blue-ray on a 60-inch plasma. Of course, for this analogy to be accurate, we’d have to remove 99 percent of the frames in the black and white version.

The increased visibility and awareness of the network, the systems, the users, and the data can only be described as staggering. This visibility almost always brings latent issues to light, issues that span operations, audit and security. On the operations front, visibility into system and application logs often provide early warnings of symptoms that, left alone, would have resulted in outages. In the event the outage wasn’t avoided, logs can be mined to see if the error message that preceded the failure has been observed on other systems.

When we combine system and network logs with security device logs, additional opportunities are presented. One of the inherent challenges with intrusion detection is accuracy and the detection of zero-day attacks. By having access to all log data, we can correlate suspicious network activity against suspicious system activity. For instance, if an IDS detects an SQL injection attack against a database, and that database server subsequently begins to generate warning and error messages in the system log, this is something worth immediately investigating. Only the combination of traditional SIEM log sources with system logs makes this possible.

Netflow and firewall logs can provide evidence of systems already compromised or verification of a successful compromise. The ability to search across all egress points for SMTP traffic leaving the organization from unauthorized servers can identify compromised hosts sending spam. The ability to detect outbound data transfers directly from file servers or any data transfer to a country where no business operation exists certainly warrants further investigation. Observing an attack where data was sent immediately back to the attacker over the next 30 minutes would certainly send up red flags.

In Conclusion

These are just a few use cases that demonstrate how next-generation log management/SIEM solutions can provide an early warning system for monitoring and responding to security threats.

However, in order for these tools to provide deep visibility into data center and network activity, they must collect the full spectrum of log data and be able to enrich these raw records with contextual information. This combination can piece together evidence to reveal threats that would otherwise remain undetected until the damage was done, and help transform daily security operations management into more of a proactive rather than reactive process.