Smoking Out Attackers Hiding in Encrypted Data

Today, most enterprises have a number of network security appliances that provide protection against attacks aimed at enterprise computing resources, as well as prevent the loss of sensitive enterprise data due to deliberate or unintentional leakage. These security appliances work by matching network traffic with threat signatures or tracking application state as a means to detect suspicious behavior.

One guaranteed way to ensure that these types of security appliances will not detect threats or data loss is to prevent signature matching and state tracking from working; the easiest way to do this is to encrypt network traffic.

If network encryption were used only for a small portion of network traffic, then the fact that threats could be hidden inside encrypted traffic would not be a major concern. However, Secure Sockets Layer (SSL) is widely used to protect network traffic, and the amount of SSL traffic has grown dramatically in the last year.

HTTPS, which is simply Web traffic running over SSL, is the basic interface for most Web 2.0 and cloud applications and is also the default interface for widely used applications such as Gmail, Yahoo mail and Google apps.

A typical enterprise network today may find that anywhere between 20 and 80 percent of the traffic on the network is SSL encrypted, and this number is only going to increase with time.

Double-Edged Sword

All SSL traffic is encrypted and, unfortunately, the same technology that is being used to encrypt traffic to prevent attacks is also enabling attacks to occur. Because SSL makes data in transit secure and encrypted, attackers are able to use the technology to effectively hide their attacks. Security infrastructure put in place by enterprises today to protect the data are blind to the contents of SSL encrypted communications.

For example, in the majority of enterprises, more than 50 percent of the traffic in and out of the network may be completely bypassing the security architecture that is in place. Attacks on enterprise Web servers will not be blocked by an Intrusion Prevention System (IPS); malware from Internet sites using HTTPS will compromise enterprise desktop machines; and outgoing webmail over HTTPS may be leaking sensitive information, despite the presence of a Data Loss Prevention (DLP) system.

For an enterprise concerned about security, which most are, addressing the risks posed by threats hidden within encrypted SSL traffic is clearly not something that can be ignored.

If SSL were being used for no purpose, then the problem could be solved by simply avoiding or preventing the use of SSL. However, the reality is that SSL is used to provide much-needed security for a wide range of network applications.

For example, allowing webmail that does not use SSL would mean that enterprise emails would cross the public network as clear text, easily accessible to any attacker. Additionally, using Google apps without SSL would expose your sensitive documents to anyone who could access them as they crossed the Internet.

With the rise in online shopping, e-commerce traffic without SSL would be unlikely to succeed, as most users know to only trust a site if their browser shows the padlock symbol, indicating SSL is in use.

The challenge is how to allow the use of encrypted traffic while also preventing it from making network security appliances useless. There are several approaches for addressing this issue. Each offers a different way to make the unencrypted content of a network flow available to the security appliance(s) while still retaining an end-to-end encrypted flow between client and server to ensure maximum security. The approaches differ in the impact they have on application performance, complexity of network configuration and requirements for client customization.

Full Proxy

One approach is to make use of a full proxy device that completely terminates and then re-originates an application layer flow. This approach requires that only applications supported by the proxy are allowed to use SSL. As long as the full proxy can support termination and re-origination of SSL and the application layer, then it can be used to gain access to the unencrypted content of the flow.

While the content is unencrypted, the full proxy can perform security checks on the data, or it can hand a copy of the unencrypted data to a separate security appliance that will perform the security checks. If the full proxy carries out security checks internally, then the checks are limited to those supported by the proxy and existing security appliances are made useless.

Full proxies also normally require that the client machine is configured to know about the proxy and direct the relevant application traffic flows directly to the proxy. However, this requires significant management overhead. It also means that it is possible for a client to be configured not to make use of the proxy, which would then bypass the security.

As full proxies terminate the complete application layer and all the underlying network protocol layers, they tend to have a negative impact on performance. In particular, they can impose significant latency, which may cause the application layer to be sluggish. The overhead of completely terminating and re-originating the application layer is significant — and that is even before taking into account the processing burden of terminating and re-originating the SSL layer.

Transparent SSL Proxy

An alternative approach is to use a transparent SSL proxy in conjunction with the existing security appliances that are already deployed in the network. A transparent SSL proxy does not need to understand or process the application layer protocols and is optimized for terminating and re-originating the SSL protocol layer.

A transparent proxy is deployed at a point in the network where all traffic of interest is present. It will detect all SSL traffic, ideally by deep packet inspection, and can decrypt and re-crypt the traffic in order to gain access to the unencrypted payload. This unencrypted payload is then packaged into a “generated” TCP flow and sent to one or more security appliances that already exist in the network.

Once the existing security appliances see a non-encrypted flow, they can do their job and detect any threats or data leakage. If the security appliance is a filtering device, such as an IPS, then it will drop any traffic containing threats, and the transparent SSL proxy will detect this and drop the corresponding SSL flow.

If the security appliance is something like an IDS or a Network Forensics device, then it will generate reports regarding the threats detected in exactly the same ways as it would for non-encrypted traffic.

Since traffic does not need to be sent to the proxy explicitly, transparent proxies do not require configuration of the client machine. Additionally, because they are not terminating and re-originating application layer protocols, transparent proxies do not introduce significant additional latency and can operate at very high speeds. It is perfectly possible to have a transparent proxy operating at line rate in a 1 GigE or 10 GigE network link.

As transparent proxies allow continued use of the existing security appliances within an enterprise network, they do not create a performance bottleneck or additional latency, and they do not require changes to network or client configuration.

As the amount of encrypted traffic continues to increase, enterprises need to find ways to ensure their security appliances installed are not made useless by SSL traffic. Transparent proxies represent the best approach to dealing with the risks from threats hidden within SSL traffic.