Kernel String Matching and IPS Performance

The iptables QUEUE and NFQUEUE targets are used by both open source and commercial Intrusion Prevention Systems. These targets allow a userspace process (the IPS in this case) to acquire packet data from the Linux kernel via a netlink socket and set verdicts on whether packets should be forwarded only after the IPS has sent the packets through its detection engine. This can provide a effective means from protecting other systems from attack (subject to the usual concerns over false positives).

Using iptables provides a flexible means for placing a QUEUE rule within an existing iptables policy. For example, if you are not running a DNS server behind an iptables firewall, then you can use the DROP target against any attempts to communicate with a DNS server before sending such packets to a userspace IPS - which would needlessly place additional load on the CPU when such traffic can just be dropped from within the kernel directly before it ever hits the QUEUE rule:

# iptables -A FORWARD -p udp --dport 53 -j DROP
# iptables -A FORWARD -j QUEUE

This is a nice consequence, but is only possible in this case because there is no internal DNS server to attack in the first place. What about using facilities provided by iptables to reduce the packet load that is queued to a userspace IPS for servers that do exist? Say, for a web server?

An IPS such as snort_inline can be deployed with a large signature set, and most Snort signatures use the content or uricontent keywords to search packet application layer data for malicious content. In the context of snort_inline, usually the QUEUE or NFQUEUE rule is the default rule in the FORWARD chain - this ensures that snort_inline inspects all packets before they are forwarded through the system. So, you could say the iptables policy has a "default QUEUE stance".

However, by using the iptables string match extension and an iptables policy built by fwsnort, we can change the "default QUEUE stance" to "only QUEUE packets that match a Snort content field". The resulting performance gain can be substantial as we will see below. Fwsnort accomplishes this by parsing the Snort signature set that is deployed in snort_inline, and then building an iptables policy that makes heavy use of the string match extension to ACCEPT all packets in the FORWARD chain that do not match any of the content fields in the signature set, and QUEUE those that do. In general, this greatly reduces the number of packets that have to be queued to snort_inline since (usually) most traffic is not malicious. As a simple example, consider the following Snort signature:

alert tcp any any -> any any (msg:"fwsnort download"; content: \
"fwsnort/download"; classtype:web-application-attack; sid:12325678; rev:1;)

This signature is for illustration purposes only and is written just to force snort_inline to inspect all TCP packets for the string "fwsnort/download". With this signature deployed in snort_inline on one of my systems (an old PIII with 100Mb Ethernet interfaces running the 2.6.20.3 kernel), I measured the following throughput with netperf. This test used a default QUEUE rule in iptables, so every TCP packet is queued to snort_inline:

# netperf -v 2 -L 192.168.50.1 -H 192.168.50.2
TCP STREAM TEST from 192.168.50.1 (192.168.50.1) port 0 AF_INET to \
192.168.50.2 (192.168.50.2) port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.02      60.07

Alignment      Offset         Bytes    Bytes       Sends   Bytes    Recvs
Local  Remote  Local  Remote  Xfered   Per                 Per
Send   Recv    Send   Recv             Send (avg)          Recv (avg)
  8       8    0      0       7.524e+07  16384.33  4592    1444.75  52076

Maximum
Segment
Size (bytes)
  1448

The throughput number above indicates that netperf was able to achieve 60.07 Mb/sec over the life of the 10 second test. Now, let's use fwsnort to convert the Snort signature into an equivalent iptables rule, and deploy it:

# fwsnort --snort-sid 12325678 --QUEUE
[+] Parsing Snort rules files...
[+] Found sid: 12325678 in test.rules
    Successful translation.

[+] Logfile: /var/log/fwsnort.log
[+] Snort rule set directory for rules to be queued to userspace:
    /etc/fwsnort/snort_rules_queue
[+] iptables script: /etc/fwsnort/fwsnort.sh
# /etc/fwsnort/fwsnort.sh
[+] Adding test rules.
    Rules added: 2
[+] Finished.

Now, executing the netperf throughput test generates the follow result:

# netperf -v 2 -L 192.168.50.1 -H 192.168.50.2
TCP STREAM TEST from 192.168.50.1 (192.168.50.1) port 0 AF_INET to \
192.168.50.2 (192.168.50.2) port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.01      94.11

Alignment      Offset         Bytes    Bytes       Sends   Bytes    Recvs
Local  Remote  Local  Remote  Xfered   Per                 Per
Send   Recv    Send   Recv             Send (avg)          Recv (avg)
    8    8      0     0       1.178e+08  16384.74  7189    1440.01  81798

Maximum
Segment
Size (bytes)
  1448

So, in this test we achieved 94.11 Mb/sec sustained throughput - that's about a 57% speed increase!

This may sound good, but as always there are some tradeoffs. First, because this strategy causes iptables to only selectively queue packets to userspace, snort_inline does not have the opportunity inspect an entire stream. This implies that the stream preprocessor needs to be disabled since otherwise signatures with the flow keyword will never match. Secondly, because the preliminary signature match is performed by iptables, taking into account application layer encodings (such as URL encoding used in web servers) is not possible. This results in a reduced ability to detect attacks. In some cases however, performance requirements may override these considerations. Also, some attacks do not need to bother with attempts at obfuscation because the target environment is so rich.

Significant work needs to be performed to validate this as a viable strategy for increasing IPS performance, but these preliminary results are encouraging. On a final note, this strategy is applicable to any IPS that makes use of the iptables queuing mechanism - not just snort_inline.

Also, thanks are in order to Hank Leininger of KoreLogic Security for suggesting the combination of the QUEUE target and string matching as a way to speed up inline Snort implementations at a talk I gave about Linux Firewalls at ShmooCon 2007. I implemented the --QUEUE and --NFQUEUE targets in fwsnort-1.0 as a result of meditating on this for a while.

fwsnort-1.0 release

The 1.0 release of fwsnort is ready for download. This release is a major update that adds the ability to build an iptables policy against the NFQUEUE or QUEUE targets in order to perform preliminary content matching in the Linux kernel using the iptables string match extension. Here is the full ChangeLog:

• Major update to include support for the NFQUEUE and QUEUE targets with new command line options --NFQUEUE and --QUEUE. This changes the default LOG target to the NFQUEUE or QUEUE targets instead, and at the same time builds a parallel Snort rule set in the /etc/fwsnort/snort_rules_queue directory. Every Snort rule in this directory has at least one "content" keyword, which fwsnort uses in the resulting iptables policy. This policy only sends those packets to snort_inline via the NFQUEUE or QUEUE target that match a content field within some Snort rule. The end result is that snort_inline should run faster because the vast majority of packets (which are not malicious) are processed via the Linux kernel without ever having to be sent to userspace for analysis. There is a tradeoff here in terms of attack detection; snort_inline does not receive all packets associated with a stream, so it cannot detect attacks quite as effectively (snort_inline does not have an opportunity to look at reassembled buffers). However, this trade off may be acceptable for large sites where performance is more important.
• Bug fix to remove any existing jump rules from the built-in INPUT, OUTPUT, and FORWARD chains before creating a new jump rules. This allows the fwsnort.sh script to be executed multiple times without creating a new jump rule into the fwsnort chains for each execution.
• Added the -X command line argument to allow fwsnort to delete all of the fwsnort chains; this emulates the iptables command line argument of the same name.
• Minor output enhancements and bugfixes to give more insight into the translation process. For example, if fwsnort is run in --snort-sid mode but is unable to translate the specified signatures, the user is notified. Also, any existing /etc/fwsnort/fwsnort.sh script is not archived and erased until fwsnort is actually going to write a new one.
• Added sid values to iptables comment match string.
• Bugfix for iptables string match --from and --to values to skip past packet headers. This is an approximation until a new --payload option can be added to the string match extension.
• Added a single iptables rule testing API internally within fwsnort; this adds a measure of consistency and removes some duplicate code.
• Added fwsnort mailing list at SourceForge.

Richard Bejtlich References Single Packet Authorization

Richard Bejtlich, founder of TaoSecurity, has made a posting to his blog that references Single Packet Authorization in the context of making it more difficult to discover and communicate with SSHD from arbitrary IP addresses. One of the points he makes in his blog posting is that it is important to force a would-be attacker to fight your fight instead of allowing an attacker to dictate the stage on which a battle is fought. For example, if an attacker possesses a zero-day exploit for SSHD, then the mere ability to access SSHD from an arbitrary source is taking a dangerous risk. One of Richard's readers posted a particularly cogent comment about the concept of "security through obscurity":

As for changing the port for SSH - my belief is that at times we get so hung up on the tags - "security by obscurity" that we cease to see the valid strategies that they can cover. Simply because we denigrate the people who rely solely on it doesn't mean that it isn't part of a valid defensive scheme.

I completely agree. Fortunately however, Single Packet Authorization does not suffer from security through obscurity. SPA essentially ties access through a default-drop packet filter to a protected service via strong cryptographic means. A nice byproduct of this is that services are not advertised to the world, but this is a consequence of the default-drop packet filter. The security of SPA does not rely on keeping the mechanism secret and hoping that it is not discovered; all source code to every SPA implementation that I am aware of is open and published to the world. SPA is no more security through obscurity than passwords or encryption algorithms themselves.

OSCON 2007 Talk on iptables Visualization

At the O'Reilly Open Source Convention of 2007 (July 23-27, Portland, Oregon) I will give a talk about visualizing iptables log data. Here is the talk abstract:

The iptables logging format provided by the Netfilter project contains a wealth of detailed information about network traffic. Nearly every interesting field in the network and transport layer headers is logged by iptables. By combining the graphing capabilities of AfterGlow with the attack detection capabilities of psad it is possible to render eye-catching graphical visualizations of network attacks. These visualizations can expose important relationships between attackers and their targets that are difficult to acquire in via non-graphical means. This talk will analyze iptables log data from two sources: the Honeynet Project, and from an Internet-facing Linux system. This data contains instances of the Nachi and Slammer worms, and suspicious outbound SSH and IRC connections from compromised systems. In addition, material from the book "Linux Firewalls: Attack Detection and Response" will be presented to show you how to deploy psad on a live firewall. As more people run Linux, mountains of iptables log data are piling up. It is time to maximize the effectiveness of this data and mine it for suspicious traffic and network-based attacks. This talk will show you how.

CipherDyne Blogspot Page Created

The current CipherDyne website is managed by a set of custom perl scripts that creates and edits a series of static HTML pages. This eliminates CGI scripts, which is good for security, but it means that people cannot leave comments for stories that are posted to the CipherDyne blog. I have created a parallel blog on Blogspot so that comments can be tied to stories. Another consequence is that Google searches for content related to CipherDyne projects and concepts may become more effective depending on how the new blog is indexed. The content on the new blog will be largely the same as the original, but occasionally I may post some stories there that are more speculative in nature than the generally technical or "software release" posts on the main blog. If people have comments about the stories, please post them on the new blog for all to see. In the spirit of open source development, this is the best way to share new ideas about the topics discussed here.

fwsnort Mailing List Started

The fwsnort project has been around for several years now, but has never had a dedicated forum for discussion of development and deployment issues that are unique to making heavy use of the Netfilter string match extension as done by fwsnort. That's changed now with the introduction of the fwsnort mailing list generously provided by SourceForge. If you are interested in learning more about fwsnort and would like to ask questions or post comments, the mailing list is the place to do it!

Protecting SSH Servers with Single Packet Authorization

The current issue (May '07) of the Linux Journal, contains part II of my series on Single Packet Authorization (SPA). The article is entitled Protecting SSH Servers with Single Packet Authorization, and appears to not be available yet through the LJ website so I suppose that it may only be in the print edition for a while. Part II is a hands-on treatment of fwknop operations, and covers SPA with both symmetric and asymmetric ciphers. here is a short excerpt from the article:

"...This article gets away from theory and concentrates on the practical application of SPA with fwknop and iptables to protect SSHD from reconnaissance and attack. With this setup on a Linux system, no one will be able to tell that SSHD is even listening under an nmap scan, and only authenticated and authorized clients will be able to communicate with SSHD.
     To begin, we require some information about configuration and network architecture. This article assumes that you have installed the latest version of fwknop (1.0.1 as of this writing) on the same system where SSHD and iptables is running. You can download fwknop from http://www.cipherdyne.org/fwknop/ and install either from the source tar archive by running the install.pl script, or install via the RPM for RPM-based Linux distributions..."

Techno Security 2007 Talk on Single Packet Authorization

At the Techno Security 2007 conference (June 3-6, Myrtle Beach, South Carolina) I will be giving a talk entitled "Zero-day Attack Prevention via Single Packet Authorization". My intention for this talk is to illustrate practical usages of fwknop with an emphasis on live demonstrations of the technology. There have also been some interesting developments in the Single Packet Authorization world since I last gave a talk on the topic at ShmooCon, 2006. In particular, Sebastien Jeanquier wrote a Master's Thesis on SPA entitled "An Analysis of Port Knocking and Single Packet Authorization" at the Information Security Group (ISG) at Royal Holloway College, University of London. His thesis is an excellent evaluation of the port knocking and SPA concepts, and is a must-read for anyone who would like to explore an authoritative treatment of the two security mechanisms. Sebastien uses a quote from Bruce Schneier's Applied Cryptography to help explain away the perception that some people have that SPA suffers from security through obscurity (which it thoroughly does not):

"...If I take a letter, lock it in a safe, hide the safe somewhere in New York, then tell you to read the letter, that's not security. That's obscurity. On the other hand, if I take a letter and lock it in a safe, and then give you the safe along with the design specifications of the safe and hundred identical safes with their combinations so that you and the worlds best safecrackers can study the locking mechanism - and you still can't open the safe and read the letter - that's security..."

Also, additional SPA projects have sprung up, such as an in-kernel implementation that is built entirely within the Netfilter framework. I will discuss these implementations, and make the case that SPA is maturing as a valuable protective mechanism against unknown zero-day exploits in server software.