How Stateful is Stateful Inspection? - Understanding the FW-1 State Table

The intent of this paper is to help you understand how  FW-1's stateful inspection  connections table works.   This table is how FW-1 maintains who is doing what and what connections are allowed based on the rule base.  The paper is based on continued research I have done with the latest version of FW-1, version 4.1.  To help you better understand your own FW-1 stateful inspection table and validate my data, I have posted all the source code I used at the bottom of this page. 

Stateful Inspection

This paper started off with a basic question.  If you have a firewall with a rule base that allows anything through it (any - any - accept), will the firewall allow a new TCP connection that is initiated with an ACK?  A part of me said yes.  If the firewall allows everything, then any packet should go through.  However, a part of me also said no.  Based on how stateful inspection works, the packet should be dropped.

My initial understanding of stateful inspection (at least on Check Point FireWall-1) worked as follows.  Whenever a firewall receives a SYN packet initiating a TCP connection, that SYN packet is reviewed against the Firewall rulebase.  Just like a router, this SYN packets is compared to the rules in sequential order (starting with rule 0).  If the packet goes through every rule without  being accepted, the packet is denied.  The connection is then dropped or rejected (RST is sent back to the remote host).  However, if the packet is accepted, the session is then entered into the firewall's stateful connection table, which is located in kernel memory.  Every packet that follows (that does not have a SYN) is then compared to the stateful inspection table.  If the session is in the table, and the packet is part of that session, then the packet is accepted.  If the packet is not part of the session, then it is dropped.  This improves system performance, as every single packet is not compared against the rule base, only SYN packets initiating a connection are compared to the rule base.  All other TCP packets are compared to the state table in kernel memory (very fast).

Now, back to our original question.  If you initiate a session with an ACK packet, will the firewall accept the packet, even with a rulebase that accepts everything?  As we said earlier, your would think yes.  But now that we have a better understanding of the connections table, maybe the answer is no.  When the firewall receives the ACK packet, it is going to compare it to the state table in kernel memory, not the rule base.  However, the firewall will not have this session in its state table, there was never a SYN packet.  So, does the firewall accept the packet, or drop it since there is no entry for it in the state table?

The Result - How FW-1 Builds a Connection.
The results were surprising.  Not only was the ACK packet accepted, but it was entered into the state table.  My understanding of the firewall state table was incorrect.  What I discovered is this, when the firewall receives a packet that is NOT part of the state connection table, that packet is checked against the rule base, regardless if it is a SYN, ACK or 'whatever' packet.  If the rule base accepts the session, then it is entered into the state table.  All subsequent packets of that session are compared to the state connection table and then accepted.  Since there is an entry in the state table for the session, the packets are accepted without being compared to the rulebase.  Below is some of the output from the tool, fwtable.pl (ver 1.0), which converts the data found in 'fw tab -t connections'.  This table is where FW-1 stores all of the concurrent connections in memory. The entries you see below are part of my firewall state connections table created by initiating connections with  ACK packets.

mozart #fwtable

                            ---- FW-1 CONNECTIONS TABLE ---
 

Src_IP          Src_Prt Dst_IP          Dst_Prt IP_prot Kbuf    Type    Flags       Timeout
192.168.7.131   10003   207.229.143.8   25      6       0       16385   02ffff00    2845/3600
192.168.7.131   10002   207.229.143.8   24      6       0       16385   02ffff00    2845/3600
192.168.7.131   10001   207.229.143.8   23      6       0       16385   02ffff00    2845/3600 

Here you see three packets accepted and entered into the firewall state table.  However, these three packets were initiated with ACK packets.   The same thing is true for Null, SYN/ACK, and various other packets, such as FIN/ACK.   If a packet is not part of the state table, it is then compared to the rulebase.  If the rulebase accepts the packet, the session is then added to the state table.  If the packet is not accepted by the rulebase, the packet is dropped/rejected, killing the session.  This is how the firewall "maintains" connections when you do a 'fwstop;fwstart'.  When you bounce the firewall, the connections table is cleared, nothing is maintained.  However, any concurrent connectins will most likely be sending ACKs.  The firewalls sees these packets, verifies them against the rulebase, and rebuilds the connections table.  All of this is transparent to the end user.  This is why you lose Authenticated and Encrypted sessions, the firewall does not have the 'initial state' for these connections. Also, notice the timeout in the right hand column, 3600 seconds.   After entering a session into its state table, the firewall leaves that entry.  That means you have 60 minutes to create and send another packet to reset the timeout clock.   The timeout properties can be set in the control properties menu.

NOTE:  valid FIN or RST packets cannot build a session, as they are used to tear a connnection down.  Also, the only packet that was NEVER added to the state table were 'Xmas' packets created with Fyodor's nmap (-sX option), however these packets were accepted and logged.

Another thing I learned, stateful inspection for FW-1 looks only at Source/Destination IP and Port numbers for determining a session.  It does NOT care about sequence numbers, as I was making up all sorts of whacked out sequence numbers, which the firewall accepted.  Nor does FW-1 maintain state about packet type when building a connection.  When you send a SYN packet initializing a session, the Firewall compares it to the rulebase.  If accepted, it adds it to the state table, as we discussed before.  At this point, the timeout is first set to 60 seconds.  The firewall then expects a return packet to build the connection.  When it sees this return packet, the timeout is then set to 3600 seconds (60 minutes). However, the firewall  is not particular about what type of packet comes back.  I initiated a connection with SYN, then sent back an ACK only, which the firewall happily accepted as part of that connection (as long as the IPs and Ports matched up).  So, the firewall does not have the intelligence to expect SYN/ACK response, nor matching of sequence numbers.   This is most likely done for performance reasons, as maintaining state of Seq numbers would require greater resources. 

Denial of Servic Potential (Bugtraq ID 549):  When building a connection, if you start a connection with an ACK (or most other non-SYN packets, such as Null, FIN/ACK, SYN/ACK, etc) the timeout is automatically set to 3600 seconds (default) see example above.  This has Denial of Service implications.   By initiating many connections with ACK packets to systems that do not exist, you quickly fill up the connections table.  Since there is no remote system, no RST or FIN is sent to teardown the connection, leaving a "dead" connection in the connections table for an hour (remember, timeout for ACK or most other non-SYN packet is 3600 seconds).  You can quickly fill up the connections table initiating connections with ACK packets.  Fortunately, this DoS attack is far more difficult to execute from the outside then from behind the firewall.  Unfortunately, it is easy to DoS yourself if you are doing any scanning from behind your Firewall (as I learned :). Check Point posted a response to this issue.  You can take the following steps to address this issue:

• Check Point has posted a solution using Inspect.  However, this can effect the functionality of the state table when reloading a policy.
• Decrease your TCP timeout to 15 minutes (900 seconds).  This decreases the 'window of opportinuty' a black-hat can use to fill your connections table.
• Increase your connections table.  This makes it more difficult to fill the connections table.
• Implement a strict rulebase that limits what can go outbound and inbound.
• Jason Rhoads has developed a PERL script, fwconwatch.pl, that will monitor your connections table for you and alert you based on criteria you define.
• Implement Fastpath (for ver 3.0) or FastMode (for ver 4.0).  This eliminates TCP connections from you connections table.  However, this could introduce additional security issues. See below for more information on Fastpath/FastMode.
• NOTE:  SynDefender does NOT protect against this threat, as it was designed to protect against SYN flooding, a different kind of attack.  This DoS issue is based on non-SYN packets.

One feature I like with FW-1 is how the firewall treats SYN packets.  If you attempt to initialize a new session that emulates an existing one, the firewall still compares it to the rulebase.  For example, lets say you attempt the following.

A --- FW --> B                    #  System A connects to system B

Now, system B can send whatever packets it wants to system A, as long as the IPs and ports match up (ie, the packets are part of the session).  However, if system B attempts to initialize a new connection (with the standard SYN), even if he uses the exact same ports of the existing session, the firewall still considers the SYN part of a new session and compares it to the rulebase.  In my opinion, this is a good thing.  In the example above, lets say the firewall allows ALL traffic from system A outbound, but no traffic from system B inbound.  The only way system B can talk to system A is if it is part of a connection.

When system A connects to system B, the connection is added to the firewalls inspection table (see example above of inspection table).  Now system B can respond by sending packets to system A.  However, the firewall has NOT blown a hole wide open.  System B cannot send any SYN packets to System A initiating another connection, even if the IPs and port numbers are the same.  When the firewall sees that SYN packet, it applies the packet to the rulebase.  In the above scenario, that packet would be dropped, even thought there is an established connection.

Fastpath:  Something else I learned is if fastpath is enabled, then the session is not added to a connections table, ie no connections table is built.  The reason for this is Fastpath only looks at the SYN packet, so there is no need for a session to be added to the connections table.  If the packet has any other flag enabled, then the packet is not filtered and is allowed through by default.  Normally, fastpath is used to improve performance (or in rare routing situations).  The idea is, if a packet does not have the SYN flag, then it must already be part of an establish connection, as only a SYN packet can start a connection.  Since only SYN packets are inspected, peformance is greatly improved.  However, enabling fastpath is normally a bad move, as this opens you up to a wide variety of attacks.  Fastpath is in FW-1 ver 3.0 only and is a global property applied to all TCP packets.  In ver 4.0, it is called Fastmode, and can be selectively applied to different TCP services.

Closing a Connection
Based on some initial testing, it seems FW-1 closes connections by timeing the connection out.  When the inspection module sees a session exchange a FIN or RST packet, it changes the timeout from 3600 seconds to 50.  If no other packets are exchanged in that 50 second period, the connection is then removed from the state table.  If any packets are sent during the timeout period, the clock is reset to 50 seconds.  By continually sending packets after a session tear down, you can keep resetting the clock to 50 seconds.  This prevents Denial of Service attacks if someone sends spoofed RST or FIN packets.   This timeout behavior can also be considered similar to the TIME_WAIT state a TCP connection enters after acknowledging (ACK) the second FIN packet in closeing a session.

UDP
UDP connections are simplier to maintain, as they are stateless.  When a UDP packet is allowed through the firewall (based on the rulebase) a entry is added to the connections table.  Any UDP packet can return within the timeout period (default 40 seconds) as long as both the SRC/DST IP addresses and SRC/DST ports match.  For example, below is a DNS query.
 
Src_IP          Src_Prt Dst_IP          Dst_Prt IP_prot Kbuf    Type    Flags           Timeout
192.168.1.10    1111    136.1.1.20      53      17      0       16386   ff01ff00        34/40
192.168.1.10    1111    136.1.1.20      0       17      0       16386   ff01ff00        34/40 

Here you see the system 192.168.1.10 doing a dns query to the server 136.1.1.20.  For 40 seconds (Timeout) that system can return as many UDP packets as it wants, as long as both the SRC/DST IPs match, and the SRC/DST ports match.  Notice how there is two entries, both are identical execept for the Dst_Prt, which is 53 and 0.  I do not know why FW-1 creates a second entry for a Dst_Prt of 0.  However, this is common for most, if not all UDP traffic that FW-1 filters.

ICMP
ICMP is a large dissapointment with FW-1.  By default, FW-1 does not statefully inspect ICMP traffic.  It is never entered into the connections table.   As a result, users are forced to blindly allow certain ICMP traffic (such as inbound ECHO_REPLIES) or hack the Inspect code (see http://www.phoneboy.com/fw1).  I believe this is one of the greatest failings of FW-1.

Fragmentation
Recently I have been playing with fragmentation, specifically how does FW-1 handle fragmented packets. Though fragmentation does not apply directly to the state table, I feel it is important enough to add to this paper. I will not be going into detail on how fragmentation works, I am assuming the reader has basic knowledge of IP Fragmentation. I will first cover general findings on how FW-1 handles fragmentation, then I will review the specifics of TCP, UDP, and ICMP.

First, FW-1 does fragmentation in, fragmenation out. What I mean by this is if FW-1 receives a fragmented packet that is accepted by the rulebase, that is what it will send out once it has completed its inspection. Thus the term "frags in, frags out". However, I also believe that FW-1 does some type of reassembly for the fragmented packets before inspecting them. This conlclusion is based on the following tests. When I initiated an allowed connection with a complete, fragmented TCP packet, the packet was accepted by the firewall, added to the state table, and then sent on its merry way (fragmented). By complete, I mean that all the fragments that make up the packet were sent. I now had a session built in the state table for 3600 seconds. I then tried to send more fragmented TCP packets that were part of the same session. These fragmented packets were accepted, the timeout setting in the state table was reset, and the accepted packets continued on. However, when I sent an incomplete TCP fragment of the same session (in other words, I sent a single fragment that did not complete a packet) the fragment was not accepted. Not only was it not accepted, but it was not logged. This leads me to believe that when FW-1 first receives a fragmented packet, it does not inspect the packet untill all the fragments have arrived and the packet is fully assembled. Once assembled, the firewall then decides what to do (accept, deny, etc), logs the packet, and adjusts the state table accordingly. Another example of this behavior is with jolt2 a DoS tool used to attack Window systems. The tool send 100's of incomplete fragmented ICMP (or UDP) packets against a selected target. When ran against FW-1, the packets neither get through the firewall (even though I was accepting ICMP) nor was logged by the firewall. I believe this is due to the fact that these fragmented ICMP packets are incomplete, they do not make up a complete ICMP packet. Since no ICMP packet can be fully assembled, nothing is inspected, nor logged.

When you think about it, some type of reassembley of fragmented packets is most likely required for a stateful firewall. Many stateful firewalls (including FW-1) inspect packets based on Src/Dst IP addresses and Src/Dst ports (TCP Header). However, only the first fragment of a fragmented packet contains all of this information, all other fragments have only the IP address information. If some form of fragment assembly did not happen, the firewall would have no way of knowing what session all the follow on fragments belong to, only the first fragment of the packet. By reassembling all the packets, the firewall inspection engine can them determine which session all the fragments belong to.

However, not inspecting the packets untill after reassembly also has a problem, the firewall is now vulnerable to fragmentation attacks that use incomplete or 'illegal' packets, such as those that are generated by jolt2. Since these incomplete or 'illegal' frag packets will never be properly reassembled, they will neither be inspected or logged. So, the firewall will continue to accept these packets and attempt to assemble them, however reassembly is impossible. Meanwhile, the firewall is vulnerable to the fragmentation attacks, system resources are consumbed trying to process all the fragments. So, the firewall can be attacked using incomplete or illegal fragments, and the attack can neither be stopped by the firewall rulebase nor logged by the firewall rulebase. This vulnerability has been assigned by bugtraq the tracking number 1312. For more information on both the vulnerability and possible solutions, read CheckPoint's advisory. For you Unix users, you can also use the command line option "fw ctl pstat" to view how many fragments the firewall has processed. See phoneboy.com for more info.

Now, on to protocol specifics. First, TCP. First, FW-1 will drop the first fragment of a fragmented TCP packet that has less then 24 bytes of data. If the first fragment of the fragmented packet has less then 24 bytes of data, the firewall drops the fragment by default and logs the packet with the message "TCP packet too short". (NOTE: remeber, when discussing data bytes with fragmentation, this does not include the 20 byte IP header.) For example, the popular network scanner nmap has a '-f' option which will fragment scanning packets into a 16 data byte packet, followed with a 8 data byte packet. These fragmented scans are dropped by default by FW-1 (regardless of your rulebase), with the message "TCP packet too short".

ICMP and UDP are different. First, both allow any standard fragment size (8 bytes, 16 bytes, 32 bytes, etc) unlike TCP, which had a requirement of at lease 24 bytes. (NOTE: like TCP, the data size does not include the 20 byte IP header). However, odd fragmentation byte sizes were not allowed (by odd I mean not increments of 8 bytes). For example, I attemped a fragmented data size of 12 bytes, but this was neither accepted nor logged.

As always, these findings are based on my own personal research, they are in no way official. In fact, I challenge the security community to conduct their own tests to validate these findings. If you find any flaws in my logic, testing methods, or technical implementation, please let me know!

Network Address Translation
I am currently working on understanding how the state table works for Network Address Translation. If you have any input, I would greatly appreciate it, as I am trying to develop both my understanding and this section of the paper on NAT. I have improved the fwtable script so that it now supports Network Address Translation tables (in large part due to the work of Brett Eldridge, beldridg@best.com). If you would like to try out the latest version of this script, download fwtable_1.1. Let me know what you think of the script. Suggestions greatly appreciated.

Conclusion
My initial impression, based on preliminary testing,  is Check Point FW-1's stateful inspection is intelligent, but only semi.  If the FW-1 receives a packet that is NOT part of the state table, that packet is checked against the rulebase. If accepted, it is added to the state table, where all subsequent packets are checked against (known exceptions are Xmas, FIN, and RST packets).  This is a good thing, as the firewall has a robust state table that will maintain connections.  What concerns me is when you initiate a connection with an ACK packet (or various other packets) the timeout is automatically set to 3600 seconds, regardless if a system responds or not.  This has  Denial of Service potential.   What I do like is all SYN packets are checked against the rulebase, regardless if its part of an existing session (this prevents 'tunneling' or 'piggybacking').  However, the inspection table does NOT keep state about sequence numbers, nor SYN - SYN/ACK - ACK sequence.    As for closing connections, its methods seem rather straight forward, similar to TCP's TIME_WAIT period. The state table looks for either a RST or FIN packet, then times the session out. Fragmentation seems to be reassebled during the inspection process. No fragmented is either inspected nor logged untill it has been fully reassembled. Hopefully, after further testing and input from the firewall community, this whitepaper can be a production document that answers many common questions concerning what stateful inspection is, and how really stateful the tables are.

Further Testing
What I have presented was tested on Check Point FireWall-1, ver 4.1 on Solaris x86 2.7.  The tools I used to read the state table and create my own packets can be found below.  I would like to do further testing to understand how the firewall interprets the 'Type' and 'Flags' columns in the state connections table.  Also, how the Firewall 'drops' a connection.  I am looking for anyone to validate (or invalidate) what I have presented here.  Also, any additional information would be greatly appreciated.

Downloads:
fwtable.pl will help you better understand the stateful inspection tables for your firewalls (only works on Check Point FW-1).  The script can be ran locally on any Firewall Module, remotely from any Management Station, or standalone on any system that has PERL.

hping2  Allows you to build your own TCP/ICMP/UDP packets, with built in traceroute and 'pinging' capabilities.

Nemesis is similar to hping, but with some different functionality. Written in C, it uses libpcap and libnet. It allows you to build any type of packet, including TCP, UDP, ICMP, DNS, OSPF, etc. Extremelly simple to use, everthing is done at the command line. Nemesis and hping2 are my tools of choice for packet building. 

http://www.windowsecurity.com/whitepapers/Understanding_the_FW1_State_Table.html