I see this problem a lot. You install a firewall policy on one of your Linux servers, and then all of a sudden you cannot update software components or generate email alerts. The rules look OK; in fact most communication protocols are working just fine. A few however seem to break as soon as you apply an outbound policy. Sound familiar? Most likely you are missing a loopback rule. In this blog post I’ll explain why the problem occurs, how to troubleshoot it, as well as how to fix it.
The loopback interface is simply a network connection a computer can use whenever it needs to talk to itself. While this can seem a bit odd, it can actually be quite handy. For example let’s say I’m creating an alerting process that needs to send email alerts. I don’t want to have to write an entire RFC compliant mail transfer agent, so I would prefer to simply leverage the mail server that has already been installed locally. The problem is I may not know which mail server is being used. Is it Sendmail? qmail? Postfix? Something else? It would be a bit cumbersome to check for every possibility.
The other option is to simply talk to the local mail server on TCP/25 and have it forward my email alerts as required. In this case I don’t care which software is being used for the mail server, as they will all listen on TCP/25. So the loopback interface is an extremely handy way for networking processes to talk to each other without the burden of having to know which specific software package is on the other end.
One of the most common software packages that leverages this connectivity is WordPress. When you update a plugin or theme, WordPress may attempt to make a connection to the local FTP port in order to transfer the required file information. If the firewall is blocking this activity, the update will fail.
Troubleshooting Loopback Problems
The easiest way to diagnose a loopback problem is to log all outbound packets that get dropped. Within the Halo interface, edit the appropriate firewall policy and insure your final rule in your outbound policy has “Log” selected. If it does not, select it now and save your changes. Then repeat the activity that has been failing.
You now need to logon to your server so you can review the firewall logs. The logs will be located in the “/var/log” directory. Depending on which flavor of Linux you are running, you want to check either the “messages” or “syslog” log file. You should only have one or the other within the “/var/log” directory.
Look for an entry similar to this one:
Jun 20 18:11:21 wordpress kernel: [16890529.490637] IN= OUT=lo SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=41643 DF PROTO=TCP SPT=40577 DPT=21 WINDOW=32792 RES=0x00 SYN URGP=0
Note that I’ve highlighted the interesting bits in red. The log entry states the packet was leaving (OUT) the “lo” interface. “lo” is the designation of the loopback interface. Also note the source and destination IP addresses are the loopback address. Since this was logged by our catch all rule at the end, the packet was being block. So we need to tweak out firewall to permit this connectivity.
Fixing Loopback Problems
This problem is pretty easy to fix. We simply need to create a permit rule that lets this connectivity through. What we need to decide is whether we want to write a specific rule for this one service, or simply permit all loopback activity. Since loopback only permits traffic to and from the same host, the risk of opening it up to all services is minimal. In order to leverage this access, an attacker would have to already be on the system in question. So by permitting all loopback activity we elevate risk only slightly but eliminate the possibility of other services encountering similar problems in the future.
Figure 1 shows an example policy with traffic being permitted on the loopback interface. The loopback rule has been circled in red. Also note this rule needs to go in the “Outbound Rules” section. Finally, it usually does not matter where in the outbound policy you place this rule, so long as it appears prior to the final rule.