Fork bombs are simple pieces of programming logic that can bring down the mightiest of servers in seconds (in fact, the faster the server, the faster it goes down). They are quite easy to protect against, but some operating systems aren’t protected by default. Below I provide an example of how fast an unprotected server is taken down by a fork bomb “going off”, as well as how to protect your own servers.
A fork bomb is simply a running process that forks other running processes, creating an exponential number of processes and rapidly using up all available processes. In other words: a program that spawns so many programs that the system becomes unusable, a denial of service attack. Fork bombs are a species of wabbit, a class of programs utilizing infinite self-replication but that aren’t worms or viruses. This bash implementation is the most famous:
While fairly hard to read for those uninitiated in the school of arcane Bashery, it is elegant, and I find it quite beautiful. This page explains how it works in some detail. In short, a function is defined that calls itself, pipes its output to another instance of itself, then pushes that child into the background, allowing the function definition to terminate. Once the parent function is then called… the fun begins. In a somewhat less obscure representation:
1
2
3
| bomb() {
bomb | bomb &
}; bomb |
There are implementations for many languages listed here. They work when an operating system doesn’t set a limit on how many processes a given user can spawn. To see if a system does have a limit set, run:
This was on Ubuntu 9.10 Desktop. Checking RHEL 5.2, CentOS 5 and 5.2, and Ubuntu 7.10 Server, there was a limit, on the order of tens of thousands of processes. Not limiting the number appropriately is unwise, for there definitely is a limit on that resource, in direct proportion with the amount of memory on your system. The file /proc/sys/kernel/threads-max provides the default allowed amount based on your physical memory. However, the 2.6.x kernel allows this value to be changed at runtime. For more information, check out TLDP’s section on kernel process management.
The Test
Now for the test. I had a server that needed to be decommissioned. It hadn’t been used for some time, everything needed was pulled off, sensitive things removed. All that was left was to press “Cancel Account” with the hosting company. But why such a boring demise? Instead I decided to test how fast a fork bomb would make the box unusable. I first created a file, “heartbeat”:
1
2
3
4
5
6
| #!/bin/bash
while [ 1 ]
do
date; uptime
sleep 1
done |
I would call this script on the remote server from my box. This would show me that it was alive every second, as well as what the load average was on the box. Then I created a fork bomb script, using the more memorable Python implementation:
1
2
3
4
5
| #!/usr/bin/env python
import os
print "Bombs away."
while True:
os.fork() |
This too would be called on the remote server from my box. I started the heartbeat in one terminal, then fired off the fork bomb script in another. The result:
In just a few seconds, the server was down. Not just stunned, mind you. On the floor in another dimension. Expiration was so rapid that my heartbeat script didn’t have time to register and return the clearly heightened load average. And I didn’t get to see the print statement output from the fork bomb script. To fix that, I might have changed line three to:
1
2
3
| import sys
print "Bombs away."
sys.stdout.flush() |
While we’re on the topic of improvements, I should have recorded the kernel version of the server. I believe it was CentOS 5.X, so likely a 2.6.X kernel. Another useful feature would have been to call both of these scripts from one script, first calling heartbeat, sleeping one second, printing an indicator, then calling the fork bomb script. If anyone tries this and gets a more accurate survival time, drop me a comment.
Protection against fork bombs
While the classic fork bomb shown above is something you aren’t likely to encounter without malicious users hanging around, the same sort of situation can arise by means of programming mistakes and poor test coverage as well. So how do you protect a server against this sort of vulnerability? It’s quite simple: set a limit on the number of processes users can initiate at any one time. I ran across several articles stating that to do this you need to edit /etc/security/limits.conf with sudo. This page describes the syntax of the file. The file should also contain an explanation in the comments at the top of the file. You’d likely want to add something similar to the following:
This will limit the number of processes that each user can initiate to 200. The “-” specifies that the value is both “hard” and “soft”. A hard value can’t be overridden with a higher number by the specified user(s), while a soft one is imposed on the user by default. “hard” and “soft” may be used separately if needed. You can also replace the * wildcard with a specific user or group, say if you have an untrusted group of users.
However, on my system (and a few others I looked at) these directives are ignored. To get the limits enforced, I had to add the following to /etc/profile:
This limited the number of processes all users logging in would be able to spawn at any one time. It also capped the limit the user could set for himself.
As to selecting the limit amount, you’ll need to do some testing to fit your needs. On my Ubuntu desktop machines at home, I have 45 – 55 running processes that were spawned by my user at any one time. This includes GNOME, Firefox, conky, IM, twitter client, my terminals, etc. Basically a full-featured modern desktop. The machine serving this blog, without a GUI, has far less processes per user on average. So, something on the order of a few hundred processes is the max you’ll likely need to set. To see how many processes are active from your system’s users, you can use the following:
1
2
3
4
5
6
| # View number of processes from all users:
sudo ps aux | sort | cut -f1 -d" " | wc -l
# View all unique users with processes
sudo ps aux | sort | cut -f1 -d" " | uniq
# Count of processes started by specified user:
sudo ps -U root | wc -l |
You can also test out various limits and their effects via:
This only sets the limit temporarily. Once you log out, it goes back to the default. To make the change permanent, edit /etc/profile as described above. Once you have a limit in place, running a fork bomb will result in an error about resources not being available, instead of a dead box. Here I switch to a test user, set its limit of number of processes, then run a fork bomb:
The bomb continues until it reaches the process cap. At this point you can Ctrl+c out, saving your box.
A benefit to placing the limit in /etc/profile instead of limits.conf is that the root user is not subject to the limits placed in limits.conf. A disadvantage is that you lose the granularity of controlling limits per group and user.
