What is Gatekeeper?
Gatekeeper is the first open source DoS protection system. It is designed to scale to any peak bandwidth, so it can withstand DoS attacks both of today and of tomorrow. In spite of the geographically distributed architecture of Gatekeeper, the network policy that describes all decisions that have to be enforced on the incoming traffic is centralized. This centralized policy enables network operators to leverage distributed algorithms that would not be viable under very high latency (e.g. distributed databases) and to fight multiple multi-vector DoS attacks at once.
The intended users of Gatekeeper are network operators of institutions, service and content providers, enterprise networks, etc. It is not intended to be used by individual Internet users.
For more information, see the Gatekeeper wiki.
How to Set Up
Configure Hugepages
DPDK requires the use of hugepages; instructions for mounting hugepages are available in the requirements documentation. On many systems, the following hugepages setup is sufficient:
$ echo 256 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
Option 1: Obtain Packages
Gatekeeper Debian packages are available for Ubuntu 20.04 LTS at the project’s Releases page.
Install
Once the packages are downloaded, they can be installed with the commands below:
$ tar -zxvf gatekeeper-ubuntu-20.04-packages.tar.gz
$ cd gatekeeper-ubuntu-20.04-packages
$ sudo dpkg -i libgkrte-*.deb \
libgkdpdk-dev_*_amd64.deb \
gatekeeper-dpdk_*_amd64.deb \
gatekeeper-dpdk-dev_*_amd64.deb \
gatekeeper-dpdk-igb-uio-dkms_*_amd64.deb \
gatekeeper-dpdk-rte-kni-dkms_*_amd64.deb \
gatekeeper-bird_*_amd64.deb \
gatekeeper_*_amd64.deb
The gatekeeper-dpdk-dev package is a dependency of the DKMS packages, which build their respective kernel modules during package installation and kernel upgrades.
Configure Gatekeeper
When installed via Debian packages, Gatekeeper configuration files are located in /etc/gatekeeper. You should edit at least the net.lua file, and set the front_ports, front_ips, back_ports and back_ips variables according to your environment.
The other Lua files configure different Gatekeeper functional blocks. Please refer to the project’s wiki for further information on whether these need to be changed in your setup.
You also need to edit the /etc/gatekeeper/envvars file and set the GATEKEEPER_INTERFACES variable to the PCI addresses of the network adapters to be bound to DPDK. These can found using the lshw command. For example:
# lshw -c network -businfo
Bus info Device Class Description
=======================================================
pci@0000:08:00.0 eth0 network I350 Gigabit Network Connection
pci@0000:08:00.1 eth1 network I350 Gigabit Network Connection
...
Given this output, set GATEKEEPER_INTERFACES as below:
GATEKEEPER_INTERFACES="08:00.0 08:00.1"
In the same file, you can optionally specify Environmental Abstraction Layer options in the DPDK_ARGS variable and Gatekeeper-specific options in GATEKEEPER_ARGS.
How to run
Run the commands below to start Gatekeeper and to ensure it is started automatically on reboots.
$ sudo systemctl start gatekeeper
$ sudo systemctl enable gatekeeper
Option 2: Build from Source
Install Dependencies
Install the following software dependencies:
$ sudo apt-get update
$ sudo apt-get -y -q install git clang devscripts doxygen libhugetlbfs-bin \
build-essential gcc-multilib linux-headers-`uname -r` libmnl0 libmnl-dev \
libkmod2 libkmod-dev libnuma-dev libelf1 libelf-dev libc6-dev-i386 \
autoconf flex bison libncurses5-dev libreadline-dev python \
libcap-dev libcap2
Note: Both libmnl0 and libmnl-dev are needed to compile and run gatekeeper, but only libmnl0 is needed for simply running gatekeeper. Both libkmod2 and libkmod-dev are needed to compile and run gatekeeper, but only libkmod2 is needed for simply running gatekeeper. libnuma-dev is needed to compile the latest DPDK and to support NUMA systems. The package libelf-dev is needed to compile DPDK with support to reading BPF programs from ELF files, but only libelf1 is needed to run it. The package libc6-dev-i386 is needed to compile the BPF programs in the folder bpf/. The autoconf, flex, bison, libncurses5-dev, and libreadline-dev packages are for BIRD. The devscripts package is used to build Gatekeeper Debian packages. python is needed to be able to run the dpdk-devbind.py script. libcap-dev is needed to compile Gatekeeper, but only libcap2 is needed to run Gatekeeper.
To use DPDK, make sure you have all of the environmental requirements.
Clone Repository
Clone the Gatekeeper repository, including the submodules that contain Gatekeeper dependencies:
$ git clone --recursive http://github.com/AltraMayor/gatekeeper.git
If you do not use the --recursive clone option, you need to obtain the submodules that contain the dependencies from within the gatekeeper directory:
$ git submodule init
$ git submodule update
Compile
This section explains how to build Gatekeeper manually. If you want to build Debian packages, refer to the section How to build packages.
While in the gatekeeper directory, run the setup script:
$ . setup.sh
This script compiles DPDK, LuaJIT, and BIRD, and loads the needed kernel modules. Additionally, it saves the interface names and their respective PCI addresses in the file lua/if_map.lua so that interface names can be used in the Gatekeeper configuration files.
It also sets two environmental variables: RTE_SDK and RTE_TARGET. They must be set before gatekeeper will compile.
After running the setup script, you may want to save the environmental variables in your shell’s preferences file. For example, in Bash, you can do:
$ echo "export RTE_SDK=${RTE_SDK}" >> ${HOME}/.profile
$ echo "export RTE_TARGET=${RTE_TARGET}" >> ${HOME}/.profile
Otherwise, each time you login you will need to set these environmental variables again.
Once DPDK is compiled and the variables are set, gatekeeper can be compiled:
$ make
Configure Network Adapters
Before gatekeeper can be used, the network adapters must be bound to DPDK. For this, you can use the script dependencies/dpdk/usertools/dpdk-devbind.py. For example:
$ sudo dependencies/dpdk/usertools/dpdk-devbind.py --bind=uio_pci_generic enp131s0f0
This command binds the interface enp131s0f0 to the uio_pci_generic driver so that frames can be passed directly to DPDK instead of the kernel. Note that this binding must take place after Gatekeeper is setup in the steps above so that the bound interface appears in the list of interfaces in lua/if_map.lua.
How to Run
Once gatekeeper is compiled and the environment is configured correctly, run:
$ sudo build/gatekeeper [EAL OPTIONS] -- [GATEKEEPER OPTIONS] Where [EAL OPTIONS] are specified before a double dash and represent the parameters for DPDK’s Environmental Abstraction Layer and [GATEKEEPER OPTIONS] are specified after the double dash and represent Gatekeeper-specific options.
The early configuration of the system, including device and memory configuration in DPDK, will be logged to stdout. Once Gatekeeper is booted, all information is output to the Gatekeeper log.
How to build packages
Gatekeeper Debian packages can be built with the commands below. They are meant to be run from the repository root and assume the git submodules have been pulled, and that the build dependencies have been installed, as instructed above. Gatekeeper and the submodules will be automatically compiled during the package build process.
$ tar --exclude-vcs -Jcvf ../gatekeeper_1.0.0.orig.tar.xz -C .. gatekeeper
$ debuild -uc -us
The Gatekeeper package will be available in the parent directory.
https://github.com/AltraMayor/gatekeeper
How does Gatekeeper work?
Gatekeeper has two components: Gatekeeper servers and Grantor servers. Gatekeeper servers are deployed throughout the Internet at locations called vantage points (VPs). Vantage points are Internet exchange points (IXPs), border and peering-link routers, and (potentially) cloud providers. The aggregated bandwidth of all Gatekeeper servers is what enables a Gatekeeper deployment to scale its incoming bandwidth to match the peak bandwidth of DDoS attacks.
Gatekeeper servers use BGP to announce the network prefixes under their protection. Thus, each traffic source is bound to a VP. Gatekeeper servers’ primary function is to enforce network policies over flows; a flow is defined by the pair source and destination IP addresses. An example of a policy decision is for IP address A to be allowed to send packets to IP address B at 1Gbps or less. An analogy that may help some to wrap their head around Gatekeeper is to think of Gatekeeper servers as reverse proxies that work at the IP layer.
When a Gatekeeper server does not have is a policy decision in its flow table to enforce over any given flow. Instead, it encapsulates the packet of that flow using IP-in-IP, assigns a priority to the encapsulated packet based on the rate of the given flow (higher priority for lower rates), and forwards it through the request channel. The request channel is reserved 5% of the bandwidth of the path that goes from a Gatekeeper server to the Grantor server responsible for the policy decision. Whenever a router forwarding the packets in the request channel needs to drop packets due to the limited bandwidth, it drops the packets of lowest priority in its queues.
A network policy is a Lua script that runs on Grantor servers. Grantor servers are co-located near the protected destination; typically in the same datacenter of the destination. One can deploy Grantor servers in other locations and even employ anycast to reach varied destinations, but we assume here (for the sake of simplicity) that the destination prefix is deployed in a single datacenter.
Grantor servers are responsible for making a policy decision on each flow in the request channel. These policy decisions are sent to the corresponding Gatekeeper servers to enforce them. As policy decisions are installed into Gatekeeper servers, the flows of legitimate senders get moved to the granted channel, in which bandwidth is allocated according to the policy. Similarly, identified malicious hosts would be blocked. This, in turn, would reduce the delay experienced by legitimate flows waiting at the request channel.
Summarizing, a Gatekeeper deployment consists of a number of vantage points forming a shield around the protected networks. Grantor servers, which reside inside of the shield but before the final destinations of the packets, run the network policy to decide the fate of all incoming traffic. The policy decisions are installed at Gatekeeper servers, which enforce these policy decisions.
Anatomy of a Gatekeeper Response
For an example of what all this means, consider the nearly 1Tbps SYN flood that Cloudflare faced in 2018. A Gatekeeper deployment with enough bandwidth at VPs and enough Gatekeeper servers to process the incoming packets would, before any policy evaluation or priority adjustments, reduce that incoming flood to 50Gbps (i.e. 5% of 1Tbps).
How? Since it was a direct attack and the attacker was targeting bandwidth capacity — that is, there was no attempt to establish connections — the request channel would start to automatically filter the flood before reaching Grantor servers in a couple of seconds. This couple of seconds would become the delay experienced by legitimate senders to have their policy decisions installed into the Gatekeeper servers.
The key points to understand how such an attack would be mitigated by Gatekeeper are: (1) ensuring that 5% of the path bandwidth is reserved for requests, (2) assigning priorities to those requests based on the time between requests in the same flow, and (3) dropping packets of lower priority when the bandwidth of the request channel is overflowing. These properties guarantee the results of Theorems 4.1 and 4.2 presented in Section 4.2 of the Portcullis paper (SIGCOMM 2007). In plain English, Theorem 4.1 states that if a network policy cannot identify malicious hosts, a legitimate sender will wait, in the worst case, a time proportional to the number of malicious hosts to have a policy decision installed at the corresponding Gatekeeper server. Theorem 4.2 states that under these conditions, that result is optimal. If a policy can identify malicious hosts, the waiting time becomes, at worst case, proportional to the number of undetected malicious hosts.
What about flows that start behaving well to receive a favorable policy decision and then abuse? There are two parts to this answer. The first is that all policy decisions eventually expire. The second is that policy decisions are BPF programs, so they can limit or act on changes of behavior. The folder bpf in our repository shows examples of BPF programs that limit the bandwidth of a flow (e.g. granted.c), penalize unwanted traffic with less bandwidth (e.g. grantedv2.c), and enforce behaviors expected by simple Web servers (e.g. web.c).
What about abuses that cannot be identified by a BPF program? If the abuse can still be identified by an intrusion detection system (IDS), one will eventually be able to combine IDSes with Gatekeeper; track the status of this feature at issue #298. Another approach is for applications to identify abuses and to feed their results into policies. For a simple example, consider a WordPress plugin that identifies a number of abuses on a WordPress site. The list of abusers from this plugin can be fed into a policy to enable Grantor servers to either bluntly reject abusers or to apply stricter rules on them.
Finally, what about abuses that cannot be identified by an IDS or even an application itself? In this case, all that can be done is to schedule the flows in such a way that enables the network and servers to serve the flows at the best rate possible. Without the scheduling of the flows, the infrastructure is going to serve nobody under such an attack. Gatekeeper is not ready for this scenario. We have many milestones in front of us before we can work on this last milestone.
Where to go from here?
The page Publications has several resources to help you understand Gatekeeper. Newcomers will find our NANOG 82 a good starting point. Those interested in learning from a large-scale Gatekeeper deployment will have plenty of information in the NextHop 2020 presentation. The technical report “Gatekeeper: The Design and Deployment of a DDoS Protection System” covers the underpinnings of Gatekeeper, and is a recommended read for those looking for the big picture.
The page Tips for Deployments describes a simple deployment and includes practical details to help those getting started. The page Overview is a quick reference of the components of Gatekeeper. Much more information is available on the pages of this wiki.
Where to find help?
The primary point of contact for bugs and issues in general is the issues page here on GitHub. The milestones
