NGINX is the best HTTP-Server

If you are a sysadmin, or have ever setup a server with a HTTP service, you will undoubtedly have used NGINX at least once in your career.

And not without good reason, NGINX is, by far, the best HTTP(S) server available on Linux, below I will share why I believe this, introduce you to some features you might not be familiar with.

Table of Contents

Rate Limiting
        Rate limiting AI-Crawlers
Reverse Proxy
        Load Balancing
Cool snippets
        Aggressively rate-limit clients without User-Agents
        Access-Control based on IP-Address

Rate Limiting

If you are anything like me, you did not initially know that this is even a thing.

I used to think that implementing rate-limiting was up to the
micro/macro/whatever service that NGINX is "managing".

In reality, NGINX ships with a very powerful and well-thought-out rate-limiting module by default, and it's pretty expressive and useful when you have some tricks up your sleeve.

I will not cover the fundamentals here, you can find the technical details in the official NGINX documentation.
Instead I want to introduce some more advanced useful use-cases I found for this module.

Rate limiting AI-Crawlers

Lets face reality for a second. OpenAIs ChatGPT, Anthropics Claude, and Googles Gemini are here to stay.
Unfortunately for us poor system administrators, this means an unusual amount of traffic we need to deal with, for as long as the arms-race for the best possible model between these behemoths runs.

Fortunately for us, we can rate limit only the AI-Crawlers without restraining actual organic traffic.

Meet the map-directive and how we can leverage this to select only
particular user agents.

map $http_user_agent $ai_crawlers {
    default "";
    ~(GPTBot|ClaudeBot|anthropic-ai|Google-Extended|GoogleOther|Bytespider|CCBot|Amazonbot|FacebookBot|AppleBot-Extended|Cohere-ai) $1;


limit_req_zone $ai_crawlers zone=ai_zone:16m rate=1r/s;
limit_req_status 429;

server {
    location / {
        limit_req zone=ai_zone nodelay burst=5;
    }

This limits all User-Agents contained in the above RegExp to 1 request per second.

I first encountered this on StackOverflow where the solutions are similar, but in my opinion not as elegant as this one.

The map matches against a regular expression that lists multiple strings contained within the AI-Crawler User-Agents.
By matching those strings in a capture list, we can access the string we found via the $1 capture variable, which is what we use as the value at the end of the line.

You could also drop the $1 from that line in favour of something else, say bad bad bot.

If you do that, all those User-Agents evaluate to the bad bad bot string, which means that every client whose User-Agent evaluates to that string shares the rate-limit. I.e. Googles Gemini would share the rate-limit with Anthropic, etc.

The value bad bad bot itself doesn't matter, this is arbitrary. I will explain shortly.

Let us understand what the limit_req_zone does.

limit_req_zone defines a shared memory zone (what the workers use to communicate, the zone=ai_zone:16m in our example means the zone will be 16MiB in size.) into which a hashmap is placed.
The first parameter ($ai_crawlers, the map) serves as the key into the hashmap.

Lets manually unravel the logic. Say a client with the User-Agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.3; +https://openai.com/gptbot
hits our server, if we follow the $ai_crawlers map, we see that this User-Agent gets mapped to GPTBot

NOTE: Actually, it maps to $1, but expanding that according to the capture list, it becomes GPTBot.

The request gets inserted in the the shared hashmap at key(GPTBot), any further requests that match the User-Agent also get sent into that key. This limits all clients matching that string (GPTBot).

As an attentive reader, you might be concerned about the default "", would this not mean that all User-Agents not matching that User-Agent also get limit to 1r/s?
Since key("") will always yield the same value?

No. NGINX got our back, as per the documentation:

Syntax: limit_req_zone key zone=name:size rate=rate; Default: — Context: http Sets parameters for a shared memory zone that will keep states for various keys. In particular, the state stores the current number of excessive requests. The key can contain text, variables, and their combination. Requests with an empty key value are not accounted.NGINX Documentation

Reverse Proxy

The most widely utilised feature of NGINX is its reverse proxy capability.
How to set this up is trivial and documented in thousands of places.

But did you know about load balancing?

Load balancing is done by the ngx_http_upstream_module module, which has a handful of nifty features improving reliability of your service under pressure.

First of all, lets see how a basic upstream can look:

upstream backend {
    server backend1.example.com weight=5;
    server 127.0.0.1:8080       max_fails=3 fail_timeout=30s;
    server unix:/tmp/backend3;

    server backup1.example.com  backup;

As you can see, servers can be a multitude of things.

1. UNIX sockets
2. External Domains/Servers
3. IP-Addresses

But each server can also have parameters, and these are where we start having fun, you can read more about all possible parameters here.

Load Balancing

Within a upstream group, we can instruct NGINX to perform load balancing for incoming requests via directives.
There a number of them we should be aware of, by default NGINX dials our upstreams servers round-robin.

1. least_conn Routes incoming requests to whatever server has the least amount of active connections.
2. least_time Routes incoming requests to whatever server has the shortest average response time and least amount of connections.
3. ip_hash Fixes incoming requests to a given upstream by the clients IP address, all requests originating from that client will end up on the same server.
4. hash key Same as ip_hash, but the key is freely definable, if you have read limiting AI-Crawlers, same principle is applicable here. Remember that we can map lots of things!

Other cool snippets

Aggressively rate-limit clients without User-Agents

On a production server, I noticed that an unusual amount of bad bots scanning for vulnerabilities do not send a User-Agent.

map $http_user_agent $ua_missing {
    ""      1;
    default 0;


location / {
    if ($ua_missing) {
        return 403;
    }

Access-Control based on IP-Address

A colleague once asked me whether it is possible to only allow specific IPs to certain routes.

It is

location /api {
    satisfy any;
    allow     10.0.0.0/8;
    allow    127.0.0.0/8;
    allow  172.16.0.0/20;
    allow 192.168.0.0/16;
    deny all;

    # ...