# DHCP Server - Performance Metrics This document explains the performance metrics of the ISC DHCP server (`dhcpd`, from the RHEL 9.6 `dhcp-server` package): what each one means, why it matters, how `test-dhcp.sh` measures it, and what to change when a result is poor. --- ## Contents 1. [DHCP in one minute](#1-dhcp-in-one-minute) 2. [The metrics at a glance](#2-the-metrics-at-a-glance) 3. [How the tests work](#3-how-the-tests-work) 4. [The metrics in detail](#4-the-metrics-in-detail) - [4.1 Startup time](#41-startup-time) - [4.2 Throughput](#42-throughput-new-leases-per-second) - [4.3 Latency](#43-latency-time-to-get-a-lease) - [4.4 Loss](#44-loss-clients-without-a-lease) - [4.5 Scaling (capacity)](#45-scaling-capacity) - [4.6 Renewal rate](#46-renewal-rate) - [4.7 CPU usage](#47-cpu-usage) - [4.8 Memory usage](#48-memory-usage) - [4.9 Lease database growth](#49-lease-database-growth) 5. [How the metrics relate to each other](#5-how-the-metrics-relate-to-each-other) 6. [How much performance do you need?](#6-how-much-performance-do-you-need) 7. [dhcpd settings that affect performance](#7-dhcpd-settings-that-affect-performance) 8. [Limits of this test](#8-limits-of-this-test) 9. [Glossary](#9-glossary) --- ## 1. DHCP in one minute A DHCP server hands out IP addresses. A device that joins the network (laptop, phone, printer, VM) has no address yet, so it asks for one with four messages, called **DORA**: ``` client DHCP server | 1. DISCOVER "I need an address" (broadcast) ---> | | <--- 2. OFFER "You can have 10.199.128.7" | | 3. REQUEST "I take 10.199.128.7" ---> | | (server writes the lease to disk) | <--- 4. ACK "It is yours for 1 hour" | ``` The address is **leased** for a limited time. At half of that time the client **renews** the lease with a short REQUEST -> ACK exchange. The server keeps all leases in a text file, the **lease database**, and must write each lease to disk *before* it sends the ACK (RFC 2131), so that no address is ever handed out twice, even after a crash. This is why DHCP performance is not only about CPU. The disk matters too. --- ## 2. The metrics at a glance | # | Metric | Unit | Question it answers | Better is | Test name | |---|---|---|---|---|---| | 1 | Startup time | ms | How fast does the server answer again after a (re)start? | lower | `startup` | | 2 | Throughput | leases/s | How many new clients can get an address per second, at full speed? | higher | `throughput` | | 3 | Latency | ms | How long does one client wait for its address? | lower | `latency` | | 4 | Loss | % | How many clients get no address at a normal load? | lower | `loss` | | 5 | Scaling (capacity) | new clients/s | Up to which rate does the server keep up without losing clients? | higher | `scaling` | | 6 | Renewal rate | renewals/s | How many lease renewals per second can it handle? | higher | `renew` | | 7 | CPU usage | % of one core | How much processor time does a normal load cost? | lower | `cpu` | | 8 | Memory usage | MB | How much RAM does dhcpd need? | lower | `memory` | | 9 | Lease database growth | bytes per lease, MB/hour | How fast does the lease file on disk grow? | information | `leasedb` | Run one metric with `./test-dhcp.sh `, or all of them with `./test-dhcp.sh`. --- ## 3. How the tests work ``` +-------------------------------+ +------------------------------+ | namespace "dhcp-perf" | virtual cable | host | | | (veth pair) | | | lib/dhcp-load.py | | dhcpd (dhcpd-perf-test | | simulates thousands of | <=============> | .service) | | DHCP clients | | listens ONLY on dhcp-srv | | interface dhcp-cli | | 10.199.0.1 | +-------------------------------+ +------------------------------+ | | | leases/s, latency, loss | CPU, memory, disk: | | /proc//... v v results//report.md (PASS / FAIL per metric) ``` - **Private test network.** `start-dhcp.sh` creates a virtual cable (a *veth pair*). One end, `dhcp-srv`, stays on the host, and `dhcpd` listens only there. The other end, `dhcp-cli`, is inside a separate *network namespace* (an isolated network stack) where the simulated clients live. No real network card is used, so the test never disturbs the real network and needs no internet access or second machine. - **Separate service.** The test server runs as `dhcpd-perf-test.service` with its own configuration (`/etc/dhcp/dhcpd-perf-test.conf`) and its own lease database (`/var/lib/dhcpd/dhcpd-perf-test.leases`). The normal `dhcpd.service` and `/etc/dhcp/dhcpd.conf` are not touched. - **Load generator.** `lib/dhcp-load.py` is a small Python 3 program (standard library only) that sends real DHCP packets. Each simulated client has its own MAC address. It works in two ways: - **Fixed rate** (`--rate`): N new clients start every second, whether or not the server keeps up. This is how real clients behave. - **Full speed** (`--outstanding`): N clients are always in the middle of an exchange. A client starts again as soon as it has its lease. - **No re-sending.** A real client re-sends after a few seconds when it gets no answer. The generator does not. It waits `EXCHANGE_TIMEOUT` seconds (2 s) and then counts the client as **lost**. So every lost packet is visible. - **Returning clients.** The generator cycles through `MAC_POOL_SIZE` (20,000) MAC addresses. A client seen before gets its old address back, exactly as on a real network, and the pool of 32,766 addresses never runs out. - **Server-side measurements**, taken once per second during the steady load: CPU time and memory of the `dhcpd` process (`/proc//stat`, `/proc//status`), size of the lease file, bytes written to disk (`/proc//io`). - **Clean start:** the test lease file is emptied at the start of each run (`RESET_LEASES_BEFORE_TEST=yes`). Then 2 seconds of light, unmeasured load run as a warm-up. The **steady load** (`LOAD_RATE`, 200 new clients/s for `DURATION` seconds) is run only once and shared by the `latency`, `loss`, `cpu`, `memory` and `leasedb` tests. --- ## 4. The metrics in detail Each metric has the same five parts: **what** it is, **why** it matters, **how** the test measures it, **how to read** the result, and **how to improve** it. ### 4.1 Startup time **What:** the time from the command that starts `dhcpd` until the first client receives an OFFER. **Why:** while the server is down, new devices get no address and the leases of existing devices cannot be renewed. Startup time decides how long that gap is during a restart, a configuration change, a package update or a reboot. **How it is measured:** `dhcpd` is stopped and started `STARTUP_ROUNDS` times (3 by default) with `systemctl stop/start dhcpd-perf-test`. Before each start, a probe starts sending a DISCOVER every 5 ms. The time runs from the start command until the first OFFER arrives. The report shows each round, the average and the size of the lease file at the time. **How to read it:** | Result | Meaning | |---|---| | under 500 ms | normal for a small or medium configuration | | 0.5 - 3 s | acceptable; big address pools or a large lease file to read | | over 3 s | investigate: very large lease file, slow disk, very large configuration | At startup `dhcpd` reads the **whole lease file**, then writes a new, compacted copy of it. A lease file of hundreds of MB therefore slows every start. **How to improve:** keep the lease file on a fast disk; do not make pools much larger than needed; shorter leases (`max-lease-time`) keep fewer old records. Check the configuration first with `dhcpd -t -cf ` so a typo does not cause a failed start. --- ### 4.2 Throughput (new leases per second) **What:** how many complete DORA exchanges (new leases) the server finishes per second when it is pushed as hard as possible. **Why:** this is the peak the server can deliver. It matters most in a **boot storm**: after a power cut or a network outage, every device on the network asks for an address at the same moment. **How it is measured:** `OUTSTANDING` clients (50 by default) are always in the middle of an exchange for `DURATION` seconds. Each client starts again with the next MAC address as soon as it has its ACK: ``` ip netns exec dhcp-perf python3 lib/dhcp-load.py dora --outstanding 50 --duration 10 ... ``` The result is the number of completed leases divided by the duration. **How to read it:** it depends most on **disk speed**, because every lease is written to disk before the ACK. `dhcpd` handles one packet at a time on one CPU core, so the core's speed matters too. Some guidance: | Result | Meaning | |---|---| | over 5,000 /s | fast disk (NVMe/SSD with fast `fsync`) and fast CPU core | | 1,000 - 5,000 /s | normal for a server with SSD storage | | 100 - 1,000 /s | slow disk (spinning disk, network storage, busy VM) | | under 100 /s | a serious bottleneck: check disk latency (`iostat -x 1`) | More outstanding clients do not raise throughput once `dhcpd` is saturated. They only make each client wait longer (see section 5). **How to improve:** put `/var/lib/dhcpd` on fast local storage; consider `delayed-ack` (section 7); split a very large network across two servers (ISC *failover* or separate scopes). --- ### 4.3 Latency (time to get a lease) **What:** how long one client waits for its address, from its DISCOVER to the server's ACK. The report also shows the two halves: - **DISCOVER -> OFFER**: the server looks for a free address. - **REQUEST -> ACK**: the server records the lease *and writes it to disk*. **Why:** until the ACK arrives, the device cannot use the network. Long waits make a device look "stuck" while connecting to Wi-Fi or booting. **How it is measured:** during the steady load (`LOAD_RATE` = 200 new clients/s), every exchange is timed. The report lists average, p50, p90, p95, p99 and the slowest. PASS requires both **p95** and **p99** to stay under their targets. **How to read it:** a percentile says how many clients were *at most* this slow. "p99 = 2 ms" means 99 of 100 clients had their lease within 2 ms. | p99 whole lease | Meaning | |---|---| | under 5 ms | excellent | | 5 - 50 ms | good; clients will not notice | | 50 - 500 ms | disk is slow or the server is near its limit | | over 1 s | real clients start re-sending; users notice delays | If REQUEST -> ACK is much slower than DISCOVER -> OFFER, the **disk** is the bottleneck. If both halves are slow, the CPU or the packet queue is. **How to improve:** the same as for throughput. Also check that `ping-check` is not adding a delay (section 7). --- ### 4.4 Loss (clients without a lease) **What:** the share of clients that got no lease at a normal, steady load: no OFFER, no ACK, or a NAK ("no"). **Why:** under a normal load **no** client should be lost. A lost packet costs the client a re-send, usually after 4 seconds or more. **How it is measured:** during the steady load, the test counts clients whose DISCOVER got no OFFER, whose REQUEST got no ACK within 2 s, and NAK answers. It also shows: - **answers dropped on the client side:** a kernel counter in the test namespace (`RcvbufErrors`). It proves that answers were not lost inside the load generator. It should be 0. - **warnings in the dhcpd log** during the run (with systemd only), for example `no free leases`. The matching lines are saved in `raw/`. **How to read it:** at the kit's normal load the result should be **0%**. Any loss at a normal load points to a problem: a full address pool, a paused VM, a very slow disk, or another process competing for the CPU. **How to improve:** make sure the pool is big enough (`no free leases` in the log); see section 7 for the kernel socket buffer. --- ### 4.5 Scaling (capacity) **What:** the highest rate of new clients per second that the server still keeps up with. **Why:** throughput (4.2) tells you the top speed. Capacity tells you the highest load you can **promise**, the rate at which (almost) no client is lost. **How it is measured:** the fixed-rate test runs at each rate in `RATE_LEVELS` (250, 500, 1000, 2000, 4000, 8000, 16000 new clients/s), `DURATION` seconds each. A step **keeps up** when: - at most `SCALING_LOSS_LIMIT_PCT` (1%) of the clients failed, **and** - at least 95% of the requested rate was actually served. The capacity is the last step that kept up, before the first step that did not. **How to read it:** the table in the report shows what happens as the load rises. ``` load below capacity -> served = asked, latency flat, 0% failed load near capacity -> latency (p99) rises first load above capacity -> served stays flat, failed % jumps ``` When the server kept up at every step, the real limit is higher. Add a larger rate to `RATE_LEVELS`. **How to improve:** the same as for throughput. Capacity is usually a little below throughput, because real (fixed-rate) clients arrive in bursts. --- ### 4.6 Renewal rate **What:** how many lease renewals (REQUEST -> ACK) the server handles per second at full speed. **Why:** on a normal network, most DHCP traffic is renewals. Every client renews at half of its lease time, so the server sees a steady stream of them all day long. **How it is measured:** first `RENEW_CLIENTS` (1,000) clients get a lease (not measured). Then those clients renew for `DURATION` seconds, with `OUTSTANDING` renewals in progress at any moment. **How to read it:** renewals are cheaper than new leases (2 messages instead of 4, no search for a free address), so this number is usually higher than throughput. Each renewal still writes a record to the lease file. The kit sets `dhcp-cache-threshold 0` so every renewal is written, like a real renewal at half the lease time (section 7). **How to improve:** longer leases mean fewer renewals per client and hour. The rest is the same as for throughput. --- ### 4.7 CPU usage **What:** the processor time `dhcpd` uses while it serves the steady load, as a percentage of **one** CPU core. **Why:** `dhcpd` runs as a **single process with a single thread**. It can never use more than one core, however many the machine has. 100% of one core is its hard limit. **How it is measured:** the CPU counters of the `dhcpd` process (`/proc//stat`) are read before and after the steady load, and once per second during it. The report shows the average, the busiest second, and the **CPU time per 1000 leases** (this does not depend on the load). It also shows the CPU used by `systemd-journald` / `rsyslogd` in the same time, because `dhcpd` logs every DHCP message to syslog and the logging work is done by those daemons. **How to read it:** | Average at 200 leases/s | Meaning | |---|---| | under 20% | plenty of headroom | | 20 - 50% | fine, but a boot storm will reach the limit sooner | | over 50% | little headroom; see "how to improve" | **How to improve:** keep the configuration simple (few `class` / `if` statements evaluated per packet, no slow `on commit` scripts); reduce logging; use a CPU with a fast single core. --- ### 4.8 Memory usage **What:** RAM used by the `dhcpd` process (resident set size), when idle and at the peak of the steady load. **Why:** memory limits how many address pools and leases one server can hold. **How it is measured:** `VmRSS` from `/proc//status`, before the steady load and once per second during it. **How to read it:** at startup, `dhcpd` creates a record in memory for **every address in every pool**. So memory depends mostly on the total pool size, not on the load. With the kit's 32,766-address pool, expect a few tens of MB. Memory that keeps growing over hours of constant load would point to a leak. Run with a longer `DURATION` to check. **How to improve:** do not make pools much larger than the number of clients needs. --- ### 4.9 Lease database growth **What:** how many bytes the lease file grows per lease granted or renewed, and how many MB per hour that makes at the steady load. **Why:** the lease file (`/var/lib/dhcpd/*.leases`) is written for every ACK. It grows until `dhcpd` compacts it (at startup and then once an hour). A large file slows startup (4.1) and needs disk space. The **number of writes** limits throughput (4.2). **How it is measured:** the lease file size before and after the steady load, divided by the number of leases granted. The report also shows the bytes `dhcpd` actually sent to storage (`write_bytes` in `/proc//io`). **How to read it:** one lease record is a few hundred bytes of text. This result is *information only*: use it to size the disk, and to estimate the file size just before the hourly compaction: ``` MB per hour = bytes per lease x (new leases + renewals per second) x 3600 / 1,048,576 ``` **How to improve:** nothing to tune in normal cases. Keep `/var/lib/dhcpd` on a disk with room for at least a few hours of growth. --- ## 5. How the metrics relate to each other One simple rule, **Little's law**, links throughput and latency: ``` clients waiting for an answer = throughput (leases/s) x latency (s) ``` Example: 50 outstanding clients and 10,000 leases/s mean an average latency of 50 / 10,000 s = **5 ms**. Once `dhcpd` is saturated, throughput cannot grow, so every extra client only adds waiting time. Typical chain of cause and effect as the load rises: ``` more clients per second --> dhcpd's single core or the disk is fully busy --> throughput stops growing (capacity reached) --> packets queue in the kernel --> latency (p99) jumps --> the queue overflows --> packets dropped --> loss > 0 --> real clients re-send --> even more load ``` When a result is poor, follow the chain back: | Symptom | Likely cause | |---|---| | low throughput, CPU near 100% | CPU bound: configuration complexity, slow core | | low throughput, CPU low, REQUEST -> ACK slow | disk bound: slow `fsync` on the lease file | | loss at a normal load, CPU low | pool empty (`no free leases`) or VM paused / starved | | slow startup | very large lease file or pools | --- ## 6. How much performance do you need? A rough way to turn the size of your network into targets: - **Renewals per second (normal day)** = clients / (lease time / 2). Example: 20,000 clients, 1-hour leases -> 20,000 / 1,800 s = **about 11 renewals/s**. - **New leases per second (boot storm)** = clients that restart together / seconds you accept for them all to get an address. Example: 10,000 devices after a power cut, all served within 60 s -> **about 170 leases/s**. Plan for **at least 5 times** these numbers as measured capacity (4.5). The test machine is idle apart from the test, while a production server also does other work, sits behind relays and sees traffic in bursts. The default targets in `settings.conf` (1,000 leases/s, 1,000 renewals/s) cover networks of tens of thousands of clients with a wide margin. --- ## 7. dhcpd settings that affect performance `start-dhcp.sh` writes `/etc/dhcp/dhcpd-perf-test.conf` from `settings.conf`. Everything not listed here is the dhcpd default. | dhcpd.conf statement | Kit value | dhcpd default | Affects | |---|---|---|---| | `ping-check` | `false` | `true` | latency of every *new* address | | `dhcp-cache-threshold` | `0` | `25` (%) | disk writes per renewal | | `authoritative` | on | off | NAK answers to wrong requests | | `default-lease-time` | `LEASE_TIME` = 3600 s | 43200 s | renewals per hour | | `max-lease-time` | `MAX_LEASE_TIME` = 7200 s | 86400 s | longest lease allowed | | `delayed-ack` | not set | `0` (off) | disk writes per ACK | Why the kit sets them: - **`ping-check false`:** by default, before offering a *new* address, dhcpd pings it and waits up to 1 second for an answer, to detect addresses that someone uses without a lease. On a real network, this protection is worth keeping. On the private test network it would only add delay, and the test would measure the ping timeout instead of the server. - **`dhcp-cache-threshold 0`:** by default, a renewal within the first 25% of the lease time is answered from memory without a disk write. Real clients renew at 50%, so their renewals *are* written. The test renews immediately, so without this setting it would measure an unrealistically fast path. Other settings worth knowing (not changed by the kit): | Setting | Effect | |---|---| | `delayed-ack 28;` + `max-ack-delay 250000;` | collect up to 28 ACKs and write them to disk together; fewer `fsync` calls, higher throughput on slow disks, slightly higher latency | | `log-facility local7;` + an rsyslog rule | move dhcpd's per-packet log messages out of the main log | | `one-lease-per-client true;` | frees an old lease when the client gets a new one; keeps pools clean | | `net.core.rmem_default` (sysctl) | size of the kernel queue for incoming packets; a larger queue absorbs bursts instead of dropping them | | lease file on fast local disk | the biggest single factor for throughput and latency | --- ## 8. Limits of this test Know what a single-host test can and cannot tell you: - **Client and server share one machine.** The load generator uses CPU too (the report shows how busy it was). When it is 90% busy or more, the report says so, and the server may be faster than measured. - **No real network, no relay.** Real clients are often behind a DHCP relay (a router with `ip helper-address`), and the network adds delay. Neither is part of this test. - **Answers are broadcast.** The simulated clients set the BOOTP *broadcast* flag, so the server answers by broadcast. The server does the same work either way. - **No re-sending, no failover.** Real clients re-send; ISC failover pairs add a second message per lease to the partner. Neither is simulated. - **Short runs.** The default is 10 seconds per test. For a capacity baseline, use `DURATION=60` or more and repeat the run 3 times. Other work on the host (backups, other VMs) can make a single run much lower. Repeat any surprising result before acting on it. - **The lease file grows during the tests** (about 140 MB in a full run). So that runs can be compared, `test-dhcp.sh` empties the test lease file at the start of each run. Set `RESET_LEASES_BEFORE_TEST=no` to keep it, for example to measure startup time with a large lease file. - **SELinux.** RHEL 9.6 normally runs SELinux in enforcing mode. The kit uses standard RHEL paths (`/etc/dhcp`, `/var/lib/dhcpd`) and the standard `dhcpd` binary, so the standard policy applies. If something is blocked, check `ausearch -m AVC -ts recent`. - **ISC DHCP is end-of-life upstream.** ISC stopped maintaining it in 2022, and Red Hat still ships it in RHEL 9. Its successor, **Kea**, is multi-threaded and behaves differently. These results do not carry over to Kea. --- ## 9. Glossary | Term | Meaning | |---|---| | **ACK / NAK** | the server's final "yes, the address is yours" / "no" | | **boot storm** | many devices asking for an address at the same moment (after a power cut or outage) | | **broadcast flag** | a bit in the DHCP packet asking the server to answer by broadcast | | **DORA** | DISCOVER, OFFER, REQUEST, ACK: the four messages that give a client a new address | | **fsync** | the system call that forces data onto the disk; dhcpd calls it before ACKs | | **lease** | an address given to one client for a limited time | | **lease database / lease file** | `/var/lib/dhcpd/*.leases`, the text file where dhcpd records every lease | | **MAC address** | the hardware address of a network card; dhcpd identifies clients by it | | **network namespace** | an isolated copy of the Linux network stack; used here to hold the simulated clients | | **outstanding** | clients that have sent a request and are waiting for the answer | | **percentile (p95)** | the value that 95% of measurements are below | | **pool / range** | the addresses the server may hand out | | **renewal** | a client asking to extend its lease (REQUEST -> ACK), normally at half the lease time | | **RSS** | resident set size: memory of a process that is in RAM | | **veth pair** | a virtual network cable: two virtual interfaces connected to each other |