# Mail Server - Performance Metrics

This document explains the performance metrics of a mail server, using the
standard RHEL 9.6 mail server as the example: **Postfix** (SMTP) together with
**Dovecot** (mailboxes, IMAP, POP3). It covers what each metric means, why it
matters, how `test-mail.sh` measures it, and what to change when a result is
poor.

---

## Contents

1. [The metrics at a glance](#1-the-metrics-at-a-glance)
2. [How the tests work](#2-how-the-tests-work)
3. [The metrics in detail](#3-the-metrics-in-detail)
   - [3.1 Startup time](#31-startup-time)
   - [3.2 SMTP throughput](#32-smtp-throughput)
   - [3.3 SMTP latency](#33-smtp-latency)
   - [3.4 Concurrency scaling](#34-concurrency-scaling)
   - [3.5 Message size](#35-message-size)
   - [3.6 End-to-end delivery](#36-end-to-end-delivery)
   - [3.7 Queue drain](#37-queue-drain)
   - [3.8 IMAP logins](#38-imap-logins)
   - [3.9 IMAP reading](#39-imap-reading)
   - [3.10 POP3 sessions](#310-pop3-sessions)
   - [3.11 Many IMAP sessions](#311-many-imap-sessions)
   - [3.12 CPU usage](#312-cpu-usage)
   - [3.13 Memory usage](#313-memory-usage)
4. [How the metrics relate to each other](#4-how-the-metrics-relate-to-each-other)
5. [Settings that affect performance](#5-settings-that-affect-performance)
6. [Limits of this test](#6-limits-of-this-test)
7. [Glossary](#7-glossary)

---

## 1. The metrics at a glance

| # | Metric | Unit | Question it answers | Better is | Test name |
|---|---|---|---|---|---|
| 1 | Startup time | ms | How fast is the mail server usable again after a (re)start? | lower | `startup` |
| 2 | SMTP throughput | messages/s | How many messages can the server accept per second? | higher | `smtp` |
| 3 | SMTP latency | ms | How long does a sender wait for one message to be accepted? | lower | `latency` |
| 4 | Concurrency scaling | % of peak | Does the server hold up when many senders connect at once? | higher | `concurrency` |
| 5 | Message size | messages/s, MB/s | How much do large messages cost? | higher | `size` |
| 6 | End-to-end delivery | messages/s, ms | How fast does mail reach the mailbox? Is any lost? | higher, lower, 0 lost | `delivery` |
| 7 | Queue drain | messages/s | How fast does the server catch up after an outage? | higher | `queue` |
| 8 | IMAP logins | logins/s | How many users can log in per second? | higher | `auth` |
| 9 | IMAP reading | messages/s, ms | How fast can users open their mail? | higher, lower | `imap` |
| 10 | POP3 sessions | sessions/s | How many POP3 downloads per second? | higher | `pop3` |
| 11 | Many IMAP sessions | sessions, MB | Can the server keep every user's mail program connected? At what memory cost? | all open, lower | `connections` |
| 12 | CPU usage | % of all CPUs, ms/message | How much processor time does mail cost? | lower | `cpu` |
| 13 | Memory usage | MB | How much RAM do Postfix and Dovecot need? | lower | `memory` |

Run one metric with `./test-mail.sh <test name>`, or all of them with
`./test-mail.sh` (about 4 minutes).

**What a mail server does, in one paragraph.** Other mail servers and mail
programs **send** mail to it over **SMTP**. Postfix checks the recipient,
writes the message into its **queue** (a folder of files, safely on disk)
and answers "250 queued" - from then on it is responsible for the message.
Its queue manager then hands the message to Dovecot over **LMTP**, and
Dovecot writes it into the user's **mailbox**. Users **read** their mail with
a mail program (Thunderbird, Outlook, a phone) over **IMAP**, which keeps the
mail on the server, or download it over **POP3**. So there are two sides to
measure: mail coming **in** (SMTP, queue, delivery - metrics 2 to 7) and
users **reading** (logins, IMAP, POP3 - metrics 8 to 11).

---

## 2. How the tests work

```
                             SMTP 127.0.0.1:25
  +--------------------+  ---------------------->  +-----------+   LMTP    +-----------+
  | lib/mailload.py    |                           |  Postfix  | --------> |  Dovecot  |
  | load generator     |                           |  (queue)  |  (socket) |           |
  | C clients =        |    IMAP 143 / POP3 110    +-----------+           +-----+-----+
  | C connections      |  <------------------------------------------------------+ |
  +--------------------+                                                           v
           |                                                 /var/spool/mail-perftest/
           | messages/s, time per operation,                   user1 ... user1000 /Maildir
           | errors                                            (100 messages each)
           v                                                         |
  results/<date-time>/report.md   <----- delivery times -----  lib/delivery-times.py
  (PASS / FAIL per metric)                CPU, memory: all Postfix + Dovecot processes
```

- **Load generator:** `lib/mailload.py`, part of this kit. It needs only
  Python 3, which every RHEL 9 system has, and speaks SMTP, LMTP, IMAP and
  POP3 itself. (RHEL has no mail benchmark tool that reports response-time
  percentiles, the most useful latency numbers.)
  - It simulates **clients**. Each client has its own connection and works
    like a normal mail program: send one command, wait for the answer, send
    the next. "50 clients" = 50 connections, each doing one thing at a time.
  - **Sending** (SMTP): every message is one complete SMTP session - connect,
    greeting, `EHLO`, `MAIL FROM`, `RCPT TO`, `DATA`, the message, `QUIT` -
    exactly how other mail servers deliver mail to this one.
  - **Reading** (IMAP, POP3): logins with a new connection each time, or
    connections that stay logged in (like a mail program left open).
  - Clients work as fast as possible, or at a fixed total **rate**
    (operations per second), like steady real traffic.
  - The clients are spread over several worker processes (`LOAD_PROCESSES`),
    each using up to one CPU core.
- **Test data:** created by `start-mail.sh`:

  | What | Example | Details |
  |---|---|---|
  | `USER_COUNT` users (1,000) | `user42@perf.test`, password `Perf-Mail-Pass-1` | "virtual" users: listed in Dovecot's users file, not Linux accounts; password stored as a SHA512-CRYPT hash |
  | one mailbox per user | `/var/spool/mail-perftest/user42/Maildir/` | Maildir format: one file per message |
  | `MAILBOX_MESSAGES` messages per mailbox (100) | 100,000 messages in total | plain-text messages of `MESSAGE_SIZE_KB` (5 KB) |
  | the list of valid addresses for Postfix | `/etc/postfix-perftest/virtual-mailboxes` | mail to other addresses is refused |

- **Test messages are tracked:** every message the load generator sends
  carries two extra headers, `X-Perf-Run` (which test sent it) and
  `X-Perf-Sent` (when, to the microsecond). `lib/delivery-times.py` finds
  them in the mailboxes afterwards and compares that time with the time in
  the Maildir file name (Dovecot writes it there, also to the microsecond).
  This gives the exact delivery time of every message, and shows whether any
  message was lost.
- **Clean mailboxes for every test:** after each test that sends mail, the
  delivered test messages are removed again
  (`doveadm expunge ... uid 101:*`), so every test starts with the same
  100 messages per mailbox.
- **Server-side measurements:**
  - CPU time of all Postfix and Dovecot processes, from the
    `postfix-perftest` and `dovecot-perftest` service cgroups (`cpu.stat`).
  - Memory (PSS) of all their processes, from `/proc/<pid>/smaps_rollup`.
  - The number of messages in the Postfix queue (files in the queue folders).
  - New warning and error lines in both logs.
- **Offline:** all traffic uses the loopback address `127.0.0.1`. No internet
  access and no second machine are needed.
- **Warm-up:** 3 seconds of light, unmeasured load run first.

---

## 3. The metrics in detail

Each metric below has the same five parts: **what** it is, **why** it matters,
**how** the test measures it, **how to read** the result, and **how to improve** it.

The example numbers ("measured on the test machine") come from a 32-core
machine with 128 GB RAM and an SSD, with Postfix 3.5 and Dovecot 2.3.16 and the
kit's default settings. Use them as a rough guide only; your hardware gives
other numbers.

### 3.1 Startup time

**What:** the time from starting Postfix and Dovecot until the mail server is
fully usable: Postfix answers SMTP connections with its greeting, and a user
can log in over IMAP.

**Why:** it decides how long mail is not accepted during a restart (after an
update, a configuration change, or a crash).

**How it is measured:** both servers are stopped and started `STARTUP_ROUNDS`
times (3). The report shows the time until SMTP was ready and until
everything was ready.

**How to read it:**

| Result | Meaning |
|---|---|
| under 1 s | normal (test machine: about 0.25 s) |
| 1 - 5 s | acceptable; slow disk, or many processes started in advance |
| over 5 s | investigate: look at both logs for errors or waits (DNS lookups that time out on an offline host are a common cause) |

A short restart loses no mail: sending servers keep the mail in their own
queues and try again a few minutes later. Senders see "connection refused"
and retry; users see their mail program reconnect.

**How to improve:** it is rarely a problem. Make sure the host's own name
resolves locally (`/etc/hosts`), so that nothing waits for DNS.

---

### 3.2 SMTP throughput

**What:** how many messages per second Postfix accepts over SMTP, at full
speed.

**Why:** this is the capacity of the server for incoming mail - newsletters,
mail from other companies, reports sent by applications. When it is too low,
senders wait, time out, and try again later: mail is delayed.

**How it is measured:** `CLIENTS` clients (50) send 5 KB messages to random
users for `DURATION` seconds (10). Every message is a complete SMTP session.
A message counts once Postfix has answered `250 ... queued as ...`.
Delivery into the mailboxes happens at the same time, in the background.
The test also sends mail to addresses that do not exist for 3 seconds: every
one must be refused at `RCPT TO` with `550 User unknown`.

**How to read it:**

| Result | Meaning |
|---|---|
| over 1,000/s | a modern server (test machine: about 2,700/s, limited on purpose by `in_flow_delay`, see 3.6) |
| 200 - 1,000/s | a small VM; enough for most organisations |
| under 200/s | investigate: slow disk (fsync), CPU, `in_flow_delay` (see 3.6) |

For comparison: a company with 1,000 users receives perhaps 50,000 messages
per day, on average under one per second; peaks are 10 - 50 times higher.
Spam waves and bulk mailings are the real peaks.

**Why unknown recipients are refused during the session:** if Postfix accepted
mail for unknown addresses first and found out later, it would have to send
a "bounce" message back - often to a forged sender address (spam). Refusing
at `RCPT TO` costs almost nothing and creates no bounce.

**How to improve:** Postfix writes and forces every accepted message to disk
before it says "250" (so no mail is lost in a power failure); the speed of
the disk under `/var/spool/postfix` matters most. Also: enough SMTP processes
(`default_process_limit`, 3.4), and spam/virus filters (not part of this test)
are often the real limit on production servers.

---

### 3.3 SMTP latency

**What:** the time for one complete SMTP session - from connecting until
Postfix says goodbye after accepting the message.

**Why:** a sending server waits this long for each message. Long waits mean
fewer messages per connection per second for the sender, and timeouts in
extreme cases (the SMTP standard allows the sender to give up after a few
minutes per step).

**How it is measured:** a steady `LATENCY_TEST_RATE` messages per second
(200) - a normal load, not the maximum - for `DURATION` seconds. The report
shows the average and the **percentiles**.

**Why percentiles and not only the average:** "p95 = 5 ms" means 95 of every
100 sessions finished within 5 ms. The average hides the slow ones; p95 and
p99 show how bad it is for the unlucky ones.

**How to read it:** over localhost the network adds almost nothing, so this
is Postfix's own time, mostly writing the queue file safely to disk.

| Result (p95) | Meaning |
|---|---|
| under 10 ms | normal (test machine: 1.8 ms) |
| 10 - 100 ms | acceptable; slow disk or busy CPU |
| over 100 ms | investigate: disk latency (`iostat -x 1`), CPU, DNS lookups of the client's name |

**How to improve:** a disk with low write latency for `/var/spool/postfix`;
working local name resolution (Postfix looks up the name of every connecting
client); keep CPU below ~70% at peak.

---

### 3.4 Concurrency scaling

**What:** how throughput and latency change as more senders connect at the
same time (1, 10, 50, 100, 200 clients by default).

**Why:** many mail servers deliver to you at once, especially during a
newsletter or spam wave. A healthy server keeps its throughput when the
number of connections rises; an unhealthy one slows down, or starts refusing
connections.

**How it is measured:** the `smtp` test repeated at each client level, with
an empty queue at the start of each level.

**How to read it:** throughput rises with more clients until the disk or CPU
is full, then levels off. Postfix runs at most `default_process_limit` SMTP
sessions at once (kit: `SMTPD_PROCESS_LIMIT`, 100). With 200 clients, 100
wait in the queue of new connections until a session ends; their waiting
time shows as higher p95/p99, while throughput should stay about the same.
The test passes when the highest level still reaches `TARGET_SCALING_MIN_PCT`
(50%) of the peak.

Measured on the test machine: 

  | Clients | Messages/s | p95 (ms) | p99 (ms) |
  |---:|---:|---:|---:|
  | 1 | 888 | 1.2 | 6.3 |
  | 10 | 2,399 | 2.0 | 2.7 |
  | 50 | 2,691 | 4.3 | 1,002 |
  | 100 | 2,605 | 13.5 | 1,005 |
  | 200 | 2,558 | 65.2 | 1,023 |

  From 50 clients on, p99 is almost exactly **1 second**: that is Postfix's
  `in_flow_delay` (see 3.6) holding a few senders back while the queue
  manager catches up. It is intended behaviour, not a fault.

**How to improve:** raise `default_process_limit` if many senders must be
served at once (each smtpd process needs only a few MB); make sure the disk
keeps up (the real limit at high concurrency).

---

### 3.5 Message size

**What:** messages per second and megabytes per second for small and large
messages (1 KB, 10 KB, 100 KB, 1 MB by default).

**Why:** small messages measure the fixed cost per message (session, queue
file, fsync, delivery); large messages measure how fast data flows. Real mail
mixes both: text messages of a few KB and attachments of several MB.

**How it is measured:** about `SIZE_TEST_MB` (200 MB) of mail per size, sent
by `CLIENTS` clients; the queue is emptied between the sizes.

**How to read it:** messages/s falls as messages grow; MB/s rises until the
disk or the copying of data becomes the limit. The test passes when the
largest messages reach `TARGET_SIZE_MIN_MBPS` (20 MB/s).

Measured on the test machine: 

  | Size | Messages/s | MB/s |
  |---:|---:|---:|
  | 1 KB | 2,177 | 2.2 |
  | 10 KB | 2,160 | 21 |
  | 100 KB | 909 | 89 |
  | 1 MB | 179 | 179 |

**How to improve:** disk throughput for `/var/spool/postfix` and the mail
store; `message_size_limit` (10 MB by default) decides the largest message
Postfix accepts at all.

---

### 3.6 End-to-end delivery

**What:** how long it takes from the moment a message is sent until it is in
the recipient's mailbox, and how many messages per second arrive in the
mailboxes. And: whether every accepted message arrived.

**Why:** this is what users experience as "mail is fast" or "mail is slow",
and it is the one metric that includes the whole server: SMTP, queue,
queue manager, LMTP and Dovecot's mailbox writes. A lost message is the
worst possible result for a mail server.

**How it is measured:** `DELIVERY_TEST_MESSAGES` (10,000) are sent at full
speed by `CLIENTS` clients. The test waits until the Postfix queue is empty,
then `lib/delivery-times.py` finds every one of them in the mailboxes and
computes its delivery time (see section 2).

**How to read it:**

| Result | Meaning |
|---|---|
| delivered/s close to accepted/s, p95 under 1 s | delivery keeps up with incoming mail (test machine: 1,932 messages/s end to end, p95 60 ms, none lost) |
| delivered/s well below accepted/s | the queue grows during peaks; delivery times grow with it |
| **any message lost** | a serious fault: look at both logs at once |

If Postfix accepts mail faster than it can deliver it, the queue grows and
every later message waits longer. Postfix then slows down the senders on
purpose (`in_flow_delay`, up to 1 s per message), so the queue cannot grow
without limit - that is why SMTP throughput can fall when delivery is slow.

Measured on the test machine (50 clients): with the default `in_flow_delay = 1s`
Postfix accepted about 2,300 messages/s with p99 = 1,003 ms; with
`in_flow_delay = 0` it accepted 7,500 messages/s with p99 = 13 ms - but then
the queue grows much faster than it can be delivered. On a production
server keep the default: a sender waiting one second is harmless, a queue
that grows without limit is not.

**How to improve:** Dovecot's mailbox writes are usually the limit: a fast
disk for the mail store; `lmtp_destination_concurrency_limit` (kit:
`LMTP_CONCURRENCY`, 20) decides how many deliveries run in parallel;
`mail_fsync` (kit: `MAIL_FSYNC`) decides how often Dovecot forces data to
disk.

---

### 3.7 Queue drain

**What:** how fast Postfix delivers a full queue once delivery is possible
again.

**Why:** after an outage of the mailbox storage or of Dovecot, or after a
network problem, thousands of messages wait in the queue. This metric says
how long users must wait until the backlog is gone.

**How it is measured:** delivery to Dovecot is held back with the Postfix
setting `defer_transports = lmtp`; `QUEUE_TEST_MESSAGES` (10,000) are sent and
wait in the queue. Then deliveries are allowed again and `postqueue -f`
("try the whole queue now") is run - what an administrator does after an
outage. The test measures the time until the queue is empty.

**How to read it:** compare with end-to-end delivery (3.6): the drain rate is
the pure delivery speed, without new mail arriving at the same time.
Test machine: 10,000 messages in 3.4 s, about 2,900/s. The report estimates how long 100,000 waiting
messages would take.

**How to improve:** the same as for delivery (3.6). Note that Postfix waits
between retries of deferred mail (`minimal_backoff_time`, 5 minutes to start
with), so after an outage run `postqueue -f` rather than waiting.

---

### 3.8 IMAP logins

**What:** how many IMAP logins per second Dovecot handles, each with a new
connection: connect, `LOGIN`, `LOGOUT`. And: wrong passwords must be refused.

**Why:** every mail program logs in when it starts, and phones reconnect
every time they change network. At 8:00 in the morning, hundreds of users
start their mail program within minutes. Webmail systems often log in for
every page.

**How it is measured:** `CLIENTS` clients (50) log in as random users with the
correct password for `DURATION` seconds. Then the same with a wrong
password: every login must be refused with `NO`.

**How to read it:**

| Result | Meaning |
|---|---|
| over 300/s | normal for Dovecot's default settings (test machine: about 305/s) |
| 100 - 300/s | a small VM; enough for thousands of users |
| under 100/s | investigate: CPU, slow password hash, auth process at 100% CPU |

**What limits the login rate:** each login needs up to three things:
a new *login* process (in Dovecot's default "high-security" mode), the
password check in Dovecot's *auth* process, and a new *imap* process for the
session. Dovecot's master process starts new processes **one at a time** per
service, so starting processes - not the password check - is usually the
limit with default settings. Measured on the test machine (50 clients):

| Dovecot set-up | Logins/s | Limit |
|---|---:|---|
| default (high-security), SHA512-CRYPT | ~310 | starting processes, one at a time |
| default, SSHA512 (fast hash) | ~310 | the same: the hash was not the limit |
| `DOVECOT_LOGIN_MODE=high-performance`, SHA512-CRYPT | ~480 | the single *auth* process, busy computing hashes |

**Wrong passwords:** Dovecot answers a failed login only after a delay
(`auth_failure_delay`, 2 s), and the delay grows up to 15 s the more failed
logins come from the same IP address (the "auth penalty"). This makes
password guessing slow - but users behind one NAT address share the penalty,
so one misconfigured phone can slow down logins for a whole office. The
kit's test clients all use 127.0.0.1, so after the wrong-password test the
kit logs in once correctly, which clears the penalty.

**How to improve:** `DOVECOT_LOGIN_MODE=high-performance` in `settings.conf`
(long-running login processes, `service imap-login { service_count = 0 }`,
plus spare imap/pop3 processes, `process_min_avail`); if the *auth* process
is then at 100% CPU, a faster password hash, or Dovecot's auth cache
(`auth_cache_size`), helps.

---

### 3.9 IMAP reading

**What:** how many whole messages per second users can read over IMAP, and
how long each takes.

**Why:** this is what users do all day: open a folder, click on a message.
Slow answers are felt immediately.

**How it is measured:** `CLIENTS` clients log in once as random users, select
their INBOX, and then read random messages (among the first 100 of the
mailbox) with `UID FETCH <n> BODY.PEEK[]` as fast as possible. `PEEK` means
the message is not marked as read, so the mailboxes stay unchanged.

**How to read it:**

| Result | Meaning |
|---|---|
| over 10,000/s | normal when the mailboxes are in the page cache (test machine: about 108,000/s, p95 0.7 ms) |
| 1,000 - 10,000/s | a small VM, or mail read from disk |
| under 1,000/s | investigate: disk, CPU, very large mailboxes |

**How to improve:** enough RAM for the Linux page cache (recently used
messages and Dovecot's index files are then read from memory); a fast disk
for the mail store; Dovecot's index files on fast storage for very large
mailboxes.

---

### 3.10 POP3 sessions

**What:** complete POP3 sessions per second: connect, `USER`, `PASS`, `STAT`,
`RETR` (download one message), `QUIT`.

**Why:** POP3 programs and devices (scanners, ticket systems, monitoring
tools) poll their mailbox every few minutes. Each poll is one complete
session, so a thousand devices polling every minute cause about 17 sessions
per second.

**How it is measured:** `CLIENTS` clients run sessions as random users for
`DURATION` seconds; the messages are downloaded but not deleted.

**How to read it:** each session includes a login, so the rate is usually
close to the IMAP login rate (3.8). Test machine: about 300 sessions/s.

**How to improve:** the same as for logins (3.8). Mailboxes with tens of
thousands of messages make every session slower (the whole list is read at
login); users of POP3 should delete downloaded mail from the server.

---

### 3.11 Many IMAP sessions

**What:** whether the server can keep many IMAP sessions open at once, and
how much memory each one costs.

**Why:** mail programs keep an IMAP connection open all day, so a server has
about one open session per active user (phones and tablets often add more).
In Dovecot every IMAP session is its own `imap` process, so this decides how
much RAM the server needs.

**How it is measured:** `CONNECTION_TEST_COUNT` clients (1,000) log in as
different users and keep their session open; together they send
`CONNECTION_TEST_RATE` (500) `NOOP` commands per second ("anything new?") for
`DURATION` seconds. Memory and the number of processes are sampled every
second.

**How to read it:** every session must open and stay open. "Memory per open
session" (test machine: about 0.8 MB per session; 1,000 sessions added about 770 MB) times the number of users online
at the busiest time gives the RAM needed for IMAP.

**How to improve:** raise the limits when more users are online at once:
`service imap { process_limit }` (kit: `IMAP_PROCESS_LIMIT`, 2,000),
`default_client_limit`, and `mail_max_userip_connections` (10 sessions per
user and IP address by default). Dovecot writes a warning to its log when a
limit is reached.

---

### 3.12 CPU usage

**What:** the processor time used by all Postfix and Dovecot processes while
mail is received and delivered.

**Why:** CPU is shared with everything else on the host (spam and virus
filters, the operating system). The **CPU time per message** is the most
useful number for planning.

**How it is measured:** `CLIENTS` clients send mail at full speed for
`DURATION` seconds; CPU is measured from the first message until the queue
is empty again (so the delivery work counts too), from the services'
cgroups.

**How to read it:** test machine: 26% of 32 CPUs (about 8 cores) at ~2,600 messages/s, **3.5 ms of CPU time per message** (receive + queue + deliver). A server that must handle 100
messages per second needs about 100 x (CPU time per message) of CPU time per
second.

**How to improve:** CPU is seldom the limit for Postfix and Dovecot
themselves; disk usually is. Content filters (SpamAssassin, ClamAV, Amavis)
often cost ten times more CPU per message than Postfix and Dovecot together.

---

### 3.13 Memory usage

**What:** the memory (PSS) of all Postfix and Dovecot processes, idle and
under the load of the CPU test.

**Why:** too little RAM leads to swapping, which makes everything slow.

**How to read it:** test machine: 62 MB idle (25 processes), 224 MB under load (139 processes). Postfix and Dovecot are made
of many small processes; memory grows with the number of processes (SMTP
sessions at once, deliveries at once, and above all open IMAP sessions -
see 3.11), not with the amount of mail. The mail itself lives in files; the
Linux page cache keeps recently used mail in RAM, which is not counted here
but is just as important (see 3.9).

---

## 4. How the metrics relate to each other

- **Accepting is not delivering.** SMTP throughput (3.2) is how fast Postfix
  takes responsibility for mail; end-to-end delivery (3.6) is how fast it
  reaches the mailbox. If the first is higher than the second for a long
  time, the queue grows, delivery times grow, and Postfix starts slowing
  down the senders (`in_flow_delay`).
- **The disk decides most mail metrics.** Every message is written safely
  to disk twice: into the Postfix queue (before "250 queued") and into the
  mailbox (before LMTP says "saved"). Fast, low-latency storage improves SMTP
  latency, throughput and delivery at the same time.
- **Clients, throughput and latency** are linked (Little's law):
  `clients in progress = messages per second x time per message`.
  Once the server is full, more clients cannot raise throughput, so the time
  per message rises instead (the concurrency table shows exactly this).
- **Logins cost more than reading.** One IMAP login (new processes, password
  hash) costs as much as reading hundreds of messages on an open session.
  Mail programs that stay connected are much cheaper for the server than
  ones that log in again and again.
- **Open sessions cost memory, not CPU.** 1,000 idle IMAP sessions hardly use
  CPU, but each one is a process with its own memory.

---

## 5. Settings that affect performance

Show a Postfix setting: `postconf -c /etc/postfix-perftest <name>`,
change it: `postconf -c /etc/postfix-perftest -e '<name> = <value>'`, then
`postfix -c /etc/postfix-perftest reload`.
Show the Dovecot settings: `doveconf -c /etc/dovecot-perftest/dovecot.conf -n`.
(For the kit's instances, change `settings.conf` and re-run `./start-mail.sh`
instead; it rewrites both configurations.)

| Setting | Program | Default | Effect |
|---|---|---|---|
| `default_process_limit` | Postfix | 100 | Most processes of one kind, e.g. SMTP sessions at once. Kit: `SMTPD_PROCESS_LIMIT`. |
| `lmtp_destination_concurrency_limit` | Postfix | 20 | Deliveries to Dovecot in parallel. Kit: `LMTP_CONCURRENCY`. |
| `in_flow_delay` | Postfix | 1s | Slows down senders when the queue manager cannot keep up. Keep it on. |
| `message_size_limit` | Postfix | 10240000 (10 MB) | Largest message accepted. |
| `smtpd_client_connection_count_limit` | Postfix | 50 | Connections at once from one client address (not applied to `mynetworks`, which includes the test clients). |
| `minimal_backoff_time`, `maximal_backoff_time` | Postfix | 300s, 4000s | Waiting time before a deferred message is tried again. |
| `defer_transports` | Postfix | (empty) | Holds back delivery for a transport; used by the `queue` test. |
| `service imap-login { service_count }` | Dovecot | 1 | 1 = new login process per connection (high-security), 0 = long-running (high-performance). Kit: `DOVECOT_LOGIN_MODE`. |
| `process_min_avail` | Dovecot | 0 | Processes started in advance, ready for new connections. Kit: `DOVECOT_LOGIN_MODE`. |
| `service imap { process_limit }` | Dovecot | 1024 | Most IMAP sessions at once. Kit: `IMAP_PROCESS_LIMIT`. |
| `default_client_limit` | Dovecot | 1000 | Clients of Dovecot's helper services; must grow with the process limits (the kit sets it). |
| `mail_max_userip_connections` | Dovecot | 10 | Sessions per user and IP address. |
| `mail_fsync` | Dovecot | optimized | When mailbox writes are forced to disk. Kit: `MAIL_FSYNC`. |
| `auth_cache_size` | Dovecot | 0 (off) | Caches password checks; helps when the auth process is the limit. |
| `auth_failure_delay` | Dovecot | 2 secs | Delay before a failed login is answered. |
| `mail_location` | Dovecot | (auto) | Mailbox format: `maildir:` (one file per message, the kit's choice), `mdbox:` (many messages per file, fewer files, faster for large mailboxes). |
| password scheme | Dovecot users file | SHA512-CRYPT | Cost of every password check. Kit: `PASSWORD_SCHEME`. |

What the kit sets in its own instances (`start-mail.sh`): listen only on
`127.0.0.1`; only the test domain `perf.test` is accepted, unknown addresses
are refused, and nothing is sent to other hosts; no TLS; the values from
`settings.conf`. Everything else is left at the RHEL defaults.

---

## 6. Limits of this test

- **Client and server on the same machine.** The load generator takes CPU
  from Postfix and Dovecot, and there is no real network. On a real network,
  add the round-trip time to every step of a session (an SMTP session has
  about 6 round trips). To test from a separate machine, copy
  `lib/mailload.py` there and run it with `--server <address>` (then
  `MAIL_HOST` must be a reachable address).
- **No TLS.** Production mail servers use STARTTLS on port 25 and 587 and
  IMAPS/POP3S. TLS adds a handshake per new connection (significant for the
  login and SMTP-session tests) and a little CPU per message.
- **No content filters.** Spam and virus filters (SpamAssassin, ClamAV,
  rspamd) usually cost far more than Postfix and Dovecot themselves.
- **No DNS, no remote delivery.** Real mail servers look up SPF, DKIM,
  DMARC and blocklists for every incoming message, and deliver outgoing mail
  to other servers over the internet. Both add network waits.
- **One recipient per message, plain text.** Real mail has several
  recipients and MIME attachments; Dovecot must then write one copy per
  recipient mailbox.
- **Mailboxes are small and in the page cache.** 100 messages per user fit
  in RAM on the test machine. Real mailboxes have tens of thousands of
  messages and gigabytes of data; reading then depends much more on the disk.
- **Synthetic mix.** Each test uses one kind of operation. A real server gets
  a mix; use the per-test numbers to estimate it.
- **Shared machine.** Other programs on the same machine change the results;
  run the tests on a quiet machine, and repeat them to see the variation.

---

## 7. Glossary

| Term | Meaning |
|---|---|
| **Auth penalty** | Dovecot's growing delay for logins from an IP address with recent failed logins. |
| **Bounce** | A message sent back to the sender because delivery failed. |
| **Deferred** | A message Postfix could not deliver yet; it stays in the queue and is tried again later. |
| **Dovecot** | The IMAP/POP3 server of RHEL; it also stores delivered mail (LMTP). |
| **EHLO** | The first command of an SMTP session: the sender introduces itself. |
| **fsync** | Forcing written data from memory to the disk, so it survives a power failure. |
| **IMAP** | Internet Message Access Protocol: users read mail that stays on the server (port 143). |
| **in_flow_delay** | Postfix slowing down senders when mail arrives faster than it can be delivered. |
| **LMTP** | Local Mail Transfer Protocol: how Postfix hands a message to Dovecot for storing. |
| **Maildir** | Mailbox format: one file per message, in the folders `new/`, `cur/`, `tmp/`. |
| **MTA** | Mail Transfer Agent: the program that receives and sends mail (Postfix). |
| **p50 / p95 / p99** | Percentiles: 50 / 95 / 99 of every 100 operations were faster than this value. |
| **POP3** | Post Office Protocol: users download mail (port 110). |
| **Postfix** | The SMTP server (MTA) of RHEL. |
| **PSS** | Proportional Set Size: a process's memory, counting shared memory by its fair share. |
| **Queue** | Postfix's folder of accepted, not yet delivered messages (`/var/spool/postfix`). |
| **Queue manager (qmgr)** | The Postfix process that decides when each queued message is delivered. |
| **SMTP** | Simple Mail Transfer Protocol: how mail is sent between servers (port 25). |
| **smtpd** | The Postfix process that handles one incoming SMTP session. |
| **Virtual users** | Mail users that exist only in the mail server's user list, not as Linux accounts. |