# Samba Server - Performance Metrics This document explains the performance metrics of the Samba file server on RHEL 9.6 (the SMB daemon `smbd` from the `samba` package): what each metric means, why it matters, how `test-samba.sh` measures it, and what to change when a result is poor. --- ## Contents 1. [Samba in one minute](#1-samba-in-one-minute) 2. [The metrics at a glance](#2-the-metrics-at-a-glance) 3. [How the tests work](#3-how-the-tests-work) 4. [The metrics in detail](#4-the-metrics-in-detail) - [4.1 Startup time](#41-startup-time) - [4.2 Login (new connections per second)](#42-login-new-connections-per-second) - [4.3 Sequential write](#43-sequential-write) - [4.4 Sequential read](#44-sequential-read) - [4.5 Random read (IOPS)](#45-random-read-iops) - [4.6 Random write (IOPS)](#46-random-write-iops) - [4.7 Latency](#47-latency-how-long-one-request-takes) - [4.8 Errors](#48-errors) - [4.9 Small files (metadata)](#49-small-files-metadata) - [4.10 Concurrency](#410-concurrency) - [4.11 CPU per request](#411-cpu-per-request) - [4.12 Memory per connection](#412-memory-per-connection) 5. [How the metrics relate to each other](#5-how-the-metrics-relate-to-each-other) 6. [How much performance do you need?](#6-how-much-performance-do-you-need) 7. [Samba settings that affect performance](#7-samba-settings-that-affect-performance) 8. [Limits of this test](#8-limits-of-this-test) 9. [Glossary](#9-glossary) --- ## 1. Samba in one minute Samba lets Windows, macOS and Linux computers use folders of a Linux server over the network, with the **SMB** protocol (Server Message Block, the protocol of Windows file sharing; the old name is CIFS). A shared folder is called a **share**. Users "map a network drive" (Windows) or **mount** the share (Linux: `mount -t cifs`) and then work with its files like local ones. ``` client computers Samba server +---------------------------+ +-----------------------------------+ | Windows PC (user anna) |---- TCP 445 -->| smbd (main process, listens) | | Linux PC (mount) |---- TCP 445 -->| | starts one process per | | Windows PC (user ben) |---- TCP 445 -->| | client connection: | +---------------------------+ | +-- smbd[10.0.0.21] (anna) | | +-- smbd[10.0.0.35] (linux) | requests: CREATE (open), READ, WRITE, | +-- smbd[10.0.0.40] (ben) | CLOSE, QUERY_DIRECTORY, LOCK, ... | | | | v | | file system (XFS) + page cache | | shared databases (*.tdb): | | open files, locks, sessions | +-----------------------------------+ ``` A few facts explain almost all Samba performance: - **One process per client connection.** `smbd` starts a new process for every connection. One busy client is served by **one** process (which uses helper threads for disk I/O); many clients spread over many processes and CPU cores. So memory grows with the number of connected clients (section 4.12), and a login costs a new process (section 4.2). - **Logging in is real work.** Every new connection negotiates the protocol version, checks the password (NTLMv2 or Kerberos) and opens the share. - **Every request is a round trip.** Opening a file, reading 4 KiB, closing it: each is one request to the server and one answer. On a real network each round trip adds the network delay (0.1 - 0.5 ms on a LAN, more over a WAN). Many small operations are therefore much slower than a few big ones. - **Windows rules on a Linux file system.** Windows file names ignore upper and lower case; Linux names do not. Before Samba creates a new file it must make sure no file with the same name in another case exists, which gets slower in very big folders (section 4.9). Samba also keeps Windows file attributes, open-file and lock information in extra databases. - **Caching hides requests.** Clients cache file data when the server gives them a *lease* (oplock). The server keeps file data in RAM (page cache). Repeated reads may never reach the disk. The tests control this on purpose (section 3). - **Security costs CPU.** SMB3 **signing** protects every message against changes; SMB3 **encryption** also hides the data. Both cost CPU for every byte (section 7). --- ## 2. The metrics at a glance | # | Metric | Unit | Question it answers | Better | Test name | |---|---|---|---|---|---| | 1 | Startup time | ms | How fast can users log in after a (re)start? | lower | `startup` | | 1b | Recovery after restart | s | How long until connected clients can work again? | lower | `startup` | | 2 | Login rate | logins/s | How many new client connections per second? | higher | `login` | | 3 | Sequential write | MB/s | How fast can a client write a big file? | higher | `seqwrite` | | 4 | Sequential read | MB/s | How fast can a client read a big file? | higher | `seqread` | | 5 | Random read | IOPS | How many small reads per second can the server answer? | higher | `randread` | | 6 | Random write | IOPS | How many small writes per second? | higher | `randwrite` | | 7 | Latency | ms | How long does one request take in normal use? | lower | `latency` | | 8 | Errors | % | How many requests fail at a normal load? | lower | `errors` | | 9 | Small files | files/s | How many small files can be created (and deleted) per second? | higher | `metadata` | | 10 | Concurrency | clients | How many clients can work at once and still be fast? | higher | `concurrency` | | 11 | CPU per request | µs | How much processor time does smbd spend per request? | lower | `cpu` | | 12 | Memory per connection | MB | How much server RAM does one connected client cost? | lower | `memory` | Run one metric with `./test-samba.sh `, or all of them with `./test-samba.sh` (about 2-3 minutes). --- ## 3. How the tests work ``` +-----------------------------------------+ 127.0.0.2:445 +--------------------------------------+ | fio (the "applications") | forwarded by | test smbd (smbd-perf-test.service) | | | | one nftables | 127.0.0.1:4450, own smb.conf | | v | rule to | share //127.0.0.2/perftest | | kernel SMB client | 127.0.0.1:4450 | = /srv/samba-perf-test | | mount /mnt/samba-perf-test ================ 1 connection ======> smbd[127.0.0.1] | | | | | | lib/smb-load.py (Samba client library) | | | | logins, small files, many clients ======== N connections =====> smbd[127.0.0.1] x N | +-----------------------------------------+ +--------------------------------------+ | | | MB/s, IOPS, latency, errors | CPU time of all smbd processes, | (fio JSON, smb-load.py, /proc/fs/cifs/Stats) | memory (PSS) per smbd process v v results//report.md (PASS / FAIL per metric) ``` - **A Samba server of its own.** `start-samba.sh` runs a second, separate `smbd` for the tests: own configuration file (`/etc/samba/perf-test/smb.conf`), own port (4450), own user database, own folders and its own systemd service (`smbd-perf-test`). It listens on `127.0.0.1` only. A Samba server that already runs on the host (port 445) is not touched, not restarted, and does not know the test user. - **Clients use the normal SMB port.** Samba's client library in Samba 4.21 (RHEL 9.6) cannot connect to another port than 445, and the Linux SMB client always uses port 445 when it reconnects. So the test clients connect to `127.0.0.2:445`, and one nftables rule, in a table of its own (`samba_perf_test`), forwards exactly these connections to the test smbd on `127.0.0.1:4450`. Connections to any other address - for example the host's own Samba - are not touched. `stop` and `cleanup` remove the rule. - **Real SMB, on one machine.** Every request passes through a real SMB client, TCP and the real `smbd` - only the network cable is missing. - **fio** (the standard Linux storage benchmark, RHEL AppStream package `fio`) runs the data tests through the test mount, which uses the Linux kernel's own SMB client (`mount -t cifs`, like a Linux client PC). Important options: - `--direct=1`: bypass the **client's** page cache, so every read and write really becomes an SMB request. (Without it, the client would answer many reads from its own memory and the server would not be tested.) - `--ioengine=libaio --iodepth=N`: keep N requests in flight at once, the way busy applications and the kernel's read-ahead do. - `--rate_iops` with `--rate_process=poisson` (latency test): requests arrive at random moments at a fixed average rate, like independent users. - `--continue_on_error=all`: a failed request is counted, not fatal. - **lib/smb-load.py** uses Samba's own client library (RHEL package `python3-samba`) to do what one mount cannot: open **many separate connections** (each gets its own `smbd` process, exactly like separate user PCs), log in again and again, create small files, and time every single request itself. It runs the `startup`, `login`, `metadata`, `concurrency` and `memory` tests. - **Server-side measurements:** - CPU time of all processes of the test smbd. With systemd, from the service's control group (`/sys/fs/cgroup/.../cpu.stat`), which also counts client processes that already ended; without systemd, from `/proc//stat`. - Memory: **PSS** of every smbd process (`/proc//smaps_rollup`), see 4.12. - Request counters of the kernel SMB client (`/proc/fs/cifs/Stats`): requests sent, requests the server answered with an error, reconnects. - **Test files** are created on the share by the first test that needs them: | File (on the share) | Size | Used by | |---|---|---| | `fio/data.0` ... `fio/data.` | `DATA_FILE_MB` (1024 MB) each | seqwrite, seqread; `data.0` also for randread, randwrite, latency, concurrency, cpu | | `metadata/worker-*/file-*` | `SMALL_FILE_KB` (4 KB) each, deleted again | metadata | | `memory-test/connection-*` | empty, deleted again | memory | - **Server page cache.** The data files were just written, so they are in the server's RAM, and reads are answered from there. That measures Samba itself. `DROP_CACHES=yes` empties the page cache before each read test, so reads come from disk. (The page cache is shared by every program on the machine. Emptying it slows other programs down for a while, so do this only on a test machine or in a quiet hour.) - The **steady load** (`LOAD_RATE`, 500 requests/s for `DURATION` seconds) is run only once and shared by the `latency` and `errors` tests. The `cpu` test re-uses the `randread` and `seqread` runs. --- ## 4. The metrics in detail Each metric has the same five parts: **what** it is, **why** it matters, **how** the test measures it, **how to read** the result, and **how to improve** it. ### 4.1 Startup time **What:** two times, both from the command that starts the test smbd: 1. **Login works**: a new client can connect, log in and open the share. 2. **Recovery**: a client that was connected before the restart (the test mount) can create a new file again. **Why:** a restart happens at every update of `samba`, every reboot, every failover in a cluster, and after some configuration changes. While smbd is down, users cannot open or save files. SMB clients that were connected **must reconnect on their own**: Windows does it at the next access (open files may survive with *durable handles*); the Linux SMB client tries every 3 seconds, so it needs up to about 3 more seconds after the server is back. **How it is measured:** the test smbd is stopped and started `STARTUP_ROUNDS` times (3) with `systemctl stop/start smbd-perf-test`. Before each start, `smb-load.py wait-login` begins trying to log in every 10 ms; it notes the moment the first login works. The test share stays mounted during the restart; after the start, `touch` tries to create a file on it until it works. (Note: the Linux SMB client of RHEL 9 always reconnects to port **445**, even when a share was mounted with `port=`. That is one reason why the kit lets clients use `127.0.0.2:445` and forwards it to the test port.) **How to read it:** | Result | Meaning | |---|---| | Login works in under 0.5 s | normal: smbd starts quickly and opens its databases | | over 3 s | investigate: `journalctl -u smbd-perf-test`, the log `/var/log/samba/perf-test/log.smbd`; slow name lookups (DNS), a very large `passdb.tdb`, or many printers being loaded (`load printers`) | | Recovery up to about 3 s | normal for Linux clients: they retry every 3 seconds | | Recovery much longer | the client did not reconnect: check `dmesg | grep -i cifs` | **How to improve:** keep `load printers = no` on pure file servers; make sure the server's own host name resolves quickly (`/etc/hosts`); on AD member servers keep `winbindd` running so it does not have to start first. Plan restarts outside working hours: open files on Windows clients may be lost unless durable handles are used. --- ### 4.2 Login (new connections per second) **What:** how many complete client logins the server handles per second, and how long one login takes, with `WORKERS` (8) clients logging in at the same time. One login = TCP connect + NEGOTIATE (agree on SMB 3.1.1) + SESSION SETUP (check user name and password with NTLMv2) + TREE CONNECT (open the share), followed by disconnecting. **Why:** every login starts a **new smbd process** and checks a password. In the morning, when hundreds of users switch on their PCs, and for programs or scripts that connect, copy one file and disconnect (scanners, backup jobs, CI pipelines), the login rate decides whether they wait. **How it is measured:** `smb-load.py login`: each of the 8 clients connects, logs in, opens the share and disconnects again, in a loop, for `DURATION` seconds. The report also shows **smbd CPU per login** (the server's CPU time during the test divided by the logins). **How to read it:** | Result | Meaning | |---|---| | p95 under 20 ms, hundreds of logins/s | normal for local users (tdbsam) | | p95 of 100 ms or more | slow password check or slow start of the smbd process: see below | | failed logins | wrong credentials, `max connections` reached, or the server ran out of processes/memory: look at the log | In a real network add the round trips of the network: a login needs about 4 of them. **How to improve:** with Active Directory, the password check goes to a domain controller (Kerberos or NTLM pass-through) and **its** speed and distance matter most; keep `winbindd` healthy and its caches warm. Avoid slow `logon script`, `root preexec` or other scripts run at every login. Programs that connect for every single file should keep their connection open instead. --- ### 4.3 Sequential write **What:** how fast one client writes a big file, from start to end, in 1 MiB requests (`SEQ_STREAMS` = 1 stream, `SEQ_QUEUE_DEPTH` = 8 requests in flight). **Why:** backups, copying ISO images or VM disks, video, log archives, build artefacts - all of this is sequential writing. **How it is measured:** ``` fio --rw=write --bs=1M --iodepth=8 --numjobs=1 --direct=1 --end_fsync=1 \ --size=1024M --runtime=10 --time_based ... ``` The file is written again and again for `DURATION` seconds; at the end fio asks for a flush (all data safe on disk). **How to read it:** Samba writes the data into the server's page cache and answers; the kernel writes it to disk shortly after. So short runs mostly measure Samba and memory speed; long runs (`DURATION=120`) also measure the disk. | Result | Typical cause | |---|---| | over 1000 MB/s | fine; over a 10 Gbit/s network the link (about 1180 MB/s) is the limit | | 200 - 1000 MB/s | enough for 1 - 10 Gbit/s networks; with long runs the disk may be the limit | | under 200 MB/s | a slow or busy disk, SMB encryption/signing on a slow CPU, or a small `wsize` (see the mount options in the report) | | p99 time per request of whole seconds | the server's page cache is full of data still waiting for the disk, so new writes must wait for the disk. On the busy validation host one run fell from 2,700 to 60 MB/s this way while another program was writing to the same disk | **How to improve:** faster disks or RAID; XFS; keep asynchronous I/O on (`aio write size = 1`, the default); more streams (`SEQ_STREAMS=4`) show whether one client connection or the server is the limit; on fast networks **SMB3 multichannel** lets one client use several connections. Encryption costs a lot here (section 7). --- ### 4.4 Sequential read **What:** how fast one client reads a big file, from start to end, in 1 MiB requests. **Why:** opening big files: restoring backups, installing software from a share, videos, CAD models, large data sets. **How it is measured:** like 4.3 with `--rw=read`, on the same file. By default the file is in the server's page cache (it was just written), so the result is the speed of **Samba itself**. With `DROP_CACHES=yes` the page cache is emptied before the test and the first pass through the file comes from **disk**. **How to read it:** on loopback, reads from the page cache reach several GB/s. Over a real network the link speed decides: | Network | Upper limit | |---|---| | 1 Gbit/s | about 118 MB/s | | 10 Gbit/s | about 1180 MB/s | | 25 Gbit/s | about 2900 MB/s | If the loopback result is **below** your network's limit, the server (CPU, disk, encryption) is the bottleneck, not the network. **How to improve:** more RAM on the server keeps hot data in the page cache; a larger read-ahead on the server's disk for reads from disk; SMB3 multichannel on fast networks; avoid encryption where the network is trusted. `use sendfile = yes` can lower CPU use for unencrypted, unsigned reads (test it with `USE_SENDFILE=yes`; it has no effect with encryption). --- ### 4.5 Random read (IOPS) **What:** how many 4 KiB reads at random places the server answers per second (IOPS = input/output operations per second), with `RANDOM_JOBS` (4) jobs, each keeping `RANDOM_QUEUE_DEPTH` (16) requests in flight. **Why:** databases (Access, SQLite, accounting software on a share), VM disk images, mail stores, and programs that open small parts of big files all produce small random reads. Here the number of **requests** matters, not the MB/s. **How it is measured:** ``` fio --rw=randread --bs=4k --iodepth=16 --numjobs=4 --direct=1 --filename=data.0 ... ``` Every 4 KiB block is one SMB READ request. All requests come from the ONE connection of the test mount, so they are all handled by ONE smbd process (which hands the reads to its helper threads). **How to read it:** with data in the page cache, the limit is the CPU time per request (section 4.11) of this one smbd process and of the client. 10,000 IOPS or more is good for a single client; many clients together reach much more (section 4.10). With `DROP_CACHES=yes`: hard disks give about 100 - 200 IOPS each, SSDs tens of thousands. **How to improve:** more RAM (hot data stays cached), faster disks (SSD / NVMe), keep `aio read size = 1` (the default) so one process can have many reads in progress; more CPU cores help only when there are many clients. --- ### 4.6 Random write (IOPS) **What:** how many 4 KiB writes at random places the server accepts per second, with the same 4 jobs x 16 requests in flight. **Why:** databases, VM images, and programs that update parts of files. **How it is measured:** like 4.5 with `--rw=randwrite`. **How to read it:** the Linux SMB client (and most Windows programs) do not ask for *write-through*, so Samba answers as soon as the data is in the server's memory; the result then looks like random read. Programs that **do** ask for safe writes - write-through, or a *flush* after every write, which many databases do - make every write wait for the disk. Then the disk's flush speed decides: from about 100 writes/s (hard disk) to tens of thousands (enterprise SSD with power-loss protection). **How to improve:** SSDs with power-loss protection for databases. Keep `strict sync = yes` (the default): it makes client flush requests really reach the disk. `strict sync = no` is faster for such programs but **data can be lost when the server crashes** - use it only for data you can recreate. Never use `sync always = yes` unless you really need it: it flushes after every write. --- ### 4.7 Latency (how long one request takes) **What:** the time from sending one request until the answer arrives, at a normal, steady load: `LOAD_RATE` (500) requests per second, 70% reads and 30% writes of 4 KiB, one at a time, arriving at random moments. **Why:** users do not feel IOPS; they feel how long each action waits. Opening a folder or a document is many requests one after the other, so a slow request is multiplied. **How it is measured:** ``` fio --rw=randrw --rwmixread=70 --bs=4k --iodepth=1 --rate_iops=350,150 \ --rate_process=poisson ... ``` The report shows the average and the **percentiles**: "p95 = 0.5 ms" means 95 of every 100 requests were done within 0.5 ms. The p95 and p99 values show the slow requests that averages hide. **How to read it:** on loopback, SMB requests take well under 1 ms. On a LAN add about 0.1 - 0.5 ms per request. p99 values much higher than p95 point to short stalls: the disk flushing, a busy CPU, or lock contention between smbd processes. **How to improve:** avoid long stalls - a disk that can keep up, enough CPU, no memory pressure (swap). Check the server with `top`, `iostat -x 1`, `vmstat 1` during the test. --- ### 4.8 Errors **What:** the share of requests that failed during the steady load of 4.7. **Why:** a failed read or write is a failed save or a broken file for the user. On a healthy server under normal load there should be **none**. **How it is measured:** fio counts the requests that returned an error to the program. The kernel SMB client counts, per share (`/proc/fs/cifs/Stats`), the reads and writes the **server** answered with an error, and how often it had to **reconnect**. **How to read it:** Linux SMB mounts are *soft* by default: when the server does not answer, the program gets an error after a while (instead of waiting for ever), so server problems show up here. Typical causes: | Symptom | Typical cause | |---|---| | reconnects > 0 | smbd stopped or crashed, a network problem, or the server was too busy to answer the client's keep-alive ("echo") | | errors without reconnects | permissions, a full disk, a quota, or SELinux (look for `avc: denied` in `ausearch -m avc -ts recent`) | **How to improve:** find the cause in `/var/log/samba/perf-test/log.smbd` (raise `log level` to 3 for details), `journalctl`, and the kernel log (`dmesg`). --- ### 4.9 Small files (metadata) **What:** how many small files (`SMALL_FILE_KB`, 4 KB) `WORKERS` (8) clients can create per second, and then delete. **Why:** source code, office documents, user profiles, home folders, mail folders: most real shares hold many small files, and operations like "copy a folder", "unzip" or "build a project" are limited by files per second, not MB/s. **How it is measured:** `smb-load.py files`: every client (own connection, own folder) creates files as fast as it can for `DURATION` seconds - each one CREATE (new file) + WRITE + CLOSE, three round trips - and then deletes them all. **How to read it:** creating a file is much more work for Samba than reading 4 KiB: - It must check that no file with the same name **in another case** exists (Windows names ignore case), and for that it reads the whole folder. The bigger the folder, the slower each new file. In the validation run (8 clients, one folder each): | Test length | Files per folder at the end | Files created per second | |---:|---:|---:| | 3 s | about 2,800 | 7,500 | | 10 s | about 5,900 | 4,700 | | 20 s | about 8,600 | 3,400 | | 20 s, share with `case sensitive = yes` | about 24,600 | 19,700 | - It stores Windows attributes in an extended attribute of the file (`store dos attributes = yes`) and updates its open-file and lock databases. - The file system must create a new folder entry. Because of this, compare metadata results only between runs with the same `DURATION`. | Result | Meaning | |---|---| | over 1000 files/s | good for a local server | | 100 - 1000 files/s | typical with slow disks, AD permissions, or big folders | | under 100 files/s | investigate: disk flush speed, huge folders, virus-scanning VFS modules, audit logging | Over a real network every file costs at least 3 - 5 round trips, so clients see far fewer files per second than this test. **How to improve:** keep folders small (tens of thousands of files, not millions). For folders with a huge number of files that are always used with the same case (for example, generated files), a share with `case sensitive = yes`, `default case = lower`, `preserve case = no`, `short preserve case = no` avoids the slow check (then all names must be lower case). Avoid expensive VFS modules on busy shares (`full_audit`, virus scanners) or limit them to the operations you need. --- ### 4.10 Concurrency **What:** how many clients can work at the same time with the requests still fast. The test runs steps of `CONCURRENCY_LEVELS` (1, 4, 16, 64, 128) clients; **each client has its own connection, and so its own smbd process**, and reads 4 KiB at random places, one request at a time. **Why:** this is what a busy office looks like: many users, each doing little, all at once. Samba's process-per-client design spreads them over all CPU cores; this test shows when the cores are full. **How it is measured:** `smb-load.py readers` for each step, `DURATION` seconds each. A step **keeps up** when every client could connect, no request failed and the p99 time stayed within `TARGET_LATENCY_P99_MAX_MS`. The result is the largest step that kept up (and all steps before it). **How to read it:** total IOPS grows with the clients until the CPU cores are busy; after that it stays flat and the time per request grows. In the validation run (32 cores) IOPS grew from 20,000 (1 client) to 290,000 (128 clients) while p99 rose from 0.07 to 1.7 ms. The test clients run on the same machine and use CPU too, so on a machine with few cores they compete with smbd. If clients could not connect ("Connected" lower than "Clients") and the log `/var/log/samba/perf-test/log.smbd` says `fork() failed: Resource temporarily unavailable`, smbd could not start more processes: a limit on processes was reached (`ulimit -u`, systemd `TasksMax`, or a container's process limit). Every client costs one smbd process plus its helper threads. **How to improve:** more CPU cores; fewer expensive features per request (encryption, signing, audit modules); on a very busy server check that no single shared database is the bottleneck (`smbstatus` works, `perf top` shows where CPU time goes). --- ### 4.11 CPU per request **What:** the CPU time smbd spends to answer one 4 KiB read, and one GB of sequential reading; the report also shows the CPU per **login** from 4.2. **Why:** tells you how many CPU cores the server needs for your load: **cores = requests per second x microseconds per request / 1,000,000**. Encryption and signing change this a lot. **How it is measured:** the CPU time of all processes of the test smbd before and after the random-read run (4.5) and the sequential-read run (4.4), divided by the requests / GB. With systemd the value comes from the service's control group, which counts every process of the service, including helper threads. **How to read it:** | Result | Meaning | |---|---| | 10 - 40 µs per 4 KiB read | normal, unencrypted | | over 100 µs | encryption or signing on a CPU without AES acceleration, audit/VFS modules, or high `log level` | | several ms per login | normal: a new process starts and a password is checked | **How to improve:** CPUs with AES-NI (all current x86 server CPUs) make AES-GCM encryption and AES-CMAC signing much cheaper; keep `log level` at 0 or 1 in production; remove unneeded VFS modules. --- ### 4.12 Memory per connection **What:** how much server memory one connected client costs, measured with `MEMORY_CONNECTIONS` (200) clients connected at once, each with one open file. **Why:** because Samba starts one process per client, memory grows with the number of connected PCs - also idle ones. A server for 1,000 users needs to know this number. **How it is measured:** `smb-load.py hold` opens the connections and keeps them. The test adds up the **PSS** of all processes of the test smbd before and during the test (`/proc//smaps_rollup`). PSS (*proportional set size*) divides memory that several processes share (program code, libraries, shared databases) fairly between them, so PSS values can be added. The report also shows the **RSS** of one process, which is what `top` shows; RSS counts shared memory again in every process and looks much bigger. **How to read it:** an idle connection with one open file costs well under 1 MB of PSS (0.56 MB in the validation run), but `top` shows 15 - 20 MB RSS per smbd process. Busy clients with many open files, locks, directory listings in progress and big read/write buffers use more. **Estimate:** connected clients x MB per connection, plus 2x headroom. One PC normally opens one connection per server, whatever the number of mapped drives on that server. **How to improve:** enough RAM; `deadtime` (minutes) closes connections of idle clients that have no open files; avoid features that keep large caches per process. --- ## 5. How the metrics relate to each other ``` per BYTE: sequential read / write (4.3, 4.4) <- disk, network, encryption per REQUEST: random read / write IOPS (4.5, 4.6) <- CPU per request (4.11), latency (4.7) disk flush for safe writes per FILE: small files (4.9) <- round trips, name check, disk per CONNECTION: login rate (4.2) <- process start + password check memory per connection (4.12) <- one smbd process per client per CLIENT: concurrency (4.10) <- CPU cores per RESTART: startup (4.1) <- smbd start, client reconnect ``` - **IOPS x CPU per request = cores busy.** One client connection is served by one process, so its IOPS is limited by that process; many clients use many cores. - **IOPS x block size = MB/s.** 60,000 IOPS of 4 KiB are only 235 MB/s; big requests are how SMB reaches high MB/s. - **Concurrency and latency:** latency is flat until the CPU cores are all busy, then rises with every extra client. - **Memory and login rate are both "per connection" costs** - the price of Samba's one-process-per-client design. --- ## 6. How much performance do you need? Work out your own targets from your users, then set them in `settings.conf`: | Question | Metric | Example | |---|---|---| | How fast is the network, and how big are the files? | sequential read / write | 10 Gbit/s link: about 1,180 MB/s is the most a client can get | | How many users log in at the busiest minute? | login rate | 600 users between 8:00 and 8:05 = 2 logins/s (plus reconnects) | | How many small requests at the busiest time? | random IOPS, CPU | 300 users x 50 requests/s = 15,000 IOPS; at 20 µs = 0.3 cores | | How long may a request take? | latency | databases on a share: p99 under 5 ms; home folders: under 20 ms | | How many files are created per hour? | small files | 1 million files per hour = 278 files/s | | How many users active at the same time? | concurrency | 200 active users: the 128 step must keep up easily | | How many PCs are connected at the same time? | memory | 1,000 PCs x 1 MB = 1 GB, plus busy clients and page cache | | How long may users wait during an update? | startup | under 1 minute including the clients' reconnect | Then add **headroom**: plan for 2x the busiest load you expect. --- ## 7. Samba settings that affect performance **The kit's settings** (`settings.conf` section 2; run `./start-samba.sh` again after a change): | Setting | smb.conf parameter | Default | Effect | |---|---|---|---| | `ENCRYPTION=yes` | `server smb encrypt = required`, mount option `seal` | no | encrypts all data (AES-GCM). In the validation run sequential read fell from about 4,400 to 2,000 MB/s (together with mandatory signing) | | `SIGNING=mandatory` | `server signing = mandatory` | default | signs every message (AES-CMAC); protects against changed messages, costs CPU per byte | | `STRICT_SYNC=no` | `strict sync = no` | yes | ignores client flush requests: faster safe-writes, **data loss if the server crashes** | | `USE_SENDFILE=yes` | `use sendfile = yes` | no | the kernel sends file data directly; less CPU for unencrypted, unsigned reads | **Other smb.conf parameters** (`[global]` or per share; `man smb.conf`): | Parameter | Samba default | Effect | |---|---|---| | `server min protocol` / `server max protocol` | SMB2_02 / SMB3_11 | SMB1 is off since Samba 4.11 and should stay off (slow and unsafe) | | `aio read size`, `aio write size` | 1 (all sizes) | asynchronous I/O: one smbd process can have many reads/writes in progress | | `aio max threads` | 100 | helper threads per smbd process for asynchronous I/O | | `server multi channel support` | yes (Linux) | lets one client use several TCP connections / network cards (SMB3 multichannel) | | `oplocks`, `smb2 leases` | yes | allow clients to cache files; much fewer requests. Turn off only for special cases (databases shared by several PCs) | | `kernel oplocks` | no | only needed when NFS or local programs change the same files | | `case sensitive` | auto (= no for Windows clients) | `yes` avoids the slow name check in big folders (4.9) | | `store dos attributes` | yes | stores Windows attributes in an extended attribute: one more write per new file | | `strict locking` | auto | checks locks on every read/write only for clients that asked for them | | `deadtime` | 0 (never) | minutes after which idle connections (no open files) are closed | | `max connections` | 0 (no limit) | limits connections per share | | `log level` | 0 | 3 and higher write a lot and slow the server down | | `vfs objects` | (none) | modules like `full_audit`, virus scanners, `recycle` add work to every operation | **Client side** (Linux mount options, `man mount.cifs`): | Option | RHEL 9 default | Effect | |---|---|---| | `vers=` | 3.1.1 (newest the server offers) | SMB dialect | | `rsize` / `wsize` | 4 MiB | largest read / write per request | | `cache=` | strict | `none` turns off client caching (more requests), `loose` caches more (less safe) | | `seal` | off | requests encryption | | `soft` / `hard` | soft | `soft` returns an error after a timeout; `hard` waits for ever | | `actimeo=` | 1 s | how long cached file attributes are trusted | | `multichannel,max_channels=N` | off | several connections to one server | **File system and disk:** XFS (RHEL default) or ext4, with extended attributes (both have them). The disk's flush speed decides safe writes and file creation; RAM decides how much hot data is read from memory. **SELinux:** smbd may only share folders labelled `samba_share_t` (or with the booleans `samba_export_all_ro` / `samba_export_all_rw`) and only listen on ports labelled `smbd_port_t` (445, 139). `start-samba.sh` labels the test share and adds port 4450 to `smbd_port_t`; `cleanup` removes the port label. The nftables forwarding rule does not need any SELinux change. SELinux has no measurable effect on speed. **Firewall:** SMB needs only TCP port 445 (`firewall-cmd --add-service=samba`). The kit uses only loopback addresses (127.0.0.1, 127.0.0.2), which the firewall does not filter, so no firewall change is needed. The forwarding rule is in its own nftables table (`nft list table ip samba_perf_test`), next to firewalld's table, and is not saved: after a reboot it is gone until `./start-samba.sh` runs again. --- ## 8. Limits of this test - **Loopback only.** Client and server run on the same machine and talk over `127.0.0.1`: no network card, no network delay, no packet loss. The results are the **upper limit of the server**. Real clients also pay the network round trip on every request, which matters most for latency, logins and small files. - **Client and server share the machine.** fio, the kernel SMB client, `smb-load.py` and smbd use the same CPU cores and memory. On a small machine the clients can be the limit. The report warns when fio itself was at least 90% busy. - **Linux clients, not Windows.** The load comes from the Linux kernel's SMB client and Samba's own client library. Windows clients send different request patterns (for example more metadata queries when a folder is opened in Explorer), so real Windows workloads can be slower. - **Local users, no Active Directory.** The test user is in the test server's own user database (tdbsam, NTLMv2). With AD, logins also depend on the domain controllers and winbindd. - **Page cache.** By default reads come from the server's RAM (`DROP_CACHES=no`). Use `DROP_CACHES=yes`, or a `DATA_FILE_MB` bigger than the RAM, to measure the disk. - **Short runs.** Each test runs for `DURATION` (10) seconds. Writes land in the page cache first, and SSDs can be fast for a short time: run `DURATION=300 ./test-samba.sh seqwrite randwrite` to see the sustained speed. - **Samba version.** RHEL 9.6 ships Samba 4.21. The kit was validated with Samba 4.21.3 (AlmaLinux 9.6) and Samba 4.23.5 (AlmaLinux 9.8); the configuration uses only parameters that both versions know. - **Shared machine.** Other programs using the CPU or the disk change the results. Run on a quiet machine, and compare results only between runs on the same machine. --- ## 9. Glossary | Term | Meaning | |---|---| | **SMB** | Server Message Block: the Windows file sharing protocol. SMB 3.1.1 is the newest version (Windows 10+, RHEL 9) | | **CIFS** | old name of SMB 1; Linux still calls its SMB client "cifs" (`mount -t cifs`) | | **Samba** | the SMB server (and client tools) for Linux; the server program is `smbd` | | **Share** | a folder the server offers to clients, for example `//127.0.0.2/perftest` | | **Mount / map a drive** | attach a share to a folder (Linux) or a drive letter (Windows) | | **Connection / session** | one client's TCP connection with a logged-in user; Samba serves it with one `smbd` process | | **NEGOTIATE, SESSION SETUP, TREE CONNECT** | the three steps of a login: pick the SMB version, check the password, open the share | | **NTLMv2 / Kerberos** | password check methods; Kerberos is used in Active Directory domains | | **Oplock / lease** | permission for a client to cache a file, because no other client uses it | | **Durable handle** | an open file that survives a short disconnect of the client | | **Signing** | a checksum on every message, so nobody can change it on the way | | **Encryption (seal)** | the whole message is encrypted (AES-GCM) | | **Multichannel** | one client uses several TCP connections to the same server | | **tdb** | "trivial database": the small database files where Samba keeps users, open files, locks and sessions (`/var/lib/samba/...`) | | **IOPS** | input/output operations per second: how many requests are answered per second | | **Latency** | the time one request takes, from sending it until the answer arrives | | **p95 / p99** | 95% / 99% of all requests were at least this fast | | **Page cache** | file data the Linux kernel keeps in free RAM, so it does not have to be read from disk again | | **PSS** | proportional set size: a process's memory, with shared memory divided among the processes that share it | | **RSS** | resident set size: all memory of a process, including shared memory (what `top` shows) | | **fio** | "flexible I/O tester", the standard storage benchmark on Linux |