# Worked example: what does `httpd` touch? Generated 2026-09-21T02:33:27+02:00 on 65-109-31-107.ptr - AlmaLinux 9.8 (Olive Jaguar), kernel 5.14.0-687.39.1.el9_8.x86_64 SELinux: Permissive. Analysed binary: `/usr/sbin/httpd` (httpd-core-2.4.62-13.el9_8.6.x86_64). A private instance was started on 127.0.0.1:8008 with its own ServerRoot, so any system httpd kept running untouched. ## The headline: ldd tells you almost nothing about httpd `ldd /usr/sbin/httpd` lists **10** shared libraries. During one startup the process actually loaded **8** files from `/usr/lib64/httpd/modules/` - every one of them opened by `dlopen()` after the loader had finished, so no amount of static inspection would have listed them. Modules loaded at runtime: ``` /usr/lib64/httpd/modules/mod_alias.so /usr/lib64/httpd/modules/mod_authz_core.so /usr/lib64/httpd/modules/mod_cgi.so /usr/lib64/httpd/modules/mod_dir.so /usr/lib64/httpd/modules/mod_log_config.so /usr/lib64/httpd/modules/mod_mime.so /usr/lib64/httpd/modules/mod_mpm_event.so /usr/lib64/httpd/modules/mod_unixd.so ``` > This is the general lesson, not an httpd quirk. Any plugin host - httpd, > nginx with dynamic modules, PAM, NSS, a JVM agent - hides its real code > surface behind dlopen. Trace it or you are guessing. ## Configuration it read ``` /etc/gai.conf /etc/mime.types /etc/nsswitch.conf /work/acb/elf_analysis/out/httpd-demo/server/conf/httpd.conf ``` ## Files it created or wrote ``` /dev/tty /work/acb/elf_analysis/out/httpd-demo/server/logs/access_log /work/acb/elf_analysis/out/httpd-demo/server/logs/error_log /work/acb/elf_analysis/out/httpd-demo/server/run/httpd.pid.KOpTqj ``` The log files are the ones that matter operationally: httpd holds them open for the lifetime of the process, which is why log rotation needs a reload. ## Content it served The request loop read these from DocumentRoot - this is how you prove which content a worker actually touched: ``` /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/.htaccess /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi/.htaccess /work/acb/elf_analysis/out/httpd-demo/server/htdocs /work/acb/elf_analysis/out/httpd-demo/server/htdocs/does-not-exist /work/acb/elf_analysis/out/httpd-demo/server/htdocs/.htaccess /work/acb/elf_analysis/out/httpd-demo/server/htdocs/index.html /work/acb/elf_analysis/out/httpd-demo/server/htdocs/payload.txt /work/acb/elf_analysis/out/httpd-demo/server/htdocs/payload.txt/.htaccess ``` ## Paths probed but absent Where httpd *would* have read configuration or content had it existed: ``` /.htaccess /etc/gai.conf /etc/ld.so.preload /etc/userdb /run/host /run/userdb /usr/condabin/cat /usr/condabin/id /usr/lib/userdb /usr/local/bin/cat /usr/local/bin/id /usr/local/lib/userdb /usr/local/sbin/cat /usr/local/sbin/id /usr/sbin/cat /usr/sbin/id /usr/share/Modules/bin/cat /usr/share/Modules/bin/id /var/lib/sss/mc/group /var/lib/sss/mc/initgroups /var/lib/sss/mc/passwd /var/lib/sss/pipes/nss /var/run/nscd/socket /var/run/setrans/.setrans-unix /work/.htaccess /work/acb/.htaccess /work/acb/elf_analysis/.htaccess /work/acb/elf_analysis/out/.htaccess /work/acb/elf_analysis/out/httpd-demo/.htaccess /work/acb/elf_analysis/out/httpd-demo/server/.htaccess /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/.htaccess /work/acb/elf_analysis/out/httpd-demo/server/htdocs/.htaccess /work/acb/elf_analysis/out/httpd-demo/server/htdocs/does-not-exist /work/acb/elf_analysis/out/httpd-demo/server/run/httpd.pid /work/acb/semgrep-offline/.venv/bin/cat /work/acb/semgrep-offline/.venv/bin/id ``` ## Processes: the forking MPM Single-process mode (`httpd -X`) is convenient for reading a trace, but it is not how httpd runs. Under the event MPM the parent forks a pool of workers: ``` - pid 447836 /usr/sbin/httpd ["/usr/sbin/httpd", "-DFOREGROUND", "-d", "/work/acb/elf_analysis/out/httpd-demo/server", "-f", "/work/acb/elf_analysis [exit=0, 13222 syscalls] - pid 447918 /usr/sbin/httpd (forked, no exec) [exit=0, 225 syscalls] +-- 1 thread(s) in this process [tid 447943] - pid 447943 /usr/sbin/httpd (thread of pid 447918) [exit=0, 54 syscalls] +-- 26 thread(s) in this process [tid 447948, 447951, 447953, 447956, 447959, 447962, 447964, 447967, +18 more] - pid 447948 /usr/sbin/httpd (thread of pid 447943) [exit=0, 23 syscalls] - pid 447953 /usr/sbin/httpd (thread of pid 447943) [exit=0, 22 syscalls] - pid 447959 /usr/sbin/httpd (thread of pid 447943) [exit=0, 63 syscalls] - pid 448125 /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi ["/work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi"] [exit=0, 155 syscalls] - pid 448126 /usr/bin/id ["id", "-un"] [exit=0, 121 syscalls] - pid 448127 /usr/bin/cat ["cat", "/etc/hostname"] [exit=0, 34 syscalls] - pid 447964 /usr/sbin/httpd (thread of pid 447943) [exit=0, 26 syscalls] - pid 447970 /usr/sbin/httpd (thread of pid 447943) [exit=0, 22 syscalls] - pid 447974 /usr/sbin/httpd (thread of pid 447943) [exit=0, 22 syscalls] - pid 447978 /usr/sbin/httpd (thread of pid 447943) [exit=0, 63 syscalls] - pid 448140 /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi ["/work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi"] [exit=0, 155 syscalls] - pid 448141 /usr/bin/id ["id", "-un"] [exit=0, 121 syscalls] - pid 448142 /usr/bin/cat ["cat", "/etc/hostname"] [exit=0, 34 syscalls] - pid 447981 /usr/sbin/httpd (thread of pid 447943) [exit=0, 26 syscalls] - pid 447985 /usr/sbin/httpd (thread of pid 447943) [exit=0, 22 syscalls] - pid 447989 /usr/sbin/httpd (thread of pid 447943) [exit=0, 22 syscalls] - pid 447993 /usr/sbin/httpd (thread of pid 447943) [exit=0, 63 syscalls] - pid 448159 /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi ["/work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi"] [exit=0, 155 syscalls] - pid 448160 /usr/bin/id ["id", "-un"] [exit=0, 121 syscalls] - pid 448161 /usr/bin/cat ["cat", "/etc/hostname"] [exit=0, 34 syscalls] - pid 447997 /usr/sbin/httpd (thread of pid 447943) [exit=0, 26 syscalls] - pid 448003 /usr/sbin/httpd (thread of pid 447943) [exit=0, 58 syscalls] - pid 447919 /usr/sbin/httpd (forked, no exec) [exit=0, 225 syscalls] +-- 1 thread(s) in this process [tid 447941] - pid 447941 /usr/sbin/httpd (thread of pid 447919) [exit=0, 54 syscalls] +-- 26 thread(s) in this process [tid 447945, 447947, 447949, 447952, 447955, 447958, 447960, 447963, +18 more] - pid 448002 /usr/sbin/httpd (thread of pid 447941) [exit=0, 31 syscalls] - pid 447920 /usr/sbin/httpd (forked, no exec) [exit=0, 225 syscalls] +-- 1 thread(s) in this process [tid 447921] - pid 447921 /usr/sbin/httpd (thread of pid 447920) [exit=0, 54 syscalls] +-- 26 thread(s) in this process [tid 447922, 447923, 447924, 447925, 447926, 447927, 447928, 447929, +18 more] - pid 447968 /usr/sbin/httpd (thread of pid 447921) [exit=0, 31 syscalls] ``` **12** child process(es) and **81** thread(s) were created in that window. The distinction matters: the event MPM runs a small number of worker *processes*, each with a pool of *threads*, and strace reports a thread as just another pid. A tool that does not read the CLONE_THREAD flag will tell you httpd forked dozens of processes, which is wrong - `parse_strace.py` separates them. Trace httpd without `strace -f` and you see the parent doing almost nothing while every request is served by a child you never recorded. `elf-trace.sh` always passes `-f`, which is the whole reason it finds them. ### The CGI child A CGI request makes httpd exec a program - the clearest case of a daemon running code that is not its own: ``` /usr/bin/cat exec'd 6 time(s) (pids 448127, 448142, 448161, 447178...) /usr/bin/id exec'd 6 time(s) (pids 448126, 448141, 448160, 447177...) /work/acb/elf_analysis/out/httpd-demo/server/cgi-bin/whoami.cgi exec'd 6 time(s) (pids 448125, 448140, 448159, 447176...) ``` Note what the trace records as the image: the **CGI script itself**, not `/bin/sh`. httpd calls `execve("whoami.cgi")` and the kernel resolves the `#!` line, so the shell never appears as a separate exec. The script's own children (`id`, `cat`) do. If you are hunting for what did this daemon run, that indirection is exactly the kind of thing a trace catches and a config review does not. ## Sockets ``` listening on : 127.0.0.1:8008 connections accepted: 48 outbound connects, grouped: unix:/var/run/nscd/socket (ENOENT) x34 unix:/var/lib/sss/pipes/nss (ENOENT) x24 inet:? (0) x4 unix:/run/systemd/userdb/io.systemd.Machine (0) x4 unix:/run/systemd/userdb/io.systemd.DynamicUser (0) x4 ``` The listen socket and the accept count are the obvious part. The interesting part is the outbound list: httpd itself never dials out, but every request that needs a user or group lookup makes the C library try `nscd` and then SSSD. Those are failed connects to sockets that do not exist here - harmless, but on a host where they *do* exist, your web server's request path depends on a name-service daemon you might not have thought of as part of it. ## Watching it as a daemon, without ptrace `elf-snapshot.sh` only reads /proc, so it is safe against a production daemon. What the parent held open: ``` 1w CHR /dev/null 2w REG /work/acb/elf_analysis/out/httpd-demo/server/logs/error_log 3u IPv4 127.0.0.1:8008 5w FIFO pipe 6w REG /work/acb/elf_analysis/out/httpd-demo/server/logs/access_log ``` Mapped modules seen from /proc/PID/maps (the dlopen list again, from a completely different angle - useful when you may not trace at all): ``` /usr/lib64/httpd/modules/mod_alias.so /usr/lib64/httpd/modules/mod_authz_core.so /usr/lib64/httpd/modules/mod_cgi.so /usr/lib64/httpd/modules/mod_dir.so /usr/lib64/httpd/modules/mod_log_config.so /usr/lib64/httpd/modules/mod_mime.so /usr/lib64/httpd/modules/mod_mpm_event.so /usr/lib64/httpd/modules/mod_unixd.so ``` Those are the same modules the trace found, recovered from `/proc/PID/maps` without touching the process at all. On a host where you may not ptrace a production daemon, this is how you still answer "what code is actually loaded in there". ## Applying this to a system httpd ```bash # 1. static first, it never runs anything tools/elf-static.sh /usr/sbin/httpd > httpd-static.md # 2. the system unit, traced from the start (stop it first) systemctl stop httpd tools/elf-trace.sh -T 20 -- /usr/sbin/httpd -DFOREGROUND # 3. or attach to the running service without stopping it tools/elf-trace.sh -T 30 -p "$(systemctl show -p MainPID --value httpd)" # 4. or watch it with zero impact for as long as you like tools/elf-audit.sh start /usr/sbin/httpd httpdwatch tools/elf-audit.sh report httpdwatch tools/elf-audit.sh stop httpdwatch ``` Three things about the *systemd* httpd that the trace will show and that surprise people: * **PrivateTmp=yes.** httpd's `/tmp` is a per-service namespace. A path logged as `/tmp/x` really lives in `/tmp/systemd-private-*-httpd.service-*/tmp/x`. Check with `systemctl show httpd -p PrivateTmp` and look inside with `nsenter -t "$MAINPID" -m ls /tmp`. * **SELinux.** On an Enforcing host, content outside the `httpd_sys_content_t` label gives `EACCES` in the trace with nothing else to explain it. Confirm with `ausearch -m AVC -ts recent | audit2why` before blaming the config. * **Workers change identity.** The parent starts as root and binds the port; the children `setuid` to `apache`. A file the parent can read may be denied to the worker, and only the worker's pid shows the `EACCES`. ## Reproducing this report ```bash examples/httpd-example.sh -o /work/acb/elf_analysis/out/httpd-demo -p 8008 ``` Raw evidence: `2-singleproc/strace.log`, `3-forking/strace.log`.