# Step-by-step offline DevSecOps guide: RPM packages and RHEL 9.6

Prepared 2026-09-11 for **RHEL 9.6 x86_64** using the existing `my-devsecops`
bundle. This guide assumes “package” means an RPM. Container archives, installation
ISOs and VM disks are different inputs; their workflows are described below.

No web application or `internal.example/app:build-001` image is required.
That image name in an earlier example was a placeholder, not a downloadable asset.
Skip ZAP unless the package exposes an HTTP/HTTPS service you want to test.

## 1. Understand the test sequence

| Stage | Tool / action | Required evidence |
|---|---|---|
| Plan | Jira: security requirements, expected behavior and remediation owner | Acceptance criteria |
| Source | Gitea reviews; Drone runs Gitleaks and SonarQube | Secret scan and quality/security gate |
| Build | Existing reproducible RPM build using internal dependencies | Signed RPM, build log, commit and SHA256 |
| Package | Verify signature, metadata, dependencies and scriptlets | Package inspection reports |
| Install | Install on a disposable RHEL 9.6 VM | Transaction and installed-file verification |
| Vulnerabilities | Syft inventory and Grype; Trivy for source/application files | SBOM and vulnerability reports |
| Host security | OpenSCAP against the installed RHEL system | Compliance XML and HTML |
| Functional | openQA and package-specific tests | Installation, service, reboot and upgrade results |
| Release | Drone requires all gates; store evidence in JFrog; track findings in Jira | Approved artifact and linked evidence |

A scanner working correctly can still produce a failed security gate. Our earlier
localhost test passed functionally, but the bundled ZAP image was blocked by both
vulnerability scanners. Do not use its test pass as approval of its vulnerabilities.

## 2. Prepare the connected transfer station

Use a subscribed RHEL **9.6**, matching architecture and approved update channel,
for Red Hat RPM collection. Do not substitute this workspace's AlmaLinux 9.8 RPMs.

Collect and transfer:

1. The complete `my-devsecops` directory, with authenticated `SHA256SUMS`.
2. Authorized RHEL 9.6 installation media and approved 9.6 errata/dependencies.
3. The candidate application RPM and its dependencies, previous release RPM if
   upgrade testing is required, and the publisher's independently trusted key.
4. Your internal CA certificates and any application test data/dependency mirrors.
5. Current scanner databases and the matching versions of OpenSCAP and RHEL SCAP
   Security Guide content. Transfer any external assessment resources separately.

Follow [offline prerequisites and RPM collection](../my-devsecops/INSTALL-OFFLINE.md)
sections 1–3. The supplied collector is
`my-devsecops/scripts/download-rhel-rpms.sh`; it collects scanner/host prerequisites,
**not all dependencies of your own application**. Resolve application dependencies
on a matching entitled staging host and test the resulting offline repository on
a clean VM. Keep RPM signature checking enabled.

Use only local repositories or an internal JFrog repository containing the approved
snapshot. A JFrog remote repository whose cache misses require Internet access is
not a complete offline mirror. This guide does not configure your Artifactory server.

The bundle's default vulnerability database age limit is seven days. Refresh data
on the connected station as described in the installation guide, regenerate the
manifest and authenticate the transfer. Do not turn off age validation.

## 3. Create a disposable RHEL 9.6 VM

Install the VM from your verified RHEL media. Use an isolated test network with
no Internet route; allow internal DNS, NTP, Gitea, JFrog and test services only as
needed. Strictly disconnected tests can use mounted media and no network adapter.
Maintain console access. Keep SELinux enforcing and record the target's normal
firewall configuration. Take a **clean snapshot before installing the candidate**.

On the VM, install `git`, `python3`, `tar`, `gzip`, `bzip2`, `coreutils`,
`openscap-scanner` and `scap-security-guide` from the approved offline repositories.
Install Podman only if you also test containers. Follow the repository commands in
INSTALL-OFFLINE.md; never disable RPM GPG checks to get installation to succeed.

Run from the transferred bundle directory:

```bash
sha256sum --check SHA256SUMS
sudo bash scripts/install-offline.sh /opt/devsecops
sudo unshare --net bash /opt/devsecops/scripts/test-offline.sh
```

The installer requires a new destination directory. For an existing installation,
use a new versioned prefix and adjust the commands below to it; do not overwrite
an active installation. Acceptance tests use synthetic fixtures, not your RPM.

## 4. Start a test session and record the baseline

The remaining VM commands use **one Bash root session on the disposable VM**.
Do not run package installation/removal examples on your JFrog server.

```bash
sudo -i
bash
. /etc/os-release
if [[ "$ID" != rhel || "$VERSION_ID" != 9.6 || "$(uname -m)" != x86_64 ]]; then
  echo 'Wrong target: use RHEL 9.6 x86_64'; exit 1
fi
umask 077
export PATH=/opt/devsecops/bin:$PATH
RUN_DIR="$(mktemp -d /var/tmp/devsecops-rpm.XXXXXXXX)"
echo "Keep this evidence directory: $RUN_DIR"
cat /etc/os-release > "$RUN_DIR/os-release.txt"
uname -a > "$RUN_DIR/kernel.txt"
rpm -qa --qf '%{NAME} %{EPOCHNUM}:%{VERSION}-%{RELEASE}.%{ARCH}\n' | sort > "$RUN_DIR/rpms-before.txt"
getenforce > "$RUN_DIR/selinux-before.txt"
systemctl --failed --no-pager > "$RUN_DIR/services-before.txt"
```

Create job-specific database copies:

```bash
mkdir -p "$RUN_DIR/cache/trivy" "$RUN_DIR/cache/grype"
cp -a /opt/devsecops/cache/trivy/. "$RUN_DIR/cache/trivy/"
cp -a /opt/devsecops/cache/grype/. "$RUN_DIR/cache/grype/"
export TRIVY_CACHE_DIR="$RUN_DIR/cache/trivy"
export GRYPE_DB_CACHE_DIR="$RUN_DIR/cache/grype"
```

Record tool versions and snapshot metadata alongside the reports. For example:

```bash
trivy --version > "$RUN_DIR/trivy-version.txt"
syft version > "$RUN_DIR/syft-version.txt"
grype version > "$RUN_DIR/grype-version.txt"
oscap --version > "$RUN_DIR/oscap-version.txt"
```

## 5. Inspect the actual candidate RPM

Enter the real transferred filename; there is no example application to download:

```bash
read -r -p 'Absolute path to your candidate RPM: ' PACKAGE
PACKAGE="$(realpath -e "$PACKAGE")"
test -f "$PACKAGE"
sha256sum "$PACKAGE" | tee "$RUN_DIR/package.sha256"
rpmkeys --checksig --verbose "$PACKAGE" | tee "$RUN_DIR/package-signature.txt"
rpm -qpi "$PACKAGE" > "$RUN_DIR/package-info.txt"
rpm -qp --requires "$PACKAGE" > "$RUN_DIR/package-requires.txt"
rpm -qp --scripts "$PACKAGE" > "$RUN_DIR/package-scriptlets.txt"
rpm -qpl "$PACKAGE" > "$RUN_DIR/package-files.txt"
PKG_NAME="$(rpm -qp --qf '%{NAME}' "$PACKAGE")"
printf '%s\n' "$PKG_NAME" > "$RUN_DIR/package-name.txt"
```

**Stop** for missing files, bad signatures, `NOKEY`, unexpected architecture or
unapproved dependencies/scriptlets. Verify the signing-key fingerprint through
an independent trusted channel before importing it with `rpmkeys --import`.
Require an approved signature: successful digest checks alone do not establish
publisher identity, and an unsigned RPM can still have valid digests.
The interactive `tee` examples are for inspection; in CI use `set -o pipefail`
and explicit signature-policy checks so command failures are not hidden by `tee`.

Read the file list and scriptlets for unexpected privileged changes, external
network downloads and broad permissions. Installing an RPM executes its scriptlets;
this is why installation belongs on a disposable VM.

## 6. Install from approved offline repositories

Configure your local candidate-dependency and RHEL repositories first. Enter the
actual enabled-for-this-test repository IDs as a comma-separated list:

```bash
read -r -p 'Approved offline repository IDs, comma-separated: ' OFFLINE_REPOS
test -n "$OFFLINE_REPOS"
dnf --disablerepo='*' --enablerepo="$OFFLINE_REPOS" \
  --setopt=localpkg_gpgcheck=1 install --assumeno "$PACKAGE"
```

The preview intentionally aborts instead of installing. Inspect the proposed
transaction and signatures; then perform the install:

```bash
dnf --disablerepo='*' --enablerepo="$OFFLINE_REPOS" \
  --setopt=localpkg_gpgcheck=1 -y install "$PACKAGE" \
  > "$RUN_DIR/install.log" 2>&1
INSTALL_RC=$?
echo "$INSTALL_RC" > "$RUN_DIR/install.exit"
if (( INSTALL_RC != 0 )); then cat "$RUN_DIR/install.log"; echo 'STOP: install failed'; fi
```

Proceed only if installation succeeded. Do not use `rpm --nodeps` to bypass
missing dependencies. Fix the mirror on the connected staging station and retry
from the clean snapshot.

```bash
rpm -q "$PKG_NAME" > "$RUN_DIR/installed-package.txt"
rpm -V "$PKG_NAME" > "$RUN_DIR/package-verification.txt" 2>&1
VERIFY_RC=$?
echo "$VERIFY_RC" > "$RUN_DIR/package-verification.exit"
rpm -qa --qf '%{NAME} %{EPOCHNUM}:%{VERSION}-%{RELEASE}.%{ARCH}\n' | sort > "$RUN_DIR/rpms-after.txt"
diff -u "$RUN_DIR/rpms-before.txt" "$RUN_DIR/rpms-after.txt" > "$RUN_DIR/rpm-changes.diff"
```

A nonzero `diff` means inventories changed and is expected. `rpm -V` checks installed
files against package metadata; review every difference, including legitimately
modified configuration. It does not test application functionality.

## 7. Generate a RHEL RPM inventory and scan vulnerabilities

Use the installed VM's RPM database so the SBOM includes package versions,
dependencies and RHEL distribution context. Scanning only extracted RPM files can
lose this context. The following **RPM-only inventory** avoids scanning reports,
scanner caches and unrelated user files:

```bash
export SYFT_CHECK_FOR_APP_UPDATE=false GRYPE_CHECK_FOR_APP_UPDATE=false
export GRYPE_DB_AUTO_UPDATE=false GRYPE_DB_VALIDATE_AGE=true
export GRYPE_DB_MAX_ALLOWED_BUILT_AGE=168h
unshare --net syft scan -c /opt/devsecops/examples/syft-offline.yaml dir:/ \
  --select-catalogers rpm-db-cataloger \
  -o "syft-json=$RUN_DIR/rhel.syft.json" \
  -o "cyclonedx-json=$RUN_DIR/rhel.cdx.json"
```

Verify the SBOM actually contains the installed candidate before accepting coverage:

```bash
python3 - "$RUN_DIR/rhel.syft.json" "$PKG_NAME" <<'PY'
import json, sys
s = json.load(open(sys.argv[1]))
packages = [p for p in s.get('artifacts', []) if p.get('type') == 'rpm']
assert packages, 'STOP: no RPM inventory'
assert any(p['name'] == sys.argv[2] for p in packages), 'STOP: candidate absent'
print('RPM components:', len(packages))
print('Distribution:', s.get('distro'))
PY
unshare --net grype -c /opt/devsecops/examples/grype-offline.yaml \
  "sbom:$RUN_DIR/rhel.syft.json" --fail-on high \
  -o json --file "$RUN_DIR/rhel-grype.json"
GRYPE_RC=$?
echo "$GRYPE_RC" > "$RUN_DIR/rhel-grype.exit"
```

Exit 0 means no findings at the configured threshold; 2 means High/Critical
findings; other nonzero values mean scanner/data errors. Both findings and errors
block approval pending triage. Inspect that distro metadata represents RHEL 9.6.
This inventories the **whole VM**, so distinguish pre-existing OS findings from
candidate/dependency findings; keep a baseline SBOM/scan from the clean snapshot.

The RPM-only scan does not inventory libraries bundled inside your application.
Also scan the application's actual installation directory and available source:

```bash
read -r -p 'Actual application installation directory (not /): ' APP_DIR
/opt/devsecops/scripts/scan-supply-chain.sh dir "$APP_DIR" "$RUN_DIR/application-sbom"
# If source is available, enter its actual checkout path:
read -r -p 'Actual source checkout directory: ' SOURCE_DIR
/opt/devsecops/scripts/scan-source.sh "$SOURCE_DIR" "$RUN_DIR/source-security"
```

Keep reports outside the scanned directory. Run source checks only when source is
available, and run SonarQube using your existing language/build configuration.
Generic CVE matching may not recognize proprietary code, statically linked
components or custom RPM names. Maintain build-time component metadata; consult
Red Hat advisories for backported fixes instead of comparing upstream versions
alone. SBOM coverage must be reviewed, even when a scanner reports no findings.

## 8. Evaluate RHEL configuration with OpenSCAP

Run on the installed VM, not inside an unrelated scanner container:

```bash
CONTENT=/usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml
test -r "$CONTENT"
oscap info "$CONTENT" > "$RUN_DIR/available-profiles.txt"
cat "$RUN_DIR/available-profiles.txt"
read -r -p 'Exact approved profile ID from that list: ' PROFILE
OSCAP_RC=0
unshare --net oscap xccdf eval --profile "$PROFILE" \
  --results "$RUN_DIR/rhel-xccdf.xml" \
  --results-arf "$RUN_DIR/rhel-arf.xml" \
  --report "$RUN_DIR/rhel-compliance.html" "$CONTENT" \
  > "$RUN_DIR/openscap.log" 2>&1 || OSCAP_RC=$?
echo "$OSCAP_RC" > "$RUN_DIR/openscap.exit"
```

Choose a profile matching your organization's policy, not simply whichever passes.
Exit 2 indicates failed rules; other errors also block assessment approval.
Review result coverage even if the exit status is zero:

```bash
python3 - "$RUN_DIR/rhel-xccdf.xml" <<'PY'
from collections import Counter
import sys, xml.etree.ElementTree as ET
ns = {'x': 'http://checklists.nist.gov/xccdf/1.2'}
r = ET.parse(sys.argv[1]).getroot()
c = Counter(n.text for n in r.findall('.//x:rule-result/x:result', ns))
print(dict(c))
if not c['pass'] or any(c[k] for k in ['fail', 'error', 'unknown', 'notchecked']):
    sys.exit('STOP: failed or incomplete compliance evaluation')
PY
```

Review `notapplicable` and `notselected` against the chosen profile and target.
Do not enable `--fetch-remote-resources` offline. If content needs external data,
import the matching resources first and configure local-resource access; missing
checks are not a pass. Configuration compliance is different from CVE matching.
If policy requires Red Hat OVAL vulnerability assessment, stage its current RHEL 9
feed separately; an OVAL command completing does not itself mean zero CVEs.

Review remediations, apply them to the disposable VM, and rerun both OpenSCAP and
functional tests. Do not automatically remediate your JFrog or production hosts.

## 9. Test package behavior and the RHEL system

Use your existing openQA worker and RHEL 9.6 test scenario. The bundle does not
supply your product's openQA tests or a universal application smoke command.
Create separate jobs/snapshots for:

1. Clean installation: dependencies resolve offline and expected files/users exist.
2. Functional behavior: execute documented CLI/library/service workflows and assert
   expected outputs, return codes, permissions and data changes.
3. Service lifecycle, if applicable: start/stop/restart and enabled-at-boot behavior.
4. Reboot: installed kernel boots, storage mounts and application behavior survives.
5. Upgrade from the supported previous release: configuration and data are preserved.
6. Uninstall: preview the transaction, remove the candidate on a disposable clone,
   and verify policy for retained configuration/data and removed services.
7. Failure cases: invalid input, restricted user access, missing dependencies,
   network unavailability, full storage and interrupted operations as applicable.
8. Performance/resource limits from your project's documented acceptance criteria.

For a service package, enter its real systemd unit name; skip this block for a
library or CLI-only package:

```bash
read -r -p 'Actual application systemd unit: ' UNIT
systemctl start "$UNIT"
systemctl is-active "$UNIT"
systemctl status "$UNIT" --no-pager > "$RUN_DIR/service-status.txt"
journalctl -u "$UNIT" -b --no-pager > "$RUN_DIR/service-journal.txt"
```

“Active” alone is not a functional pass. Invoke the application's real operation
and assert its result. Compare `systemctl --failed` and relevant SELinux/audit logs
with the baseline. Document expected warnings separately from regressions.
Do not disable SELinux just to obtain a successful test.

For uninstall testing, inspect `dnf remove --assumeno "$PKG_NAME"` on a disposable
clone, then perform the approved removal transaction there. For upgrade testing,
start with the previous package installed; a clean install does not prove upgrades.
Record `$RUN_DIR` before reboot because shell variables do not survive a new login.

## 10. If the deliverable is an OS image or container

**RHEL ISO / QCOW2 / raw VM disk:** verify the release checksum/signature, then
boot/install it in the isolated VM or openQA scenario. Perform steps 4, 7, 8 and 9
inside that running system. Scanning an ISO's file list cannot establish bootability,
installed package state or effective host compliance. Assess the final built OS,
not just its original installation media. Keep the disk/ISO hash in the evidence.

**Container archive:** use an actual archive produced by
`podman save --format docker-archive`. Run the following on the scanner worker:

```bash
read -r -p 'Actual container archive path: ' IMAGE_TAR
test -s "$IMAGE_TAR"
IMAGE_REPORTS="$(mktemp -d "$HOME/devsecops-image.XXXXXXXX")"
/opt/devsecops/scripts/scan-image.sh "$IMAGE_TAR" "$IMAGE_REPORTS/trivy"
/opt/devsecops/scripts/scan-supply-chain.sh docker-archive "$IMAGE_TAR" "$IMAGE_REPORTS/sbom"
```

These scan the archive without starting the application. They do not accept a
QCOW2/ISO as a Docker archive and do not replace host compliance or functional tests.

## 11. Enforce gates in Drone and retain evidence in JFrog

Adapt your existing Drone pipeline; the supplied
[exec example](../my-devsecops/examples/drone-exec.yml) is an integration template,
not a configured RHEL VM provisioning/openQA pipeline.

1. Source scans and SonarQube gate must pass before promotion.
2. Build the RPM once and record its hash. Test that exact candidate on a clean VM.
3. Collect install, SBOM, vulnerability, OpenSCAP and openQA results even on failure.
4. Block release on scanner errors, missing/stale data, missing coverage, unapproved
   High/Critical findings, failed/incomplete compliance or failed functional tests.
5. Apply only reviewed, scoped exceptions with an owner and expiry in Jira.
6. Promote the exact tested package only after every required gate passes. Keep
   signing/promotion credentials in protected release jobs, away from untrusted PRs.

Suggested existing JFrog repository layout (a naming convention, not a repository
created by this guide): `project/build-id/{packages,sbom,reports,provenance}`.
Use your configured internal Artifactory URL/repository and trusted CA. Store the
candidate RPM, its signature/hash, source commit, tool/database versions, SBOMs,
scan reports, openQA job ID and approval/exception records. Restrict access to
reports that contain application paths, system configuration or logs.

Package evidence without redistributing caches or temporary test material:

```bash
tar --exclude=./cache -C "$RUN_DIR" -czf "$RUN_DIR.evidence.tar.gz" .
sha256sum "$RUN_DIR.evidence.tar.gz" > "$RUN_DIR.evidence.tar.gz.sha256"
```

Use your existing authenticated JFrog upload process. Do not paste tokens into
Git or shell command literals. No repository upload or Jira ticket creation is
performed by this guide. Open a Jira finding with CVE/rule, affected package,
severity, build/hash, evidence link, owner and remediation deadline.

## 12. Final acceptance checklist

- [ ] Genuine RHEL 9.6 x86_64 target and approved offline dependencies confirmed.
- [ ] Candidate signature verified against an independently trusted key.
- [ ] Clean install, functional, reboot, upgrade and uninstall scenarios assessed.
- [ ] Candidate and bundled dependencies appear in reviewed SBOM coverage.
- [ ] Vulnerability reports triaged; no unapproved blocking findings.
- [ ] OpenSCAP profile coverage complete and policy satisfied.
- [ ] All required openQA/SonarQube/Drone gates passed.
- [ ] Exact tested artifact and evidence retained; release approval recorded.

## References and validation limits

- [Existing offline bundle guide](../my-devsecops/QUICKSTART.md)
- [Local offline test results](../devsecops-local-test/RESULTS.md)
- [Red Hat RHEL 9 security hardening](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html-single/security_hardening/index)
- [Syft scan targets](https://oss.anchore.com/docs/guides/sbom/scan-targets/)
- [Anchore RPM inventory and vulnerability coverage](https://oss.anchore.com/docs/capabilities/rpm/)

The command interfaces were checked against the bundled Syft/Grype tools.
This guide has not installed or tested your RPM on RHEL 9.6. Previous executable
acceptance tests ran on AlmaLinux 9.8; they are not RHEL certification or proof
that your product's functional/security requirements pass.