> **Start with [QUICKSTART.md](QUICKSTART.md)** for the full diagram, including > newly added Syft/Grype, the combined offline test and Drone exec integration. > The following detailed guide is inherited from the existing bundle. # RHEL 9.6 offline installation and operation This guide targets x86_64 RHEL 9.6. No target server has been modified. CLI checks run on the AlmaLinux 9.8 preparation host do not constitute RHEL certification. Validate on a RHEL 9.6 test VM before rollout, including SELinux, your runner account and any FIPS requirements. These upstream Go binaries are not represented as FIPS-validated cryptographic modules. ## 1. Prepare the transfer Allow about **30 GB** for the expanded bundle, installed caches and loaded scanner images, plus additional space for builds, application images and reports. Start a scanner worker at 4 vCPU/8 GB RAM and measure actual workloads; ZAP/browser-heavy tests may need more. The Java DB occupies approximately 1.5 GB expanded. Give concurrent Trivy jobs separate writable caches to avoid shared BoltDB locks. On the connected preparation station, finish the Red Hat prerequisites below, then generate the final manifest: ```bash cd /path/to/my-devsecops python3 scripts/make-manifest.py sha256sum --check SHA256SUMS ``` Transfer the entire directory using approved media. Record the manifest hash outside the media (or sign it using an already trusted organizational key). On the offline host, verify the trusted manifest and run `sha256sum --check SHA256SUMS` before executing any bundle binaries. A manifest stored with its files detects corruption; it does not independently authenticate their publisher. ## 2. Obtain genuine RHEL 9.6 prerequisites Two supported preparation paths are available. The bundle's `rpms/` starts empty. **Path A: your RHEL 9.6 Binary DVD ISO.** Transfer your authorized x86_64 DVD ISO and verify its Red Hat-published checksum. Mount it read-only: ```bash sudo mkdir -p /mnt/rhel96 sudo mount -o loop,ro /path/to/rhel-9.6-x86_64-dvd.iso /mnt/rhel96 ``` Create `/etc/yum.repos.d/rhel96-offline.repo`: ```ini [rhel96-media-baseos] name=RHEL 9.6 DVD BaseOS baseurl=file:///mnt/rhel96/BaseOS enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release [rhel96-media-appstream] name=RHEL 9.6 DVD AppStream baseurl=file:///mnt/rhel96/AppStream enabled=0 gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release ``` Install the dependencies from those repositories only: ```bash sudo dnf --disablerepo='*' \ --enablerepo=rhel96-media-baseos --enablerepo=rhel96-media-appstream \ install openscap-scanner scap-security-guide podman skopeo git python3 \ tar gzip bzip2 coreutils findutils ca-certificates audit ``` DVD packages are a baseline; bring approved RHEL 9.6 errata into the offline environment through Satellite or the matching entitled repositories. Avoid downgrading already patched hosts to DVD package versions. **Path B: subscribed RHEL 9.6 staging host.** Match target architecture, minor release and approved update channel. Confirm available repository IDs with `subscription-manager repos --list`; a system held at 9.6 may require EUS entitlement/repositories. Do not assume the ordinary moving RHEL 9 repositories still serve the required 9.6 package set. Install `dnf-plugins-core` and `createrepo_c` on that staging host from approved RHEL repositories, then run: ```bash cd /path/to/my-devsecops # These IDs are examples for an entitled EUS host; select your approved channel. export RHEL_BASEOS_REPO=rhel-9-for-x86_64-baseos-eus-rpms export RHEL_APPSTREAM_REPO=rhel-9-for-x86_64-appstream-eus-rpms bash scripts/download-rhel-rpms.sh python3 scripts/make-manifest.py ``` The script uses `--resolve --alldeps` to collect dependencies even if already installed on the staging host. Test the resulting repository in a clean RHEL 9.6 VM with network disabled. Repository/package availability still depends on your subscription; do not substitute AlmaLinux, CentOS, Fedora or another RHEL minor. After transfer, set `baseurl=file:///absolute/path/to/my-devsecops/rpms` in a local repository named `devsecops-rpms`, with `enabled=0`, `gpgcheck=1`, and the same Red Hat GPG key path used above. Then: ```bash sudo dnf --disablerepo='*' --enablerepo=devsecops-rpms \ install openscap-scanner scap-security-guide podman skopeo git python3 \ tar gzip bzip2 coreutils findutils ca-certificates audit ``` Do not use `--nogpgcheck`. Keep the ISO/local repository mounted during install. ## 3. Install the CLI tools and cache ```bash cd /path/to/my-devsecops sudo bash scripts/install-offline.sh /opt/devsecops export PATH=/opt/devsecops/bin:$PATH trivy --version gitleaks version syft version grype version ``` The installer verifies the manifest and copies binaries, scripts, configurations and databases into a **new** directory. It does not contact external services or change system security settings. For upgrades use a new versioned directory and switch your pipeline path after tests; retain the previous version for rollback. Keep `/opt/devsecops` owned by administrators. As your scanner account, prepare an individual writable cache for each job: ```bash mkdir -p "$HOME/devsecops-cache/job-001" cp -a /opt/devsecops/cache/trivy/. "$HOME/devsecops-cache/job-001/" export TRIVY_CACHE_DIR="$HOME/devsecops-cache/job-001" ``` Provide internal DNS, NTP, CA trust and authenticated access to Jira/Gitea/your registry. Offline means no public Internet; internal service connectivity is still needed. Keep SELinux enforcing. For bind-mounted container data use an appropriate `:Z` label (private) or `:z` (shared), and narrowly scoped permissions. ## 4. Source, secret, image and SBOM checks ```bash /opt/devsecops/scripts/scan-source.sh /srv/checkout/app /srv/reports/build-001 ``` This scans Git history when present, scans working files for secrets, blocks on HIGH/CRITICAL vulnerabilities or IaC findings, and emits a CycloneDX SBOM. Reports must live outside the source directory. Supply full Git history from internal Gitea; shallow clones deliberately fail. Scanner errors also fail the job. Report files are generated even when findings block promotion; preserve them in an always-run evidence upload step in your existing Drone setup. Use lockfiles and mirror exact build dependencies internally. Trivy cannot discover dependencies that are absent from its input, and offline identification can have reduced coverage. Scan both source/lockfiles and the actual built artifact. C/C++ vendored or statically linked dependencies need maintained component inventory and appropriate build metadata in addition to generic scans. For an application container built using Podman: ```bash podman save --format docker-archive -o /srv/artifacts/app.tar internal.example/app:build-001 /opt/devsecops/scripts/scan-image.sh /srv/artifacts/app.tar /srv/reports/image-001 ``` The scanner reads the local archive and does not pull images or require a Docker socket. Retain the image SBOM with the exact tested artifact. A source SBOM is not necessarily the inventory of the final release. Licensing policy and VEX exceptions require explicit review; the default scripts do not enforce them. ## 5. Load and run ZAP for web/API applications Load the image **as the same account that will run ZAP**; rootless and rootful Podman use different stores: ```bash podman load -i /path/to/my-devsecops/images/zap-stable-amd64.tar podman run --rm --pull=never --network=none \ localhost/devsecops/zap:20260909 zap.sh -version ``` The archive contains the Java runtime and bundled add-ons. Do not run an add-on update online from the offline worker. For a staging baseline scan, set an internal staging URL and use a dedicated report directory. The following is for a **rootless** Podman account; `--userns=keep-id` maps the process to the invoking user so it can write reports: ```bash mkdir -p "$HOME/zap-reports/build-001" podman run --rm --pull=never --userns=keep-id \ --user "$(id -u):$(id -g)" \ -v "$HOME/zap-reports/build-001:/zap/wrk:Z" \ localhost/devsecops/zap:20260909 \ zap-baseline.py -t https://staging.example.internal \ -r baseline.html -J baseline.json \ -z '-config autoupdate.checkOnStart=false -config autoupdate.checkAddonUpdates=false' ``` Permit container access only to authorized staging systems and required internal DNS. Configure your actual internal CA, routing and authentication/context before using results as a release gate. Baseline exit codes: 0 success, 1 policy FAIL, 2 WARN, 3 other error. CI should fail on nonzero until you define reviewed rule policy; never append `|| true` to a release gate. Baseline uses a spider and passive analysis; it is **not a complete active DAST test**. Use `zap-api-scan.py` for your API specification, or an authenticated ZAP Automation Framework plan / `zap-full-scan.py` on a disposable authorized test environment for active testing. Import API schemas, contexts, scripts and needed add-ons at the connected station. Trigger ZAP after deployment to staging, alongside openQA; application login and business workflows must be covered. ## 6. OpenSCAP on RHEL ```bash oscap --version oscap info /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml ``` Choose an available profile matching your policy from `oscap info` (for example CIS Server Level 1 if listed), and evaluate: ```bash sudo oscap xccdf eval \ --profile xccdf_org.ssgproject.content_profile_cis_server_l1 \ --results-arf /var/tmp/rhel96-arf.xml \ --report /var/tmp/rhel96-compliance.html \ /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml ``` Run on the target VM/host or a representative built system image. Exit 2 means failed rules; errors/incomplete checks must also block compliance approval. Review failed, error and notchecked results. Avoid `--fetch-remote-resources` offline. If your chosen content references external OVAL resources, stage those exact resources and use the scanner's local-resource options, then prove no checks were silently skipped. RHEL CVE OVAL feeds and SCAP configuration profiles serve different purposes. Trivy is not a replacement for Red Hat's host compliance content. Do not apply `--remediate` automatically to production; review changes, test the VM and rerun openQA after hardening. ## 7. Prepare and verify the approved release After every security and test gate succeeds, preserve the exact candidate and SBOM with SHA256 checksums. The scanner adapter provides: ```bash ci-security package /artifacts/app.tar /security/image/supply-chain/sbom.cdx.json /release/candidate ci-security verify /release/candidate ``` The output directory must be new. Checksums detect accidental changes; they do not authenticate the producer or protect against replacement of both artifact and manifest. Restrict release storage and promotion credentials. Preserve commit/build identity, reports and test results alongside the candidate. ## 8. Integrate with your existing Drone/Gitea/Jira setup The included `examples/drone-exec.yml` is a **Drone exec-runner** template. It runs commands on the host without container isolation and therefore belongs on a dedicated runner for trusted repositories. It does not run on a Docker runner simply by copying the YAML. Merge the scan scripts into your existing pipeline stages and retain your current SonarQube/openQA build logic. If your runner is Docker-based, build a dedicated scanner image offline from your approved, already mirrored base image containing Bash, Git and Python 3. COPY `bin/`, `scripts/`, and `examples/` into `/opt/devsecops/`. COPY the refreshed Trivy cache into the image or mount an administrator-managed read-only snapshot and copy it into a per-job writable cache. Push to your internal registry; pin the image digest in `.drone.yml`. Ensure Drone's clone/plugin/build images and all application dependencies are mirrored too. Host volumes require trusted repositories; never expose the host container socket just to scan an archive. Do not treat Podman as an automatically supported replacement for your existing Drone Docker runner daemon. Recommended gate order: 1. Gitleaks and SonarQube security/quality gate on reviewed source. 2. Trivy dependency/IaC scan; deterministic build using internal package mirrors. 3. Built artifact/container scan and release SBOM generation. 4. Deploy staging; openQA, authenticated ZAP tests where applicable, OpenSCAP. 5. Preserve the exact approved artifact, SBOM, checksums and evidence. 6. Deployment checks hashes and requires all gates before promotion. Set Gitea branch protections to require the security pipeline's successful status and reviews. Protect pipeline files and release branches. Restrict registry promotion credentials and production credentials to trusted release events; do not expose them to untrusted pull requests. Use Jira findings with severity, component/version, CVE/rule ID, source/build commit, owner, due date, evidence and an exception expiry if applicable. Deduplicate recurring findings. Prefer links to restricted report storage; even redacted reports can reveal sensitive code paths. This bundle does not connect to or change Jira, Gitea or Drone server settings. ## 9. Refresh databases, rules and packages Offline vulnerability knowledge becomes stale. Import fresh data at least weekly (daily if your transfer process allows). The scan scripts default to blocking when either database is more than **7 days** old, or metadata is missing/invalid. `MAX_DB_AGE_DAYS` is an explicit organizational policy setting, not an excuse to silently accept old databases. Never use `DownloadedAt` as the freshness clock. On a connected station with the **same pinned Trivy binary**, use a fresh cache: ```bash export TRIVY_CACHE_DIR=/path/to/new-cache/trivy /path/to/my-devsecops/bin/trivy image --download-db-only /path/to/my-devsecops/bin/trivy image --download-java-db-only mkdir -p /path/to/rules-probe printf 'FROM scratch\nUSER 10001\n' > /path/to/rules-probe/Dockerfile /path/to/my-devsecops/bin/trivy config --disable-telemetry --skip-version-check /path/to/rules-probe ``` Transfer `db/`, `java-db/`, and `policy/` together, with their metadata. Rebuild and authenticate the transfer manifest. On the offline side, install a new snapshot and switch new jobs to it; avoid changing databases during active scans. Retain previous snapshots for audit/rollback. Upgrade scanner versions and DB schemas together after validation. `scripts/download-tools.py` re-downloads the pinned versions; it deliberately does not silently upgrade to latest. Refresh ZAP by resolving an approved upstream tag to a digest, then use `skopeo copy --override-os linux --override-arch amd64 docker://...@sha256:... docker-archive:/absolute/path/zap.tar:localhost/devsecops/zap:APPROVED_VERSION`. Record the digest, test bundled add-ons and import the new archive. Refresh Red Hat RPMs/content through your authorized 9.6 update channel. Preserve licenses, upstream metadata and release notes when upgrading any tool. ## 10. Acceptance checks and limitations Run the included smoke test with networking disabled on a test host: ```bash sudo unshare --net bash /path/to/my-devsecops/scripts/test-offline.sh ``` It verifies detection of a deliberately old Python dependency, SBOM output, Syft/Grype vulnerability and SBOM gates. Synthetic test fixtures remain only in the printed temporary test directory and must never be reused. Read `metadata/current-offline-test.log` and `metadata/MY-VALIDATION.md` for preparation results. This is not an application penetration test or a production deployment. Before adoption, run your actual Drone build, check a deliberately failing security gate prevents promotion, verify a tampered release is rejected, scan your staging application with real authentication, and validate OpenSCAP on RHEL 9.6 using your selected profile. Test offline with egress denied and check logs for attempted external access. Keep public package fetching, update checks and scanner telemetry disabled in offline jobs.