187 lines
9.9 KiB
Markdown
187 lines
9.9 KiB
Markdown
# OQE Collection Plan — Automated Objective Quality Evidence for the IRAP Assessment
|
|
|
|
> **Protective marking:** `STARFLEET RESTRICTED — CLEARANCE LEVEL 2 — BRIDGE CREW / SECURITY DIVISION`
|
|
> *(marking scheme per the [SAR](irap-sar.md) banner.)*
|
|
>
|
|
> **Status:** working plan (v0.1, 2026-07-11) — recorded from assessor guidance;
|
|
> not yet implemented. Companion to the [SAR](irap-sar.md) and
|
|
> [POA&M](irap-poam.md); feeds the continuous-monitoring plan (POA&M P-08 M2)
|
|
> and the delta-assessment witness points (SAR §7.2).
|
|
|
|
## 1. Purpose
|
|
|
|
Define how Objective Quality Evidence (OQE) for the assessment is captured
|
|
**automatically**, so that control effectiveness is demonstrated by
|
|
machine-collected, attributable artifacts rather than screenshots and prose
|
|
assertions — and so the delta assessment (and later, ongoing authorisation)
|
|
can sample and re-verify evidence instead of regenerating it by hand.
|
|
|
|
This system is unusually well-positioned for automated OQE because it already
|
|
has the three prerequisites:
|
|
|
|
1. **A digest that uniquely names the thing being assessed** (one release =
|
|
one OCI digest = one manifest commit).
|
|
2. **An authenticated channel from every host** (`fleet-agent` over TPM-mTLS —
|
|
evidence is cryptographically attributable to a specific hardware host).
|
|
3. **A CI pipeline that produces the security-relevant state** (signing, STIG
|
|
scan, verity coupling, content assertions).
|
|
|
|
## 2. What makes an artifact OQE
|
|
|
|
Every evidence artifact must answer four questions **without a human
|
|
interpreting it**:
|
|
|
|
| Question | Requirement |
|
|
|---|---|
|
|
| **What** does it demonstrate? | Mapped to a control/claim (ISM control ID + SSP/Annex A row + POA&M milestone). |
|
|
| **Which** system produced it? | Exact attribution: OCI digest for builds; host serial (TPM cert serial) for hosts — never "the test box". |
|
|
| **When** was it captured? | Timestamp from the authenticated time source (NTS, post-P-18). |
|
|
| **How** was it produced? | The collection method is **in-repo and inspectable** — the assessor reads the collector before trusting its output. |
|
|
|
|
Evidence strength ordering (prefer higher): **signed attestation** > raw
|
|
command output with attribution > screenshot/prose assertion. Automation
|
|
removes the human from the capture path — but only if the automation itself is
|
|
auditable.
|
|
|
|
## 3. The four collection planes
|
|
|
|
### 3.1 Build-time evidence — attached to the OCI artifact
|
|
|
|
CI already produces most of it. Formalise into a per-build **evidence bundle**,
|
|
emitted as a CI artifact and (target state) attached to the OCI image as a
|
|
**signed attestation** (cosign/in-toto via the registry referrers API), so OQE
|
|
is *keyed to the digest*: assessing release `sha256:abc…` means pulling its
|
|
evidence chain from the registry.
|
|
|
|
| Evidence | Source | Exists today? |
|
|
|---|---|---|
|
|
| STIG ARF + HTML report + deviations register (0 unjustified REVIEW) | CI artifacts (`stig-report.html`, `stig-arf.xml`, `stig-deviations.md`) | Yes |
|
|
| roothash coupling (UKI cmdline ↔ `verity.json`) | `build-oci.sh` hard gate result | Yes (gate; log it) |
|
|
| Initramfs content assertions (drivers, module, nvmf unit, NM hook) | `lsinitrd` assertions in the template | Yes (assert; capture listing) |
|
|
| SELinux labels survived export | `getfattr` build assertion | Yes (assert; capture output) |
|
|
| UKI + iPXE signature verification | `sbverify` output | Add |
|
|
| Extracted (immutable) cmdline + its hash | `extract-artifacts.sh` dump | Add |
|
|
| RPM manifest (+ size manifest) | `rpm -qa` dump (size manifest exists) | Partial |
|
|
| Reproducibility proof | second build → identical `verity.json` root hash | Add (CI job) |
|
|
| Production-profile assertion (which toggles/gates were active) | CI Production build gate output | Partial (gate exists — emit a machine-readable record) |
|
|
| Signed provenance over the whole bundle | cosign/in-toto attestation on the OCI image | Add (target state) |
|
|
|
|
Supports: SAR §7.2 items 2 and 7; POA&M P-03 M2 (profile-tagged artifacts),
|
|
P-04 M2, P-05.
|
|
|
|
### 3.2 Bench evidence — the harness emits structured results
|
|
|
|
Bench gates are witnessed live, but the harness should capture as it runs: a
|
|
**JSON results file per run** — each pass criterion with its raw output,
|
|
stamped with host serial, UKI digest, and timestamps. The assessor verifies the
|
|
harness once, watches one live run, and keeps the bundle.
|
|
|
|
| Bench gate | Captured criteria (raw outputs) | POA&M |
|
|
|---|---|---|
|
|
| Secure path (802.1X/MACsec) | `wpa_cli status` = `Authorized`; `ip macsec show`; macsec0 DHCP + renewal journal lines; `nvme list-subsys` (targeted NQN); supplicant PID alive post-switch-root | P-01 |
|
|
| Application control | novel non-RPM binary in `/tmp` → fapolicyd `FAN_DENY` audit record | SAR §7.2(7) |
|
|
| Device control | USB mass-storage block event (LinuxAudit) + console HID still functional | SAR §7.2(7) |
|
|
| SSH host certificates | `ssh -v` host-cert offer; `@cert-authority` validation; broker signing log line | P-17 M3 |
|
|
| Authenticated time | `chronyc authdata` (NTS cookies present); tcp/4460 handshake; cold-clock `nocerttimecheck` path | P-18 M3 |
|
|
| Agent actuators inert | desired-state/command received → log-only, no mutation | P-23 / SAR §7.2(7) |
|
|
|
|
### 3.3 Runtime/continuous evidence — the fleet-agent as evidence pipeline
|
|
|
|
The agent already streams SELinux state, chrony health, image version, and
|
|
cert serial over TPM-mTLS — continuous, hardware-attributed OQE for those
|
|
controls. Extending it per POA&M P-21 (SAR F-07) makes the InfluxDB store a
|
|
rolling posture record: **every host, hourly, attributed to its TPM.**
|
|
|
|
Facts to add (P-21): Secure Boot state (`SecureBoot` EFI var), fapolicyd
|
|
enforcement mode, kernel cmdline hash, USBGuard status.
|
|
|
|
This time-series **is** the substance of the continuous-monitoring plan (P-08
|
|
M2) and the ongoing-authorisation story; its retention follows the P-08/P-09
|
|
log-retention policy so the evidence itself survives. It also provides
|
|
machine-checkable closure for other items — e.g. P-05 M2 closes when telemetry
|
|
shows `mokutil --sb-state` = enabled for 100% of enrolled hosts.
|
|
|
|
### 3.4 Detect-pillar evidence — synthetic control tests
|
|
|
|
Exercise the detection path on a schedule and record the result: a probe runs
|
|
`sudo` on a canary host, then confirms the `USER_CMD` event (a) landed in the
|
|
collector's per-host file and (b) fired the Wazuh `fleet_audit` rule, within N
|
|
seconds. Log each probe result. A control tested weekly with recorded outcomes
|
|
is far stronger OQE than "the rule existed at assessment time."
|
|
|
|
Supports: Annex A §2.6 rows; POA&M P-09.
|
|
|
|
## 4. The mapping layer (where automation usually fails)
|
|
|
|
Evidence without a control map is a folder of files. Maintain a
|
|
machine-readable matrix in-repo — `docs/oqe-matrix.yaml` (to be scaffolded) —
|
|
one row per claim:
|
|
|
|
```yaml
|
|
- id: OQE-013
|
|
control: ISM-1542 # key against the ACSC OSCAL ISM edition
|
|
claim: "Only fleet-signed UKIs boot under Secure Boot" # SSP/Annex A row
|
|
plane: build # build | bench | runtime | detect
|
|
collector: "sbverify --cert fleet-db.pem fleet-<ver>.efi"
|
|
frequency: per-build
|
|
closes: P-05 # POA&M milestone this evidences
|
|
```
|
|
|
|
Two decisions recorded with it:
|
|
|
|
1. **Key controls against the ACSC OSCAL edition of the ISM.** The ISM is
|
|
published in OSCAL; stable machine-readable identifiers per release make the
|
|
May-2026 renumbering reconciliation (P-10) a computable diff instead of a
|
|
manual crosswalk.
|
|
2. **Define closure evidence per POA&M milestone.** Each M-milestone states
|
|
its machine-checkable "done" signal in the matrix, so POA&M status stops
|
|
being self-asserted.
|
|
|
|
## 5. Integrity and chain of custody
|
|
|
|
Evidence about a security system is itself a target. Proportionate measures at
|
|
this classification:
|
|
|
|
- **Build evidence:** covered by the CI attestation signature (target state);
|
|
until then, SHA256 manifests of each bundle **committed to git** — the
|
|
one-commit-per-release convention makes git history the tamper-evident
|
|
ledger.
|
|
- **Runtime evidence:** inherently attributed by the agent's mTLS session +
|
|
TPM-bound identity.
|
|
- **Bench bundles:** hashed into a committed manifest per run.
|
|
- **Timestamps:** ride the NTS-verified clock (P-18).
|
|
- Storage: append-only files on the collector + committed manifests. An
|
|
elaborate WORM store is **not** proportionate at STARFLEET RESTRICTED.
|
|
|
|
## 6. What the assessor still does in person
|
|
|
|
Automation changes the character of the SAR §7.2 witness points, it does not
|
|
replace them. At the delta assessment the assessor will:
|
|
|
|
1. **Read the collectors** — confirm they measure what they claim.
|
|
2. **Watch one live run** of each critical bench (P-01 above all).
|
|
3. **Independently re-run a sample** of collection commands and diff against
|
|
the automated bundles.
|
|
|
|
If the automated evidence survives that sampling, everything it produced
|
|
*between* assessments is credible — which is what makes ongoing authorisation
|
|
cheap instead of an annual re-assessment.
|
|
|
|
## 7. Implementation backlog (feeds TODO.md / POA&M)
|
|
|
|
| # | Task | Plane | Pairs with |
|
|
|---|---|---|---|
|
|
| O-1 | CI `evidence` job: bundle existing artifacts + add `sbverify`/cmdline/rpm-manifest dumps; emit machine-readable production-gate record | Build | P-03 M2 |
|
|
| O-2 | Reproducibility CI job (double build, compare root hash) | Build | SAR §7.2(7) |
|
|
| O-3 | Signed attestation on the OCI image (cosign/in-toto, referrers API) | Build | P-03 M2 |
|
|
| O-4 | Bench harness structured JSON output (`serving/dot1x-bench/` first) | Bench | P-01 |
|
|
| O-5 | Agent posture facts: SB state, fapolicyd mode, cmdline hash, USBGuard | Runtime | P-21 |
|
|
| O-6 | Synthetic detect probe (canary `sudo` → collector file + Wazuh alert, scheduled) | Detect | P-09 |
|
|
| O-7 | Scaffold `docs/oqe-matrix.yaml` from current POA&M rows; adopt OSCAL ISM keys | Mapping | P-10, P-08 M2 |
|
|
| O-8 | Define closure-evidence signal for every open POA&M milestone in the matrix | Mapping | all |
|
|
|
|
---
|
|
|
|
*End of OQE collection plan. Review alongside the POA&M at each monthly
|
|
review; promote O-items into TODO.md as they are scheduled.*
|