# OQE Collection Plan — Automated Objective Quality Evidence for the IRAP Assessment

> **Protective marking:** `STARFLEET RESTRICTED — CLEARANCE LEVEL 2 — BRIDGE CREW / SECURITY DIVISION`
> *(marking scheme per the [SAR](irap-sar.md) banner.)*
>
> **Status:** working plan (v0.1, 2026-07-11) — recorded from assessor guidance;
> not yet implemented. Companion to the [SAR](irap-sar.md) and
> [POA&M](irap-poam.md); feeds the continuous-monitoring plan (POA&M P-08 M2)
> and the delta-assessment witness points (SAR §7.2).

## 1. Purpose

Define how Objective Quality Evidence (OQE) for the assessment is captured
**automatically**, so that control effectiveness is demonstrated by
machine-collected, attributable artifacts rather than screenshots and prose
assertions — and so the delta assessment (and later, ongoing authorisation)
can sample and re-verify evidence instead of regenerating it by hand.

This system is unusually well-positioned for automated OQE because it already
has the three prerequisites:

1. **A digest that uniquely names the thing being assessed** (one release =
   one OCI digest = one manifest commit).
2. **An authenticated channel from every host** (`fleet-agent` over TPM-mTLS —
   evidence is cryptographically attributable to a specific hardware host).
3. **A CI pipeline that produces the security-relevant state** (signing, STIG
   scan, verity coupling, content assertions).

## 2. What makes an artifact OQE

Every evidence artifact must answer four questions **without a human
interpreting it**:

| Question | Requirement |
|---|---|
| **What** does it demonstrate? | Mapped to a control/claim (ISM control ID + SSP/Annex A row + POA&M milestone). |
| **Which** system produced it? | Exact attribution: OCI digest for builds; host serial (TPM cert serial) for hosts — never "the test box". |
| **When** was it captured? | Timestamp from the authenticated time source (NTS, post-P-18). |
| **How** was it produced? | The collection method is **in-repo and inspectable** — the assessor reads the collector before trusting its output. |

Evidence strength ordering (prefer higher): **signed attestation** > raw
command output with attribution > screenshot/prose assertion. Automation
removes the human from the capture path — but only if the automation itself is
auditable.

## 3. The four collection planes

### 3.1 Build-time evidence — attached to the OCI artifact

CI already produces most of it. Formalise into a per-build **evidence bundle**,
emitted as a CI artifact and (target state) attached to the OCI image as a
**signed attestation** (cosign/in-toto via the registry referrers API), so OQE
is *keyed to the digest*: assessing release `sha256:abc…` means pulling its
evidence chain from the registry.

| Evidence | Source | Exists today? |
|---|---|---|
| STIG ARF + HTML report + deviations register (0 unjustified REVIEW) | CI artifacts (`stig-report.html`, `stig-arf.xml`, `stig-deviations.md`) | Yes |
| roothash coupling (UKI cmdline ↔ `verity.json`) | `build-oci.sh` hard gate result | Yes (gate; log it) |
| Initramfs content assertions (drivers, module, nvmf unit, NM hook) | `lsinitrd` assertions in the template | Yes (assert; capture listing) |
| SELinux labels survived export | `getfattr` build assertion | Yes (assert; capture output) |
| UKI + iPXE signature verification | `sbverify` output | Add |
| Extracted (immutable) cmdline + its hash | `extract-artifacts.sh` dump | Add |
| RPM manifest (+ size manifest) | `rpm -qa` dump (size manifest exists) | Partial |
| Reproducibility proof | second build → identical `verity.json` root hash | Add (CI job) |
| Production-profile assertion (which toggles/gates were active) | CI Production build gate output | Partial (gate exists — emit a machine-readable record) |
| Signed provenance over the whole bundle | cosign/in-toto attestation on the OCI image | Add (target state) |

Supports: SAR §7.2 items 2 and 7; POA&M P-03 M2 (profile-tagged artifacts),
P-04 M2, P-05.

### 3.2 Bench evidence — the harness emits structured results

Bench gates are witnessed live, but the harness should capture as it runs: a
**JSON results file per run** — each pass criterion with its raw output,
stamped with host serial, UKI digest, and timestamps. The assessor verifies the
harness once, watches one live run, and keeps the bundle.

| Bench gate | Captured criteria (raw outputs) | POA&M |
|---|---|---|
| Secure path (802.1X/MACsec) | `wpa_cli status` = `Authorized`; `ip macsec show`; macsec0 DHCP + renewal journal lines; `nvme list-subsys` (targeted NQN); supplicant PID alive post-switch-root | P-01 |
| Application control | novel non-RPM binary in `/tmp` → fapolicyd `FAN_DENY` audit record | SAR §7.2(7) |
| Device control | USB mass-storage block event (LinuxAudit) + console HID still functional | SAR §7.2(7) |
| SSH host certificates | `ssh -v` host-cert offer; `@cert-authority` validation; broker signing log line | P-17 M3 |
| Authenticated time | `chronyc authdata` (NTS cookies present); tcp/4460 handshake; cold-clock `nocerttimecheck` path | P-18 M3 |
| Agent actuators inert | desired-state/command received → log-only, no mutation | P-23 / SAR §7.2(7) |

### 3.3 Runtime/continuous evidence — the fleet-agent as evidence pipeline

The agent already streams SELinux state, chrony health, image version, and
cert serial over TPM-mTLS — continuous, hardware-attributed OQE for those
controls. Extending it per POA&M P-21 (SAR F-07) makes the InfluxDB store a
rolling posture record: **every host, hourly, attributed to its TPM.**

Facts to add (P-21): Secure Boot state (`SecureBoot` EFI var), fapolicyd
enforcement mode, kernel cmdline hash, USBGuard status.

This time-series **is** the substance of the continuous-monitoring plan (P-08
M2) and the ongoing-authorisation story; its retention follows the P-08/P-09
log-retention policy so the evidence itself survives. It also provides
machine-checkable closure for other items — e.g. P-05 M2 closes when telemetry
shows `mokutil --sb-state` = enabled for 100% of enrolled hosts.

### 3.4 Detect-pillar evidence — synthetic control tests

Exercise the detection path on a schedule and record the result: a probe runs
`sudo` on a canary host, then confirms the `USER_CMD` event (a) landed in the
collector's per-host file and (b) fired the Wazuh `fleet_audit` rule, within N
seconds. Log each probe result. A control tested weekly with recorded outcomes
is far stronger OQE than "the rule existed at assessment time."

Supports: Annex A §2.6 rows; POA&M P-09.

## 4. The mapping layer (where automation usually fails)

Evidence without a control map is a folder of files. Maintain a
machine-readable matrix in-repo — `docs/oqe-matrix.yaml` (to be scaffolded) —
one row per claim:

```yaml
- id: OQE-013
  control: ISM-1542            # key against the ACSC OSCAL ISM edition
  claim: "Only fleet-signed UKIs boot under Secure Boot"   # SSP/Annex A row
  plane: build                  # build | bench | runtime | detect
  collector: "sbverify --cert fleet-db.pem fleet-<ver>.efi"
  frequency: per-build
  closes: P-05                  # POA&M milestone this evidences
```

Two decisions recorded with it:

1. **Key controls against the ACSC OSCAL edition of the ISM.** The ISM is
   published in OSCAL; stable machine-readable identifiers per release make the
   May-2026 renumbering reconciliation (P-10) a computable diff instead of a
   manual crosswalk.
2. **Define closure evidence per POA&M milestone.** Each M-milestone states
   its machine-checkable "done" signal in the matrix, so POA&M status stops
   being self-asserted.

## 5. Integrity and chain of custody

Evidence about a security system is itself a target. Proportionate measures at
this classification:

- **Build evidence:** covered by the CI attestation signature (target state);
  until then, SHA256 manifests of each bundle **committed to git** — the
  one-commit-per-release convention makes git history the tamper-evident
  ledger.
- **Runtime evidence:** inherently attributed by the agent's mTLS session +
  TPM-bound identity.
- **Bench bundles:** hashed into a committed manifest per run.
- **Timestamps:** ride the NTS-verified clock (P-18).
- Storage: append-only files on the collector + committed manifests. An
  elaborate WORM store is **not** proportionate at STARFLEET RESTRICTED.

## 6. What the assessor still does in person

Automation changes the character of the SAR §7.2 witness points, it does not
replace them. At the delta assessment the assessor will:

1. **Read the collectors** — confirm they measure what they claim.
2. **Watch one live run** of each critical bench (P-01 above all).
3. **Independently re-run a sample** of collection commands and diff against
   the automated bundles.

If the automated evidence survives that sampling, everything it produced
*between* assessments is credible — which is what makes ongoing authorisation
cheap instead of an annual re-assessment.

## 7. Implementation backlog (feeds TODO.md / POA&M)

| # | Task | Plane | Pairs with |
|---|---|---|---|
| O-1 | CI `evidence` job: bundle existing artifacts + add `sbverify`/cmdline/rpm-manifest dumps; emit machine-readable production-gate record | Build | P-03 M2 |
| O-2 | Reproducibility CI job (double build, compare root hash) | Build | SAR §7.2(7) |
| O-3 | Signed attestation on the OCI image (cosign/in-toto, referrers API) | Build | P-03 M2 |
| O-4 | Bench harness structured JSON output (`serving/dot1x-bench/` first) | Bench | P-01 |
| O-5 | Agent posture facts: SB state, fapolicyd mode, cmdline hash, USBGuard | Runtime | P-21 |
| O-6 | Synthetic detect probe (canary `sudo` → collector file + Wazuh alert, scheduled) | Detect | P-09 |
| O-7 | Scaffold `docs/oqe-matrix.yaml` from current POA&M rows; adopt OSCAL ISM keys | Mapping | P-10, P-08 M2 |
| O-8 | Define closure-evidence signal for every open POA&M milestone in the matrix | Mapping | all |

---

*End of OQE collection plan. Review alongside the POA&M at each monthly
review; promote O-items into TODO.md as they are scheduled.*