GA Pre-Launch Penetration Test Checklist

This is the checklist promised by Threat model follow-up item 6 ("Publish a GA-readiness penetration-test checklist document"). It scopes an external or manual penetration test against the BoardReadyOps control plane (apps/web) and release-signing surface ahead of general availability, and separates what tests/integration/security-adversarial.test.ts and other automated suites already exercise from what still needs a human or external tester.

This document does not itself claim a pen test occurred. See Assurance Case claim 4 for current residual risk, which this checklist is meant to close.

Scope

In scope

apps/web control plane: bearer-token and session-cookie authentication (apps/web/lib/api-auth.ts, apps/web/lib/user-session.ts), API token scope enforcement, tenant/repository isolation, GitHub OAuth login and webhook handlers, Stripe billing webhook handler.
API token lifecycle (packages/db/src/api-token-store.ts): issuance, scope assignment, hashing/storage.
Webhook signature verification (packages/cloud-core/src/index.ts: verifyGitHubWebhook, verifyStripeWebhook) as deployed behind the real route handlers, not just as unit-tested functions.
Artifact storage path handling (resolveLocalArtifactPath) and evidence ledger integrity (buildEvidenceLedger / verifyEvidenceLedger).
Release signing-key trust store (src/release/signing.ts, release verify --trust-store) and its local file distribution.
Container/deployment configuration for apps/web/Dockerfile and apps/container/Dockerfile, and the publish workflows' OIDC/Trusted-Publishing configuration.

Out of scope

The local CLI's parsing of KiCad/BOM/pinmap project files — covered by Input Validation fuzz/property tests, not a network attack surface.
Plugin runtime sandboxing — explicitly documented as a trusted-code model in ADR 0009; not a GA blocker to re-litigate here.
GitHub's own infrastructure (Actions runners, GitHub App platform, Marketplace billing internals) — only BoardReadyOps' use of them is in scope.
Denial-of-service / load testing — tracked separately, not a pen-test checklist item.
Cross-OS/cross-CI-runner reproducible-build divergence — tracked as its own open item in Release Integrity, not a pen-test finding.

Checklist

Each item lists whether it is Automated (already exercised by a test in this repo — re-verifying it manually is low value) or Manual (needs a human/external tester against a real deployment, because it depends on runtime configuration, infrastructure, or logic no test currently asserts).

Authentication / session

#	Item	Status
A1	Invalid/forged bearer token (`bro_live_...`) is rejected.	Automated — `tests/unit/web/api-auth-repository-scope.test.ts` and token-hash lookup in `packages/db/src/api-token-store.ts`.
A2	Session cookie (`brops_session`) with a tampered payload or invalid HMAC signature is rejected; signature comparison is constant-time.	Manual — confirm `timingSafeEqual` usage in `apps/web/lib/user-session.ts` actually gates every code path (no early-return short-circuit reintroduced by a future edit) against a live deployment, not just source reading.
A3	`SESSION_SECRET` is enforced at minimum length and is not a shared default/example value in the deployed environment.	Manual — this is an environment/config check, not something a unit test can assert.
A4	Revoking a user's GitHub App installation does not immediately invalidate their existing BoardReadyOps session; confirm the session's remaining lifetime (`sessionLifetimeMs`, currently 8h) is an acceptable exposure window.	Manual — accepted-risk judgment call, not a pass/fail bug in isolation. Flag if a tester finds the window materially different from 8h in production config.
A5	Failed bearer-token attempts are rate-limited (default 20/60s, `BOARDREADYOPS_AUTH_RATE_LIMIT_PER_MINUTE`); the rate-limit key is not trivially resettable by rotating IPs.	Automated (limiter logic) — `tests/unit/web/auth-rate-limit.test.ts`. Manual — the limiter keys on the first hop of `X-Forwarded-For` (`clientIdentifierFromRequest`); confirm the production reverse proxy strips/overwrites client-supplied `X-Forwarded-For` before it reaches Next.js, otherwise the limit is spoofable per request.
A6	Failed session-cookie authentication attempts are also rate-limited, not just failed bearer-token attempts.	Manual/gap — `recordFailedAuthAttempt` is currently only called from the bearer-token failure path in `apps/web/lib/api-auth.ts`; confirm whether this is an accepted gap or needs closing before GA.
A7	OAuth `state` parameter in the GitHub login flow prevents CSRF/session-fixation on callback.	Manual — `apps/web/lib/oauth-state.ts` and the login/callback routes; no automated adversarial test currently targets this flow.

Tenant isolation

#	Item	Status
T1	A bearer token scoped to repository A cannot read or write resources for repository B, even by supplying B's id directly in the request.	Automated (partial) — `tests/integration/security-adversarial.test.ts` ("rejects cross-tenant cursor tampering", "enforces tenant scope in DB queries"). `resolveRepositoryApiContext` in `apps/web/lib/api-auth.ts` rejects a caller-supplied `repositoryId` that doesn't match the token's bound repository.
T2	Every store under `packages/db/src/` that holds repository-scoped data filters by `repository_id` in its SQL, not just in an in-memory array filter after an unscoped query.	Manual — spot-check stores beyond `review-store.ts` (which the automated test covers): `api-token-store.ts`, `review-comment-store.ts`, `runner-artifact-store.ts`, `finding-decision-store.ts`, and any added since.
T3	Every `apps/web/app/api/**` route handler that touches repository- or review-scoped data calls `resolveRepositoryApiContext` / `resolveReviewApiContext` rather than trusting a raw `repositoryId`/`reviewId` route or query parameter.	Manual — this is a per-route audit; no automated test enumerates route handlers to assert this structurally today.
T4	Session-cookie auth (which has no token-bound `repositoryId`) correctly derives tenant access from `session.installationIds` via the GitHub installation lookup, and cannot be tricked into acting on a repository outside those installations by URL/parameter manipulation.	Manual — exercise `resolveReviewApiContext` (`apps/web/lib/api-auth.ts`) against a session with a known, narrow installation set.

API / webhook surface

#	Item	Status
W1	An API token's scope (`runs:write`, `reviews:read`, `reviews:write`, `admin`) is enforced on every route that requires it; a `reviews:read` token cannot perform a write.	Manual — `tests/unit/web/api-auth-repository-scope.test.ts` covers repository scoping, not an exhaustive per-route scope matrix. Build/execute that matrix manually against a running deployment.
W2	Token creation without an explicit `scopes` argument does not silently grant broader access than the caller intended.	Manual — `packages/db/src/api-token-store.ts` defaults to all three non-admin scopes when scopes are unspecified; audit every token-creation call site to confirm none relies on this default unintentionally.
W3	Invalid Stripe webhook signature is rejected; valid signature with a stale timestamp (replay) is rejected outside the tolerance window.	Automated — `tests/integration/security-adversarial.test.ts` ("rejects invalid Stripe signature", "accepts valid Stripe signature", "rejects Stripe replay outside tolerance"); `tests/unit/cloud-core/webhook.test.ts`.
W4	GitHub webhook signature (`X-Hub-Signature-256`, `verifyGitHubWebhook`) is verified before any event processing, and a missing header is rejected (not just a malformed one).	Manual — `verifyGitHubWebhook` itself has unit coverage, but confirm the route handlers (`apps/web/app/api/github/webhook/route.ts`, `.../marketplace/webhook/route.ts`) reject a request with the header absent entirely, not only a request with a bad signature.
W5	Webhook secrets (`STRIPE_WEBHOOK_SECRET`, GitHub App webhook secret) are distinct per environment (dev/staging/prod) and not reused.	Manual — deployment/secret-management check, not testable from the repo.
W6	A notifier `webhookEnv` value that resolves to an environment variable whose name doesn't look webhook-related is surfaced, not silently used.	Automated — covered by notifier tests referenced in Threat model (`notifier.webhook.unrecognized-env-name` warning); `tests/unit/notifiers/notifiers.test.ts`.
W7	Stored content rendered back to users (e.g. review discussion comments) is escaped, not executed, including nested payloads.	Automated — `tests/integration/security-adversarial.test.ts` ("renders a stored-XSS comment payload as inert escaped text"). Manual — spot-check any other user-supplied free-text field rendered in `apps/web` beyond the discussion tab.
W8	`accepted_risk` finding dispositions require a substantive reason (not a token string) before being accepted.	Automated — `tests/integration/security-adversarial.test.ts` ("requires reason for accepted_risk disposition"), `createFindingDecisionRequestSchema` in `packages/contracts`.
W9	Source-code upload requires an explicit source-upload mode; a manifest cannot smuggle source-classified content under a metadata-only upload mode.	Automated — `tests/integration/security-adversarial.test.ts` ("rejects source upload without explicit source mode"), `uploadManifestSchema` in `packages/contracts`.

Artifact storage / signing

#	Item	Status
S1	Artifact keys containing `..` traversal segments or absolute paths are rejected when resolved against the artifact storage root.	Automated — `tests/integration/security-adversarial.test.ts` ("rejects path traversal in artifact key"), `resolveLocalArtifactPath` in `packages/cloud-core/src/index.ts`.
S2	A single-byte tamper to a persisted evidence ledger is detected on verification.	Automated — `tests/integration/security-adversarial.test.ts` ("detects single-byte tampering in evidence ledger"), `buildEvidenceLedger`/`verifyEvidenceLedger` (SHA-256 over canonical JSON in `packages/contracts/src/evidence-ledger.ts`).
S3	A revoked signing key in the trust store no longer verifies new signatures, without breaking verification of past releases signed by other still-valid keys.	Manual — exercise `verifyManifestSignatureAgainstTrustStore` end-to-end via `release verify --trust-store` with a trust store containing one revoked and one active key; no automated test in `security-adversarial.test.ts` currently drives this through the CLI.
S4	The trust-store JSON file's own integrity/distribution is protected at the point it reaches a consumer's machine (file permissions at minimum; a signed trust-store bundle is explicitly not yet designed, per `src/release/signing.ts` and Assurance Case claim 3).	Manual, known gap — confirm this is documented as accepted risk, not silently assumed safe. Do not treat as a "finding" to fix during the pen test itself; it is already tracked.
S5	An incomplete required-check state cannot produce a "ready" (green) release-gating result.	Automated — `tests/integration/security-adversarial.test.ts` ("prevents incomplete check from turning green"), `isWdrrReady` in `packages/cloud-core`.

Infra / deployment

#	Item	Status
I1	The npm publish workflow (`.github/workflows/publish-npm.yml`) uses OIDC Trusted Publishing and fails closed if a long-lived `NPM_TOKEN` or basic-auth `.npmrc` entry is injected.	Manual — Release Integrity documents this as passed for the documented flow; confirm no fallback token secret remains configured in the live GitHub environment/repo secrets.
I2	Container images (`apps/web/Dockerfile`, `apps/container/Dockerfile`) do not run as root and do not bake in secrets or long-lived credentials.	Manual — image build/runtime audit, not something the repo's tests assert.
I3	GitHub Actions workflow permissions are minimum-necessary at the job level, not only declared broadly at the workflow level.	Manual — spot-check `.github/workflows/*.yml` job-level `permissions:` blocks, especially any workflow that runs on `pull_request_target` or handles fork PRs, per the "Malicious workflow change" threat in Threat model.
I4	The custom Next.js worker build (`scripts/build-control-plane-worker.mjs` output) does not expose any endpoint or capability beyond what the documented API routes provide.	Manual — review generated `.next/worker.mjs` behavior against a running deployment.

Pass / fail criteria and exit conditions

Automated items are exit conditions of the existing test suite, not of the pen test: a pen tester should confirm they still pass (corepack pnpm vitest run tests/integration/security-adversarial.test.ts) rather than re-deriving them by hand, and should treat a regression in one of them as a release blocker on its own.
Manual items are the actual deliverable of a GA pen test. The test passes when every Manual item above has an explicit recorded result (pass, fail, or accepted-risk-with-justification) — "not tested" is not a passing state.
Any Manual item that fails is a GA release blocker unless the maintainer explicitly records it as an accepted risk with a reason, consistent with the accepted_risk disposition discipline this codebase already enforces for findings (W8 above).
Known, already-tracked gaps (S4; cross-OS reproducibility in Release Integrity; private vulnerability reporting/push-protection settings confirmation in Assurance Case claim 4) are not new findings if rediscovered — cross-reference before filing a duplicate.
On completion, record the tester, date, scope covered, and per-item results as release evidence (see Evidence Bundles) rather than only as an external report, so it participates in the same audit trail as other release verification.