Verified on-prem releases and updates
This guide prepares verified artifacts for both the
single-server installer and the
Kubernetes deployment. The installer consumes
verified-images.json; Helm additionally uses the verified chart archive.
The Helm commands below apply to Kubernetes. Return to the single-server guide
after verification if you are using the installer.
For sites with internet access, use an external OCI registry and immutable image
digests. The On-prem Release workflow prepares application and CPU Whisper GHCR
images for Linux amd64 and arm64, an amd64 NVIDIA vLLM image, application and
inference chart archives, and release.json. It runs only for the current source
repository main commit after successful push CI. It can also be dispatched for
that commit, for example to rebuild after distribution package fixes.
Distribution uses the public packages
ghcr.io/videoq-jp/videoq-api, videoq-worker and
videoq-web and videoq-transcription-gateway.
Docker and Kubernetes can pull them without a GHCR login
or imagePullSecrets. Site-owned private registries can still use
imagePullSecrets when required. The images include the repository's existing
LICENSE; making distribution public does not change those terms.
Package administrators initially set these packages to Public and grant
videoq-jp/videoq the Write role under Package settings → Manage
Actions access. The workflow checks public visibility before and after every
push and before assembling indexes. Missing, private or internal packages stop
publication. Staging verifies the signed release and pulls each reviewed digest
with an empty Docker credential configuration, proving anonymous image access.
Publishing still requires repository-scoped write credentials. GitHub does not
allow a public package to become private again.
What the release verifies
Application builds have read-only repository access. They scan each image for HIGH/CRITICAL vulnerabilities, including unfixed findings, and export the image and a CycloneDX inventory. Separate fresh runners publish the exact build artifacts without running application containers or installing app dependencies with publication credentials. GitHub OIDC signs provenance and per-architecture SBOM attestations. A signed multi-architecture index binds the two architectures; the signed schema-v5 release manifest binds all four image index digests, their supported architectures and both charts' bytes. The inference chart separately pins the unmodified Linux amd64 upstream vLLM digest for chat, embeddings and speech. VideoQ does not build, publish or sign a replacement engine image.
The mandatory upstream inspection retains unsuppressed Trivy scans of the image
and the separate installed audio wheels, verifies actual image files and runtime
imports, and evaluates findings against the constrained chart workloads. Only
exact, current, evidence-backed applicability assessments are accepted; new or
unassessed HIGH/CRITICAL/UNKNOWN findings block publication. See the
upstream assessment.
The upstream-vllm-inspection-* artifacts retain raw reports, inventories, runtime
checks, dependency locks and assessments for 90 days.
The production API and worker use digest-pinned Alpine images. In addition to the interpreter, the canonical runtime updater refreshes these pins and the verified npm/pip/uv build tools. A distribution change must pass the image tests: native Python wheels, PostgreSQL, FFmpeg, auth and the Kubernetes lifecycle. OS package versions are captured in the SBOM; a rebuild is a new artifact even at the same source commit. Release runs and attempts use distinct registry tags and artifact names. On a failed release, rerun all jobs so every artifact belongs to the same attempt; partial retries fail closed when their inputs are from an older attempt. Installation always uses digests.
On-prem Validation runs on selected changes and daily against current main,
building and scanning images. Lifecycle rehearsals, including the disposable
Kubernetes environment, run only when deployment, migration, storage or recovery
mechanisms change and during release validation. Daily Operations checks do not
run lifecycle rehearsals.
A daily failure requires triage; it does not approve or deploy a replacement.
Every release also verifies the signed artifacts and runs the Kubernetes rehearsal
on a fresh hosted runner. A successful release run therefore includes this staging
check and a separate real CPU model acceptance job. Production promotion remains operator-approved.
Existing dependency and runtime PR automation remains the review path for source
changes. No blanket ignore-unfixed or CVE ignore file is used.
Download and verify a specific release
Public Releases include the manifest, both Helm charts and Sigstore/SLSA verification bundles. No GitHub account, token or CLI is required. Use Python 3.10+ and Helm 4.3.1. Run the verifier from a reviewed checkout, not from the downloaded release. Review the full source SHA and run/attempt in the public release tag; do not take the trusted SHA only from the manifest being checked.
SOURCE_SHA=REPLACE_WITH_REVIEWED_40_CHARACTER_COMMIT
RUN_ID=REPLACE_WITH_PUBLIC_RELEASE_RUN
RUN_ATTEMPT=REPLACE_WITH_PUBLIC_RELEASE_ATTEMPT
bash scripts/onprem/install-verify-tools.sh
python3 scripts/onprem/public_release.py --source-sha "$SOURCE_SHA" \
--run-id "$RUN_ID" --run-attempt "$RUN_ATTEMPT" --directory /secure/videoq-release
The public downloader verifies that the run succeeded on main and matches the source and attempt. Sigstore verifies signatures, transparency-log evidence, repository/owner, workflow, main ref, source commit, GitHub-hosted runner, run/attempt and the subject names/digests. It validates the manifest, both charts and all five image indexes before writing either Helm values file. Trust roots come independently from Sigstore production's TUF configuration. Missing bundles, changed bytes or a different signer fail closed and preserve existing values.
The installer selects the public release automatically. Releases are published
only after the signed-artifact and real-CPU-model rehearsal jobs succeed; public
assets are not overwritten. CI and maintainers can still use the legacy
onprem-release.py/GitHub CLI path to inspect Actions-only artifacts or SBOMs,
but end-user installation does not use it. Preserve promoted digests and bundles
in your archive; GitHub Actions transfer artifacts expire after 90 days.
Merge the generated image values last so site settings cannot accidentally replace the reviewed digests:
helm template videoq /secure/videoq-release/videoq-0.1.0-sha.$SOURCE_SHA.tgz \
-n videoq -f /secure/site.yaml -f /secure/videoq-release/verified-images.json
helm upgrade --install videoq /secure/videoq-release/videoq-0.1.0-sha.$SOURCE_SHA.tgz \
-n videoq -f /secure/site.yaml -f /secure/videoq-release/verified-images.json \
--wait --timeout 15m
helm test videoq -n videoq --timeout 2m
For fully local AI, install the verified inference archive with
verified-inference.json last, and the application with the matching local AI
overlay before verified-images.json. See the
local inference guide. Hosted OpenRouter remains the application
default; downloading the inference chart alone does not switch AI endpoints.
The chart enforces digests but does not itself enforce signatures. The verifier is an operator-side gate; a cluster administrator can bypass it. If that must be prevented, add a site-owned admission policy for the same signer identity.
Staging and promotion
Start with a separate staging namespace and backing services. Record the source commit, release run ID, chart checksum, image digests, site-values revision and migration version in the site's change record. Verify the release, rehearse an upgrade with a representative backup, and check upload, transcription, indexing, cited Q&A, SSE and timestamp playback. Production promotion requires an operator approval of this concrete set of artifacts and the migration/restore plan.
A GitOps repository can store verified-images.json in a reviewable PR after
verification. Let the staging controller reconcile approved values, then promote
the same digests to production after approval. Do not auto-promote a mutable tag
or regenerate a production Secret from release metadata. No site deployment or
GitOps credentials are configured in this repository while the target is unset.
Reproduce the local lifecycle rehearsal
Build the three application images as videoq-api:onprem-ci,
videoq-worker:onprem-ci and videoq-web:onprem-ci. Then, with Docker, kind,
kubectl, Helm, Node and Python available:
npm ci --ignore-scripts --workspaces=false
kind create cluster --name videoq-onprem-lab --kubeconfig /tmp/videoq-lab-kubeconfig
HELM_BIN=helm python3 .github/scripts/onprem-lab.py \
/tmp/videoq-lab-kubeconfig /tmp/videoq-lab-results
kind delete cluster --name videoq-onprem-lab --kubeconfig /tmp/videoq-lab-kubeconfig
The script restricts kubectl to that dedicated context. It creates disposable
PostgreSQL/pgvector, Garage, ElasticMQ and a deterministic inference HTTP fixture
outside the application chart. It exercises real uploads, FFmpeg audio extraction,
queue delivery, transcript storage, vector indexing, cited Q&A and SSE; then S3
object backup/delete/restore with checksum comparison, a database dump restored
to a separate database, application reconnection, a two-replica upgrade, Helm
tests and a one-replica rollback retaining the restored data. It writes
result.json only after all checks pass. Results and database dumps are local
artifacts; do not commit credential-bearing database dumps.
This rehearsal tests application and protocol integration. Real inference model quality, GPU sizing, TLS/ingress, storage and queue persistence across host loss, and the site's backup RPO/RTO require validation in the eventual target site.
Real CPU model acceptance
The release's Real CPU models with public egress blocked job separately verifies
the signed bundle and pulls the app, CPU gateway and pinned llama.cpp images.
It provisions the exact chat, embedding and Whisper model revisions/checksums
from the verified inference chart before starting a dedicated kind cluster.
That cluster attaches only to a new dedicated Docker network. After bootstrap
and image loading, its node's IPv4/IPv6 OUTPUT and FORWARD rules reject public
destinations and new Docker-host connections; CoreDNS public forwarding is removed.
Kind needs a default gateway during bootstrap, so Docker --internal cannot be
used directly. No VideoQ processing begins until the firewall is installed.
A worker-Pod probe must reach private inference while
public IP connections and external DNS resolution fail.
The disposable PostgreSQL, S3 and queue fixtures are pulled by their reviewed
upstream digests, then imported into kind under explicit local aliases with
imagePullPolicy: Never. This avoids re-resolving a remote multiarch index after
public egress is blocked. Results retain their original references and native
Docker image IDs. These aliases are only for the disposable lab; operator installation
still uses the verified release digests. Failure artifacts retain Pod/node
status and warning events, excluding Pod specs and Secrets.
For containerd-backed Docker stores the importer uses native-platform export
(Docker API 1.48+); classic stores must already contain the daemon's native
architecture. It checks each alias through the node's CRI cache before blocking
public traffic, and records the actual imported platform and CRI image ID.
The job uploads the checksum-pinned public Japanese FLEURS clip through the app,
uses real whisper.cpp transcription and llama.cpp embeddings/chat, checks cited
answers and SSE, and uses Chromium to decode and play the uploaded video at an
actual citation timestamp. The transcript must have valid source-relative times
and normalized character error rate below 0.25 against the public reference.
It repeats Q&A after database restore, upgrade and
rollback. onprem-cpu-acceptance-* artifacts retain the public transcript,
answers, processing time, playback result, model checksums and image digests for
90 days. They exclude credentials, model files and database dumps.
This is one short public fixture, not site-specific quality or capacity approval. The node firewall does not test the site's CNI enforcement. With a reviewed checkout, the tools listed in the script, and Chromium installed, reproduce it:
npm ci --ignore-scripts --workspace @videoq/api --workspace @videoq/web
node node_modules/playwright/cli.js install --with-deps chromium
HELM_BIN=helm python3 .github/scripts/onprem-cpu-acceptance.py \
"$SOURCE_SHA" /secure/videoq-release /tmp/videoq-real-cpu-results
The script refuses to replace an existing videoq-onprem-lab cluster. Initial
downloads need internet access. Allow at least 16 GiB RAM for this disposable
test with backing services; production sizing still needs representative videos.
Package names and historical releases
The namespace migration is complete. Current releases publish the four public
videoq-jp/videoq-{api,web,worker,transcription-gateway} packages through
On-prem Release. Their package settings must retain repository Actions Write
access and public visibility. Bootstrap tags are not deployment releases.
The migration-only Package Namespace Bootstrap and Retired Package Cleanup workflows have been retired. Keep historical packages available for rollback and verification; removing a workflow does not delete images, releases or attestations. The verifier still validates historical schemas against their original repository, workflow and exact source SHA, including all eight signed subjects. Mixed image namespaces remain invalid. Use the normal signed release workflow for new builds.
Distribution retention
Automatic releases compare the selected main source with the newest published
on-prem release backed by a successful main release run. Documentation, adopted
deployment pins and retention housekeeping alone keep the existing release.
Unreleased runtime or distribution changes still publish even if the newest
commit only edits documentation. Unknown, divergent or large diffs publish
conservatively. A manual workflow_dispatch requests a full release after the
same current-main CI validation.
After a successful release, On-prem Distribution Retention keeps the latest two
published on-prem releases whose source/run/attempt match a successful main
release workflow. Other release types are preserved. It protects their image
indexes, platform manifests, signatures/SBOM referrers, operational pins in main
and open same-repository PRs, and seven days of build grace. Only versions outside
that complete reference graph may be deleted. The retired videoq-vllm package
remains while any retained release or protected source/deployment pin refers to
it. Once all those references disappear, its entire package is deleted, including
its last tagged version and recent builds from the retired publisher. The build
grace still applies to all four active application packages. Package retirement
checks the complete version inventory again before deletion. Later runs accept
an exact HTTP 404 for the retired package only after verifying all four active
package identities. Protected references to that missing package still stop
cleanup. Other API errors stop cleanup. It does not require organization-wide
package listing access, and it requests registry tokens only for packages still
present.
The workflow first saves a deletion plan, then re-reads the live inventory and
reference graph before applying it. Missing metadata, changed references or an
expired plan stop deletion. Exact successful deletions are journaled and kept as
workflow artifacts for 90 days. A daily scheduled run clears expired build grace
even when no new release is published. Manual runs default to plan-only; set apply=true
only when actual cleanup is intended. The installer never runs registry cleanup
on a customer's host.