Forge
A self-hosted NuGet package registry, npm registry, Docker / OCI registry and raw file repository written in Rust. It speaks the NuGet V3 protocol, the npm registry protocol (npm, pnpm, Yarn, Bun) and the OCI Distribution Spec v1.1 (Docker, Podman, containerd, crane, Helm, cosign), and supports the three repository types you expect from an enterprise registry (more package formats come as format plugins, see below):
- hosted — push and restore your own
.nupkgpackages. - proxy — cache packages on demand from an upstream NuGet V3 feed —
api.nuget.orgor a private/custom feed (Azure Artifacts, GitHub Packages, Artifactory, Nexus, …) with Basic/PAT credentials — with an optional publication delay that holds back new upstream releases. - group — fan a single feed URL out across several member repos by priority; tolerates a failing member and, by default, keeps public feeds from shadowing internal package ids (dependency-confusion guard).
The same three types exist for npm packages: publish with npm publish,
log in with npm login (approved in the browser), proxies of
registry.npmjs.org or any registry, and groups that keep your scopes
internal and pass npm audit on — see docs/npm.md;
for container images and other OCI artifacts,
Helm charts included (helm push, helm install oci://…): hosted image
repositories, pull-through proxies of Docker Hub, GHCR, Quay or any OCI
registry, and groups — see docs/docker.md — and for plain files
under arbitrary paths (binaries, installers, fixtures), like Nexus’ raw
repositories, with ranges, checksum sidecars and proxies of any HTTP(S)
download URL such as GitHub release assets — see
docs/raw.md.
Storage is content-addressed (CAS) on top of any S3-compatible object store; package metadata lives in PostgreSQL. Access control is fine-grained RBAC (users, nested groups, roles, package/content selectors). Authentication is organized into ordered, web-managed auth realms — local passwords, OIDC / Microsoft Entra browser login, and credential-forwarding against an existing registry — with an identifier-first login flow. CI jobs can publish without stored API keys through Forge’s direct-token OIDC trusted-publishing flow for GitHub Actions, GitLab CI, and compatible workload issuers; its bearer token, cache, and audit limitations are documented below. A retention engine reaps cold versions and garbage-collects dereferenced blobs. Format plugins (preview), crates built into the server, add further package formats that reuse all of the above (docs/format-plugins.md). The admin + browse UI is a TanStack Router SPA served by the same Rust process on a single origin — there is no Node server in production.
Architecture
A Cargo workspace of small crates plus the web SPA:
Crate (crates/…) | Package | Responsibility |
|---|---|---|
db | forge-db | PostgreSQL data access (sqlx), row models, DAOs. |
storage | forge-storage | CAS over S3, blob outbox / GC reconciler. |
cache | forge-cache | In-process L1 + optional Redis L2 with cross-instance invalidation (no Postgres advisory locks). |
core | forge-core | Domain logic: repo dispatch, auth chain, authz/RBAC, sessions, OIDC, retention. |
plugin-api | forge-plugin-api | The format-plugin contract: the FormatPlugin trait and its types. |
plugin | forge-plugin | The host side of format plugins: an installed plugin and its adapter to the repository machinery. |
formats/<id> | forge-format-* | Format plugins, built into the server: docker (the distribution-spec /v2 API, manifests, upstream registries), nuget, npm, raw. |
server | forge-server | Axum HTTP server: the format plugins’ feeds under /repository/{repo} (except Docker at /v2), the file store of files formats, the image store and registry tokens of images formats, /admin + /auth REST, background workers, static SPA serving. |
e2e | forge-e2e | End-to-end suite (testcontainers + mock upstream/IdP). |
xtask | — | The task runner (cargo xtask …); replaces Make/Just. |
web/ | — | TanStack Router SPA (pnpm + Vite), built into web/dist/. |
Every format is a plugin in crates/formats/. A format is a crate in
crates/formats/<id>/ implementing FormatPlugin, linked into the server.
Docker / OCI is one too, an images format: the plugin speaks the protocol,
and the server’s image store (crates/server/src/images) keeps the
images.
Single-origin serving. forge-server serves every non-Docker format plugin
at /repository/{repo}, the Docker / OCI registry at /v2,
the admin/browse APIs (/admin, /browse, /auth) and then serves the built SPA
from web/dist/ via a ServeDir SPA fallback. The browser talks to the API on
the same origin with relative URLs; no routing proxy is required (production
still puts a TLS-terminating proxy in front — see
docs/deployment.md).
Quickstart
Prerequisites: Docker, a Rust toolchain (pinned in rust-toolchain.toml),
and Node 26 + pnpm 11 (npm install -g corepack && corepack enable).
Optionally the .NET SDK to drive the feed with a real client.
cargo xtask dev
That one command uses your local Postgres on :5432 (creating the forge
role + database if missing), brings up an S3 gateway (:8422) via
docker-compose.yml, runs migrations, and starts the Rust API on :8421 plus
Vite HMR on :8420. A bootstrap admin admin@example.com / changeme is created on
first run (dev only: the bundled forge.toml sets
server.allow_insecure_defaults = true; any other config refuses to start with
the changeme password or minioadmin S3 keys). (Requires a running local PostgreSQL with permission to create the
role/db; use --db docker if you don’t have one — see below.)
- UI: http://localhost:8420
- NuGet feed:
http://localhost:8421/repository/{repo}/index.json - npm registry:
http://localhost:8421/repository/{repo}/
Add demo data (anonymous-readable nuget repo + a pushed sample package) in a
second terminal:
cargo xtask seed
# feed: http://localhost:8421/repository/nuget/index.json
Choosing where Postgres comes from
cargo xtask dev # default: host Postgres on :5432 (forge role+db auto-created)
cargo xtask dev --db docker # self-contained docker-compose Postgres on :5433
cargo xtask dev --db none # assume the DB is already up; don't start one
Run via the Docker image
Every release publishes the signed image to GitHub Packages as
ghcr.io/forgepkg/forge,
for linux/amd64 and linux/arm64 (from 2.0.0 on; up to 1.1.0
ghcr.io/schmitch/forge, amd64 only; see
docs/deployment.md). To build it from a
checkout instead, the root Dockerfile packages the SPA and the Rust
server into one image (no Node in prod):
cargo xtask docker-build # docker build -t forge:latest .
cargo xtask docker-run # run the image against the compose db + s3 on :8421
Or directly, against your own infra (configure with FORGE_* env vars, e.g.
FORGE_DATABASE__URL, FORGE_S3__ENDPOINT/__BUCKET/__ACCESS_KEY/__SECRET_KEY):
docker run --rm -p 8421:8421 \
-e FORGE_DATABASE__URL=postgres://forge:forge@db:5432/forge \
-e FORGE_S3__ENDPOINT=http://s3:8422 -e FORGE_S3__BUCKET=forge \
-e FORGE_S3__ACCESS_KEY=… -e FORGE_S3__SECRET_KEY=… \
-e FORGE_AUTH__BOOTSTRAP_ADMIN__USERNAME=admin@example.com \
-e FORGE_AUTH__BOOTSTRAP_ADMIN__PASSWORD=… \
forge:latest
The image refuses to start without S3 credentials and with placeholder
credentials (minioadmin S3 keys, a changeme bootstrap password). Instead
of S3 keys it can use short-lived credentials from the platform’s workload
identity (IRSA, EKS Pod Identity, STS web identity; see
docs/deployment.md). It
bakes no bootstrap admin: FORGE_AUTH__BOOTSTRAP_ADMIN__PASSWORD creates one on the first start,
named by FORGE_AUTH__BOOTSTRAP_ADMIN__USERNAME (a UPN, default admin@example.com), and
once it is removed the image simply starts without one. A new install has
no anonymous access and accepts no passwords over HTTP Basic (clients use API
tokens); an administrator changes both in the web UI under Admin →
Settings (see
docs/deployment.md).
Point a dotnet client at it
dotnet nuget add source http://localhost:8421/repository/nuget/index.json -n forge
dotnet nuget push -s forge -k <TOKEN> your.1.0.0.nupkg
dotnet add package <PackageId> # in a project
For a non-anonymous repo, pass --username admin@example.com --password <TOKEN> --store-password-in-clear-text to the source (the password is a token, not
the user password).
Push and pull container images
Create a docker repository ("format": "docker") under Admin →
Repositories, then log in with an API token as the password:
docker login localhost:8421 -u admin@example.com # password: an API token
docker tag alpine:3.20 localhost:8421/docker-hosted/team/alpine:3.20
docker push localhost:8421/docker-hosted/team/alpine:3.20
docker pull localhost:8421/docker-hosted/team/alpine:3.20
Upload and download files
Create a raw repository ("format": "raw"), then upload with an API token as
the password and download with its checksum:
curl -u admin@example.com:<TOKEN> --upload-file tool.tgz http://localhost:8421/repository/binaries/tools/1.0/tool.tgz
curl -u admin@example.com:<TOKEN> -O http://localhost:8421/repository/binaries/tools/1.0/tool.tgz
curl -u admin@example.com:<TOKEN> http://localhost:8421/repository/binaries/tools/1.0/tool.tgz.sha256 | sha256sum -c
Documentation
The guides below are also a site, with navigation and search: https://forgepkg.github.io/forge/.
- Production deployment — required settings, TLS
and reverse proxy, secret encryption, health probes, backup/restore and
upgrades. Kubernetes: a Helm chart published with every release
(deploy/helm/forge, on GitHub Packages
as
oci://ghcr.io/forgepkg/charts/forge), or kustomize manifests (deploy/kubernetes). - CHANGELOG.md — release notes, upgrade notes and known limitations; SECURITY.md — how to report a vulnerability.
- DEVELOPMENT.md — full local-dev guide: every
cargo xtaskcommand, configuration knobs, OIDC/Entra setup, the web UI dev server, tests, and Docker packaging details. - Docker / OCI registry — image repositories (hosted,
proxy, group), naming and host-based routing,
docker login, retention and garbage collection. - npm registries — npm packages (hosted, proxy, group)
for npm, pnpm, Yarn and Bun:
.npmrc, logins, publishing, dist-tags, deprecations, unpublishing, proxies ofregistry.npmjs.org, internal scopes, audits and retention. - Raw file repositories — files under arbitrary paths (hosted, proxy, group): uploads with curl, ranges and checksum sidecars, listings, the deployment policy, proxies of release downloads (the pdfium example) and retention.
- Retention size budgets — keep a repository under a logical size: the budget in the retention policy, how size is counted, and what goes first.
- Trusted publishing — configure short-lived
workload identity for GitHub Actions, GitLab CI, and compatible OIDC issuers,
and exchange a job’s OIDC token for a short-lived API key (
POST /api/v2/token). - GitLab CI with workload identity — step by step: every job of a GitLab project restores from Forge’s NuGet feeds, and only the release job, on a protected tag or environment, pushes the packages a trusted publisher names. No secret in GitLab.
- CI with service accounts — for pipelines that cannot use workload identity: a read token and a release token of service accounts in GitLab’s CI/CD variables, and how both methods run side by side.
- Format plugins — further package formats, built into the server: design, operation, and writing one.
- Importing from Nexus — copy a Nexus hosted repository (NuGet, Docker) into Forge, with its publish dates: dry run, conflicts, resuming, and the cutover.
Run cargo xtask help for the complete command list.
License
Forge is licensed under the MIT License (SPDX MIT).
Contributions are accepted under the Contributor License Agreement, which grants the forgepkg organization the rights to distribute them under the MIT License or other terms. See CONTRIBUTING.md for how to sign it.
Production deployment
This is the operator guide for running the Forge image in production. For local development see DEVELOPMENT.md; for CI publishing without stored API keys see trusted publishing.
What you need
| Dependency | Notes |
|---|---|
| PostgreSQL 18+ | The authority for all metadata, sessions, RBAC and coordination. Migrations run automatically at startup. public grants CREATE to the database’s owner only, so the role in FORGE_DATABASE__URL must own the database (CREATE DATABASE forge OWNER forge) or hold CREATE on its schema. |
| S3-compatible object store | Package blobs (content-addressed). AWS S3, MinIO, Ceph RGW, versitygw, … A missing bucket is created in the configured region; see S3 permissions. Static keys or short-lived credentials (IRSA, EKS Pod Identity, STS web identity). |
| TLS termination | A reverse proxy or load balancer in front of Forge (nginx, Traefik, an ingress, an ALB). Forge itself speaks plain HTTP. |
| Valkey or Redis 6+ (optional) | Shared L2 cache and cross-instance invalidation. Recommended as soon as you run more than one instance. TLS via rediss://. The Helm chart (valkey.enabled) and the kustomize component valkey can run a Valkey in the cluster. |
Forge is a single stateless process. The API, the NuGet feed and the web UI
share one port (8421), so one upstream on the reverse proxy is enough.
On Kubernetes, install the Helm chart
(oci://ghcr.io/<owner>/charts/forge, see
deploy/helm/forge) or start from the
kustomize manifests in deploy/kubernetes.
Both set the probes, the shutdown timing and the security context described
below, and pin the image to the release they belong to.
S3 permissions
Forge’s S3 credentials need these actions on its bucket (an AWS IAM policy
for the bucket forge; MinIO and Ceph RGW use the same action names):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListBucket",
"Resource": "arn:aws:s3:::forge"
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
"Resource": "arn:aws:s3:::forge/*"
}
]
}
s3:ListBucketis what the bucket check (HeadBucket, at startup and in/readyz) needs, and without it S3 answers a lookup of a missing key with 403 instead of 404.s3:PutObjectcovers multipart uploads too (large pushes and proxied downloads);s3:AbortMultipartUploaddiscards an interrupted one.s3:DeleteObjectis not optional. When a package is deleted or retention evicts a version, garbage collection deletes the blob through the outbox. With append-only credentials that delete fails, the blob’s reservation stays fenced, and every later ingest of the same bytes retries the delete itself and fails: a package deleted and pushed again with identical bytes, or a proxied version evicted by retention, can then never be stored again.s3:CreateBucket(on the bucket, next tos3:ListBucket) only if Forge should create the bucket. Forge creates a missing bucket inFORGE_S3__REGION(outsideus-east-1it sends the region as the location constraint), and refuses to start when the name belongs to another account (BucketAlreadyExists). On AWS, create the bucket beforehand (with the lifecycle rule below); then Forge needs nos3:CreateBucket.
These are the S3 calls behind the policy: HeadBucket, CreateBucket (only
when the bucket is missing), HeadObject, GetObject, PutObject,
DeleteObject, CreateMultipartUpload, UploadPart,
CompleteMultipartUpload and AbortMultipartUpload. The policy is the same
for static keys and for the role behind short-lived credentials.
Short-lived S3 credentials
Instead of a long-lived access key in a Secret, Forge can get temporary S3
credentials from the platform’s workload identity and refresh them itself.
Then there is no S3 secret to store, rotate or leak. s3.credentials
(FORGE_S3__CREDENTIALS) chooses the source:
s3.credentials | Credentials from |
|---|---|
static (default) | FORGE_S3__ACCESS_KEY / FORGE_S3__SECRET_KEY. |
default_chain | The AWS SDK default credentials chain, in this order: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN, the shared profile (AWS_PROFILE), web identity (AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE, which IRSA injects), the container credentials endpoint (EKS Pod Identity, ECS task roles), and EC2 instance metadata (instance profiles). |
web_identity | A token file exchanged for temporary keys with STS AssumeRoleWithWebIdentity, against AWS STS or the STS of MinIO or Ceph RGW ([s3.web_identity], below). |
With default_chain and web_identity, leave FORGE_S3__ACCESS_KEY and
FORGE_S3__SECRET_KEY unset. Forge refuses to start with keys it would not
use, so a long-lived key does not stay behind in a Secret after the switch.
The insecure-default check (minioadmin) applies only to static. Endpoint,
region, path-style and checksum settings work the same with every source.
| Variable | [s3.web_identity] key | Value |
|---|---|---|
FORGE_S3__WEB_IDENTITY__TOKEN_FILE | token_file | Required. The token, e.g. a projected service-account token. |
FORGE_S3__WEB_IDENTITY__ROLE_ARN | role_arn | The role to assume. AWS and MinIO’s role policies need one; MinIO’s claim-based policies take none. |
FORGE_S3__WEB_IDENTITY__STS_ENDPOINT | sts_endpoint | Unset: AWS STS in s3.region. For MinIO and Ceph RGW, their URL (usually the S3 endpoint). |
FORGE_S3__WEB_IDENTITY__SESSION_NAME | session_name | RoleSessionName, recorded by CloudTrail. Default forge. |
FORGE_S3__WEB_IDENTITY__DURATION_SECONDS | duration_seconds | 900 to 43200. Unset leaves it to STS (one hour on AWS). |
How Forge handles them:
- Refresh. The S3 client caches the credentials and fetches new ones 30
to 60 seconds before they expire, so no request is signed with expired
keys. Each fetch reads
token_fileagain, which picks up a token the kubelet rotated. Every request is signed with the credentials current when it is sent, so a multipart upload that outlives one set of credentials completes. - Failures. A failed fetch fails the S3 call that needed it with an error
that names the source, the role, the token file and the STS endpoint, and
the STS error code (
InvalidIdentityToken,AccessDenied, …). It is also logged as a warning. Neither ever contains the token or the derived keys, and neither does the configuration’s debug output. Like an S3 failure, it makes the startup bucket check abort and/readyzanswer 503. With the local blob cache, an STS that cannot be reached counts as S3 being down (start degraded, serve the cache), while a refused token still fails startup. - Network. Forge must reach the STS (
sts.<region>.amazonaws.com, or the store’s endpoint), and for EKS Pod Identity the node’s agent at169.254.170.23:80. Allow them in a restrictive egress policy.
IRSA (EKS). Create an IAM role with the policy above
and a trust policy for the cluster’s OIDC provider and Forge’s service
account (system:serviceaccount:<namespace>:<service account>). Annotate the
service account with eks.amazonaws.com/role-arn: <role ARN> and set
FORGE_S3__CREDENTIALS=default_chain. The EKS webhook injects
AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and the token volume.
EKS Pod Identity. Create a Pod Identity association between the role
(with the policy above, trusted by pods.eks.amazonaws.com) and Forge’s
service account, and set FORGE_S3__CREDENTIALS=default_chain. The webhook
injects AWS_CONTAINER_CREDENTIALS_FULL_URI and
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE.
Both work with automountServiceAccountToken: false, which the Helm chart
and the kustomize base set. The webhook (amazon-eks-pod-identity-webhook)
selects pods by the service account’s role annotation or the Pod Identity
association. It does not read automountServiceAccountToken, and it adds
its own projected token volume, separate from the Kubernetes API token that
setting controls.
ECS, EC2. default_chain picks up the task role or the instance profile.
MinIO. Its STS accepts tokens from an OpenID Connect issuer. To trust
the Kubernetes service-account issuer, set
MINIO_IDENTITY_OPENID_CONFIG_URL to the cluster’s discovery document
(<issuer>/.well-known/openid-configuration), and
MINIO_IDENTITY_OPENID_CLIENT_ID to the audience Forge’s token carries
(e.g. forge-s3). A projected service-account token cannot carry a policy
claim, so give the issuer a role policy (MINIO_IDENTITY_OPENID_ROLE_POLICY=forge)
and set role_arn to the role ARN MinIO logs at startup
(IAM Roles: arn:minio:iam:::role/…). Any pod in the
cluster can request a token for that audience, so restrict the policy to
Forge’s service account with a jwt:sub condition (namespace forge,
service account forge, bucket forge):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::forge"],
"Condition": { "StringEquals": { "jwt:sub": "system:serviceaccount:forge:forge" } }
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
"Resource": ["arn:aws:s3:::forge/*"],
"Condition": { "StringEquals": { "jwt:sub": "system:serviceaccount:forge:forge" } }
}
]
}
With an issuer whose tokens name the policy in a claim
(MINIO_IDENTITY_OPENID_CLAIM_NAME, MinIO’s claim-based policies), leave
role_arn unset.
Point Forge at it:
FORGE_S3__CREDENTIALS=web_identity
FORGE_S3__WEB_IDENTITY__TOKEN_FILE=/var/run/secrets/forge/s3/token
FORGE_S3__WEB_IDENTITY__STS_ENDPOINT=https://minio.example.com:9000
FORGE_S3__WEB_IDENTITY__ROLE_ARN=arn:minio:iam:::role/<id from the MinIO log>
Forge then needs no s3:CreateBucket: create the bucket with MinIO’s root
user (mc mb).
On Kubernetes, the token is a projected service-account token with that
audience. The Helm chart mounts it with s3.credentials: web_identity, and
the kustomize component s3-workload-identity does the same. Ceph RGW works
the same way with its OpenID Connect provider and role_arn.
Not covered. GCS’s S3 interoperability API accepts only HMAC keys, and Azure Blob Storage has no S3 API, so neither works with short-lived credentials here.
Image
docker pull ghcr.io/<owner>/<repo>:1.0.0
The image is published for linux/amd64 and linux/arm64 (from 2.0.0 on;
earlier releases are amd64 only) as one multi-platform image: its tags and
digest name an index of both platforms’ images, from which docker pull and
Kubernetes take the image of the node’s platform. The Helm chart and the
manifests therefore deploy to amd64 and arm64 nodes, mixed ones included,
without a change. Each platform’s image is built natively, not under
emulation.
Release images are signed with keyless cosign and carry an SBOM and a
provenance attestation for each platform (docker buildx imagetools inspect ghcr.io/<owner>/<repo>:1.0.0 lists them). The release signs the index and
each platform’s image, and its notes list their digests. The exact cosign verify command is in each GitHub release’s notes, next to the commit the
release was built from; it passes for each of those digests. It pins the
signer to this repository’s release workflow at that release’s tag, run by
the push of the tag, on that commit:
cosign verify ghcr.io/<owner>/<repo>@<digest> \
--certificate-identity https://github.com/<owner>/<repo>/.github/workflows/release.yml@refs/tags/v1.0.0 \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-github-workflow-sha <commit> \
--certificate-github-workflow-trigger push
The identity alone names only the tag; pinning the commit as well rejects a signature from a release run for a tag that was later moved or re-created on another commit.
Each release also publishes the Helm chart of the same version,
oci://ghcr.io/<owner>/charts/forge, which pulls the image by the digest the
release signed, the index’s, so each node pulls its platform’s image. It is
signed the same way; its cosign verify command is in the release notes too.
:latest always names the newest stable release and :<major>.<minor> the
newest release of that line; prereleases and patches to older lines never move
them. Pin deployments to a version or digest. GET /version on a running
instance reports its version and source revision.
Required configuration
The image ships a secret-free base config (/app/forge.toml, from
docker/forge.docker.toml, loaded through
FORGE_CONFIG). Everything is overridden with FORGE_* environment variables
(__ separates nesting levels). If you point FORGE_CONFIG at your own file,
that file must exist: a missing file fails startup instead of being ignored.
Secret values (FORGE_DATABASE__URL, FORGE_REDIS__URL, the S3 keys, the
bootstrap password, FORGE_OIDC__CLIENT_SECRET, the credential-forwarding
pepper) are read verbatim as strings, so a numeric or bracketed password needs
no quoting, and configuration errors never print their values.
| Variable | Value |
|---|---|
FORGE_SERVER__PUBLIC_BASE_URL | The https URL clients use, e.g. https://nuget.example.com. It is used for every absolute URL in NuGet responses, and it decides whether session cookies are Secure. |
FORGE_DATABASE__URL | postgres://forge:<password>@db:5432/forge |
FORGE_S3__ENDPOINT | e.g. https://s3.eu-central-1.amazonaws.com or http://minio:9000 |
FORGE_S3__REGION | e.g. eu-central-1 |
FORGE_S3__BUCKET | e.g. forge |
FORGE_S3__ACCESS_KEY / FORGE_S3__SECRET_KEY | Object-store credentials, or instead FORGE_S3__CREDENTIALS for short-lived credentials. |
FORGE_S3__FORCE_PATH_STYLE | true for MinIO and most self-hosted stores, false for AWS. |
FORGE_AUTH__BOOTSTRAP_ADMIN__USERNAME / FORGE_AUTH__BOOTSTRAP_ADMIN__PASSWORD | First start only: the initial admin, named by a UPN (default admin@example.com), and its password. Without the password no bootstrap admin is created. |
FORGE_CRYPTO__ROOT__PROVIDER + root key | Required: the root key of the secret encryption, e.g. FORGE_CRYPTO__ROOT__PROVIDER=env with FORGE_ROOT_KEY=<openssl rand -base64 32>. |
FORGE_AUTH__TRUSTED_PROXY_CIDRS | The CIDRs of your reverse proxies, e.g. [10.0.0.0/8]. Without it, every request appears to come from the proxy’s IP, so the login lockout sees one client, and the headers of a trusted-header realm are ignored. Every address in these ranges chooses the client IP Forge sees: a whole pod network trusts every pod in the cluster, which can then get around the login lockout. Keep the ranges as narrow as the proxies allow, and let only the proxies reach Forge (see Reverse proxy: on Kubernetes the Helm chart’s networkPolicy.from, the kustomize network-policy component). |
Startup refuses the known placeholder credentials (a changeme bootstrap
password, minioadmin S3 keys, and a secret still holding a <…> placeholder
of the example env files, such as <initial admin password>) and a
configuration without a root key. It logs a warning when public_base_url
is not https or points at localhost. Treat those warnings as deployment
errors.
Never set FORGE_SERVER__ALLOW_INSECURE_DEFAULTS=true in production. It is
meant only for the local compose stack.
The bootstrap admin
The image bakes no bootstrap admin; it comes only from the environment, and
only while a password is set. Set FORGE_AUTH__BOOTSTRAP_ADMIN__PASSWORD for
the first start, and FORGE_AUTH__BOOTSTRAP_ADMIN__USERNAME to a
UPN of your organization if admin@example.com will not do: the
account is created once. Startup refuses a name that is no UPN. Sign in
to the web UI, invite personal admin accounts (below) or
connect OIDC, then delete the bootstrap account. It is not re-created on later
restarts. Then remove the password from the environment: with no bootstrap
password configured, Forge simply starts without a bootstrap admin.
Forge never lets the last active administrator go: deleting or deactivating
that account, or removing its admin role, is refused with 409 Conflict.
So the replacement admin has to exist first; an OIDC or credential-forwarding
admin counts once they have signed in.
Access levels
What a user may do follows from the roles they hold (Admin → Roles,
/admin/roles). A role’s grants each give one of three ordered levels on a
repository, or on every repository, and each level includes the ones below
it:
| Level | Covers (in Nexus’s terms) | In Forge |
|---|---|---|
read | browse, read | Download, list and search packages, images and files. |
write | browse, read, add, edit | read, plus push and relist: a NuGet push, an npm publish, a container image or Helm chart push, a raw file upload; and npm’s dist-tags and deprecations. |
delete | browse, read, add, edit, delete | write, plus unlist and delete versions (npm unpublish too), delete image tags, delete or replace raw files, and redeploy a version. |
A check passes when a grant covering the repository has the level it needs or
a higher one. A role holds one grant per repository (or every repository) and
content selector: giving it another level there replaces the level
(PUT /admin/roles/{id}/grants with {"repository": "<name>", "level": "write"}, or Add grant on the role’s page). A content selector narrows a
grant to the packages it matches, for every level the grant includes: a
write grant scoped to acme.* reads and publishes acme.* only.
Administration is no level: it is membership in the built-in admin role.
An API token carries its owner’s levels, never admin, unless its scope says
otherwise. "scope": {"level": "read"} caps every grant at that level (a
delete grant reads only), "repositories": [<id>, ...] limits the
repositories, and "admin": true keeps the owner’s admin role, for
automation, when the scope does not limit repositories. A scope can only
narrow what its owner may do.
Built-in roles
Every install has three built-in roles, which cannot be deleted or renamed:
| Role | Grants | For |
|---|---|---|
admin | Everything: every repository at every level, and administering the instance (repositories, users, roles, authentication, settings). It holds no grant; membership is the administration. | Administrators. |
writer | write on every repository, ones created later included. No delete, no administration. | People and CI that publish packages: a NuGet push, an npm publish, a container image push, a Helm chart upload, a raw file upload, of a package or image the repository does not have yet too. |
anonymous | What unauthenticated callers may read (anonymous_read repositories add to it). Its grants reach signed-in users too, as read, under the anonymous mode all. | Anonymous access. |
A user who holds neither admin nor writer is a reader: they read what
their other roles and anonymous access allow. Admin → Users shows each
user’s level (admin, writer or reader), and the user’s page switches it.
A writer cannot unlist or delete package versions, delete image tags, or
delete or replace raw files: deleting stays with administrators and custom
roles that grant delete, so a leaked CI token cannot remove published
versions. Nor can a writer reach the
admin API (/admin/* answers 403; only /admin/me answers, and reports the
repo.write capability). A writer mints their own API tokens on the
Tokens page; a token never carries more than its owner’s rights.
Inviting users
An administrator never sets another person’s password. Admin → Users →
New user (/manage/users/new) takes the username, the email
address the invitation goes to (the username unless entered: a UPN is not
always a mailbox) and the access level (reader, writer or admin). Forge
creates the account without a password and emails its user a link,
<server.public_base_url>/invite#<token>, with which they choose their own
password, at least 12 characters, and then sign in.
- The link works once, until the lifetime set on the
email settings page (72 hours unless changed). The token is in the
URL fragment, which reaches no server or proxy log and no
Referer; Forge stores only its SHA-256 and never logs it. An unknown, used, expired, replaced or revoked link gets one answer, which names no account. Failed attempts are throttled per client network with the login lockout’s budget (login_max_failuresperlogin_failure_window_seconds, among the access and sign-in settings). - Until the user has chosen the password, the account signs in by no path: the login and HTTP Basic refuse every password, nothing is forwarded to a credential-forwarding realm, and an OIDC login does not link to it by name. Admin → Users shows it as invited, or invitation expired once its link no longer works.
- The user’s page offers Resend invitation (a new email, to another address if needed), Copy invitation link (a new link shown once, for an install without email or a mailbox that does not receive it) and Revoke invitation. A new link replaces the previous one, which stops working. Deactivating or deleting the account revokes its invitation.
- Without an email service, or when the send fails, the account is still created and the link is shown once on the page to pass on yourself.
- Accounts of a credential-forwarding realm (see Usernames) and OIDC accounts are not invited: they sign in through their realm. The bootstrap admin keeps its configured password.
The admin API is POST /admin/users with {"username", "email", "role"}
(role is reader, writer or admin), or "realm" for a
credential-forwarding account. It takes no password: the answer carries
invitation (email_sent, error, expires_at, and the link when no
email went out). POST /admin/users/{id}/invitation ({"send_email": true | false, "email": "…"}) makes a new invitation and DELETE revokes it. The
invited user’s page calls POST /auth/invitations/lookup and
POST /auth/invitations/accept ({"token", "password"}).
Deactivating users
To cut a user off without deleting the account, deactivate it on the user’s
page (Admin → Users → <user> → Deactivate, or PATCH /admin/users/{id}
with {"active": false}):
- every sign-in is refused: the password, the browser, OIDC, credential forwarding, the headers of a trusted-header realm and API tokens, and no realm provisions the name back to life;
- the user’s browser sessions end, their cached upstream verdicts are dropped, and a pending invitation is revoked;
- their tokens are refused but not revoked: reactivating
(
{"active": true}) lets the unrevoked ones work again, so revoke those that should not before reactivating.
Deactivate, rather than delete, an account to suspend during an incident or to keep after its user left: deleting cannot be undone and takes the account’s tokens, roles and identity links along, and an OIDC realm that provisions accounts re-creates a deleted one at its next login.
Sign out everywhere (POST /admin/users/{id}/reauth) ends the user’s
browser sessions without deactivating the account. For a
credential-forwarding account the next password is checked against the
upstream again, and the tokens the user minted from a forwarded sign-in are
revoked; tokens an administrator issued keep working.
Service accounts
A CI pipeline, a build agent or a network the load balancer admits is no
person, and should not have a person’s account that someone could sign in
to, invite or convert. Give it a service account (Admin → Service
accounts, /manage/service-accounts): an account for a machine or a
network that nobody signs in to.
- It has a username (a UPN, in the name space every account shares, so a name always means one account), a description of what it is for and who owns it, roles and grants like a user’s, and an access level of reader or writer.
- It never has a password, an invitation or an email address, and nobody signs in to it: the login screen, a password over HTTP Basic, credential forwarding (its name is never forwarded upstream) and OIDC (no identity is linked to it by name) all refuse it. Sign out everywhere, invitations and the switch to a local account do not apply.
- It never holds the built-in
adminrole. For admin automation, issue an administrator a token with theadminscope instead. - It authenticates with the API tokens an administrator issues on its page (each secret shown once; expiry, scope and revocation as for a user’s), and through the trusted-header realm it belongs to, if any. A service account belongs to at most one such realm, and only that realm may name it: another trusted-header realm, even one that provisions accounts, gets a 401 for its name. No realm creates service accounts.
- Deactivating it (Deactivate… on its page) stops its tokens and its realm from authenticating it; reactivating lets both work again. The list shows when each service account was last used: its tokens’ latest use, or its realm’s last request naming it.
A CI pipeline that publishes packages: create ci@example.com as a writer
(or with a role that grants write on its repositories), then issue it a
token on its page, scoped to those repositories if you like, and store the
token in the pipeline’s secrets:
dotnet nuget push -s https://forge.example.com/repository/nuget/index.json \
-k "$FORGE_TOKEN" MyLib.1.2.3.nupkg
A network the load balancer admits, reading without credentials: create
office-lan@example.com with the trusted-header realm of the load balancer
as its realm, give it the grants the network should have, and let the load
balancer name it for the network’s clients (see the examples under
Trusted-header realms).
The admin API is the users API: POST /admin/users with "kind": "service", the username, an optional description and role (reader
or writer), and the optional realm (the name of a trusted-header realm).
GET /admin/users?kind=service lists the service accounts; every user view
carries kind, description, realm and last_used_at. PATCH /admin/users/{id} sets a service account’s description and realm
(null clears either) besides active. Tokens are
POST /admin/users/{id}/tokens, roles PUT /admin/users/{id}/roles.
Usernames
Every account is named by a UPN, local-part@domain (alice@example.com):
the name says which organization the account belongs to, two
organizations’ alice cannot collide, and it is the name Entra and Active
Directory sign the person in with. After canonicalization (trimmed, Unicode
NFC, lowercased) a username must have exactly one @, a non-empty local part
without whitespace, and a domain of at least two DNS labels without a
trailing dot. Each label is 1 to 63 letters, digits or -, and does not start
or end with -; the last one is not all digits. A Unicode domain is accepted
as written (jürgen@müller.example), and so is its xn-- form; the two are
different names.
The rule applies wherever an account is created:
- Admin → Users and
POST /admin/usersanswer 400 (“username must be a UPN such as alice@example.com”). - The bootstrap admin (see above).
oidcrealms. The realm’s claim chain (the username claim, else the email claim whenemail_verified, elsesub) must yield a UPN for a new account, or the login is refused and the login page names the claim. With Entra,preferred_usernameis the UPN. Elsewhere, map a claim that holds one, such asemail. A name that is taken gets-2,-3, … on its local part. Whether a realm creates the account at all is set for each realm: Auto-provision users on first login on its page under Admin → Authentication (auto_provisionin its config). The[oidc]block does not set it; the realm it seeds starts with it on.- Credential-forwarding realms create no accounts. Create each one under
Admin → Users with the realm as its sign-in (
POST /admin/userswith"realm": "<name>"and no password), named by a UPN whose local part is the upstream’s username: Forge forwards the local part, soalice@example.comsigns in as the upstream’salice. A login the upstream accepts but that names no account of the realm is refused. A realm config that setsauto_provision: trueis refused by the admin API. - Trusted-header realms (below) name an account
by the UPN in their username header. With Create a user’s account on its
first request (
auto_provision) the realm creates a missing one; otherwise create each account under Admin → Users with the realm as its sign-in. A name that is no UPN is refused. - Service accounts, under Admin → Service accounts.
Access and sign-in settings
Who may read without signing in, and how people sign in, is not deployment
configuration: an administrator sets it in the web UI (Admin → Settings,
/manage/settings) or through GET/PUT /admin/settings/auth. The values
are kept in the database, so every instance applies the same ones: the
instance that takes a change at once, the others within
auth.realm_cache_ttl_seconds (10 s), without a restart. A PUT changes the
fields its body names; a value out of range is refused with 400 and what is
wrong.
| Setting | Default | Meaning |
|---|---|---|
anonymous_mode | off | off or all: who is served without credentials. A repository is read anonymously only when it also allows it (Allow anonymous read, or a grant of the anonymous role). Under all, signed-in users also read what anonymous callers may. To let a network read without credentials, have the load balancer name a service account through a trusted-header realm. |
login_mode | identifier_first | The login screen: the username first (then a password prompt or the identity provider whose login domains match), or combined (username and password, plus the OIDC button). |
remember_accounts | true | The login screen offers the accounts signed in on the device before, so signing in again is one click (see below). false for kiosks and shared computers. |
allow_password_basic | false | Accept a user’s password over HTTP Basic on the feeds (see below). |
session_idle_ttl_seconds | 28800 (8 h) | A browser session ends this long after its last use. |
session_absolute_ttl_seconds | 604800 (7 d) | A browser session ends this long after sign-in. New lifetimes apply to sessions signed in afterwards; an open session’s idle window moves to the new idle timeout on its next use. |
login_max_failures | 10 | Failed passwords per client network and username before the lockout; 0 turns it off. Each instance counts its own. |
login_failure_window_seconds | 300 | The failure window, and how long a lockout lasts (at most a day). |
A configuration that still sets one of these (auth.anonymous_mode,
FORGE_AUTH__ALLOW_PASSWORD_BASIC, …, or the oidc.session_* lifetimes)
is refused at startup, with a message naming the keys to remove.
Remembered accounts
After someone signs in, the login screen shows their account the next time
instead of an empty username field, like Microsoft’s “Pick an account”. A
click on it continues as that account: a password account at the password
step, an OIDC account at its identity provider, which gets the username as
login_hint and so does not ask for the account again (the username typed
into the identifier-first form is passed the same way). Use another
account shows the usual form, and each account’s ✕ forgets it. Signing out
keeps the account; the browser keeps at most five, the latest first.
The accounts are kept in the browser only (localStorage): the username, the
name the identity provider gave, the realm and when it was last used, never a
password, a token or the session. The server never lists accounts, so
POST /auth/identify answers as before, from the realms’ configuration alone.
An account whose OIDC realm was disabled or renamed is routed like a typed
username. With remember_accounts off (Admin → Settings → Remember accounts
on the login screen), the login screen keeps none and forgets those a
browser kept, the next time it is opened there.
Password Basic auth
allow_password_basic is off by default: the web UI signs in through its
login form (not affected by this setting), and NuGet clients and scripts
authenticate with API tokens (a token in the Basic password field is always
accepted). An enabled credential-forwarding realm turns the Basic password
path on by itself, because its users have no tokens until they first sign in
— and that path runs the whole password chain, so local users’ passwords
are then accepted over Basic too, whatever this setting says. Turn it on only
if a client must send a local user’s password over Basic without such a
realm.
A password sent over Basic is a feed credential only: the admin API
(/admin/*) and token minting refuse it with 403, even for an administrator,
because NuGet clients store that password and replay it on every request.
Administer through a browser session (the web UI, or POST /auth/login plus
the returned CSRF token), or mint an admin-scoped token
({"scope": {"admin": true}}) for automation.
Forge sends email for invitations and notifications through one email
service, set up under Admin → Email (/manage/settings/email). The
settings are stored in the database only: there is no forge.toml section
and no FORGE_* variable for them, so they can change while Forge runs and
hold the same value on every instance (another instance picks a change up
within auth.realm_cache_ttl_seconds). Until they are saved, Forge sends no
email.
- Sender. The
Fromaddress, an optional display name andReply-To, and how long an invitation link stays valid (default 72 hours, at most 720). - SMTP server. Host, port and encryption: STARTTLS (port 587, refused when the server does not offer it) or TLS from the first byte (SMTPS, port 465). The server’s certificate is always verified against the system’s trusted roots; a private CA is not supported. Without TLS Forge connects only to a loopback host, a relay on the same machine (port 25). A username and password sign in with AUTH PLAIN or LOGIN, whichever the server offers; without a username Forge sends without signing in. IMAP is not used: a mail server whose IMAP takes an app password (Stalwart, for example) takes it on SMTP submission too.
- Microsoft 365 (Graph). Forge signs in as an Entra app registration and sends with Microsoft Graph, as the sender mailbox; see Microsoft 365 below.
- SMTP server with OAuth. SMTP submission with an OAuth access token from the mail server’s own authorization server instead of a password, Stalwart’s for one; see SMTP with OAuth below.
- Secrets (the SMTP password, the Graph client secret, the OAuth client
secret and refresh token) are write-only: the page and
GET /admin/settings/emailsay only whether one is set. Saving without it keeps the stored one, unless the SMTP host or username, the Graph tenant or app, or the OAuth issuer or mailbox changed. They are sealed at rest (secret encryption). - Send test email sends a message with the stored settings, to the address you enter or to your own (your UPN), even while sending is turned off, and shows the server’s error when it fails. The page also shows the last send that failed.
- Turning it off (uncheck Send email) keeps the settings; Remove deletes them.
Each send is logged with its purpose, the recipient’s domain and the
outcome; message bodies, links and secrets never are. The admin API is
GET, PUT and DELETE /admin/settings/email and
POST /admin/settings/email/test ({"to": "..."}, optional).
Microsoft 365
For Exchange Online, send through Microsoft Graph rather than SMTP: SMTP
AUTH is off under security defaults, Microsoft recommends turning it off
tenant-wide, and its Basic authentication is being retired. Forge signs in
with the OAuth client credentials grant as an app registration and calls
sendMail for the sender mailbox, without saving the message to its Sent
Items; no mailbox password is involved.
-
Register the app. In the Entra admin center, App registrations → New registration, single tenant. Note the Application (client) ID and the Directory (tenant) ID.
-
Let it send as the sender mailbox only, with RBAC for Applications in Exchange Online:
Connect-ExchangeOnline New-ServicePrincipal -AppId <client id> -ObjectId <enterprise app object id> -DisplayName "Forge" New-ManagementScope -Name "Forge sender" -RecipientRestrictionFilter "PrimarySmtpAddress -eq 'forge@contoso.com'" New-ManagementRoleAssignment -App <client id> -Role "Application Mail.Send" -CustomResourceScope "Forge sender" Test-ServicePrincipalAuthorization -Identity <client id> -Resource forge@contoso.comThe object id is the enterprise application’s (Enterprise applications → Forge → Object ID), not the app registration’s.
-
Do not consent
Mail.Sendin Entra. The Microsoft GraphMail.Sendapplication permission with admin consent works too, but lets the app send as every mailbox in the tenant, and Exchange adds it to the scoped role: with both, the scope narrows nothing. (Application Access Policies,New-ApplicationAccessPolicy, are the legacy way to narrow an Entra consent.) -
Give Forge a credential, one of:
- A federated credential (no secret to store or rotate). Choose
A federated credential on the page and save: Forge then shows the
issuer (its
server.public_base_url), the subject and the audience. On the app, Certificates & secrets → Federated credentials → Add credential → Other issuer, and enter them. Forge signs its client assertions with the issuer keys a federated OIDC realm uses, published at/.well-known/openid-configurationand rotated automatically; Entra must reach that URL over https. The subject is the one a federated OIDC realm of the same app and tenant presents, so one credential covers both. - A client secret: Certificates & secrets → New client secret, and enter its value on the page. Renew it before it expires.
- A federated credential (no secret to store or rotate). Choose
A federated credential on the page and save: Forge then shows the
issuer (its
-
Enter the tenant, the client id and the sender (its UPN or object id; blank sends as the
Fromaddress) on Admin → Email, and Send test email.
A failed send shows Entra’s or Graph’s error: invalid_client with an
AADSTS… description names a wrong secret or credential;
ErrorAccessDenied means the sender mailbox is outside the app’s
Application Mail.Send scope (Test-ServicePrincipalAuthorization shows
it); a 401 from Graph usually means the sender has no
Exchange Online mailbox or licence. Each instance caches its access token
until shortly before it expires. National clouds and certificate
credentials are not supported yet.
SMTP with OAuth
A mail server with its own OAuth authorization server, Stalwart for one,
takes an access token on SMTP submission (SASL OAUTHBEARER, RFC 7628, or
XOAUTH2) where it would take a password. Forge then stores no mailbox
password: it registers itself as an OAuth client, the mailbox authorizes it
once, and Forge refreshes its access token from then on.
- Enter the server on Admin → Email with SMTP server with OAuth:
the SMTP host, port and encryption as for SMTP, the mailbox Forge
sends as (the account that authorizes it), and the authorization
server, its issuer URL (
https). Forge reads its metadata from/.well-known/oauth-authorization-server(RFC 8414), or/.well-known/openid-configuration, and the document must name that issuer. A scope is optional. Save. - Give Forge a client, one of:
- Register Forge (RFC 7591). Forge registers a confidential client
named Forge (
server.public_base_url) with the redirect URI the page shows, for the authorization code, refresh and (when offered) device and client credentials grants. A server that does not take registrations from anyone wants them authenticated: with an initial access token, or as an account of the mail server (HTTP Basic). Either is used for that request only and never stored. - Enter a client registered on the server by hand: its client id
and secret, with the redirect URI the page shows
(
https://<forge>/admin/settings/email/oauth/callback).
- Register Forge (RFC 7591). Forge registers a confidential client
named Forge (
- Authorize Forge, one of:
- In this browser: Forge sends you to the authorization server (the
code flow with PKCE, the mailbox as
login_hint); sign in as the mailbox and allow Forge, and the server sends you back to the page. The flow is bound to your Forge sign-in and this browser, and expires after 10 minutes. - On another device (the device authorization grant, RFC 8628): the page shows a code and where to enter it, and waits until the mailbox has allowed Forge.
- With the client credentials grant, when the server issues tokens to a client itself: no mailbox authorizes Forge.
- In this browser: Forge sends you to the authorization server (the
code flow with PKCE, the mailbox as
- Send test email.
Forge keeps the refresh token sealed with the settings and stores a new one
whenever the server rotates it; instances take turns refreshing, so two
never spend the same refresh token. When the server refuses a refresh
(invalid_grant: the authorization was revoked or expired), sending stops
and the page asks for a new authorization. A token the mail server refuses
is dropped and the next send gets a new one. Saving the settings keeps the
client while the issuer stays the same, and the authorization while the
SMTP host and the mailbox stay the same too: another host gets no token
before the transport is authorized again. Forge deletes a client it registered (RFC 7592) when the settings stop
using it, or names it so that it can be removed on the server when the
server offers no way to.
Stalwart. Stalwart 0.16 has all of this built in: its issuer is its
public https URL. Registration wants an account allowed to register OAuth
clients (the administrator account is), so choose An account of the mail
server and enter it. Stalwart offers no RFC 7592 management: remove a client
Forge no longer uses under its OAuth clients. Authorize in the browser on
Stalwart’s sign-in page, or on another device at Stalwart’s /device page.
Stalwart issues no tokens by the client credentials grant. The mailbox’s
account needs the permission to send email; a refused token is answered
535 5.7.8, which the test send shows.
The admin API is POST /admin/settings/email/oauth/register (body
{"initial_access_token": "…"}, {"username": "…", "password": "…"} or
{}), DELETE /admin/settings/email/oauth/client,
POST /admin/settings/email/oauth/authorize ({"method": "authorization_code" | "device" | "client_credentials"}),
GET /admin/settings/email/oauth/callback and
POST /admin/settings/email/oauth/device/{id} (one poll of a device
authorization).
Secret encryption at rest
Upstream proxy passwords, authentication-realm secrets (OIDC client
secrets, credential-forwarding peppers), the email service’s password or
client secret (Email), the generated registry-token key
(without docker.token_secret) and the keys Forge signs its OIDC client
assertions with (a realm with a federated credential) are stored with
double-envelope encryption:
DEK → KEK → a root key that never enters the database. Encryption is
required: the [crypto] section names exactly one root provider, and
Forge refuses to start without a usable root key, naming what to set:
openssl rand -base64 32 # generate the root key once; store it in your secret manager
FORGE_CRYPTO__ROOT__PROVIDER=env
FORGE_ROOT_KEY=<base64 32 bytes>
# or, for a mounted secret file:
# FORGE_CRYPTO__ROOT__PROVIDER=file
# FORGE_CRYPTO__ROOT__ROOT_KEY_PATH=/run/secrets/forge_root_key
The Helm chart uses the env provider and refuses to render without
FORGE_ROOT_KEY (in secretEnv, existingSecret or env); the kustomize
base sets the provider, and forge.env holds the key.
The root key is part of your backup. Without it, stored secrets cannot be recovered: they have to be entered again (proxy passwords, realm secrets, the email credentials), and Forge generates a new registry-token key and new issuer keys. Packages and metadata are unaffected. AWS KMS is not supported yet.
There is no plaintext mode: every instance needs the same root key, and a secret is never written unencrypted.
Each encrypted secret is bound to its row and to where it is sent: a proxy
password to the origin (scheme, host and port) of the repository’s
remote_url, an OIDC client secret to the realm’s issuer and client_id,
the email password to the SMTP host and username, the Graph client secret to
the tenant and app, the OAuth mailbox’s secrets to the issuer, client,
mailbox and SMTP host.
Changing those through the admin API asks for the secret again. Changing
them directly in the database makes the secret undecryptable (the proxy
then fails instead of sending the password to the new host), and it has to
be entered again through the API. For a realm, that is an edit
(PATCH /admin/realms/{id} or the realm form) that includes the secret:
until then the realm is skipped at login, and an edit without the secret
answers 400 naming it.
GET /admin/crypto/keys reports the keys’ metadata and the current
root_key_version. Key rotations (POST /admin/crypto/rotate/{dek,kek,root}
and the worker’s age-based rotation) are serialized across all instances; a
rotation requested while another one runs is answered with 409 Conflict and
can simply be retried.
Rotating the root key on a running cluster
Every instance must be able to unwrap the KEKs with whichever root they were last wrapped under, so rotate the root in two phases, with a complete rolling restart after each:
- Add the new key as the previous key everywhere. Keep the current key
in
FORGE_ROOT_KEYand add the new key as the previous one:FORGE_CRYPTO__ROOT__ROOT_KEY_ENV_PREVIOUS=FORGE_ROOT_KEY_PREVIOUSplusFORGE_ROOT_KEY_PREVIOUS=<new key>(with the file provider,FORGE_CRYPTO__ROOT__ROOT_KEY_PATH_PREVIOUS). Roll this out to every instance. Nothing is re-wrapped yet, but every instance can now read keys wrapped under either root. - Swap. Put the new key in
FORGE_ROOT_KEYand the old one inFORGE_ROOT_KEY_PREVIOUS, and roll this out. Each instance re-wraps the KEKs under the new root at startup;POST /admin/crypto/rotate/rootdoes the same on demand. During the roll-out old and new instances both reach both roots, so neither fails to unwrap (an instance that has not restarted yet may wrap a KEK back under the old root; that is harmless and converges once every instance runs phase 2). - Once
GET /admin/crypto/keysshows every non-revoked KEK on the newroot_key_version(revoked KEKs keep the old one and are never used again), remove the previous key and roll out once more. Keep the old root key with your backups for as long as you keep a database backup taken before the rotation: restoring such a backup needs it.
Swapping in one step makes the instances that still run with only the old key fail to unwrap KEKs that a new instance has already re-wrapped.
Reverse proxy
- Terminate TLS and forward to
http://forge:8421, preservingHost. ForwardX-Forwarded-For, and list the proxy inFORGE_AUTH__TRUSTED_PROXY_CIDRS. - Let only the proxy reach Forge. Forge believes the
X-Forwarded-Forof every peer intrusted_proxy_cidrs, and the headers of a trusted-header realm, so anything else that can connect from those ranges can pick the client address the login lockout sees. On Kubernetes the range is usually the whole pod network: admit only the ingress controller or gateway with a NetworkPolicy (the Helm chart’snetworkPolicy.from, the kustomizenetwork-policycomponent; the CNI must enforce it). - Allow large request bodies for
PUT /repository/{repo}/package(package pushes), for image pushes under/v2, where docker sends a whole layer in one request, and for file uploads under/repository(and the web UI’s/browse/raw). In nginx that isclient_max_body_size. Match it to the largest upload limit a repository has (set per repository in the web UI; 512 MiB for package and raw repositories and 1 GiB per blob for docker by default). - Serve
/v2at the root of the host: Docker clients always talk tohttps://<host>/v2/, whatever pathpublic_base_urlhas. A docker repository with ahost(docs/docker.md) needs that name routed to Forge withHostpreserved, and covered by the certificate. - Disable response buffering or give it enough room. Package downloads are streamed.
- Do not rewrite the
OriginorSec-Fetch-Siteheaders. Forge refuses state-changing browser requests that are cross-site, so a proxy that strips or fakes these headers breaks that protection. - With an
https://public_base_urlForge sendsStrict-Transport-Security: max-age=31536000; serve it over https only. API responses and every feed and registry response to a request with credentials areCache-Control: private, no-store; feed responses carryVary: Authorization, Cookie, X-NuGet-ApiKeyand registry responsesVary: Authorization, Cookie, Accept: keep a caching proxy from overriding either. - Forge bounds slow clients itself (
server.header_read_timeout_seconds,body_idle_timeout_seconds,request_timeout_seconds); keep the proxy’s own timeouts at least as long for package pushes. A proxy repository’s first download of a package or layer may take up toproxy.download_timeout_seconds(30 min) before Forge answers, plusrequest_timeout_secondsto store it: allow that long a wait for the response on the feed and registry paths too, or the client is cut off first. - Forge speaks HTTP/1.1 to the proxy (no h2c).
header_read_timeout_seconds(default 150) also closes keep-alive connections that stay idle that long. If the proxy keeps upstream connections open longer than that (Envoy and Istio default to 1 hour), lower the proxy’s upstream idle timeout or raise this setting; otherwise the proxy can reuse a connection Forge is closing and answer 502, which it does not retry for a push. AWS ALB and ingress-nginx (60 s), Traefik (90 s) and Caddy (120 s) are below the default.
Minimal nginx example:
server {
listen 443 ssl http2;
server_name nuget.example.com;
# ssl_certificate …; ssl_certificate_key …;
client_max_body_size 1g; # the largest repository upload limit (docker: 1 GiB by default)
proxy_request_buffering off;
proxy_buffering off;
location / {
proxy_pass http://forge:8421;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
}
}
Trusted-header realms
A trusted_header realm (Admin → Authentication → New realm → Trusted
header) authenticates a request as the account a header of the load
balancer or reverse proxy in front of Forge names, in the style of
REMOTE_USER. The proxy decides who gets which name:
- an SSO proxy (oauth2-proxy, Authelia, …) names the person it signed in;
- a load balancer names a service account for a
network or machines it admits (
ci-runners@example.com,office-lan@example.com), by source network, mTLS or VPN. This is how a network reads without credentials: the load balancer filters by network next to the rest of the network policy, and Forge only checks the headers. Create the service account under Admin → Service accounts with the realm as its trusted-header realm, and give it the grants the network should have. Only that realm may name it.
| Config key | Meaning |
|---|---|
username_header | The header naming the account, e.g. X-Forwarded-User. Its value is a UPN. |
secret_header | The header the proxy proves itself with, e.g. X-Forge-Proxy-Secret. |
secret | Its value, 16 to 1024 printable characters without spaces (openssl rand -hex 32, or Generate a secret in the form). Write-only and sealed at rest like the other realms’ secrets; compared in constant time. |
auto_provision | Create a user’s account on its first request when the realm has none of the name (off by default). It never creates a service account, nor takes over the name of one. |
What Forge believes, and when:
- The headers count only when the socket peer is in
auth.trusted_proxy_cidrsand the request’sHostis one of Forge’s own names (the host ofpublic_base_url, or one inserver.allowed_hosts: a DNS-rebinding page reaches the proxy under its own name). From any other request they are ignored. - They name the caller only when the request has no credential of its
own: an API token, a password or a signed-in session is that
credential’s, whatever the headers say, and one that is refused is a 401,
never the account the headers name. That includes an API token sent as
Authorization: Bearerto a feed, as npm sends its_authToken: an SSO proxy must not forward a Bearer token of its own to/repository(oauth2-proxy’s--pass-authorization-headerstays off). - The headers present but no realm’s secret among them, the right secret
with no, an empty, a repeated or an invalid name, a name of no account of
the realm (without
auto_provision), of a deactivated account, or of an account of another kind (local, OIDC, credential forwarding) or another realm, a service account of another realm or of none included: the headers name nobody. The request goes on as one without credentials, never anonymous whatever the anonymous mode: what needs a principal answers 401, while a credential of its own still counts (an API token sent asAuthorization: Bearer, a registry token) and its sender can sign in. Deactivating the service account a load balancer names for a network so cuts the network’s access, not the access of the people on it. The refusal is logged (trusted-header credentials refused). A realm never maps a name onto another source’s account, and never creates or names the built-inanonymousprincipal. - The headers are an ambient credential: the proxy adds them to every
request of a client it admits, the ones a browser sends for a foreign page
included. So a request they authenticate holds the account’s grants but
never the
adminrole: the admin API and minting tokens refuse it (403), and publishing packages or images takes an API token, which an administrator issues on the account’s page (Admin → Users → <user> → Tokens, or Admin → Service accounts → <account>). State-changing requests a browser marks as cross-site are refused (403) before authentication. An account of the realm never holds theadminrole either (creating or making it an admin is a 400): it would count as an administrator nobody can act as, and the last real one could go. - A feed request the named account may not make is challenged like an
anonymous one (401 with
WWW-Authenticate: Basic), so a client with credentials of its own, such as NuGet, sends them. - When the realm last named each account is kept, written at most once a minute: Admin → Service accounts shows it as the account’s last use.
- Forge answers every feed and registry request the headers authenticated
with
Cache-Control: private, no-store: no cache varies on the headers. Keep caches from overriding it, and do not let one between the proxy and Forge store/repositoryresponses regardless of their caching headers. - In the web UI a user the SSO proxy signed in is signed in without a session; they sign out at the proxy.
The proxy must remove both headers from every request a client sends, and set them only for the clients it admits, the secret only together with a name. Most proxies replace a header they set; make sure they also drop the client’s when they set none.
nginx, admitting a network as a service account (proxy_set_header
replaces the client’s header, and sends none for an empty value):
# The networks admitted as a service account, and the secret only with a name.
geo $forge_account {
default "";
10.20.0.0/16 "ci-runners@example.com";
}
map $forge_account $forge_proxy_secret {
"" "";
default "<the realm's secret>";
}
server {
# … as above …
location / {
proxy_pass http://forge:8421;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-User $forge_account;
proxy_set_header X-Forge-Proxy-Secret $forge_proxy_secret;
}
}
nginx behind oauth2-proxy (--set-xauthrequest), naming the person it
signed in; every request it lets through has a name:
location /oauth2/ {
proxy_pass http://oauth2-proxy:4180;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location = /oauth2/auth {
internal;
proxy_pass http://oauth2-proxy:4180;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
}
location / {
auth_request /oauth2/auth;
error_page 401 = /oauth2/sign_in;
auth_request_set $forge_user $upstream_http_x_auth_request_email;
proxy_pass http://forge:8421;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-User $forge_user;
proxy_set_header X-Forge-Proxy-Secret "<the realm's secret>";
}
NuGet, npm and docker clients cannot sign in to an SSO proxy: serve the
feeds and /v2 to them past it (another host name, or the proxy’s own rule
for requests with an Authorization or X-NuGet-ApiKey header), with
their API tokens.
ingress-nginx runs the same directives in a configuration-snippet
annotation where the controller allows snippets. Without snippets, use
nginx.ingress.kubernetes.io/auth-url and
nginx.ingress.kubernetes.io/auth-response-headers: the controller copies
the listed headers of the auth service’s answer onto the request, in place
of the client’s, so let the auth service answer both the name and the
secret.
Traefik (v3), admitting a network: a router for the network’s clients sets both headers, the other router removes them (an empty value removes a header):
http:
routers:
forge-ci:
rule: "Host(`forge.example.com`) && ClientIP(`10.20.0.0/16`)"
priority: 100
service: forge
middlewares: [forge-ci-account]
forge:
rule: "Host(`forge.example.com`)"
service: forge
middlewares: [forge-no-account]
middlewares:
forge-ci-account:
headers:
customRequestHeaders:
X-Forwarded-User: "ci-runners@example.com"
X-Forge-Proxy-Secret: "<the realm's secret>"
forge-no-account:
headers:
customRequestHeaders:
X-Forwarded-User: ""
X-Forge-Proxy-Secret: ""
Behind an SSO proxy, Traefik’s forwardAuth with
authResponseHeaders: [X-Auth-Request-Email] replaces the client’s header
with the auth service’s answer; add the secret with a headers middleware
after it, and set username_header to X-Auth-Request-Email.
HAProxy, admitting a network:
frontend fe_forge
bind :443 ssl crt /etc/haproxy/forge.pem
option forwardfor
http-request del-header X-Forwarded-User
http-request del-header X-Forge-Proxy-Secret
acl ci_network src 10.20.0.0/16
http-request set-header X-Forwarded-User ci-runners@example.com if ci_network
http-request set-header X-Forge-Proxy-Secret <the realm's secret> if ci_network
default_backend be_forge
Health probes
| Endpoint | Use |
|---|---|
GET /healthz | Liveness. Never touches a dependency. |
GET /readyz | Readiness. PostgreSQL and S3 are each checked with a 3 s deadline; the endpoint returns 503 with the failing check names. With the local blob cache, an unreachable S3 is not a failure: 200 with "status": "degraded" and "object_store": "degraded". |
GET /version | The running version and revision. |
The image has no shell tools such as curl. Use your orchestrator’s HTTP
probes rather than a Docker HEALTHCHECK.
Forge starts listening only after startup work is done: database
migrations, the bucket check, the bootstrap admin, the secret-encryption
bootstrap. On the first start of a new version (migrations) or
against a slow database this can take a while, and /healthz does not answer
until then. Give the container a startup probe on /healthz with a
generous budget, so the liveness probe does not restart it mid-migration.
(With the local blob cache, an unreachable S3 does not
hold up the start: the bucket check is retried in the background.)
startupProbe:
httpGet: { path: /healthz, port: 8421 }
periodSeconds: 5
failureThreshold: 60 # up to 5 minutes for migrations
livenessProbe:
httpGet: { path: /healthz, port: 8421 }
periodSeconds: 10
readinessProbe:
httpGet: { path: /readyz, port: 8421 }
periodSeconds: 10
The image runs as the numeric user 10001:10001, so
securityContext.runAsNonRoot: true works without further settings.
On SIGTERM, Forge stops accepting connections and lets in-flight requests finish before exiting. Give it a termination grace period long enough for your largest package download.
Telemetry
Forge exports OpenTelemetry metrics, traces and logs over OTLP/HTTP,
with the official OpenTelemetry SDK. Each signal is off until [telemetry]
turns it on:
[telemetry]
metrics = true # FORGE_TELEMETRY__METRICS=true
traces = true # FORGE_TELEMETRY__TRACES=true
logs = true # FORGE_TELEMETRY__LOGS=true
# per_repository = true # the repository's name on the metrics
# inventory_interval_seconds = 300
Where the signals go, and how, is set by the standard OpenTelemetry variables, read by the SDK:
| Variable | Meaning |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | The OTLP/HTTP endpoint, usually an OpenTelemetry Collector beside Forge; default http://localhost:4318. Forge appends /v1/metrics, /v1/traces and /v1/logs; the OTEL_EXPORTER_OTLP_{METRICS,TRACES,LOGS}_ENDPOINT variables set one signal’s full URL. |
OTEL_EXPORTER_OTLP_HEADERS | Headers sent with every export (api-key=... for a vendor’s backend). Keep it in a secret. |
OTEL_EXPORTER_OTLP_PROTOCOL | http/protobuf, the only protocol Forge speaks: Forge refuses to start with grpc. |
OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_TIMEOUT | gzip; the export timeout in milliseconds (default 10 000). |
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES | The resource. Forge sets service.name=forge, service.version, vcs.ref.head.revision, host.name and a random service.instance.id; what these variables say wins. |
OTEL_METRIC_EXPORT_INTERVAL | Milliseconds between two metric exports (default 60 000). |
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG | Which traces are kept (default parentbased_always_on: all, and a caller’s decision); e.g. parentbased_traceidratio and 0.1. |
OTEL_SDK_DISABLED | true turns every signal off, whatever [telemetry] says. |
An export that fails is logged and never fails a request: metrics are cumulative, so the next export carries what a failed one held, while a failed batch of spans or log records is dropped. Each instance exports its own telemetry.
Metrics. The names follow the OpenTelemetry semantic conventions where
one exists (http.server.request.duration, db.client.connection.count),
and are forge.* otherwise. Durations are histograms in seconds, sizes in
bytes (By). Attributes stay bounded: none carries a package id, a
version, an image or tag, a username, a token or a path. A request is
counted under the surface of the route it matched (forge.surface:
nuget, oci, plugin_npm, plugin_raw, plugin_other, admin,
browse, auth, health, ui; repository for a /repository/… request
that named no repository), never its URL. The repository’s name
(forge.repository) is the one attribute an administrator chooses; with
very many repositories turn telemetry.per_repository off and the
repositories add up.
| Area | Instruments (attributes) |
|---|---|
| HTTP | http.server.request.duration (forge.surface, http.request.method, http.response.status_code), to the response headers; http.server.active_requests (http.request.method); http.server.response.body.size |
| Proxies | forge.proxy.lookups (forge.repository, forge.proxy.kind: metadata, package, manifest, blob, file; forge.proxy.outcome: hit, fetched, stale_served, negative_cached, delay_gated, not_found, offline, upstream_error, error); http.client.request.duration of the requests to the upstreams (forge.repository, http.request.method, http.response.status_code, or error.type when none came); forge.proxy.upstream.fetched (bytes) |
| Storage | forge.s3.operation.duration (forge.s3.operation, forge.s3.outcome: ok, not_found, precondition_failed, unreachable, breaker_open, error); forge.s3.circuit_breaker.open (with the local blob cache); the counters of GET /admin/blob-cache as forge.blob_cache.* (hits, served, hits_while_store_down, misses, fills, fill_failures, corrupt, evictions, evicted, bypassed, verifications, and entries, usage, reserved, limit); forge.staging.usage against forge.staging.limit (server.max_staging_bytes); forge.outbox.pending, forge.outbox.due, forge.outbox.oldest_due.age; forge.gc.deleted (forge.gc.source) |
| Cache | forge.cache.lookups (forge.cache.tier: l1 in process, l2 Redis; forge.cache.result); forge.cache.backplane.connected (with Redis); forge.cache.invalidations (forge.cache.direction: sent, received) |
| Database | db.client.connection.count (db.client.connection.state: idle, used), db.client.connection.max, db.client.connection.wait_time, db.client.connection.timeouts |
| Workers | forge.worker.runs (forge.worker, forge.worker.outcome), forge.worker.run.duration, forge.worker.last_success.time (Unix seconds), forge.worker.interval, forge.worker.restarts, for the outbox, reconciler, proxy-refresh, retention, auth-reaper, cred-reaper, registry-sweeper, crypto-rotation, blob-cache and metrics-inventory workers (restarts for every worker) |
| Retention | forge.retention.runs (forge.repository, forge.retention.outcome), forge.retention.versions (forge.retention.action: unlisted, deleted), forge.retention.freed (the repository’s size before a run less its size after) |
| Auth | forge.auth.sign_ins (forge.auth.realm_type, forge.auth.outcome): a password through the realms, on the login form or over HTTP Basic, and OIDC logins; forge.auth.throttled (forge.auth.throttle.reason), forge.auth.token_exchanges (forge.auth.exchange.outcome), forge.auth.jwks.refresh_failures, forge.auth.forwarding.errors |
| Inventory | forge.repositories (forge.repository.format, forge.repository.kind), forge.repository.versions (package versions, image tags, files) and forge.repository.size (forge.repository), counted by a worker every telemetry.inventory_interval_seconds, never per export |
The inventory and the outbox backlog are database-wide: every instance
reports the same numbers, so aggregate them with max, not sum. The pool’s
wait times come from sqlx’s pool events, which forge-server reads whatever
RUST_LOG says.
In Prometheus (the Collector’s prometheus or prometheusremotewrite
exporter, or Prometheus’ own OTLP receiver), the names become
http_server_request_duration_seconds, forge_proxy_lookups_total,
forge_repository_size_bytes and so on, the attributes forge_surface,
forge_repository, …, and job and instance come from service.name and
service.instance.id. deploy/monitoring
has an example Collector configuration, a Grafana dashboard and alerting
rules written against those names: a worker without a successful pass for
three of its intervals, the S3 circuit breaker open, a high 5xx rate, a
growing outbox backlog, upstream errors and rate limits per proxy, an
exhausted connection pool, a lost Redis backplane and nearly full staging
space.
Traces. Each request is a server span (GET /repository/{repo}/{*rest}:
the route, never the URL, names it), a child of the caller’s trace when the
request carries a W3C traceparent. Beneath it are the spans of a proxy’s
lookup (proxy package, with its outcome), its requests to the upstream
(client spans, which send the trace on in their own traceparent, so a
Forge proxying another Forge continues the trace), and the S3 operations
(S3.GetObject, …). Each pass of a background worker is a trace of its own
(outbox pass, …). Health probes make no spans. A span’s URL never holds a
query string or credentials, and the secret in the path of an npm web login
(the URL the client polls for its token, the page that approves the login
and the API behind it) and the token in the path of npm logout are
recorded as {secret}, there and in the audit line of the request.
Logs. With logs on, what Forge logs (filtered by RUST_LOG, as the
console log is) is exported too, each record with the trace and span it was
written in, so a backend links a request’s log lines to its trace. The
console log stays as it is.
Running more than one instance
Instances coordinate through PostgreSQL (row locks and leases), so any number
can run behind the load balancer. Configure FORGE_REDIS__URL so that caches
and invalidations converge immediately, not only after a TTL expires. The
access and sign-in settings are read from the
database: the other instances follow a change within
auth.realm_cache_ttl_seconds. The password-failure lockout is kept in memory
on each instance; the networks a username signed in from, which that
username’s backoff exempts, are shared through the database.
Redis is optional at runtime: an instance starts while Redis is unreachable (after waiting a few seconds for it) and works from its own memory until Redis is back. A Redis that stops answering slows only the requests that talk to it, and an instance whose invalidation subscription goes silent notices within about 15 s, drops its in-memory cache and resubscribes.
- Connections. Every instance opens up to
database.max_connections(default 20); keep instances × that below PostgreSQL’smax_connections. Through PgBouncer, use session pooling: migrations and key rotation take session-level advisory locks, which transaction pooling breaks. - Upgrades. Migrations run at startup, serialized by an advisory lock. Once a newer version has migrated, an instance of the older version refuses to start, so roll forward and do not restart old instances in the middle of a rollout.
- Rolling deploys. On SIGTERM Forge stops accepting connections at once
and drains in-flight requests. Give the load balancer time to stop routing
to the instance first (for Kubernetes a
preStopsleep of 5 to 10 s), otherwise requests routed in that gap are refused, and a refused push is not retried. - Temp space. Pushes and first-time proxy downloads are staged in
server.staging_dir(defaultforge-stagingin the system temp directory). All staged files of an instance shareserver.max_staging_bytes(default four times the largest item: a push under the default upload limits,proxy.max_package_bytesordocker.max_blob_bytes, so 4 GiB;0removes the limit); past it, pushes and downloads are answered503withRetry-Afteruntil space frees up. A repository’s upload limit may not exceed the budget (lessnuget.max_nuspec_bytes), so raise the budget before giving a repository a larger limit. An explicit budget below the 1 GiB docker defaults lowers them to what it holds: docker repositories without a limit of their own accept layers up to the budget (lessnuget.max_nuspec_bytes), and so do docker proxy downloads unlessdocker.max_blob_bytesis set (startup warns). Size the volume above that budget. Each process stages in a locked subdirectory of its own, and the files a killed process left are removed at the next start however recent they are, so they do not add to the space a restarted instance needs. Give each instance a volume of its own (the Helm chart and the Kubernetes base do). Instances on different hosts may share one only if its file locks work across those hosts (NFSv4, or NFSv3 withlockd; not a mount withnolockorlocal_lock): otherwise one instance’s startup can remove another’s files mid-request. Where locks do not work at all, a killed process’s files are removed once idle for an hour instead. The staging directory must belong to Forge’s user and must not be writable by group or others (unless it is sticky, like/tmp); Forge refuses to start otherwise, and creates a missing one, and each process’s subdirectory, with mode0700. A staged file is read back through the handle it was written with, never by its name. On a host shared with other users, pointserver.staging_dirat a directory of Forge’s own: another user could create the default one under the system temp directory first, and Forge then refuses to start. (The container image and the Kubernetes manifests are not affected.) - Audit log. Package mutations and admin changes are logged at INFO
under the
forge::audittarget, with the user/token or workload that made them; keep those lines. - Outbound proxy. Proxy repositories send upstream traffic through
HTTPS_PROXY/HTTP_PROXY/ALL_PROXYunlessNO_PROXYexempts the host. Forge never sends an upstream request, or follows a redirect, to a link-local or cloud-metadata address (169.254.0.0/16,fe80::/10,metadata.google.internal, …). Through an outbound proxy the proxy resolves the host name, so Forge resolves it first and refuses it when any of its addresses is such a target. A name that resolves differently for the proxy than for Forge, or that only the proxy can resolve, still gets through: block those addresses on the proxy too. A proxy’s format chooses which upstream URLs to request, but Forge sends them only to the origin of the repository’sremote_urland to origins the upstream’s own documents name (format-plugins.md); to keep internal services out of reach even of an upstream that names them, block them on the outbound proxy or with network policy.
Redis is a trust boundary
Anything that can write to Redis can write Forge’s shared cache. Forge therefore keeps every value a security decision or a client rests on out of Redis: authorization sets, credential-forwarding verdicts, OIDC discovery documents and signing keys, workload-identity token verdicts, and a proxy’s upstream service index, manifests and registrations are cached in each instance’s memory only. Redis carries only their invalidations, so the worst a Redis writer can do to them is evict them early. Publishing invalidations for ever new names cannot grow an instance’s memory without bound either: the per-name bookkeeping is capped (100,000 names), and an instance that reaches the cap clears it together with its in-memory cache, which costs a round of cache misses, not correctness. The only values still shared through Redis are two proxy markers (“listing refreshed”, “version rejected”); forging them can hide a proxied package until its TTL expires, not change what is served.
Still treat Redis like the database: a dedicated instance (or ACL user) for
Forge, requirepass/ACL credentials in the URL, not reachable from outside
the deployment, and TLS when it crosses a network you do not control
(FORGE_REDIS__URL=rediss://user:password@redis.internal:6380; the server
verifies the certificate against the system trust store).
Local blob cache
Optionally, Forge keeps a copy of the blobs it serves on a local disk. Hot
downloads are then served without S3 (no request, no egress), and packages in
the cache keep resolving while S3 is unreachable. S3 stays the source of
truth: the cache only ever holds copies of objects S3 has, and since blobs are
content-addressed and immutable, a copy needs no invalidation. The disk can be
anything from an emptyDir to a dedicated or persistent volume.
| Setting | Meaning |
|---|---|
FORGE_BLOB_CACHE__PATH | The cache directory, e.g. /var/cache/forge/blobs. Unset (the default) = no cache. Each instance needs a directory of its own. It must belong to Forge’s user, and neither group nor others may write to it: startup refuses it otherwise, since a file placed there is served as the blob it is named after. Forge creates it (mode 0700) when it is missing, so point it at a subdirectory of a mounted volume, whose root belongs to root. Entries are readable by Forge’s user alone (0600). On Kubernetes, give the pod fsGroupChangePolicy: OnRootMismatch next to its fsGroup (the shipped manifests and chart do): otherwise the kubelet makes the directory group-writable at each pod start. Forge then restores it to 0700 and its entries to 0600, logging a warning, and checks each entry against its hash on its first hit. |
FORGE_BLOB_CACHE__MAX_BYTES | auto (default): the size of the filesystem holding the path, minus min_free_bytes, minus server.max_staging_bytes when staging is on the same filesystem; re-read every minute, so a resized volume is picked up without a restart. Or a fixed number of bytes. |
FORGE_BLOB_CACHE__MIN_FREE_BYTES | Free space the cache always leaves on its filesystem, evicting instead of filling it, even when something else uses the disk too. Default: 5 % of the filesystem. On a shared disk with less free space than that, nothing is cached (logged as a warning at startup). |
How it fills. A download looks in the cache first; a hit never contacts
S3. A miss streams the blob from S3 to the client and into the cache at the
same time; concurrent misses of one blob share that single S3 read. The copy
is checked against the blob’s SHA-256 and size before it joins the cache. A
push or a first-time proxy download writes its staged file through to the
cache once S3 has it (a hard link when staging shares the cache’s
filesystem, a copy otherwise), so freshly published packages are cached too.
Package manifests (.nuspec) go through the cache as well. With the cache
on, package downloads also answer Range requests.
Eviction is first in, first out: over the limit, the blobs filled longest ago go first, however often they are read (a read changes nothing on disk). A file’s modification time is its fill time, so the order survives a restart on a persistent volume. A blob that garbage collection deletes from S3 is dropped from the cache of the instance that deletes it at once, also when a download was filling it at the time. Every other instance checks its cached blobs against the database at startup and every 10 minutes, and drops those garbage collection deleted (one query per 1,000 cached blobs).
While S3 is down:
- Cached blobs are served as usual. Package metadata comes from PostgreSQL, which is still required.
- Uncached downloads, pushes and first-time proxy downloads answer
503withRetry-After. A circuit breaker makes them fail at once once S3 is known to be down, instead of each waiting for the S3 timeout. Nothing is ever acknowledged that exists only on one instance’s disk. /readyzanswers 200 with"object_store": "degraded", so instances stay in the load balancer and serve what they have (a database failure is still 503).- An instance (re)started while S3 is unreachable starts anyway, serves its cache, and retries the bucket check in the background.
- When S3 is back, misses fill again and
/readyzreportsok, without a restart.
Robustness. Any cache I/O error (a full disk, a read error, a file of the wrong size) is logged, the entry is dropped and the request continues from S3 (or answers 503 while S3 is down): the cache never turns a servable request into an error. Blobs found at startup are checked against their hash on their first hit, so a file damaged while Forge was down is fetched again; partial fills a killed process left are removed at startup.
Statistics. GET /admin/blob-cache (admin) reports the size against the
effective limit (and whether it is auto), the hit ratio, bytes served from
the cache, fills, fill failures, evictions and hits served while S3 was down.
The same numbers are logged every ten minutes (local blob cache).
Each instance has a cache of its own, so the numbers are those of the
instance that answered the request: instance names it by its host name
(HOSTNAME, the pod’s name on Kubernetes) and start time. Behind a load
balancer, consecutive requests may be answered by different replicas; to see
one replica’s numbers, ask it directly (its pod IP, or kubectl port-forward pod/<name>), or read its log.
On Kubernetes (see deploy/kubernetes):
emptyDirwith asizeLimitand a fixedmax_bytesbelow it — the simplest option, and it works with any number of replicas.statvfson anemptyDirreports the node’s filesystem, not the volume’ssizeLimit, so withautothe cache could outgrow the limit and get the pod evicted. The cache goes with the pod: it bridges an S3 outage only while the pod keeps running.- A generic ephemeral volume (
ephemeral.volumeClaimTemplate) on a fast StorageClass gives each pod a disk of its own whereautoworks; it is deleted with the pod, too. - A PersistentVolume lets the cache survive pod restarts during an S3
outage. A ReadWriteOnce volume needs a single replica replaced with
strategy: Recreate(thelocal-cache-persistentcomponent), or Forge as a StatefulSet with avolumeClaimTemplateper pod. KeepfsGroupChangePolicy: OnRootMismatchin the pod’s security context (the default of the manifests and the chart): with the kubelet’s default,Always, every pod start walks the whole cache to change its group.
With the Helm chart, set config.blob_cache and mount the volume with
extraVolumes/extraVolumeMounts:
config:
blob_cache:
path: /var/cache/forge/blobs # Forge creates it in the volume
max_bytes: 8589934592 # below the emptyDir sizeLimit
extraVolumes:
- name: blob-cache
emptyDir: { sizeLimit: 10Gi }
extraVolumeMounts:
- name: blob-cache
mountPath: /var/cache/forge
Backup and restore
PostgreSQL and the object store are one backup unit. Metadata references blobs by content hash, so the restored bucket must still contain every object the restored database references.
The safe order is: take the database snapshot first, and make sure no object that existed at that moment can be missing from the bucket copy. Garbage collection deletes objects after packages are deleted or retained away, so a bucket copied later could already lack them. Either:
- enable bucket versioning (or an equivalent point-in-time feature) and
restore the bucket to a point at the database snapshot’s time, and no
later than the snapshot time plus
outbox.grace_seconds(default 3600 s; garbage collection deletes an object only after that grace period, so a later bucket state can already lack objects the restored database still references). Keep noncurrent versions longer than your backup retention; or - stop every Forge instance while both copies are taken (garbage collection runs on every instance and has no separate switch; switching retention runs off in Admin → Retention does not stop it).
Objects written after the database snapshot are harmless: nothing in the restored database references them. They only use storage.
Keep the secret-encryption root key with the backup set, stored separately from it.
S3 bucket lifecycle
Large pushes and proxied downloads are uploaded with S3 multipart uploads. An upload that is interrupted (client disconnect, process restart) leaves its parts in the bucket, invisible but billed. Add a lifecycle rule that aborts incomplete multipart uploads, for example after one day:
{
"Rules": [{
"ID": "abort-incomplete-multipart-uploads",
"Status": "Enabled",
"Filter": {},
"AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 }
}]
}
(aws s3api put-bucket-lifecycle-configuration --bucket forge --lifecycle-configuration file://lifecycle.json; MinIO and Ceph RGW accept
the same rule.) Do not add expiration rules for current objects: Forge’s own
garbage collection decides when a blob may go.
To restore: stop every Forge instance, restore PostgreSQL, restore the bucket as described above, then start Forge. Practise this before you need it.
Locked out of every realm (break-glass)
The admin API and the web UI refuse (409 Conflict) to disable or delete a
realm when no enabled realm people can sign in with interactively (local or
OIDC) would remain. They cannot tell whether that realm works, though: with an
enabled OIDC realm whose IdP is down or misconfigured, local can still be
disabled, and then nobody can sign in. API tokens keep working, so restores
and pushes continue. To turn the local realm back on:
-
With an admin-scoped API token, minted beforehand with
{"scope": {"admin": true}}(keep one for this): look up the realm’s id, then enable it. The token can also go in the Basic password.curl -H "X-NuGet-ApiKey: $TOKEN" https://nuget.example.com/admin/realms curl -H "X-NuGet-ApiKey: $TOKEN" -X PATCH https://nuget.example.com/admin/realms/<id> \ -H 'content-type: application/json' -d '{"enabled": true}' -
With SQL, when there is no such token:
UPDATE auth_realm SET enabled = true WHERE kind = 'local';Every instance picks the change up within
auth.realm_cache_ttl_seconds(default 10 s); no restart is needed.
Then sign in as a local administrator and fix or disable the broken realm.
Upgrades
- 2.0.0 does not upgrade a 1.x install: it is installed fresh, with an empty database, as the 2.0.0 upgrade notes say. Upgrades are from 2.0.0 on.
- Read the changelog for each version you skip over. A release that needs something other than a rolling restart says so there.
- Migrations run automatically on the first start of a new version and only move forward. Rolling back to an older binary after a newer one has migrated the database requires restoring the backup taken before the upgrade.
- Back up (see above) before every upgrade.
Known limitations
See the “Known limitations” section of the changelog.
Helm chart
A Helm chart for running Forge on Kubernetes (1.30 or later), next to the
kustomize manifests in deploy/kubernetes. Both
set the same probes, shutdown timing and restricted security context; read
docs/deployment.md for what every Forge setting
means.
Each release publishes the chart to GitHub Packages as an OCI artifact,
oci://ghcr.io/forgepkg/charts/forge,
with the same version as the image (from 2.0.0 on; the releases up to 1.1.0
are at oci://ghcr.io/schmitch/charts/forge and ghcr.io/schmitch/forge):
helm install forge oci://ghcr.io/forgepkg/charts/forge --version 2.0.0 \
--namespace forge --create-namespace --values my-values.yaml
Which image runs
The chart’s appVersion is the release it belongs to, and the image tag
defaults to it. The release workflow also writes the digest of the image it
built and signed into the published chart (the forge/image-digest
annotation in Chart.yaml), so a published chart deploys exactly that image
(ghcr.io/forgepkg/forge):
ghcr.io/forgepkg/forge:2.0.0@sha256:…
Setting image.tag or image.digest replaces that pin. Upgrading Forge is
upgrading the chart (helm upgrade --version <new>); read the changelog
first, since migrations only go forward. Both the image and the chart are
signed with keyless cosign; the commands are in each GitHub release.
Configuration
Forge reads forge.toml plus FORGE_* environment variables. The chart maps
them like this:
| Values | Becomes |
|---|---|
config | forge.toml in a ConfigMap, loaded through FORGE_CONFIG. Any key of forge.toml.example, as YAML, nested by section. |
host | server.public_base_url (https://<host>), and the default host of the HTTPRoute, the ListenerSet listener and the Certificate. |
secretEnv | A Secret the chart creates (<fullname>-secret-env), loaded with envFrom: FORGE_DATABASE__URL, FORGE_S3__ACCESS_KEY, FORGE_ROOT_KEY (required), … |
existingSecret | An existing Secret loaded with envFrom (External Secrets, Sealed Secrets, SOPS, …); it wins over secretEnv for a key in both. Any name but the chart’s own <fullname>-secret-env. |
env, envFrom | Further variables; valueFrom reads single Secret keys. They win over the chart’s own. |
Environment variables override forge.toml. Keep secrets out of config:
the chart warns when it finds one there. The chart sets server.bind from
containerPort, and a key set to null in config removes a chart default.
A change of config or secretEnv rolls the pods through pod annotations.
Anyone who can read the pods can read those, and a plain hash of a password
can be tested against guesses, so secretEnv is hashed with a random salt
that the chart generates on install and keeps in the Secret
<fullname>-checksum-salt (looked up in the cluster on every upgrade).
Tools that render without a cluster (Argo CD, helm template) cannot look
it up and would roll the pods on every render: set secretEnvChecksumSalt
there (openssl rand -hex 16). A change inside existingSecret rolls
nothing; restart the pods yourself.
A production setup with an external PostgreSQL and S3 and a Valkey in the cluster:
host: forge.example.com
replicaCount: 2
# Only the gateway may reach Forge: every address in trusted_proxy_cidrs
# chooses the client IP Forge sees. Listing a peer turns the policy on.
networkPolicy:
from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: gateway-system # your gateway's namespace
config:
auth:
trusted_proxy_cidrs: [10.0.0.0/8] # the gateway's pod network
bootstrap_admin:
username: admin@example.com # a UPN of your organization
s3:
endpoint: https://s3.eu-central-1.amazonaws.com
region: eu-central-1
bucket: forge
force_path_style: false
existingSecret: forge-credentials # FORGE_DATABASE__URL, FORGE_S3__ACCESS_KEY,
# FORGE_S3__SECRET_KEY, FORGE_ROOT_KEY,
# FORGE_AUTH__BOOTSTRAP_ADMIN__PASSWORD
valkey:
enabled: true
podDisruptionBudget:
enabled: true
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
httpRoute:
enabled: true
parentRefs:
- name: gateway
namespace: gateway-system
sectionName: https
Secret encryption at rest is required: Forge seals the secrets it stores
(proxy passwords, realm secrets, the email credentials, its signing keys)
under a root key, FORGE_ROOT_KEY (openssl rand -base64 32). The chart
defaults to the env provider (config.crypto.root.provider) and refuses to
render unless secretEnv, existingSecret or env provides the key (or
config.crypto.root names a key file). Keep the root key with your backups:
without it the stored secrets cannot be read. Every other value is documented
in values.yaml.
Short-lived S3 credentials
Instead of FORGE_S3__ACCESS_KEY / FORGE_S3__SECRET_KEY in a Secret, Forge
can get temporary S3 credentials from the pod’s identity and refresh them
itself (s3.credentials; the IAM policy and the store-side setup are in
docs/deployment.md).
Leave the S3 keys out of secretEnv and existingSecret then: Forge refuses
to start with keys it would not use, and the chart refuses them in
secretEnv.
IRSA on EKS: annotate the service account with the role and use the AWS SDK default credentials chain, which reads what the EKS webhook injects:
s3:
credentials: default_chain
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/forge
EKS Pod Identity: s3.credentials: default_chain and a Pod Identity
association for the release’s service account (<fullname>, or
serviceAccount.name); nothing else in the values.
MinIO, Ceph RGW, or AWS without EKS: web_identity exchanges a projected
service-account token at the store’s STS. The chart mounts the token
(audience s3.webIdentity.projectedToken.audience) at
/var/run/secrets/forge/s3/token:
s3:
credentials: web_identity
webIdentity:
stsEndpoint: http://minio.storage.svc:9000 # empty: AWS STS
roleArn: arn:minio:iam:::role/<id> # the role ARN MinIO logs at startup
projectedToken:
audience: forge-s3 # MINIO_IDENTITY_OPENID_CLIENT_ID
automountServiceAccountToken stays false in every mode. The chart’s
token is a separate projected volume for the STS audience, not a Kubernetes
API token. The EKS IRSA and Pod Identity webhook also adds its own projected
token volume and environment variables. It chooses pods by the service
account’s role annotation or a Pod Identity association, and does not read
automountServiceAccountToken (amazon-eks-pod-identity-webhook,
pkg/handler/handler.go). With a restricted networkPolicy.egress, allow
the STS, or for EKS Pod Identity the node’s agent at 169.254.170.23:80.
Forge believes the X-Forwarded-For header of every peer in
trusted_proxy_cidrs, which on Kubernetes is usually the whole pod network.
Without a NetworkPolicy any pod in the cluster can call the Service with a
forged client address and get past the login lockout. List the pods that proxy to Forge (the namespace of your gateway’s
data plane or ingress controller) in networkPolicy.from, and the chart
renders the policy (networkPolicy.enabled is null by default, which means
“on as soon as from lists a peer”; false turns it off). The CNI must
enforce NetworkPolicy. The install notes warn while trusted proxies are set
without it.
Routing: HTTPRoute and ListenerSet
The chart has no Ingress; it routes through the Gateway API.
-
httpRouteattaches an HTTPRoute forhostto the Gateways inhttpRoute.parentRefs. A rule withoutbackendRefsroutes to the forge Service, so rules can addtimeouts,filtersormatcheswithout repeating it. -
listenerSetadds the release’s own HTTPS listener to a shared Gateway as a ListenerSet (Gateway API 1.5+), so the Gateway’s owner does not add a listener per application. The Gateway must allow it (spec.allowedListeners.namespaces.from). The HTTPRoute attaches to the ListenerSet automatically, and the listener terminates TLS with the Secret<fullname>-tls, whichcertificate.enabledhas cert-manager issue:host: forge.example.com httpRoute: enabled: true listenerSet: enabled: true parentRef: name: gateway namespace: gateway-system certificate: enabled: true issuerRef: name: letsencrypt
Request size limits and timeouts are implementation-specific: allow pushes of
the largest repository upload limit (set per repository in the web UI; by
default 512 MiB, and 1 GiB per image layer), and keep the gateway’s upstream
idle timeout below server.header_read_timeout_seconds (150 s; Envoy and
Istio default to one hour). Add the policy objects of your implementation (Envoy
Gateway BackendTrafficPolicy, …), or an Ingress if you need one, with
extraObjects.
extraObjects
extraObjects renders further objects with the release, given as YAML or as
strings. Both go through tpl, so they can use values and the chart’s
helpers. It is a list or a map; the keys of a map only name its entries,
which render in key order:
extraObjects:
envoy-policy:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: '{{ include "forge.fullname" . }}'
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: '{{ include "forge.fullname" . }}'
timeout:
http:
connectionIdleTimeout: 120s
Helm merges a map across values files (-f base.yaml -f production.yaml):
each file can add its own objects, and a later one replaces an entry by its
key or drops it with null (--set extraObjects.envoy-policy=null). A list
in a later file replaces the whole list.
Optional dependencies
Three single-instance backing services ship inside the chart (charts/), for
evaluation and small installs. Each is a chart dependency, off by default,
and wired into Forge when enabled:
| Dependency | Enable with | Sets |
|---|---|---|
| PostgreSQL 18 (StatefulSet) | postgresql.enabled | FORGE_DATABASE__URL |
| Valkey, cache only (Deployment) | valkey.enabled | FORGE_REDIS__URL |
| versitygw S3 gateway (StatefulSet) | versitygw.enabled | s3.endpoint, s3.bucket, FORGE_S3__ACCESS_KEY / _SECRET_KEY (static root credentials only: not with short-lived s3.credentials) |
None has backups, replication or failover. For production use a managed
PostgreSQL (or an operator such as CloudNativePG; point
FORGE_DATABASE__URL at its app Secret with env) and a real object store.
Leaving valkey.enabled off and setting FORGE_REDIS__URL (in secretEnv or
existingSecret) uses your own Valkey or Redis instead; Forge speaks the
Redis protocol to either.
Each one only accepts connections from this release’s Forge pods (a
NetworkPolicy), and every value is listed in
charts/<name>/values.yaml; set them under the dependency’s key.
Their images come from ghcr.io, like Forge’s: ghcr.io/forgepkg/forge/<name>
holds byte-for-byte copies of the Docker Hub images (library/postgres,
valkey/valkey, versity/versitygw) under the same digests, so a cluster
needs no access to Docker Hub. To pull the originals instead, set
image.registry: docker.io and image.repository to the Docker Hub name;
the pinned digest stays valid. They are pinned by digest: image.tag is
<tag>@sha256:… (the multi-arch index), which Dependabot bumps together with
the tag. A digest in image.digest replaces the pinned one; a plain
image.tag such as "16" pulls by tag.
Passwords are generated on install and kept across upgrades by looking up the
existing Secret; the PostgreSQL and S3 Secrets also survive helm uninstall
together with their volumes. Tools that render without a cluster (Argo CD,
helm template) cannot look them up and would generate new ones on every
render: set auth.password (or auth.existingSecret) there. Passwords are
pasted into Forge’s connection URLs unescaped, so use URL-safe values
(openssl rand -hex 24). Valkey restarts when valkey.auth.password
changes, through a pod annotation salted like Forge’s (the salt is kept in
the Valkey Secret): with valkey.auth.password in Argo CD, set
valkey.auth.checksumSalt too.
Evaluation install, everything in the cluster:
password=$(openssl rand -hex 12); echo "admin password: $password"
helm install forge oci://ghcr.io/forgepkg/charts/forge --version 2.0.0 \
--namespace forge-eval --create-namespace \
--set postgresql.enabled=true --set versitygw.enabled=true \
--set config.auth.bootstrap_admin.username=admin@example.com \
--set secretEnv.FORGE_AUTH__BOOTSTRAP_ADMIN__PASSWORD="$password"
kubectl -n forge-eval rollout status deployment/forge
kubectl -n forge-eval port-forward svc/forge 8421
Open http://localhost:8421 and sign in as admin@example.com with that
password. The bootstrap admin is named by a UPN and has no default name.
Telemetry
Forge exports OpenTelemetry metrics, traces and logs over OTLP/HTTP (see docs/deployment.md for what each holds). Turn on the signals you want and point them at your OpenTelemetry Collector’s OTLP/HTTP port; Forge does not speak OTLP/gRPC:
telemetry:
metrics: true
traces: true
logs: true
endpoint: http://otel-collector.observability:4318
resourceAttributes:
deployment.environment.name: production
# Keep one trace in ten (a caller's sampling decision is kept).
sampler: parentbased_traceidratio
samplerArg: "0.1"
The chart writes the signals into [telemetry] and sets the standard
OTEL_* variables: OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_SAMPLER, and
OTEL_RESOURCE_ATTRIBUTES with the pod, its namespace, node and Deployment
besides yours; the instance’s id (service.instance.id, Prometheus’
instance label) is <namespace>.<pod>.forge. For a Collector on every node (a DaemonSet), use
http://$(HOST_IP):4318: the chart defines HOST_IP from the downward API.
A backend that wants an API key on every export gets it from a Secret
holding the OTEL_EXPORTER_OTLP_HEADERS value (api-key=...):
telemetry.headersSecret: {name: otlp-auth, key: headers}. Any other
OTEL_* variable goes in env. With networkPolicy.egress set, allow the
Collector’s port. config.telemetry.per_repository: false takes the
repository names off the metrics.
Checking changes
CI lints the chart and renders it with the default values and each file in
ci/, validates the output with kubeconform, parses every rendered
forge.toml, and checks that the pod annotations hashing secrets are
salted. Locally, with helm, kubeconform and python3 on your PATH:
.github/scripts/helm-check.sh 1.33.0
Kubernetes manifests
Kustomize manifests for running Forge on Kubernetes (1.30 or later). Read docs/deployment.md first: it explains every setting these manifests set or leave to you. If you deploy with Helm, use the chart in deploy/helm/forge instead: it is published with every release and needs no copied overlay.
deploy/kubernetes/
├── base/ the Forge server: Deployment, Service, ServiceAccount, ConfigMap
├── components/
│ ├── ingress/ Ingress with TLS (ingress-nginx annotations for large, slow pushes)
│ ├── httproute/ Gateway API HTTPRoute instead of an Ingress
│ ├── network-policy/ only the ingress controller / gateway reaches Forge
│ ├── valkey/ in-cluster Valkey (shared cache), password-protected
│ ├── high-availability/ two replicas, spread over nodes, PodDisruptionBudget
│ ├── local-cache/ local blob cache on an emptyDir (faster downloads, serves through an S3 outage)
│ ├── local-cache-persistent/ the same on a PersistentVolume (one replica, survives restarts)
│ ├── postgres/ in-cluster PostgreSQL 18 (evaluation / small installs)
│ ├── s3-versitygw/ in-cluster S3 gateway on a PersistentVolume (evaluation / small installs)
│ ├── s3-workload-identity/ short-lived S3 credentials from a projected service-account token (no S3 keys)
│ ├── opentelemetry/ metrics, traces and logs over OTLP to an OpenTelemetry Collector
│ └── image-latest/ run :latest instead of the pinned release
└── overlays/
├── production/ external PostgreSQL + S3, Valkey, two replicas, Ingress, NetworkPolicy
└── evaluation/ everything in-cluster, no Ingress
Which image runs
The base pins the image to the release it belongs to:
images:
- name: ghcr.io/forgepkg/forge
newTag: 1.0.0
cargo xtask release <version> rewrites that newTag in the same commit as
the version bump, before it creates the v<version> tag, and the release
workflow refuses a tag whose base names another version. So the manifests at
a release tag always deploy that release’s image, and upgrading is a change
of one ref:
resources:
- https://github.com/schmitch/forge//deploy/kubernetes/base?ref=v1.0.0
components:
- https://github.com/schmitch/forge//deploy/kubernetes/components/valkey?ref=v1.0.0
To follow :latest instead, add components/image-latest. Nothing restarts
the pods when :latest moves; they pick up the new release when they are
recreated (kubectl rollout restart deployment/forge), and since migrations
only go forward, read the changelog before that happens.
Release images are signed; the cosign command is in each GitHub release. If
the GHCR package is not public, give the forge ServiceAccount an
imagePullSecrets entry. The postgres, valkey and s3-versitygw components
pull from ghcr.io too: ghcr.io/forgepkg/forge/<name> holds byte-for-byte
copies of the Docker Hub images under the same digests, pinned by digest
(tag@sha256:…); Dependabot proposes updates. To pull the originals
instead, add an images: entry to your overlay (name: ghcr.io/forgepkg/forge/valkey, newName: valkey/valkey); the digest stays
valid.
Configuration and secrets
Forge reads its configuration from FORGE_* environment variables
(docs/deployment.md lists them):
- Non-secret settings go into the
forge-configConfigMap. Overlays add to it with aconfigMapGeneratorusingbehavior: merge; set at leastFORGE_SERVER__PUBLIC_BASE_URL. - Secrets come from a Secret named
forge-env, loaded withenvFrom. The overlays generate it from a git-ignoredforge.envnext to theirkustomization.yaml(copyforge.env.example). If you manage secrets elsewhere (External Secrets, Sealed Secrets, SOPS), create a Secret namedforge-envand drop thesecretGenerator.
The components that run a backing service read their password from the same
Secret: POSTGRES_PASSWORD (postgres), VALKEY_PASSWORD (valkey), and the
gateway uses Forge’s own FORGE_S3__ACCESS_KEY/FORGE_S3__SECRET_KEY
(s3-versitygw). The Postgres and Valkey passwords are pasted into a URL, so
generate URL-safe values: openssl rand -hex 24.
Secret encryption at rest is required: the base sets
FORGE_CRYPTO__ROOT__PROVIDER=env, and forge.env holds the root key,
FORGE_ROOT_KEY (openssl rand -base64 32; both examples have it). Forge
refuses to start without it. Keep the root key with your backups: without it
the stored secrets cannot be read.
Quick start (evaluation)
cd deploy/kubernetes/overlays/evaluation
cp forge.env.example forge.env # fill in the passwords
kubectl apply -k .
kubectl -n forge-eval rollout status deployment/forge
kubectl -n forge-eval port-forward svc/forge 8421
Open http://localhost:8421 and sign in as the bootstrap admin (FORGE_AUTH__BOOTSTRAP_ADMIN__USERNAME) with its password.
The evaluation overlay has no backups and single instances of everything.
Production
Copy overlays/production into your deployment repository, switch its
resources and components to the remote, tag-pinned URLs shown above, and
adjust:
- the host in
FORGE_SERVER__PUBLIC_BASE_URLand the Ingress patch, and theforge-tlscertificate Secret (or a cert-manager annotation); FORGE_AUTH__TRUSTED_PROXY_CIDRS: the pod network of your ingress controller or gateway, so Forge sees the real client IP;- the namespace in the NetworkPolicy patch: that of your ingress controller (or your gateway’s data plane), the only pods that may reach Forge;
- the S3 endpoint, region and bucket (on AWS, create the bucket beforehand;
docs/deployment.md lists the S3 permissions Forge needs), and
forge.env.
Points that matter on Kubernetes:
- Probes. A startup probe gives migrations up to five minutes on the first start of a new version before the liveness probe takes over.
- Rolling updates.
maxUnavailable: 0, a 10 spreStopsleep so the Service stops routing before Forge stops accepting connections, and a 300 s termination grace period for in-flight downloads. Do not roll back to an older image once a newer one has migrated the database. - Temp space.
/tmpis anemptyDir(the root filesystem is read-only). Pushes and first-time proxy downloads are staged there, up toserver.max_staging_bytes(4 GiB by default); the volume’ssizeLimitis 6 GiB. Raise both together. - Two or more replicas need Valkey or Redis (
components/valkeyor your ownFORGE_REDIS__URL), and replicas ×database.max_connections(20) must fit PostgreSQL’smax_connections. - Reverse proxy. Allow 1 GiB request bodies without buffering (image layers are up to 1 GiB and package pushes up to 512 MiB unless a repository’s upload limit, set in the web UI, says otherwise), and keep the proxy’s upstream idle timeout below 150 s (Envoy and Istio default to one hour; see the httproute component).
- Who reaches Forge. Forge believes the
X-Forwarded-Forof every peer inFORGE_AUTH__TRUSTED_PROXY_CIDRS, usually the whole pod network, so any other pod could call the Service with a forged client address and get past the login lockout.components/network-policyadmits only the ingress controller or gateway; use it whenever you set trusted proxies (the CNI must enforce NetworkPolicy). - Pod security. Every workload here satisfies the
restrictedPod Security Standard; the overlays’ namespaces enforce it.
Short-lived S3 credentials
Instead of FORGE_S3__ACCESS_KEY / FORGE_S3__SECRET_KEY in forge-env,
Forge can get temporary S3 credentials from the pod’s identity and refresh
them itself (FORGE_S3__CREDENTIALS; the IAM policy and the store-side setup
are in docs/deployment.md).
Take the keys out of forge-env then: Forge refuses to start with keys it
would not use.
-
IRSA / EKS Pod Identity (EKS): set
FORGE_S3__CREDENTIALS=default_chaininforge-config. For IRSA, annotate the ServiceAccount; for Pod Identity, create an association for theforgeServiceAccount instead. The EKS webhook injects its own token volume andAWS_*variables. It does not readautomountServiceAccountToken, which staysfalse.patches: - patch: |- apiVersion: v1 kind: ServiceAccount metadata: name: forge annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/forge -
MinIO, Ceph RGW, AWS without EKS:
components/s3-workload-identitymounts a projected service-account token for the STS audience and setsFORGE_S3__CREDENTIALS=web_identity. AddFORGE_S3__WEB_IDENTITY__STS_ENDPOINT(MinIO / Ceph RGW) andFORGE_S3__WEB_IDENTITY__ROLE_ARNtoforge-config, and patch the token’s audience for anything but AWS (the component shows how).
With components/network-policy and your own egress rules, allow the STS
(for EKS Pod Identity, the node’s agent at 169.254.170.23:80). The
s3-versitygw component keeps its static root credentials.
Local blob cache
Forge can keep a copy of the blobs it serves on a local disk: hot downloads
skip S3, and cached packages keep resolving while S3 is unreachable (see
docs/deployment.md). The
components mount a volume at /var/cache/forge and set
FORGE_BLOB_CACHE__PATH to /var/cache/forge/blobs, a directory Forge creates
in it (the cache refuses one its user does not own), and __MAX_BYTES where
needed. The existing
/readyz readiness probe needs no change: with the cache, an S3 outage
reports degraded but keeps the pod in the Service.
components/local-cache: anemptyDir(sizeLimit: 10Gi) with a fixed 8 GiB cache.statvfson anemptyDirsees the node’s disk, not thesizeLimit, so the cache must not size itself (auto) there: raise both numbers together. Works with any number of replicas; the cache goes with the pod, so it bridges an S3 outage only while the pod keeps running.components/local-cache-persistent: a 20 GiB ReadWriteOnce PersistentVolumeClaim, so the cache survives restarts, and an instance restarted during an S3 outage serves what it had. One pod can mount the volume, so it runs one replica withstrategy: Recreate(a short gap on every rollout); it cannot be combined withcomponents/high-availability. The cache sizes itself from the volume (auto). The base’sfsGroupChangePolicy: OnRootMismatchkeeps the kubelet from walking the volume at each pod start, which would make the cache directory group-writable (Forge restores it, with a warning) and take long on a large cache.
For a dedicated disk per pod (e.g. local NVMe) with any number of replicas,
use components/local-cache and swap its volume for a generic ephemeral
volume, where auto works because the cache sees the volume itself:
patches:
- patch: |-
apiVersion: apps/v1
kind: Deployment
metadata:
name: forge
spec:
template:
spec:
containers:
- name: forge
env:
- name: FORGE_BLOB_CACHE__MAX_BYTES
value: auto
volumes:
- name: blob-cache
emptyDir: null
ephemeral:
volumeClaimTemplate:
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-nvme
resources: { requests: { storage: 50Gi } }
Telemetry
components/opentelemetry turns on Forge’s OpenTelemetry export: metrics,
traces and logs over OTLP/HTTP to an OpenTelemetry Collector (see
docs/deployment.md for what is
exported). It sets FORGE_TELEMETRY__METRICS, __TRACES and __LOGS, the
Collector’s address in OTEL_EXPORTER_OTLP_ENDPOINT, and resource attributes
naming the pod, its namespace and its node. The address,
http://otel-collector.observability.svc:4318, is a placeholder: patch it to
your Collector’s OTLP/HTTP endpoint, or to http://$(HOST_IP):4318 for one
running on every node. Any other OTEL_* variable (OTEL_TRACES_SAMPLER,
OTEL_EXPORTER_OTLP_HEADERS from a Secret for a vendor’s API key) goes in
the same way.
patches:
- patch: |-
apiVersion: apps/v1
kind: Deployment
metadata:
name: forge
spec:
template:
spec:
containers:
- name: forge
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://otel-collector.monitoring.svc:4318
- name: OTEL_TRACES_SAMPLER
value: parentbased_traceidratio
- name: OTEL_TRACES_SAMPLER_ARG
value: "0.1"
With components/network-policy and your own egress rules, allow the
Collector’s port.
Checking changes
CI renders every overlay, and the base with each component, validates the
output with kubeconform, and checks that an overlay with trusted proxies has
the NetworkPolicy. Locally, with kustomize and kubeconform on your PATH:
.github/scripts/kustomize-check.sh 1.33.0
Monitoring
An example OpenTelemetry Collector configuration, a Grafana dashboard and example Prometheus alerting rules for Forge’s telemetry. docs/deployment.md → Telemetry describes how Forge exports metrics, traces and logs over OTLP, and every metric.
| File | What it is |
|---|---|
otel-collector.yaml | An OpenTelemetry Collector (otelcol-contrib) receiving Forge’s OTLP/HTTP on port 4318: the metrics for Prometheus to scrape on port 8889, the traces and logs sent on over OTLP (placeholders for Tempo and Loki). |
grafana-dashboard.json | A Grafana dashboard: requests, proxies, storage, cache and database, workers, retention, authentication, inventory. Import it (Dashboards → New → Import) and pick your Prometheus data source; the job, instance and repository variables narrow it down. |
alerts.yaml | Example alerting rules: a stalled, failing or restarting worker, S3 down, a high 5xx rate, a growing outbox backlog, upstream errors and rate limits per proxy, an exhausted connection pool, a lost Redis backplane, nearly full staging space. |
The dashboard and the rules use the names Prometheus gives OpenTelemetry
metrics: dots become underscores and the unit a suffix
(http.server.request.duration is http_server_request_duration_seconds,
forge.proxy.lookups is forge_proxy_lookups_total), attributes become
labels (forge.repository is forge_repository), and job and instance
come from the resource’s service.name and service.instance.id. That is
what the Collector’s prometheus and prometheusremotewrite exporters
write, and what Prometheus’ own OTLP receiver (--web.enable-otlp-receiver,
which Forge can export to directly) stores by default.
Scrape the Collector with honor_labels: true: otherwise Prometheus
replaces each series’ job and instance with the Collector’s own. With
the prometheus-operator, a ServiceMonitor or PodMonitor for the Collector’s
port 8889 with honorLabels: true.
To load the rules with the prometheus-operator, wrap them in a
PrometheusRule (labeled for your Prometheus’s ruleSelector):
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: forge
spec:
groups: [] # the `groups` of alerts.yaml
The thresholds are starting points; check edits with
promtool check rules deploy/monitoring/alerts.yaml, and the Collector’s
configuration with
otelcol-contrib validate --config deploy/monitoring/otel-collector.yaml.
Retention size budgets
Hosted and proxy package repositories, raw file repositories, and Docker
repositories can set a logical size budget in their retention policy. Configure
it through the repository’s Retention screen or
PUT /admin/repositories/<name>/retention; it is stored with the policy, not
in forge.toml. Group repositories do not have their own retention policy.
Add size_budget to the policy envelope:
{
"version": 1,
"size_budget": {
"max_bytes": 536870912000,
"target_bytes": 483183820800
},
"rules": []
}
max_bytes must be greater than zero. If target_bytes is omitted, it
defaults to 90% of max_bytes; it cannot be greater than max_bytes. The
budget may be the policy’s only content, so rules can be empty when a budget
is set. Without a budget, at least one rule is required. enabled and
dry_run_only apply to the budget as they do to retention rules.
The budget measures logical repository size, not physical object-store usage: package and raw-file size is the sum of their asset/file sizes. Docker size is the sum of distinct blobs linked by the repository’s manifests. A blob shared between repositories counts toward each repository’s logical size; evicting it from one repository may not free physical storage while another repository still references it.
Rules are applied first. If content still exceeds max_bytes, the budget
evicts unprotected content in least-recently-used order until it reaches
target_bytes or has no eligible content left:
- Packages: versions are ordered by last download, or publication/first-seen time if never downloaded. Hosted repositories retain each package’s newest listed version; proxy versions are evicted from the cache and can be fetched again.
- Raw files: ordered by the later of their last download and their deployment, so a file deployed again counts as used then.
- Docker: untagged manifests go first, ordered by last pull, or push time if
never pulled. Then tagged manifests, ordered by the later of their last pull
and their newest tag’s push (as
tag_not_pulled_for_daysjudges a tag): the budget deletes all of a manifest’s tags together, and the manifest with them. It never deletes a tag whose manifest would stay, since that frees nothing: not an alias of a manifest a protected tag also names, and not the tags of a manifest that holds its image’s last tags. An image always keeps a tag. Freed size counts only blobs no remaining manifest in the repository links.
The protections of ordinary rules also apply to budget eviction, including keep-N rules, protected tags, and Docker index children and referrers. If protected content alone is larger than the target, eligible content is still removed and the run reports that it could not reach the target.
Retention preview reports budget decisions with reason size_budget and
includes the size before and projected size after. The retention API reports
the repository’s current logical size, and the run summary records the
before/after sizes, target, and whether it was reached. The Retention
screen edits the maximum and the target in GB or GiB, displays the current size
against the configured maximum, and warns when the last run could not reach
its target.
Importing from Nexus
Forge replaces Nexus Repository 3. Logins carry over with credential forwarding, and a proxy repository refills from its upstream. The content of Nexus’ hosted repositories (your internal packages and images) is copied by an import, one repository at a time: point a Forge hosted repository at a Nexus hosted repository of the same format, and Forge copies its content.
| Format | What is copied |
|---|---|
| NuGet | the .nupkg of every version |
| Docker | every tag with everything it references: config and layers, the child manifests of a multi-platform index; signatures and SBOMs (cosign’s sha256-<digest>.sig tags, and referrers Nexus holds by digest) |
apt follows once Forge serves apt repositories (#176). Proxy repositories are not imported (they refill from their upstream), nor are groups (recreate them as configuration), Nexus users, roles and privileges (credential forwarding covers sign-in) or cleanup policies (see retention).
Running an import
In the web UI, open the hosted repository under Admin → Repositories and choose Import from Nexus. Or with the admin API:
curl -u admin@example.com:<admin-scoped token> -X POST \
https://forge.example.com/admin/repositories/nuget-internal/imports \
-H 'content-type: application/json' \
-d '{
"url": "https://nexus.example.com",
"repository": "nuget-hosted",
"username": "migrator",
"password": "…",
"dry_run": true
}'
| Field | |
|---|---|
url | The Nexus server, with its context path if it has one. |
repository | The Nexus hosted repository. |
username, password | A Nexus user that may browse and read it (the privileges nx-repository-view-<format>-<name>-browse and -read); both, or neither for anonymous access. |
registry_url | Docker only: the repository’s registry API. Default <url>/repository/<name>, Nexus’ path route, which serves reads on Nexus’ own port; otherwise its connector, e.g. http://nexus.example.com:8082. |
on_conflict | A version Forge already has with other bytes: skip it and report it (default), overwrite it with Nexus’ bytes, or fail the import. |
concurrency | Items copied at a time, 1–16 (default 4), so Nexus is not overwhelmed. |
dry_run | Decide every item and write nothing. |
Before an import is accepted, Forge checks the source (the preflight): that Nexus answers its status endpoint (its version is recorded), that the repository exists and is hosted and of the Forge repository’s format, and for Docker that its registry API answers. A failed check answers 400 (502 when Nexus cannot be reached) with what is wrong, and nothing is stored. A repository runs one import at a time.
A dry run reports how many components and assets the Nexus repository holds and their size, what would be copied and how much, and the conflicts with versions Forge already has, without writing anything. It downloads nothing whose SHA-256 Nexus lists.
What an import does
- Scan. Nexus’ component listing is walked page by page
(
GET /service/rest/v1/components). Each page’s components are recorded as the import’s items, and the page’scontinuationTokenas its cursor, in one transaction: an import resumes after a restart, a failure or a cancel at the page it had not recorded. For Docker, the asset listing follows, for the manifests Nexus holds by digest only. - Copy. The items are copied,
concurrencyat a time:- A package is streamed from Nexus into staging, checked against the size and checksums Nexus lists (SHA-256 and SHA-512; SHA-1 where an older Nexus lists nothing else), and written through the same path as a push: format validation, the package selectors of the administrator who started the import, the repository’s upload limit, deduplication in the content-addressed store, and the audit trail. A version Forge has with the same SHA-256 is skipped without downloading it.
- A Docker tag is copied through the Nexus repository’s registry API,
like
crane copy: digests are kept, and the manifests go through the same checks as adocker push(every blob and child present, the repository’s immutable tags). A layer the repository holds already is not downloaded again. A referrer (a signature or SBOM with asubject) Nexus holds by digest is copied once its subject is in Forge. An untagged manifest that refers to nothing is not copied, and a Docker schema 1 manifest is refused as on a push (its tag fails).
- Metadata. What Nexus records (in recent Nexus 3 versions) is kept:
blobCreatedbecomes the version’s creation time (for a Docker tag,lastModified, when the tag last moved), which retention’solder_than_daysreads;lastDownloadedbecomes its last download (not_downloaded_for_days); the uploader is recorded with the item and in the audit trail. Download counts are not in Nexus’ API, sokeep_n_most_downloadedstarts from zero.
Every imported version is audited as an import: a forge::audit log line
with action = "import", the administrator, the source and the uploader.
Progress, cancel, resume, retry
GET /admin/repositories/{name}/imports (and the web UI, which refreshes
while an import runs) shows each import’s state (queued, running,
completed, failed, cancelled), its phase, and its items by outcome:
imported, replaced, identical, conflict, failed, planned (dry
run), pending, with the bytes copied. …/imports/{id}/items?state=conflict
lists the items of one outcome, with the reason of each conflict or failure.
POST …/imports/{id}/cancelstops an import after the items in flight.POST …/imports/{id}/resumecontinues a cancelled or failed import where it stopped; with{"retry_failed": true}(or"items": [<ids>]) its failed items are copied again, on their own.DELETE …/imports/{id}deletes a stopped import with its items.
An import fails (and can be resumed) when Nexus’ listing cannot be read after
a few attempts, when 25 items in a row fail (Nexus went away, the
credentials stopped working), on a conflict with on_conflict: fail, or when
the administrator who started it is no longer one.
Running an import again later copies only what is new in Nexus: everything
Forge already has is identical. A cutover is therefore an import ahead of
time, then a final one just before DNS or the clients switch to Forge.
Credentials and workers
The Nexus password is stored sealed with the secret encryption
([crypto]), sent only to the Nexus origin (and the registry’s), and
deleted when the import completes. A cancelled or failed import keeps it, to
be resumed; retrying the failed items of a completed import asks for it
again.
Every instance runs the import worker; an import runs on one instance at a time, which holds it with a lease in Postgres that it renews while it works. When that instance dies, another takes the import over once the lease has expired (two minutes) and continues it.
Docker / OCI registry
Forge serves container images and other OCI artifacts (Helm charts, cosign
signatures, SBOMs) under /v2/, the way Nexus’ docker-hosted, docker-proxy
and docker-group repositories do. It speaks the
OCI Distribution Spec
v1.1 that Docker, Podman, containerd, crane, skopeo, oras, Helm and cosign
use. CI runs the spec’s conformance suite against Forge on every change.
Images live in repositories of format: "docker", a format plugin built
into Forge like NuGet and raw (GET /admin/formats lists it, with storage: images). It is an images format: the
plugin, crates/formats/docker, speaks the
protocol, and Forge’s image store keeps the images. They come in the same
three kinds as NuGet repositories:
- hosted: push and pull your own images;
- proxy: a pull-through cache of an upstream registry: Docker Hub, GHCR, Quay, MCR, or any registry that speaks the distribution spec;
- group: one name over several members, resolved by priority.
Layers, configs and manifests are blobs in the content-addressed store. A layer shared by many images, or by images in several repositories, is stored once.
Names
Docker clients always talk to https://<host>/v2/. Forge finds the
repository in the image name:
- By path (always available): the first component of the name is the
Forge repository, the rest the image.
forge.example.com/docker-hosted/team/app:1.2is the imageteam/appin the repositorydocker-hosted. - By host (optional): a repository with a
host("host": "docker.example.com") also answers at the root of that host, so its images keep short names:docker.example.com/team/app:1.2. Point the host name at Forge, include it in the TLS certificate, keepHostintact in the reverse proxy, and add it toserver.allowed_hostsif a trusted-header realm names its clients.
A docker repository’s name is part of every image name, so it follows the
spec’s name grammar: lowercase letters and digits, separated by ., _,
__ or - (for example docker-hosted). Image names follow the same
grammar, with / between components.
Nexus-style per-repository connector ports are not supported: they are awkward behind an Ingress.
Logging in
Forge implements Docker’s Bearer token service (Nexus’ “Docker Bearer Token
Realm”): GET /v2/ answers 401 with a challenge naming the token endpoint
/v2/token, the client sends its credentials there once and receives a
short-lived token (docker.token_ttl_seconds, 5 minutes by default) for the
repositories and actions it asked for.
docker login forge.example.com -u <username> # password: an API token
docker pull forge.example.com/docker-hosted/team/app:1.2
docker push forge.example.com/docker-hosted/team/app:1.3
- API tokens (created on the Tokens page, or
POST /admin/me/tokens) are the password to use. A token’s repository scope applies. - Passwords work where Basic passwords are allowed at all: the
allow_password_basicsetting (Admin → Settings), or an enabled credential-forwarding realm (so users of an existing Nexus or LDAP sign in with their upstream password). The password goes to the token endpoint only; the registry itself sees the short-lived token. - CI workloads with a workload-identity realm pass their OIDC token, or
the workload key it was exchanged for, as the password with the username
oidc(trusted publishing); their pushes and deletes are recorded in the trusted-publishing ledger. - Anonymous pulls work for repositories that allow anonymous read (with the anonymous mode on, Admin → Settings): clients get an anonymous token without logging in.
- A browser session never pushes, even an administrator’s.
A registry token grants only what the caller may do when it is issued:
pull needs read on the repository, push needs write, and delete
needs delete; content selectors match image names (team/*). One token
request may name at most 32 repositories (clients ask for one to three); a
request naming more is refused with 400 UNSUPPORTED. Every
request is authorized again against the current permissions, and the user
and API token behind a registry token must still be active; a workload’s
realm must still be enabled with the configuration it was verified under,
and a token got with a workload key stands only while that key does
(checked at least every 10 seconds). A token got with a CI token directly
does not depend on any key that CI token was exchanged for. Either kind
expires no later than the key or CI token it was got with. Registry tokens
are accepted under /v2 only.
Registry tokens are signed with docker.token_secret when set, otherwise
with a key Forge generates once and keeps in the database (sealed by the
secret encryption, and re-sealed by DEK rotation like the other secrets), so every instance accepts every
instance’s tokens. An instance replaces the stored key only when it can
never be opened (a database or key-store error fails startup instead); the
running instances pick the new key up within 30 seconds and keep accepting
tokens of the previous one.
Hosted repositories
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/repositories \
-d '{"name":"docker-hosted","kind":"hosted","format":"docker"}'
All of the spec’s push paths work: monolithic and chunked (resumable)
uploads, cross-repository mounts (?mount=&from=, needs read on the source
image), pushes by tag or digest, ?tag= parameters on a push by digest, and
deletes of tags, manifests and blobs. Manifests may be Docker v2 schema 2
manifests and manifest lists, OCI image manifests and indexes, or artifacts
with any artifactType; Docker schema 1 is refused.
- Immutable tags:
"immutable_tags": truerefuses to move an existing tag to another manifest (re-pushing the same manifest is fine). With"latest_mutable": true,latestmay still move. - A manifest must name blobs the image has (
MANIFEST_BLOB_UNKNOWNotherwise), except non-distributable layers and the OCI empty config ({}), which Forge stores itself when it is missing. - A blob a manifest of the image references cannot be deleted (
409): delete the manifest first. DELETE /v2/<name>/manifests/<tag>deletes the tag alone: the manifest stays, under its other tags or untagged. ADELETEby digest deletes the manifest and every tag that names it.- Uploads are limited per blob by the repository’s upload limit (Maximum
upload size in the web UI,
max_upload_bytesin the admin API; 1 GiB when unset, or what an explicitserver.max_staging_bytesbelow that holds) and staged on disk like package pushes (seeserver.max_staging_bytes, which also caps the limit). Upload sessions are shared by all instances; an abandoned one expires afterdocker.upload_ttl_seconds.
The images of a Nexus docker-hosted repository move over with an import, digests, multi-platform indexes, signatures and SBOMs included (nexus-import.md).
Signatures, SBOMs and attestations
The referrers API (GET /v2/<name>/referrers/<digest>, with the
artifactType filter) lists every manifest whose subject is the digest,
with its annotations. cosign 3 stores signatures and attestations there by
default; older cosign versions use tags (sha256-<digest>.sig), which work
as well.
Helm charts
Helm (3.8 and later) stores charts in OCI registries, and Forge serves them
from docker repositories like any other artifact: a chart is a manifest
whose config is the chart’s Chart.yaml (application/vnd.cncf.helm.config.v1+json)
and whose layer is the packaged chart. helm push tags it with the chart’s
version. Forge serves charts this way only; it has no classic chart
repositories (index.yaml, helm repo add).
helm registry login forge.example.com --username <username> # password: an API token
helm push mychart-1.2.3.tgz oci://forge.example.com/docker-hosted/charts
helm install myrelease oci://forge.example.com/docker-hosted/charts/mychart --version 1.2.3
helm pull oci://forge.example.com/docker-group/charts/mychart --version '~1.2'
The chart’s name is the last component of the image name
(docker-hosted/charts/mychart); content selectors and retention match it
like any image name. Proxies of a registry that serves charts (Docker Hub’s
registry-1.docker.io/bitnamicharts, GHCR) and groups work as for images.
helm registry login keeps its own login, so log in with Helm even when
docker login already did.
Proxy repositories
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/repositories -d '{
"name": "docker-hub", "kind": "proxy", "format": "docker",
"remote_url": "https://registry-1.docker.io",
"upstream_username": "<docker hub user>", "upstream_password": "<docker hub token>"
}'
docker pull forge.example.com/docker-hub/nginx:1.27
remote_urlis the registry’s base URL:https://registry-1.docker.io(notdocker.io),https://ghcr.io,https://quay.io,https://mcr.microsoft.com.- Docker Hub’s one-component names live in
library/:nginxandlibrary/nginxare the same image, cached once. - Upstream credentials are optional; give Docker Hub ones to raise its pull
rate limit. They are sent to the upstream and to the token service its
401challenge names (Docker Hub’sauth.docker.io), over https when the upstream is https. They are sealed at rest. - Blobs and manifests pulled by digest are verified against their digest and
cached for good. A tag is served from the cache for
metadata_ttl_seconds(default 600) and then re-validated with aHEAD, which Docker Hub does not count against the rate limit. While the upstream is unreachable or rate limits, the cached tag is served; an upstream “not found” is remembered fornot_found_ttl_seconds(default 60). - Multi-arch indexes are cached as they are; each platform’s manifest and
layers are fetched when a client asks for them. Tag lists and referrers
come from the upstream, or from the cache while it is unreachable. A tag
list follows the upstream’s own pages (
Link: rel="next", up to 10 requests) to fill the client’s page, and links to the next page when the upstream has more; an upstream that refuses a largenis asked without one. - An offline proxy (
"online": false) serves only what it cached. Pointing a proxy at anotherremote_urlempties its cache; a fetch from the previous upstream still running then caches nothing, and its client is asked to retry (503). - To ask the upstream again before those TTLs run out (a tag moved, an image
published after the proxy answered “not found”), invalidate the proxy’s
cache: Invalidate cache on its page in the web UI, or
curl … /admin/repositories/docker-hub/proxy-cache/invalidate -d '{"package": "library/nginx"}'(with-d '{}': every image). The proxy forgets its “not found” answers, on every instance, and re-validates the image’s tags on their next pull; blobs and manifests stay cached. - A proxy is read-only (
405on pushes).
Group repositories
curl … /admin/repositories -d '{"name":"docker","kind":"group","format":"docker",
"writable_member":"docker-hosted",
"reserved_prefixes":["acme/"]}'
curl … /admin/repositories/docker/members -d '{"member":"docker-hosted","priority":10}'
curl … /admin/repositories/docker/members -d '{"member":"docker-hub","priority":5}'
docker pull forge.example.com/docker/nginx:1.27
- A pull is served by the highest-priority member that has the image and the caller may read; a failing member (an unreachable upstream) is skipped while another answers. Tag lists and referrers are merged.
- Members must be docker repositories. An offline hosted or nested group member is skipped; an offline proxy member still serves its cache.
- Pushes to a group go to its
writable_member, a hosted member (Nexus Pro’s group deployment). Without one, pushes are refused. The caller needs read on the group and write on that member. The group keeps the member when it is renamed; once it is deleted, the group has no writable member until one is set again. - Dependency-confusion guard: an image under one of the group’s
reserved_prefixes(namespaces such asacme/) is never served by a proxy member, and withprefer_hosted(on by default) neither is an image a hosted member already has. Flag trusted members asownerwhen hosted members have different writers, as for NuGet groups.
Browsing
The web UI lists a docker repository’s images, and per image its tags and
manifests with digests, sizes, platforms, referrers, push and pull times, and
ready-to-paste docker login / docker pull commands. The same data is in
the browse API: GET /browse/images?repo=<name> and
GET /browse/images/<repo>/<image>. An offline hosted or group repository
answers 503 there, as it does to pulls and pushes.
An image of Helm charts shows its charts instead: the versions, the
helm install and helm pull commands of the selected one (the newest
release by default), its Chart.yaml facts (version, appVersion,
description, home, sources, maintainers, dependencies), its README and its
default values. The browse API reads them from the chart’s blobs:
GET /browse/chart?repo=<name>&image=<image>&digest=<manifest digest>. A
proxy stores a chart’s blobs when it is first pulled, so until then it shows
the versions and commands only. A README or values.yaml over 512 KiB is cut.
Deleting from the web UI
In a hosted repository, an administrator deletes from the image page: the whole image (Delete image…), a tag, a manifest, or a chart’s version (its manifest). The admin API does the same with a session or an admin-scoped token:
# A tag: its manifest stays, pullable by digest.
curl -u admin@example.com:<admin-token> -X DELETE \
'https://forge.example.com/admin/repositories/docker-hosted/images?image=team/app&reference=1.0'
# {"manifests": 0, "tags": 1}
# A manifest, with its tags: reference=sha256:<hex>. The whole image: no
# reference (an empty `reference=` is refused with 400).
A manifest goes with its tags and with what only it kept: the manifests an index lists and the referrers (signatures, SBOMs, attestations) of it, unless a tag or another manifest of the image still needs them, as image retention decides. A manifest an index lists is deleted when asked for, even though pulling that platform through the index then fails. The answer counts the manifests and tags deleted (404 when nothing matched). A hosted repository is deleted only once it holds no image, so emptying one is deleting its images.
The catalog API (/v2/_catalog) is not supported.
Retention and garbage collection
A manifest keeps its config and layers alive through the image’s blob links.
Deleting a tag only untags its manifest; deleting a manifest releases the
manifest, and the registry sweeper (every docker.sweep_interval_seconds)
releases the blobs no manifest of the image references any more once they
have been unused for docker.unreferenced_blob_ttl_seconds (24 h by
default: the time a client has between pushing layers and pushing their
manifest). Pushing or mounting a blob the image already has counts as a
use, and so does a pull or HEAD of it, recorded at most every tenth of
that TTL (at most hourly). The blob outbox deletes a blob after its grace period once no
image of any repository links it, so deleting an image never removes a layer
another image uses.
A docker repository’s retention policy (PUT /admin/repositories/<name>/retention) has image rules:
{"policy": {
"version": 1,
"rules": [
{"id": "keep-10", "image_prefix": "team/", "keep_last_n_tags": 10,
"protected_tags": ["latest", "stable"]},
{"id": "stale-tags", "tag_older_than_days": 90, "tag_not_pulled_for_days": 30},
{"id": "untagged", "delete_untagged_after_days": 7}
]
}}
- Tag rules delete tags. Per image (only those under
image_prefix, when set), thekeep_last_n_tagsmost recently pushed tags and theprotected_tagsstay; any other tag goes when every trigger the rule sets holds (tag_older_than_dayssince its push,tag_not_pulled_for_days). AGETor aHEADof a tag’s manifest counts as a pull, so a tag that containerd (Kubernetes, nerdctl, BuildKit) only resolves stays pulled. A rule with onlykeep_last_n_tagskeeps N tags per image. - Tags of one digest: a tag rule deletes the tag alone. Its manifest
stays while another tag names it, and is untagged (for
delete_untagged_after_days) from when its last tag goes. A pull is recorded per manifest, so pullinglatestcounts for every tag of its digest: whilelatestis pulled,tag_not_pulled_for_daysdeletes no other tag of that digest, whereastag_older_than_daysandkeep_last_n_tagsjudge each tag by its own push. Every tag takes a place among thekeep_last_n_tags; name aliases such aslatestinprotected_tagsso that they do not. - Untagged rules (
delete_untagged_after_days) delete manifests without a tag for that long, except what something kept refers to: the platforms of a kept index, and the signatures, SBOMs and attestations (referrers) of a kept manifest. dry_run_only,enabled,min_run_interval_seconds, preview, run and audit work as for NuGet repositories; audit rows carry the image aspackage_nameand the tag or digest asversion. Group repositories have no policy of their own. The repository’s Retention screen in the web UI (Repositories → the repository → Retention) edits the rules, previews a draft before it is saved (POST …/retention/previewwith{"policy": {…}}), runs the policy and lists the audit log.- A run applies its decisions image by image, in batches of up to 100. As for NuGet repositories, a batch first checks that the policy is still the one the run evaluated and that the run still holds its lease; otherwise the run stops, so disabling the policy, switching it to dry run or editing it takes effect at once. The batch then evaluates the image’s current tags and manifests again and skips what they no longer warrant: a tag pulled or pushed since, a tag that newer tags deleted meanwhile now keep, or a signature whose image was tagged again.
An image retention policy may also include a size_budget; it evicts
least-recently used unprotected manifests and tags after its ordinary rules.
See Retention size budgets for configuration and logical size
accounting.
Configuration
[docker]
max_blob_bytes = 1073741824 # (default: 1 GiB, or what an explicit server.max_staging_bytes holds) largest layer a proxy downloads
token_ttl_seconds = 300 # (default: 300) registry token lifetime, 60-3600
# token_secret = "…" # (default: a generated key kept in the database)
upload_ttl_seconds = 86400 # (default: 24 h) abandoned upload sessions expire
unreferenced_blob_ttl_seconds = 86400 # (default: 24 h, at least 1 h)
sweep_interval_seconds = 300 # (default: 300)
Pushes follow the repository’s upload limit, not max_blob_bytes.
Raising max_blob_bytes raises the default server.max_staging_bytes
(four times the largest item) as well; size the staging volume for it. The
reverse proxy must allow request bodies of the largest repository upload
limit on /v2 (docker sends a whole layer in one request, 1 GiB by default)
and stream them without buffering; see
deployment.
Known limitations
- A proxy downloads a blob completely (and verifies it) before it answers the client, so a client waits for a large layer’s first fetch. Concurrent pulls of one layer share a download per instance.
- An upload of several chunks is assembled into a new blob when it finishes (the chunks are read back from the object store); one-chunk uploads, what docker, podman and containerd send, are stored once.
- Mounting a blob without
fromsucceeds only when the image already has it. - The registry is served over plain http only behind a TLS-terminating proxy
or for
localhost; Docker refuses other plain-http registries unless they are listed as insecure registries in the daemon.
npm registries
Forge serves npm packages to npm, pnpm, Yarn (1 and 2+) and Bun: publish,
install, dist-tags, deprecations, unpublishing, search, npm audit and the
logins. Repositories of format: "npm" are served at
https://<host>/repository/<repository>/, in the usual three kinds:
- hosted: publish and install your own packages;
- proxy: a cache of another registry,
https://registry.npmjs.orgor a private one; - group: one registry URL over several members, which keeps your scopes
internal and passes
npm auditon to the public registry.
npm is a format plugin built into Forge (crates/formats/npm,
format-plugins.md): tarballs are blobs in the
content-addressed store, and repositories, access levels, package selectors,
retention and the web UI are the same as for every other format.
Contents
- Repositories
- Connecting a client
- Logging in
- Publishing
- Dist-tags, deprecations and unpublishing
- Installing and searching
- Proxy repositories
- Group repositories
- Audits
- Permissions
- Retention
- Limits
- Limitations
Repositories
Create them in the web UI (Admin → Repositories → New repository, format npm) or with the admin API:
api() { curl -fsS -u admin@example.com:<admin-token> -H 'content-type: application/json' "$@"; }
api https://forge.example.com/admin/repositories \
-d '{"name": "npm-hosted", "kind": "hosted", "format": "npm"}'
api https://forge.example.com/admin/repositories \
-d '{"name": "npm-proxy", "kind": "proxy", "format": "npm", "remote_url": "https://registry.npmjs.org"}'
api https://forge.example.com/admin/repositories \
-d '{"name": "npm-all", "kind": "group", "format": "npm",
"reserved_prefixes": ["@acme"], "writable_member": "npm-hosted"}'
api https://forge.example.com/admin/repositories/npm-all/members -d '{"member": "npm-hosted", "priority": 10}'
api https://forge.example.com/admin/repositories/npm-all/members -d '{"member": "npm-proxy", "priority": 5}'
Most teams point their clients at the group: it serves the internal packages and the public ones, takes publishes (into its writable member) and audits.
Connecting a client
A repository’s page in the web UI shows its registry URL and these lines, ready to copy.
npm, pnpm, Bun and Yarn 1 read .npmrc (the project’s, or ~/.npmrc):
registry=https://forge.example.com/repository/npm-all/
//forge.example.com/repository/npm-all/:_authToken=<token>
Only one scope from Forge, the rest from elsewhere:
@acme:registry=https://forge.example.com/repository/npm-all/
//forge.example.com/repository/npm-all/:_authToken=<token>
Yarn 2 and later read .yarnrc.yml:
npmRegistryServer: "https://forge.example.com/repository/npm-all/"
npmAlwaysAuth: true
npmAuthToken: "<token>"
The token is an API token: one npm login mints (below), or one you create
under Your tokens in the web UI. An API token of yours works for every
repository you may read, unless its scope limits it; Forge sees whose it
is, not which repository it was minted for. A repository that allows anonymous reads needs no token to
install from.
Logging in
npm login (npm 9 and later, and pnpm login) opens a page of Forge in
the browser. Sign in there as always (a password or your organization’s
OIDC provider), check the client, repository and host name it shows, and
approve; the client then stores a new API token in .npmrc:
npm login --registry https://forge.example.com/repository/npm-all/
npm whoami --registry https://forge.example.com/repository/npm-all/
Without a browser on the machine (--browser=false), npm prints the page’s
URL to open elsewhere. A login waits for its approval for
cli_login.web_login_ttl_seconds (10 minutes), and the page answers only
the signed-in user who opens it. Logins in the browser need the web UI
([web] enabled); without it npm falls back to asking for a password.
A username and a password: yarn npm login (Yarn 2+) and npm login --auth-type=legacy send them to Forge, which checks them like the web
login does (local passwords, credential forwarding, the lockout after
failed attempts) and mints a token. No account is ever created by a login.
npm’s legacy login refuses a username containing an @, which every Forge
account has (an e-mail address): use the browser login, Yarn’s, or a token
from the web UI.
A token a login mints carries your rights, expires after
cli_login.token_lifetime_days (90 days; 0: never), and is listed and
revocable under Your tokens like every other. npm logout revokes
the token it logged in with. Logins, the tokens they mint and logouts are
audit-logged (forge::audit).
[cli_login]
token_lifetime_days = 90
web_login_ttl_seconds = 600
Publishing
npm publish --registry https://forge.example.com/repository/npm-hosted/
pnpm publish --registry https://forge.example.com/repository/npm-hosted/
yarn npm publish # with npmRegistryServer / npmPublishRegistry set
bun publish --registry https://forge.example.com/repository/npm-hosted/
Publishing needs the level write and an API token (a password typed
at the web login is not a publishing credential); CI jobs can use a
trusted publishing token instead: npm 11.5.1+
automatically exchanges GitHub Actions or GitLab CI (including self-hosted
GitLab) identity tokens, with no stored _authToken. Publishing to a
group goes to its writable member.
Forge checks what it stores:
- the tarball’s
package/package.jsonnames the package and version the document publishes, and the integrity (sha512) andshasumthe client claims match the tarball; - the name is a valid npm name (scoped or not; at most 214 characters, lowercase for new names) and the version valid SemVer;
- the tarball is a gzip’d tar that unpacks to at most
npm.max_unpacked_bytes, read as npm unpacks it: one that npm would read otherwise than its headers say (a header npm rejects, data after the end of the archive, a PAX record npm reads otherwise, a file some client may unpack topackage.jsonother than<first>/package.json) is refused, whether published or fetched through a proxy.npm packwrites none of these; - the tarball fits the repository’s upload limit (Maximum upload size,
512 MiB by default), and the rest of the document
npm.max_document_bytes.
A version that exists is refused (409), as on the public registry, unless
the repository’s deployment policy allows redeploys (which needs the
level delete). The policy read-only refuses every publish,
unpublish, deprecation and dist-tag change:
api -X PUT https://forge.example.com/admin/repositories/npm-hosted \
-d '{"deployment_policy": "read_only"}'
The dist-tags of the publish (npm publish --tag next; latest by default)
are set with it. Package selectors apply as for every format: a caller
whose selector covers the @acme scope only cannot publish left-pad.
Dist-tags, deprecations and unpublishing
npm dist-tag add @acme/ui@2.0.0-rc.1 next
npm dist-tag rm @acme/ui next
npm dist-tag ls @acme/ui
npm deprecate @acme/ui@1.0.0 "use 2.x"
npm deprecate @acme/ui@1.0.0 "" # undeprecate
npm unpublish @acme/ui@1.0.0
npm unpublish @acme/ui --force # every version
- A dist-tag names one version.
latestis always there: it cannot be removed, and when the version it named is gone it moves to the highest version left. Deleting a version drops its tags. - A deprecated version still installs; clients warn with its message.
- Unpublishing deletes the version (or every version) and needs the level delete. Unlike on the public registry there is no time limit, and a version unpublished may be published again.
Dist-tags and deprecations need write. The web UI shows a
package’s dist-tags, deprecated versions with their messages, and the
README of its latest version.
Installing and searching
Every client installs from a hosted repository, a proxy or a group:
abbreviated packuments (application/vnd.npm.install-v1+json) for clients
that ask for them, full ones (with each version’s publish time and the
README of latest) for the others, tarballs, npm view, npm search
(GET /-/v1/search, over the repository’s packages; a proxy searches its
upstream too) and npm ping. Tarballs are served from the store with
counted as downloads. Tarball URLs in a packument point at the
repository it was read from, so a lockfile written through the group keeps
resolving through it.
Proxy repositories
A proxy reads another registry on demand, https://registry.npmjs.org or
any registry that speaks npm’s protocol:
- Credentials. With a username and a password, it sends Basic
authentication; with a password alone (a registry token, such as an npm
automation token or a GitHub Packages token),
Authorization: Bearer. They go to the origin of the remote URL only, never to a tarball host elsewhere. - Caching. A packument is read again after the repository’s metadata
TTL; installs read the abbreviated one,
npm viewand clients that want publish times (pnpm’sminimumReleaseAge,npm install --before) the full one. A package the upstream does not know is remembered for the not-found TTL. A tarball is downloaded once, checked against theintegrity(orshasum) the upstream publishes and against its version, and kept; one that does not hold up is refused (502) and not stored. A packument larger thanproxy.max_metadata_bytes(32 MiB by default) is not read: until the setting is raised, its package is served from what is cached of it, or answers 502 when nothing is. A few packages publish that many versions (renovate’s abbreviated packument is about 36 MB, its full one about 73 MB). - The upstream’s dist-tags and deprecations are served as it gives them, and the last dist-tags read are kept for when it is down.
- Publication delay.
package_delay_secondsholds back versions published upstream less than that long ago (counted from the upstream’s publish time, not from when Forge first saw them): a new release, possibly a compromised one, reaches your builds only after the delay. - Outages. While the upstream is down, what is cached is served: the
packument as last read, and every tarball downloaded before. A tarball
never downloaded answers 503 with
Retry-After. An upstream that refuses the proxy’s credentials answers 502. - Invalidate cache on the repository’s page (or
POST /admin/repositories/<name>/proxy-cache/invalidate) makes it read the upstream again now.
Publishing, dist-tag changes and unpublishing are refused on a proxy.
Group repositories
A group serves its members, by priority:
- a packument merges the versions of every member that has the package (the highest-priority member’s copy of a version wins), with the dist-tags of the first member, by priority, that has the package and tags for it;
- reserved prefixes (
@acme, the scope@acme/, or a name prefix such asacme-) are never served from proxy members, also before the first internal version is published, so a public package that claims your scope or name cannot slip into a build (dependency confusion); - prefer hosted (on by default) hides proxy versions of a package a hosted member has;
- the writable member, a hosted npm repository that is also a member, takes publishes, dist-tag changes, deprecations and unpublishing through the group; without one they are refused.
Audits
npm audit (and pnpm audit) asks the registry for advisories. A proxy
passes the request on to its upstream with its credentials, and a group to
its first proxy member; a group leaves the packages under its reserved
prefixes out, so your internal package names stay inside. A hosted
repository has no upstream to ask (404). Auditing needs read.
Permissions
Each access level includes the ones below it (deployment.md):
| What | Level |
|---|---|
Install, npm view, search, audit | read |
Publish, npm dist-tag add / rm, npm deprecate | write |
| Unpublish (a version or the whole package); a redeploy | delete |
npm login, npm logout, npm whoami | none (a login checks the password) |
Content selectors match package names (@acme/*), for every level a grant
includes. An unknown repository answers like one the caller may not read.
A read-only deployment policy refuses every change, whatever the caller’s
level.
Retention
Retention policies apply to npm repositories like to every other format,
with one rule more: a version a dist-tag names is never deleted (a
proxy’s tags are its upstream’s, as last read), so latest, next and
every other tag keep installing. The preview of a policy lists such
versions as kept.
Package retention policies may include a size_budget to evict least-recently
used eligible versions when a repository exceeds its logical size limit. See
Retention size budgets for configuration and size accounting.
Limits
[npm]
max_document_bytes = 33554432 # 32 MiB: a publish document without its tarball, an update, an audit
max_unpacked_bytes = 1073741824 # 1 GiB: the most a tarball may unpack to
A tarball’s size is bounded by its repository’s upload limit. A login’s body is at most 64 KiB.
Limitations
- No provenance. A publish with a provenance statement (
npm publish --provenance) attaches a second file, which Forge refuses: publish without it. - No package access management.
npm access,npm owner,npm team,npm org,npm starandnpm tokenare not served; access is Forge’s roles and package selectors, tokens are managed in the web UI. - npm’s legacy login and
@usernames. See Logging in. - One registry per URL. Clients address a repository by its URL; a scope maps to one registry, as in npm.
Raw file repositories
Forge hosts arbitrary files under arbitrary paths, the way Nexus’ raw-hosted, raw-proxy and raw-group repositories do: the binaries a build downloads that have no package manager of their own (the pdfium binaries, for example), installers, firmware images, test fixtures, tool downloads.
Files live in repositories of format: "raw", a format plugin built into
Forge like NuGet (crates/formats/raw;
GET /admin/formats lists it), served at
https://<host>/repository/<repository>/<path>. The plugin
speaks the protocol below; the host’s file store keeps the files
(format-plugins.md). They come in the
usual three kinds:
- hosted: upload, download and delete your own files;
- proxy: a cache of files under an HTTP(S) base URL, such as a GitHub project’s release downloads;
- group: one URL over several members; the first member, by priority, that has a path serves it.
Files are blobs in the content-addressed store: the same content under several paths, or in several repositories, is stored once.
Contents
- Hosted repositories
- Downloads
- Listings
- Paths
- Proxy repositories
- Group repositories
- Permissions
- Retention
- Example: the pdfium binaries
- Limitations
Hosted repositories
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/repositories \
-d '{"name": "binaries", "kind": "hosted", "format": "raw"}'
Upload a file with PUT to its path, with an API token as the password (a
password typed at the web login is not an upload credential, and neither is
a browser session):
curl -u ci:<token> --upload-file pdfium-linux-x64.tgz \
https://forge.example.com/repository/binaries/pdfium/7350/pdfium-linux-x64.tgz
A new file answers 201, a replaced one 204. The Content-Type stored
with a file is the one the upload declares, unless it only says “bytes”
(none, application/octet-stream, or the application/x-www-form-urlencoded
of curl --data-binary): then the file name decides (.json, .txt,
.zip, …), else application/octet-stream.
Several files go in one multipart/form-data upload to a directory, each
stored under its part’s file name. Nexus’ field names for its component
upload work too: raw.directory names a subdirectory, and
raw.assetN.filename the name of the file part raw.assetN. Every file of
the form is stored, or none (an existing file the policy refuses fails the
whole form):
curl -u ci:<token> -F raw.directory=7350 \
-F raw.asset1=@pdfium-linux-x64.tgz -F raw.asset1.filename=pdfium-linux-x64.tgz \
-F raw.asset2=@pdfium-win-x64.tgz -F raw.asset2.filename=pdfium-win-x64.tgz \
https://forge.example.com/repository/binaries/pdfium/
At most 100 files per form; each one is limited by the repository’s upload
limit (Maximum upload size, max_upload_bytes; 512 MiB when unset) and
staged on disk like every push (see server.max_staging_bytes).
DELETE /repository/<repository>/<path> deletes a file (204). A directory, with
every file below it, is deleted through the admin API (or the web UI):
curl -u admin@example.com:<admin-token> -X DELETE \
'https://forge.example.com/admin/repositories/binaries/files?path=pdfium/7350/'
# {"deleted": 2}
A path without a trailing / deletes that one file, and an empty path every
file of the repository, which a hosted repository with files needs before it
can be deleted.
Deployment policy
As in Nexus, deployment_policy decides what an upload to a path that holds
a file does:
deployment_policy | |
|---|---|
disable_redeploy (default) | Refused with 409. Uploading the same content again is accepted and changes nothing, so a retried upload succeeds. |
allow_redeploy | The file is replaced. Replacing needs the delete level, which includes write. |
read_only | Every upload and delete is refused with 405. |
curl -u admin@example.com:<admin-token> -X PUT -H 'content-type: application/json' \
https://forge.example.com/admin/repositories/binaries \
-d '{"deployment_policy": "allow_redeploy"}'
Displayed or downloaded
Files are served as downloads (Content-Disposition: attachment) by
default. With "content_disposition": "inline" a browser displays them
instead (a PDF, a text file, an HTML report). Either way every answer
carries Content-Security-Policy: sandbox and X-Content-Type-Options: nosniff: an HTML or SVG file cannot run script on Forge’s origin.
Downloads
GET and HEAD of a file answer with its bytes and
ETag: the file’s SHA-256, a strong validator;If-None-MatchandIf-Modified-Sinceanswer304while it is current;Last-Modified: when its content was last uploaded (or, in a proxy, cached);Content-Length,Content-Type, andAccept-Ranges: bytes.
One Range is served (206, 416 past the end), and If-Range with the
ETag or the date, so an interrupted download resumes where it stopped:
curl -u me:<token> -C - -O https://forge.example.com/repository/binaries/pdfium/7350/pdfium-linux-x64.tgz
Checksum sidecars
<file>.sha256, <file>.sha512, <file>.sha1 and <file>.md5 are
answered from the stored file’s digests, as Nexus does, without being
uploaded: the digest and the file name, the way sha256sum prints them, so
the -c mode of the matching tool checks a download:
curl -u me:<token> -O https://forge.example.com/repository/binaries/pdfium/7350/pdfium-linux-x64.tgz
curl -u me:<token> https://forge.example.com/repository/binaries/pdfium/7350/pdfium-linux-x64.tgz.sha256 \
| sha256sum -c
# pdfium-linux-x64.tgz: OK
A file actually stored under such a name is served as it is.
Listings
GET of a directory (a path ending in /, or the repository’s root
/repository/<repository>/) lists it: as an HTML page for browsers, as JSON with
Accept: application/json (or ?format=json). A directory named without
its / redirects to it.
curl -u me:<token> -H 'Accept: application/json' https://forge.example.com/repository/binaries/pdfium/
{
"repository": "binaries",
"path": "pdfium/",
"entries": [
{"name": "7350/", "path": "pdfium/7350/", "type": "directory"},
{"name": "README.txt", "path": "pdfium/README.txt", "type": "file", "size": 1204,
"content_type": "text/plain", "sha256": "…", "last_modified": "2026-10-21T10:58:44Z"}
],
"next": null
}
Entries come in byte order of their names, a directory’s ending in /. A
page holds limit entries (100 by default, at most 1,000); next (and a
Link: <…>; rel="next" header) names the after of the next page. A
listing costs one index lookup per entry, however many files are below the
directory. A proxy lists the files it has cached, and a group the merged
entries of the members the caller may read.
The web UI’s repository screen is a tree browser of the same listings: a
file’s size, type, SHA-256 and URL with the curl commands to download,
check and upload it; an upload of files into the directory and the delete of
a file (with the same permissions as the repository URL); and, for administrators, the
delete of a directory.
Paths
A path is a list of segments separated by /. A request whose path has an
empty segment (a//b), a . or .. segment (also spelled %2e%2e), an
encoded / (%2F) or a backslash (also %5C) in a segment, a control
character, a segment over 255 bytes or more than 1,024 bytes in all is
refused with 400: nothing is collapsed or resolved, so every stored file
has exactly one spelling. Percent-encoded characters are decoded once
(a%20b is a b), and paths are case-sensitive.
Proxy repositories
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/repositories -d '{
"name": "pdfium-releases", "kind": "proxy", "format": "raw",
"remote_url": "https://github.com/bblanchon/pdfium-binaries/releases/download/",
"max_age_seconds": 0
}'
A file <path> of the proxy is <remote_url>/<path>, its segments
percent-encoded. The first request fetches and caches it; later ones are
served from the cache.
- Redirects are followed (GitHub release assets redirect to
release-assets.githubusercontent.com), up to five hops, each one through the upstream client’s guard against cloud-metadata addresses. Theupstream_username/upstream_passwordare sent to the origin ofremote_urlonly, never to the host a redirect leads to. Theremote_urltakes no query or fragment. - Re-validation. A cached file is served for
max_age_seconds(a day by default) after it was fetched or last confirmed. Then the proxy asks the upstream with a conditional GET (If-None-Matchwith the upstream’sETag,If-Modified-Sincewith itsLast-Modified): unchanged, the copy is kept; changed, it is replaced.0never asks again, for upstreams whose files never change (release assets). - Misses are remembered for
not_found_ttl_seconds(600 seconds by default;0remembers none). - An unreachable upstream (or one answering an error) does not fail a
file that is cached: the cached copy is served. A file it never cached
answers
502. A file the upstream answers404for on re-validation is served from the cache too, until it is deleted (admin API) or retention evicts it. - Invalidate cache on the repository’s page (
POST /admin/repositories/<name>/proxy-cache/invalidate, optionally with{"package": "<file or directory/>"}) forgets the misses and makes the cached files (of that file or directory) due for re-validation. - Pointing the proxy at another
remote_urlempties its cache. - An offline proxy serves what it cached and never asks its upstream.
- A proxy takes no uploads. Files larger than
proxy.max_package_bytes(1 GiB by default) are not cached and answer502.
Group repositories
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/repositories -d '{"name": "files", "kind": "group", "format": "raw"}'
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/repositories/files/members -d '{"member": "binaries", "priority": 100}'
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/repositories/files/members -d '{"member": "pdfium-releases", "priority": 10}'
A request for a path is answered by the first member, highest priority first, that has it (a proxy member: that has it cached or whose upstream has it), the checksum sidecars included. Members are raw repositories (a group cannot get a member of another format), nested groups are allowed, and an offline hosted or group member is skipped. A member that fails (an unreachable upstream) is skipped while another has the path. As for every format, reading a group grants nothing on its members: the caller is served from the members it may read itself. A raw group has no owners, preferred hosted members or reserved prefixes: paths have no namespaces to confuse.
A group takes no uploads; upload to a hosted member.
Permissions
GET and HEAD need the level read on the repository, PUT and POST
need write, DELETE needs delete (and so does replacing a file with
allow_redeploy); each level includes the ones below it. Content selectors
match file paths, so a role can be limited to part of a repository:
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/content-selectors \
-d '{"name": "pdfium-only", "includes": ["pdfium/**"]}'
A file outside the caller’s selector is not found, its sidecars neither, and a listing leaves it out; a directory is listed when the caller may see a file in it (among its first 1,000).
Uploads and deletes are recorded in the audit log (forge::audit, “package
mutation”), and a workload identity’s in the trusted-publishing ledger, with
the file’s path and SHA-256.
Retention
A raw repository’s retention policy has the envelope of a package policy
(version, enabled, dry_run_only, min_run_interval_seconds) with file
rules:
| Field | |
|---|---|
path_prefix | Only files whose path starts with it (nightly/); all files without one. |
keep_newest_per_directory | The N most recently uploaded files of each directory (among the rule’s) stay. Alone, it keeps N per directory and deletes the others. |
older_than_days | Trigger: uploaded (or cached) at least this many days ago. |
not_downloaded_for_days | Trigger: not downloaded for this many days (counted from the upload when that is later, or when it was never downloaded). |
A file goes when any rule selects it; within a rule, every trigger it sets must hold. On a proxy the same rules evict cached files, which the upstream serves again on their next download.
curl -u admin@example.com:<admin-token> -X PUT -H 'content-type: application/json' \
https://forge.example.com/admin/repositories/binaries/retention -d '{"policy": {
"version": 1,
"rules": [
{"id": "nightlies", "path_prefix": "nightly/", "keep_newest_per_directory": 5},
{"id": "cold", "older_than_days": 365, "not_downloaded_for_days": 90}
]}}'
POST …/retention/preview lists what a run would delete (file_decisions),
and every run’s decisions are audited like other formats’. The preview
evaluates the stored policy, or a draft sent as {"policy": {…}}, which is
checked like a PUT of it and not stored. The repository’s Retention
screen in the web UI (Repositories → the repository → Retention) does all of
this: it edits the rules, previews them before they are saved, runs the
policy and lists the audit log.
Retention policies may also include a size_budget to evict least-recently
used files when the repository exceeds a logical size limit. See
Retention size budgets for configuration and size accounting.
Example: the pdfium binaries
The pdfium binaries are
published as GitHub release assets under tags such as chromium/7350. A
proxy caches them as they are, and a hosted repository keeps re-packaged
copies; a group serves both under one URL:
# The proxy of the release downloads (assets of a release never change).
curl -u admin@example.com:<admin-token> -H 'content-type: application/json' \
https://forge.example.com/admin/repositories -d '{
"name": "pdfium-releases", "kind": "proxy", "format": "raw",
"remote_url": "https://github.com/bblanchon/pdfium-binaries/releases/download/",
"max_age_seconds": 0}'
# A build downloads through it and checks the download.
curl -u ci:<token> -O \
https://forge.example.com/repository/pdfium-releases/chromium/7350/pdfium-linux-x64.tgz
curl -u ci:<token> \
https://forge.example.com/repository/pdfium-releases/chromium/7350/pdfium-linux-x64.tgz.sha256 | sha256sum -c
# A re-packaged copy (only the library and the headers) in a hosted repository.
mkdir pdfium && tar -xzf pdfium-linux-x64.tgz -C pdfium
tar -czf pdfium-linux-x64.tgz -C pdfium lib include
curl -u ci:<token> --upload-file pdfium-linux-x64.tgz \
https://forge.example.com/repository/binaries/pdfium/7350/pdfium-linux-x64.tgz
CI runs this example against GitHub (.github/scripts/raw-e2e.sh), next to
the other curl checks of this page.
Limitations
- Listings and searches. A listing is per directory; there is no search across a repository’s paths. A proxy lists only what it has cached: an upstream’s own directory listings are not read.
- Ranges. One range per request; a multi-range request is answered in full.
- No Nexus import. Importing a Nexus raw repository is not supported yet (nexus-import.md covers NuGet and Docker).
- No directory delete through the repository URL. A directory is deleted through the admin API or the web UI (administrators).
Format plugins
Every format Forge serves comes from a format plugin: a Rust crate that
teaches Forge the format’s rules, its client protocol and, for proxies, how
to read an upstream, by implementing one trait,
FormatPlugin. The repository machinery
stays Forge’s own. That covers storage, metadata, authentication, RBAC and
package selectors, hosted, proxy and group repositories, retention, and the
streaming of package bytes.
Every plugin is built in: a crate under crates/formats/<id>/ linked into
the server and called directly, like any other Rust code. Plugins need no
configuration. Forge ships four: Docker / OCI
(crates/formats/docker,
Images formats, docker.md), NuGet
(crates/formats/nuget), npm
(crates/formats/npm, npm.md) and raw
files (crates/formats/raw,
Files formats). Helm charts are pushed to Docker
repositories (docker.md).
Status: preview. The plugin contract may change in any release. The plugins build with the server, so the compiler checks each of them against the contract it is linked with. Contract 0.4.0 added files formats (Files formats), and
renderreturns an action, likehandle; for npm, request bodies, document uploads, changes (tags, deprecations), logins and forwards followed (npm as a plugin), and images formats (Images formats).
Contents
- Layout
- How it fits together
- Files formats
- Images formats
- Proxy repositories
- Using a plugin
- Security model
- Writing a plugin
- Limitations and roadmap
Layout
A plugin is a workspace crate that describes itself in Rust:
crates/formats/nuget/
├── Cargo.toml # the plugin's version
├── src/lib.rs # DESCRIPTOR (id, name, kinds hosted/proxy/group)
│ # and impl FormatPlugin for Nuget
├── src/protocol.rs # the V3 protocol: handle and render
├── src/upstream.rs # proxies: reading another V3 feed (upstream)
└── tests/ # the protocol against canned answers (forge-plugin-api `testing`)
crates/formats/raw/
├── Cargo.toml
├── src/lib.rs # DESCRIPTOR (storage files, kinds hosted/proxy/group)
│ # and impl FormatPlugin for Raw: paths, methods, upstream URLs
├── src/listing.rs # directory listings as HTML or JSON
├── src/sidecar.rs # .sha256/.sha512/.sha1/.md5 from the stored digests
├── src/form.rs # multipart uploads, Nexus' field names
└── tests/ # the protocol against canned answers
crates/formats/npm/
├── Cargo.toml
├── src/lib.rs # DESCRIPTOR (kinds hosted/proxy/group, Bearer
│ # upstream tokens) and impl FormatPlugin for Npm
├── src/protocol.rs # the registry API: packuments, publishes, dist-tags, logins
├── src/publish.rs # the tarball inside a publish document, as it streams
├── src/proxy.rs # proxies: reading another registry (upstream)
└── tests/ # the protocol against canned answers
crates/formats/docker/
├── Cargo.toml
├── src/lib.rs # DESCRIPTOR (storage images, kinds hosted/proxy/group)
│ # and impl FormatPlugin for Docker: paths, manifests, charts
├── src/protocol.rs # the distribution-spec API: handle and render
├── src/upstream.rs # proxies: URLs, tag pages and referrers of an upstream registry
├── src/error.rs # the distribution-spec error answers
└── src/manifest.rs, chart.rs, path.rs, reference.rs, digest.rs
The pieces:
| Crate | What it holds |
|---|---|
forge-plugin-api | The contract: the FormatPlugin trait and its types, and, behind its testing feature, helpers that run a plugin’s protocol against canned answers. Depends on nothing but std. |
crates/formats/<id> | One crate per plugin, implementing FormatPlugin. |
forge-plugin | The host side: Plugin (an installed plugin, which describes itself), PluginFormat, which adapts it to Forge’s Format seam, and the checks of a descriptor (descriptor::check). |
forge-server | The plugins it links in (plugins::builtin), the /repository/{repo}/... dispatcher, the file store files formats keep their files in (crates/server/src/files), and the image store and token service of images formats, under /v2 (crates/server/src/images). |
How it fits together
| Forge (the host) | The plugin | |
|---|---|---|
| Identity | Calls the format for every id and version it sees | canonical_id, validate_id, normalize_version, compare_versions |
| Publishing | Streams the upload to a staging file, enforces the format’s upload limit, authorizes against the package selector, stores the blob content-addressed, writes the version in one transaction, records trusted-publishing provenance | parse_package: reads the staged file, returns id, version, manifest, dependencies. For a package that comes inside a document (npm’s JSON with a base64 tarball), unpack_upload takes the package out while the body streams by, and parse_upload reads it with the rest of the document |
| Reads | Runs the queries the plugin asks for through the repository read path (hosted, proxy or group). Package selectors, group membership, prefer-hosted and reserved prefixes all apply | handle decides what a request is; render turns the loaded data into the response body |
| Downloads | Streams the blob from S3, counts the download, answers HEAD without touching S3, and a single Range when the local blob cache is on | Names the version (download) |
| Unlist, relist, delete | Hosted write path, with the repository’s permissions and selectors; a package outside the caller’s selector answers 404, like an unknown one | Names the version and the success response |
| Tags and deprecations | Keeps a package’s tags (npm’s dist-tags: a name for one version) and each version’s deprecation, drops a version’s tags with the version, and never lets retention delete a tagged version | Names the changes (tag, untag, deprecate); published names those a publish document carries |
| Logins | Checks a password through the realm chain, or runs a login approved in the browser, and mints, lists and revokes the API tokens; accepts Authorization: Bearer API tokens | Says what a request asks (auth) and renders the outcome (auth_response) |
| Proxies | Sends every upstream request (credentials only to the upstream’s origin, bounded, cached), downloads and verifies packages, keeps the package delay, the not-found markers and the offline mode | upstream: which documents an operation needs, and what they say (Proxy repositories) |
| Groups | Merges members by priority; the dependency-confusion guards apply | Nothing: a group of plugin repositories just works |
| Retention | Evaluates policies, ordering versions with the plugin’s compare_versions | Nothing more |
| Files (a files format) | Keeps files under paths, content-addressed with their digests: serving, ranges and validators, uploads under the deployment policy, listings, proxies and groups, selectors on paths, retention by path (Files formats) | Which path a request names, which file or directory it needs, what the response looks like |
A request to every non-Docker repository is served under
/repository/{repo}/...; the repository name selects its format. Docker
repositories are served only through the registry under /v2.
The dispatcher
(crates/server/src/plugins.rs) handles it
in this order:
-
Resolve the repository and the caller (a session, Basic credentials, or an
Authorization: BearerAPI token). Unknown repositories answer like unreadable ones, and offline ones get 503 (to a reader). A files format’s request is authorized by its method first: GET and HEAD need read, PUT and POST write, DELETE delete; a package format’s actions are checked one by one (below). -
Call the plugin’s
handlewith the repository (name, kind, base URL, a group’s reserved prefixes) and the request: method, path, query, and a short allowlist of headers. The plugin never sees credentials or cookies. Protocol calls run on a blocking thread, never on the async workers. -
Act on the plugin’s answer, checked against what the caller may do:
respond: send the response as given, keeping only the allowed headers. Needs read.load(queries): run the queries, then callrenderwith every query and its answer.renderanswers likehandledoes: a response, more data to load, or one of the actions below. One request gets at most 4 rounds and 2,048 answers. Needs read.download: stream the stored package. Needs read.serve-file: stream a stored file (a files format).read body: read the request’s body, bounded (at most 64 MiB, 64 KiB for a login), and askhandleagain with it. Its use says what it needs: a login nothing, a read read, a change the permission of the method.publish,change(unlist,relist,delete,deprecate,tag,untag): go through the hosted write path, but only for a mutating method, with the permission each needs (delete for a delete, write for the rest), never for a GET. The repository must be hosted, or a group with a writable member (where they go), its deployment policy must not be read-only, and the caller must present an API token or a workload identity.auth: a login, a logout or awhoami, performed by the host; the plugin renders the outcome (auth_response).forward: a POST to the repository’s upstream (a proxy’s, or a group’s first proxy member’s), with the proxy’s credentials; its answer is passed back. npm’s audits go so. Needs read.
A host error (a refusal, a conflict, a missing package) is rendered by the plugin’s
error_responsewhen it has one: npm clients print{"error": "..."}, NuGet’s plain text.
The contract is crates/plugin-api/src/lib.rs.
Its queries are targeted:
| Query | Answer |
|---|---|
versions(id) | Every version of a package |
packages(id) | Every version with its metadata: publish time, SHA-256, SHA-512, size, the stored manifest summary, the deprecation |
dated-packages(id) | packages, with every version’s publish time even where that costs more: a proxy asks its upstream with dating instead of metadata (npm’s full packument, whose time installs with a minimum release age read) |
packages-of(id, versions) | Some versions with their metadata (a page of a long listing) |
package(id, version) | One version with its metadata |
manifest(id, version) | The manifest stored at publish |
search(text, skip, take, prerelease, semver2, upstream) | Matching packages with their versions, the latest version’s summary and the download total |
tags(id) | The package’s tags: a hosted repository’s own, a proxy’s upstream’s, a group’s from its first member that has the package and tags for it |
A files format asks for files instead (Files formats); a query of the other kind fails the request.
Prerelease filtering is the host’s, and its rule is that a normalized version
is a prerelease when it contains a -; semver2 adds SemVer 2.0.0 versions
(dotted prerelease labels, build metadata). Publish times are RFC 3339 in UTC
with milliseconds. A take of 0 is the default page (20).
Manifest summaries and the web UI
At publish, Forge stores the JSON object the plugin’s manifest_summary
returns with the version. render gets it back as summary, so a listing
never re-reads manifests (for a version stored before summaries were, the
host asks manifest_summary of its stored manifest). The web UI’s package detail reads three keys from it,
if present: description, deprecated (a proxy’s upstream deprecation),
and dependency_groups (the same shape as NuGet’s). It shows the latest
version’s README, which the plugin’s readme reads from the stored
manifest, and the package’s tags.
Every listing hands each version’s summary to the plugin, so a summary should stay small.
Files formats
A format whose descriptor says storage: Storage::Files holds files under
paths, not package versions: raw is one. It keeps the defaults of
the identity rules and parse_package (which refuse every id, version and
package) and implements the protocol over the host’s file store, which keeps
everything a file repository is: the files, content-addressed, with their
SHA-256, SHA-512, SHA-1 and MD5; uploads, under the repository’s upload
limit and deployment policy (allow_redeploy, disable_redeploy,
read_only); serving with one Range, If-Range, ETag, Last-Modified
and conditional requests, under the repository’s content disposition;
listings that walk an index; proxies that fetch on first use, follow
redirects and re-validate after max_age_seconds; groups; package selectors
on paths; retention by path prefix, age, downloads and the newest N per
directory; and the web UI’s upload, delete and the admin API’s directory
delete.
The plugin decides what a request is, in the vocabulary of paths. A path is
segments joined by /, without a leading or trailing /, none of them
empty, . or .., nor holding a backslash or a control character
(forge_plugin_api::files parses
request paths by these rules); a directory is empty (the root) or a path
ending in /. The host refuses any other path a plugin names.
| Query | Answer |
|---|---|
file(path) | The stored file with its size, content type, digests, modification and last download time, or none. A proxy re-validates its copy once it is due; a group answers with its first member, by priority, that has the path |
fetch-file(path) | file, and a proxy that has not cached the path asks its upstream (upstream’s file operation) |
directory(directory, after, limit) | A page of a directory: files and subdirectories in byte order of their names, and the after of the next page; none for a directory that does not exist. A group merges its members’ entries |
form | Receives the request’s multipart/form-data body: file parts are staged, other fields read |
store(files) | Stores files, each from the request body or a form part, all or none, under the deployment policy; what became of each (created, replaced, unchanged) |
remove(path) | Deletes a file; whether there was one |
form, store and remove change the repository: each comes alone in its
round, needs the method’s permission (write, or delete for remove), a
hosted repository and a package-client credential, and when one fails (403,
405, 409, 413) the host answers with its own status. The action
serve-file(path) serves a file as fetch-file finds it. The repository
a request is for says whether its files are shown inline (inline), for
responses the plugin renders itself.
Raw maps its URLs onto these: GET /repository/<repo>/<path> is file(path)
(and file of the file a checksum sidecar names), then serve-file or
the sidecar rendered from the file’s digests, else fetch-file in a proxy
or group, else a directory probe for a redirect; GET …/<dir>/ is a
directory page rendered as HTML or JSON; PUT is a store of the body;
POST …/<dir>/ is a form, then a store of its parts with Nexus’ field
names applied; DELETE is a remove.
crates/formats/raw is the whole protocol, in a few
hundred lines; the file store is
crates/server/src/files.
Images formats
A format whose descriptor says storage: Storage::Images holds images:
content-addressed blobs and manifests linked into each image (team/app),
tags that point at manifests, indexes that name other manifests, referrers,
and uploads in sessions of several requests. Docker / OCI is
one (crates/formats/docker). Its mount is
/v2 (IMAGES_MOUNT), where every registry client asks, so a server holds
one images format at most.
The split is the one files formats have. The format owns the protocol: the
grammar of paths, names, references and digests, what a manifest names and
says, the error answers, the response headers, and the URLs of an upstream
registry. The host owns the image store
(crates/server/src/images), which keeps
everything else:
/v2/(the version check),/v2/token(the token service and its signing keys) and/v2/_catalog(refused), answered by the host itself;- resolving a name to a repository and an image, by path
(
<repository>/<image>) or on a repository’s own host (host); - authorization: the Bearer challenge and a token’s scopes, RBAC and package selectors on image names, and where a change lands (a group’s writable member);
- the store: blobs and manifests by digest in the content-addressed store, tags (immutable ones too), upload sessions and their chunks, mounts from other images, the references between manifests, and the trusted-publishing ledger;
- groups, a proxy’s caching, retention, garbage collection, and the Nexus import;
- serving a blob or a manifest, with one
Range,Docker-Content-DigestandETag, and the registry’s response hardening on every answer.
The plugin never sees credentials, never touches storage and never holds a blob in memory. A request goes through three steps:
image_pathsplits the path below the mount into the name the client wrote and the endpoint (/manifests/latest), and says what the request needs: read, write (every request of an upload session included) or delete.- The host resolves the name and authorizes the request. What it refuses
(an unknown name, a missing token, a denied scope, an offline
repository) it hands to the format as an
ImageError, andimage_errorrenders it, the host adding its Bearer challenge. handlegets the request with the endpoint as its path and the resolved image asrequest.image, and answers as for any format: a response, queries, or an action.
| Query | Answer |
|---|---|
manifest(image, reference) | A manifest by tag or digest: digest, media type, size, artifact type, subject, the stored summary |
blob(image, digest) | A blob’s digest and size; a manifest is content too |
tags(image, last, n) | A page of the image’s tags in byte order, and whether more follow |
referrers(image, digest, artifact_type) | The manifests whose subject is a manifest, of one artifact type when one is given |
start-upload(image, mount) | An upload session, or a blob mounted from another image without one |
upload-status(id), append-upload(id, range) | How far a session got, after appending the request body (at range, when the client gave one) |
finish-upload(id, digest, range) | Appends the body, ends the session and links its content into the image as digest |
cancel-upload(id) | Ends a session without a blob |
put-manifest(image, reference, tags) | Stores the request body as a manifest under reference and points tags at it |
delete-manifest(image, reference), delete-blob(image, digest) | Deletes a tag or a manifest, or unlinks a blob no manifest names |
A query that changes the repository comes alone in its round. Every answer
is a result: a query that failed (a blob unknown, a digest that does not
match, a manifest naming a blob the image lacks, a tag that is immutable, an
upload too large) comes back to render as its ImageError, and the format
answers it with its own error codes and status, which registry clients read.
The actions serve-blob(image, digest) and serve-manifest(image, reference) stream what a query of the same request found. A response the
format renders keeps content-type, docker-content-digest,
docker-upload-uuid, link, location, oci-filters-applied,
oci-subject, oci-tag, range and retry-after; location and link
only when they stay under /v2/.
Manifests are the format’s to read. The host stages a pushed manifest (at
most 4 MiB), and parse_manifest says what it is served as, the blobs and
manifests it names (which the push needs in the image), its subject and
artifact type, and a JSON summary stored with it. A proxy’s manifests are
read the same way. When the manifest names a config small enough to read
(an image’s, a Helm chart’s), the host reads it once the image holds it,
and merges what config_summary makes of it (the platform, the chart’s
metadata) into the summary; the summary names the config under
config_digest and config_media_type, so a proxy completes the summaries
of the manifests it cached before their config. The web UI reads
layers_size, platform, platforms and chart from a summary, and shows a
chart’s README and values with artifact_files, over the archive blob the
host lends it (at most 16 MiB).
A proxy’s upstream operations name where things are, and the host fetches them:
| Operation | The plugin answers |
|---|---|
image-name(image) | The name the upstream knows the image by, which the proxy caches it under (Docker Hub’s library/ namespace) |
image-manifest(image, reference) | The manifest’s URL, the Accept header to send and the scope of a token for it. The host re-validates a cached tag with a HEAD, downloads the manifest, checks it against its digest and reads it with parse_manifest |
image-blob(image, digest) | The blob’s URL; the host streams it into the store and checks its digest |
image-tags(image, last, n) | A page of the upstream’s tags, following the upstream’s Link pages (a conversation of fetches) |
image-referrers(image, digest) | The upstream’s referrers of a manifest |
An images format’s fetches go to the origin of remote_url only, carry the
Accept header the plugin names, and come back with their status and Link
headers. When the upstream answers 401 with a Bearer challenge, the host asks
the realm the challenge names for a token (with the repository’s
credentials, if it has any) and sends the request again, as Docker Hub
requires; the token is the host’s, cached for its lifetime.
testing::converse_answered runs upstream against canned answers with
their status and headers, as the Docker plugin’s tests do.
Proxy repositories
A plugin that lists proxy among its kinds implements the upstream side of
the contract, upstream. The host keeps everything a proxy is, whatever the
format: the proxy_cached_version rows, the package delay, the not-found
markers and their TTL, the download singleflight, the content-addressed
store, credentials (decrypted per request, sent only to the origin of the
repository’s remote_url), where requests may go (below), the SSRF and
cloud-metadata guards, the size and time bounds of every request, and the
offline mode. What the plugin knows is which documents an operation needs and
what they say:
| Operation | The plugin answers |
|---|---|
index | A JSON object of its own making, from remote_url (NuGet: the service index’s resources). The host persists it, refreshes it once per metadata TTL, falls back to the last good one when a refresh fails, and hands it to every other operation |
versions(id) | The versions the upstream lists, or none when it does not know the package |
package(id, version) | The package’s URL, and the SHA-512 or SHA-1 the upstream publishes for it when it does (located). The host downloads it, checks the digest and, with parse_package, the version asked for, and stores it |
manifest(id, version) | The manifest, verified to be that version’s, without downloading the package |
metadata(versions), dating(versions) | Per version: listing state, publish time, a manifest summary for a version not downloaded yet, and (dating) when it was created upstream, which the package delay counts from |
search(query) | A page of the upstream’s search results |
file(path) | A files format: the file’s URL, or none when the upstream cannot have it. The host downloads it (following redirects; credentials go to remote_url’s origin only), caches it, and re-validates it with a conditional GET once the repository’s max_age_seconds has passed (raw: <remote_url>/<path>) |
tags(id) | The tags the upstream gives the package. The host keeps the last ones read, for when the upstream is down |
Each operation is a conversation: upstream either answers or asks for
GETs (fetch), and is called again with every fetch so far and how it went
(a body, a 404, or a failure), so a plugin keeps no state between calls. A
fetch says how long its document may be reused: for the repository’s
metadata TTL, or for a day when it never changes once published, and may
name the Accept it sends (npm’s abbreviated and full packuments share a
URL). One conversation gets at most 4 rounds and 256 fetches, 4 at a time,
and keeps at most 4 × proxy.max_metadata_bytes of documents, all rounds
together: a document past that comes back as a failure, and so does every
later fetch of the conversation, which is not sent.
An unavailable upstream is an outage, not a fault of Forge. Unavailable
means a fetch or a download got no answer (connection refused, a name that
does not resolve, a timeout) or a 5xx or 429. When nothing cached answers in
its place, the request answers 503 with Retry-After, and the host logs
a warning that names the upstream URL. So does a group whose answer waits
on such a member. An upstream that refuses the proxy (a 401 or 403 for its
credentials), or serves a package the proxy does not take (too large,
unreadable, not the version asked for, not matching its published digest),
answers 502: retrying does not help until the upstream or the proxy
changes. Any other failure answers 500: an answer the plugin cannot use, a
document over its size bound.
The host sends a fetch only where the upstream points: to the origin (scheme,
host and port) of remote_url, or to an origin that an absolute http or
https URL names in a document the host fetched for the same operation (in an
earlier round), or while resolving index. The URLs package and file
answer are held to the same rule. Anything else is not sent: the fetch comes back as a
failure, a package URL fails the download like an upstream error, and the host
logs a warning naming the format and the repository. What the plugin answers
names no origin: its index is its own making (and the host keeps fields of its
own beside it, so an index may not use the keys remote_url and
named_origins). So a NuGet service index can put its resources on other
hosts (nuget.org’s search service is on azuresearch-*.nuget.org), and a
chart repository’s index its charts (on a project’s GitHub releases), but a plugin cannot make
Forge request a host its upstream does not point at. Redirects are the
upstream’s to choose, and are followed under the cloud-metadata guard.
An administrator makes a proxy of any format ask its upstream again before
those TTLs run out with Invalidate cache on the repository’s page, or
POST /admin/repositories/{name}/proxy-cache/invalidate: the proxy forgets
its not-found markers (of one package with {"package": "<id>"}, or of
all) and every document it fetched, on every instance, and invalidating the
whole repository also refreshes its index. Downloaded packages stay
cached.
Using a plugin
GET /admin/formats lists every format the server can create repositories
of: the plugins (Docker, NuGet, npm, raw files), each from its descriptor, with
the repository kinds it supports, what its repositories hold (storage:
packages, files or images) and where its repositories are served. The
web UI’s New repository form offers them. Plugins serve hosted and
group repositories, and proxies when they implement the upstream
side (all four do).
A repository of a plugin’s format is created like any other, with the
plugin’s format id, and served under /repository:
# Create a hosted repository of the plugin `example` (or use the web UI).
curl -u admin@example.com:$ADMIN_TOKEN -H 'content-type: application/json' \
-d '{"name": "pkgs", "kind": "hosted", "format": "example"}' \
https://forge.example.com/admin/repositories
# Its clients use https://forge.example.com/repository/pkgs/
As with every format, the password is an API token. A password typed at the web login is not a publishing credential. The admin API refuses to create a repository of a format the server does not have, and to put repositories of different formats in one group. A repository whose format the server no longer has stays in the database: it answers 503, is never read as another format, and retention skips it (a failed run is recorded).
The web UI’s Browse screen filters repositories by format
(GET /browse/repos?format=<id>), and a repository’s screen shows its
client commands: dotnet nuget for NuGet, the .npmrc / .yarnrc.yml
lines and logins for npm, the repository’s URL for other plugin formats.
Upload limits. Each hosted repository has its own upload limit, the
largest push it accepts (413 above it), whatever its format: set it under
Maximum upload size when creating or editing the repository in the web UI,
or as max_upload_bytes in the admin API (null on an update clears it).
Without one a repository accepts 512 MiB. Every push is staged on disk, so
a limit may be at most what a push can stage, server.max_staging_bytes less
nuget.max_nuspec_bytes (GET /admin/formats reports it as
max_upload_bytes_limit, with each format’s default_max_upload_bytes).
Security model
The host enforces these for every plugin:
- No data beyond the caller’s. The plugin reads only through host queries, and those run with the caller’s package selector and group visibility. An id the format does not validate is unknown, not an error.
- No mutations it was not asked for. The host checks every mutating action against the request’s method, the permission it needs, the repository kind and the credential type, before the hosted write path applies its own selector and redeploy checks.
- No credentials. The plugin never sees a password or a token: a
login’s body goes to the host’s login (
auth), which checks it and mints the token the plugin renders. A body is read only when the plugin asks, bounded, and for a caller with the permission its use needs. - Bounded ids and versions. Whatever the plugin’s own rules, an id longer than 512 bytes or a version longer than 256 bytes, as given or as the plugin normalizes it, is not one: a push of a package carrying one answers 400, a request naming one answers like one naming an unknown package (404), and a proxy leaves such versions of an upstream listing out. Ids and versions are keys of database indexes, whose entries PostgreSQL limits to about 2,700 bytes. NuGet allows less: 100 characters for an id and 64 for a version.
- Upstream requests go where the upstream points. A proxy’s plugin
reaches the network only through the GETs the host sends for it, and the
host sends them, and downloads packages, only from the origin of the
repository’s
remote_urland from origins that the documents it fetched from that upstream name (Proxy repositories). A plugin cannot have Forge readhttp://localhost:8421/admin/…or an internal search cluster that its upstream does not point at, and pass the answers on through the proxy’s metadata. - Responses cannot attack the origin. Only
allow,cache-control,content-disposition,content-language,content-type,etag,last-modified,link,location,retry-afterandvarysurvive;locationandlinksurvive only when they stay inside the repository. Every response gets the feed’s hardening headers (Content-Security-Policy: sandbox,nosniff,Content-Disposition: attachment). A plugin cannot set cookies, CORS headers, or redirect a client elsewhere.
A plugin is part of the server: it is reviewed, tested and released with it, and runs in-process like the rest of Forge’s code.
The documents an upstream serves decide which hosts a proxy’s plugin can
reach. A public feed whose package metadata names an internal URL (a
project URL or a chart’s home page pointing at http://10.0.0.5:9200/) makes
that host reachable for a proxy of that feed, and the plugin reads what it
answers back into what the proxy serves. The rule bounds a plugin to the
hosts its upstream points at, public or internal; where internal services
must stay out of Forge’s reach altogether, confine its egress with network
policy (an outbound proxy, a Kubernetes NetworkPolicy).
Writing a plugin
A plugin is a crate in crates/formats/<id>/, a member of the workspace,
depending on forge-plugin-api and implementing its trait
(crates/formats/nuget is a complete one). It
describes itself with a 'static Descriptor
that descriptor() returns:
use forge_plugin_api::{
Action, Assets, Descriptor, FormatPlugin, Loaded, Repository, RepositoryKind, Request,
Storage, UpstreamToken, /* … */
};
pub const DESCRIPTOR: Descriptor = Descriptor {
id: "example", // the format id repositories are created with
name: "Example packages",
version: env!("CARGO_PKG_VERSION"),
description: "Example packages: push and pull with the example client.",
// The repository kinds it serves (`Proxy` too when it implements
// `upstream`).
kinds: &[RepositoryKind::Hosted, RepositoryKind::Group],
storage: Storage::Packages, // or `Storage::Files` (Files formats)
// or `Storage::Images` (Images formats)
// How a version's files are stored: `package` and `manifest`, both
// application/octet-stream, unless it names its own.
assets: Assets::DEFAULT,
// What a proxy given a password and no username sends upstream:
// nothing more (`None`), or the password as a Bearer token (npm).
upstream_token: UpstreamToken::None,
};
#[derive(Default)]
pub struct MyFormat;
impl FormatPlugin for MyFormat {
fn descriptor(&self) -> &'static Descriptor {
&DESCRIPTOR
}
// identity: canonical_id, validate_id, normalize_version, compare_versions
// packages: parse_package, manifest_summary (optional: readme, reserved_prefix)
// (a files format keeps the defaults of these, which refuse everything)
// protocol: handle, render (optional: error_response, auth_response)
// uploads in a document (optional): unpack_upload, parse_upload, published
// proxies (optional): upstream
}
An id is 2–32 lowercase letters, digits and -, starting with a letter,
and no other format’s. Every non-images format is served under
/repository/{repo}/…; only one format of a server may store images, which
the registry serves at /v2. List Proxy among the kinds only
when the plugin implements upstream. Asset names are plain file names, and
their content types type/subtype.
Then build it in: add the crate to forge-server’s dependencies and to
plugins::builtin() in
crates/server/src/plugins.rs, whose
tests check every built-in format’s descriptor against these rules. A few
rules hold:
- Keep methods pure. The host may call any method from any thread, in any order, any number of times.
- Read packages as a stream.
parse_packagegets aRead + Seekover the staged package. Read only what you need, and bound everything you decompress: a gzip bomb should fail fast. - Ask for data, don’t assume it.
handlereturnsAction::Load(queries);rendergets each query with its answer and may load more, respond, or hand the request to the host (a download, a publish, serving a file). Build absolute URLs from the repository’sbase_url. Groups and kinds are already resolved by the time data reaches you. - Test the protocol without a server.
forge-plugin-api’stestingfeature runshandleandrenderagainst canned answers to the queries (testing::serve), ascrates/formats/nuget/tests/protocol.rsdoes;testing::converserunsupstreamagainst canned upstream documents (crates/formats/nuget/tests/upstream.rs). - Test it in a server. The crate’s unit tests run with the workspace.
crates/server/tests/plugin_contract.rsruns a server with a small plugin of its own installed; copy it for a new plugin.
Limitations and roadmap
These are the current limits of the preview, roughly in the order we expect to lift them:
- Proxies fetch whole documents with GET. A package or files format’s
upstream request carries no headers but the repository’s credentials
(Basic, or a Bearer token) and the
Acceptthe fetch names, and its body is read whole up toproxy.max_metadata_bytesbefore the plugin sees it; only aforwardPOSTs. An upstream that streams a listing too large to hold needs more from the contract. (An images format’s fetches get a Bearer token exchange, and the host streams the blobs and manifests they name: Images formats.) - One package file per version. Formats that publish several files per version (PyPI wheels per platform, Maven’s jar, pom and sources) need multi-asset versions.
- Prerelease is “contains a
-”. This is right for SemVer formats; a format with another rule (PEP 440’s1.0a1) needs a contract function for it. - Listings are bounded. A response may load at most 2,048 answers, so a listing of every package of a repository covers at most about 2,000 packages.
- An upstream document is read whole. A proxy reads each upstream
document it needs (cached for the metadata TTL, and at most
proxy.max_metadata_bytes, 32 MiB by default); the contract has no ranged or streamed reads of an upstream document.
NuGet as a plugin
NuGet is a built-in plugin: crates/formats/nuget
implements FormatPlugin, and the dispatcher serves it at
/repository/{repo}/.... The pieces it needed from the contract are general
ones: its own asset names,
ranged downloads, paged registrations (packages-of), the search options of
the V3 search service, and publishing the first file of a multipart body.
Its proxies read another V3 feed through the plugin’s upstream side
(src/upstream.rs): the service
index’s resources, the flat container, the registration (bounded to the
newest pages a request needs), catalog entries for the package delay of
unlisted versions, and the search service.
npm as a plugin
npm is a built-in plugin too: crates/formats/npm serves
registries at /repository/{repo}/... for npm, pnpm, Yarn and Bun (npm.md).
Contract 0.4.0 grew for it too, and every piece is general:
- Bodies on demand (
read body): logins, audits, dist-tag changes and packument updates are small JSON bodies the plugin reads. - Packages inside a document (
Upload::Document):npm publishsends the tarball base64-encoded inside a JSON document. The plugin’sunpack_uploaddecodes it while the body streams into the store, and the rest of the document (itsdist-tags, its integrity claims) comes back toparse_uploadandpublished. - Tags and deprecations (
change,tags): stored by the host for any format, kept by retention, shown by the web UI. - Logins (
auth): a password login through the realm chain, and a login approved in the browser (/cli-login/<secret>), each minting an API token; a logout revokes it, andwhoaminames the caller. - Forwards: an audit goes to the upstream with the proxy’s credentials, without a group’s reserved names.
- Proxies: tags from the upstream, packages located with their digests,
documents fetched with an
Acceptof their own, and a Bearer token for a registry that wants one. - Groups: a writable member, where publishes and changes through the
group go, and reserved prefixes in the format’s own words (
@acmeis the scope@acme/,reserved_prefix).
Trusted publishing with workload identity
Forge can authenticate a CI job with its short-lived OIDC identity token instead of a stored API key. The token is validated against the configured issuer’s discovery document and JWKS, matched against explicit claim rules, and mapped to an ordinary Forge RBAC role. A matching trusted publisher can add package-scoped write or delete rights on one hosted repository. No user or browser session is created.
A job uses its token in one of two ways:
- Token exchange (recommended). The job sends the token once to
POST /api/v2/tokenand gets a short-lived workload API key back, exactly as nuget.org’s trusted publishing works. It publishes withdotnet nuget push --api-key <key>and restores with the key as the Basic password of the usernameoidc. npm performs its own native exchange automatically. Each token is redeemed once. - Direct token. The job presents the token itself on every request, as the
Basic password (username
oidc), theX-NuGet-ApiKeyor a Bearer token.
The implementation supports workload-only discovery documents containing just
issuer and jwks_uri, including GitHub Actions. Interactive OIDC login keeps
its stricter discovery requirements and remains a separate realm kind.
Security model
A workload token is accepted only when all of these checks pass:
- RSA signature and an explicit, non-empty
kid; a missingkidfails before metadata fetch. An unknownkidmay claim one cooldown-gated JWKS refresh and then fails closed, so random-key traffic cannot force a refetch per request; - exact configured issuer and at least one exact allowed audience (wildcards are rejected);
- required
exp,iat, andsub, plusnbfwhen present; iatis not in the future, token age is withinmax_token_age_seconds, and the declaredexp - iatlifetime is also within that bound;- optional
azpis inallowed_authorized_partieswhen that list is configured; - the first matching rule has at least one explicit claim condition; catch-all
rules are rejected. A rule must also contain a workload identity condition:
issuer/audience/time/ref/environment conditions alone are rejected because
they can match unrelated jobs. For GitHub Actions, every rule must include
both
repository_idandrepository_owner_id; GitLab.com requires a numeric project ID (job_project_idorproject_id) and namespace ID (job_namespace_idornamespace_id). Mutable paths andsubare optional extra boundaries, not identity anchors. Any other issuer (self-managed GitLab, GitHub Enterprise Server, other CI systems) must pin one of the built-in stable identity claims (sub,repository_id,repository_owner_id,repository,repository_owner,job_workflow_ref,workflow_ref,project_id,job_project_id,namespace_id,job_namespace_id,project_path,namespace_path) or a claim the realm declares inidentity_claims(for a CI system with its own workload identifier); claims such asactororevent_namecan be shared by unrelated jobs and never count as an identity. The one exception is an issuer-wide rule ("issuer_wide": true, Every project of this issuer in the web UI), the opt-in for a self-managed issuer whose workloads all may read. It matches every token of the realm’s issuer without an identity condition; conditions on context such asref_protectedmay narrow it, and none is required. Forge refuses it for GitHub Actions and GitLab.com, whose tokens anyone can mint, and a self-managed GitLab with open sign-up is just as public: every account that can create a project and run a pipeline, in a personal namespace or a fork too, gets the role. Rules are first-match, so an issuer-wide rule must come after every other rule. A workload it matched holds its role’s grants capped atread, with the CI token, an exchanged key and the registry tokens minted from either: awriteordeletegrant the role holds, or is given later, never reaches every project. Writes stay pinned to a workload identity: a rule for the project ahead of the issuer-wide one picks a role of its own, and a trusted publisher still addswritefor its own project on top of the issuer-wide read; and - a realm rule must accept the token and map it to a role; a trusted publisher
alone never authenticates a workload. The role continues to grant base access,
commonly
readon hosted and proxy repositories, while a trusted publisher adds write or delete rights only for its hosted repository and matching canonical package IDs.
Positive signature-verification results may be cached only when the token has a
jti. The cache key includes the complete token digest and validation-policy
digest, and its TTL is the smaller of positive_cache_seconds and the remaining
time to exp. Claim rules and role lookup still run for every request.
In the direct-token flow a valid token can be reused for the multiple HTTP
requests a native NuGet operation makes until exp; jti is not a one-time
consumption ledger there. Keep provider token lifetimes short and set
max_token_age_seconds accordingly.
Every successful workload publish — and every unlist, relist and delete a
workload performs — appends a trusted_publish_audit row (its action says
which) in the same PostgreSQL transaction as the package change, behind the
same realm and publisher fences described below. It snapshots repository,
realm, issuer, role, canonical package ID/version, validated token iat, and
SHA-256 digests of subject and jti; raw claims and bearer tokens are not
retained. When a trusted publisher authorized the write, the audit also records
its id and name snapshot; both are null when the role authorized it. Immediately
before commit, Forge locks the current realm and role and compares the complete
normalized realm-config digest with the config that authenticated the request.
It also rechecks matched publisher definition digests. A realm edit revokes that
realm’s keys; editing or deleting a publisher withdraws only its rights from
already issued keys and registry tokens. Repository grants and selectors are
enforced before this hook; they are not re-evaluated by the audit insert itself.
The table has no cascading foreign keys and a database trigger rejects UPDATE
or DELETE, so ordinary application-level repository/realm/role removal preserves
the snapshotted names. This is an internal append-only application ledger, not a
tamper-evident or external transparency log: a database owner can still disable
the trigger or truncate the table, and the row does not capture every provider
claim, matched rule, source commit, or ref. Externally verifiable attestations
remain future work.
Token exchange
POST /api/v2/token takes the job’s token as Authorization: Bearer, with
nuget.org’s request body (optional here):
POST /api/v2/token
Authorization: Bearer <CI OIDC token>
Content-Type: application/json
{"username": "oidc", "tokenType": "ApiKey"}
Add a trusted publisher to the hosted releases repository. GitLab’s protected
environment deploy access determines which jobs may use nuget-release; Forge
checks the signed claims and does not make that provider-side decision:
{
"name": "widgets-release",
"description": "Protected GitLab release environment for Acme.Widgets",
"realm": "gitlab-com",
"packages": { "includes": ["Acme.Widgets.*"], "excludes": [] },
"actions": ["write"],
"conditions": [
{ "claim": "job_project_id", "any_of": ["1001"] },
{ "claim": "job_namespace_id", "any_of": ["2001"] },
{ "claim": "environment", "any_of": ["nuget-release"] },
{ "claim": "environment_protected", "any_of": ["true"] }
]
}
{"apiKey": "forge_wif.…", "expires": "2026-10-21T15:04:05Z"}
- The token passes every check above (the realm’s claim rules pick the role),
and it must carry a
jti: each(issuer, jti)is exchanged once, so a token captured from a job log cannot be redeemed again. A second exchange, a token withoutjti, an API key in place of a token and a token of an issuer no realm trusts all get 401. usernameis optional and, when given, must beoidc;tokenType, when given, must beApiKey(400 otherwise). Identity comes only from the verified token, never from the body.- The key lives for the realm’s
api_key_ttl_seconds(60 to 86400, default 3600, nuget.org’s lifetime), independent of the token’s ownexp. That matters for GitLab, whose token lives as long as the job’s timeout. - The key authorizes as the token’s workload: same realm, role, matched trusted
publishers and audit identity. A publish with it records the exchanged
token’s subject and
jtidigests and itsiatintrusted_publish_audit. - Revocation: a key is honoured only while its realm is enabled with the
very configuration it was issued under. Editing anything in the realm
(rules, audiences, lifetimes, its name) or disabling or deleting it revokes
all of its outstanding keys on the next request, for good: enabling the
realm again or reverting the edit does not bring them back. A key issued
under the new configuration works on every instance at once, also on
those that have not reloaded the realm yet. Deleting the role leaves the
key without role rights. Editing or deleting a trusted publisher similarly
revokes only that publisher’s rights from existing keys and registry tokens;
the realm’s other keys keep working. A registry token
docker login -u oidcgot with a key stops working within 10 seconds of these changes, and expires no later than the key. Forge stores only SHA-256 digests of the key, subject andjti(tableworkload_api_key), and the auth reaper deletes a row once the key has expired and the token has been expired for 15 minutes (the most clock skew a realm accepts, plus a margin). Whether a token has expired is decided on the database’s clock, so an instance whose clock runs behind it cannot redeem a token twice. - A key starts with
forge_wif., which no user API token can (user tokens are URL-safe base64, without a dot). Secret scanners can matchforge_wif\.[A-Za-z0-9_-]{43}. - The key is accepted as
X-NuGet-ApiKey(--api-key, or theNUGET_API_KEYenvironment variable from .NET SDK 10.0.300), as a Bearer token, or as the Basic password of the usernameoidc.
This implements the exchange profile of the OpenSSF Trusted Publishers
guidance: a short-lived, repository-controlled publish credential and
one-time jti redemption. The direct-token flow stays available for clients
that cannot run an exchange step.
The oidc username
oidc is a reserved account name. HTTP Basic with the username oidc (in
any case) is only ever checked as a workload credential: the password must be
a token of an enabled workload_identity realm or a workload API key. Anything
else, including a user’s API token, is refused with 401, and it never reaches
the password realms, so it is never forwarded to a credential-forwarding
upstream. No account can be created, bootstrapped or provisioned under the name
(it is no UPN either), and the browser login refuses it.
Other Basic usernames still work with a direct token, as before. Prefer oidc:
it is the one name that can never mean an account.
GitHub Actions
Create a workload_identity realm under Admin → Authentication. Give its
reader role read access to this hosted repository and the feeds the project
restores from. Its realm rule accepts the project’s jobs and maps them to that
role. Use stable numeric identity claims rather than repository/owner names,
which can be renamed.
Also pin which workflow may publish and how it was triggered:
job_workflow_ref (e.g.
acme/widgets/.github/workflows/release.yml@refs/heads/main) keeps any other
workflow in the repository — including one a contributor adds in a pull
request — from minting a usable token, and event_name (e.g. push or
release) excludes pull_request_target, workflow_dispatch and similar
triggers you did not intend.
{
"issuer": "https://token.actions.githubusercontent.com",
"allowed_audiences": ["https://forge.example.com"],
"clock_skew_seconds": 60,
"max_token_age_seconds": 900,
"positive_cache_seconds": 60,
"rules": [
{
"role": "ci-reader",
"conditions": [
{ "claim": "repository_id", "any_of": ["987654321"] },
{ "claim": "repository_owner_id", "any_of": ["1234567"] }
]
}
]
}
Then add a trusted publisher on the hosted repository at Admin → Repositories → releases → Trusted publishers. This pins publishing to the release workflow while the realm rule above keeps all jobs in the repository eligible for read access:
{
"name": "widgets-release",
"description": "Protected GitHub release workflow for Acme.Widgets packages",
"realm": "github-actions",
"packages": { "includes": ["Acme.Widgets.*"], "excludes": [] },
"actions": ["write"],
"conditions": [
{ "claim": "repository_id", "any_of": ["987654321"] },
{ "claim": "repository_owner_id", "any_of": ["1234567"] },
{ "claim": "ref", "any_of": ["refs/heads/main"] },
{
"claim": "job_workflow_ref",
"any_of": ["acme/widgets/.github/workflows/release.yml@refs/heads/main"]
},
{ "claim": "event_name", "any_of": ["push"] }
]
}
The workflow needs only read access to the checkout plus permission to request
an OIDC token. The official NuGet/login action requests it with the audience
configured above and exchanges it at Forge’s token endpoint. Push to the hosted
repository’s publish URL with --no-service-endpoint: the client then sends a
single PUT carrying the key, without first fetching the private service
index, so no source credentials are needed for the push.
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
- name: Exchange the OIDC token for a Forge API key
uses: NuGet/login@v1
id: forge
with:
user: oidc
audience: https://forge.example.com
token-service-url: https://forge.example.com/api/v2/token
- name: Restore and build
shell: bash
env:
# `forge` is the repository's nuget.config source (no credentials in it).
NuGetPackageSourceCredentials_forge: Username=oidc;Password=${{ steps.forge.outputs.NUGET_API_KEY }}
run: |
dotnet restore
dotnet pack -c Release -o artifacts --no-restore
- name: Publish
shell: bash
run: >-
dotnet nuget push "artifacts/*.nupkg"
--source https://forge.example.com/repository/releases/package
--no-service-endpoint
--api-key "${{ steps.forge.outputs.NUGET_API_KEY }}"
NuGetPackageSourceCredentials_<name> applies to the source of that name in
nuget.config; the name must be a valid environment-variable name (no -),
and NuGet ignores a malformed value without a message.
Without the exchange
Pass the token itself as the NuGet API key and the source password:
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
- name: Request Forge identity token
id: forge-token
shell: bash
run: |
response="$(curl --fail --silent --show-error \
-H "Authorization: Bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
"${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=https%3A%2F%2Fforge.example.com")"
token="$(jq -r '.value' <<<"${response}")"
echo "::add-mask::${token}"
echo "token=${token}" >> "${GITHUB_OUTPUT}"
- name: Publish
shell: bash
env:
FORGE_TOKEN: ${{ steps.forge-token.outputs.token }}
NUGET_CONFIG: ${{ runner.temp }}/forge-nuget-${{ github.run_id }}-${{ github.run_attempt }}.config
run: |
umask 077
printf '%s\n' '<?xml version="1.0" encoding="utf-8"?><configuration><packageSources><clear /></packageSources></configuration>' > "${NUGET_CONFIG}"
chmod 600 "${NUGET_CONFIG}"
dotnet nuget add source \
https://forge.example.com/repository/releases/index.json \
--name forge-oidc \
--username oidc \
--password "${FORGE_TOKEN}" \
--store-password-in-clear-text \
--configfile "${NUGET_CONFIG}"
dotnet nuget push artifacts/*.nupkg \
--source forge-oidc \
--api-key "${FORGE_TOKEN}" \
--configfile "${NUGET_CONFIG}"
- name: Remove temporary Forge credentials
if: always()
shell: bash
env:
NUGET_CONFIG: ${{ runner.temp }}/forge-nuget-${{ github.run_id }}-${{ github.run_attempt }}.config
run: |
if [[ -f "${NUGET_CONFIG}" ]]; then
dotnet nuget remove source forge-oidc --configfile "${NUGET_CONFIG}" >/dev/null 2>&1 || true
rm -f "${NUGET_CONFIG}"
fi
See GitHub’s OIDC reference for the current claim set and subject customization options.
npm
npm 11.5.1 and later can exchange a CI identity token automatically during
npm publish, without a stored _authToken or a curl step. npm 11.21 and
later also supports the exchange for npm dist-tag. Use a current npm release
with this support; pnpm, Yarn and Bun do not perform npm’s exchange.
Configure a workload_identity realm under Admin → Authentication, using
the GitHub Actions or GitLab claim rules shown in this document, but set
allowed_audiences to ["npm:forge.example.com"]. The audience is
npm:<registry hostname>, not the registry URL, port or repository path.
If the realm also serves NuGet, add this audience alongside its existing one.
Give the mapped role write on the hosted npm repository and, when publishing through a group, read on the group and write on its writable member. Package selectors on that role can restrict the package namespace.
GitHub Actions
Use the GitHub issuer and immutable repository/owner IDs from the
GitHub Actions configuration above, with the npm audience.
Do not configure NODE_AUTH_TOKEN or _authToken:
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '26'
- run: npm ci
- run: npm publish --provenance=false --registry https://forge.example.com/repository/npm-all/
npm requests the GitHub job’s ID token with the npm:forge.example.com
audience, exchanges it at Forge and uses the resulting key for the command.
For private dependencies, npm ci still needs separate read credentials;
the automatic exchange is for publishing, not installing.
GitLab CI, including self-hosted GitLab
Use the GitLab realm configuration below with the npm audience.
GitLab supplies the token through NPM_ID_TOKEN:
publish:
image: mirror.gcr.io/library/node:26
timeout: 10m
id_tokens:
NPM_ID_TOKEN:
aud: npm:forge.example.com
script:
- npm ci
- npm publish --registry https://forge.example.com/repository/npm-all/
rules:
- if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_COMMIT_REF_PROTECTED == "true"'
Self-hosted GitLab works the same way: set the realm’s issuer to the exact
iss in that instance’s job tokens, for example https://gitlab.example.com,
instead of https://gitlab.com. Pin numeric project and namespace IDs in the
claim rules, plus protected-ref or environment restrictions. Older GitLab
versions without job_project_id / job_namespace_id can use project_id /
namespace_id. Forge must be able to reach that issuer’s HTTPS discovery
document and JWKS and trust its TLS certificate, including for a private
instance. Do not enable allow_http_issuer for production. Set
max_token_age_seconds to cover the job timeout, as described below.
Exchange and limitations
npm calls POST /-/npm/v1/oidc/token/exchange/package/<escaped-name> at the
host root, even when the registry is /repository/npm-all/. Reverse proxies must
forward this root path to Forge, not only /repository/. The answer is
{"token":"forge_wif.…"}; invalid requests or credentials receive a JSON
error and 400 or 401 (temporary authentication failures receive 503).
Both exchange endpoints share the same one-redemption-per-(issuer, jti)
ledger, realm checks and api_key_ttl_seconds. The named package does not
scope the issued key: the realm’s role and matched trusted publishers authorize
its use. The exchange has no repository, so a key may be issued even when the
workload has no publisher for that package; its publish is refused with 403.
npm silently falls back to configured credentials when the exchange fails;
omit stored tokens in jobs intended to require trusted publishing. Forge
does not serve npm’s package-visibility endpoint. npm 11.5.1 can enable
provenance automatically for public GitHub repositories without checking that
endpoint, so pass --provenance=false to npm publish for those clients.
Forge rejects provenance statements until attestations can be stored.
GitLab CI
GitLab CI with workload identity walks through this setup step
by step, from the role to the .gitlab-ci.yml;
CI with service accounts covers pipelines that
cannot use workload identity.
Give the realm’s reader role read access to the hosted repository and the
feeds the project restores from. The realm rule accepts jobs from the project,
so every pipeline can restore; a separate trusted publisher limits writes to
the protected release environment and selected packages.
{
"issuer": "https://gitlab.com",
"allowed_audiences": ["https://forge.example.com"],
"clock_skew_seconds": 60,
"max_token_age_seconds": 39600,
"positive_cache_seconds": 60,
"rules": [
{
"role": "ci-reader",
"conditions": [
{ "claim": "job_project_id", "any_of": ["1001"] },
{ "claim": "job_namespace_id", "any_of": ["2001"] }
]
}
]
}
GitLab gives an ID token the job’s timeout as its lifetime, so
max_token_age_seconds must be at least the longest timeout of a job that
publishes. The default, 39600 (11 hours), covers jobs up to 11 hours; set a
lower value when your release jobs are shorter. The exchanged key lives for the
realm’s api_key_ttl_seconds instead, whatever the job’s timeout.
Ask GitLab to mint an ID token with that exact audience, exchange it, and push
to the publish URL with --no-service-endpoint (see GitHub Actions above):
publish:
environment:
name: nuget-release
id_tokens:
FORGE_ID_TOKEN:
aud: https://forge.example.com
script:
- >-
export NUGET_API_KEY="$(curl --fail --silent --show-error
-X POST https://forge.example.com/api/v2/token
-H "Authorization: Bearer $FORGE_ID_TOKEN"
-H 'Content-Type: application/json'
-d '{"username":"oidc","tokenType":"ApiKey"}' | jq -r .apiKey)"
- export NuGetPackageSourceCredentials_forge="Username=oidc;Password=$NUGET_API_KEY"
- dotnet restore
- dotnet pack -c Release -o artifacts --no-restore
- >-
dotnet nuget push "artifacts/*.nupkg"
--source https://forge.example.com/repository/releases/package
--no-service-endpoint
--api-key "$NUGET_API_KEY"
Without the exchange
publish:
id_tokens:
FORGE_ID_TOKEN:
aud: https://forge.example.com
script:
- export NUGET_CONFIG="$CI_PROJECT_DIR/.forge-nuget-$CI_JOB_ID.config"
- umask 077
- printf '%s\n' '<?xml version="1.0" encoding="utf-8"?><configuration><packageSources><clear /></packageSources></configuration>' > "$NUGET_CONFIG"
- chmod 600 "$NUGET_CONFIG"
- >-
dotnet nuget add source
https://forge.example.com/repository/releases/index.json
--name forge-oidc
--username oidc
--password "$FORGE_ID_TOKEN"
--store-password-in-clear-text
--configfile "$NUGET_CONFIG"
- >-
dotnet nuget push artifacts/*.nupkg
--source forge-oidc
--api-key "$FORGE_ID_TOKEN"
--configfile "$NUGET_CONFIG"
after_script:
- export NUGET_CONFIG="$CI_PROJECT_DIR/.forge-nuget-$CI_JOB_ID.config"
- dotnet nuget remove source forge-oidc --configfile "$NUGET_CONFIG" >/dev/null 2>&1 || true
- rm -f "$NUGET_CONFIG"
See GitLab’s ID-token authentication reference for provider claim semantics.
In the direct-token examples the temporary Basic source is intentional. The
.NET NuGet client resolves the service index before it sends the push API key,
and --api-key is not attached to that initial GET (pushing to the publish URL
with --no-service-endpoint, as the exchange examples do, skips that GET). Supplying the same short-lived OIDC token as the source
password authenticates both phases; the cleanup step removes it even after a
failed push. The examples use a job-unique, mode-0600 config rather than the
runner’s default/global NuGet.Config, so concurrent jobs cannot reuse it and
ordinary failure cleanup removes the bearer. Cleanup is best-effort: hard runner
termination can leave the file until the token expires, so prefer ephemeral
runners. An anonymous_read repository can omit this temporary source and push
directly by URL, but that deliberately exposes the repository’s read-side
metadata according to the configured anonymous policy.
Operational guidance
- Give each CI trust policy its own repository-scoped role. Forge rejects WIF
rules that map to the built-in global
adminrole. - Keep all rules for one issuer in one enabled realm. Forge rejects duplicate enabled WIF issuers so routing cannot depend on realm order.
- Grant a selector-scoped
writelevel (which includesread) on the intended hosted repository, using an include/exclude package selector such asAcme.Widgets.*. Thereadit includes serves a private feed’s service-index GET; it does not widen access beyond that selector on package-addressed routes. - Prefer numeric provider IDs and add a protected ref/environment condition.
A ref or environment is an additional boundary, not a workload identity by
itself. Forge does not require one: a rule that pins only the repository or
project lets any branch and any workflow of it publish (bypassing branch
protection), including
pull_request_targetruns and, on GitLab, a pipeline for a fork’s merge request run in the parent project. On GitHub addjob_workflow_refplus a protected ref or anenvironmentto every publishing rule; GitLab tokens have nojob_workflow_ref, so there addref_protected = trueor anenvironment, and prefer thejob_*ids:project_idandnamespace_idname a merge request’s source project. - Use a dedicated, exact audience for Forge; never reuse a broad cloud audience.
- Set
positive_cache_seconds = 0to disable the per-token positive verdict cache. Discovery and JWKS have their own caches (JWKS defaults to one hour), so this does not force signing-key revocation to be fetched on every request. The positive verdict cache itself never extends past its configured TTL or token expiry. - When the issuer’s discovery document or JWKS cannot be fetched (a timeout,
a refused connection, a 5xx) and is not cached yet, a workload token
cannot be checked: the exchange, the feeds and
/v2/tokenanswer 503 withRetry-Afterrather than 401, so clients that retry do. This happens on a cold cache only: after a restart, on a new replica, or when the JWKS cache expires (hourly by default) during the issuer’s outage. - Treat a token as a bearer credential until
exp, and a workload key until itsexpires: mask both in CI logs and do not persist them as an artifact, cache entry, or repository secret. Exchange right before restoring or publishing; a key outlives the job only up toapi_key_ttl_seconds, so lower it when your jobs are short. - To revoke every outstanding workload key of a realm at once, edit or disable the realm; the keys stay revoked when you enable it again.
GitLab CI with workload identity
This guide connects a GitLab project’s NuGet builds to Forge without storing a secret in GitLab. Every job of the project can restore from Forge’s feeds, and only the release job can push, and only the packages you name:
| Job | Identity | Can |
|---|---|---|
| Every job (build, test, merge requests) | the realm rule’s read-only role | restore from the feeds the role reads |
| The release job | the same role, plus a trusted publisher | restore, and push the named packages to one hosted repository |
Both identities come from the same GitLab ID token and the same Forge realm. A
job holds exactly one role, the one the realm’s first matching rule names,
and an issuer has exactly one enabled realm. The write identity is therefore
no second realm or role: it is a trusted publisher on the hosted
repository, which adds write there for the jobs its conditions match. A job
that matches no publisher keeps the read-only role.
Trusted publishing is the reference behind this guide: the security model, the token exchange and every setting. For pipelines that cannot use workload identity, see CI with service accounts.
The examples use these names; substitute your own:
| Name | Example |
|---|---|
| Forge | https://forge.example.com |
| GitLab, the token issuer | https://gitlab.example.com (or https://gitlab.com) |
| Hosted repository the packages go to | releases |
| Proxy of nuget.org | nuget-org |
Group the builds restore from (releases + nuget-org) | nuget |
| GitLab project ID / namespace ID | 1001 / 42 |
| Packages the project publishes | Acme.Widgets, Acme.Widgets.* |
Before you start
-
GitLab issues ID tokens (the
id_tokenskeyword, GitLab 15.7 and later). Forge must reach the issuer’s/.well-known/openid-configurationand its JWKS over HTTPS and trust the certificate, also for a GitLab on a private network. Plain HTTP is for test instances only. -
You know the issuer and the IDs. The issuer is the exact
issof the instance’s tokens, normally the instance URL ($CI_SERVER_URLin a job). The project ID is on the project’s overview page (⋮ → Copy project ID) and is$CI_PROJECT_IDin a job. The namespace ID is the ID of the group the project sits in directly,$CI_PROJECT_NAMESPACE_IDin a job. -
Which claims your GitLab puts in its tokens. Current GitLab versions carry
job_project_idandjob_namespace_id; older self-managed versions have onlyproject_idandnamespace_id. If in doubt, run this job once and read its log. It prints the token’s claims without its signature, which is no credential; delete the job afterwards, since the claims name the user who ran it.show-forge-claims: image: mcr.microsoft.com/dotnet/sdk:10.0 when: manual id_tokens: FORGE_ID_TOKEN: aud: https://forge.example.com script: - | payload="$(printf '%s' "$FORGE_ID_TOKEN" | cut -d. -f2 | tr '_-' '/+')" while [ $(( ${#payload} % 4 )) -ne 0 ]; do payload="$payload="; done printf '%s' "$payload" | base64 -d echo -
The repositories exist in Forge (Admin → Repositories): the hosted NuGet repository the packages go to, and the feeds the builds restore from.
1. The read-only role
Under Admin → Roles, create a role, for example gitlab-ci-read. On its
page, under Grants, add one grant per repository the builds restore from,
each with the level read:
releases, so builds can use the project’s own packages;nuget-org;nuget, the group.
A group serves a caller only the members that caller may read, so grant
read on the group and on each member it should serve from. Leave
All repositories alone unless every job of the project may read every
repository: a role’s grants reach every format, npm, Docker and raw
repositories as well as NuGet.
The role needs no write: the trusted publisher in step 4 adds it for the
release job only.
2. The workload identity realm
Under Admin → Authentication, create a realm of the kind Workload identity:
| Field | Value |
|---|---|
| Name | gitlab (the trusted publisher names the realm) |
| Issuer URL | https://gitlab.example.com, exactly the tokens’ iss |
| Allowed audiences | https://forge.example.com, an audience used for Forge alone |
| Maximum token age (seconds) | at least the longest timeout of a job that uses Forge; the default, 39600 (11 hours), covers most |
| Exchanged API key lifetime (seconds) | how long a job may use its key; the default is 3600 (1 hour) |
Under Rules, add one rule with the role gitlab-ci-read and two
conditions:
| Claim | Any of |
|---|---|
job_project_id | 1001 |
job_namespace_id | 42 |
Every job of project 1001 now matches the rule and gets gitlab-ci-read; a
job of any other project gets 401. The realm’s configuration, as the admin
API takes it (POST /admin/realms):
{
"kind": "workload_identity",
"name": "gitlab",
"config": {
"issuer": "https://gitlab.example.com",
"allowed_audiences": ["https://forge.example.com"],
"max_token_age_seconds": 39600,
"api_key_ttl_seconds": 3600,
"rules": [
{
"role": "gitlab-ci-read",
"conditions": [
{ "claim": "job_project_id", "any_of": ["1001"] },
{ "claim": "job_namespace_id", "any_of": ["42"] }
]
}
]
}
}
Instead of filling in the rule field by field, paste this document into Edit as JSON, at the top of Rules, and apply it. The dialog keeps only the rules of what it is given (the rules alone, a realm’s config, a whole realm, or one rule) and fills in the form with them; saving the realm saves them. It also shows a realm’s rules as JSON, to copy them into another realm or another Forge.
- Numeric IDs, not paths. A project or group can be renamed, and another
can take its old path. On
https://gitlab.comevery rule must contain a project ID and a namespace ID; a self-managed GitLab accepts any stable identity claim, but pin both IDs there too. job_*claims where your GitLab has them. In a merge request pipeline run in the parent project,project_idandnamespace_idname the merge request’s source project,job_project_idandjob_namespace_idthe project the job runs in. Useproject_idandnamespace_idonly on a GitLab without thejob_*claims.- More projects go into the same realm: add their IDs to the rule’s lists, or add a rule per team with a role of its own. Rules are ordered, and the first rule whose conditions all match picks the role. On a self-managed GitLab, a last rule can take every project of the instance, read-only.
- Saving the realm revokes its keys. Any change to the realm revokes every key it has issued, so jobs running at that moment fail and must be retried.
3. Who may release, in GitLab
The write identity trusts what GitLab signs into the token, so GitLab decides who may run the release job. Protect the ref the release runs on, and with GitLab Premium or Ultimate also the environment it deploys to:
- Protected tags (Settings → Repository → Protected tags): protect
v*and choose who may create such tags, for example Maintainers. A job of a pipeline for a protected tag getsref_protected: truein its token. - Protected environment (Settings → CI/CD → Protected environments):
protect
nuget-releaseand choose who may deploy to it. A job that deploys to it getsenvironment: nuget-releaseandenvironment_protected: true. The environment must exist first: the first job that names it creates it, or create it under Operate → Environments.
4. The trusted publisher
Open the hosted repository under Admin → Repositories → releases, and in its Trusted publishers panel choose Add trusted publisher:
| Field | Value |
|---|---|
| Publisher name | widgets-release |
| Realm | gitlab |
| Description | who owns it and what it releases |
| Package includes | Acme.Widgets and Acme.Widgets.*, one glob per line |
| Package excludes | empty |
| Claim conditions | the JSON below |
| Allow unlist and delete | off |
[
{ "claim": "job_project_id", "any_of": ["1001"] },
{ "claim": "job_namespace_id", "any_of": ["42"] },
{ "claim": "ref_type", "any_of": ["tag"] },
{ "claim": "ref_protected", "any_of": ["true"] },
{ "claim": "environment", "any_of": ["nuget-release"] },
{ "claim": "environment_protected", "any_of": ["true"] }
]
On GitLab Free, which has no protected environments, leave out the two
environment conditions; the protected tag then decides alone.
- Pin the packages. An empty include list means every package. Patterns match the package ID without regard to case.
- The publisher only adds. It never authenticates a job by itself: a job
must match the realm rule first, and then holds
gitlab-ci-readpluswriteonreleasesfor the included packages. ref_protectedbelongs next to the environment. It keeps a merge request pipeline from a fork, which a maintainer may run in the parent project, from publishing even when it deploys to the environment: a merge request’s ref is never protected.- Unlist and delete add the
deletelevel for the same packages. Leave them off unless the pipeline removes versions. - Editing or deleting the publisher withdraws its rights from the keys already issued; the realm’s other keys keep working.
The admin API takes the same publisher at
POST /admin/repositories/releases/trusted-publishers:
{
"name": "widgets-release",
"description": "Tag releases of Acme.Widgets from the protected nuget-release environment",
"realm": "gitlab",
"packages": { "includes": ["Acme.Widgets", "Acme.Widgets.*"], "excludes": [] },
"actions": ["write"],
"conditions": [
{ "claim": "job_project_id", "any_of": ["1001"] },
{ "claim": "job_namespace_id", "any_of": ["42"] },
{ "claim": "ref_type", "any_of": ["tag"] },
{ "claim": "ref_protected", "any_of": ["true"] },
{ "claim": "environment", "any_of": ["nuget-release"] },
{ "claim": "environment_protected", "any_of": ["true"] }
]
}
5. The project
The project’s nuget.config names Forge’s group feed and holds no
credentials:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear />
<add key="forge" value="https://forge.example.com/repository/nuget/index.json" />
</packageSources>
</configuration>
In .gitlab-ci.yml, a hidden job requests the ID token and exchanges it for a
key in before_script; every job that uses Forge extends it:
variables:
FORGE_URL: https://forge.example.com
.forge:
image: mcr.microsoft.com/dotnet/sdk:10.0
id_tokens:
FORGE_ID_TOKEN:
aud: https://forge.example.com
before_script:
- |
FORGE_KEY="$(curl --fail --silent --show-error -X POST "$FORGE_URL/api/v2/token" \
-H "Authorization: Bearer $FORGE_ID_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"username":"oidc","tokenType":"ApiKey"}' \
| sed -n 's/.*"apiKey":"\([^"]*\)".*/\1/p')"
test -n "$FORGE_KEY"
export FORGE_KEY
export NuGetPackageSourceCredentials_forge="Username=oidc;Password=$FORGE_KEY"
build:
extends: .forge
script:
- dotnet restore
- dotnet build -c Release --no-restore
- dotnet test -c Release --no-build
publish:
extends: .forge
stage: deploy
environment:
name: nuget-release
rules:
- if: $CI_COMMIT_TAG
script:
- dotnet restore
- dotnet pack -c Release -o artifacts --no-restore
- >-
dotnet nuget push "artifacts/*.nupkg"
--source "$FORGE_URL/repository/releases/package"
--no-service-endpoint
--api-key "$FORGE_KEY"
-
The audience under
id_tokensmust be one of the realm’s allowed audiences, exactly. -
The exchange runs once per job. A token can be exchanged only once, and the job’s later commands share the key.
FORGE_KEY="$(…)"stays a plain assignment so that a failed exchange fails the job:export FORGE_KEY="$(…)"would succeed whatevercurlreturned, and the job would go on without a key. The .NET SDK image hascurlbut nojq, sosedpicks the key out of the answer,{"apiKey":"forge_wif.…","expires":"…"}. -
A job with a
before_scriptof its own replaces the hidden job’s, and then has no key. List the hidden job’s first:test: extends: .forge before_script: - !reference [.forge, before_script] - dotnet tool restore script: - dotnet test -
NuGetPackageSourceCredentials_forgegives theforgesource ofnuget.configthe key as its password; the part after the_must be the source’s key. NuGet ignores a malformed value without a message. -
The push goes to the hosted repository’s publish URL with
--no-service-endpoint, so NuGet sends a singlePUTwith the key instead of first reading the service index. -
The publish job restores with the read-only role, which every job of the project holds, and pushes with the write the trusted publisher adds.
Every project on a runner
The hidden job above lives in each project. An organization that runs its own
GitLab Runner for its .NET builds can run the exchange on the runner instead,
once for every job it builds. Each job then restores with the realm rule’s
read-only role, and only a job that a trusted publisher matches can push. The
runner’s pre_build_script runs in the job’s shell right before the job’s
before_script and script, so the key it exports reaches the job.
The runner cannot ask for the token itself: GitLab issues an ID token only to
a job that declares id_tokens. Each project keeps that part.
The realm rule
The rule of step 2 names every project the runner builds. On a self-managed GitLab a rule may pin namespaces alone: every project directly in one of the groups matches, and so does a project created there later.
{
"role": "gitlab-ci-read",
"conditions": [
{ "claim": "job_namespace_id", "any_of": ["42", "43", "57"] }
]
}
- A subgroup is a namespace of its own.
job_namespace_idis the group a project sits in directly, so a group’s ID does not cover its subgroups: list each of them. - On
https://gitlab.comevery rule needs project IDs as well: list the projects there. - Keep the role read-only. Every job of those groups holds it, merge request pipelines included.
- Rules are ordered. A project that needs a role of its own goes into a rule before this one.
Every project of the instance
On a self-managed GitLab whose projects all may restore, one rule takes every job the instance signs a token for, a project or subgroup created later included, where a list of groups must be kept current:
{ "role": "gitlab-ci-read", "issuer_wide": true, "conditions": [] }
In the web UI it is the rule’s Every project of this issuer switch.
- Everyone who runs a pipeline gets the role. Every account that can create a project and run a pipeline on the instance reads with it, in a personal namespace or a fork too. A GitLab with open sign-up is as public as GitLab.com: close sign-up, or pin the groups as above.
- Read-only, whatever the role grants. A job this rule matched holds the
role’s grants at
read, and so do its exchanged key and the registry tokens minted from either: awriteordeletegrant on the role does not apply to it, so a role givenwritelater does not hand it to every project. - Projects that publish keep their trusted publisher
(step 4), which adds
writefor the project’s own packages on top, or get a rule of their own ahead of this one, whose role may write. - Last. Rules are first-match, so this rule comes after every other rule; Forge refuses the realm otherwise.
- Conditions narrow it. Add context conditions, such as
ref_protectedtruefor the pipelines of protected branches and tags only; none is required. - Not on
https://gitlab.com(nor for GitHub Actions): anyone can mint its tokens, and Forge refuses the rule there.
The runner
In the runner’s config.toml, in its [[runners]] entry:
[[runners]]
# name, url, token, executor, …
environment = ["FORGE_URL=https://forge.example.com"]
pre_build_script = '''
if [ -n "${FORGE_ID_TOKEN:-}" ] &&
[ -n "$(find "$CI_PROJECT_DIR" -maxdepth 3 -iname nuget.config -exec grep -liF "$FORGE_URL" {} + 2>/dev/null)" ]; then
FORGE_KEY="$(curl --fail --silent --show-error -X POST "$FORGE_URL/api/v2/token" \
-H "Authorization: Bearer $FORGE_ID_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"username":"oidc","tokenType":"ApiKey"}' \
| sed -n 's/.*"apiKey":"\([^"]*\)".*/\1/p')"
test -n "$FORGE_KEY"
export FORGE_KEY
export NuGetPackageSourceCredentials_forge="Username=oidc;Password=$FORGE_KEY"
fi
'''
- Only jobs with a token.
FORGE_ID_TOKENexists only in a job that asks for it, so the runner leaves every other job alone. - Only projects that use Forge. The exchange runs only when a
nuget.configup to three directories deep names$FORGE_URL, whatever the case of the file’s name. Thefindlists nothing both when there is no such file and when none of them names Forge. - A failed exchange fails the job, as in the hidden job:
test -nstops it beforedotnet restoregets 401. - The image of such a job needs
curl,sed,findand a POSIX shell; the .NET SDK image has them. A Windows runner needs a PowerShell version. - The source key is
forgein every project’snuget.config:NuGetPackageSourceCredentials_forgereaches that source alone. - Your own runners only. GitLab.com’s instance runners take no configuration of yours: register a project, group or instance runner of your own.
The projects
Each project asks for the token in every job, and runs no exchange of its own:
default:
image: mcr.microsoft.com/dotnet/sdk:10.0
id_tokens:
FORGE_ID_TOKEN:
aud: https://forge.example.com
build:
script:
- dotnet restore
- dotnet build -c Release --no-restore
- dotnet test -c Release --no-build
- No
extends: .forge. A token is exchanged once, so the hidden job’s exchange would get 401 after the runner’s. - The release job of step 5 stays as it is, without
extends: .forge: it pushes with the$FORGE_KEYthe runner exported, which the trusted publisher of step 4 giveswrite. - The jobs run on that runner: give them its tags, or let it pick up untagged jobs.
The runner writes the NuGet configuration
The runner above exchanges only for a project whose own nuget.config names
Forge. It can write the configuration itself instead, so that a project needs
none: a job with a token gets Forge’s feed and the key, and every other job a
NuGet.Config of your own, which an instance CI/CD variable, CI_NUGET_CONFIG,
holds base64-encoded.
[[runners]]
# name, url, token, executor, …
environment = ["FORGE_URL=https://forge.example.com"]
pre_build_script = '''
nuget_config="$HOME/.nuget/NuGet/NuGet.Config"
if [ -n "${FORGE_ID_TOKEN:-}" ]; then
FORGE_KEY="$(curl --fail --silent --show-error -X POST "$FORGE_URL/api/v2/token" \
-H "Authorization: Bearer $FORGE_ID_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"username":"oidc","tokenType":"ApiKey"}' \
| sed -n 's/.*"apiKey":"\([^"]*\)".*/\1/p')"
test -n "$FORGE_KEY"
export FORGE_KEY
export NuGetPackageSourceCredentials_forge="Username=oidc;Password=$FORGE_KEY"
mkdir -p "$(dirname "$nuget_config")"
cat > "$nuget_config" <<EOF
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<clear />
<add key="forge" value="$FORGE_URL/repository/nuget/index.json" />
</packageSources>
</configuration>
EOF
elif [ -n "${CI_NUGET_CONFIG:-}" ]; then
mkdir -p "$(dirname "$nuget_config")"
printf '%s' "$CI_NUGET_CONFIG" | base64 -d > "$nuget_config"
fi
'''
- The user’s configuration. The runner writes
~/.nuget/NuGet/NuGet.Config, the file NuGet reads beneath every project’s own, in place of the one the image brings (the .NET SDK image’s names nuget.org). A project without anuget.configrestores from what the runner wrote. A project’s ownnuget.configstill comes first: one with<clear />drops the runner’s sources, and then names Forge under the keyforgeitself, as in step 5. - A job with a token gets Forge’s group feed under the key
forge. Its key goes inNuGetPackageSourceCredentials_forge, as above, and stays out of the file. The token decides alone: a project that asks for it uses Forge, whether or not it has anuget.config. - A failed exchange fails the job. The job does not fall back to
CI_NUGET_CONFIG: a job that asks for a token is meant to reach Forge with it. - Every other job gets
CI_NUGET_CONFIG, decoded as it is: the jobs of a project that asks for no token. Add the variable under Admin → Settings → CI/CD → Variables, or in a group’s CI/CD variables for that group’s projects alone;base64 -w0 NuGet.Configprints its value. A value that is no base64 fails the job. While the variable is unset, a job without a token keeps the image’s configuration. CI_NUGET_CONFIGis a secret when it holds credentials. An instance variable reaches every job of every project on every runner, not only this one’s, and any of those jobs can print it, as with the runner’s environment. Give a token in it the read level alone and an expiry.- A container per job. With the Docker or Kubernetes executor every job
runs in a fresh container, and the file and the key go with it. A shell
executor shares
$HOMEbetween all its jobs, concurrent ones too: they would overwrite each other’s configuration, and the last one’s would stay. Use this runner only with an executor that gives each job a container. - The image of a job needs
curl,sed,base64and a POSIX shell; the .NET SDK image has them.
Without workload identity
A runner can also hand every job a service account’s read token, with no change to the projects:
[[runners]]
environment = ["NuGetPackageSourceCredentials_forge=Username=gitlab-ci@example.com;Password=<read token>"]
The token is then a long-lived secret in the runner’s configuration, and every job the runner runs can print it, whatever project the job belongs to. Give it the read level alone, an expiry, and only the repositories the builds restore from (step 3 there).
What each identity can do
| Every job | The release job | |
|---|---|---|
Restore from releases, nuget-org and nuget | yes | yes |
Push Acme.Widgets* to releases | no, 403 | yes |
| Push another package, or to another repository | no | no |
| Unlist or delete | no | only with Allow unlist and delete |
| The admin API, the web UI, API tokens | never | never |
- No account. A job is a workload, not a user: it signs in nowhere, and
/auth/tokensneeds a browser session, so it cannot mint an API token. Forge refuses a realm rule that maps to the built-inadminrole, so no token reaches the admin API. - Public reads. Like every caller, a job may also read what anonymous access allows.
- Lifetimes. The key lives for the realm’s exchanged API key lifetime, whatever the job’s timeout. The ID token itself lives until the job’s timeout but can be exchanged only once.
- Revoking. Disabling or editing the realm revokes all its keys for good;
enabling it again does not bring them back. Removing a grant from
gitlab-ci-readtakes effect on the next request. - Audit. Every publish by a job is recorded in
trusted_publish_auditwith the realm, the role, the publisher and digests of the token’s subject andjti.
Troubleshooting
| Symptom | Cause |
|---|---|
| The exchange answers 401 | No rule matched: check the issuer (the tokens’ exact iss), the audience, and the IDs; on an older GitLab use project_id / namespace_id. Or the token is older than the maximum token age, or was exchanged already (a retried curl, a second exchange in the job, the project’s exchange after the runner’s). |
| The exchange answers 503 | Forge cannot fetch GitLab’s discovery document or JWKS: the network between them, a proxy, or a certificate Forge does not trust. |
dotnet restore gets 401 | NuGetPackageSourceCredentials_<key> does not name the nuget.config source, or the key expired during a long job: raise the exchanged API key lifetime. |
| A package from a group member is missing | The role reads the group but not that member. |
| The push gets 403 | The job did not match the publisher (not a protected tag, not the protected environment), or the package ID is outside its includes. |
| The push gets 405 | It went to a proxy, or to a group that names no writable member: push to /repository/releases/package. |
| The push gets 401 | The key was revoked by an edit of the realm, or expired. |
Without the exchange
A client that cannot run the exchange can present the ID token itself on
every request, as the Basic password of the username oidc and as the API
key. Trusted publishing shows
the GitLab job; the exchange is preferred, since the token then cannot be
replayed from a job log.
CI with service accounts
Some pipelines cannot use workload identity: a GitLab older than 15.7, a GitLab whose discovery document and JWKS Forge cannot reach, a CI system without OIDC tokens. They authenticate with an API token of a service account instead: an account for a machine that nobody signs in to.
Prefer workload identity wherever it works. A token stored in GitLab is a long-lived secret: any job that sees it can use it, until it expires or is revoked, from anywhere. This guide keeps the same split as the workload identity guide, so that a leaked token reveals as little as possible:
| Token | Account | Stored in GitLab | Can |
|---|---|---|---|
| Read token | gitlab-ci@example.com | masked variable, every pipeline | restore from the feeds the account reads |
| Release token | gitlab-release@example.com | masked and protected variable, protected refs only | push the named packages to one hosted repository |
The examples use the names of the workload identity guide:
Forge at https://forge.example.com, the hosted repository releases, the
proxy nuget-org, the group nuget, and the packages Acme.Widgets and
Acme.Widgets.*.
1. The roles
Reading. The read-only role of the workload identity guide serves here
too: gitlab-ci-read, with read on releases, nuget-org and nuget
(see step 1 there). One role for both
methods keeps what CI may read in one place; give each method its own role
when they should differ.
Publishing. Pin the packages with a content selector, then grant write
through it:
- Under Admin → Content selectors, create
acme-widgetswith the includesAcme.WidgetsandAcme.Widgets.*. - Under Admin → Roles, create
widgets-publish, and on its page add the grant: repositoryreleases, level write, content selectoracme-widgets.
Without the selector, the grant reaches every package of releases.
2. The service accounts
Under Admin → Service accounts, create two accounts:
| Name | Description | Access | Roles (on its page) |
|---|---|---|---|
gitlab-ci@example.com | restores for the GitLab project acme/widgets; who owns it | Reader | gitlab-ci-read |
gitlab-release@example.com | releases Acme.Widgets from acme/widgets; who owns it | Reader | widgets-publish |
Reader gives an account no built-in role: it may do what its roles allow.
Writer would let it publish anything to every repository; leave it for
accounts that really need it. A service account never holds the admin role.
Two accounts rather than one keep the release rights out of the read token, and the logs tell restores and releases apart.
3. The tokens
On each account’s page, under Tokens, create a token:
| Read token | Release token | |
|---|---|---|
| Token name | gitlab acme/widgets | gitlab acme/widgets release |
| Token expiry | 90 days, or 1 year | 90 days, or 1 year |
| Level | at most read | at most write |
| Repositories | every repository the owner can reach | only releases |
The secret is shown once: put it into GitLab straight away. A token’s scope only narrows its account’s roles. Here it keeps the tokens to their purpose even when someone later gives the account a wider role.
4. The GitLab variables
In the project, under Settings → CI/CD → Variables, add:
| Key | Value | Settings |
|---|---|---|
FORGE_READ_TOKEN | the read token | Masked, not protected |
FORGE_RELEASE_TOKEN | the release token | Masked and Protected |
A protected variable reaches only pipelines of protected branches and tags.
Protect the release tags (Settings → Repository → Protected tags, for
example v*, created by Maintainers only), so the release token reaches the
pipeline of such a tag and nothing else. The read token is in every pipeline,
including those of branches anybody with the Developer role can push: anyone
who can change .gitlab-ci.yml can print it. It carries read alone for
that reason.
5. The pipeline
nuget.config is the one of the workload identity guide, naming the source
forge without credentials. The jobs pass the read token as the source’s
password and the release token as the push’s API key:
variables:
FORGE_URL: https://forge.example.com
.forge:
image: mcr.microsoft.com/dotnet/sdk:10.0
before_script:
- export NuGetPackageSourceCredentials_forge="Username=gitlab-ci@example.com;Password=$FORGE_READ_TOKEN"
build:
extends: .forge
script:
- dotnet restore
- dotnet build -c Release --no-restore
- dotnet test -c Release --no-build
publish:
extends: .forge
stage: deploy
rules:
- if: $CI_COMMIT_TAG
script:
- dotnet restore
- dotnet pack -c Release -o artifacts --no-restore
- >-
dotnet nuget push "artifacts/*.nupkg"
--source "$FORGE_URL/repository/releases/package"
--no-service-endpoint
--api-key "$FORGE_RELEASE_TOKEN"
- The username in the source credentials is the account’s name. Forge
identifies the account by the token, but the name
oidcis reserved for workload identity and refuses every API token. - The push goes to the hosted repository’s publish URL with
--no-service-endpoint: NuGet then sends a singlePUTwith the API key, without reading the service index first.
What each token can do
- Its account’s roles, narrowed by its scope. The read token restores
from
releases,nuget-organdnuget, and pushes nothing (403). The release token pushesAcme.Widgets*toreleases, and nothing else. - Never more than an API token. Nobody signs in to a service account, it has no password, and its tokens reach no admin API.
- Until it ends. A token works until its expiry, until it is revoked on the account’s page, or while the account is deactivated (Deactivate… on its page stops all its tokens at once). The list of service accounts shows when each was last used.
Rotating a token: create a new one on the account’s page, replace the variable’s value in GitLab, run a pipeline, then revoke the old token. With an expiry on every token, a forgotten one ends by itself.
Using both
Workload identity and service accounts work side by side, without any setting: Forge tells the credentials apart.
- HTTP Basic with the username
oidcis only ever a workload credential. - A JWT whose issuer has a workload identity realm goes to that realm, and a failure there is final: it never falls through to the password checks.
- Anything else, such as a service account’s token, is checked as an API token.
So each project can use whichever method it can, and switch when it can. To move a project from a service account to workload identity:
- Add its project and namespace IDs to the realm rule, and a trusted publisher for its packages (GitLab CI with workload identity, steps 2 to 4).
- Change its pipeline to the token exchange (step 5 there), and run it.
- Revoke the tokens, delete the GitLab variables, and deactivate the service accounts once no project uses them.
| Workload identity | Service account | |
|---|---|---|
| Secret stored in GitLab | none | the tokens, in CI/CD variables |
| Identity | each job, from claims GitLab signs | whoever holds the token |
| Rights | one role, plus trusted publishers | the account’s roles, narrowed by the token’s scope |
| Limited to protected refs or environments | by the token’s claims, which Forge checks | by GitLab’s protected variables |
| Credential lifetime | the exchanged key’s, one hour by default | until expiry or revocation |
| Audit | the publish audit, with the job’s subject and jti digests | the account and token in the logs |
| Needs | Forge reaches GitLab’s discovery document and JWKS | nothing |
Changelog
All notable changes to Forge are documented here. The format follows Keep a Changelog, and versions follow Semantic Versioning: breaking changes come only in a major release, and each one is called out under Upgrade notes.
The release workflow publishes the section whose heading matches the tag
(## [1.0.0] for v1.0.0) as the GitHub release notes. Entries for changes
not released yet are fragments under docs/changes/
until the release folds them in.
Unreleased
2.1.0
Upgrade notes
- The anonymous mode
allowlistand itsanonymous_ip_allowlistare removed; the anonymous modes areoffandall. An instance whose stored mode isallowlistcomes up with anonymous accessoff, and the settings API refuses"anonymous_mode": "allowlist"andanonymous_ip_allowlist. To keep letting a network read without credentials, admit it at the load balancer instead: create a service account under Admin → Service accounts in a trusted-header realm, give it read grants on the repositories the network read anonymously (those that allow anonymous reads, and those theanonymousrole has grants on), and have the load balancer name that account for the networks that were on the allowlist. See the deployment guide, “Trusted-header realms”. (#507) - Stop every 2.0.0 instance before the first 2.1.0 instance starts,
rather than rolling the upgrade. The first 2.1.0 instance migrates the
database. While 2.0.0 instances still serve, the migration can wait behind
a 2.0.0 database connection left inside a transaction, and every sign-in
and token check of every instance then waits behind it, for up to 30
minutes. An OIDC sign-in on a 2.0.0 instance can also link its identity to
an account of a trusted-header realm meanwhile. With the Helm chart,
upgrade with
--set strategy.type=Recreateor scale the deployment to zero first; with the Kubernetes manifests, scale theforgedeployment to zero before applying 2.1.0. (#615)
Features
- OpenTelemetry metrics, traces and logs, exported over OTLP/HTTP with the
official OpenTelemetry SDK.
[telemetry]turns each signal on (all are off by default), and the standardOTEL_*variables say where they go. The metrics cover requests by surface, proxy lookups by outcome with the upstream requests and bytes, S3 operations and the circuit breaker, the blob cache’s counters, staging space, the outbox backlog, cache hits, the connection pool, every background worker’s passes and restarts, retention, sign-ins, and an inventory of repositories, versions and stored bytes; no attribute carries a package, version, image, user or path, andtelemetry.per_repository = falsetakes the repository names off. A request is a trace, joining the caller’s from itstraceparent, with the proxy lookup, the upstream requests (which carry the trace on) and the S3 operations beneath it; each worker pass is a trace of its own. A span’s URL holds no query string, and the secret in the path of an npm web login is recorded as{secret}. Exported log records carry the trace they were written in. The Helm chart (telemetry.*) and the kustomize componentcomponents/opentelemetryturn it on, anddeploy/monitoringhas an example Collector configuration, a Grafana dashboard and alerting rules. See the deployment guide’s Telemetry section. (#178) - Trusted-header realms: a
trusted_headerrealm (Admin → Authentication) authenticates a request as the account a header of the load balancer or SSO proxy in front of Forge names, when the proxy proves itself with the realm’s secret in a second header. The headers count only from a peer inauth.trusted_proxy_cidrs, for a request naming one of Forge’s own host names, and only when the request has no credential of its own. A realm names only its own accounts (created under Admin → Users, or on the first request withauto_provision) and never makes anyone an administrator: its accounts never hold theadminrole. Headers with a wrong secret or a name it may not use (a deactivated account’s) name nobody: the request is served as one without credentials, never anonymously, so its own credential and signing in still work. A load balancer can so admit a network as an account with its own grants. See the deployment guide, “Trusted-header realms”. (#506) - Service accounts: an account for a machine or a network that nobody signs in
to, such as a CI pipeline, a build agent or a network the load balancer
admits (Admin → Service accounts, or
POST /admin/userswith"kind": "service"). A service account has a description, roles and grants, and no password, invitation or email. The login screen, password Basic, credential forwarding and OIDC all refuse it, and it never holds theadminrole. It authenticates with the API tokens an administrator issues for it, and through the one trusted-header realm it may belong to, which alone may name it. Deactivating it stops both.GET /admin/usersgainskind,description,realmandlast_used_atin its views, and a?kind=filter. The accounts that exist are users. See the deployment guide, “Service accounts”. (#508) - Administrators delete images from the web UI: on the image page of a hosted
docker repository, the whole image, a tag, a manifest (with the platform
manifests and referrers only it kept) or a Helm chart’s version. The admin
API has it as
DELETE /admin/repositories/<name>/images?image=<image>, withreference=<tag|digest>for a tag or a manifest. A hosted docker repository, deleted only once it holds no image, can now be emptied without the registry API. (#509)
Fixed
- Of two first pushes of the same package version that arrived at the same moment (two CI jobs, or two replicas), the one that lost the race failed with 409 even where the repository allows redeploys. It now replaces the version, as it would have a moment later. This holds for NuGet, npm and the other package formats; a repository without redeploys still answers 409. (#226)
- The retention preview of a package or proxy repository evaluated its rules
at the Forge host’s clock, while a run evaluates them at the database’s.
With the two clocks apart, a version near an
older_than_days,not_downloaded_for_daysornot_pulled_for_daysboundary could be shown as kept and then be deleted by the run. The preview now evaluates at the database’s clock, as image and file previews already did. (#227) - A proxy operation that reads several upstream documents (a version listing,
dating the package delay of a package with many versions) kept every
document it read until it answered: up to 256 of them, each up to
proxy.max_metadata_bytes. An upstream serving large pages could make a few such operations hold gigabytes. One operation now keeps at most 4 ×proxy.max_metadata_bytesof documents. The documents past that are not read, and the format answers with the others: NuGet leaves the versions of an unread registration page to the listing. (#230) - The page of an image with many tags and manifests (a CI that tags every
build, or a proxy of an image with thousands of tags upstream) took seconds
to answer and held a server thread all the while: it compared every tag
with every manifest, 24 s for 50,000 of them. It now answers in
milliseconds, and lists the newest 1,000 tags and 1,000 manifests, saying
how many the image has (
tags_totalandmanifests_totalinGET /browse/images/<repo>/<image>). (#240) - The browse of a docker group listed the images and tags of every member the
reader could read, ignoring the group’s guards. An image a proxy member had
cached under one of the group’s
reserved_prefixes, or of a name an owning hosted member has (prefer_hosted), was listed with adocker pullcommand that the group then refused, which is exactly the dependency-confusion image the guards keep out. The group’s own content selector was not applied either. The images list, an image’s page and a chart’s page now show what a pull through the group serves. A manifest of a lower priority member no longer wears a tag that a higher priority member’s manifest answers. (#241) - Browse kept listing repositories as they were for up to 30 seconds after an administrator created, renamed, deleted or took one offline, or changed a group’s members, under Manage: a renamed repository was still listed under its old name, which then showed as empty. Those changes now refresh what Browse shows. (#244)
- A docker proxy’s images showed when Forge first cached them as “Pushed” (the manifests on an image’s page) and “Last pushed” (the images of the repository), as if pushed to the upstream then. A proxy labels them “Cached” and “Last cached”, as package versions already were. (#248)
- The last run of a package or proxy repository’s retention policy counted the
versions it decided on as
evaluated(“of N” on the Retention screen), not the versions it inspected. It now counts every candidate version, as the preview does and as docker and raw runs already did. (#439) - npm repositories read a package’s tarball as npm unpacks it, contiguous-file
entries (type
7) included: apackage.jsonof that type after a regular one is the version’s manifest, as it is the one npm installs, and such entries count indist.fileCount. Forge skipped them, so a tarball could show other dependencies and install scripts in Forge than run on install. (#504) - A NuGet package that repeats an entry name was refused, whether pushed or
fetched through a proxy, though NuGet restores it. On nuget.org,
Microsoft.Extensions.Caching.Memory,Microsoft.Extensions.Hosting.AbstractionsandMicrosoft.Extensions.Logging.Abstractions3.1.0 holdpackageIcon.pngtwice, so a restore that needed them failed with 502. Forge now reads every entry of the archive as NuGet does, and refuses only a repeated manifest, which NuGet refuses too. (#533) - A request or background task dropped while Forge began a database transaction (a client that went away, a request past its deadline) could leave a database connection inside an open transaction that Forge did not know of. What later ran on that connection was not committed and not seen by others, held its locks, and was rolled back when the connection closed 10 to 30 minutes later: requests could hang that long, and writes made in between were lost, such as a resumed Nexus import’s record of an item it had copied. Beginning a transaction is now safe against cancellation. (#570)
Security
- npm repositories read a tarball’s directories as npm unpacks them. To npm, a
directory entry (of type
5, or of type0with a path ending in/) has no body, whatever size its header gives: the next entry starts right after its header. Forge skipped the size the header gave, so a tarball could hide from Forge thepackage.jsonnpm installs, with other dependencies and install scripts than Forge showed, and Forge read an entrypackage/package.json/as the manifest npm takes for a directory. Such a directory is no longer read as a file, and a tarball whose directory entry has a body is refused. (#540) - A NuGet package whose entry carries the ZIP64 extra field twice is refused. The field can give where the entry’s data starts, and NuGet reads the first one while Forge read the last: a package could so hold two manifests, Forge checking and showing one identity and its dependencies while NuGet installed another. Neither NuGet nor .NET writes the field twice. (#541)
- npm repositories read a tarball as npm unpacks it in more ways, and refuse
one they cannot follow. A tarball could still hold a
package.jsonnpm installs that Forge never read, with other dependencies and install scripts than Forge showed: after a single zero block, in the body of a header npm rejects (a checksum npm reads over 12 bytes, a link target on a file, an empty path, a base-256 number npm cannot read, a name that goes on past its end), in a second gzip member, or behind a PAX record npm reads otherwise. And a file such as./package.json,package\package.jsonorpackage/PACKAGE.JSONispackage.jsonto npm, or to npm on another system, but was not to Forge. Forge now reads every gzip member, and refuses data after the end of the archive, such headers and records, and a file it cannot tell npm does not take forpackage.json.npm packwrites none of these. (#554) npm logoutsends the API token it revokes in the request’s path, and theforge::auditline of the request recorded that path, so the console log held the token, and with telemetry on the request’s span (url.path) and the exported log record did too. The token stays valid when the logout fails (another account’s token, a workload key, a database error). Both now record it as{secret}. (#557)- npm repositories refuse a tarball holding a
package.jsonbehind a Windows root. npm (node-tar) strips a drive, a UNC root or a device root at the start of an entry’s path, also on Linux and macOS: before the first component is stripped (npm 10), or after it (npm 11.20 and later). A file such aspackage/c:package.json,package/\\server\share\package.jsonorC:C:package.jsonis so thepackage.jsonnpm installs, while Forge read the tarball’s other one and showed other dependencies and install scripts than ran on install.npm packwrites no such path. (#571) - NuGet repositories refuse a package whose end-of-central-directory record NuGet reads otherwise than Forge: one marking ZIP64 in its central directory’s size alone, or in its disk numbers alone, and a last record whose comment runs past the end of the file. Forge and NuGet then read different central directories, so a package could hold two manifests, Forge checking and showing one identity and its dependencies while NuGet (.NET 8 and .NET Framework, for the comment) installed the other. Neither NuGet nor .NET writes such a record. (#572)
- A NuGet package whose entry carries a ZIP64 extra field of 28 bytes or more,
with a value its header does not mark before one it does, is refused. .NET 8
and later read each value of such a field at its fixed place, Forge one marked
value after the other: a package could so point the two at different local
headers and hold two manifests, Forge checking and showing one identity and
its dependencies while NuGet (
dotnet restore) installed the other. Neither NuGet nor .NET writes such a field. (#592)
Documentation
- The Docker guide says what deleting one of several tags of a digest leaves,
by a retention rule or the registry API: the manifest stays under its other
tags. It also says that a pull counts for every tag of its digest, so
tag_not_pulled_for_dayskeeps a tag while another tag of the same digest is pulled, and that aliases such aslatesttake places among thekeep_last_n_tagsunless they are protected. (#510)
2.0.0
A new major release, installed fresh: 2.0.0 does not upgrade 1.0 or 1.1.
Install it on an empty PostgreSQL database and an object store of its own;
docs/deployment.md
lists the required settings. It adds npm and raw (generic file)
repositories, serves every package repository at /repository/<name>/,
keeps Helm charts as OCI artifacts in docker repositories, invites users by
email, and manages roles, access levels and retention in the web UI. Every
format is built in: Docker / OCI, NuGet, npm and raw. Forge is licensed
under the MIT License (up to 1.1.0: the GNU AGPL v3 or a commercial
license), and is published as ghcr.io/forgepkg/forge for linux/amd64
and linux/arm64, with the Helm chart oci://ghcr.io/forgepkg/charts/forge.
Upgrade notes
- 2.0.0 does not upgrade 1.0 or 1.1. There is no upgrade path: 2.0.0 is installed fresh, with an empty PostgreSQL database and an object-store bucket (or prefix) of its own. Startup refuses a database created by 1.x or by a build before 2.0.0 (“the database was created by Forge 1.x or by a build before 2.0.0, and 2.0.0 does not upgrade it”). Upgrades are from 2.0.0 on. (#490, #498)
Features
- Credential forwarding: a realm can keep an argon2id hash of every password
its upstream (e.g. a Nexus) accepts (Keep a password hash of successful
logins,
capture_password_hash, off by default; unchecking it deletes the hashes). Once the hashes are in place the realm can stop asking the upstream and sign its users in with them (captured_password_only), and its accounts can be switched to local accounts with the same password, one by one or all at once (POST /admin/users/{id}/convert-to-local,POST /admin/realms/{id}/convert-to-local), so the upstream can be retired without asking every user for a new password. - OIDC login without a client secret: an
oidcrealm can authenticate at Microsoft Entra with a federated identity credential instead (client_authentication = "federated_credential"). Forge signs a short-lived client assertion itself, as an OpenID Connect issuer of its own (server.public_base_url), publishing/.well-known/openid-configurationand its keys at/.well-known/jwks.json. The keys are RSA-2048, shared by every instance through the database (sealed with[crypto]), and rotated every 30 days, each new key published two days before it signs. The realm shows the issuer, subject (<client id>@<realm issuer>) and audience to register. See DEVELOPMENT.md. (#302) - Helm:
extraObjectscan also be a map, whose keys only name its entries. Helm merges a map across values files, so each file can add its own objects, and a later one replaces an entry by its key or drops it withnull; a list in a later file replaces the whole list. Lists keep working. (#288) - Import from Nexus: copy a Nexus 3 hosted repository into a Forge hosted
repository of the same format, one repository at a time: NuGet packages,
and Docker images with their digests, multi-platform indexes, signatures
and SBOMs. In the web UI (a hosted repository’s
Import from Nexus) or the admin API
(
/admin/repositories/{name}/imports). The source is checked before an import starts; a dry run reports the counts, the size and the conflicts without writing. Packages are verified against Nexus’ checksums and written through the normal push path, keep their publish dates, last downloads and uploaders, and are audited as imports. A version Forge has with other bytes is skipped and reported, overwritten, or fails the import, as chosen. An import resumes after a restart, a failure or a cancel, failed items can be retried on their own, and running it again copies only what is new. The Nexus password is sealed with the secret encryption and deleted when the import completes. See docs/nexus-import.md. (#183) - npm registries: repositories of
"format": "npm", served at/repository/<repository>/for npm, pnpm, Yarn and Bun. Hosted ones takenpm publish(the tarball checked against the integrity the client claims and the version it names, under the repository’s deployment policy), dist-tags andnpm deprecatewith the levelwrite, andnpm unpublishwithdelete. Proxies cacheregistry.npmjs.orgor any registry, with Basic credentials or a token sent asAuthorization: Bearerto its own origin only, and serve the upstream’s dist-tags, deprecations and publish times, verified tarballs and the publication delay. Groups merge their members, take publishes into a writable member, keep reserved scopes (@acme) away from proxy members and passnpm auditon to their proxy without the reserved names.npm loginis approved in the browser on a page of the web UI, or checks a password (yarn npm login), and mints an API token thatnpm logoutrevokes; npm clients send API tokens asAuthorization: Bearer, which every format plugin’s repositories now accept. Retention never deletes a version a dist-tag names. The web UI shows a package’s dist-tags, deprecated versions and README, and a repository’s.npmrcand.yarnrc.ymllines and install commands. npm is a built-in format plugin (crates/formats/npm). See docs/npm.md. (#41) - Hosted NuGet repositories have a deployment policy, like raw and npm ones:
allow redeploys, refuse them (the default), or read-only, which refuses
every push, unlist and delete.
allow_redeploysays the same and follows it. A NuGet group may have a writable member, a hosted member that pushes, unlists and deletes through the group go to. The web UI’s repository form offers both. (#41) - Raw (generic file) repositories, like Nexus’ raw-hosted, raw-proxy and
raw-group:
"format": "raw"repositories serve arbitrary files under arbitrary paths at/repository/<repository>/<path>. Uploads are aPUTwith an API token or a multipart form (Nexus’raw.directory/raw.assetNfields work); downloads answer oneRange, conditional requests, anETag(the SHA-256) andLast-Modified, and<file>.sha256,.sha512,.sha1and.md5from the stored digests, sosha256sum -cchecks a download. Directories list as HTML or JSON, a page at a time. A hosted repository’s deployment policy allows redeploys, refuses them (the default) or makes it read-only; files download unless the repository displays theminline, and are sandboxed either way. Paths with..,., empty segments, backslashes or encoded slashes are refused. A proxy fetches from any HTTP(S) base URL, following redirects (GitHub release assets) with its credentials sent to its own origin only, re-validates cached files aftermax_age_seconds(0: never), remembers misses, and serves its cached copy while the upstream is down; a group serves a path from its first member that has it. Content selectors match paths, retention deletes files by path prefix, age and downloads or keeps the newest per directory, and the web UI browses a repository’s tree, uploads and deletes files and shows each file’s URL andcurlcommands. See docs/raw.md. (#42) - Helm charts pushed to a docker repository (
helm push) have a page of their own in the web UI: the chart’s versions, thehelm installandhelm pullcommands of the selected one (the newest release by default) with thehelm registry loginto run first, itsChart.yamlfacts (appVersion, description, home, sources, maintainers, dependencies), its README and its default values. The browse API serves them atGET /browse/chart?repo=<name>&image=<image>&digest=<manifest digest>; a proxy shows them once the chart has been pulled through it. See docs/docker.md. (#43) - A proxy’s retention rules (
scope: proxy_cached) can keep a working set of each package:keep_last_nkeeps the newest N cached versions the upstream lists, andkeep_n_most_downloadedthe N cached versions pulled most recently (by when they were last pulled, not how often). A version any proxy rule keeps is never evicted, however long nobody pulls it, so a quiet package no longer loses every cached version and stays restorable while the upstream is unreachable. The Retention screen offers both on proxy rules, and a new proxy policy starts by keeping the newest three versions of each package. (#59) - Forge runs on arm64: each release publishes
ghcr.io/forgepkg/forgeforlinux/amd64andlinux/arm64as one multi-platform image under the same tags, so arm64 nodes (AWS Graviton, Azure Cobalt, Ampere, Raspberry Pi) and Apple Silicon machines pull an image of their own platform, without emulation. Each platform’s image carries its own SBOM and provenance attestations. The release signs the multi-platform index and each platform’s image, so the release notes’cosign verifycommand passes for each of their digests, and the Helm chart pins the index digest.cargo xtask docker-build --platform linux/arm64builds the arm64 image from a checkout. (#186) - Admin → Settings (
/manage/settings) sets who may read without signing in (the anonymous mode and its IP allowlist, next to the repositories that allow anonymous reads) and how people sign in (the login screen, password Basic on the feeds, the session lifetimes and the lockout), for every instance and without a restart: the instance that takes a change applies it at once, the others withinauth.realm_cache_ttl_seconds. The admin API offers the same asGET/PUT /admin/settings/auth; aPUTchanges the fields it names and refuses a wrong value with 400 and what is wrong. A repository’s page shows the server-wide anonymous mode next to its own anonymous access. (#294) - Hosted repositories can grant package-scoped write and delete access to workload identities through trusted publishers, independently of their realm reader role. Manage trusted publishers through the admin API or the repository admin page; publisher changes revoke their rights from outstanding workload keys and registry tokens without revoking the realm’s other keys. (#306)
- A built-in
writerrole:readandwriteon every repository, nodeleteand noadmin. A writer publishes NuGet packages, container images, Helm charts and raw files (new ones too) to every repository but cannot unlist or delete versions, delete tags, delete or replace raw files, or reach the admin API;/admin/mereports therepo.writecapability. Admin → Users shows each user’s access level (admin, writer or reader) from their roles, and the user’s page switches it.docs/deployment.mddescribes the built-in roles. (#318) - Forge can send email: Admin → Email (
/manage/settings/email) sets up the email service, starting with an SMTP server (STARTTLS or implicit TLS, with a username and password or as a relay), with a test send and the last failed send on the page. The settings live in the database only, with noforge.tomlsection or environment variable, apply on every instance without a restart, and the password is sealed at rest with secret encryption and never returned. The admin API is/admin/settings/email(GET,PUT,DELETE) andPOST /admin/settings/email/test. (#319) - The email service can send through Microsoft 365: a Microsoft 365 (Graph)
transport on Admin → Email signs in as an Entra app registration with
the client credentials grant and sends with Graph’s
sendMail, so no mailbox password is stored and SMTP AUTH can stay off. The app authenticates with a client secret (sealed at rest, bound to the tenant and app) or with a federated credential: Forge signs the client assertion with the issuer keys federated OIDC realms use, and the page shows the issuer, subject and audience to register.docs/deployment.mdwalks through the setup, including limiting the app to the sender mailbox with RBAC for Applications. (#320) - The email service can send through an SMTP server that takes OAuth tokens
(SASL
OAUTHBEARER, orXOAUTH2), such as Stalwart: the SMTP server with OAuth transport on Admin → Email stores no mailbox password. Forge registers itself with the mail server’s authorization server (RFC 7591, authenticated by an initial access token or an account, which are never stored) or uses a client entered by hand, and the mailbox authorizes it once, in the browser (the code flow with PKCE) or with a code on another device (RFC 8628); the client credentials grant works too. Forge refreshes its access token, stores a rotated refresh token sealed at rest, and asks for a new authorization on the page when a refresh is refused.docs/deployment.mdwalks through the setup with Stalwart. (#321) - Users are invited instead of given a password: Admin → Users → New user
creates the account with its email address and access level (reader, writer
or admin), and Forge emails its user a single-use link,
/invite#<token>, with which they choose their own password before signing in. The user’s page resends the invitation, copies a new link (shown once, for an install without email) or revokes it, and the users list shows who is still invited. Only the token’s hash is stored; the link expires after the lifetime set on the email settings page (72 hours by default), and deactivating or deleting the account revokes it. Until the password is chosen the account signs in by no path, and no credential-forwarding or OIDC realm can take its name. (#322) - The web UI manages RBAC without the
/adminAPI: roles with their grants and members (deleting a role first names the users and grants it takes along), content selectors (with a box that tests a package name against the patterns, and a delete that first names the grants it removes), each user’s roles and effective access, and token scopes, which a token gets when it is created and an administrator can change later. The user list shows each user’s roles instead ofadminorreader, a repository’s page shows the anonymous role’s grants on it next to its anonymous-read flag, and a workload-identity rule picks its role from a list. For these screens,GET /admin/rolescounts each role’s grants and users,GET /admin/roles/{id}/userslists its members,GET /admin/content-selectors/{id}/grantslists the grants a selector scopes,POST /admin/content-selectors/matchtests a package name, a user lists itsroles, andGET /browse/reposgives each repository’sid. (#369) - Tokens created in the web UI now expire after 90 days by default, with 30-day, one-year, custom-date and no-expiry choices. Expiry is shown when the secret is created and in token lists; the API rejects an expiry that is not in the future. (#371)
- A user’s page (Admin → Users →
) deactivates, reactivates and signs the user out everywhere, each after a confirmation that says what happens, without the /adminAPI. Deactivating refuses every sign-in, ends the user’s sessions and revokes a pending invitation; their tokens are refused but not revoked, so reactivating lets them work again, and the confirmation counts them. Deactivating the last active administrator is refused with the server’s reason. Signing out everywhere ends the user’s browser sessions; for a credential-forwarding account it also checks the password against the upstream again and revokes the tokens minted from a forwarded sign-in. The delete confirmation suggests deactivating an account that should be kept. (#372) - Each hosted or proxy repository has a Retention screen (Repositories →
→ Retention ): it edits the repository’s policy rule by rule (package, proxy eviction, raw file or image rules, by the repository’s format), previews what the policy in the form would remove before it is saved, asks before saving a policy whose runs delete and before removing one, starts a run, shows the last run, and lists the audit log, filtered to one run and paged back.POST /admin/repositories/{name}/retention/previewtakes a draftpolicyto evaluate in place of the stored one: it is checked like aPUTof it, evaluated even when disabled, and never stored. Until now retention was managed through the admin API only. (#373) - Administrators can permanently delete a hosted package version from its package page after confirming the repository, package and version. (#374)
- The web UI has a design of its own: Forge’s design system replaces the Catalyst kit on every screen. A navigation rail takes the place of the top bar, with the places, the administration screens and the repositories you can read, each with its kind and format; on phones it opens as a drawer. Screens open with a header of breadcrumbs, title, badges and actions, lists are tables, and detail screens show their connect and install commands next to the content. The interface uses IBM Plex Sans and JetBrains Mono, served by Forge itself, and offers three themes (Light, Light with dark rail and Dark; System follows the operating system). Deleting files and revoking tokens ask for confirmation in a dialog. (#397)
- Administration has a System screen: it lists the keys of the secret
encryption at rest, rotates the data key or the key-encryption key and re-wraps the
keys under the current root key after asking first, and shows the local blob
cache of the instance that answered: its hit ratio, what it served, its size
against the limit, its evictions and whether S3 is reachable. Until now these
were
GET /admin/crypto/keys,POST /admin/crypto/rotate/{dek,kek,root}andGET /admin/blob-cacheonly. (#404) - Grants and token scopes give ordered access levels,
read<write<delete, each including the ones below it: one grant gives full access to a repository, a role can no longer push to a repository it cannot pull from, and a content selector on awritegrant filters what its holder reads as well as what it publishes. A role holds one grant per repository and selector. The role page and the token dialogs pick one level, labelled with the Nexus actions it covers (read: browse, read;write: add, edit;delete: delete), and a token’s admin opt-in is a checkbox of its own.docs/deployment.mddescribes the levels. (#411) - Retention policies can set a per-repository logical size budget. When a repository exceeds its maximum, retention evicts least-recently used eligible content toward a configurable target, while preserving rule protections. The Retention screen shows the current repository size and budget status. (#417)
- Admin → Retention switches retention runs off and on for every repository and instance, and sets how long the retention audit log is kept. While runs are off no run starts, a run in progress stops before its next batch, Run now answers 409, and each repository’s Retention screen says so; policies can still be edited and previewed. A failed run is retried after its policy’s interval rather than the old global default. (#429)
- Raw file repositories are a format plugin,
crates/formats/raw, like NuGet: the plugin speaks the raw protocol (paths, checksum sidecars, listings, Nexus’ form fields, a proxy’s upstream URLs) and the server’s new file store keeps the files. The plugin contract gains a second storage model for this: a format whose descriptor saysStorage::Filesasks the host for files and directories, stores and deletes files, and serves a file withAction::ServeFile;rendermay now answer with any actionhandlecan.GET /admin/formatsand the repositories API report what a repository holds (storage:packages,filesorimages). See docs/format-plugins.md. (#431) - Docker / OCI is a format plugin,
crates/formats/docker, like NuGet and raw: the plugin speaks the distribution-spec protocol (paths, names, manifests, error answers, Helm charts, an upstream registry’s URLs), and the server’s image store keeps the images, the registry tokens and the authorization./v2answers as before. The plugin contract gains a third storage model for this, images: a plugin whose descriptor saysStorage::Imagesasks the host for manifests, blobs, tags and referrers, runs upload sessions and stores manifests through it, and serves a blob or a manifest withserve-blobandserve-manifest; a proxy’s plugin names the upstream’s URLs and the host fetches them, exchanging the upstream’s Bearer tokens.GET /admin/formatslists Docker from its descriptor, with its version and description. See docs/format-plugins.md. (#447) - npm can publish from GitHub Actions and GitLab CI, including self-hosted GitLab, without a stored token through its native OIDC token exchange. Exchanged keys use the workload realm’s lifetime, claim rules and repository permissions, including a group’s writable member. (#467)
Fixed
- A database failure while checking a browser session or its CSRF token
answered 500; it answers 503 with
Retry-After, like the token and password checks, which now sendRetry-Aftertoo. (#205) - Trusted publishing: a workload token answered 401 “invalid credentials”
while its issuer’s discovery document or keys could not be fetched and
were not cached yet (after a restart, on a new replica, at the hourly key
refresh), so CI jobs failed instead of retrying. It answers 503 with
Retry-After, at the token exchange too. (#202) - Trusted publishing: after a realm edit, the other instances refused the
workload keys the editing instance issued under the new configuration
for up to
auth.realm_cache_ttl_seconds, and could refuse an exchange with 401 “realm changed during the exchange”. An instance now reloads the realms when a key names a configuration it does not have yet. (#203) - Trusted publishing: an instance whose clock ran behind the database’s could exchange a CI token a second time just after the record of its first exchange was cleaned up. Whether a token has expired is decided on the database’s clock, and the record is kept 15 minutes past the token’s expiry. (#204)
- Docker registry: a registry token got with a workload key
(
docker login -u oidc) worked again once the key’s record was cleaned up, and one got with a CI token directly stopped working when a key exchanged from that CI token ended. A registry token stands and falls with the credential it was got with, and expires no later than it. (#201) - S3
default_chaincredentials: an STS or container credentials endpoint that could not be reached counted as a refusal, so requests answered 500 instead of 503, and with the blob cache an instance restarted during the outage failed to start instead of starting degraded. (#208) - Blob cache: making room deleted the evicted files while every cache hit on the instance waited, on the threads that serve requests, and each miss read the disk’s free space. Evicted files are deleted in the background, and the free space is read once a minute. (#212)
- Blob cache: after a restart, concurrent downloads of a cached blob each read and hashed the whole file before serving it; each file is checked once. (#210)
- Blob cache: a blob garbage collection deleted stayed in the other instances’ caches until it was evicted. Every instance drops such blobs at startup and every 10 minutes. (#211)
- Docker registry: while S3 was unavailable, uncached pulls, pushes and the
finish of a chunked upload answered 500 instead of 503 with
Retry-After, and an upload whose body the client aborted answered 500 instead of 400. (#209) - Blob cache: a docker blob pushed in several chunks was cached about twice: every chunk was written through to the cache next to the assembled blob, and evicted as much content that is downloaded. Only the blob is cached. (#213)
- Blob cache: with several replicas,
GET /admin/blob-cacheanswered with the statistics of whichever replica served the request, without saying which one. The response names it ininstance(host name and start time), and the deployment guide says the numbers are per instance. (#214) - Docker image retention kept deleting what a run had decided after its policy was disabled, switched to dry run or changed, after the run lost its lease, and when the image changed meanwhile: it deleted a tag pulled since, older tags after the newest were deleted, and the signature of an image tagged again. Like package retention, a run now stops when its policy or lease changes, and applies an image’s decisions only after evaluating its current state again. (#215)
- Docker proxies: a tag list was cut short, without a link to the next
page, when the upstream paged its tags (
Link: rel="next", as Quay does) or refused a largen(then only the cached tags were listed), so Renovate, Flux orskopeo list-tagsmissed tags; groups inherited it. A proxy follows the upstream’s pages, asks again withoutnwhen one is refused, and links to its next page when the upstream has more. A group ends its page where a member left tags out. (#216) - Docker proxies: a fetch from the previous upstream that was still running
when the proxy was pointed at another
remote_urlcached its tag, manifest or layer after the repoint had emptied the cache, so the previous upstream’s tag was served formetadata_ttl_seconds. Such a fetch caches nothing; its client is asked to retry (503). (#217) - Docker registry: the sweeper could release a layer a client had just
confirmed, so the manifest push answered
MANIFEST_BLOB_UNKNOWN: a pull orHEADof a blob recorded its use at most hourly, which withdocker.unreferenced_blob_ttl_seconds = 3600left no time, and pushing or mounting a blob the image already had recorded none. Pushes and mounts record the use, and pulls record it every tenth of the TTL. (#218) - Docker groups named their
writable_memberby name: renaming that repository broke group pushes, and a repository created later under the old name silently became the push target. The member is stored by id, and the admin API and web UI show its current name. (#219) - Proxies of a format plugin (NuGet, npm): an upstream that refuses the proxy’s credentials (401, 403), or serves a package the proxy does not take (too large, unreadable, not the version asked for, not matching the digest the upstream publishes), with nothing cached for the request, answered 500. It answers 502: the upstream or the proxy’s settings need fixing, retrying does not help. (#41)
- Proxies:
server.request_timeout_seconds(900 s) cut off a first download long beforeproxy.download_timeout_seconds(1800 s), since a proxy stores the whole package or layer before it answers; a large layer from a slow upstream answered 408 and could never be cached. A download from the upstream extends its request’s deadline to its own, and the request hasrequest_timeout_secondsagain after it. (#220) - NuGet proxies: a package id the upstream answered 404 for stayed hidden for
metadata_ttl_seconds, whatevernot_found_ttl_secondssaid (0 included): the “unknown id” listing was cached for the metadata TTL on every instance. With a shorter not-found TTL, a package published upstream meanwhile answered 404 to restores, and with a package delay its downloads failed too. The cached answer now stands only while the id’s not-found marker does. (#222) - Proxies: a failed upstream version listing was not remembered, so during an upstream outage every listing, registration or package-delay check waited for the upstream to time out (15–60 s) before serving the cached versions; a restore of hundreds of cached packages took many minutes or timed out. For a minute after the upstream was unavailable for a listing (no answer, a timeout, a 5xx or a 429), ids with cached versions are answered from the cache without asking the upstream; ids without still ask it. A listing that failed for one id alone (an unusable document) does not hold back the others. (#223)
- A search through a group asked each proxy member’s upstream for 1,000 hits,
even for a page of 20: every search through a group that fronts nuget.org was
as slow as a 1,000-hit one, and an answer over
proxy.max_metadata_bytessent the proxy’s searches to its cache for a minute. The group asks an upstream for the hits its page needs (32 for a page of 20), and for more only while the proxy or the group’s guards leave hits out. A group search lists its hosted members’ packages first, then the upstreams’ in the upstreams’ own (relevance) order rather than by id, and its total is an estimate while an upstream has more hits than it returned. (#224) - Format plugins: a version several kilobytes long passed validation and failed its database insert, so a push answered 500 instead of 400, and so did a proxy asked for such a version. Every format plugin’s ids are now at most 512 bytes and its versions at most 256: a longer one is refused with 400 on push, answers 404 in a URL, and is left out of a proxy’s upstream version listings. (#231)
- Helm chart:
strategy.type: Recreate, which a blob cache on a ReadWriteOnce volume needs, kept the default’srollingUpdate, so the API server refused the Deployment andhelm install/helm upgradefailed. The chart leavesrollingUpdateout of a Recreate strategy. (#235) - Image listings now show accurate totals and allow paging through every image, including in repository groups and with content selectors, instead of stopping at 5,000 images per member. Image search is now case-insensitive. (#242)
- The repository edit screen loads format metadata, restoring upload-size defaults and client-side staging-ceiling validation, and showing plugin format names and feed paths correctly. (#243)
- The image page’s pull commands started with the first tag by name: for a
signed image a cosign
sha256-….sigsignature, otherwise often the oldest version. They start withlatest, else the most recently pushed image tag, and skip signature, attestation and SBOM tags. (#245) - Admin → Authentication: one click on a workload-identity realm’s Enabled switch disabled it at once, which revokes every API key its token exchange issued, for good. Disabling one now asks first, and so does saving an enabled one on its edit page, which revokes the keys too. (#247)
- Proxies of a format plugin (NuGet): an upstream that could not be
reached (connection refused, a name that does not resolve, a timeout) or
answered a 5xx or 429, with nothing cached for the request, answered 500
and logged an error, as if Forge were broken. It answers 503 with
Retry-After, like the docker registry’s proxy, and logs a warning naming the upstream. A group whose member failed this way answers the same. Credentials the upstream refuses answer 502. (#283) - Web UI: a link whose query value looked like a number or a boolean lost
it, because the router read the value as JSON:
?tag=1.36on an image page showed the pull commands of the default tag,?tag=1.10was even read as1.1,?q=1.0on a repository page searched for1, and?path=2026missed the directory2026/of a raw repository. Every query value is now read as the text it is. Links the web UI wrote before, such as?tag=%221.36%22, still work, and the links it writes are plain (?tag=1.36). (#403) - Widen the L1 cache TTL tests’ timing windows to avoid spurious failures on busy CI runners while preserving their cache reuse and expiry checks. (#471)
- Proxies: a version listing the upstream answered unusably (a document over
proxy.max_metadata_bytes, or one without the versions) answered 500 when nothing was cached; it answers 502, like an upstream that refuses the proxy. (#488)
Security
- Proxy search no longer lets rejected search parameters disable upstream search for other callers for a minute. Only upstream outages trigger search backoff; NuGet search offsets above 3,000 use locally cached packages instead. (#225)
- A proxy repository’s format plugin could make Forge send a GET to any
address Forge reaches, a loopback or private service included, and read the
answer back into what the proxy serves. Forge now sends the requests a
format asks for, and downloads its packages, only from the origin of the
repository’s
remote_urland from origins that documents fetched from that upstream name (a NuGet service index’s resources on other hosts); any other request is refused and logged as a warning. The documents an upstream serves still decide which hosts become reachable, so confine Forge’s egress with network policy where internal services must stay out of reach. (#233) - The S3 client no longer compiles in SigV4a signing, and with it
crypto-bigint0.5.5 (RUSTSEC-2026-0387): several of its operations meant to run in constant time did not. The AWS SDK used it only to derive SigV4a signing keys, which sign requests to AWS S3 Multi-Region Access Points (a bucket given as anarn:aws:s3::…:accesspoint/….mrapARN), a setup Forge never documented or tested; requests to such a bucket now fail. - The
ctutilscrate moves from 0.4.2 to 0.4.5 (RUSTSEC-2026-0385): 0.4.2 generated its masks and comparedu128values in non-constant time. Forge uses it through the AES-GCM encryption of stored secrets and the hash crates.
Documentation
forge.toml.examplesuggestedblob_cache.path = "/var/cache/forge", where the Kubernetes manifests mount the cache volume. A volume’s mount root belongs to root, and startup refuses a cache directory Forge’s user does not own. The example suggests/var/cache/forge/blobs, a directory Forge creates in the volume, like the deployment docs. (#236)- Contributions from outside the forgepkg organization now require signing the
Contributor License Agreement (
CLA.md) once, which grants forgepkg the rights to distribute them under the MIT License or other terms.CONTRIBUTING.mdexplains how to sign.
Known limitations
- npm’s password login (
npm login --auth-type=legacy) refuses a username with an@, which every Forge account has: log in in the browser (npm login), withyarn npm login, or with an API token from the web UI. A publish with a provenance statement (npm publish --provenance) is refused, andnpm access,npm owner,npm team,npm org,npm starandnpm tokenare not served. (#41)
1.1.0
The Docker / OCI registry, format plugins (Helm built in, NuGet ported to one), the trusted-publishing token exchange, short-lived S3 credentials and a local blob cache, and the fixes for the findings of the 1.0.0 release review. The 1.0.0 known limitations still apply, except where an entry below says otherwise.
Upgrade notes
- Startup refuses a
server.staging_dirthat does not belong to Forge’s user, or that group or others can write unless it is sticky like/tmp. The container image and the Kubernetes manifests are not affected. (#127) - Durations above 100 years (
outbox.grace_seconds,retention.default_run_interval_seconds,retention.lease_seconds,crypto.cache.unwrapped_key_ttl_secs,crypto.rotation.dek_max_age_days,crypto.rotation.kek_max_age_days, and the newdocker.upload_ttl_seconds,docker.unreferenced_blob_ttl_secondsanddocker.sweep_interval_seconds) fail validation instead of panicking later. (#128) - Migration 0032 deletes grants of the
adminaction on a single repository, which never had an effect, and the grants API refuses them with 400. Administration stays global. (#124) - While a credential-forwarding realm is enabled, an account that an OIDC
realm provisions is named
<claim>@<realm>. Existing accounts keep their names. (#123) - Proxy search now calls the upstream’s search service, which may live on
another host than the feed (on nuget.org,
azuresearch-*.nuget.org). Where outbound traffic is restricted, allow that host, or proxy search keeps answering from the cache only. (#112) - The production kustomize overlay adds a NetworkPolicy that admits only
the ingress controller’s namespace (set it in the overlay), and the Helm
chart renders one as soon as
networkPolicy.fromlists a peer. (#129) - Helm: the first upgrade rolls the Forge pods (when
secretEnvis set) and restarts the in-cluster Redis once, because the checksum annotations are now salted. (#133) - The published
cosign verifycommands also pin the tagged commit and the push trigger; use the commands of the release you verify. (#130) - Migration 0033 adds the Docker / OCI registry’s tables; it changes no
existing data. To push images through a reverse proxy, allow request
bodies of a docker repository’s upload limit (1 GiB by default: docker
sends a whole layer in one request) on
/v2, and serve/v2at the root of the host. The shipped Ingress now allows 1 GiB. The defaultserver.max_staging_bytes(four times the largest item) grows with it to 4 GiB; size the staging volume for it. (#40) - An explicit
server.max_staging_byteskeeps working as it is. When it is below the new 1 GiB docker defaults, they shrink to fit it (lessnuget.max_nuspec_bytes): docker repositories without an upload limit of their own accept layers up to that size, and so do docker proxy downloads unlessdocker.max_blob_bytesis set; startup logs a warning. Raise the budget for larger layers. A budget too small for the package limits is refused as before, with an error that names the setting that decides the minimum. (#198) - The container image’s baked
forge.tomlno longer holds the placeholderminioadminS3 keys. An image started withoutFORGE_S3__ACCESS_KEY/FORGE_S3__SECRET_KEY(and without short-lived credentials) still refuses to start, now with an error that names the missing keys. (#60) server.max_upload_bytesis deprecated: upload limits are now set per hosted repository, in the web UI or the admin API. The setting still works as the default of package repositories without a limit of their own, and startup logs a warning while it is set; give the repositories that need it their own limit, then remove it. A repository’s limit may be at mostserver.max_staging_bytes(lessnuget.max_nuspec_bytes); raise the staging budget first if a repository needs more.- A workload identity realm’s
max_token_age_secondsdefaults to 39600 (11 hours) instead of 3600, so GitLab jobs with a timeout over an hour can publish. Existing realms keep the value they were saved with; to use the new default, clear the field in the realm form (or drop the key from the config) and save. - The username
oidcis reserved for workload identity (trusted publishing), and 1.0.0 allowed it. An account namedoidccan no longer sign in with its password, in the browser or over Basic, and its API tokens work only asX-NuGet-ApiKey, no longer as the Basic password; startup logs a warning that names it. A config withauth.bootstrap_admin.username = "oidc"refuses to start. Rename the account before upgrading; there is no API for it, so use SQL:UPDATE app_user SET username = 'oidc-admin' WHERE username = 'oidc';(the account keeps its id, password, roles and tokens). Then setauth.bootstrap_admin.usernameto the new name if it wasoidc. (#199) - Migration 0034 adds the
workload_api_keytable for the trusted-publishing token exchange; it changes no existing data./api/v2/tokenis a new route, so a reverse proxy that only forwards known prefixes needs/api. (#169) cache.l1_max_capacitycounts an entry over 16 KiB once per started 16 KiB, so it bounds the in-process cache’s memory as well as its entry count (about 160 MiB of large entries at the default 10000). Small entries count once, as before.
Features
-
Format plugins (preview): further package formats, served under
/repository/{repo}/(or a mount the plugin names) with the same storage, RBAC and package selectors, hosted, proxy and group repositories, retention, ranged downloads and audit log as NuGet. A plugin is a crate implementing one contract (forge-plugin-api’sFormatPlugin, mirrored by theforge:format@0.3.0WIT package; a plugin’supstreamside tells the host what to fetch for its proxies) and runs either built into the server or as a WebAssembly component run by Wasmtime. Components are loaded from[plugins]only when pinned by SHA-256, and each call runs with fuel, memory and time limits and no WASI capabilities. Helm chart repositories (plugins/helm) are built in:index.yaml, chart downloads and ChartMuseum-compatible push, delete and listing, and proxies of other chart repositories. The index lists chart metadata with the types Helm reads, so a Chart.yaml withappVersion: 2or an annotationfalsedoes not break it (#206). A Helm proxy parses its upstream’s index once while the upstream serves the same document, one parse at a time, instead of once per operation (#207). Parsed indexes are kept up to 128 MiB per instance, shared by all Helm proxies: about 60 MB ofindex.yaml, enough for every upstream of a typical group. Past that, the indexes in use stay kept and only the others are parsed again for each operation (#279, #284). NuGet is a built-in plugin too (plugins/nuget), served at/v3/{repo}/by the same dispatcher with the same paths, responses and proxy behaviour as before; its proxies read the upstream feed through the plugin’s upstream side.GET /admin/formatslists the installed formats; the web UI offers them when creating a repository. See docs/format-plugins.md. -
Docker / OCI registry under
/v2/(OCI Distribution Spec v1.1, passing its conformance suite), for container images, Helm charts and other OCI artifacts, as a native format next to NuGet ("format": "docker", listed byGET /admin/formats) with all three repository kinds:- hosted: monolithic, chunked and cross-repository-mount pushes,
deletes, the referrers API for cosign signatures and attestations, and
optional immutable tags (with a
latestexception); - proxy: a pull-through cache of Docker Hub (with its
library/names), GHCR, Quay or any distribution-spec registry, with upstream credentials, tag re-validation byHEADaftermetadata_ttl_seconds, and stale serving while the upstream is down or rate limits; - group: members by priority, pushes to a
writable_member, and theprefer_hosted/reserved_prefixes/ owner guards of NuGet groups.
docker logingoes through a Bearer token service (API tokens, passwords where allowed, or a workload OIDC token), anonymous pulls follow the repository’s anonymous read, and repositories are addressed by path or by an optional host name of their own. Image retention rules (keep the last N tags, stale tags, untagged manifests; aHEADof a manifest counts as a pull, #197) and a sweeper release layers only when no image references them. The web UI lists images, tags, manifests, platforms and referrers. New[docker]settings; see docs/docker.md. (#40) - hosted: monolithic, chunked and cross-repository-mount pushes,
deletes, the referrers API for cosign signatures and attestations, and
optional immutable tags (with a
-
Per-repository upload limits: every hosted repository (NuGet, Helm, docker or a WebAssembly plugin format) has its own upload limit, the largest push it accepts (docker: per blob), editable as Maximum upload size in the web UI and as
max_upload_bytesin the admin API. Without one, a repository accepts 512 MiB (docker 1 GiB per blob).docker.max_blob_bytesnow bounds proxy downloads only. -
Browse by format:
GET /browse/repostakesformatand names each repository’s client URL; the web UI’s Browse screen filters by format, and NuGet and Helm repositories show theirdotnet nugetandhelmcommands (other plugin formats, their URL) like docker ones show theirs. (#38) -
Short-lived S3 credentials from the platform’s workload identity, so no long-lived S3 key has to be stored or rotated.
s3.credentials(FORGE_S3__CREDENTIALS) isstatic(the default,s3.access_key/s3.secret_keyas before),default_chain(the AWS SDK chain: IRSA, EKS Pod Identity, ECS task roles, EC2 instance profiles,AWS_*variables), orweb_identity: a token file exchanged with STSAssumeRoleWithWebIdentityagainst AWS or MinIO / Ceph RGW ([s3.web_identity]:token_file,role_arn,sts_endpoint,session_name,duration_seconds). Credentials are refreshed before they expire and the token file is read again on each refresh. A failed fetch names the credential source and the STS error code, never the token. The short-lived modes refuse S3 keys, and startup warns about a plain-httpsts_endpoint(the token and the temporary keys would cross the network in clear text). The Helm chart hass3.credentialsands3.webIdentity(with an optional projected service-account token), and the kustomize components3-workload-identitymounts one. See docs/deployment.md. (#60) -
Trusted-publishing token exchange:
POST /api/v2/token(nuget.org’s path and request/response shape, soNuGet/loginworks withtoken-service-url) exchanges a CI job’s OIDC token, validated by aworkload_identityrealm, for a short-lived workload API key (forge_wif.…). Each token is redeemed once (itsjtiis required), the key lives for the realm’s newapi_key_ttl_seconds(default an hour) whatever the token’s lifetime, it authorizes and is audited as the token’s workload, and any edit, disabling or deletion of the realm revokes its keys for good.dotnet nuget push --source <feed>/v3/<repo>/package --no-service-endpoint --api-key <key>publishes with the key alone; restore takes it as the Basic password of the usernameoidc. (#169) -
Reserved account names:
oidcis the workload-identity Basic username. Basicoidc:<secret>is only ever checked as a workload credential (an OIDC token or a workload key), never as a user token or a password and never forwarded to a credential-forwarding upstream, and no account can be created, bootstrapped or provisioned under the name (an OIDC login whose username claim isoidcgetsoidc-2). (#169) -
Proxy cache invalidation:
POST /admin/repositories/{name}/proxy-cache/invalidate, and Invalidate cache on a proxy’s page in the web UI, make a NuGet, Helm or docker proxy ask its upstream again before its TTLs run out, for example once a package it answered “not found” for is published upstream. The proxy forgets the upstream’s “not found” answers (of one package with{"package": "<id>"}, or of all) and the listings and documents it cached from the upstream, on every instance; invalidating the whole repository also refreshes the upstream’s service index, and a docker proxy re-validates its tags (of one image, or of all) on their next pull. Downloaded packages, blobs and manifests stay cached, and so does the package-delay clock. -
Local blob cache (optional,
[blob_cache]): a copy of downloaded and pushed blobs on a local disk, sized from the disk (max_bytes = "auto") or fixed, with FIFO eviction that survives a restart. Hot downloads skip S3, concurrent misses of one blob share a single S3 read, and cached packages keep resolving while S3 is unreachable: uncached downloads and pushes then answer 503 withRetry-Afterat once (a circuit breaker),/readyzstays 200 and reports the object storedegraded, and an instance starts without S3. With the cache, package downloads also answerRangerequests. Statistics atGET /admin/blob-cache; Kubernetes componentslocal-cache(emptyDir) andlocal-cache-persistent(a PersistentVolume). The cache directory must belong to Forge’s user and be writable by no one else (startup refuses it otherwise; a directory a KubernetesfsGrouppass made group-writable is restored instead), and its entries are readable by that user alone. The kustomize base and the Helm chart setfsGroupChangePolicy: OnRootMismatch(#195). See docs/deployment.md.
Fixed
- Proxy and group search find packages nobody has restored through Forge yet: a proxy forwards the search to its upstream’s search service, applies the caller’s package filter, not-found markers and the package delay, and answers from its cache when the upstream search fails. The browse UI keeps listing what Forge holds. (#112)
- A proxy pointed back to its previous upstream within the metadata TTL served empty version listings. (#105)
- A proxied package id’s casing could flip between reads; it is the newest version’s, as on nuget.org. (#108)
- The package delay held back old unlisted upstream versions, such as ones a lockfile pins, for the whole delay. They now count from when the upstream created them, and old versions of large packages are dated at their first sighting too. (#111)
- A push with a version longer than 64 characters answered 500; it is refused with 400. (#134)
- An offline nested group was still served through its outer group; it now fails like an offline hosted member. (#106)
- When a group was reachable along two paths, a guard on one path hid packages the other path admits. (#107)
- Downloads are counted in memory and written in batches every few seconds. A burst of downloads of one version no longer queues on its row and holds every database connection. (#119)
- Creating a missing S3 bucket failed on AWS outside us-east-1, and a bucket name owned by another account counted as created. (#113)
- Unknown paths under
/admin,/auth,/browseand/v3answered 200 with the web UI’s page; they answer 404. (#121) PUT /admin/realms/orderanswered 500 when it raced another realm change; it answers 409. (#117)- The browse API served offline hosted and group repositories; it answers 503 like the feed, and the web UI says the repository is offline. (#120)
- A failed OIDC sign-in ended on a plain-text page; it returns to the login page with a message and a way to try again. (#109)
- Web UI: a proxy’s version times are labelled “Cached”, not “Published” (#110); disabling a sign-in realm asks first (#104); unflagging a group’s last owner and removing a member ask first (#122); the user list shows how each user signs in (#123).
Security
- Behind an outbound
HTTP(S)_PROXY, a proxy refuses an upstream host name that resolves to a link-local or cloud-metadata address, and checks every redirect hop the same way. (#125) - Forged Redis invalidations can no longer grow every instance’s memory: the invalidation bookkeeping is bounded. (#126)
- The staging directory is private to Forge’s user, and a staged package is parsed and uploaded from the file that was hashed, not reopened by name. (#127)
- An OIDC-provisioned account can no longer take a credential-forwarding user’s username. (#123)
- Only the ingress proxies can reach Forge when it trusts
X-Forwarded-Forfrom a pod network (NetworkPolicy, see above). (#129) - Release: every
cosign verifypins the tagged commit (#130); re-running a release reuses what it already pushed instead of moving the tags to a new build (#131); the build helpers and the dependency images are pinned by digest (#132); the Helm checksum annotations are salted (#133). - Docker registry tokens are HS256 JWTs bound to
public_base_url, live fordocker.token_ttl_seconds(5 minutes), grant only the scopes the caller held when they were issued, and are accepted under/v2only. A token request names at most 32 repositories (#200). Every registry request is authorized again against current permissions and content selectors, the user and API token behind a token must still be active, and a browser session never pushes or deletes. The token endpoint takes the same Basic credentials as the feeds, so it adds no new password path. The signing key isdocker.token_secret, or a generated key kept in the database, sealed when[crypto]is enabled and re-sealed by DEK rotation; an instance replaces it only when it can never be opened, and the others pick the new key up within 30 seconds (#196). - Registry digests are verified on every upload and every proxied blob and
manifest. Blobs are linked per image: knowing a layer digest grants no
read on another image’s copy, and a mount needs read on the source image.
A docker proxy sends upstream credentials only to its
remote_urlorigin and the token realm the upstream’s challenge names (over https when the upstream is https), and dropsAuthorizationon a redirect to another host. Registry responses carry the feeds’ hardening headers andVary: Authorization, Cookie, Accept.
Documentation
- The deployment guide lists the S3 permissions and the PostgreSQL role
Forge needs, and an S3 delete that is denied names the missing
s3:DeleteObject. (#114)
Known limitations
- With a package delay, proxy search shows a version only once the proxy has recorded it and the delay has passed, because a search answer names no publish times. Totals of a proxy search are the upstream’s, less the ids dropped from the returned page; for a caller whose content selectors hide packages, they count what it sees, plus one while more may follow.
- Behind an outbound proxy, a host name that resolves differently for the proxy than for Forge, or that only the proxy can resolve, still gets past the metadata guard; block those addresses on the proxy as well.
- Format plugins receive no request bodies other than the published
package (which must be the whole body or one multipart part), accept only
Basic credentials, and treat a version with a
-as a prerelease; a response of a plugin may load at most 2,048 query answers. Helm refuses aChart.yamlorindex.yamlwhose YAML aliases would add more than 100,000 nodes, or that nests deeper than 64 levels. Their proxies read the upstream with plain GETs of whole documents (Basic credentials only, no token exchange). - A docker proxy downloads a blob completely and verifies it before it
answers, so a client waits for a large layer’s first fetch; concurrent
pulls of one layer share a download per instance only. A registry upload
of several chunks is assembled into a new blob when it finishes (one-chunk
uploads, what docker, podman and containerd send, are stored once). The
registry has no catalog API (
/v2/_catalog) and no per-repository connector ports.
1.0.0
First production release. envisia-forge is dual-licensed under the GNU AGPL
v3 (AGPL-3.0-only, see LICENSE) or the envisia license, at the user’s
choice. Install it on an empty PostgreSQL database;
docs/deployment.md
lists the required settings.
Features
- NuGet V3 registry with three repository kinds:
- hosted: push, restore, unlist/relist and delete your own packages;
- proxy: on-demand caching of an upstream V3 feed (nuget.org or a private feed with Basic/PAT credentials), with an optional publication delay and negative caching;
- group: one feed URL over several members, by priority. A failing member is tolerated, and a dependency-confusion guard keeps public feeds from shadowing internal package ids.
- Storage: content-addressed blobs on any S3-compatible store; metadata in PostgreSQL; race-free garbage collection with an outbox and a reconciler.
- Access control: fine-grained RBAC with users, nested groups, roles, repository grants, and package/content selectors, enforced on every read and publish path. Tokens can be scoped per repository.
- Authentication: ordered, web-managed auth realms (local passwords, OIDC / Microsoft Entra, and credential forwarding to an existing registry), an identifier-first login, and API keys.
- Trusted publishing: CI jobs on GitHub Actions, GitLab CI and compatible issuers publish with their workload OIDC token, with no stored API key. See docs/trusted-publishing.md.
- Retention: policies with preview, run and audit, and download tracking.
- Secret encryption at rest: double-envelope encryption of upstream passwords and realm secrets, with key rotation.
- Web UI: browse, repository and package detail, self-service tokens, and administration of repositories, users, tokens and realms. It is served by the same process on the same origin.
- Operations:
/healthz,/readyzand/version, graceful SIGTERM, supervised background workers, an audit log of every package change and admin action (theforge::auditlog target), and an optional Redis L2 cache with cross-instance invalidation. - Container image: signed with cosign, with a provenance attestation and
an SBOM that lists the Debian packages, Rust crates and npm packages it
ships. The image carries
LICENSEand the third-party notices in/usr/share/doc/forge/. Each release’s notes print the exact verification command. - Kubernetes manifests in
deploy/kubernetes: a kustomize base for the server, components for an Ingress or Gateway API HTTPRoute, in-cluster Redis, PostgreSQL and S3 (versitygw), two replicas with a PodDisruptionBudget, and following:latest; plus a production and an evaluation overlay. The base pins the image to the release it is tagged with. - Helm chart in
deploy/helm/forge, published with every release tooci://ghcr.io/schmitch/charts/forge(same version as the image, signed with cosign, and attached to the GitHub release). The published chart pulls the release image by the digest the release workflow signed. Forge’s whole configuration is settable asconfig(rendered toforge.toml), secrets through a chart-managed (<fullname>-secret-env) or existing Secret; routing through a Gateway API HTTPRoute and an optional ListenerSet (plus a cert-manager Certificate); PodDisruptionBudget, HorizontalPodAutoscaler, NetworkPolicy andextraObjects; and optional in-cluster PostgreSQL, Redis and versitygw dependencies.
Security
Signing in
- Passwords are verified behind a bounded queue
(
auth.password_verify_concurrency). A memory-bounded login lockout refuses usernames over 256 bytes, keys IPv6 clients by their /64, and backs off a username guessed from many networks, except from networks it signed in from in the last 30 days. Unknown and federated usernames take the same time, so response timing does not reveal which accounts exist. - Usernames are canonical (trimmed, Unicode NFC, lowercased) in every login
path, the lockout, provisioning and admin user creation. Empty names,
control characters and
:are refused. - Browser sessions: a CSRF token on every cookie-authenticated mutation, and
cross-site browser mutations are refused (
Sec-Fetch-Site/Origin), which also covers Basic credentials a browser cached from the feed’s login prompt. On https the session and OIDC flow cookies carry the__Host-prefix, and a request that carries either cookie twice is not authenticated by it. An expired or revoked session cookie is cleared. - OIDC login uses the authorization-code flow with PKCE (S256), a nonce and
a flow bound to the browser. Only RSA signatures are accepted, and
iss,aud,expandnbfare checked. Discovery, key-set and token requests use https and follow no redirects. A realm can require claims (required_claims, e.g. a tenant id), an email claim becomes a username only withemail_verified,link_existing_by_usernamenever takes over an account another credential owns, and a realm’s client secret is dropped when itsissuerorclient_idchanges. Login starts are rate-limited per client network (auth.oidc_login_starts_per_minute). - Credential forwarding requires https unless the upstream is a loopback
address or the realm sets
allow_http. It follows no redirects, ignoresHTTP(S)_PROXY, holds no database connection during the upstream call, and is bounded per realm (auth.credential_forward_max_concurrent) and by the realm’s timeouts. Setidentity_pathfor upstreams that accept a secret under any username (GitHub, Azure DevOps or GitLab personal access tokens): without it, any token holder can sign in as any user. Forge sends the canonical, lowercased username, so an upstream that compares names case-sensitively (an htpasswd file, for example) must know its users in lowercase. Names that Unicode compatibility normalization (NFKC) would change, or that hold invisible characters, are never forwarded. - The users of a deleted credential-forwarding realm are never adopted by another realm. Upstream and IdP admin-group verdicts apply in both directions, and a demotion ends the user’s browser sessions.
- Users can be deactivated (
PATCH /admin/users/{id}with{"active": false}): their sessions and cached verdicts are revoked and their tokens refused. A forced re-authentication (POST /admin/users/{id}/reauth) ends their sessions and revokes the tokens they minted from a credential-forwarding session. The last active administrator cannot be removed, and the last enabled realm people sign in with interactively cannot be disabled or deleted. - The bootstrap admin is created once and stays deleted after removal. Boot
logs a security warning while it still accepts the placeholder password.
Startup refuses placeholder credentials, including a secret that still
holds a
<…>placeholder of the example env files, and warns about a non-https or loopbackpublic_base_urland about disabled secret encryption.
Tokens and the admin API
- The admin API accepts a browser session or an admin-scoped token
(
{"actions": ["admin"]}), never a password sent over Basic, and minting tokens (/admin/me/tokens) takes a session. An unscoped token of an admin can push and restore but gets 403 on/admin/*, so a leaked CI token cannot mint tokens or create admins. An administrator’s password still signs in through/auth/login: give administrators an OIDC realm, and NuGet clients tokens. - An expired or revoked API token, or one of a deactivated user, sent as the Basic password is refused and never tried as a password. A token typed into the web login’s password field is refused the same way, live or not.
- Admin request bodies refuse unknown fields (422 naming the field), and a
realm’s
configrefuses unknown keys (400), as do the[oidc]and[auth.upstream_basic]config sections that seed realms (at startup), so a typo never falls back to an unsafe default. Token listings show each token’s scope. - Every package change (push, redeploy, unlist, relist) and every admin
change is logged at INFO under the
forge::audittarget with the principal’s ids.
Reading and publishing
- RBAC applies on every feed, browse and admin path. A repository name that does not exist answers like one the caller may not read (401 or 403), so names cannot be enumerated.
- With
anonymous_mode = "all", signed-in users can also read what anonymous callers can; underoffandallowlistthey hold only their own grants. Underallowlist, the anonymous grant requires aHostthat matchespublic_base_urlorserver.allowed_hosts(a DNS-rebinding guard), so the reverse proxy must preserve theHostheader.X-Forwarded-Foris honoured only fromauth.trusted_proxy_cidrs, and a malformed hop leaves the client unknown, never allowlisted. - With
allow_redeployon, replacing an existing version needs the delete permission as well as write, so a push-only token cannot overwrite a released version. - Groups guard against dependency confusion:
prefer_hosteddecides over every hosted member, whatever the caller can read; ids underreserved_prefixesare never served from a proxy member; and once a group flags owner members, only owners claim ids. A nested group keeps its own guards when it is read through an outer group. - Packages are read the way NuGet reads them: exactly one
.nuspecat the package root, found with percent-decoded entry names and\as a separator, and one<id>and<version>. Archives with repeated entry names, Info-ZIP Unicode path fields, a<!DOCTYPE>manifest, inconsistent ZIP64 records, or a version NuGet clients cannot read are refused, so Forge and a NuGet client never see different identities for one package. - Feed responses are inert in a browser (
Content-Security-Policy: default-src 'none'; sandbox,Content-Disposition: attachment,nosniff). Package ids are validated against the NuGet id grammar on every route and before a proxy builds an upstream URL. Searchqis capped at 256 bytes and 8 terms.
Proxies, secrets and caches
- A proxy refuses upstream redirects to, and resolutions of, link-local and
cloud-metadata addresses, sends its credentials only to the origin of its
remote_url, drops them whenremote_urlmoves to another origin, and serves an upstream.nuspeconly if it names the requested package. Inside the package-delay window it does not download a version at all. - Encrypted secrets are bound to where they are sent: a proxy password to
the origin of
remote_url, an OIDC client secret to the realm’s issuer and client id. Key rotations are serialized across instances, key material is wiped from memory after use, and an instance without[crypto]refuses to start on a database that holds encrypted secrets. Config errors and theDebugoutput of the configuration never contain secrets. - Redis is a trust boundary: authorization sets, credential verdicts, OIDC
discovery documents and signing keys, workload-identity verdicts and a
proxy’s upstream metadata are cached in each instance’s memory only, and
Redis only relays their invalidations.
rediss://(TLS) URLs are supported. - Trusted publishing: a rule for a custom issuer must pin a stable identity
claim, or one the realm lists in
identity_claims. Unlist, relist and delete by a workload pass the same checks as a publish and are recorded intrusted_publish_audit.
HTTP and supply chain
- Responses carry
X-Content-Type-Options: nosniff,X-Frame-Options: DENY,Content-Security-Policy: frame-ancestors 'none'andReferrer-Policy: same-origin; an https deployment sends HSTS. The web UI has a strict Content Security Policy: scripts only from its own origin plus the hashed inline theme bootstrap, noeval, and fetches only to the same origin. - API responses (
/admin,/auth,/browse) and feed responses to a request with credentials areCache-Control: private, no-store, and feed responses carryVary: Authorization, Cookie, X-NuGet-ApiKey(foranonymous_mode = "allowlist"see Known limitations). - Slow clients are bounded by a header-read timeout
(
server.header_read_timeout_seconds, 150 s), a request-body idle timeout (body_idle_timeout_seconds, 60 s) and a response timeout (request_timeout_seconds, 900 s). Connections are HTTP/1 only. - Release images are signed by digest before any tag points at them, and the verification command pins the exact release workflow. Actions and base images are pinned by digest, CI audits the Rust and shipped web dependencies, and Dependabot keeps them current.
Good to know
- Repository names must match
^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$;newis reserved. In repository updates, an absent field keeps its value and JSONnullclears it. - The retention worker runs a repository automatically at most once a
minute, whatever
default_run_interval_secondsand a policy’smin_run_interval_secondssay.
Known limitations
- NuGet is the only package format.
- Trusted publishing uses the direct-token model: the workload token is
accepted until it expires; there is no exchange for a short-lived Forge
token and no single-use
jtiledger. - Browser OIDC login validates
iss,aud,exp,nbf, signature and nonce but notiatorazp. - Rate limiting is limited to a per-instance, in-memory lockout after failed
passwords. There is no limit on
/auth/identify, token creation or workload-token failures. - Secret encryption is optional (plaintext without a
[crypto]section). Only the env/file root-key providers exist; AWS KMS is not supported yet. - A proxy fetches a new package version once per instance. Several instances that miss the same version at the same time each download it.
- The package delay is dated from the upstream publish time only for the versions in the newest 10 pages (640 versions on nuget.org) of a package’s upstream registration; older versions, and versions of an upstream without a registration, count from the proxy’s first sighting.
- A group’s search is capped at 1,000 results per member, so totals and paging are exact only below that.
- A token whose scope lists a group’s repository id reads through the group only from the members it lists as well: reading a group grants no read on its members.
- Retention has no per-run delete cap.
- No metrics endpoint, request IDs or JSON log output. Shutdown drains in-flight requests without a deadline.
- The web UI requires sign-in, even for repositories that allow anonymous reads; NuGet clients can still read those anonymously.
- There is no web console yet for RBAC roles and grants, retention, user
deactivation or key rotation; use the
/adminAPI. - CI does not scan the container image for OS-package vulnerabilities.
- A proxy trusts its upstream: it follows the resource
@ids and redirects the upstream names, also to private or plain-http addresses (credentials go only to the upstream’s own origin), and parses its metadata documents (up toproxy.max_metadata_bytes, 32 MiB) in memory. Point proxies only at upstreams you trust. Behind an outbound proxy (HTTP(S)_PROXY) the proxy resolves host names, so Forge refuses only literal cloud-metadata addresses and the well-known metadata host names there. - Unless a group flags owner members, it trusts all its hosted members
equally: anyone who can push to any hosted member can publish under the
group’s
reserved_prefixes, and withprefer_hostedcan claim any public package id for the whole group. Flag the trusted members as owners when a group mixes feeds with different writers. Owners claim only the ids they have: an id no owner has is still served by every member, so reserve the internal prefixes as well. - With
prefer_hosted, a caller who cannot read a group’s hosted members still learns whether one of them has a package id the upstream also has: they get 404 instead of the public package. - A role grant of the
adminaction on one repository has no effect: every/adminoperation needs a global administrator. - An OIDC realm that provisions accounts can take a username before a credential-forwarding user of that name first signs in, where users choose their own username claim at the IdP. That user is then refused until an administrator removes the other account.
- Under
anonymous_mode = "allowlist", anonymous feed responses carry noCache-Control: no-store, and whether one is served depends on the client’s address, which a cache does not key on. A shared cache in front of Forge that stores responses without caching headers (Varnish’s default does) can serve a package an allowlisted client fetched to a client outside the allowlist. Do not let such a cache store/v3responses. - While Redis is unreachable, an instance hears no invalidations from the others and converges by TTL (60 s for authorization). Anything that can publish to Redis can grow every instance’s memory: the invalidation bookkeeping is not bounded.
- Every
.nupkgdownload updates its version’s download counters in PostgreSQL, unthrottled. Hundreds of downloads of one version per second queue on those rows and can hold every database connection. - Forge does not check the owner or permissions of
server.staging_dir. On a host shared with other users, use a directory only Forge’s user can write (the container image and the Kubernetes manifests are not affected). - The Helm chart rolls the pods on a change of
secretEnv(and of the in-cluster Redisauth) through an unsalted SHA-256 of the values in a pod annotation: whoever can read the Deployment can test guesses of a weak secret against it. - The release build pulls the Dockerfile frontend, BuildKit and the SBOM scanner images by tag, not by digest.
- Login lockouts and failure counts are per instance and lost on restart (the networks a username signed in from are kept in the database).
- A first download through a proxy runs inside the request, so
server.request_timeout_seconds(900) caps it belowproxy.download_timeout_seconds(1800); NuGet clients give up far earlier anyway.