Alwafi administrator and operator manual

Audience: Alwafi owners, administrators, and operators. This is not a consumer client document and must not be bundled into Expressly with administrator credentials.

Verified 2026-07-14 against the testing deployment.

Claude Code instruction

Give Claude Code this file and GET /openapi.json. Tell it:

Implement or automate only the Alwafi control-plane functions authorized for this task. Use administrator sessions only against /admin/v1/*. Never expose administrator sessions, provider credentials, customer tokens, webhook secrets, license signing material, or the encryption master key. Treat the OpenAPI document as the wire-schema authority. Preserve auditability and the disabled-by-default behavior of newly installed routes.

Administration endpoints

Purpose	URL
Local admin portal	http://127.0.0.1:3001
Testing admin portal	https://alwafi-admin.expressly.tools
Testing API	https://alwafi-api.expressly.tools
API contract	https://alwafi-api.expressly.tools/openapi.json

The testing administrator is protected first by Cloudflare Access and then by an Alwafi administrator session. Cloudflare Access identity does not replace the application login. Do not put email, password, session, or API keys in URL query parameters.

Roles

• owner — full control plane, administrator management, and destructive governance operations.
• operator — day-to-day providers, routes, users, credits, jobs, and operational actions allowed by the API authorization policy.
• viewer — read-only monitoring and audit access.

Use the least privileged role. Create a separate administrator identity for automation instead of sharing the owner's session.

Daily operating sequence

1. Check Overview for readiness, request volume, provider health, worker heartbeat, recent failures, and wallet pressure.
2. Check Providers for stale tests, degraded catalogs, failed credentials, and circuit-breaker failures.
3. Check Routes & models for enabled state, priorities, public aliases, reservations, and current prices.
4. Check Async jobs for old queued/retrying work and exhausted webhooks.
5. Check Usage & cost and Request logs for unexpected spend, latency, failures, or estimated usage.
6. Check System operations for worker heartbeat, backup, and restore checkpoints.

Provider management

Provider credentials entered in the portal are encrypted with AES-256-GCM and stored as ciphertext in PostgreSQL. The deployment master key remains outside the database. Never paste credentials into logs, issues, screenshots, URLs, or documentation.

For every provider:

1. Create the connection with a clear name, provider kind, base URL, timeout, and enabled state.
2. Save the credential securely.
3. Run Test credentials & models.
4. Verify Last tested, Last successful, health, failure count, discovered models, and assigned models.
5. Create routes only after the model appears or its separate native endpoint has been tested.
6. Re-test after credential rotation or provider catalog changes.

Current tested connections include OpenAI, ElevenLabs, and DigitalOcean Serverless Inference. OpenAI-compatible, Anthropic, and Gemini adapters are supported when configured.

DigitalOcean

Use the DigitalOcean preset with https://inference.do-ai.run/v1. The recommended pack installs six disabled routes: GPT OSS 20B, Qwen3 32B, Qwen3 embeddings, Flux Schnell, Stable Diffusion 3.5 Large, and GPT Image 2.

Keep the pack disabled until prices and reservation ceilings are approved. Flux uses DigitalOcean asynchronous invocation and status polling; it is not expected in the OpenAI-compatible model catalog. Run scripts/smoke-digitalocean-inference.mjs before activation.

Routes, prices, and concurrency

A route maps a stable public alias and capability to an upstream provider/model. Clients see only the public alias. Review:

• provider connection and upstream model;
• capability and priority;
• enabled/disabled state;
• input/output token rates or native meter rate;
• image price configuration;
• minimum charge and maximum reservation;
• fallback public model; and
• token-policy access.

New recommended routes are disabled by design. Enable incrementally and monitor health, request logs, usage settlement, and wallet debits. Alwafi worker concurrency does not override provider rate or concurrency quotas.

Users, tokens, and credits

Users & credits manages service identities used by Expressly and other API clients. Customer accounts manages customer login, licensing, plans, subscriptions, and devices. They are related but not interchangeable.

For a service user:

1. Create or activate the user.
2. Grant only the required microcredits.
3. Mint a token with explicit scopes, allowed models, rate/spend policy, and expiry.
4. Copy the token once into the client's secret store.
5. Verify /v1/models, /v1/pricing, /v1/me/balance, and one low-cost call.
6. Revoke unused or exposed tokens immediately.

The portal never recovers a token secret after creation. Token lists show safe metadata only. Credit adjustments are immutable ledger entries; correct a mistake with a compensating entry instead of editing history.

Customer accounts, billing, and provisioning

The customer plane covers registration, verification, login, BYOK selection, trial/subscription flows, license issuance, and device revocation. These flows fail closed until their production integrations are configured: remote P-256 signing, transactional email, Google OAuth, Stripe, and reviewed credit grants.

Do not advertise customer signup, billing, or production license issuance as live merely because the pages render. Verify each integration end to end and record the checkpoint.

Playground

The playground has two different purposes:

• Client/routed mode uses the same public model aliases, authentication, routing, metering, and normalized response that a consumer sees.
• Direct provider mode is an owner/operator diagnostic. It bypasses normal client routing and billing so the stored credential and provider-native response can be inspected.

Always run the routed mode before declaring a service client-ready. Direct mode alone does not prove wallet reservation, route selection, normalization, settlement, SSE, or artifact behavior. The preview may render text, images, audio, embeddings, jobs, artifacts, or SSE, while the JSON panel preserves the wire response.

Jobs, SSE, and webhooks

Use Async jobs to inspect queue state, attempts, retry state, terminal error code, settled usage, charge, and artifacts. Do not resubmit a client's work to repair an observation problem. SSE and polling are observation channels over the same durable job.

Webhook delivery is at least once. Investigate pending, retrying, or exhausted deliveries without revealing the signing secret. A receiver deduplicates by X-Alwafi-Event-ID and fetches the canonical job.

Usage and request telemetry

Usage records report the native meter, measured quantity, source, and charged microcredits. Request logs contain bounded operational metadata and must not contain prompts, generated content, credentials, or customer secrets. Separate:

• authentication/policy errors (401, 403);
• wallet errors (402);
• rate policy (429);
• upstream/transport failure (502 or durable retry/failure); and
• operator configuration errors such as disabled routes.

System operations

The testing runtime is native systemd plus PostgreSQL—no Docker. Gateway and worker scale independently through PostgreSQL leases. Cloudflare Tunnel is the only HTTP ingress and application/database ports remain loopback-only.

Operators verify:

• API readiness and worker heartbeats;
• stale reservations and reconciliation;
• retention cleanup;
• nightly PostgreSQL backup validation;
• periodic isolated restore verification;
• private artifact storage health; and
• graceful worker drain before deployment.

Do not mark a backup green because a file exists. A successful archive check and isolated restore checkpoint are required.

Security boundaries

• Never give an Expressly client an administrator session.
• Never expose direct-provider diagnostics on the customer hostname.
• Never log provider keys, consumer token secrets, passwords, session cookies, signing keys, webhook secrets, or the provider encryption master key.
• Keep the administrator behind Cloudflare Access.
• Keep routes disabled until credential, model, price, and reservation checks pass.
• Preserve audit events for administrator mutations and direct diagnostics.
• Use separate recovery storage for the provider-secret master key; database backups contain only encrypted provider credentials.

Change and release checklist

• OpenAPI reflects every changed endpoint and schema.
• Admin portal tests, lint, typecheck, optimized build, and exact production source-line coverage pass.
• Gateway Clippy, integration tests, and exact source-line coverage pass.
• Provider live tests use sanitized evidence and approved small spend.
• Database migration is backward compatible and serialized.
• Backup succeeds before binary replacement.
• Worker drains and restarts with a current heartbeat.
• Public API, customer hostname isolation, and Access-protected admin are probed.
• Documentation and the Expressly implementation handoff are updated.

Incident triage

Symptom	First checks
Provider unavailable	Last test, credential, discovered/assigned models, circuit failures
Jobs stuck queued	Worker heartbeat, concurrency, database lease, enabled route
Repeated retrying	Provider health, timeout, quota, sanitized job error
402	Available balance, reservation ceiling, pending reservations
Missing model	Route enabled state, token policy, provider catalog
SSE reconnect loop	Last event ID, job ownership, database notifications, polling result
Artifact failure	Stored metadata, backend health, authorization, content type
Billing mismatch	Settled usage source, price effective time, ledger entry, reservation

The public OpenAPI document is the canonical field-level reference. The portal API reference renders it with searchable operations and language examples.