feat(pki): reconcile recovered signing operations

Add bounded durable reconciliation with retry metadata, fencing-safe
status and cancellation handling, and server-managed background recovery.

Include versioned persistence migration, bounded keyset paging, lifecycle-safe
worker shutdown, redacted diagnostics, and restart/failure coverage.

Closes #10
This commit is contained in:
2026-08-12 01:48:11 +02:00
parent 67989b232f
commit 0312cf699f
42 changed files with 2318 additions and 275 deletions

View File

@@ -8,7 +8,7 @@ An ACME directory is an immutable policy revision bound to one realm, logical au
Direct deployments use server-authenticated TLS. A client TLS certificate is not ACME account authority; ACME identity is the account key authenticated by JWS. Trusted-reverse-proxy deployments require the established mutually authenticated proxy-to-ZeroEcho TLS hop and a dedicated enabled proxy principal with `FORWARD_AUTHENTICATED_CLIENT_IDENTITY`. Forwarded administrative identity is not used as an ACME account. Source addresses and `Forwarded` or `X-Forwarded-*` headers never authorize ACME.
The server configuration schema is version 5. ACME is disabled explicitly with:
The server configuration schema is version 6. ACME is disabled explicitly with:
```json
"acmeListener": {"enabled": false}

View File

@@ -84,8 +84,23 @@ Responses are deterministic version-one JSON. They carry only safe typed results
Transport workers, transport backlog, concurrent operation workers, waiting operation capacity, admitted requests, request body size, deadlines, and both shutdown periods are finite configuration values. Saturation never runs a request on the caller thread and a rejected request never invokes the gateway.
Configuration schema version 6 adds `signingReconciliation`. Its production
defaults are enabled, 256 examined records, 64 provider calls, a 30-second pass
timeout, a 10-second provider-call timeout, and a fixed 2-second idle interval.
`maxRecords` is 1 through 4096, `maxCalls` is 1 through the record
bound, all durations are positive, and the provider timeout cannot exceed the
pass timeout. Version 5 configurations migrate to these defaults. Set `enabled`
to `false` only when an external owner deliberately invokes the transport-neutral
session reconciliation API.
Shutdown first removes readiness and admission, stops listener acceptance, drains for the configured graceful period, cooperatively cancels remaining work, waits only for the forced period, then closes the one realm context. Already committed PKI changes remain committed.
Signing-provider deadlines and cancellation are cooperative for trusted
in-process implementations. A provider call that remains live after forced
shutdown causes `SIGNING_RECONCILIATION_SHUTDOWN_TIMEOUT`; the session, provider,
and store stay open and `close` may be retried after the call returns. Hard
isolation for non-cooperative providers requires an out-of-process boundary.
## Request correlation
Clients may supply `X-ZeroEcho-Request-Id` using 16128 ASCII letters, digits, underscore, or hyphen. Otherwise the server creates a random opaque identifier. It is correlation metadata—not an idempotency key, object identity, approval, or capability token.

View File

@@ -243,9 +243,47 @@ retires advisory state before reporting that marker. The retained record does no
regain a live content reference. Store restart recovery uses authoritative live
references to reclaim the orphaned staged file safely.
Each fully opened realm owns exactly one signing-reconciliation worker. The
worker starts immediately before the server becomes ready, runs one synchronous
bounded pass at a time, and self-schedules the next pass only after the current
pass returns. It has no queue of missed ticks and creates no active-active lease;
the existing realm, provider, and filesystem ownership locks remain the
single-active authority.
The reconciliation cursor is an advisory exclusive key into a metadata snapshot
ordered by canonical submission identifier. A pass examines no more than its
record bound, makes no more than its independent provider-call budget, and wraps
to the beginning after reaching the end so deferred records cannot starve other
records. Per-record failures are isolated and persisted separately from the
provider detail code with a store-time retry schedule of 2, 4, 8, 16, then 30
seconds. Terminal and retired records carry no retry metadata.
Shutdown first stops reconciliation admission and scheduling, then cooperatively
cancels and drains the active pass before closing the realm session, providers,
or store. If a provider ignores cancellation beyond the forced-shutdown period,
the server reports `SIGNING_RECONCILIATION_SHUTDOWN_TIMEOUT` and leaves those
dependencies open so close can be retried without use-after-close behavior.
`SignatureWorkflow` implementations are trusted in-process components and every
submit, status, verify, and cancel call now requires a `CallControl` carrying an
absolute deadline and cancellation signal. This is an intentionally breaking
provider-SPI migration: implementations must check the control before and
between I/O, before an externally visible effect, and before returning. These
controls are cooperative bounds; the server does not kill provider threads or
close their dependencies while a violating call remains live. Providers that
need enforcement against untrusted or non-cooperative code require a future
out-of-process provider boundary with process or RPC isolation.
Payload staging remains streaming `O(n)` time and `O(1)` aggregate auxiliary heap
excluding the signature. Synchronous waiting performs
`O(TTL / polling interval)` status observations.
`O(TTL / polling interval)` status observations. For `M` metadata keys and a
bound `B`, an ordered reconciliation page is `O(log M + B)` traversal and each
candidate point access is `O(log M)`, for `O(B log M)` candidate access in the
worst case. Normal pass auxiliary memory is `O(B)`. A concurrent metadata commit
may detach the currently pinned index generation once in `O(M)` time and memory;
later commits do not copy again merely because an older generation remains
pinned. A pass uses at most `maximumProviderCalls` external calls, and each realm
owns `O(1)` scheduler state.
## 7. Authorization architecture

View File

@@ -1,5 +1,5 @@
{
"version": 5,
"version": 6,
"serverName": "zeroecho-admin",
"realm": {
"realmId": "production",
@@ -74,6 +74,7 @@
"gracefulShutdownMillis": 30000,
"forcedShutdownMillis": 10000
},
"signingReconciliation": {"enabled": true, "maxRecords": 256, "maxCalls": 64, "passTimeoutMillis": 30000, "providerCallTimeoutMillis": 10000, "idleIntervalMillis": 2000},
"publicListener": {
"enabled": true,
"address": "127.0.0.1",

View File

@@ -1,5 +1,5 @@
{
"version": 5,
"version": 6,
"serverName": "zeroecho-admin",
"realm": {
"realmId": "production",
@@ -74,6 +74,7 @@
"gracefulShutdownMillis": 30000,
"forcedShutdownMillis": 10000
},
"signingReconciliation": {"enabled": true, "maxRecords": 256, "maxCalls": 64, "passTimeoutMillis": 30000, "providerCallTimeoutMillis": 10000, "idleIntervalMillis": 2000},
"publicListener": {"enabled": false},
"acmeListener": {"enabled": false},
"runtime": {}

View File

@@ -1,5 +1,5 @@
{
"version": 5,
"version": 6,
"serverName": "zeroecho-admin-nginx",
"realm": {
"realmId": "production",
@@ -40,6 +40,7 @@
"maximumForwardedChainBytes": 524288
},
"execution": {"transportWorkers":8,"transportQueueCapacity":64,"operationWorkers":4,"operationQueueCapacity":32,"maximumAdmittedRequests":96,"defaultDeadlineMillis":30000,"maximumDeadlineMillis":120000,"gracefulShutdownMillis":30000,"forcedShutdownMillis":10000},
"signingReconciliation": {"enabled":true,"maxRecords":256,"maxCalls":64,"passTimeoutMillis":30000,"providerCallTimeoutMillis":10000,"idleIntervalMillis":2000},
"publicListener": {"enabled":false},
"acmeListener": {"enabled":false},
"runtime": {}

View File

@@ -1,5 +1,5 @@
{
"version": 5,
"version": 6,
"serverName": "zeroecho-admin-proxy",
"realm": {
"realmId": "production",
@@ -41,6 +41,7 @@
"maximumForwardedChainBytes": 524288
},
"execution": {"transportWorkers":8,"transportQueueCapacity":64,"operationWorkers":4,"operationQueueCapacity":32,"maximumAdmittedRequests":96,"defaultDeadlineMillis":30000,"maximumDeadlineMillis":120000,"gracefulShutdownMillis":30000,"forcedShutdownMillis":10000},
"signingReconciliation": {"enabled":true,"maxRecords":256,"maxCalls":64,"passTimeoutMillis":30000,"providerCallTimeoutMillis":10000,"idleIntervalMillis":2000},
"publicListener": {
"enabled":true,
"address":"127.0.0.1",