feat(pki): reconcile recovered signing operations
Add bounded durable reconciliation with retry metadata, fencing-safe status and cancellation handling, and server-managed background recovery. Include versioned persistence migration, bounded keyset paging, lifecycle-safe worker shutdown, redacted diagnostics, and restart/failure coverage. Closes #10
This commit is contained in:
@@ -8,7 +8,7 @@ An ACME directory is an immutable policy revision bound to one realm, logical au
|
||||
|
||||
Direct deployments use server-authenticated TLS. A client TLS certificate is not ACME account authority; ACME identity is the account key authenticated by JWS. Trusted-reverse-proxy deployments require the established mutually authenticated proxy-to-ZeroEcho TLS hop and a dedicated enabled proxy principal with `FORWARD_AUTHENTICATED_CLIENT_IDENTITY`. Forwarded administrative identity is not used as an ACME account. Source addresses and `Forwarded` or `X-Forwarded-*` headers never authorize ACME.
|
||||
|
||||
The server configuration schema is version 5. ACME is disabled explicitly with:
|
||||
The server configuration schema is version 6. ACME is disabled explicitly with:
|
||||
|
||||
```json
|
||||
"acmeListener": {"enabled": false}
|
||||
|
||||
@@ -84,8 +84,23 @@ Responses are deterministic version-one JSON. They carry only safe typed results
|
||||
|
||||
Transport workers, transport backlog, concurrent operation workers, waiting operation capacity, admitted requests, request body size, deadlines, and both shutdown periods are finite configuration values. Saturation never runs a request on the caller thread and a rejected request never invokes the gateway.
|
||||
|
||||
Configuration schema version 6 adds `signingReconciliation`. Its production
|
||||
defaults are enabled, 256 examined records, 64 provider calls, a 30-second pass
|
||||
timeout, a 10-second provider-call timeout, and a fixed 2-second idle interval.
|
||||
`maxRecords` is 1 through 4096, `maxCalls` is 1 through the record
|
||||
bound, all durations are positive, and the provider timeout cannot exceed the
|
||||
pass timeout. Version 5 configurations migrate to these defaults. Set `enabled`
|
||||
to `false` only when an external owner deliberately invokes the transport-neutral
|
||||
session reconciliation API.
|
||||
|
||||
Shutdown first removes readiness and admission, stops listener acceptance, drains for the configured graceful period, cooperatively cancels remaining work, waits only for the forced period, then closes the one realm context. Already committed PKI changes remain committed.
|
||||
|
||||
Signing-provider deadlines and cancellation are cooperative for trusted
|
||||
in-process implementations. A provider call that remains live after forced
|
||||
shutdown causes `SIGNING_RECONCILIATION_SHUTDOWN_TIMEOUT`; the session, provider,
|
||||
and store stay open and `close` may be retried after the call returns. Hard
|
||||
isolation for non-cooperative providers requires an out-of-process boundary.
|
||||
|
||||
## Request correlation
|
||||
|
||||
Clients may supply `X-ZeroEcho-Request-Id` using 16–128 ASCII letters, digits, underscore, or hyphen. Otherwise the server creates a random opaque identifier. It is correlation metadata—not an idempotency key, object identity, approval, or capability token.
|
||||
|
||||
@@ -243,9 +243,47 @@ retires advisory state before reporting that marker. The retained record does no
|
||||
regain a live content reference. Store restart recovery uses authoritative live
|
||||
references to reclaim the orphaned staged file safely.
|
||||
|
||||
Each fully opened realm owns exactly one signing-reconciliation worker. The
|
||||
worker starts immediately before the server becomes ready, runs one synchronous
|
||||
bounded pass at a time, and self-schedules the next pass only after the current
|
||||
pass returns. It has no queue of missed ticks and creates no active-active lease;
|
||||
the existing realm, provider, and filesystem ownership locks remain the
|
||||
single-active authority.
|
||||
|
||||
The reconciliation cursor is an advisory exclusive key into a metadata snapshot
|
||||
ordered by canonical submission identifier. A pass examines no more than its
|
||||
record bound, makes no more than its independent provider-call budget, and wraps
|
||||
to the beginning after reaching the end so deferred records cannot starve other
|
||||
records. Per-record failures are isolated and persisted separately from the
|
||||
provider detail code with a store-time retry schedule of 2, 4, 8, 16, then 30
|
||||
seconds. Terminal and retired records carry no retry metadata.
|
||||
|
||||
Shutdown first stops reconciliation admission and scheduling, then cooperatively
|
||||
cancels and drains the active pass before closing the realm session, providers,
|
||||
or store. If a provider ignores cancellation beyond the forced-shutdown period,
|
||||
the server reports `SIGNING_RECONCILIATION_SHUTDOWN_TIMEOUT` and leaves those
|
||||
dependencies open so close can be retried without use-after-close behavior.
|
||||
|
||||
`SignatureWorkflow` implementations are trusted in-process components and every
|
||||
submit, status, verify, and cancel call now requires a `CallControl` carrying an
|
||||
absolute deadline and cancellation signal. This is an intentionally breaking
|
||||
provider-SPI migration: implementations must check the control before and
|
||||
between I/O, before an externally visible effect, and before returning. These
|
||||
controls are cooperative bounds; the server does not kill provider threads or
|
||||
close their dependencies while a violating call remains live. Providers that
|
||||
need enforcement against untrusted or non-cooperative code require a future
|
||||
out-of-process provider boundary with process or RPC isolation.
|
||||
|
||||
Payload staging remains streaming `O(n)` time and `O(1)` aggregate auxiliary heap
|
||||
excluding the signature. Synchronous waiting performs
|
||||
`O(TTL / polling interval)` status observations.
|
||||
`O(TTL / polling interval)` status observations. For `M` metadata keys and a
|
||||
bound `B`, an ordered reconciliation page is `O(log M + B)` traversal and each
|
||||
candidate point access is `O(log M)`, for `O(B log M)` candidate access in the
|
||||
worst case. Normal pass auxiliary memory is `O(B)`. A concurrent metadata commit
|
||||
may detach the currently pinned index generation once in `O(M)` time and memory;
|
||||
later commits do not copy again merely because an older generation remains
|
||||
pinned. A pass uses at most `maximumProviderCalls` external calls, and each realm
|
||||
owns `O(1)` scheduler state.
|
||||
|
||||
## 7. Authorization architecture
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"version": 5,
|
||||
"version": 6,
|
||||
"serverName": "zeroecho-admin",
|
||||
"realm": {
|
||||
"realmId": "production",
|
||||
@@ -74,6 +74,7 @@
|
||||
"gracefulShutdownMillis": 30000,
|
||||
"forcedShutdownMillis": 10000
|
||||
},
|
||||
"signingReconciliation": {"enabled": true, "maxRecords": 256, "maxCalls": 64, "passTimeoutMillis": 30000, "providerCallTimeoutMillis": 10000, "idleIntervalMillis": 2000},
|
||||
"publicListener": {
|
||||
"enabled": true,
|
||||
"address": "127.0.0.1",
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"version": 5,
|
||||
"version": 6,
|
||||
"serverName": "zeroecho-admin",
|
||||
"realm": {
|
||||
"realmId": "production",
|
||||
@@ -74,6 +74,7 @@
|
||||
"gracefulShutdownMillis": 30000,
|
||||
"forcedShutdownMillis": 10000
|
||||
},
|
||||
"signingReconciliation": {"enabled": true, "maxRecords": 256, "maxCalls": 64, "passTimeoutMillis": 30000, "providerCallTimeoutMillis": 10000, "idleIntervalMillis": 2000},
|
||||
"publicListener": {"enabled": false},
|
||||
"acmeListener": {"enabled": false},
|
||||
"runtime": {}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"version": 5,
|
||||
"version": 6,
|
||||
"serverName": "zeroecho-admin-nginx",
|
||||
"realm": {
|
||||
"realmId": "production",
|
||||
@@ -40,6 +40,7 @@
|
||||
"maximumForwardedChainBytes": 524288
|
||||
},
|
||||
"execution": {"transportWorkers":8,"transportQueueCapacity":64,"operationWorkers":4,"operationQueueCapacity":32,"maximumAdmittedRequests":96,"defaultDeadlineMillis":30000,"maximumDeadlineMillis":120000,"gracefulShutdownMillis":30000,"forcedShutdownMillis":10000},
|
||||
"signingReconciliation": {"enabled":true,"maxRecords":256,"maxCalls":64,"passTimeoutMillis":30000,"providerCallTimeoutMillis":10000,"idleIntervalMillis":2000},
|
||||
"publicListener": {"enabled":false},
|
||||
"acmeListener": {"enabled":false},
|
||||
"runtime": {}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"version": 5,
|
||||
"version": 6,
|
||||
"serverName": "zeroecho-admin-proxy",
|
||||
"realm": {
|
||||
"realmId": "production",
|
||||
@@ -41,6 +41,7 @@
|
||||
"maximumForwardedChainBytes": 524288
|
||||
},
|
||||
"execution": {"transportWorkers":8,"transportQueueCapacity":64,"operationWorkers":4,"operationQueueCapacity":32,"maximumAdmittedRequests":96,"defaultDeadlineMillis":30000,"maximumDeadlineMillis":120000,"gracefulShutdownMillis":30000,"forcedShutdownMillis":10000},
|
||||
"signingReconciliation": {"enabled":true,"maxRecords":256,"maxCalls":64,"passTimeoutMillis":30000,"providerCallTimeoutMillis":10000,"idleIntervalMillis":2000},
|
||||
"publicListener": {
|
||||
"enabled":true,
|
||||
"address":"127.0.0.1",
|
||||
|
||||
Reference in New Issue
Block a user