Operations and emergencies
Summary. This page describes how WKL TrustBridge responds to outages and compromise: which function remains available when which component fails, how much time there is for recovery, how revocation of certificates and relationships works when central services are missing, and how a compromised CA key is handled. The guiding principle is that status information is more robust than issuance: a missing issuance causes delay, a missing revocation causes harm. The default values at the end of the page apply as long as an organization sets nothing else.
Terms
- RPO (recovery point objective) – the maximum tolerable data loss, measured in time.
- RTO (recovery time objective) – the maximum tolerable time until recovery.
- Standby – a prepared second instance that takes over on failure but does not run in parallel.
- Emergency revocation – revocation directly at the worker host, without database or management.
Failure domains
| One-day outage of | Issuance | Revocation | Status information | Management and connections |
|---|---|---|---|---|
| SQL Server | Stopped | Emergency revocation at the worker host | Runs over the second path | Stopped |
| Management | Automatic classes run; manual ones stop | By the holder through a protocol; emergency revocation | Runs | Stopped |
| Enrollment | Automatic classes stop | Through management | Runs | Runs |
| CA worker | Stopped | Orders wait | Last CRL stays valid for 66 h; the standby takes over | Runs |
| HSM | Stopped | Stopped | As for the worker | Runs |
| Distribution | Runs | Runs | Relying parties use the last fetched list until nextUpdate; designed for high availability | Runs |
| A host's key agent | Runs | Runs | Runs | Runs; this host's existing connections run until the end of their lifetime, and new ones are opened as long as a valid secret is available |
| A host's TLS adapter | Runs | Runs | Runs | Runs; this host's connections to partners are down |
graph TD W[CA worker fails] --> L[last CRL valid<br/>until nextUpdate] L --> T1[0 to 66 h:<br/>validations succeed] L --> T2[after 66 h without a new list:<br/>validations fail closed] B[Standby worker] -->|target: well under 66 h| N[new CRL] N --> T1 classDef ok fill:#dff6dd,stroke:#107c10,color:#201f1e classDef ko fill:#fde7e9,stroke:#a4262c,color:#201f1e class T1 ok class T2 ko classDef tcb fill:#fde7e9,stroke:#a4262c,color:#201f1e class B,W tcb classDef neu fill:#f3f2f1,stroke:#605e5c,color:#201f1e class L,N neu
Fail closed. If a CRL expires and no new one is available, relying parties reject. That is intentional: a relying party that accepts without valid status would accept revoked certificates. The price is an availability risk, which the 66-hour buffer and the standby contain.
Recovery objectives
| Component | RPO | RTO | Means |
|---|---|---|---|
| Worker ledger | Close to zero | 4 hours (standby switchover time) | Continuous backup; standby; reconciliation after restore |
| HSM | – (keys only in the HSM and its backup) | As for the worker | Standby HSM; backup under a multi-person rule |
| Distribution | – (pre-signed artifacts) | Minutes | Several instances |
| Database | 15 minutes | 4 hours | Backup; missing audit events from the external copy |
| Offline root | – | Days | Two safes, recovery table |
Restoring the ledger
The ledger is the single source of truth about issuance and revocation. After a restore from backup it may lag behind the database. The worker then proceeds as follows:
- It records the restore, together with the last known anchor, as an event of its own; falling back behind an anchor is thus explained, not hidden.
- It takes over projected certificates from the database whose CA signature is valid before processing new orders. That way a serial number that has already been used is never assigned again.
- It produces a new CRL with a CRL number above every number already published.
Emergency revocation
If the database or the management interface is unavailable, an authorized person revokes at the console of the worker host. The revocation order goes straight into the ledger; the new CRL reaches distribution over the second path. This requires physical or secured remote access to the worker host and a rehearsed procedure; the steps are in Revoke a certificate.
Revoking a relationship in an emergency
Relationship revocation needs neither the other side nor the database. If management is unavailable, an authorized person revokes at the console of the worker host; the worker produces a new trust bundle without the relationship, and the TLS adapters fetch it over the second path. Until the adapters have received the new bundle, the lifetime of the old bundle bounds the effect. The steps are in Revoke a trust relationship.
Compromise of a CA key
A compromise shows up in the hourly reconciliation of the HSM's signature counters against the ledger, or is reported from outside. The response has six steps – detect, contain, report, revoke, rebuild, follow up – described in Respond to a compromised CA key. An issuing CA is revoked only in a ceremony at the root; the partners' hierarchies are not affected.
If the root is compromised, the organization's hierarchy has to be rebuilt and the new anchor brought into all of its relationships. The partners' hierarchies are not affected. Offline operation, the multi-person rule, and the safes protect against this.
Rotation and changeover
| Item | Rule |
|---|---|
| Certificates in automatic classes | Renewal once only a third of the lifetime remains; minimum lifetime three days |
| Issuing CA | A new generation before the remaining lifetime falls below the longest profile lifetime; the old one then signs only CRLs |
| Algorithm change | A new generation with the new algorithm, overlapping anchors (see Cryptography and the post-quantum transition) |
| HSM | Replacement changes only the reference; keys move through the HSM's backup mechanism |
| Personal keys for management | An expiry date per key; revocable at any time, with an audit entry |
| Approval keys | Changed only in the rule set, under the four-eyes principle |
| The key agents' additional secret | Renewed on the relationship's schedule; connections are rotated at their maximum lifetime |
| Agent certificates | Automatic renewal through ACME or EST, like system certificates |
Monitoring
| Signal | Meaning |
|---|---|
| Age of the most recent CRL per CA | Approaching nextUpdate means the worker or the publication path is disrupted |
Orders in state claimed above a threshold | The worker is hanging, or poison orders |
| Discrepancy between signature counter and ledger | Suspected compromise |
| Time since the last external anchor | The unanchored period is growing |
| Remaining lifetime of the CAs | Lead time for the generation change |
| Age of the trust bundle per TLS adapter | The adapter is no longer receiving relationship changes |
| Rejected connections per relationship and reason | Misconfiguration, expired relationship, or attack |
| Age of the most recent secret per agent pair | The other side is unreachable or the exchange is disrupted |
Emergency and recovery tests
Every organization runs the following tests before going live and at least once a year after that. Exercises involving the root use a test root.
| Test | Procedure | Passed when |
|---|---|---|
| Worker outage | Stop the worker, let the standby take over | New CRL well before the old one expires; no duplicate issuance; no gap in CRL numbers |
| Database outage | Stop the database; emergency revocation at the console | The revocation appears in the next list; the list reaches distribution over the second path |
| Ledger restore | Restore the ledger from an older backup | Reconciliation takes over projected certificates; no serial number used twice; restore recorded as an event |
| Database restore | Restore the database from backup | Missing audit events filled in from the external copy; chain consistent from the last anchor |
| CA compromise | Simulate a counter discrepancy on a test CA | Alert; issuance stopped; test CA revoked in the test ceremony; reporting path walked through |
| Distribution outage | Shut down one instance | CRL retrieval still possible |
| Trial ceremony | Complete ceremony at the test root | Record complete, roles observed, artifacts as prescribed |
| Relationship revocation | Revoke a test relationship during live traffic | New connections rejected immediately; existing ones end within the period |
| Key agent outage | Stop one side's agent | Existing connections run until their lifetime ends; after that, no new connection without a valid secret; alert |
Default values
The following values apply as long as an organization sets nothing else. Deviations are entered in the rule set or the relationship policy and recorded in the ledger.
| Value | Default | Rationale |
|---|---|---|
| Validity of an issuing CA's CRL | 72 hours | A 66-hour buffer for restarting the worker |
| CRL generation interval | 6 hours, and immediately after every revocation | Limits the time until relying parties learn of a revocation |
| Renewal of automatic classes | Once only a third of the lifetime remains; lifetime at least three days | Renewal well before expiry, even during a short disruption |
| Root lifetime | 15 years, hybrid roots only | Only with ML-DSA does it outlast the PQ roadmap (2030, 2035); a classical root could be valid only until the end of 2030 at most |
| Lifetime of an issuing CA | 5 years | Enough for a 4-year issuing phase with system certificates of up to 1 year; limits the impact of a compromise |
| Lifetime of manual system certificates | At most 1 year | Annual renewal keeps inventory and profiles current |
| Lifetime of automatic system certificates and agent certificates | 30 days | A compromised key stays usable only briefly, with automatic renewal |
| Order deadline | 7 days | Enough for approvals over a weekend; an abandoned order expires |
| Backdating limit | 5 minutes | Covers clock skew, prevents deliberate backdating |
| Maximum check attempts per order | 5 | Separates transient errors from poison orders |
Alert threshold for orders in state claimed | 15 minutes | An issuance takes seconds; anything longer means a disruption |
| Size limit of an order | 64 KiB | A hybrid order with ML-DSA-87 and CSR stays well below it |
| Root quorum | 3 of 5 custodians | Survives the loss of two people and prevents solo actions |
| Quorum for administering issuing CAs' HSMs | 2 of 3 | Four-eyes principle with one deputy |
| Lead time of a CA's changeover alert | 6 months before the end of the issuing phase | Time for the ceremony and for adding the new anchor to relationships |
| Interval for reconciling HSM against ledger | Hourly | Limits the time until a signature made around the worker is detected |
nextUpdate of the root's CRL | 12 months | With scheduled ceremonies every 6 months, an overlap of 6 months remains |
| Interval of scheduled ceremonies | Every six months | New root CRL before the old one expires |
| Interval of external anchors | Hourly | Limits the unanchored period |
| Standby switchover time | At most 4 hours | Far below the 66-hour buffer |
| Session duration at the management interface | 8 hours, sign-out after 30 minutes of inactivity | One working day; limits shared use of an open session |
| Renewal interval of the additional secret | 10 minutes | Every new connection gets fresh material, at no measurable load |
| Maximum lifetime of a relationship connection | 60 minutes | Limits traffic under one secret; rotation without interruption |
| Lifetime of a trust bundle | 24 hours, regenerated hourly and immediately after every change | An adapter without a connection to distribution works with an old state for one day at most |
| Deadline for existing connections after a relationship revocation | 15 minutes | Running operations can finish; new ones cannot start |
| Retention of the operations log | 90 days | Troubleshooting; contains IP addresses (data minimization) |
| Retention of the security audit log and connection log | 7 years | Evidence beyond the lifetime of a CA |
| Retention of the ledger | Permanent | The single source of truth about issuance, revocation, and relationships |
| Number of safes | 2, at separate locations | Losing one site does not endanger the root |
What this page does not define
- The operations organization of any particular organization. Each organization names its own roles, custodians, and standby; in the Managed model, the operating partner takes on the agreed roles, never approval or custody.
- An operating partner's response times. They are governed by the contract with that partner.
- Values outside the limits of the rule set; an organization can tighten defaults and adjust them within the limits, but cannot remove them.