Validator infrastructure that does not page you at 3 AM

Validator infrastructure is the only kind of production system where a single configuration error can destroy the asset it secures. The cost of getting it wrong is denominated in the same units as the asset itself. After running validators for foundations, exchanges, and DAOs across EVM and Cosmos chains, we have a small number of patterns that we trust. Here they are.

Treat keys like production secrets, because they are

Validator private keys should never exist in plaintext on disk. We use Hardware Security Modules (YubiHSM2, Ledger Flex, or cloud HSMs like AWS KMS) for signing. The mnemonic, if one exists, lives in a sealed envelope in a safe with documented dual-custody access. We have never regretted a single dollar spent on key protection, and we have watched teams lose millions by skipping it.

Double-sign protection is non-negotiable

A double-sign slash on most proof-of-stake networks is 100% of the validator's stake. The minimum protection is to ensure two instances of the validator client never run with the same key at the same time. We enforce this at the infrastructure layer: only one signing key reference exists per data center, with remote attestation and process-level locking. We test this monthly by deliberately trying to start a second instance and watching it refuse.

Sentry nodes versus sentry architecture

A sentry node is a hop between the public internet and your validator. A sentry architecture is the full design: a network of public-facing nodes that shield one private validator. The latter is what we build. The former, deployed without rate limiting and DDoS protection, is how validators get slashed for missed attestations during an attack.

MEV is a feature and a threat

MEV-aware block building can increase validator rewards materially. It also adds a non-trivial attack surface. We run mev-boost with at least three relays, a circuit breaker on anomalous relay responses, and a path to fall back to local block building in seconds. The circuit breaker has saved us from at least two relays that were censoring or returning malformed bundles.

RPC redundancy you can measure

Validator health depends on RPC availability. We run at least three independent RPC endpoints: our own sentry node, a paid provider, and a public fallback. The client is configured to round-robin with a 1-second timeout. RPC outages should never cause validator downtime if you spent two days on this layer.

Upgrades are the riskiest hour

Every team has been bitten by an upgrade that silently changed a default. We do all upgrades in a staging network first; we triple-check consensus-breaking version transitions; we always have a downgrade path. We have watched production validators lose hours of attestations because someone shipped a config change without restarting the right service.

Runbooks that survive the on-call being on vacation

The on-call engineer at 3 AM is not the engineer who built the system. They are tired, half-asleep, and reading your runbook on a phone. We write runbooks with explicit commands, expected outputs, and exactly-one branch for the common failure modes. If your runbook has more than three pages, it is too long.