Resilience and chaos testing
Checks how your Kubernetes or VM workloads behave and recover when something fails: a pod evicted, a service stopped or restarted, CPU or memory under pressure, disk or network trouble, a dependency blocked. Included in the Complete plan.
Safety model
- Locked by default. Real faults are off for every organization. Request access from the Resilience page; DVSentry (the platform owner) grants it from the Creator Console for named targets and faults, after your organization accepts the current resilience testing authorization (version and SHA-256 recorded). Until then the page shows "Requested — waiting for DVSentry".
- Your agent does the work. You install the DVSentry agent in your cluster or on your VM. DVSentry sends it Ed25519-signed instructions; the agent checks them against its own trust file and acts only on targets and faults pinned there. DVSentry never holds a cluster or VM credential and never imports a cluster client.
- Two people. A run starts as
pending_approvaland needs a different Admin or Owner to approve it. A single-operator exception exists only for one-member organizations, granted for at most 7 days, for pod eviction or VM CPU/memory stress, with a fresh password and MFA check and a 10-minute cooling period. - Checked again and again. The lock is rechecked at creation, approval, precheck, right before dispatch and on every observation poll. One active run per organization.
- Stop at any time. Every run has a hard time limit, a watchdog and an emergency stop; revoking access cancels a run at once.
- Use staging or test environments. Do not point destructive faults at production.
First run
- Connect an environment: Resilience → Connect an environment → Install with one command (the same installer as Infrastructure), or the manual steps. Choose Check again until the agent shows connected with an inventory.
- Add a target (Owner or Admin) from the agent's inventory and request access for it.
- When access is granted, set up a run: the fault, its intensity and hold time, the health URL pinned in the agent's trust file, and the hard time limit.
- Submit for approval. Once approved, the run shows each phase live: fault active, watching your service, recovery.
- Read the signed report: timeline, health checks, exactly what was changed and whether the undo was verified. A pod eviction cannot be undone, so its report describes the replacement and recovery instead of claiming an undo.
Faults
| Fault | Where | Needs |
|---|---|---|
| Pod eviction | Kubernetes | Agent with the eviction role |
| Service stop / restart | VM | Agent; a process-kill target with Restart=no also needs a platform-granted agent restart |
| CPU / memory stress | VM | Agent (stress-ng --timeout, unprivileged, with an independent local supervisor) |
| CPU, memory, disk, network, dependency faults inside a pod | Kubernetes | The customer-installed sidecar, a pod-UID-bound signed spec, matching trust pins and the optional pod-patch RBAC overlay; network faults also need the separate NET_ADMIN manifest |
| Process kill / pause inside a pod | Kubernetes | Not available |
Catalog entries without a reviewed executor are descriptions only and stay unsupported.
GameDays
A GameDay is a plan of 1-8 steps, approved once by a different Admin or Owner over the plan's digest (valid 12 hours). Each step still starts only after the previous one passed and after a member confirms the target. Profiles may name up to five steady-state hypotheses; the agent runs the Prometheus query, operator and threshold pinned in its own trust file and returns one number per hypothesis. "Unknown" never counts as holding.
Reports and release gates
Every finished, refused or cancelled run has a frozen signed report (JSON with an Ed25519 envelope, plus a PDF). A policy's posture rule resilience (no_failure or proven) or resilience_gameday_days can gate releases on resilience results (Policies and gates).
Something missing or unclear? Tell us with Feedback in the app.