--- title: "Reliability Architecture: One Execution Core, Golden Journeys, Injected Faults" description: "Local-first visual environment for building and running AI workflows. Build agents visually, deploy anywhere, privacy by design." canonical: https://docs.nodetool.ai/RELIABILITY_ARCHITECTURE markdown: https://docs.nodetool.ai/RELIABILITY_ARCHITECTURE.md product: NodeTool source: https://github.com/nodetool-ai/nodetool/blob/main/docs/RELIABILITY_ARCHITECTURE.md --- # Reliability Architecture: One Execution Core, Golden Journeys, Injected Faults # Reliability Architecture: One Execution Core, Golden Journeys, Injected Faults > Status: proposal. Author: Principal Reliability Engineering review, 2026-07. > Scope: the architectural investment that buys the most stability over the > next several years, designed in enough depth to start building. > Execution plan: [RELIABILITY_TASKS.md](RELIABILITY_TASKS.md) — the same > work cut into six parallel tracks of one-PR tasks. NodeTool's recent failures — CLI runs behaving differently from WebSocket runs, packaged Electron builds dying with green CI, half-open sockets, fake workers accepting frames real workers reject, orphaned actors, provider drift — are not coverage gaps. They are **drift between execution surfaces and unvalidated boundaries**, and no amount of additional unit testing fixes either. This document names the root causes, picks one primary investment, and designs it. The one-paragraph verdict up front: **the highest-leverage investment is a Reliability Harness that runs a small suite of golden workflow journeys through every real execution surface, differentially compares the message streams, asserts lifecycle invariants the kernel currently only logs, and injects faults at the five seams where the real world differs from tests. Its prerequisite — and the first PR — is collapsing the eight hand-wired `WorkflowRunner` construction sites into one `ExecutionSession` facade and making the streaming protocol runtime-validated.** Everything else in this document is the design of those two things. --- ## 1. Why the unit tests didn't catch these bugs The unit tests are good. `packages/kernel/tests/` has ~50 files including `controlled-cancel-hang.test.ts`, `execution-reliability.test.ts`, and mutation-kill suites. They didn't catch the recent failures because those failures live in places unit tests structurally cannot reach: **The bugs are in the wiring, not the units.** All surfaces call the same `WorkflowRunner` (`packages/kernel/src/runner.ts`), but each surface wires it by hand. The CLI (`packages/cli/src/nodetool.ts:751`), the debug harness (`packages/cli/src/debug/server-runner.ts:101`), the WebSocket runner (`packages/websocket/src/unified-websocket-runner.ts:2788`), the headless job runner, the HTTP API, the MCP server, and the browser workflow-runner each independently repeat graph hydration, executor resolution, param seeding, and result handling. A unit test exercises the kernel; it cannot notice that the CLI forgot `hydrateGraphNodeFlags` (the in-code comment says exactly what happens: "or streaming nodes run as one-shots") while the WS surface remembered it. "CLI execution differed from websocket execution" is this, mechanically. **The bugs are at unverified boundaries.** The entire streaming protocol — ~30 message types in `packages/protocol/src/messages.ts` — is TypeScript interfaces only. Zero runtime validators, zero type guards. The server's `receiveMessage` blind-casts `unpack(...) as Record` and narrows with hand-rolled `typeof` checks. TS types are erased at the msgpack boundary, at the Python stdio boundary, at the mobile client, and at any non-JS consumer. A fake worker that "accepts protocol shapes real workers reject" is only possible because nothing executable defines the shape. Unit tests on either side of an unspecified boundary both pass while disagreeing with each other. **The bugs are in environments tests don't run in.** The sharp/@img skew that killed `v0.7.1-nightly.20260728.713` was invisible to every test because tests run against `node_modules`, not the flat `_modules/` staging that `scripts/bundle-backend.mjs` produces. The release smoke-launch missed it too, because it watched the Electron main process, which survives its backend child dying. Packaged-artifact bugs require running the packaged artifact. **The bugs are emergent timing, and the kernel deliberately tolerates them.** `_checkPendingInboxWork` (`runner.ts:1319`) detects data loss after all actors settle — and logs a warning. Backpressure release (`inbox.ts:571`) exists because a parked producer once hung `Promise.allSettled` forever. Three separate control-response hang guards exist. These are known failure classes with advisory-only detection: unit tests assert the happy path around them, and production hits the warning path nobody asserts on. **The test doubles are flattering.** `FakeProvider`/`ScriptedProvider` return well-formed shapes by construction. The Python WS test helper speaks msgpack-over-WS with no length prefix, while the real stdio bridge is length-prefixed — and there is no stdio fake at all, so the framing, spawn-timeout, and EPIPE paths of `python-stdio-bridge.ts` are untested. Provider tests are 87 hand-written per-provider files; nothing enumerates the registry and asserts one contract, so a new provider ships with a different error shape and no test fails. The pattern: unit tests verify components against their author's mental model. The recent bugs are disagreements *between* mental models. Only tests that run the real components against each other can see those. ## 2. The highest-leverage investment One investment, two layers, in dependency order: **Layer 1 (prerequisite): a single `ExecutionSession` facade plus a runtime-validated protocol.** Collapse the eight construction sites into one package that owns hydration, executor resolution, param/streaming seeding, cancellation, timeout, persistence hooks, and message emission. Generate Zod schemas for every `ProcessingMessage` and validate at every process boundary. This removes the drift *mechanism*; without it, the harness would forever chase symptoms. **Layer 2 (the harness): golden workflow journeys executed through the real production stack on every surface, with differential comparison and fault injection.** Not a new test framework — a promotion and unification of infrastructure the repo already has: the hermetic `e2e-server.ts` + `fake-runtime.ts`, the `nodetool debug` harness whose `diff.ts` already compares kernel-vs-browser runs, the e2e_runner Playwright suite, and `backend:smoke`. Today these are separate, partially nightly, and none injects faults. The harness makes them one system with teeth. Why this beats the alternatives considered: - *More integration tests per package* — still tests components against local assumptions; doesn't touch cross-surface drift. - *A full staging environment with synthetic traffic* — right instinct, wrong cost for a local-first product; most NodeTool instances are a laptop, not a fleet. The harness delivers staging-grade confidence hermetically, in CI, per PR. - *Formal protocol specification (protobuf/JSON-schema-first)* — the Zod layer gets 90% of the value without a wire-format migration; revisit IDL-first only if a non-TS client beyond the Python worker appears. ## 3. Should there be golden journeys through the production stack? Yes. Unambiguously. The repo has already proven the approach piecemeal: the `bundle` CI leg (`backend:smoke`) was added after a packaging incident and has been catching what layout checks can't; the e2e-runner suite exercises the real server, real kernel, real ReactFlow canvas, real WS protocol hermetically. The gap is that these are not organized as *journeys* — named, versioned, cross-surface, invariant-asserting scenarios — and the journey-shaped suite that exists (`user-journeys.yml`) runs nightly with `continue-on-error: true`. A golden journey is defined as: **a fixed workflow + a scripted interaction + an expected message-stream shape + a set of invariants, executed through a real entry point with no mocked internal seams.** Only the world's edges are faked: provider HTTP (cassettes/scripted), external media, the clock where determinism demands it. Everything from the entry point inward — protocol parsing, job queue, kernel, actors, inboxes, persistence, streaming — is production code. ## 4. Journey structure Each journey is a directory under `reliability/journeys//`: ``` reliability/journeys/streaming-llm-chat/ journey.json # manifest: workflow ref, params, interactions, # surfaces, fault matrix, timeout budget workflow.json # the graph (or a ref to a shipped example) interactions.json # scripted actions: run, stream_input, cancel-at-, # reconnect-at- — same shape as `app debug --interact` expected/ stream.shape.json # normalized expected message sequence (see §12) outputs.json # terminal outputs after normalization ``` Rules that keep journeys trustworthy: - **Declarative, not imperative.** A journey is data interpreted by the harness, so the same journey runs on every surface without per-surface test code. Per-surface *drivers* exist once, in the harness. - **Event-anchored, not time-anchored.** "Cancel after the second `chunk`", never "cancel after 500 ms". Wall-clock anchors are the root of flake, and per `docs/DEVELOPMENT_STANDARDS.md`, flaky tests are bugs. - **Normalized comparison.** Timestamps, ids, durations, and asset URLs are normalized before diffing; ordering is compared per-node-channel, not globally, because the actor model legitimately interleaves. - **Every journey states its invariant set explicitly** (§6) and the harness refuses a journey that asserts nothing beyond "completed". ## 5. Which workflows belong in the suite Fifteen to twenty journeys, chosen for failure-mode coverage rather than feature coverage. Seed set, mapped to the recent bug classes: 1. **Linear text pipeline** (input → transform → output) — the baseline; if this diverges across surfaces, everything is suspect. 2. **Streaming LLM chat** with chunked output and `stream_input` / `end_input_stream` incremental seeding — exercises the WS-only seeding path that no other surface has (the kind of asymmetry §7 exists to kill). 3. **Fan-out/fan-in DAG** with a slow branch — completion detection, EOS dedup, `Promise.allSettled` semantics. 4. **Streaming producer → slow consumer** with a finite `bufferLimit` — backpressure park/release (`inbox.ts:571`), the "producer never returns" hang class. 5. **Controlled/agent node with tool calls** — the control-response hang guards (`runner.ts:1259`, `:1778`). 6. **Mid-run cancellation** at three anchors: while queued, during a node, during streaming — including the tRPC DB-cancel/start race the code comments at `unified-websocket-runner.ts:2834` describe. 7. **Python-node workflow** over the real stdio bridge against the real worker (nightly ring) and a *faithful* length-prefixed fake (PR ring) — closing the "no stdio fake exists" gap. 8. **Provider failure mid-stream** — provider returns 429/500/truncated stream after N chunks; asserts terminal `job_update failed` with a sane error, no orphan actors. 9. **Client reconnect mid-run** — kill the socket after node 2 of 4; assert the client's queue-during-backoff behavior (`WebSocketManager.ts:249`) and that the run's terminal state is observable after resubscribe. 10. **Concurrent jobs beyond `MAX_CONCURRENT_JOBS`** — queue, drain, slot handoff (`startingJobs`→`activeJobs`), the leaked-slot class the code comments memorialize. 11. **Mini-app journey** — an `ApplicationBundle` through the shared `app-runtime` fold on web, mobile (jest), and headless `app debug`; asserts the fold's job-id isolation. 12. **Suspend/resume trigger workflow** — `WorkflowSuspendedError` path, durable inbox, trigger wakeup. 13. **Error-in-one-branch** — sibling branches complete, failed node reported, `finalize()` ran everywhere. 14. **Malformed-protocol journey** — client sends near-valid frames (wrong field type, unknown `type`, oversized, truncated msgpack); asserts rejection without connection or job corruption. 15. **Packaged-backend journey** — journeys 1, 2, and 7 executed against the staged bundle's `server.mjs` (extending `backend:smoke` from "boots and serves /health" to "runs real workflows"). Each shipped example workflow that `example-smoke-debug.mjs` curates is a candidate for promotion; journeys should prefer shipped examples so the suite doubles as product validation. ## 6. Invariants to assert Journeys assert invariants *in addition to* expected outputs. These are the checks that convert today's advisory logging into failures: **Lifecycle.** - Every `node_update running` is eventually paired with exactly one terminal `node_update` (`completed`/`failed`/`cancelled`) for that invocation. No node is left `running` after `job_update` terminal. - Terminal precedence is exactly cancel > suspend > failed > completed (`runner.ts:625`) — asserted, not assumed. - Exactly one terminal `job_update` per run, and nothing follows it on that job's stream. **Cleanup and leaks.** - `_checkPendingInboxWork` finding pending work is a journey **failure**, not a log line. The harness runs the kernel with a strict mode flag (§12) that surfaces it. - After the run: zero live actors, zero pending control responses, zero timers beyond the trigger timeout, Python bridge has zero pending requests, WS slot accounting returns to zero (`activeJobs`/`startingJobs` empty), no growth in open handles across repeated journey iterations (leak journeys run N=20 and compare). - Cancellation completes within a budget (e.g. 5 s) and `finalize()` ran for every started actor. **Determinism.** - Two runs of the same journey with the same cassettes produce identical normalized streams per node-channel. Nondeterminism outside declared normalization fields is a failure — this is what makes drift visible. **State transitions.** - Client `WebSocketManager` transitions only along its declared state machine (`WebSocketManager.ts:95`); the harness driver records transitions and diffs against the machine. - Job table state (when `persistence: "job"`) matches the final `job_update` — no "resurrected" cancelled jobs. **Protocol correctness.** - Every frame crossing every boundary validates against the generated Zod schema — both directions, all surfaces. One shared assertion, run everywhere, kills the fake-worker-accepts-what-real-workers-reject class permanently. - Bridge frames respect `BRIDGE_PROTOCOL_VERSION` negotiation. **Resource/flow control.** - Outbound WS buffer never exceeds the configured cap; sends beyond it wait for drain (asserted via the health layer's counters). - Inbox depth never exceeds `bufferLimit` when set. ## 7. One execution implementation across surfaces? Yes — and it's closer than it looks. The kernel is already shared; the drift is in the ~200 lines each surface hand-writes around it. The answer is not "merge the surfaces" — the WS runner legitimately needs streaming, queueing, and persistence the CLI doesn't — it is **one facade that owns everything below the transport**: ``` @nodetool-ai/execution (new package, between kernel and its consumers) ExecutionSession.create({ graph, // raw saved JSON — hydration happens HERE, once registry, bridgeFactory, params | inputStream, // both seeding modes for every surface persistence: JobPersistenceHook | null, limits: { runTimeoutMs, nodeTimeoutMs, bufferLimit }, }) session.messages // AsyncIterable — validated session.pushInput(...) / session.finishInputStream(...) session.cancel(reason) // the ONLY cancel; timeout = cancel("timeout") session.result // Promise ``` Surfaces become thin adapters: the WS runner maps commands to session calls and pumps `session.messages` through its transform pipeline; the CLI awaits `session.result`; the debug harness's `withTimeout` — which today *abandons* a still-running kernel — becomes `session.cancel`, fixing a real leak. Executor resolution (currently copy-pasted three times), `hydrateGraphNodeFlags` (currently remembered by some callers), and output-name rewriting (currently WS-only) move inside and cannot be forgotten. Mobile and web clients stay consumers of the WS protocol — they should never embed the engine — but their *message folding* is already unified in `@nodetool-ai/app-runtime/fold.ts`, whose header comment ("existed three times and they drifted") is this document's thesis proven in miniature. Extend that precedent: the graph-editor message reduction in web should fold through a shared reducer the same way. ## 8. Eliminating drift structurally A facade stops drift only if bypassing it is harder than using it: 1. **Dependency lint.** Extend the existing key-boundary/coupled-dep CI guards (`quality-checks.yml` `deps` leg): only `@nodetool-ai/execution` may construct `WorkflowRunner`. Direct construction anywhere else fails CI. 2. **Schema-first protocol.** `messages.ts` becomes generated output: Zod schemas are the source, TS types are `z.infer`, and a generated JSON Schema artifact is published for the Python worker to validate against in *its* tests. Adding a message type without a schema is impossible because the type no longer exists separately. 3. **Contract tests from the registry, not by hand.** One parameterized suite enumerates `provider-registry.ts` and asserts the `BaseProvider` contract (streaming shape, error taxonomy, cost fields, cancellation) against every registered provider via cassette; a new provider is covered on registration or fails CI. Same pattern for node executors and for bridge transports (stdio and WS fakes generated from one framing spec). 4. **The differential journey run is the drift alarm.** Surfaces sharing one facade can still diverge in their adapters; the harness's cross-surface diff (§12) is the standing detector for whatever the lints miss. 5. **Fakes derive from contracts.** `fake-runtime.ts` executors and the Python fakes must be validated by the same Zod schemas and framing code as production — a fake that emits an invalid frame fails its own build. Delete the msgpack-no-length-prefix WS test helper divergence. ## 9. Fault injection points Inject at the five real seams, controlled declaratively per journey (`faults` block in `journey.json`), implemented once in the harness: | Seam | Mechanism | Faults | |---|---|---| | Provider HTTP | fault-capable `CassetteProvider` | 429/500/timeouts, truncated stream after N chunks, malformed SSE, slow-drip tokens, mid-stream disconnect, cost-field omission | | WS transport | proxy in front of `e2e-server` | drop (no FIN — the half-open case), delay, reorder within limits, fragment frames, stall reads (slow consumer → server buffer growth → drain-timeout path), abrupt close during `run_job` ack | | Python bridge | fake worker + real-bridge harness | exit mid-request (pre- and post-startup), stderr flood, framing violation (bad length prefix), never-ready (spawn timeout), EPIPE on write, protocol-version mismatch | | Process/host | harness-level | SIGKILL the server mid-run then restart (journeys 9/12 assert recovery/observability), disk-full via injectable storage adapter (asset write fails mid-run), DB locked/failed during job persistence (today silently swallowed at `unified-websocket-runner.ts:2868` — a journey must pin what the *intended* behavior is) | | Client | driver-level | cancel at every event anchor, reconnect storms (connect/drop loops), duplicate commands, out-of-order `end_input_stream` | Priority order for building: provider faults and WS faults first (they map to the most recent incidents), bridge faults second, process/disk third. Fault journeys assert the same §6 invariants — that is the point. A dropped socket may fail the *client's* view of the run; it must never orphan an actor, leak a slot, or corrupt the job table. ## 10. Invariants that must never be violated (system-wide) The short list, worth printing: 1. A started run reaches exactly one terminal state, always, under every fault in §9. No run hangs forever; no run reports two terminals. 2. No orphaned execution: when the last consumer is gone (client, CLI, harness), the actors, bridge requests, and timers of that run are released within a bounded time. 3. Cancellation is total and prompt: cancel → all actors observe the signal, all inboxes close, `finalize()` runs, terminal `cancelled` emitted, within budget. 4. No silent data loss: pending inbox work after settle is an error, not a warning (behind the strict flag until burn-in, then default). 5. Every frame on every boundary is schema-valid; invalid frames are rejected at the boundary without corrupting the connection or the run. 6. Resource accounting returns to zero: job slots, sockets, child processes, file handles, pending promises. 7. Persisted job state and emitted terminal state never disagree. 8. The packaged artifact executes the same journeys with the same results as the dev tree. 9. One workflow, one params set, one cassette set → one normalized result, on every surface. ## 11. Release engineering Restructure validation into three rings; a release is a promotion through rings, not a tag plus hope: **Ring 0 — PR gate (minutes).** Today's quality gate, plus: protocol schema validation tests, the facade dependency lint, and the five fastest journeys (1, 3, 6, 13, 14) on the kernel surface with strict lifecycle mode. Hermetic, no network, no flake budget. **Ring 1 — merge-to-main (tens of minutes).** Full journey suite on all hermetic surfaces (kernel, WS server, e2e_runner browser, headless app, CLI), full fault matrix, cross-surface differential diff, leak iterations, plus the packaged-backend journey ring (bundle → stage → run journeys against `server.mjs`). Runs before the Docker image is considered deployable — i.e., it gates `fly-deploy.yml`, not just informs it. **Ring 2 — release candidate (hours, per-OS).** Everything `release.yaml` does today (packed-tree smoke boots on win/mac/linux, mcpb smoke, updater assets), extended to run journeys 1/2/7 against the *packed* backend on each OS — the ring that would have caught the sharp incident on the first nightly instead of in a shipped build. Plus the real-provider nightly: journeys 2/7/8 against live APIs with a spend cap (promoting `example-smoke-debug.yml` from dispatch-only to scheduled), and the real Python worker instead of the fake. Two policy changes with more effect than any new job: - **Journeys are required checks, not `continue-on-error`.** The `user-journeys.yml` comment already promises promotion "after a clean streak" — set the date. A reliability suite that can't fail the build is documentation. - **Every incident retires into a journey or a fault.** The `backend:smoke` header comment is the model: the regression test *is* the incident writeup. `docs/KERNEL_PARITY_GAPS.md` shows the discipline already exists for parity; apply it to reliability. ## 12. The Reliability Harness architecture ``` reliability/ journeys//... # §4 harness/ core/ journey.ts # manifest loader + Zod validation normalize.ts # stream normalization (ids, ts, urls) diff.ts # per-channel stream diff (grow from # packages/cli/src/debug/diff.ts) invariants/ # one module per §6 invariant, each a # pure function (RunRecord) -> Violation[] faults/ # provider, ws-proxy, bridge, storage, client record.ts # RunRecord: every frame, both directions, # timestamped, surface-tagged drivers/ # ONE per surface, each ~"start, script, kernel.ts # collect, teardown" against real entry pts ws-server.ts # (wraps e2e-server.ts + real msgpack client) cli.ts # (spawns the real CLI binary) browser.ts # (Playwright on e2e_runner — exists today) app-headless.ts # (wraps app-debug runtime) packaged.ts # (bundle-backend staging + server.mjs) electron.ts # (ring 2; drives packed app) compare.ts # cross-surface differential: run journey on # N drivers, normalize, diff, report cli.ts # `nodetool reliability run # [--surface ...] [--faults ...] [--diff]` ``` Design decisions: - **Drivers speak entry points, never internals.** The kernel driver is the only one allowed to touch `ExecutionSession` directly (it *is* the reference surface). Every other driver goes through the same door a user does: the WS protocol, the CLI process, the browser page, the packed binary. That's what makes a green run meaningful. - **The kernel surface is the oracle.** Differential comparison is every-surface-vs-kernel, not all-pairs — O(n) diffs, and disagreement assigns blame to a specific adapter layer. - **`RunRecord` is the universal currency.** Every driver produces the same record shape (superset of the debug harness's bundle and the e2e_runner's `record.json`); invariants and diffs are pure functions over it, so they run identically in CI, in `nodetool reliability`, and over a record captured from a user's debug bundle — turning field incident bundles into replayable inputs. - **Strict mode is a kernel flag, not a fork.** `runner.run({ strict })` turns the advisory checks (`_checkPendingInboxWork`, pending control responses, active-edge drain) into thrown violations. The harness always sets it; production gains it as a default once the suite is clean. - **Reuse, don't rebuild.** `e2e-server.ts`, `fake-runtime.ts`, `cassette-provider.ts`, `debug/diff.ts`, the app-debug runtime, and `smoke-backend-bundle.mjs` are the harness's organs. The new code is the journey format, the invariant modules, the fault layer, the ws proxy, and the drivers' shared record shape. ### Build order 1. `ExecutionSession` facade; migrate CLI + debug + headless (the simplest three); dependency lint. *(unblocks everything; fixes the debug-harness abandonment leak immediately)* 2. Zod-first `messages.ts` + boundary validation + fake conformance. *(kills the protocol-drift class)* 3. Harness core + kernel and ws-server drivers + journeys 1–6 + strict mode → Ring 0. 4. Fault layer (provider, WS proxy) + journeys 8–10, 14; browser/CLI/app drivers + differential compare → Ring 1; gate deploys. 5. Bridge faults + real stdio fake + journey 7; packaged driver + journey 15 → Ring 2; migrate the WS runner onto the facade (last, because it's the richest surface). 6. Promote `user-journeys.yml` to required; retire incident backlog into journeys. --- ## What this buys At millions of workflow executions, the failure modes that matter are exactly the ones listed at the top: drift, boundary disagreement, timing, environment. This design attacks each with structure — one facade so drift has nowhere to live, executable schemas so boundaries can't disagree silently, event-anchored fault journeys so timing bugs reproduce on demand, and packaged-artifact rings so the thing tested is the thing shipped. The unit tests stay what they are: fast proofs of local logic. The harness is the proof of the system.