Failure-mode roadmap

Failure-mode roadmap

Validated against main on 2026-08-22. Open items name the remaining code, tests, and acceptance conditions. Completed work is recorded at the end.

Do not implement a check until it fails on a reproduction or positive-control fixture. An audit must enumerate its targets and assert that it found them.

1. Add an authenticated production /mcp endpoint

Status: OPEN. Issue #5126.

packages/websocket/src/server.ts mounts /mcp only outside production. The development mount always uses user "1", so it cannot be enabled unchanged. Python is not part of this item; its production opt-in already exists.

Implementation:

  1. Add NODETOOL_ENABLE_MCP=1. Keep the production route absent by default.
  2. When enforceAuth is true, use the existing global bearer-token hook and build agentToolsScope from req.userId with source "http-session".
  3. When enforceAuth is false, require a new NODETOOL_MCP_TOKEN of at least 32 characters. Validate it inside the global onRequest hook, before the generic remote-client denial. Compare it in constant time. Do not add a general /mcp public-route exemption. Add "http-token" to the MCP scope source union and use user "1" only after the token passes.
  4. Return 401 for missing or invalid credentials. Keep the global rate limit, CORS policy, and JSON-RPC input validation on the route.
  5. Add both variables to the config setting catalog and its Zod validation.
  6. Add production route tests under packages/websocket/tests/ for both auth modes, a non-loopback client, invalid credentials, the default 404 result, and authenticated GET/DELETE requests after session initialization.
  7. Add an opt-in probe to scripts/docker-smoke.mjs. Send a complete MCP initialize request, including jsonrpc, id, protocol version, capabilities, and client information.
  8. Document the two MCP flags in docs/configuration.md and AGENTS.md.

Acceptance:

  • Authenticated production derives the MCP user from the validated session.
  • Local production requires the dedicated token.
  • Production without the enable flag returns 404.
  • npm run test --workspace=packages/websocket passes.

2. Include missing secrets in validate_workflow

Status: OPEN.

validate_workflow calls validateGraph in packages/agents/src/capabilities/workflows.ts. Changing modelSelectionError does not change this capability.

Implementation:

  1. Add optional availableSecrets(keys) to CreateCapabilityRunOptions and propagate it through createCapabilityRun to CapabilityRun.
  2. Pass it to validateGraph from validate_workflow.
  3. Enumerate every createCapabilityRun call site in an audit test. Server, CLI, and MCP hosts with a real secret resolver must inject a callback that uses their ProcessingContext. Hermetic eval hosts must omit it explicitly.
  4. A missing_secret issue must name the key and direct the agent to Settings > Credentials. Mention request_secret only when that capability is present in the run’s registered capability set.

Tests must show that a missing key produces an issue, an available key clears it, and an absent callback preserves current output. Run npm run test --workspace=packages/agents -- workflows.

3. Make media resolution the browser rendering boundary

Status: OPEN. Follow-up to #4873, #4929, #5028, #5078, #5122, and #5123.

The reported leaks are fixed. Canonical resolution exists in web/src/utils/resolveMediaUri.ts and web/src/hooks/useResolvedMediaUri.ts. asset:// remains a valid stored locator, so a zero-hit grep is invalid.

Implementation:

  1. Add a ResolvedMediaUrl branded string to resolveMediaUri.ts. Brand only a non-empty resolved URL; keep missing results null or undefined.
  2. Extend ResponsiveImage and VideoPlayer instead of creating parallel image/video primitives. Add one audio primitive. They accept a MediaLocator and resolve it before setting src.
  3. Migrate chat media, storyboard cards, sketch layers, script shot chips, node previews, and App Builder media widgets.
  4. Check in a consumer inventory for the named surfaces and assert that each uses a locator-aware primitive. Add a design-lint rule that rejects an asset:// string literal in a JSX src. Include a positive-control fixture.

Acceptance: upload -> graph input -> render works for image, video, and audio; the primitives cover stored locators and HTTPS URLs; the lint fixture proves the rule can fail.

4. Preserve provider behavior through decorators

Status: OPEN. Issue #5109.

CassetteProvider inherits generateLoop and can drop an inner override. A method-existence test cannot detect this.

  1. For each decorator method, either forward it or reject construction when the behavior cannot be preserved.
  2. Cover the actual BaseProvider contract: generateMessage, generateMessages, traced wrappers, generateLoop, model discovery, media generation, tool support, error classification, lifecycle, and cost.
  3. Replace the diagnostic expectation in packages/runtime/tests/providers/provider-decorator-inertness.test.ts with behavioral equivalence checks using unique sentinel results.
  4. Start with CassetteProvider and add each later wrapper to an explicit inventory. In record mode, compare forwarded behavior with the inner provider. In replay mode, compare the recorded normalized contract and cost; do not require calls to the inner provider.

The current implementation must fail the new test before the fix. Then run npm run test --workspace=packages/runtime -- provider-decorator-inertness.

5. Inventory and consolidate SSRF screening

Status: DONE, shipped 2026-08-22. See SSRF screening inventory and consolidation under Completed work. The numbering stays so older references still resolve.

6. Add property tests to seven pure helpers

Status: OPEN. Use one PR per row. Before adding fast-check, record why the current dependencies and table-driven tests are insufficient, check its latest maintenance date, and inspect its transitive tree with npm ls fast-check. Add it only to the owning workspace.

Issue Helper Workspace Property
#4909 normalizeGraph and graph validation execution, node-sdk Malformed arrays never execute; normalization is idempotent.
#4910 deriveImageSizePreset web Rotated dimensions preserve the preset; positive dimensions return a valid preset.
#5035 clampTimeoutSeconds agents Positive finite inputs stay in bounds; sub-second values never become zero.
#5091 matchesFileWatchPattern automation-nodes Line terminators and glob metacharacters do not alter escaped filename semantics.
#4939 hasYieldStatement node-sdk Comment markers in strings do not change executable yield detection.
#4987 findMissingModelNodes web A missing provider is reported; a valid provider/model pair is not.
#5116 mergeIntoSequence web Reassembly preserves track indexes and order, including duplicate indexes.

Each PR pins the original counterexample, defines generator bounds and a replay seed, runs in the normal workspace suite, and proves failure by restoring the original defect once. Before generating values, the test must define its independent oracle: the complete valid preset set for #4910, parser-derived executable yield locations for #4939, documented glob semantics for #5091, and stable source order as the tie-breaker for duplicate indexes in #5116. Split #4909 into one execution normalization property and one node-sdk validation property.

7. Complete editor input-path coverage

Status: OPEN. Use three separate PRs.

7A. Shortcut action mapping

web/src/config/shortcuts.ts already owns shortcuts and rejects duplicate slugs. Add a typed action ID used by menus and handlers. Test missing actions, duplicate normalized combinations within the same active editor context and OS, Electron-only actions in web menus, and the command-menu shortcut on Windows and macOS. Duplicate combinations in disjoint contexts remain valid.

7B. Canvas drop journeys

Add web Playwright journeys for file drop, node creation from a dropped connection at the cursor, and run-selected with multiple nodes. Use npm --prefix web run test:e2e. For the Windows Explorer defect, first add an Electron Playwright dependency, config, script, packaged-app fixture, and Windows CI job. Update AGENTS.md and Electron testing docs with the command. Run the case on Windows; a Windows-style string on macOS is not sufficient.

7C. Dialog containment audit

PositionedDialog already clamps to the viewport. Enumerate other dialog primitives and direct users in a checked-in audit with a non-zero count. A dialog fails the audit when its rendered bounds extend outside a 600 x 600 px viewport or its content cannot scroll into view. Migrate failing callers and add one 600 x 600 px test per migrated primitive.

Completed work

  • An eval case or a suite for each agent capability: shipped 2026-08-22. packages/cli/src/harness/capability-table.ts names all 201 exported capabilities — 193 covered, 8 carrying a written gap note — with the implementation file, the suites the capability-suites selfcheck runs, and the eval cases whose expect.requiredTools demand them. scripts/sync-capability-coverage.mjs (npm run capabilities:sync / :check) derives everything but the gap notes from the live registry, the agent suites, and the eval case files, so a new capability with no check fails the check rather than review. Each entry carries a fingerprint of what the capability declares, and nodetool harness gate --base <ref> refuses a contract change whose coverage mapping stood still while saying nothing about a refactor. Fixtures in packages/cli/tests/harness-registry.test.ts, the table’s own audit in capability-coverage.test.ts, the rule in packages/agents/AGENTS.md, and .github/pull_request_template.md.

  • Live provider contract probes: shipped 2026-08-22. The manifest packages/runtime/src/providers/contract/probe-manifest.ts names, per entry, the provider, the endpoint, the production decoder, a checked-in raw HTTP response fixture, the required fields whose removal must break the check, and — for one entry per provider — the single live request, capped at one request and USD 0.05 per provider. The decoders were extracted out of the provider methods (decodeChatCompletion, decodeOpenAIModelList, decodeGeminiGenerateContent, decodeGeminiModelsPage, decodeFalLanguageCatalog, the fal extract*Url family, kieEnvelopeError, decodeKieTaskSubmission, decodeKieRecordInfo, decodeKieResultUrls), so fixtures and live responses parse through the code a run uses. runProbes reports network failures apart from schema failures, and retains only the response shape (summarizeShape) plus redacted messages, so no credential, prompt, request id, or signed URL reaches an artifact. Nightly: .github/workflows/provider-contract-probe.yml; offline half: packages/runtime/tests/providers/provider-contract-probes.test.ts, wired into harness gate through the new provider-clients surface. Docs: docs/provider-contract-probes.md.
  • Generated provider metadata drift: shipped 2026-08-22. Both generators have a fixture mode — packages/{fal,kie}-codegen/src/fixture-generate.ts — that reads only the schema fixtures under fixtures/, named by fixtures/generator-manifest.json, with no network, no pricing, and no timestamps. scripts/provider-codegen-check.mjs generates into a temporary directory and diffs every declared node-source and static-metadata output against fixtures/expected/, failing on a difference, an absent fixture, or a run that compared nothing. Root commands generate:fal:check and generate:kie:check and the workflow .github/workflows/provider-codegen.yml run it; fixture-generate.test.ts in each package covers the same comparison, byte stability, and the absent-fixture failures.
  • Missing secrets in validate_workflow: shipped 2026-08-22. CapabilityRun.availableSecrets carries the host’s answer, contextSecretAvailability builds it from a context that can reach a store, and getAllMcpTools takes the factory as secretAvailability. The missing_secret issue names the key and Settings → Credentials, and adds request_secret only where the run can raise the dialog. Covered by packages/agents/tests/mcp-tools.test.ts and the call-site audit capability-run-secrets-audit.test.ts.
  • Workflow credential preflight: shipped 2026-08-22 in node-sdk, execution, and CLI. The two execution credential suites cover it.
  • ExecutionSession preflight: shipped 2026-08-22. The model and credential checks moved to packages/execution/src/preflight.ts (no models import), ExecutionSession.create selects its context and refuses through ExecutionPreflightError before the Python bridge and persistence.onAccepted, and runWorkflow refuses through the same contract. session-preflight.test.ts covers both directions; execution-session-hydration-audit.test.ts audits every host.
  • Production Python opt-in: NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 is implemented in python-stdio-bridge.ts and documented in docs/configuration.md.
  • tRPC POST batches: issue #3979 is closed. Both tRPC clients set methodOverride: "POST".
  • Vite preload recovery: issue #4203 is closed. Dedicated tests cover stale deploys, non-stale failures, and the guard window.
  • Stale Prompt value: issue #3786 is closed. PromptComposerBody commits edits synchronously; its commit and run-from-here suites cover the defect.
  • Media resolution as the browser rendering boundary: shipped 2026-08-22. ResolvedMediaUrl in web/src/utils/resolveMediaUri.ts is the brand; ResponsiveImage, VideoPlayer, and the new AudioPlayback take a locator and resolve it. The rendering surfaces are inventoried in web/src/__tests__/mediaResolutionBoundary.test.ts, and design-tokens/no-unresolved-media-src (fixture web/scripts/test-media-src-rule.mjs) rejects a locator literal in a JSX url attribute.
  • SSRF screening inventory and consolidation: shipped 2026-08-22. isBlockedIpLiteral in packages/runtime/src/providers/safe-url.ts is the one address table; the sandbox’s network-guard.ts imports it and the websocket runner’s private copy (isSafeExternalUrl) is gone. fetchExternalMedia (packages/runtime/src/external-media-fetch.ts) is the media-ref egress policy — safeFetch, with NODETOOL_ALLOW_PRIVATE_MEDIA_FETCH=1 as the self-hosted LAN opt-out — and the fifteen media and result-download sites that used a bare fetch now call it or safeFetch. The inventory is packages/runtime/tests/url-egress-inventory.ts (86 entries: guarded, fixed-host, private-integration, browser, sandbox-bridge, screening); the reader-facing doc, with the DNS-rebinding decision and the contribution checklist, is docs/url-egress-inventory.md; and url-egress-audit.test.ts discovers URL surfaces from source in both directions, so a new unclassified fetch(url) fails.
  • Cross-origin media CSP: issue #5125 is fixed and covered by web/src/__tests__/contentSecurityPolicy.test.ts.
Edit this page on GitHub