# NodeTool Documentation — Full Text > Local-first visual environment for building and running AI workflows. Build agents visually, deploy anywhere, privacy by design. This file concatenates the NodeTool guide documentation for LLM ingestion. The full per-node reference (900+ nodes) is omitted for size — browse it at https://docs.nodetool.ai/nodes/ or search at https://docs.nodetool.ai/search.json.--- # index.md Source: https://docs.nodetool.ai/ NodeTool documentation Open-source agent-first creative workspace Create and edit images, video, audio, and text with agents that work alongside you. Let them build and revise workflows, then inspect and edit the results yourself. Your project keeps the brief, assets, and edits together. One sentence to a finished short: beat sheet, entities, storyboard, clips, and the cut. Get started Examples Cookbook Agents Describe it. The agent builds the project. Give the agent a brief. It drafts the script, boards the shots, renders the takes, and cuts them on a timeline. What comes back is a project you can open, not a finished file. Write the brief "Turn this story idea into a storyboard and a short trailer." Watch it work The plan, every tool call, and every render appear as they run. A permission mode decides what needs your approval. Take over, or send a note Change any part yourself, or ask the agent to redo the one shot you want different. Chat & Agents → The agent loop → Editors Seven editors. One project. Everything the agent makes opens in an editor. The agent works each editor with the same tools you click. Storyboard Board the film shot by shot. Generate cheap stills to lock the look, then animate only the shots you approved. Storyboard Editor → Timeline Arrange, trim, and layer generated video and audio across tracks. The agent edits the same document when you ask it to tighten the opening. Video Editor → Script & voice Draft the dialogue and cast a voice per character. Change the words and the take flags itself stale. Script Editor → Sketch Paint and blend hand-drawn elements with generated layers. Bind a layer to a prompt and regenerate that layer alone. Sketch Editor → 3D Block out a set with simple shapes and lights, by hand or by asking the agent. Render it from any angle as a shot reference. 3D Editor → Game Build a 2D or 3D game and play it in the editor. Ask the agent for a new level, then export a web player. Game Editor → Nodes Every editor above sits on a graph you can open. Wire typed ports, press Run, and read the output at every step. Workflow Editor → Examples Start from an example Every video below is a NodeTool timeline. Open one, swap in your own product, copy, and footage, and render your version. Ad recipes Sound-off vertical ad formats with a beat sheet, an asset list, and a rendered sample cut. Offer Kinetic offer wall One offer stays readable while the product range moves behind it. Range Product cutout shuffle Product cutouts swap on a fixed baseline to show every flavour. Premium Editorial image panels Three photos and short copy make a calm, sound-off brand ad. Software Integration puzzle App tiles snap into one grid to show a connected product. Timeline examples Finished films that ship with NodeTool. Open one in the timeline editor and change the scenes, text, and motion. Software launch film Serein An inbox becomes a product story through animated scenes, type, and interface details. App motion graphics Kite A savings app with kinetic type, a growing chart, and an animated goal ring. Animated event poster Tidewater A jazz festival poster in motion, with layered inks, cut-paper shapes, and a swing score. Vertical data story Cadence A bike-share year in review, with a route that rides itself, counting stats, and animated charts. Running-shoe campaign Prism A shoe launch built from product stills, colour trails, and orbiting type. Motorcycle launch ad Voltra An electric motorcycle campaign with parallax, animated gauges, and a cut timed to the beat. All ad recipes → · Video Editor → · Use-case walkthroughs → · Cookbook → What you can do Let the AI build it — Describe what you want, and it puts everything together, chooses the tools, runs them, and shows you the result. Edit what it built — Click any part, change its settings, and run it again. Nothing is hidden; you own everything it creates. Put AI inside your workflows — You can make an agent act as one step in a larger process, letting it plan and use tools. Mix models from anywhere — Use different AI models together, like an image maker with a text writer and a voice generator, all in one place. Choose the best tool for each specific job. Run advanced models on your computer — Run powerful models directly on your hardware. It works without the internet, and your files stay private. Use your own accounts — Pay AI companies like OpenAI or Google directly. We do not add any extra fees or charges. Share your workflow as a Mini-App — Turn your complex work into a simple app with just inputs and outputs. Share a link, and others can use it without installing anything. Chat with your documents — Search and talk to your own files securely. Your data stays safely on your computer. Studio or Cloud Same code, same workflows. Both AGPL-3.0. NodeTool Studio — desktop Mac, Windows, Linux. Local inference via Ollama, MLX, and GGUF. Works offline. Prompts and outputs stay on disk. BYOK for cloud providers when you want them. Download Studio → NodeTool Cloud — browser Hosted, no install. Same canvas, same nodes. BYOK for every cloud provider — OpenAI, Anthropic, Gemini, Replicate, FAL, ElevenLabs, HuggingFace. No local models. Open Cloud → No credit markup. Cloud hosts the same code in this repo. Self-host the Docker images any time. You pay providers directly. What you can build Image and video Flux, Qwen, Wan, Seedance, Sora, Veo, Kling on one canvas. Movie Posters → Story to video Prompt to storyboard to narration to animation to score. Movie Trailer Generator → Sound and voice Music, sound design, narration. ElevenLabs, MusicGen, Whisper in the graph. Image to Audio Story → Agents Agents that plan, call tools, and drive pipelines — in chat or as a node. Fetch Papers → More creative patterns — directed films, entity-consistent batches, script-driven cuts — in the Cookbook. Get started Download NodeTool for macOS, Windows, or Linux. Connect a language, image, and video model. Ask the agent for a storyboard, render it, and export the film. Find what you need Get startedInstall NodeTool, connect a model, and make a first short film with the agent. Core conceptsAgents, nodes, workflows, assets, and how they fit together. Workflow editorBuild, run, and debug node graphs on the canvas. Creative editorsStoryboards, timelines, sketches, 3D, games, and audio. Chat and agentsChat, skills, memory, and the MCP server. Models and providersLocal and cloud models, API keys, and costs. Node referenceEvery node, grouped by what it works with. CookbookWorked patterns you can copy into your own projects. Mini appsTurn a workflow into a small app other people can run. Self-hostingRun NodeTool on your own server with Docker. DevelopersCustom nodes, the TypeScript DSL, and the APIs. HelpError messages, fixes, FAQ, and keyboard shortcuts. Open source AGPL-3.0. Discord · GitHub.--- # 3D Editor Source: https://docs.nodetool.ai/3d-editor.html Build a rough 3D set from boxes, spheres, planes, cylinders, cones, tori, and lights. Move things by hand or ask the agent to do it, then render the scene as a reference image for a shot. Quick access: Click the + button in the workspace tab bar and choose New 3D model, or open a .glb or .gltf asset and switch its tab to edit mode. What a 3D model is A 3D model in NodeTool is a glTF asset. The editor reads and writes .glb and .gltf files only. A new model is a .glb, and Save always writes binary glTF back over the same asset. The asset id does not change, so anything that points at it keeps working. File Opens in the editor Notes .glb, .gltf Yes Editable and saveable .obj, .fbx, .stl, .ply, .usdz No The tab opens, but the viewer reports that the format is not supported. Supported formats are .glb and .gltf When a model is not editable, or the tab is in view mode, you get the viewer instead. The viewer orbits, zooms, and toggles the grid, axes, and wireframe. See Asset Management. Creating or opening a model There are four ways in. New model. Click + in the workspace tab bar and choose New 3D model. NodeTool creates an asset in the current project that holds a single box named “Box”, and opens it in edit mode. From an asset. Open a .glb or .gltf in the asset viewer and click Edit in 3D Editor. This button appears for assets in your library. From the examples. On the Examples page, open the 3D models tab. It lists low-poly game models in six packs: Adventure, Dungeon, Platformer, Seaside, Sci-fi, and Sci-fi outpost. Open model copies the model into the current project and opens the copy in edit mode, so the original stays untouched. From the agent. Ask the agent to create a model, as described under Asking the agent. Layout The editor has a toolbar across the top, three panels under it, and a status bar along the bottom. Scene on the left is the outliner. The viewport is in the middle. Inspector is on the right. Assistant is an optional chat panel on the far right. You can resize it. The status bar names the selection, tells you what the mouse does in the current mode, and counts the objects, lights, vertices, and triangles in the scene. The viewport starts with the camera framing the whole model, shows a ground grid with a red X axis and a blue Z axis, and lights the scene with a studio environment. That lighting is for editing only. It is not saved into the model and does not count as a light in the scene. It is built in, so it also works offline. Toolbar Control What it does Undo, redo Steps back and forward through your edits. The tooltip names the step Move, rotate, scale Sets the gizmo mode Transform space Switches the gizmo between world axes and the object’s own axes Snapping Turns snapping on or off: 0.25 units, 15 degrees, and 0.1 scale. Holding Ctrl while you drag inverts the setting Add Adds a shape, light, or empty group. See Adding and deleting objects Duplicate, delete Act on the selected object and its children Keyboard shortcuts Lists every shortcut Assistant Shows or hides the chat panel. NodeTool remembers whether you left it open Export Downloads the scene or the selected object as a file. See Exporting Save Writes the scene back to the asset. A dot next to the model name marks unsaved changes Close Returns the tab to view mode. With unsaved changes, it asks before discarding them Viewport controls The bar in the top-left corner of the viewport holds View (front, back, right, left, top, and bottom views), frame all, focus on the selection, and three display toggles: the grid, a wireframe overlay, and light icons. The display toggles are remembered between sessions. The axis widget in the bottom-right corner turns the camera to an axis when you click it. Drag to orbit, right-drag to pan, and scroll to zoom. Camera moves from View, frame, and focus are animated, so you keep your bearings. Each light has an icon in the viewport. Click it to select the light. Directional and spot lights also draw a dashed line in the direction they shine. Keyboard shortcuts The shortcuts work only while this editor is the visible tab and no text field has focus. Press ? to see them in the editor. Keys Action Ctrl+Z, Ctrl+Shift+Z or Ctrl+Y Undo, redo Ctrl+S Save G or W, R or E, S Move, rotate, scale X Toggle world and local space J Toggle snapping Ctrl+D or Shift+D Duplicate Delete or Backspace Delete Esc Clear the selection H, Alt+H Hide or show the selection, show everything F Focus the selection Home or Shift+A Frame the whole scene 1, 3, 7 Front, right, and top views. Add Ctrl for back, left, and bottom Shift+G, Shift+W Toggle the grid, toggle the wireframe overlay On macOS, Cmd works in place of Ctrl for undo, redo, save, and duplicate. Undo and redo Every edit can be undone: gizmo drags, Inspector values, adding, duplicating, deleting, renaming, hiding, and moving objects in the outliner. Edits the agent makes go into the same history, so you can undo them too. A slider drag or a run of typing in one field undoes as one step. History keeps the last 200 steps and is cleared when the model is reloaded. The outliner The Scene panel lists every object in the scene as a tree, with children indented under their parent. Each row shows an icon for the object type (mesh, light, camera, or group), its name, and an eye button that appears on hover and hides or shows the object. Hidden objects and their children are dimmed. Click a row to select the object. Click empty space in the outliner or the viewport to clear the selection. Type in the filter box to show only matching objects and their parents. Click the arrow next to a parent to collapse or expand it. Double-click a name, or press F2, to rename it. Use the arrow keys to move through the list. Left and right collapse and expand. Drag a row onto another row to make it a child of that object. Drop it on empty space to move it back to the scene root. The object keeps its place in the world. Right-click a row for rename, duplicate, focus, hide, move to scene root, and delete. Adding and deleting objects Add opens a menu in three groups: Entry Starts as Box 1 x 1 x 1 Sphere Radius 0.5 Plane 2 x 2, laid flat Cylinder Radius 0.5, height 1 Cone Radius 0.5, height 1 Torus Radius 0.5, tube 0.2 Directional Light White, intensity 1, aimed at the view center Point Light White, intensity 1, 2 units above the view center Spot Light White, intensity 10, 30-degree cone, aimed at the view center Empty An empty group to organize other objects under New objects appear at the point the camera orbits around, and shapes rest on the ground. New shapes get a light grey material with low metalness and high roughness. Each new object gets a unique name, so you can address it later by name. Duplicate copies the selection with its children, its own geometry, and its own materials, so editing the copy leaves the original alone. The copy gets the next free name, for example “Crate 3” after “Crate 2”. There is no entry for cameras. A camera that comes in with an imported model shows up in the outliner. Importing a model To add another model to the scene, choose Import model… at the bottom of the Add menu, drop a model file onto the viewport, or drag a model asset from the Asset Explorer onto the viewport. The model arrives as one group named after the file, centered on the view and resting on the ground, and is selected. Its objects are listed under that group, so it moves, hides, and deletes as one. Its animations are not imported. Importing can be undone like any other edit. Format Extensions Notes glTF .glb, .gltf Keeps materials, textures, lights, and editor settings Wavefront OBJ .obj Materials come from the .mtl file the OBJ names, when you pick it too FBX .fbx Units are not converted, so a file modeled in centimeters arrives 100 times too large. Scale the group down STL .stl Geometry only. Gets a grey material PLY .ply Keeps vertex colors. A PLY without faces arrives as a point cloud Collada .dae   3MF .3mf   USD .usdz, .usda, .usdc   Pick or drop the files a model refers to together with it: the .mtl of an OBJ, the .bin of a .gltf, and texture images. The editor matches them to the model’s references by file name. A model asset dragged from the Asset Explorer resolves its references next to the asset, so a .gltf asset must be self-contained. OBJ, Collada, and 3MF materials are converted to the PBR material the Inspector edits, keeping color, textures, and transparency. Animation playback When the model has animations, a playback bar sits along the bottom of the viewport. Press play to run the clip in a loop, drag the time slider to scrub, and pick another clip from the list when there is more than one. Playback moves objects without editing them. The gizmo is hidden while the preview runs. Press stop to put every object back in the pose it had before playback, then edit as usual. Saving stops the preview first, so the saved model keeps that rest pose, not the frame on screen. The animations themselves are kept on save. Transforms and the gizmo Select an object and a gizmo appears on it, with a box around its bounds. Drag the gizmo handles to move, rotate, or scale the object. For exact values, use the Transform section of the Inspector. Position, Rotation, and Scale each have X, Y, and Z fields. Rotation is in degrees. The numbers update while you drag the gizmo. Drag an X, Y, or Z label left or right to scrub its value. Hold Shift for fine steps or Ctrl for coarse ones. The reset button next to each row sets it back to zero, or to one for scale. Children move with their parent, because transforms are relative to the parent in the tree. Inspector The Inspector shows the selected object’s name and type, then collapsible sections. Which sections appear depends on what you selected. Transform Position, rotation, and scale, as described above. Light A light has Color and Intensity. A point or spot light also has Range and Decay, and a spot light has Angle and Penumbra. Directional and spot lights shine down their own forward axis. Use the rotate gizmo to aim them. Lights are written into the file with the glTF KHR_lights_punctual extension, so other glTF tools read them. Geometry Shapes you added from the Add menu get a Geometry section. It edits the size and segment counts of that shape and rebuilds the mesh live. A box has width, height, depth, and segments. A sphere has radius, segments, and angle ranges. The plane, cylinder, cone, and torus have their own sets. Meshes that arrive inside an imported model have no editable geometry parameters, so they have no Geometry section. Material Meshes get a Material section. A mesh with several materials shows a Slot picker first. The section has: Colors: Base Color, Emissive, and a few specialty colors Sliders: Metalness, Roughness, Emission intensity, Opacity, and the physical-material set (Clearcoat, Sheen, Transmission, Iridescence, IOR, and others) Toggles: Transparent, Wireframe, Flat Shading, Depth Test, Depth Write, and others Side, with Front, Back, and Double Lowering Opacity turns on transparency. Raising it back to 1 turns transparency off again, unless the material was transparent before you changed it. For a blockout, Base Color, Roughness, and Metalness are usually enough. Rendering Visible, Cast Shadow and Receive Shadow for meshes, Frustum Cull, and Render Order. Asking the agent Click the sparkle button in the toolbar to open the Assistant panel. It is labeled “3D Assistant” and suggests prompts such as “add a red box and a sphere above it”, “make the floor blue”, and “list everything in the scene”. While the editor is the visible tab, the agent works on the live scene, and you see each change as it happens. It can: List the scene, read the selection, and read one object in full: its children, every material slot, its light settings, and its shape parameters Add shapes, lights, and empty groups, and select, rename, duplicate, show, hide, or delete objects Move an object under another object, or back to the scene root Set position, rotation (degrees), and scale Change a material’s color, emissive color and strength, metalness, roughness, and opacity, for one slot or all of them Change a light’s color and intensity, a point or spot light’s range and decay, and a spot light’s cone angle and penumbra Resize a shape from the Add menu by changing its geometry, for example a box’s width or a cone’s height Undo and redo, including your own edits Frame the scene and capture a screenshot of the viewport, which it can look at to check its own work. The screenshot shows the model only, without the grid, gizmo, or light icons Objects are addressed by id or by name, and names are matched without regard to case. Duplicate names make that ambiguous, so give objects distinct names. The agent changes the open scene, and each change is a step you can undo. Changes are not on the asset until you press Save. Outside the editor, the agent can also work on a model by asset id with no editor open. It can list models, create a model (optionally building the whole scene in the same call), read the scene with its bounds, apply up to 60 edit operations in one call, and validate the glTF. These edits save straight to the asset, and an open editor reloads it. See the nodetool-3d-scene skill for the operation list. A few things need an open editor with a working WebGL context: framing the scene and capturing the viewport. Rendering a shot reference The editor viewport is a quick preview. For a rendered image, the agent has a render_model3d tool that renders a saved model through headless Blender and store the result as a PNG image asset. The editor has no Render button. You ask the agent for the render in chat, for example “render the set from a low angle at 1920 by 1080”. The render takes these settings: Setting Default Notes Camera Scene camera if the model has one, otherwise an orbit camera Orbit cameras take azimuth (45 degrees), elevation (25 degrees), field of view (35 degrees), and a zoom multiplier Size 1024 x 1024 Plus a resolution percentage Lighting studio Presets are studio, soft, and flat. Used only when the scene has no lights of its own Light intensity 1 Multiplies the preset lights Background #808080 Or transparent Engine eevee cycles is slower and higher quality Samples 16 Denoise is on by default and applies to Cycles Requirements and behavior: Blender 5.2 or newer must be installed on the machine that runs the NodeTool server. NodeTool looks at BLENDER_PATH first, then blender on PATH, then the usual install locations. Set BLENDER_PATH to the executable if it is not found. Rendering is off on cloud deployments. It is available on the desktop app, a local server, and a self-hosted install. A server without Blender answers with an error that names the missing install. The render reads the saved asset. Press Save before you ask for a render, or the image will not show your latest edits. A render can run in the background. The agent gets a generation id and collects the finished image later. The result is an image asset in your library, so you can use it anywhere an image goes. If you prefer a graph, the Blender nodes (nodetool.blender.RenderImage, RenderAnimation, RenderPasses, PrepareForEngine, and ExportModel) use the same Blender install. Exporting Save serializes the scene to a .glb and updates the asset, then shows “Saved” with the asset name. Animations that came with an imported model are kept, also after you rename the animated object. Hidden objects are saved too, marked with a nodetool_hidden value in the node’s extras, and come back hidden when the model is reopened. Other glTF tools ignore that value and show them. Names keep their spaces and punctuation. If saving fails, a banner over the viewport shows the error with a button to report it. Open the asset in the Asset Explorer to download or reuse the file like any other asset. See Asset Management. Export downloads a file for another tool and leaves the asset alone. Pick Scene as a format to export everything, or Selection as a format to export the selected object and its children, placed where it sits in the scene. The Cmd+K menu lists the scene exports too. Hidden objects are left out. Unsaved edits are included. If an animation preview is playing, it stops first, so the file holds the rest pose. Format What it keeps GLB Geometry, materials, textures, lights, and, for the whole scene, the model’s animations glTF The same as GLB, as one JSON file with its data embedded OBJ Geometry. No materials STL Geometry, in binary STL, for 3D printing PLY Geometry with vertex colors, in binary PLY USDZ Geometry, materials, and textures, for Apple AR Quick Look A selection exported to GLB or glTF has no animations. OBJ, STL, and PLY need at least one visible mesh. If an export fails, a banner over the viewport shows the error with a button to report it. Using a model elsewhere Workflows. Drag the asset from the Asset Explorer onto the canvas to add a nodetool.constant.Model3D node. Timelines. The timeline add menu has a 3D model entry that lists the glTF assets in your library and places the one you pick as a clip on a video or overlay track. You can also drag the asset onto a track. See Video Editor. Image references. A rendered PNG is an ordinary image asset, so you can drop it into a sketch or use it as a reference in an image workflow. The editor has no direct link to the storyboard surface. Use a rendered image asset to bring a set into one. Limits Only .glb and .gltf assets open in the editor. Other formats can be imported into an open model. You cannot add cameras from the Add menu, and you cannot add new materials or textures. You can change existing material values. The viewport camera is not saved. It is for looking around, not for authoring a shot. The editor has no render button. Rendering goes through the agent and needs Blender 5.2 or newer on the server, and it is unavailable on cloud deployments. The agent can work on the live scene only while the editor is the visible tab. A render uses the saved asset. Unsaved edits do not appear in it. One edit_model3d call applies at most 60 operations. Related Asset Management Video Editor Sketch Editor Game Editor, where a 3D game binds glTF models to its entities--- # CLAUDE.md Source: https://docs.nodetool.ai/CLAUDE.html @AGENTS.md--- # Harness-First Engineering Source: https://docs.nodetool.ai/HARNESS_FIRST.html Harness-First Engineering NodeTool is built by agents as much as by people. An agent cannot click a button, watch a spinner, or eyeball a canvas — it can run a command, read a report, and act on a verdict. Harness-first engineering makes that the primary way every surface is exercised: the headless harness is the interface; the UI is a client of it. Taken to its conclusion — and this repo takes it there — the harness is also the merge gate: a diff selects the checks that must pass, and the registry, not the author, decides what “verified” means. This is already true for most of NodeTool. Workflows have nodetool validate and nodetool debug, mini apps have app debug and app build, timelines have timeline validate/debug, single nodes have node run, the planners and tool contracts have eighteen eval suites, the kernel has the Ring 0 reliability journeys, the packaged backend and deploy image have smoke harnesses. This document names the doctrine those grew into and the machinery that keeps it true: a registry, an audit, and a gate. The rules No surface without a harness. Every product surface — a thing a user or agent can author, run, or edit — is either covered by a headless harness or carries a written gap note in the registry saying why not and what its harness would look like. An uncovered surface with no note fails the build. The harness ships with the surface, not after it. A feature PR that adds a surface adds its harness (or its gap note) in the same PR. “We’ll add the harness later” is the debt the registry exists to make visible. The gate selects your checks — you don’t. A diff maps onto surfaces by the paths it touches, and every harness covering a touched surface runs its selfcheck: nodetool harness gate --base main. Hand-picking which checks to run is how the check you skipped becomes the bug you shipped. Harnesses without a selfcheck are listed as manual work the gate cannot do for you. A capability exists when its harness can drive it. Nothing lands UI-first: the CLI or agent-tool form of a capability is the capability, and the UI renders it. debug_workflow and nodetool debug share one reducer; the app simulator serves the CLI, the build loop, and the server; the Code-node analysis serves the validator, the planner, and the editor. A harness never reimplements the behavior it checks — the core lives in a shared package (@nodetool-ai/execution, @nodetool-ai/node-sdk, app-runtime) and every host calls it, so harness and product cannot drift. A bug fix ships its harness reproduction. Not a description of the bug — a case that fails before the fix and passes after, at the harness level: a validator rule, a golden journey, a deterministic eval case, a debug fixture. A fix without a reproduction is a claim, not a fix. Reports are for machines first. Every harness has --json (or writes a JSON report into its bundle) and an exit code that is the verdict. Human-readable output is a rendering of the report, never the only form. Fail closed. A judge that times out scores the case as failed, an unanswered escalation fails on its timeout, a budget that runs out ends the build as failed. A harness that cannot decide never reports success — see the “gates that reported success without checking anything” class of bug this repo has already paid for. The failure path is drivable. Where a run can escalate, an agent can sit on the failure path (--supervise, interactive: true, resolve_workflow_escalation) instead of the harness only observing the wreckage. The loop is live. Iterative harnesses grow --watch with a verdict diff, so edit→verify is a running process, not a fresh full report each save. A harness runs what the runner runs. Rule 4 keeps the logic shared; it does not stop a harness from constructing the run differently. That is a distinct failure, and a nastier one: the harness and the product call the same core, so nothing looks reimplemented, and the report is confidently wrong. nodetool debug handed ExecutionSession a registry but no resolveNodeType, which hydrates node flags and leaves propertyTypes empty. Correlation analysis reads list-ness only from that map, so every list[...] handle read as non-list and a stream arriving on one collapsed to the last value. Same graph, two answers: workflows run keyframe=2 animate=2 debug keyframe=2 animate=1 Directed Film to Timeline looked like it generated N keyframes and animated one. It does that under debug and not under the runner — a debugging session went looking for a kernel bug that did not exist, and a validator rule was written for it before anyone reproduced the claim through the real runner. So: when a harness reports a defect, confirm it through the canonical runner before diagnosing; if they disagree, the harness is the first suspect, and the thing to diff is how each builds the run, not what the graph contains. Construction differences are auditable — packages/execution/tests/execution-session-hydration-audit.test.ts walks every ExecutionSession.create and fails on any that passes a registry without a resolver, with a KNOWN_UNHYDRATED map for the surfaces that deliberately differ and why. The registry packages/cli/src/harness/registry.ts is the machine-readable inventory: every harness (id, canonical command, capabilities, agent tool, selfcheck, docs pointer) and every surface (the harnesses covering it, the code paths it owns, or its gap note). A surface with a harness is only useful if the gate can find it. paths routes a diff to the surfaces it touches, so a package with no entry anywhere is invisible to nodetool harness gate even when the harness that would have covered it exists — a diff there selects nothing, silently. UNCLAIMED_PATHS closes that hole from the other side: every directory under packages/, plus web/, electron/, mobile/, must appear in some surface’s paths or in UNCLAIMED_PATHS, which names the reason nothing claims it yet and what closing that gap would need. packages/cli/tests/harness-registry.test.ts walks the real repo tree and fails the build on one that is in neither — the same shape as the harness-coverage invariant above, one layer below it. npm run dev:nodetool -- harness list # every harness + capabilities npm run dev:nodetool -- harness audit # surface coverage + gaps npm run dev:nodetool -- harness audit --json # for agents npm run dev:nodetool -- harness audit --strict # exit 1 while any gap remains audit always exits non-zero on a broken registry (an uncovered surface with no gap note, or a surface referencing a harness that does not exist); --strict also fails on documented gaps — the ratchet to pull once a gap is closed. packages/cli/tests/harness-registry.test.ts enforces the same invariants in CI, so the registry cannot rot silently. The capability table A surface is coarse: packages/agents/ is one path, so the gate cannot tell a diff that adds a capability from one that renames a local variable. Five capabilities shipped through that hole with nothing exercising them. packages/cli/src/harness/capability-table.ts closes it, with the registry’s invariant one rung down: no capability without a check or a documented gap. Each entry names the implementation file, the suites a selfcheck runs over the capability, the eval cases whose requiredTools demand it, or the gap note saying what a check for it would look like. npm run capabilities:sync # rewrite from the live registry npm run capabilities:check # fail when stale or uncovered npm run dev:nodetool -- harness capabilities # coverage + documented gaps Only gap is written by hand; the sync derives the rest and preserves the notes. Each entry also carries a fingerprint of what the capability declares — name, description, input schema, category, needsToolCallId — which is what lets harness gate --base <ref> demand a mapping change from a contract change and nothing from a refactor. The comparison is against the merge base of <ref> and HEAD, not <ref>’s tip, so a branch that is merely behind is not asked to answer for a contract that moved on the base branch meanwhile. Details: packages/agents/AGENTS.md § Capability coverage. The gate npm run dev:nodetool -- harness gate # gate the working tree npm run dev:nodetool -- harness gate --base main # gate a branch diff npm run dev:nodetool -- harness gate --dry-run # plan only, run nothing npm run dev:nodetool -- harness gate --all # every selfcheck, diff ignored npm run dev:nodetool -- harness gate --expensive # include bundle-staging class checks npm run dev:nodetool -- harness gate --strict # fail when a touched surface has no harness npm run dev:nodetool -- harness gate --deps # also route through workspace dependents npm run dev:nodetool -- harness gate --include-suites # also run suite-only selfchecks npm run dev:nodetool -- harness gate --jobs 4 # run cheap selfchecks concurrently The gate reads the diff, maps files to surfaces by path prefix, and runs the selfcheck of every harness covering a touched surface. A selfcheck is keyless, deterministic, and target-free: validate runs over every shipped example workflow, reliability-ring0 replays the golden journeys on the kernel, app-debug wiring-checks a shipped bundle, app-build runs its two deterministic cases with a provider that is never called, node-run executes a real node hermetically. Touch packages/kernel/ and the kernel’s journeys run; touch packages/agents/src/app-build/ and the build harness proves itself; touch nothing mapped and nothing runs. A root file that changes every workspace’s behavior (package.json, package-lock.json, root tsconfig*.json, turbo.json, .nvmrc, the root Vitest config, the shared scripts/run-*.mjs launchers) is a global file: it selects every selfcheck, the way npm run test:affected runs every suite for it. A selfcheck marked suiteOnly only re-runs workspace suites that test:affected already runs in full for the same diff, so the gate skips it unless --include-suites is passed. The registry test derives suiteOnly from the real workspace graph and fails when an entry claims it wrongly. For a diff containing a global file, the gate also skips commands that only run workspace test suites covered by the full test:affected selection. Commands containing other work or unknown workspaces still run. This assumes test:affected runs for the same diff. Use --include-suites to run those checks again. --all alone does not establish that coverage. CI adds --include-suites when its skip-tests input disables package tests. --deps also touches a surface when one of its paths sits in a workspace downstream of a changed one. It is opt-in because the hub workspaces (cli, agents, websocket) depend on nearly everything, so most package changes then select most of the registry. The plan step loads no built package, so --dry-run, list, and audit work before npm run build:packages. CI runs selfchecks serially with --jobs 1, while NODETOOL_TEST_WORKERS=1 keeps each check’s test files serial. This avoids competing selfchecks exhausting the game runtime’s wall-clock script batch budgets. Each check reports its start and completion on stderr, including with --json, and its result includes durationMs. JSON results stay on stdout so logs can show progress without changing machine-readable output. The per-check timeout and CI job deadline still apply. Exit code: non-zero when any selfcheck fails, when a capability’s declared contract moved without its coverage mapping moving with it, or — with --strict — when the diff touches a surface only a gap note covers. Harnesses that need a target, key, or model are printed as manual work; the gate never silently narrows what “checked” means (rule 7). The Quality Gate’s examples, reliability, app-build, and bundle legs are hand-instantiated selfchecks from this registry; the direction of travel is for CI to converge on harness gate so the workflow file stops being a second, driftable copy of the selection logic. Adding a surface or a harness Build the core in a shared package; keep the CLI to flags, target resolution, and bundle writing (the pattern every existing harness follows). Add a HarnessEntry with the canonical command, and add or update the SurfaceEntry it covers — including the paths the surface owns, which is what routes future diffs to your harness. Closing a documented gap means deleting its gap note and pointing at the new harness. Give the harness --json and a meaningful exit code from day one, and a selfcheck if any invocation can run keyless and target-free — a harness the gate can run is worth ten the author has to remember. Document it in AGENTS.md next to the other harnesses, in the same PR. Current gaps Run nodetool harness audit for the live surface gaps: the mobile app (the tool contract exists — drive it headlessly like the tool-loop evals do) and the Electron shell (unit tests only; a harness would boot the packaged shell under Playwright’s Electron driver). For the path-claim layer below that — a package with no harness and no surface even naming it — read UNCLAIMED_PATHS in the registry directly: each entry says what a diff there cannot verify headlessly today and what closing it would need.--- # External document changes merge into the draft, and the draft wins Source: https://docs.nodetool.ai/adr/0001-document-sync-draft-wins.html External document changes merge into the draft, and the draft wins An agent, the CLI, or another tab writes a document while an editor holds a dirty draft. Before this decision the editor kept the draft and the next autosave failed the updated_at CAS with no way forward except a refresh. We now merge the external change into the draft per merge unit (shot, clip, layer, line, node, edge). Where both sides changed the same unit the draft wins and the external value is listed in a document-level conflict banner. An external change never enters the undo stack: Cmd-Z only reverts the user’s own edits, so undo cannot silently delete an agent’s work. Considered options Server-side merge: rejected, the server does not know which units the user is still typing in. Locks for the duration of an agent run: rejected, a run takes minutes and would block the editor. Agent wins on a conflict: rejected, it drops keystrokes the user can see. External changes on the undo stack: rejected, undo would remove work the user did not make and write that removal back to the server. Consequences web/src/stores/documentSync.ts is the canonical editor synchronization seam. Each editor supplies its document adapter while the shared module owns debounce, serialization, flush, compare-and-swap recovery, and routing of external changes. The checked inventory is in docs/document-sync.md. The resource_change payload carries the ops the write was made with. The headless bridge attaches its ui_* op list to the document mutation, and the server broadcasts it as the patch. The document mutations keep their whole-section inputs. A write with no ops (another tab’s autosave, a CLI restore) is one merge unit: a dirty draft keeps everything and the banner lists one conflict, “document replaced elsewhere”. Draft wins also over an external delete: a unit the user is editing survives, and the next autosave recreates it. A shared editor controller that meets a CAS rejection reloads the latest document before it retries a still-dirty draft. Headless capability retry behavior remains specific to the capability that issued the write. Accepting an external value from the conflict banner is the user’s own edit and goes on the undo stack. A dangling edge (its node deleted in the draft) is dropped and listed as a conflict.--- # CodeAct is the one agent loop; Agent is retired Source: https://docs.nodetool.ai/adr/0002-codeact-is-the-one-agent-loop.html CodeAct is the one agent loop; Agent is retired packages/agents/src/agent.ts runs a second loop next to CodeAct: TaskPlanner → ParallelTaskExecutor → TaskExecutor → CodeActExecutor → CompilerAgent, wrapped in a plan approval gate, a plan cache, a checkpoint store, per-phase model selection, and skill auto-select by objective word overlap. One command constructs it. Everything Agent alone wires is therefore reachable only from the CLI, while the loop the product ships — a CodeAct session over a gated belt — is what chat, plan mode, and every sub-agent run on. Two loops means every fix lands twice or in the wrong one: budgets, the permission ladder, transcript compaction, and the scheduler each have to be built for both, and the one nobody uses is where they rot. The repo made this call once when CodeAct replaced script mode (codeact-design.md); this applies it to the loop that survived. D1. nodetool agent run becomes one CodeAct session over the belt chat uses, with create_plan and execute_plan reachable the way they are in chat. What goes with Agent: CompilerAgent, because synthesis is the parent loop’s next turn and it already reads task:<id> from memory; the plan approval gate, because the permission ladder classifies execute_plan as external and that classification is the approval; the plan cache and the checkpoint store, which no caller supplies; planningModel and reasoningModel, which no caller sets; and skill auto-select, whose replacement is the chat’s skill catalog plus load_skill. What stays: TaskPlanner, ParallelTaskExecutor, TaskExecutor, CodeActExecutor, SubAgentTool, and the supervisor. Chat plan mode runs on the planner and the parallel executor, so those are a product path rather than Agent’s scaffolding. D2. The workflow AgentNode (packages/llm-nodes/src/nodes/agents.ts) stays a provider-native JSON tool loop, and joins the shared permission gate and the shared RunBudget. A node inside a saved graph wants a small loop a graph author can bound and predict, and provider-native tool calls give that with no sandbox in the path. Handing it the sandbox and the nodetool.* object model is a separate product question — what a node that runs arbitrary code means for a graph somebody else opens and runs — and it is not answered here. Evidence Every host that runs a model loop, and where each one is constructed: Host Loop Constructed in Gate Budget Chat turn (web, Telegram, MCP) one CodeAct session over provider.generateLoop packages/websocket/src/session/chat-turn.ts (createChatCodeActSession) gateTools createChatTurnBudget Chat plan mode create_plan → TaskPlanner.planMultiTask, execute_plan → ParallelTaskExecutor packages/agents/src/capabilities/agents.ts inherited belt the run’s budget, via subAgentRuntime Workflow AgentNode JSON tool calls over provider.generateLoop packages/llm-nodes/src/nodes/agents.ts none its own createRunBudget, or the caller’s via budgetFromContext CLI nodetool agent run Agent: TaskPlanner → ParallelTaskExecutor → TaskExecutor → CodeActExecutor → CompilerAgent packages/cli/src/commands/agent.ts none none The call graph is one grep: grep -rn "new Agent(" packages web electron --include=*.ts --include=*.tsx It answers with packages/cli/src/commands/agent.ts and the packages/agents/tests/ suites that exercise Agent directly. There is no other production caller, so the CLI is the whole audience for what Agent wires. The options only Agent passes are dead on the same evidence. checkpointStore and planCache are supplied at one site each, both inside agent.ts, and FileCheckpointStore.load resolves without reading anything until a loadFromDisk() that Agent never awaits, so a resumed run resumes from an empty store. planningModel and reasoningModel appear in agent.ts and task-planner.ts and are set by no caller, so the per-phase model selection they exist for has never selected a different model. The plan approval gate is orphaned from the other side. PLAN_APPROVAL_CONTEXT_KEY has one reader, Agent.resolveApprovalCallback, and the websocket installs a callback under it (attachPlanApproval, packages/websocket/src/session/chat-history.ts) for an Agent that is never constructed in that process. Retiring Agent deletes the reader and the installer together. Considered options Keep Agent as a second supported host: rejected, this is the status quo and it is what makes each of A1, A2, A4, and A7 land twice. Delete TaskPlanner and the DAG executors with Agent: rejected, create_plan and execute_plan run on them from chat. Keep CompilerAgent as the synthesis step under CodeAct: rejected, it stores the finish_step payload without validating it against outputSchema, and the memory read that replaces it is already what execute_plan documents. Converge AgentNode on CodeAct in the same change: rejected, it changes what a node in a saved graph is allowed to do, which is D2’s separate question. Give AgentNode its own gate and its own budget rather than the shared ones: rejected, a second classification table is exactly the duplication D1 removes. Consequences nodetool agent run gains the chat belt’s gate and budget, and loses the plan-first-then-execute shape it had by construction. The rewrite has to give that shape back — an objective that asks for a plan, or a permission mode the command exposes. Plan approval moves from a bespoke callback to the permission ladder. A host with nobody to ask fails closed at execute_plan instead of inside Agent. The planning_update message type stays, because create_plan emits it. The compiler phases stop being produced, so the consumers that render them (packages/cli/src/useExecutionState.ts) lose those branches. AgentPolicy’s bounds have one consumer left, execute_plan. They either fold into the run budget or survive as the constants file the two executors read. The memory contract is unchanged. “Final synthesis by CompilerAgent” becomes “the calling loop reads task:<id> after execute_plan returns” in packages/agents/AGENTS.md and docs/agent-memory.md. AgentNode still needs the gate. Its budget already comes from the shared RunBudget, its own or the calling run’s.--- # Agent CLI Source: https://docs.nodetool.ai/agent-cli.html The nodetool agent command runs one CodeAct turn from the terminal. It takes an objective, hands it to the model as the user message, and prints the final answer. The model acts by writing sandboxed JavaScript over the toolbelt — the same loop the chat runs — so what it does is up to it: call a tool, decompose the objective with create_plan and run the DAG with execute_plan, or answer straight away. Every run streams a live trace of tool calls, plan events, and task events to stderr. There is no configuration file. Everything the command needs is an argument. Subcommands nodetool agent run --objective "..." # run the agent loop nodetool agent diagnose <job_id> # aggregate a failed run into one report nodetool agent run The objective comes from -o, --objective or, when that is absent, from piped stdin. nodetool agent run -p anthropic -m claude-sonnet-5 \ --objective "Research the latest AI trends" echo "Research the latest AI trends" | \ nodetool agent run -p anthropic -m claude-sonnet-5 Options: -o, --objective <text> — Objective for the agent (else read from stdin) -p, --provider <id> — Provider id (required) -m, --model <id> — Model id (required) -w, --workspace <path> — Workspace dir for file tools (default: cwd) --max-iterations <n> — Tool-calling rounds in the turn (default 25) --permission-mode <default|auto|plan> — How tool calls are gated (see Permission mode) --cost-cap <usd> — Ceiling on provider spend for the whole run; 0 lifts it --timeout <s> — Wall-clock bound on the run in seconds; 0 leaves it none --json — Emit each agent event as a JSON line on stderr -v, --verbose — Include low-level chunk and other events in the trace Run budget --cost-cap and --timeout override two of the five NODETOOL_AGENT_* settings a chat turn reads (NODETOOL_AGENT_TURN_COST_CAP_USD, NODETOOL_AGENT_TURN_DEADLINE_MS); concurrency, total turns, and the unpriced-token ceiling come from the settings alone. The run holds one budget and shares it downward — a sub-agent, an execute_plan DAG, an AgentNode reached through run_node — so the ceiling bounds the run, not each loop. A cap is admission: a turn whose worst case would cross it is refused before the call, not noticed after the money is spent. A model with no catalog price has no worst case and is admitted against a prompt-token ceiling instead, never as free. A ceiling that stops the run prints its reason and exits non-zero (agent stopped: turn budget of $0.01 reached); with --json the same reason arrives as an error event. Every run ends with what it committed — spent $0.0123. nodetool agent run -p openai -m gpt-5.4-mini -o "Research AI trends" \ --cost-cap 0.50 --timeout 120 The final result goes to stdout and the trace goes to stderr, so you can capture the result on its own: nodetool agent run -p openai -m gpt-5.4-mini \ -o "Summarize NodeTool" > result.txt Permission mode Every tool call passes one gate. default runs reads and asks before each write, execute, or external call. auto runs everything. plan runs reads and blocks the rest. An unrecognized value is an error. With a terminal on stdin, the default mode is default. The prompt goes to stderr and takes y (this call), n (refuse), or a (this tool for the rest of the run). When the objective arrives down a pipe nobody can answer, so the run uses the headless gate (auto), prints a notice once, and denies anything the gate escalates. An explicit --permission-mode still applies. With --json the notice and any prompt arrive as log_update events. See Permission mode in the CLI reference. Tools The agent gets the default toolbelt: files, search, browser, PDF, vision, critique, memory, assets, todo, plus the platform tools — workflows, nodes, jobs, apps, models, and media generation. This is the same belt the chat agent and the MCP bridge assemble, so the surfaces cannot drift. The belt is not narrowable from the command line — nodetool agent run has no --tools flag. The belt is assembled by buildCliAgentBelt, shared with the CLI’s own chat, and carries the capabilities that spawn loops of their own: run_subtask, start_subtask, wait_subtasks, run_search, and — here, unlike chat — both create_plan and execute_plan. The provider itself is offered execute_code plus the direct tools; the rest of the belt lives in the sandbox. What the planner used to do agent run used to plan a task DAG up front, execute it, and have a compiler model write the final answer. The model now decides: it calls create_plan when the objective wants decomposing, and execute_plan runs that DAG on the same executor. Four things went with the old pipeline and have no replacement flag: the plan approval prompt (the permission gate on execute_plan is the approval), checkpoints and the plan cache, the separate planning and reasoning models, and skill auto-selection by word overlap (the model loads a skill itself). --max-steps is gone with them; --max-iterations now bounds tool-calling rounds in the turn rather than action rounds per step. nodetool agent diagnose Aggregates a failed run — the failing node or step, its error, the last LLM call, and the agent’s memory — into one report. nodetool agent diagnose <job_id> nodetool agent diagnose <job_id> --json nodetool agent diagnose <job_id> --trace-file trace.jsonl Messages and spans come from a debug bundle in nodetool-debug/<job_id>-*/ when one exists. --trace-file wins over NODETOOL_TRACE_FILE, which wins over the bundle’s own trace.jsonl. --api-url points the job lookup at a remote server. Output # Capture the result, discard the trace nodetool agent run -p openai -m gpt-5.4-mini -o "Task" 2>/dev/null > result.txt # Machine-readable event stream on stderr nodetool agent run -p openai -m gpt-5.4-mini -o "Task" --json 2>events.jsonl API keys Keys come from the secret store (nodetool secrets store OPENAI_API_KEY) or the environment: export OPENAI_API_KEY=sk-... export ANTHROPIC_API_KEY=sk-ant-... Chaining runs nodetool agent run -p anthropic -m claude-sonnet-5 \ -o "Research topic X" > research.md nodetool agent run -p anthropic -m claude-sonnet-5 \ -o "Write an article based on: $(cat research.md)" Related Documentation Chat & Agents — Agent system overview Chat CLI — Interactive chat interface NodeTool CLI — Complete CLI reference--- # Agent Memory System Source: https://docs.nodetool.ai/agent-memory Navigation: Chat & Agents → Agent Memory Not the same as the memory_* capabilities. This page describes per-run scratch space (context.memory) shared between the steps of one workflow. For the notes an agent saves durably, across every conversation, see Memory. The agent memory system is the single source of truth for everything that flows between agents, tasks, steps, sub-agents, and tools during a workflow run. One AgentMemory instance lives on every ProcessingContext as context.memory. All executors read from and write to it through a single namespaced API, and every agent accesses it through three auto-attached tools: list_shared — discover what’s available (metadata only) read_shared — fetch full values for specific keys share_result — publish a value under the shared: namespace This page is the full reference. For a quick orientation, jump to Quick Reference or Examples. Why This Exists Earlier versions of the agent system kept results in three uncoordinated stores (context._variables, ParallelTaskExecutor.taskResults, TaskBoard.task.result) and had each agent type deliver upstream results to the LLM differently. Plan mode replaced the default execution prompt with a “dependency context” block that stripped the finish_step discipline, so downstream tasks routinely lost results. The fix is one store, one API, and one access pattern — progressive disclosure via tool calls: Auto-injection is wasteful: dumping all upstream results into every prompt bloats context with data the step rarely needs. Tool-mediated access is selective: the agent sees a tiny “what’s available” hint in the system prompt and pulls only the values it actually needs. Specific declared dependencies still get a nudge: if the planner declared task.dependsOn or step.dependsOn, those exact memory keys appear in the user message as a hint — but the values are not included. Architecture ┌──────────────────────────────────┐ │ ProcessingContext.memory │ │ │ │ Map<string, MemoryEntry> │ │ │ │ step:<id> step_result │ │ task:<id> task_result │ │ input:<key> input │ │ shared:<k> shared │ └────────────┬─────────────────────┘ │ ┌─────────────────────────┴─────────────────────────┐ │ │ ▼ ▼ CodeActExecutor ParallelTaskExecutor (writes step:, task: on finish-task (passes task.dependsOn IDs as steps; auto-attaches list_shared / upstream key hints to read_shared / share_result tools) TaskExecutor) Every executor writes results into context.memory. Every step has the three memory tools available automatically, and the system prompt instructs the model when to use them. Key Namespaces Namespace Helper Written By Used For step:<id> memoryKeys.step(id) CodeActExecutor, TaskExecutor (terminal step failure) Per-step results task:<id> memoryKeys.task(id) CodeActExecutor (finish-task steps), ParallelTaskExecutor Per-task results input:<key> memoryKeys.input(key) TaskExecutor, ParallelTaskExecutor Caller-supplied inputs shared:<key> memoryKeys.shared(key) share_result tool Cross-agent communication, scratch space Always use the helper functions when constructing keys — they prevent typos and make grep-able call sites. import { memoryKeys } from "@nodetool-ai/runtime"; context.memory.has(memoryKeys.task("research_phase")); context.memory.getValue(memoryKeys.step("step_1")); Memory Entry Shape export interface MemoryEntry { /** Globally unique key (use memoryKeys.*). */ key: string; /** Categorization for filtering and rendering. */ kind: "task_result" | "step_result" | "input" | "shared"; /** Stored value (any JSON-serializable structure). */ value: unknown; /** Optional ID of the producer (task / step / agent / tool). */ source?: string; /** Optional human-readable title shown in `list_shared`. */ title?: string; /** Optional brief description. */ description?: string; /** Wall-clock ms when the entry was first written. */ createdAt: number; } title and description flow through to list_shared output, so set them when you want the LLM to see a friendly label rather than a UUID. Memory Tools (the LLM-facing API) Three tools are auto-attached to every step executor. Inside a CodeAct step the model reaches them as nodetool.shared.list(opts), nodetool.shared.read(keys), and nodetool.shared.publish(key, value, opts) (opts carries title and description). The wire names below are what the tool calls and the supervisor’s StepExecutor use. list_shared Discover available entries without paying for their values. // args { "kind": ["task_result", "shared"], // optional filter "key_prefix": "task:", // optional filter "sources": ["research", "summary"] // optional filter } // result { "total": 4, "returned": 4, "truncated": false, "entries": [ { "key": "task:research", "kind": "task_result", "title": "Research findings", "description": "Top three sources from the web search step.", "source": "research", "valueBytes": 142, "createdAt": "2026-05-07T14:00:01.234Z" } // ... ] } valueBytes is the character length of the value (the string itself, or its JSON serialization). The model can use it to budget reads. Each description is cut at 240 characters with a trailing …. The result is capped at 200 entries, the first 200 in insertion order, and truncated: true reports that more exist. createdAt is an ISO timestamp here. read_shared Fetch full values for one or more keys. // args { "keys": ["task:research", "step:summary"] } // result { "entries": { "task:research": { "key": "task:research", "kind": "task_result", "value": { "findings": ["alpha", "beta"] }, "source": "research", "title": "Research findings", "createdAt": 1700000000000 } }, "missing": ["step:summary"] } Missing keys are reported in missing so the model can decide whether to retry, list again, or proceed without them. A key with no : is retried under shared:, so the suffix given to share_result reads back as written. Each entry is the full MemoryEntry, so createdAt is a millisecond number here. share_result Publish a value under the shared: namespace so other agents and steps can discover it via list_shared. // args { "key": "top_source", // suffix; stored as "shared:top_source" "value": "https://example.com", "title": "Top source URL", "description": "Picked by the researcher agent." } // result { "ok": true, "key": "shared:top_source", "kind": "shared", "createdAt": "..." } A key that already starts with shared: has the prefix stripped, so writing back a key from list_shared does not produce shared:shared:<key>. title defaults to the key suffix, and source is always share_result. Writes are restricted to the shared: namespace to prevent agents from spoofing step / task / input results. Direct API Reference The AgentMemory class lives in packages/runtime/src/agent-memory.ts and is re-exported from @nodetool-ai/runtime. Tools and executors use this API directly; agents reach it only through the three memory tools above. Other methods: delete(key) returns whether the key existed, size() returns the entry count, and formatForPrompt(filter?) renders matching entries as a Markdown block ordered by createdAt (empty string when none match). Writing context.memory.set({ key: memoryKeys.task("research"), kind: "task_result", value: { findings: ["alpha", "beta"] }, source: "research", title: "Research findings", description: "Top three sources from the web search step." }); set returns the persisted MemoryEntry. Repeated writes for the same key overwrite the value but preserve the original createdAt. Reading context.memory.get(memoryKeys.task("research")); // MemoryEntry | undefined context.memory.getValue<T>(memoryKeys.task("research")); // T | undefined context.memory.has(memoryKeys.task("research")); // boolean context.memory.snapshot(); // MemoryEntry[] Listing & Filtering context.memory.list(); // all context.memory.list({ kind: "task_result" }); context.memory.list({ kind: ["task_result", "input"] }); context.memory.list({ keys: ["task:research", "task:report"] }); context.memory.list({ keyPrefix: "step:" }); context.memory.list({ sources: ["research"] }); Subscriptions const unsubscribe = context.memory.subscribe((entry) => { console.log(`memory write: ${entry.key} (${entry.kind})`); }); // later... unsubscribe(); UIs can use this to render a live memory side panel. Snapshot and restore const saved = context.memory.snapshot(); // MemoryEntry[] in insertion order context.memory.restore(saved); // rewrites each entry, keeps createdAt, notifies subscribers Persist a snapshot at a checkpoint and call restore on a fresh AgentMemory to rebuild it. Existing keys are overwritten. Clearing context.memory.clear(); // wipe everything context.memory.clear({ kind: "step_result" }); // selective context.memory.clear({ keyPrefix: "input:" }); // by prefix How Each Agent Type Uses Memory CodeActExecutor (packages/agents/src/codeact/codeact-executor.ts) The execution engine for a single step. Writes: Trigger Key Kind Always step:<step.id> step_result Last step of a task (finish-task) task:<task.id> task_result Step failed (error, budget stop, iterations exhausted, or no finish()) step:<step.id> (with { error }, title Failed: <instructions>) step_result LLM access: The memory tools ride in the toolbelt like any other, but the object model is their one documented form: the prompt teaches nodetool.shared.list() / nodetool.shared.read(keys) / nodetool.shared.publish(key, value), and the three wire names drop out of the raw tool catalog like every other wrapped tool. The user message includes only specific declared upstream keys as a hint, under the heading “Required upstream context — read these before acting (via await nodetool.shared.read([...]))”: step:<id> for every entry of the step’s dependsOn (intra-task deps). any key supplied via CodeActExecutorOptions.upstreamMemoryKeys (typically task:<id> from the parent task’s dependsOn). Values are not included, and the keys are listed whether or not they exist yet. The agent calls nodetool.shared.read to fetch them. Tool attachment: getSharedTools() (packages/agents/src/tools/shared-tools.ts) — a belt built from the shared capability module’s specs — is auto-pushed into the step’s tool list at construction time, alongside any caller-supplied tools. Mount policy stays with the executor: the host never mounts these. Completion is finish(result) in the sandbox, validated host-side against the step’s schema. Custom prompts are preambles, not replacements: A caller-supplied systemPrompt is layered before the default execution prompt, so the contract — including the tool catalog and the finish() discipline — is non-bypassable. TaskExecutor (packages/agents/src/task-executor.ts) Walks the step DAG of a single task. On startup it seeds caller inputs: for (const [key, value] of Object.entries(this.inputs)) { this.context.memory.set({ key: memoryKeys.input(key), kind: "input", value, title: key }); } Keys already seeded by ParallelTaskExecutor are skipped. Each step runs on a copy of the context that shares the task’s memory (context.copy({ shareMemory: true })). The only step: key it writes itself is a terminal failure: failStepEvents records { error } under memoryKeys.step(step.id) so the parent can read the reason. Every successful step result is written by the CodeActExecutor running it. TaskExecutor accepts an optional upstreamMemoryKeys array (e.g. task:<id> keys from the parent plan). It forwards this verbatim to every step executor it creates. ParallelTaskExecutor (packages/agents/src/parallel-task-executor.ts) Runs a TaskPlan of multiple tasks as a DAG. It owns no private result map — everything lives in context.memory. Operation Memory Action Startup: seed inputs set each as input:<key> For each task: derive upstream keys task.dependsOn.map(memoryKeys.task) → forwarded as upstreamMemoryKeys to TaskExecutor After task executor completes If no is_task_result was emitted, fall back to step:<lastStepId> Idempotent task write set task:<task.id> only if not already present Read final result getFinalResult() returns getValue(task:<lastTaskId>) Read all results getAllResults() lists all task_result entries Read specific task getTaskResult(id) Downstream tasks see their declared upstream task keys as hints in the step user message and pull values via read_shared when needed. StepExecutor (packages/agents/src/step-executor.ts) The step executor of the supervisor agent and the app-build harness, not of execute_plan. It attaches the same three tools. Its system prompt has a “Memory Tools (progressive disclosure)” section that names list_shared, read_shared, and share_result directly. Its user message lists declared upstream keys that exist in memory, with titles: # Required upstream memory (call `read_shared` with these keys): - task:research_phase — Research findings Steps finish through finish_step. The calling loop (execute_plan’s caller) There is no synthesis stage that reads memory on the run’s behalf. When execute_plan returns, each task’s value is in context.memory under task:<id> and in the call’s own results object; the CodeAct session that made the call writes the answer on its next turn, calling read_shared for anything the return did not carry. That is what execute_plan’s description tells the model to do, and it is why the planner is told not to plan an assembly task. Propagation Flow This is the canonical end-to-end flow for a multi-task plan: 1. Caller: execute_plan(plan) 2. ParallelTaskExecutor.execute() ├─ Seed inputs: context.memory.set({ kind: "input", ... }) └─ For each executable task: └─ TaskExecutor.executeTasks() └─ For each step: └─ CodeActExecutor.execute() ├─ system prompt teaches nodetool.shared.* ├─ buildUserMessage() → instructions │ + "Required upstream context" hint listing │ declared dependency keys (no values) ├─ model writes actions that may call: │ - nodetool.shared.list → returns metadata │ - nodetool.shared.read → returns values for chosen keys │ - nodetool.shared.publish → publishes shared facts ├─ finish(result) accepted → storeCompletionResult() │ ├─ context.memory.set({ key: "step:<id>", ... }) │ └─ if useFinishTask: │ context.memory.set({ key: "task:<id>", ... }) └─ yield step_result 3. ParallelTaskExecutor: ensure task: entry exists (idempotent) 4. Mark task.completed = true → unblocks downstream tasks 5. Next iteration: downstream tasks now executable. Their step user messages name the upstream task keys; agents call nodetool.shared.read when they actually need the values. 6. execute_plan returns the task results; the calling session's next turn reads task:<id> via read_shared for anything it still needs and writes the answer. Examples Inspect memory at the end of a run import { memoryKeys } from "@nodetool-ai/runtime"; // After the turn that called execute_plan has finished: console.log("All task results:", context.memory.list({ kind: "task_result" })); console.log( "Specific task:", context.memory.getValue(memoryKeys.task("research_phase")) ); What the LLM sees (StepExecutor system prompt extract) The StepExecutor default execution system prompt includes the block below. A CodeAct step is taught the same three operations as nodetool.shared.list, read, and publish. ## Memory Tools (progressive disclosure) - Shared agent memory holds results from prior steps and tasks, original inputs, and facts published by other agents. - Memory contents are NOT auto-included in your prompt. If you need upstream context, discover it on demand: 1. Call `list_shared` to see what's available (returns metadata only — keys, titles, kinds, byte sizes). 2. Call `read_shared` with the specific keys you actually need; it returns full values. 3. Call `share_result` to publish a value under `shared:<key>` so other agents can find it via `list_shared`. - Pull only what you need — don't fetch every entry by reflex. And the StepExecutor user message for a step that depends on task:research_phase looks like: Write a report from the upstream findings. # Required upstream memory (call `read_shared` with these keys): - task:research_phase — Research findings The model then chooses whether to read the key or proceed. Pre-populate memory before running Useful in tests, and for handing a run what an earlier one already found: context.memory.set({ key: memoryKeys.task("prior_research"), kind: "task_result", value: cachedFindings, source: "prior_research", title: "Cached prior research", description: "Findings from a previous run; reuse instead of re-researching." }); The first list_shared call in any step under this context surfaces the entry, and read_shared fetches it. Tool that publishes to shared memory directly (without an LLM round-trip) For deterministic publish-from-code use cases, write to the API directly: context.memory.set({ key: memoryKeys.shared("top_source"), kind: "shared", value: "https://example.com/article", source: "data_pipeline", title: "Top source URL" }); Subsequent agents will see this via list_shared and can fetch it via read_shared. Subscribe in a host application (UI sidebar) const unsubscribe = context.memory.subscribe((entry) => { ui.appendMemoryEntry(entry); // render the new entry in a side panel }); Quick Reference import { AgentMemory, memoryKeys, type MemoryEntry } from "@nodetool-ai/runtime"; // Already mounted on every ProcessingContext context.memory; // AgentMemory // Write context.memory.set({ key: memoryKeys.task("t1"), kind: "task_result", value: result, source: "t1", title: "Task One" }); // Read context.memory.get(memoryKeys.task("t1")); // MemoryEntry | undefined context.memory.getValue(memoryKeys.task("t1")); // unknown | undefined context.memory.has(memoryKeys.task("t1")); // boolean // List / filter context.memory.list({ kind: "task_result" }); context.memory.list({ keyPrefix: "step:" }); context.memory.list({ sources: ["research"] }); // Subscribe const off = context.memory.subscribe((entry) => { /* ... */ }); // Clear context.memory.clear({ kind: "step_result" }); LLM-facing tools (auto-attached to every step): Tool Purpose Returns list_shared Discover entries (metadata only) { total, returned, entries: [...] } read_shared Fetch full values for specific keys { entries: { ... }, missing: [...] } share_result Publish under shared:<key> { ok, key, kind, createdAt } Design Decisions Why progressive disclosure and not auto-injection? Auto-injecting every memory entry into every prompt is wasteful. Most steps need one or two specific upstream values; dumping all of them costs tokens and pollutes attention. Progressive disclosure mirrors how a human researcher works: scan the index, fetch the specific document, ignore the rest. Why expose memory through tools instead of a dedicated channel? Models in 2026 are excellent tool callers — and tools come with rich JSON schemas that make discovery, filtering, and parameter validation trivial. Adding bespoke prompt syntax for memory access would be reinventing function calling. Tools also give us first-class observability: every memory access shows up as a tool_call_update in the message stream. Why is share_result restricted to shared:? Step results, task results, and inputs are owned by the executors. Letting an agent overwrite a task:<id> entry would let it spoof the result of work it didn’t actually do, breaking the audit trail. Agents publish under shared: and the executor namespaces stay tamper-proof. Why does the user message still mention specific upstream keys? A pure tool-only design would force the agent to call list_shared even when the planner already declared the dependency. That’s an unnecessary round trip for a known-relevant key. The user message names exactly the keys the planner pinned (step.dependsOn plus parent-task dependsOn), so the agent can go straight to read_shared for declared deps and use list_shared only when it needs to discover beyond them. Why a single map and not a relational store? Agent runs are short-lived and the data is small (typically tens of entries). A Map plus structured rendering covers every observed use case without the operational cost of a database. Persistent storage belongs in the asset / vector layer. Why namespaced string keys instead of typed IDs? The LLM has to read keys back from the tool result and pass them to read_shared. Strings round-trip cleanly through prompts and logs without serialization games. Why isn’t memory shared across context.copy()? Copies are designed for isolated sub-runs. If a sub-run should inherit memory, the caller can set entries from the parent before kicking it off. Default isolation is the safer choice. Why does systemPrompt no longer replace the default execution prompt? Replacing it stripped the memory-tool documentation and the completion discipline. The fix layers any caller preamble before the default prompt rather than replacing it. The execution contract (memory tools, output schema, completion protocol, conclusion-stage rules) is now non-bypassable. Troubleshooting Symptom: A downstream task’s prompt shows the upstream key hint but the agent never reads it. The model may have decided it doesn’t need the value. If you know it should, make the user instructions more explicit (“read the upstream findings via read_shared before writing”). Symptom: Task result key is missing after the step yielded step_result with is_task_result: true. A step executor only writes task:<id> for steps where useFinishTask === true. That flag is set by TaskExecutor.isFinishStep() for the last step in the task. Steps in the middle of a task only write step:<id>. For belt-and-suspenders, ParallelTaskExecutor performs an idempotent task-result write after each task, falling back to the last step’s value. Symptom: Tests pass but memory entries seem stale across test runs. A new AgentMemory instance is constructed for every ProcessingContext. If you reuse a context across tests, call context.memory.clear() between them. Symptom: Agent calls list_shared and gets truncated: true. The list response is capped at the first 200 entries in insertion order. Narrow the filter (kind, key_prefix, or sources) to reach later ones. Related Chat & Agents — agents overview packages/runtime/src/agent-memory.ts — AgentMemory implementation packages/agents/src/capabilities/shared.ts — list_shared / read_shared / share_result (the shared capability module; packages/agents/src/tools/memory-tools.ts is the belt executors mount them from) packages/agents/tests/memory-propagation.test.ts — end-to-end propagation tests including the tool round-trip packages/agents/tests/memory-tools.test.ts — unit tests for the memory tools packages/runtime/tests/agent-memory.test.ts — AgentMemory unit tests--- # Agent Resource Links in Chat — Concept Source: https://docs.nodetool.ai/agent-resource-links-design.html Agent Resource Links in Chat — Concept Author: Matti Georgi Status: Draft — for review Last updated: 2026-08-03 1. Summary When the agent builds something — an asset, a storyboard, a script, a sketch, a timeline, a mini app — the chat has no way to point at it. Tool results carry ids inside JSON (that’s fine, and stays as-is), but the agent’s prose can only describe the thing: “I’ve added three shots to the storyboard” with nothing to click. Following progress means expanding tool cards and reading JSON; giving feedback means describing the target in words. The fix is a resource URI the agent can drop into ordinary markdown, generalizing the asset:// scheme the codebase already uses — the kind is the scheme: storyboard://sb_01hxyz asset://as_01hab2 timeline://tl_01hqrs#clip=cl_9 The chat renders any such link as a compact inline resource chip — kind icon, label, a thumbnail for assets — and clicking it opens the resource in its editor tab or viewer. The agent learns the scheme from its system prompt and from tool results, which gain one string field (url) carrying the ready-made link. The same URIs work inbound: one pasted or inserted into the composer tells the agent exactly which document — or which shot, clip, or layer — the user means. No new message types, no tool-result restructuring, no server changes beyond a line in the system prompt. 2. Design goals Progress is legible in prose. “Rendered the intro → [intro v2]” reads like a sentence, with the artifact one click away. The narrative stays the agent’s; links decorate it. Feedback has a target, both directions. The user clicks a chip to open the resource, or references it in a reply; the agent receives the same URI back and can act on precisely that document and sub-target. Tool results stay JSON. No card layer derived from result shapes. The only result change is additive: a url string field so the agent can copy a correct link instead of composing one. One vocabulary. URI kinds map 1:1 onto the existing UiSurfaceType / UiDocumentRef types and WorkspaceTabsStore.openTab — no parallel naming scheme. Degrades to text. In any renderer that doesn’t know the scheme (logs, CLI, eval transcripts, old clients), the link is still a legible [label](timeline://…) markdown link. 3. Current state (what the concept builds on) ChatMarkdown.tsx already special-cases links. Its a component override renders image hrefs as inline images and audio hrefs as players (web/src/components/chat/message/ChatMarkdown.tsx:81-109). A resource scheme is a third branch in the same switch. An asset scheme already exists. asset://<id> resolves to /api/storage/<id> via resolveUri (web/src/utils/imageUtils.ts), and backend media tools already return asset_uri: "asset://…" from persistOutput. The document vocabulary exists. UiSurfaceType / UiDocumentRef in packages/protocol/src/api-types.ts name every editor surface; WorkspaceTabsStore.openTab({type, ref}) opens any of them — the pattern ui_storyboard_assemble_timeline already uses. Sub-targets are addressable. Every surface has selection tools (ui_timeline_select_clip, ui_storyboard_select_shot, ui_sketch_select_layer, …), so a link fragment can both name and select a sub-target on open. Caveat: react-markdown’s default urlTransform strips unknown URL schemes for safety — resource links vanish unless their schemes are explicitly allowlisted in the ReactMarkdown props. 4. The URI scheme <kind>://<id>[#<param>=<value>] kind := asset | workflow | timeline | storyboard | sketch | script | app | model3d | collection | thread id := the resource's database id #… := optional sub-target, kind-specific: #shot=<id> (storyboard) #clip=<id>|t=<seconds> (timeline) #layer=<id> (sketch) #scene=<n> (script) #node=<id> (workflow) #component=<id> (app) Rules: kind values are the UiSurfaceType union plus asset and collection. Adding a kind means extending one union in @nodetool-ai/protocol. Parsing and formatting live in one module in protocol (parseResourceUri / formatResourceUri), shared by web, the eval bridges, and the CLI — no per-surface regex. asset:// is simply the asset kind’s scheme — existing tool results keep working unchanged. The legacy long form nodetool://<kind>/<id> still parses (links persisted by earlier builds), but nothing emits it; the nodetool:// scheme is otherwise reserved by the mobile app’s deep links. Unknown kind or malformed URI → rendered as plain text, never a broken chip. 5. Architecture Agent emits markdown: "Added the opening shots — [Beach intro](storyboard://sb_x#shot=s3)" │ ▼ ChatMarkdown `a` override href parses as resource URI? → <ResourceChip uri=… label=children/> else → today's <a> handling (image / audio / plain) │ ▼ ResourceChip icon + label (+ thumbnail for assets, via resolveUri) click → openResource(uri): WorkspaceTabsStore.openTab({type: kind, ref: id}) then apply sub-target via the surface's selection bridge │ ▼ inbound Composer: chip pasted/inserted → URI travels as plain text in the user message; the agent parses it with the same module. Optionally the client mirrors it into ui_context.referenced for the system prompt. 5.1 Where the agent gets the links Two sources, both cheap: Tool results. Every mutating tool’s result gains one field: url: "<kind>://<id>" (sub-target variants where natural, e.g. ui_storyboard_add_shot → …#shot=<newId>). Backend media tools emit it from persistOutput next to the existing asset_uri. The JSON stays JSON — the model copies the string into its prose when it wants to link. System prompt. buildChatAgentSystemPrompt gets a short section: the scheme, the rule (“when you create or change a resource, link it once in your reply with a human label”), and the anti-rule (don’t link on every sentence; one link per resource per reply). The per-surface agent panels (timeline, storyboard, sketch) inherit the same section. 5.2 Rendering: ResourceChip New component in web/src/components/chat/message/, built from ui_primitives, small enough to sit inline in a sentence: All kinds: kind icon (reuse getToolVisual’s icon/accent map) + the link’s own text as label. No lookup needed to render — the chip is useful even offline. Assets: a small inline thumbnail when the mime is imagey — resolved through the existing resolveUri; hover shows a larger preview. This covers “generated images should be visible in chat” without touching tool result rendering. Optional enrichment: a TanStack Query lookup (existing useAssets/document queries) can replace a stale label with the current name and mark deleted resources (struck-through chip, no navigation). Failure of the lookup never breaks the chip. Same treatment everywhere markdown renders: chat thread, memory sidebar, agent-panel chats — they all go through ChatMarkdown. urlTransform is extended to pass resource-URI schemes through untouched; everything else keeps react-markdown’s default sanitization. 5.3 Navigation: openResource One helper (web/src/lib/chat/openResource.ts): kind → openTab({type, ref: id}) for document kinds; assets open the asset viewer; collection opens the collections panel; thread switches chat threads. Sub-target application reuses the agent bridges’ selection handlers after the tab mounts (getTimelineAgentHandler(id).selectClip(...) etc.). If the editor isn’t mounted yet, the sub-target is applied on bridge registration — a one-shot pending-selection map, mirroring how useAssembleTimeline already sequences open-then-act. 5.4 Inbound: the user links back The composer accepts the same URIs: pasting one renders a chip in the input; an “insert reference” action on each rendered chip (and on the editor tabs’ context menus, later) drops the URI into the composer. On send, the URI simply rides in the message text — the agent already reads text, and the system-prompt section teaches it to parse the scheme. Additionally the client appends the parsed refs to the existing ui_context (extend UiContext with referenced?: UiDocumentRef[]), so the server’s formatUiContext can state it plainly in the system prompt. This makes “make this darker” precise: the message carries storyboard://sb_x#shot=s3, and the agent’s first tool call can be ui_storyboard_select_shot on exactly that shot. 5.5 What this is not Not a card system. Tool results render as they do today. The chip is an inline text element, not a block; there is no coalescing, no preview pipeline, no per-kind summary machinery. Not automatic navigation. The agent linking a resource never opens or focuses it; clicking does. Not a web URL. The scheme is internal to the app. If shareable deep links are wanted later, the same parseResourceUri can back a /#/resource/<kind>/<id> route — out of scope here. 6. Server involvement One additive section in buildChatAgentSystemPrompt (packages/websocket/src/session/chat-prompt.ts). processToolResult already passes string fields through — url survives as-is. Nothing to change. Persistence: URIs live inside ordinary message text. Old threads render chips retroactively wherever an agent happened to emit a URI; no migration. 7. Phasing Phase 1 — render + navigate. parseResourceUri in protocol; urlTransform allowlist; ResourceChip (icon + label, asset thumbnails); openResource without sub-targets; system-prompt section. Agent-emitted links become clickable. Phase 2 — tools hand out links. url field from persistOutput and the mutating ui_* tools (per-bridge helper, one line per return site); sub-target fragments + pending-selection on open. Phase 3 — inbound references. Composer chips, insert-reference action, UiContext.referenced. Phase 4 (optional). Label freshness via query lookup; deep-link route; links in memory summaries. 8. Testing Unit: parseResourceUri/formatResourceUri round-trip incl. malformed input (Vitest, in protocol); ResourceChip per kind and ChatMarkdown integration incl. the urlTransform allowlist (Jest + RTL, next to the existing markdown tests). Eval: the tool-loop suites can assert that mutating-tool results carry a well-formed url, and a prompt-adherence case can check the model links the resource it created (string match on the transcript — no browser). Navigation: a jsdom test that openResource calls openTab with the right {type, ref} per kind, and that a fragment queues a selection until the bridge registers. 9. Open questions Link discipline. Is the system-prompt rule enough to keep models from over-linking, or do we cap chips rendered per message defensively? thread and collection kinds are speculative — include in the union from day one, or start with the six editor surfaces plus asset? Sub-target selection on open touches per-surface bridge code — worth Phase 2, or ship links that open the document only and add fragments when a surface asks for them?--- # Studio: the agentic-video carve-out Source: https://docs.nodetool.ai/agentic-video-product.html Studio: the agentic-video carve-out A niche product on the NodeTool platform: agentic video creation and editing for absolute beginners. One shell, two creation paths (storyboard and script), one finishing surface (the timeline editor), a curated model list, and a credits display instead of provider pricing. Everything below the shell is reused NodeTool. Status: prototype. The shell ships inside the main web app at /studio; carving it into a standalone deployable is a later step (see “From prototype to product”). What the user sees /studio — home. Two cards: Start with a storyboard (describe an idea; the director agent breaks it into shots, renders stills, animates clips) and Start with a script (write or generate lines, cast voices, voice them). Below: recent projects across all three document types. The header always shows the credit balance. /studio/storyboard/:id — the storyboard editor (board + agent panel + generation queue) without the workspace sidebar or tabs. Assemble navigates to the Studio timeline page. /studio/script/:id — the script editor (document pane + cast / assistant dock). Create video in the header assembles the voiced takes into a timeline and navigates there. /studio/timeline/:id — the full timeline editor, the one finishing surface. Preview, tracks, inspector, agent panel and export all come from the platform editor unchanged. What is reused (all of it) Product piece Platform code Storyboard editing + agent web/src/components/storyboard/*, ui_storyboard_* tools, StoryboardStore, server sync Script editing + voicing web/src/components/script/*, ui_script_* tools, ScriptStore, server sync Timeline editing web/src/components/timeline/TimelineEditor, ui_timeline_* tools Storyboard → timeline useAssembleTimeline → buildStoryboardTimeline (@nodetool-ai/timeline) Script → timeline useAssembleScriptTimeline → buildScriptTimeline (@nodetool-ai/timeline) Linked board + script → timeline buildLinkedTimeline (@nodetool-ai/timeline) — shot-aligned picture, one voiceover clip per line, no narration draft clip Agent chat in every editor ChatView + per-document agent panels and bridges Persistence tRPC storyboards / scripts / timeline routers, autosave hooks Cost data prediction ledger (nodetool_predictions) via costs.dashboard Headless QA nodetool timeline validate, storyboard/script render tools, the tool-loop evals, and the script-storyboard-link harness selfcheck (packages/cli/src/harness/registry.ts) the gate runs on diffs touching either surface The Studio module itself is five small files in web/src/studio/ plus four routes in web/src/index.tsx. Nothing under web/src/components/ changed. Curated models web/src/studio/curatedModels.ts is the entire model policy: one language model (screenplay direction + assistants), one still model, one clip model. New Studio storyboards get them stamped on before the first generation, so beginners never meet a model picker; the platform pickers still exist inside the reused editors for users who dig, and validate_workflow’s provider/model check rejects any id the catalogs don’t know. Changing the lineup is editing that file. If the product later needs per-plan lineups (e.g. faster models on the free tier), the same shape can move to a server-delivered config. Credits and plans Server-owned, in packages/models/src/credits.ts (@nodetool-ai/models): 1 credit = $0.01 of provider spend. Spend is never double-booked: the balance is sum(grant ledger) - ceil(prediction spend / 1¢), read straight from the nodetool_predictions rows every provider call already writes. Ledger (nodetool_credit_ledger) holds grants only: the welcome grant, monthly plan accruals, top-ups, adjustments. Grants are keyed so the lazy accrual (run on every status read) is idempotent by primary key — no cron: signup:<userId> once ever, plan:<userId>:<planId>:<YYYY-MM> per month. New users get free credits. The first balance read for a user inserts NODETOOL_SIGNUP_CREDITS (default 500) as a welcome grant, on top of the free plan’s monthly accrual. Raising the amount later reaches users who have not signed up yet; it never re-grants one who already has the row. 0 switches the welcome grant off. Plans (nodetool_user_subscriptions, catalog CREDIT_PLANS): Free 300/mo, Creator 3,000/mo ($12), Pro 10,000/mo ($40). Switching is instant and unbilled; a payment provider integration replaces the topup mutation with a checkout session and writes ledger rows from its webhook. API: trpc.credits.status | setPlan | topup (packages/websocket/src/trpc/routers/credits.ts, schemas in packages/protocol/src/api-schemas/credits.ts). The nodetool provider is what gets metered. NodeTool’s own managed models are a real provider (packages/runtime/src/providers/nodetool-provider.ts, id nodetool): each curated model (NODETOOL_MODELS in @nodetool-ai/protocol) names a delegate provider+model, and the provider runs the delegate on platform-owned keys (NODETOOL_PLATFORM_FAL_KEY, NODETOOL_PLATFORM_ANTHROPIC_KEY) rather than the user’s. Cost is absorbed from the delegate at the delegate’s price, and @nodetool-ai/model-pricing translates nodetool/... ids to the delegate before lookup so estimates are real numbers. Models appear in the pickers only when their platform key is set. The managed provider is cloud-only. nodetool is in CLOUD_ONLY_PROVIDER_IDS (packages/protocol/src/cloud-profile.ts) and the provider index unregisters it whenever the cloud profile is off — a desktop install and a self-hosted server (NODETOOL_NODE_PROFILE=full) both have platform keys they do not own and no account to bill, so the provider would only ever produce an error. Those users bring their own keys, unmetered. The operator picks which models the platform sells. NODETOOL_CREDIT_MODELS is a comma/whitespace-separated list of curated model ids (nodetool/flux-schnell nodetool/kokoro); unset means the whole catalog, the right default for a staging server holding platform keys. A model outside the list is hidden by the provider’s listers, refused by delegateFor before a platform key is used, and refused by the gate before a run starts (MODEL_NOT_AVAILABLE, not BUDGET_EXCEEDED — a full balance does not change the answer). Parsing is packages/config/src/credit-policy.ts; the surviving ids reach clients as spendableModels on credits.status, and Studio’s dropdowns show only those. nodetool/director drives the in-editor assistants and is not user-selectable, so a whitelist that leaves it out turns them off — name it alongside the models you sell. Enforcement follows the provider, not the deployment (packages/websocket/src/credit-gate.ts): a workflow run is gated only on the slice of its estimate whose provider is nodetool (estimateNodetoolSpend → admitCreditRun, next to the application-budget gate); the direct generate_media/generate_text/ transcribe_audio RPCs are gated only when called with provider: "nodetool". BYOK providers are never gated — credits and bring-your-own-key coexist on one server, per user, per call. The gate fails open on its own errors. UI: the header chip reads credits.status and links to /studio/account — balance, usage, plan cards, and the (clearly labeled) test top-up. Still open, in order of value: Per-action estimates. @nodetool-ai/model-pricing (getModelUnitPrice) prices the curated models per unit, so shot cards and the voice-all button can show “≈ 3 credits” before the click. Payments. Stripe (or similar) in front of setPlan/topup; the ledger and gate don’t change — a webhook writes topup rows and flips plan_id, and the balance is read the same way. Direct-RPC metering. Done: generate_media, generate_text and transcribe_audio each write a prediction row on the managed provider (direct.<mode>, direct.text/direct.structured, direct.transcription), so their spend decrements the balance the gate reads. From prototype to product Standalone shell. The Studio routes live in the main bundle today. To ship a separate product surface: a studio.html Vite entry (the app-preview.html pattern) with only the Studio routes, or keep one bundle and gate by hostname/runtime config. The second is cheaper and keeps one deploy pipeline (the GHCR image). Access control. A Studio-only auth mode hides /workspace, the node editor, and settings; the reused editors already work without them. Curated voices. Script casting still offers the full TTS catalog; curate it the same way the image/video models are curated. Known prototype seams. The reused assemble hooks also open a workspace tab (harmless in Studio; the tab is simply there if the user ever visits /workspace). Model stamping re-applies if a user clears all three storyboard models. The storyboard/script pages are desktop-first; the workspace surfaces’ mobile pane-switchers were not carried over.--- # Agents and Automation Source: https://docs.nodetool.ai/agents/ NodeTool is an open-source visual workflow runtime for AI models, tools, agents, media generation, and data processing. This guide is the entry point for software agents and automated integrations. Start here Install NodeTool and start a server with npm run dev:server. Find node schemas or fetch the node catalog. Create and validate a workflow. Run and inspect a workflow with the CLI or API. Machine-readable resources Documentation index Complete documentation Node catalog Workflow API WebSocket API Verify a workflow Run the cheap structural check before execution: npm run dev:nodetool -- validate workflow.json After a valid result, execute it and inspect its output: npm run dev:nodetool -- debug workflow.json The command prints the workflow status, output values, errors, and job logs. A successful run has a completed status and no execution errors.--- # AI Timeline Editing Source: https://docs.nodetool.ai/ai-timeline-editing.html A NodeTool timeline keeps editorial timing separate from generated media. You can compare variations, repair a shot, and adapt a cut for another format without rebuilding the sequence. Compare takes without moving the edit Every generated result is a take on its clip. A take records the generated asset and its generation context. Selecting another take changes the media in the clip, not its start time, duration, track, effects, or linked media. Use takes when you want to compare alternate performances, prompts, or model results in the same cut. Generate a clip, then create another take with Regenerate as new clip or generate again. Select the clip, open History in the Inspector, and choose Preview Candidate to audition a take in place. Choose Use take on the take that belongs in the cut. Keep useful alternatives or remove candidates you no longer need. The active take is the one used by preview, render, and interchange export. If a generated take fails, the previously active take remains in place. Track a subject A media track stores a subject’s position over source time. Bind a clip’s position, scale, or mask to that track when a title, graphic, or mask must follow a moving subject. Create the track with the agent’s track_object tool (it needs a provider that implements track_object, and a box around the subject at the start of the range), then attach the dependent clip. The Inspector’s Tracking section lists a clip’s tracks. Its Follow object section binds the clip to a track in Position or Position + scale mode, with offsets and smoothing. The Subject matte section uses a track for a mask. The binding follows trims and clip movement because it uses source time rather than the timeline’s absolute position. A changed source asset makes its media tracks stale. Re-run tracking before relying on a stale track. Adapt a cut for another frame Choose Adapt format in the timeline’s overflow menu (the three-dot button in the tab row). Pick one or more target formats (9:16, 1:1, 4:5), a strategy, and a safe margin of 0 to 30 percent. Each format becomes a new sequence that shares the source media. The source sequence stays unchanged. Strategy Behavior Center Crops each clip around the center. Smart Reframe Uses subject tracks or authored framing. Unavailable until every visual clip has a current subject track or authored framing. Follow selected object Follows one ready subject track. Unavailable until a track exists. Review the derived cut, then open Smart Reframe in the Inspector for a clip. Choose its subject (auto, center, or a track) and add manual framing keys where the automatic crop needs direction. Framing is stored in source time, so it follows trims and moves. The section flags automatic framing as stale when the source asset changes. Preview and render use the same framing data. Repair or extend a shot Generative timeline edits (the AI Edit section for a video clip, or the agent’s ui_timeline_generatively_edit_clip tool) create a candidate take. They do not replace the active media until you select the result. Available operations include: Extend a shot at its start or end (extend). Replace a source range (replace_range). Remove or replace a tracked object (remove_object, replace_object). Restyle a shot (restyle) or regenerate it (regenerate). The clip must be a video clip with a current asset and a positive duration. Object operations require a ready media track for the active source. Before running an edit, NodeTool checks the requested operation against the selected model’s capabilities. If the model cannot perform the edit, choose another model or change the operation. Hand off a sequence exportInterchange in @nodetool-ai/timeline writes an editable project document for another editor. No editor menu, CLI command, or agent tool calls it yet. Use it from code until one does. Target Output Carries Final Cut Pro FCPXML sequence format, active media, timing, tracks, and markers Premiere Pro Legacy FCP XML sequence format, active media, audio and video tracks, timing, and markers The export report names missing media and edits that need baking, including effects, transforms, masks, retimes, fades, captions, and unsupported clip types. Resolve or bake those items before delivery. NodeTool never reports an incomplete editable export as lossless. Work with the agent The agent can read each clip’s active take id through ui_timeline_get_state, select a take with ui_timeline_apply_take, create a media track with track_object, adapt a sequence format with ui_timeline_retarget_format, set framing with ui_timeline_set_reframe_subject and ui_timeline_add_reframe_keyframe, and request a generative edit with ui_timeline_generatively_edit_clip. Give it the editorial intent and the target clip. For example: “Track the product in the hero shot, attach the price graphic, then make a vertical version of this cut.” Check the resulting takes, tracks, and framing before export. Related features Video Editor — timeline, playback, clip editing, and MP4 export Workflow Editor — edit a workflow bound to a generated clip Asset Management — organize imported media--- # AI Video Production Source: https://docs.nodetool.ai/ai-video-production Turn a product brief into an editable video without leaving NodeTool. Start with a beat plan, a storyboard, or a script. Add the references that should guide the images, review every shot before generation, then choose which generated takes enter the cut. Generated takes are previews until you set one as current, choose Use take, or approve a draft. You can compare options without changing the current edit. Before you start Connect the models needed for your route: A language model plans beats, scenes, and shots. An image model renders storyboard stills. A video model generates clips. Reference-based shots need a model that supports reference-to-video input. An on-camera performance needs a model that supports audio-driven or lip-sync video, plus prepared speech audio and source footage of the performer. See Models and providers for setup. The Quick Start explains the language, image, and video model roles and how provider billing works. Add product photos, character images, logos, and style frames to the Asset Explorer before you begin. See Asset Management for import and organization. Choose where to start Video: plan beats and land on the timeline Choose Video when you want a short ad, explainer, or social clip organized as timed beats. Describe the video in What’s the video? Open Creative context (optional). Add the product name, audience, objective, tone, approved claims, and claims to avoid. Choose Add product reference, Add character reference, or Add style reference. After adding an asset, you can change its role to product, character, location, or style. Continue to Review your beats. Edit the direction, duration, transition, and voiceover for each beat. Set the production choices described in Review the production plan, then continue to the look and generation step. The result opens in the timeline, where each beat remains a separate clip. Storyboard: direct each shot before animation Choose Storyboard when reference consistency and shot-by-shot control matter more than speed. Describe the story in What’s your story? and add creative context and references. Review Your screenplay. Edit scene headings, action, dialogue, and shot duration before rendering. Set the editorial purpose, visual treatment, speech mode, production direction, and requested takes for each shot. In Entities, assign reusable characters and other references to the shots that need them. In Look, choose the aspect ratio, art style, still model, and video model. Add your own style can build a look from one to three reference images. Render stills, approve the framing you want, then generate clips. A product close-up needs a product reference. A reference-based character or lifestyle shot needs the matching character, product, location, or style asset and a video model that accepts references. Script: write and cast the words first Choose Script when dialogue, narration, or voice casting should drive the video. Write or import the script. In Production context, add the product, audience, objective, tone, and claim rules. These notes travel with the script without changing its words. Assign speakers and voices, then generate the speech takes you want to use. Create a storyboard from the script when you want visual shots. The linked storyboard keeps the script lines, speakers, entities, voices, and selected speech takes. Add or confirm visual references in the storyboard before generating clips. Edit linked words and voices in Script. The storyboard treats them as source material instead of silently rewriting them. Review the production plan The Video and Storyboard review screens add the same production choices to each beat or shot: Editorial purpose identifies the job of the section, such as Hook, Demonstration, Proof, or Call to action. Visual treatment chooses Actor to camera, Product close-up, Lifestyle B-roll, or Generated scene. Speech mode chooses None or Off-camera. On-camera appears in the list but is disabled in the guided editors. Production direction adds local camera or performance notes. Requested takes asks for one, two, or three alternatives. Review these choices before continuing. If you change the brief, references, or production choices later, NodeTool asks you to review the plan again before it generates media. Narration and on-camera speech Use Off-camera for narration or voiceover that plays over a product shot, B-roll, or another scene. The speaker does not need to appear to say the line. This is the usual choice for a product demo assembled in the guided editors. The guided Video and Storyboard editors do not support On-camera. They show it as unavailable and block generation with “On-camera speech is unavailable in this guided flow” if a shot carries it. Choose Off-camera or None there. The agent can run an on-camera performance when the visible performer must match the recorded line. That route (audio_driven_performance) needs all of the following: the exact speech take or recorded audio, a character entity with a character reference, a source face video for the performance, an audio-driven or lip-sync model available through a connected provider, enough time in the shot for the complete speech take. NodeTool does not turn an unsupported on-camera request into narration. If the agent route cannot resolve the required audio, character, source video, or model capability, the request is rejected with the missing requirement. Use off-camera speech for that shot, or prepare the missing inputs first. Speech must fit inside its beat or shot. Shorten the line, choose a shorter speech take, or lengthen the section when NodeTool reports a timing conflict. Generate and compare takes Generation creates the requested one to three options for each destination. Options can finish in any order without changing their take numbers. In Storyboard: Open View takes on a shot. Select a Preview chip to audition that clip without replacing the accepted clip. Choose Set as current clip beside the take you want. Stills have the same choice as Set as current still. In the timeline: Select a clip and open History in the Inspector. Choose Preview Candidate to audition an option in place. Choose Use take to make that option active for the selected clip. The agent can also collect one preview for each destination into a draft. Ask it to show the draft first, then tell it to Use draft when the selection is right. NodeTool checks the complete draft before applying it, so a failed or outdated choice leaves the timeline unchanged. Accepting a take changes only its destination. It does not replace other clips, rewrite the source script, or modify the character or product entity. Other generated options remain available until you remove them. Retry failed generations A failed option does not discard successful takes from the same run. Use Retry N failed in the Storyboard toolbar to retry failed shots. Use Retry N failed above the timeline to retry failed generated clips. Retry a single destination when only one take needs attention. The error appears on the affected shot or clip. Common causes include a missing reference, a model without the required video capability, missing performance audio, a missing character or source face video, and speech that is longer than the target slot. Example: a 15-second vertical product ad This example makes a three-beat social ad for a reusable water bottle. Add a clean product photo, a lifestyle photo of the recurring character, and two warm daylight style frames to the Asset Explorer. Start Video and choose a 15-second vertical social format. Enter: “A quick product ad for the Northline bottle, aimed at commuters who want cold water all day.” In Creative context, add the product name, audience, tone, the approved claim “keeps drinks cold for 24 hours,” and any claim the ad must avoid. Attach the product, character, and style references. Review three five-second beats: Hook: Lifestyle B-roll of the character leaving home, with off-camera narration. Demonstration: Product close-up showing the cap and condensation. Call to action: Product and character together with the final line. Request two takes for the product close-up and one take for the other beats. Choose a reference-capable video model and generate. Preview both close-up options. Choose Set as current clip (Storyboard) or Use take (timeline) for the better one. Retry only the failed beat if a generation fails. Continue in the timeline. Trim the accepted clips, adjust audio, add captions, and export the vertical cut. To make the final line an on-camera performance, first generate or record its speech in Script and prepare a source face video for the character. Then use a connected audio-driven model. If those inputs or that model route are missing, keep the line off-camera. Finish the cut Accepted takes are ordinary timeline media. You can trim and reorder clips, mix narration and music, add captions and transitions, and export without changing the unused candidates. See the Video Editor for timeline controls and AI Timeline Editing for generation history and local media edits.--- # api.md Source: https://docs.nodetool.ai/api This page has moved. Please go to API Reference.--- # API Reference Source: https://docs.nodetool.ai/api-reference.html Server Architecture NodeTool runs a single Fastify HTTP + WebSocket server (@nodetool-ai/websocket — packages/websocket/src/server.ts). The same process serves: REST routes under /api/* (workflows, assets, storage, and the surfaces in the matrix below). Jobs, models, and settings are tRPC procedures, not REST. See tRPC Procedures. OpenAI-compatible /v1/chat/completions and /v1/models. WebSocket endpoints for workflow execution, chat, the browser extension, and downloads. An MCP endpoint at /mcp and its OAuth routes. In production /mcp registers only with NODETOOL_ENABLE_MCP=1. Health and liveness probes. The built web app, when the server finds one, for every GET outside /api, /ws, /v1, /trpc, and /mcp. Start it with nodetool serve (default 127.0.0.1:7777). serve accepts only --host and --port — there is no --mode flag. Endpoint Matrix For detailed schemas, see Chat API and Workflow API. Area Path Method / Protocol Auth Streaming Notes Models /v1/models GET Session auth no OpenAI-compatible model listing. A fixed list, not a live catalog Chat /v1/chat/completions POST Session auth SSE when "stream": true OpenAI-compatible chat; SSE or single JSON Workflows /api/workflows GET, POST Session auth no GET lists the caller’s workflows with graphs (limit, cursor, run_mode, project_id). POST creates one from name and graph Workflows /api/workflows/{id} GET, PUT, DELETE Session auth no Read, save, or delete one workflow. PUT creates it when the id is new and takes expected_updated_at for optimistic saves. DELETE answers 204 Workflows /api/workflows/{id}/run POST Session auth no Run a workflow once, return final outputs as one JSON response Workflows /api/workflows/{id}/debug POST Session auth no Same body as run. Returns the debug report: per-node summary and a verdict Workflows /api/workflows/names GET Session auth no {id: name} for the caller’s workflows (up to 1000) Workflows /api/workflows/tools GET Session auth no Workflows saved with run_mode: "tool", as {name, tool_name, description} Workflows /api/workflows/{id}/dsl-export GET Session auth no Graph as TypeScript DSL source (text/plain) Workflows /api/workflows/{id}/export-bundle GET Session auth no One workflow and its assets as a .nodetool zip Workflows /api/workflows/export-bundle POST Session auth no Several workflows in one .nodetool zip, by workflow_ids Workflows /api/workflows/import-bundle POST Session auth no Import a .nodetool zip into the caller’s library Workflows /api/debug/sessions/{id} GET Session auth no State of an interactive run: the escalation it is parked on, or its final report Workflows /api/debug/sessions/{id}/verdict POST Session auth no Answer the parked escalation, then wait for the next one or the final report Workflows /api/debug/sessions/{id}/cancel POST Session auth no Cancel the run and return its final report SDK /api/sdk/v1/workflows/{id}/interface GET Session auth no One workflow’s input and output pins; ?version=1 is required Workflows /api/workflows/public GET none no Workflows the owner marked access: "public" Workflows /api/workflows/public/{id} GET none no One public workflow; 404 when it is not public Examples /api/workflows/examples GET none no Shipped example templates — metadata only, graph is empty Examples /api/workflows/examples/search GET none no Same list filtered by ?query= over name, description, tags Examples /api/workflows/examples/thumbnails/{filename} GET none no Example thumbnail; .jpg and .png only Examples /api/workflows/examples/{package}/{example} GET none no One example with its full graph, unlike the list; 404 when the package has no example by that name Triggers /api/webhooks/{token} POST x-webhook-secret header (no session) no Deliver an event to a webhook trigger registration; wakes the workflow without waiting for the next poll Integrations /api/integrations/{provider}/link/start POST NODETOOL_INTEGRATION_TOKEN bearer (no session) no Mint a one-time link code and the URL that redeems it; 10-minute TTL Integrations /api/integrations/{provider}/link/complete POST NODETOOL_INTEGRATION_TOKEN bearer (no session) no Redeem a link code, binding the external account to a NodeTool user Integrations /api/integrations/{provider}/token POST NODETOOL_INTEGRATION_TOKEN bearer (no session) no Exchange a linked external id for a one-hour delegated user token; 409 in local single-user mode Integrations /api/integrations/{provider}/link DELETE NODETOOL_INTEGRATION_TOKEN bearer (no session) no Unlink an external account; {"unlinked": false} when it was not linked MCP OAuth /.well-known/oauth-protected-resource GET none no RFC 9728 resource metadata for /mcp; 404 unless the flow is enabled. The path-inserted form …/mcp returns the same document MCP OAuth /.well-known/oauth-authorization-server GET none no RFC 8414 authorization-server metadata — the endpoint URLs a client discovers; 404 unless the flow is enabled MCP OAuth /oauth/authorize GET none (PKCE) no Park an authorization request and 302 to the consent page /oauth/consent?request_id=… MCP OAuth /oauth/token POST none (PKCE) no Exchange an authorization code or refresh token for an nta_ access token MCP OAuth /oauth/register POST none no RFC 7591 dynamic client registration MCP OAuth /oauth/revoke POST none no RFC 7009 revocation of an access or refresh token Nodes /api/nodes/metadata GET none no The node registry the editor loads at boot; slim summaries by default, one node’s full metadata with ?node_type= Workspaces /api/workspaces/{id}/download/{path} GET Session auth streaming One file out of a workspace as an attachment; 403 in production unless the workspace is the server-managed one Assets /api/assets POST Session auth no Create an asset. Multipart with a file part and an optional json part for name, content_type, parent_id, workflow_id, node_id, job_id, project_id, metadata. Cloud backends upload through assets.createUpload instead, see Uploading an Asset Assets /api/assets/{id}/extract-audio POST Session auth no Extract a video asset’s audio track into a new WAV asset Assets /api/assets/{id}/peaks GET Session auth no Waveform peaks of an audio or video asset, computed with ffmpeg and cached on disk Assets /api/assets/packages/{package}/{file} GET none streaming Bytes behind a package:// ref, from a node pack’s assets directory Assets /api/assets/packages GET none no Stub — always {"assets": [], "next": null}; there is no package listing Assets /api/assets/packages/{package} GET none no Stub — same empty page. Fetch a package’s files by name, not by listing Assets /api/assets/download POST Session auth no Bulk ZIP download; 501 on this server Apps /api/applications GET, POST Session auth no List the caller’s apps (project_id filter) or create one Apps /api/applications/{id} GET, PUT, PATCH, DELETE Session auth no Read, update, or delete one app Apps /api/applications/{id}/released-document GET Session auth no The snapshot a published app should run, with each operation’s pinned graph; null when nothing is published Apps /api/applications/{id}/export-bundle GET Session auth no One app and the full graph of every workflow it binds, as a downloadable ApplicationBundle Apps /api/applications/{id}/export-project GET Session auth no One app as a standalone JavaScript project (Vite UI plus a Node server that runs its workflows and scripts), as a .zip Apps /api/applications/import-bundle POST Session auth no Create an app, and the workflows it binds, from an ApplicationBundle Apps /api/applications/build POST Session auth no Build a mini app from a prompt or a pinned spec; returns the BuildReport. poll: true returns a session id instead Apps /api/applications/debug POST Session auth no Simulate a saved app by application_id, or a draft posted inline as document; returns the compacted debug report Apps /api/applications/examples GET Session auth no The shipped example apps — slug, name, description, workflow names, operation count, thumbnail URL Apps /api/applications/examples/{slug} GET Session auth no One example’s full ApplicationBundle; 404 when the slug names nothing shipped Apps /api/applications/examples/{slug}/install POST Session auth no Install an example into the caller’s library, creating the workflows it binds Apps /api/applications/examples/{slug}/scripts/{key}/run POST Session auth NDJSON with Accept: application/x-ndjson Run a JS script an example bundle carries, without installing the example Storyboards /api/storyboards/{id}/export-zip GET Session auth streaming One board as a zip of Markdown plus its stills and clips; 404 when the caller does not own it Games /api/games/{id}/export-zip GET Session auth streaming A game’s published build as a zip: target=web (default) or target=tauri; 404 when the caller does not own it Games /api/games/{id}/build-executable POST Session auth NDJSON Compile the desktop app project with cargo on this machine; 403 on a cloud profile, 503 without cargo Timelines /api/timelines/{id}/export-zip GET Session auth streaming One sequence and the bytes of every asset its clips name, as a zip; 404 when the caller does not own it Timelines /api/timelines/{id}/export-premiere GET Session auth streaming One sequence as a Premiere Pro XML (xmeml) plus its media and an export report, as a zip; 404 when the caller does not own it Timelines /api/timelines/{id}/export-media GET Session auth streaming Every clip’s media file plus clips.csv, as a zip; 404 when the caller does not own it Timelines /api/timelines/import-zip POST Session auth no Multipart upload of such a zip, or of a zip holding a Premiere Pro XML and its media; stores the assets and creates a new timeline pointing at them Providers /api/fal/credits GET Session auth no The server’s fal.ai account balance; 204 when no FAL_API_KEY is configured Providers /api/fal/pricing GET Session auth no Unit price per fal.ai endpoint, one or more ?endpoint_id=; cached an hour Providers /api/fal/pricing/estimate POST Session auth no What a fal.ai endpoint costs for a given quantity; 204 when no FAL_API_KEY is configured Providers /api/kie/credits GET Session auth no The server’s kie.ai credit balance; 204 when no KIE_API_KEY is configured Providers /api/kie/pricing GET Session auth no Credit price per kie.ai model, one or more ?model_id=; cached an hour Providers /api/kie/resolve-dynamic-schema POST Session auth no Pasted kie.ai model docs to a node’s dynamic properties, inputs, and outputs Providers /api/comfy/resolve-workflow POST Session auth no A ComfyUI API-format workflow (workflow) or exported PNG (png_base64) to the Comfy runners’ dynamic inputs, outputs, and exposable params JS Scripts /api/js-scripts/{id}/run POST Session auth no Run a saved JS script document in the sandbox and return its outputs SDK /api/sdk/v1/capabilities GET Session auth no What this server supports — profiles, encodings, execution options, limits SDK /api/sdk/v1/node-types GET Session auth no Paged inventory of the pin types in the registry and which nodes use them SDK /api/sdk/v1/workflows GET Session auth no Workflow summaries with the revision an interface was read at SDK /api/sdk/v1/workflow-interfaces POST Session auth no Up to 100 workflows’ interfaces in one call, with per-workflow errors SDK /api/sdk/v1/preflight POST Session auth no Whether a workflow is runnable, what it needs, and what it will cost SDK /api/sdk/v1/assets/temporary POST Session auth no Store one execution input in temporary storage; creates no asset row SDK /api/sdk/v1/models GET Session auth no Paged model catalog with per-model availability and the wire value a node property takes SDK /api/sdk/v1/model-downloads GET Session auth no Snapshot of this caller’s model downloads, running and finished SDK /api/sdk/v1/model-downloads POST Session auth no Start a model download; 202 with the operation’s first state SDK /api/sdk/v1/model-downloads/cancel POST Session auth no Cancel one download by operation_id Extension /api/extension/download GET Session auth no The built Chrome extension as a zip; 404 when the server has no build Workflow WS /ws WebSocket Session auth. Bearer header or api_key query yes Workflow execution, chat, job control, live editor tools, and live updates (MessagePack or JSON) Download WS /ws/download WebSocket Session auth. Off in production yes Model/file downloads Storage /api/storage/* HEAD/GET Session auth streaming for GET Asset bytes at <userId>/<assetId>.<ext>, scoped to the caller. Read-only: writes and deletes go through the asset API (tRPC assets.delete removes an asset’s stored objects). storage.signUrl is the only tRPC storage procedure Account /api/account/export GET An authenticated identity, even in Local mode no Everything this server holds about the caller, as one JSON download Account /api/account DELETE An authenticated identity, even in Local mode no Erase the caller’s data. The body must be {"confirm": "DELETE MY ACCOUNT"} Files /api/files/local GET, HEAD Session auth streaming One local file by absolute path, with Range support, for file:// previews. 403 when NODETOOL_ENV=production OAuth /api/oauth/{provider}/* see Provider Sign-in Session auth, except the two callbacks no Hugging Face, GitHub, OpenAI, Claude, and Google credential flows Collections /api/collections/{name}/index POST Session auth no Multipart upload of one text file into a vector collection. Other collection operations are tRPC Documents /api/documents/extract-text POST Session auth no Extract text from an uploaded PDF or DOCX (25 MB cap). Stores nothing Apps /api/apps/{token} GET none (the token is the credential) no A deployed mini app’s released document. Production only Apps /api/apps/{token}/session POST none (the token is the credential) no Mint the short-lived nda_ session a deployed app’s visitor runs on. Production only Workflows /api/shared-workflows/{token} GET none (the token is the credential) no A workflow’s name, description, tags, and graph behind a public share link Timelines /api/timelines/{id}/track-object POST Session auth no Track a subject through a clip with the track_object capability Timelines /api/timelines/{id}/bake-audio-animation POST Session auth no Turn an audio clip’s loudness or beats into keyframes on another clip Timelines /api/timelines/{id}/isolate-subject POST Session auth no Cut a clip’s subject out with a matte model Timelines /api/timelines/animations/bake POST Session auth no Run a custom animation’s JavaScript once and return keyframe curves Timelines /api/assets/packages/timeline/fonts/{file} and .../google/{family}/css, .../google/{slug}/{file} GET none streaming Bundled timeline fonts, and Google Fonts CSS and files cached on the server Providers /api/kie/webhook POST none no kie.ai task callback. Needs a taskId, answers {"status":"accepted"} Providers /api/providers/fal/webhook/{token} POST Ed25519 signature headers (no session) no fal.ai queue callback, rate-limited to 120 per minute Providers /api/providers/atlascloud/webhook POST Ed25519 signature header (no session) no AtlasCloud prediction callback, rate-limited to 120 per minute Sandbox /api/sandbox-modules/* GET Session auth no Guest module source for the browser sandbox, by module id MCP /mcp GET, POST, DELETE Session auth, or an ntk_ or nta_ token SSE MCP over HTTP. See MCP in production Config /api/config GET none no How this server is configured, for a client that has not signed in yet: auth mode, Supabase URL and anon key, Google Workspace scopes, version Admin /admin/secrets/import POST Session auth no Stub — always 501. Bulk secret import is not part of the standalone server Health /health GET none no JSON: {status, timestamp, uptime, turns, jobs, services}. 503 when the database check fails or the server is draining Health /api/health GET none no JSON: {version, uptime} Liveness /ready GET none no Always 200 with {status:"ok"} Session auth means the request must carry a credential unless Local mode trusts the caller. Rows marked none skip session auth. The server does not read an AUTH_PROVIDER variable. The mode comes from SUPABASE_URL and SUPABASE_KEY, as described in Authentication. The SDK discovery routes (capabilities, models, node-types, workflows, workflow-interfaces) also skip session auth unless the server enforces auth or the SDK authentication flag is set. Authentication and Headers The server picks one of two modes at startup. When both SUPABASE_URL and SUPABASE_KEY are set it runs in Supabase mode and validates a Supabase JWT on every request that is not public. Otherwise it runs in Local mode. Loopback callers are trusted as user 1 with no token, and other callers need an access token or an address listed in NODETOOL_TRUST_LOCAL_NETWORKS. The full rules are in Authentication. Credentials a client can present: Credential Prefix Where it works Supabase JWT none Everything, in Supabase mode Access token minted in Settings → MCP ntk_ Everything, in both modes MCP OAuth access token nta_ /mcp only Deployed app session nda_ /ws only, confined to one app Delegated token from a messaging bridge ndt_ Everything, for the linked user. See Linking an External Messaging Account How to include credentials HTTP requests: Authorization: Bearer <token> header on all non-public routes WebSocket: Authorization: Bearer <token> header (preferred) or api_key query parameter. The query parameter is read on WebSocket upgrades only SSE streams (/v1/chat/completions): Authorization: Bearer <token> Requests without a valid credential get 401. The public routes are listed in the matrix with none in the Auth column. Local development: When running locally with the default config, loopback requests need no token. You can omit the Authorization header entirely. See Authentication for full token handling rules. Streaming Behavior /v1/chat/completions uses OpenAI-style SSE when stream is true; otherwise it returns a single JSON response. POST /api/workflows/{id}/run does not stream — it runs the workflow to completion and returns one JSON response. The /ws WebSocket streams workflow/job events (job_update, node_update, node_progress, output_update, chunk, …) and chat tokens/tool calls. See the WebSocket API for the full protocol. Storage routes stream file contents for large assets. tRPC Procedures The web app, the CLI, and the mobile app talk to the server over tRPC at /trpc. Procedures are plain JSON with no transformer. A query is GET /trpc/<router>.<procedure>?input=<url-encoded JSON>, and a mutation is POST /trpc/<router>.<procedure> with the input as the JSON body. Batching joins paths with commas (/trpc/a.x,b.y?batch=1, one input per index) and is capped at 20 calls per batch. Because long batched inputs overflow URLs behind reverse proxies, the server also accepts POST for queries. A result is {"result": {"data": ...}}. An error carries the tRPC code and message, plus data.apiCode (an API error code such as NOT_FOUND or INVALID_INPUT) and data.zodError (field errors, or null). Every procedure needs a signed-in user except healthz, extension.status, nodes.replicateStatus, workflows.public.list, and workflows.public.get. Procedures by router. Nested routers appear as a dotted path. Router Queries Mutations agentAccess mcpConnection, listTokens, getOauthRequest, listOauthGrants createToken, revokeToken, approveOauthRequest, denyOauthRequest, revokeOauthGrant applications list, get, versions, released, releasedDocument, deployment, budget, usage, invocations create, update, delete, publish, deploy, undeploy, setBudget, beginInvocation, settleInvocation, release assets list, get, externalImportConfig, recursive, search createUpload, finalizeUpload, createExternal, relinkExternal, ensureProxy, update, delete codeGen — generate collections list create, update, delete costs dashboard — credits status setPlan, topup customProviders list save, delete, test documents index — extension status installHost externalMcp list save, delete, probe files list createFolder fonts list — games list, examples, get, getDraft, draftChanges, draftBeforeChange, revisions create, installExample, saveDraft, previewAuthoring, applyAuthoring, publish, restore, restoreDraft, installAsset, installCandidate integrations list, describeLinkCode createLinkCode, confirmLink, unlink jobs list, get, triggersRunning cancel, triggerStart, triggerStop jsScripts list, palette, get, documentVersions.list, documentVersions.get create, update, delete, documentVersions.create, documentVersions.restore, documentVersions.delete mcpConfig status install, uninstall memories list, search delete messages list — models providers, recommended, recommendedImageTextToImage, recommendedImageImageToImage, recommendedLanguageEmbedding, recommendedAsr, recommendedTts, recommendedVideoTextToVideo, recommendedVideoImageToVideo, availableForKind, all, huggingfaceList, huggingfaceSearch, huggingfaceHubSearch, huggingfaceByType, transformersJsByType, transformersJsRecommended, ollama, llmByProvider, mediaOptions, imageByProvider, tts, ttsByProvider, music, musicByProvider, audioToAudio, audioToAudioByProvider, asr, asrByProvider, video, videoByProvider, embeddingByProvider huggingfaceDelete, ollamaDelete, huggingfaceCacheStatus, pullOllamaModel nodes replicateStatus, list, get — packs list, runtimeStatuses, getTrust, sandboxModules, sandboxPackageDocs, listBuiltins setTrust, setBuiltinEnabled, reload projects list, archived, summaries, get, documents, restoreTabs, unassigned thread, create, update, archive, restore, delete, assignDocument, copyDocument resources list, read create, update, delete scripts list, get create, update, delete segmentation — segment settings secrets.list, secrets.get, history.get, list secrets.upsert, secrets.delete, secrets.validate, history.update, history.cleanup, history.compact, update sketch list, get, versions.list, documentVersions.list, documentVersions.get create, update, delete, versions.append, versions.setFavorite, versions.delete, documentVersions.create, documentVersions.restore, documentVersions.delete, layers.create, layers.duplicate skills list, get create, update, delete storage signUrl — storyboards list, get, examples create, update, delete, installExample, stylePresets threads list, get create, update, delete, summarize timeline examples, list, get, versions.list, versions.get, code.get installExample, create, update, delete, versions.create, versions.restore, versions.delete, code.set, code.rebake, code.detach, clips.create triggers listByWorkflow fire users list create, remove, resetToken worker profiles.list, instances.list, apiKeyStatus, health profiles.create, profiles.delete, provision, stop, resume, terminate, stopAll, reconcile, attach, detach workflows list, get, examples, recipes, public.list, public.get, terminalOutputs, versions.list, sharing.get, sharing.sharedWithMe create, update, delete, autosave, versions.create, versions.restore, versions.delete, sharing.createLink, sharing.revokeLink, sharing.setRole, sharing.removeCollaborator, sharing.accept, sharing.duplicatePublic workspace list, listFiles, readFile create, update, delete, writeFile healthz is a root-level query that returns {"ok": true}. Input and output shapes are the Zod schemas in packages/protocol/src/api-schemas/ and in the router files under packages/websocket/src/trpc/routers/. The AppRouter type exported by @nodetool-ai/websocket/trpc gives a typed client every shape. Headless Mode: Running Workflows via CLI/API NodeTool can run entirely without the UI—perfect for automation, CI/CD pipelines, and programmatic integrations. This section shows how to execute workflows from the command line or via HTTP requests. Quick Start: Run a Workflow via cURL # Run a workflow and get results (non-streaming) curl -X POST "http://localhost:7777/api/workflows/YOUR_WORKFLOW_ID/run" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{ "params": { "prompt": "A cyberpunk cityscape at sunset", "style": "photorealistic" } }' Response: { "job_id": "job_abc123", "workflow_id": "YOUR_WORKFLOW_ID", "status": "completed", "outputs": { "image": { "type": "image", "uri": "http://localhost:7777/api/storage/assets/abc123.png" }, "caption": "Generated image of a cyberpunk cityscape..." }, "error": null, "message_count": 12, "background": false } outputs is an object keyed by output-node name. The route does not stream — for real-time progress, run the workflow over the WebSocket API. Interactive Runs: Answering a Failed Node By default a node that throws ends the run. Post "interactive": true to /api/workflows/{id}/run or /api/workflows/{id}/debug and the failure comes back to you instead: the run parks on the node and the response returns at once with the escalation. Your client decides what happens next, standing where the --supervise LLM supervisor otherwise would. Reach for it when the caller (an agent, a test harness, a batch script) is the one that knows whether a given failure should be retried, skipped, or fatal. curl -X POST "http://localhost:7777/api/workflows/YOUR_WORKFLOW_ID/run" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"interactive": true}' { "status": "escalated", "session_id": "8629a9bc-052f-4af9-96fd-56779be0d94e", "job_id": "180129ebe0864d7ea5f70c1de80b22af", "workflow_id": "YOUR_WORKFLOW_ID", "escalation_id": "esc-1", "escalation": { "nodeId": "work", "nodeType": "nodetool.code.Code", "correlationLineage": [], "invocationKey": "", "allowedActions": ["skip", "fail"], "detail": "boom 42", "inputs": { "code": "throw new Error(\"boom 42\");" }, "declaredOutputs": {}, "attempt": 1, "spentCostUsd": 0, "createdAssets": false, "retrySafe": false, "emitted": false }, "resolve": "POST /api/debug/sessions/8629a9bc-052f-4af9-96fd-56779be0d94e/verdict with {\"escalation_id\": \"esc-1\", \"verdict\": {\"action\": ...}} — allowed actions: skip, fail. The run is parked on this node until a verdict arrives; an unanswered escalation fails closed on the decision timeout." } inputs, detail, and candidateOutput are redacted and truncated before they leave the kernel. allowedActions is computed per invocation and enforced there too, so a verdict outside the list is refused however it was produced. retry appears only when the invocation is replayable — retrySafe, nothing spent, no assets created. substitute appears only when the node produced a malformed value a validator can check. A node reading a stream gets end_stream and fail and nothing else; skip and fail are always available. Answer on the session. escalation_id identifies the escalation you are answering, and verdict.action is one of retry, substitute (with outputs), skip, end_stream, or fail: curl -X POST "http://localhost:7777/api/debug/sessions/SESSION_ID/verdict" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"escalation_id": "esc-1", "verdict": {"action": "skip"}}' The call blocks until the run escalates again, in the same "status": "escalated" shape, or finishes. A finished run returns the ordinary run payload with session_id added: { "job_id": "180129ebe0864d7ea5f70c1de80b22af", "workflow_id": "YOUR_WORKFLOW_ID", "status": "completed", "outputs": { "out": [] }, "error": null, "message_count": 9, "background": false, "session_id": "8629a9bc-052f-4af9-96fd-56779be0d94e" } Two more calls round out the session: # Where is the run right now? Never blocks. curl "http://localhost:7777/api/debug/sessions/SESSION_ID" \ -H "Authorization: Bearer YOUR_TOKEN" # Stop it. Returns the final report with status "cancelled". curl -X POST "http://localhost:7777/api/debug/sessions/SESSION_ID/cancel" \ -H "Authorization: Bearer YOUR_TOKEN" Sessions belong to the user who started the run, so an unknown, foreign, or expired id answers 404 with {"detail": "Debug session not found"} rather than 403. A verdict whose action is not one of the five is a 400. Every boundary resolves as fail, including an escalation nobody answers: the decision timeout is ten minutes by default. Three optional fields in the run body move the bounds, each clamped server-side — decision_timeout_ms (up to 30 minutes), max_decisions (default 10, up to 100), and max_retries_per_node (default 2, up to 20). A settled session stays readable for ten minutes. Agents reach the same machinery through the run_workflow and debug_workflow tools with interactive: true, answering with resolve_workflow_escalation. See Workflow Debugging and the supervisor design for the verdict vocabulary. Triggering a Workflow by Webhook A workflow whose graph carries a WebhookTrigger node gets a registration with its own path token and shared secret. POST /api/webhooks/{token} is where an outside system delivers the event, and the route sits on the public allowlist — there is no session, so the x-webhook-secret header is the only thing standing between the caller and the run. curl -X POST "http://localhost:7777/api/webhooks/YOUR_TOKEN" \ -H "x-webhook-secret: YOUR_SECRET" \ -H "Content-Type: application/json" \ -d '{"order_id": 41, "status": "paid"}' { "status": "accepted", "input_id": "webhook:YOUR_TOKEN:9f2c…", "duplicate": false } The trigger node receives {body, headers, query, method}. headers has the shared secret stripped out before anything is stored. A body that is not JSON is passed through as the raw string, so form posts and plain text work without a content type the server has to recognize. Send x-webhook-id to make retries safe: it becomes the delivery’s idempotency key, and a repeat answers 200 with "duplicate": true having stored nothing. Without that header the key is a hash of token, body, and the current minute — an identical body resent within the same minute is one event, the same body an hour later is two. The failures are distinct on purpose: Status Meaning 404 No registration carries this token. Checked before the secret, so a wrong token never reveals whether a secret would have matched 401 x-webhook-secret missing or wrong; compared in constant time against the stored sha256 410 The registration exists but is disabled 413 Body over 1 MiB 429 Over the route’s own limit of 120 deliveries per minute curl -X POST "http://localhost:7777/api/webhooks/nosuchtoken" \ -H "Content-Type: application/json" -d '{"hello":1}' { "error": "Unknown webhook token" } Linking an External Messaging Account A messaging bridge — the nodetool telegram bot, and later Discord — is not a browser and holds no user credential. It proves which external account is speaking, and the server decides which NodeTool user that is. The four /api/integrations/{provider}/* routes are that exchange. {provider} is telegram or discord; anything else is 400. These routes authenticate with NODETOOL_INTEGRATION_TOKEN — the server’s own service token, sent as a bearer and compared in constant time — rather than a session, which is why they sit outside the session-auth hook the way the webhook route does. A server with that variable unset, or set to fewer than 16 characters, never registers them: every path answers 404, not 401. Set the same value on the bridge process. Linking runs in one of two directions, and which one you are in decides which route redeems the code. Either way the code is 24 random bytes, base64url, good for ten minutes and spent on first use. Bot-initiated. The bridge mints a code bound to the external account and sends the URL into the chat: curl -X POST "http://localhost:7777/api/integrations/telegram/link/start" \ -H "Authorization: Bearer $NODETOOL_INTEGRATION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"external_id": "482913044"}' { "code": "Yb3xK9_qLm2vR7nT4pWzA1sD6fG8hJ0c", "url": "http://localhost:7777/integrations/link?code=Yb3xK9_qLm2vR7nT4pWzA1sD6fG8hJ0c", "expires_at": "2026-08-23T12:41:07.000Z" } url is built from the request’s own Host header unless NODETOOL_PUBLIC_URL is set — set that when the bridge reaches the server at an address the user’s browser cannot, such as http://nodetool:7777 inside a compose network. The bridge stops there. The user opens that URL, and the confirmation page spends the code over tRPC (integrations.describeLinkCode, then integrations.confirmLink) under their own session, so the account that gets linked is the one they are signed in as — never one the code named. Web-initiated. The mirror image, and the one /link/complete exists for. Settings → Integrations mints a code bound to the signed-in user (integrations.createLinkCode) and renders it as t.me/<bot>?start=<code>. Pressing Start delivers /start <code> to the bot, which redeems it with the external id it can see: curl -X POST "http://localhost:7777/api/integrations/telegram/link/complete" \ -H "Authorization: Bearer $NODETOOL_INTEGRATION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"external_id": "482913044", "code": "Yb3xK9_qLm2vR7nT4pWzA1sD6fG8hJ0c"}' { "linked": true } A user-bound code already carries the user who minted it, so a user_id in the body is ignored. A code that link/start minted names no user and is refused here with 400: only the signed-in browser can redeem it. A refused call does not spend the code. With the link in place either way, the bridge exchanges identity for access on every connection: curl -X POST "http://localhost:7777/api/integrations/telegram/token" \ -H "Authorization: Bearer $NODETOOL_INTEGRATION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"external_id": "482913044"}' { "token": "ndt_eyJ2IjoxLCJ1IjoiM2Y5YSIsImUiOjE3ODc0OTE4Njd9.9c1f…", "expires_at": "2026-08-23T13:31:07.000Z", "user_id": "3f9a…" } The token lasts one hour and authenticates as that user on /ws, /trpc, and asset URLs. Tenant isolation is then the server’s usual rules — threads, tools, permissions, and cost tracking all stay server-side, and the bridge holds no conversation state of its own. Unlinking takes the external id in the body of a DELETE: curl -X DELETE "http://localhost:7777/api/integrations/telegram/link" \ -H "Authorization: Bearer $NODETOOL_INTEGRATION_TOKEN" \ -H "Content-Type: application/json" \ -d '{"external_id": "482913044"}' { "unlinked": true } "unlinked": false means there was nothing to remove. The failures across all four routes: Status Meaning 404 The routes are not registered (NODETOOL_INTEGRATION_TOKEN unset or under 16 characters), or — on /token — the external id is not linked to any user 401 Missing or wrong service token 400 Unknown provider, missing external_id, or a code issued for a different account 410 The link code expired or was already used 409 /token only: the server runs in local single-user mode, where every request is already user 1, so a delegated token would isolate nothing. Run with an enforcing auth provider (Supabase) Design and the bot’s side of the flow: Telegram bot design. The bridge command is nodetool telegram. MCP OAuth Discovery For /mcp, NodeTool is both the OAuth authorization server and the protected resource. An MCP client that meets a 401 on /mcp follows the WWW-Authenticate challenge to two discovery documents and takes the endpoint URLs from there — so a client needs no NodeTool-specific configuration beyond the server’s address. curl "https://nodetool.example.com/.well-known/oauth-protected-resource" { "resource": "https://nodetool.example.com/mcp", "authorization_servers": ["https://nodetool.example.com"], "scopes_supported": ["mcp"], "bearer_methods_supported": ["header"], "resource_name": "NodeTool MCP" } curl "https://nodetool.example.com/.well-known/oauth-authorization-server" { "issuer": "https://nodetool.example.com", "authorization_endpoint": "https://nodetool.example.com/oauth/authorize", "token_endpoint": "https://nodetool.example.com/oauth/token", "registration_endpoint": "https://nodetool.example.com/oauth/register", "revocation_endpoint": "https://nodetool.example.com/oauth/revoke", "response_types_supported": ["code"], "grant_types_supported": ["authorization_code", "refresh_token"], "code_challenge_methods_supported": ["S256"], "token_endpoint_auth_methods_supported": ["none"], "scopes_supported": ["mcp"], "client_id_metadata_document_supported": true, "authorization_response_iss_parameter_supported": true } Both documents come back with Cache-Control: public, max-age=3600. Deployed under a sub-path, RFC 8414 and RFC 9728 put them at the path-inserted forms instead — issuer https://host/base publishes at /.well-known/oauth-authorization-server/base, and the resource document for https://host/base/mcp at /.well-known/oauth-protected-resource/base/mcp. At the origin root the resource form is /.well-known/oauth-protected-resource/mcp, which returns the same document as the bare path above. All six routes answer 404 unless the flow can actually complete, and four conditions decide that: NODETOOL_DISABLE_MCP_OAUTH is not 1, the /mcp mount is enabled (in production that needs NODETOOL_ENABLE_MCP=1; in dev it is on by default), NODETOOL_PUBLIC_URL is set, and that URL is HTTPS or loopback. The server never advertises a discovery document it would then refuse to serve — with the flow off, /mcp sends no WWW-Authenticate challenge at all. The flow itself is authorization code with PKCE (S256) and no client secret. GET /oauth/authorize parks the request and 302s to /oauth/consent?request_id=…, a page the web app renders; approving it mints the code that POST /oauth/token exchanges for an nta_ access token. Replaying a code revokes the grant, per the OAuth 2.1 rule. Pasting an ntk_ token minted in Settings → MCP → Connect an agent remotely stays the alternative for a client that does not do OAuth. See MCP in production for the operator’s walkthrough and MCP OAuth design for the full flow. Chat API (OpenAI-Compatible) NodeTool exposes OpenAI-compatible endpoints, so you can use standard OpenAI clients: # Simple chat completion curl -X POST "http://localhost:7777/v1/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{ "model": "gpt-5.6", "messages": [ {"role": "user", "content": "Explain quantum computing in simple terms"} ] }' Response: { "id": "chatcmpl-abc123", "object": "chat.completion", "created": 1699000000, "model": "gpt-5.6", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Quantum computing uses quantum mechanics..." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0 } } usage is always zeros, because the endpoint does not count tokens. The provider comes from the model name, and a request without model uses llama3.2:latest. See the Chat API for the rules. Streaming Chat # Streaming chat (prints tokens as they arrive) curl -X POST "http://localhost:7777/v1/chat/completions" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{ "model": "gpt-5.6", "messages": [ {"role": "user", "content": "Write a haiku about programming"} ], "stream": true }' Streaming response. Each event is a chat.completion.chunk, the last one carries finish_reason, and the stream ends with [DONE]: data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1699000000,"model":"gpt-5.6","choices":[{"index":0,"delta":{"content":"Code"},"finish_reason":null}]} data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1699000000,"model":"gpt-5.6","choices":[{"index":0,"delta":{"content":" flows"},"finish_reason":null}]} data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1699000000,"model":"gpt-5.6","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} data: [DONE] List Available Models curl "http://localhost:7777/v1/models" \ -H "Authorization: Bearer YOUR_TOKEN" The route returns a fixed list. It does not query providers or reflect configured keys: { "object": "list", "data": [ {"id": "llama3.2:latest", "object": "model", "created": 0, "owned_by": "ollama"}, {"id": "gpt-4", "object": "model", "created": 0, "owned_by": "openai"}, {"id": "gpt-4o", "object": "model", "created": 0, "owned_by": "openai"}, {"id": "gpt-4o-mini", "object": "model", "created": 0, "owned_by": "openai"}, {"id": "claude-sonnet-4-20250514", "object": "model", "created": 0, "owned_by": "anthropic"}, {"id": "claude-opus-4-20250514", "object": "model", "created": 0, "owned_by": "anthropic"} ] } A chat request accepts any model name, not only these. For the live catalog with per-model availability, use GET /api/sdk/v1/models. Node Metadata GET /api/nodes/metadata is the node registry — every node type the server can run, with its properties, inputs, and outputs. The editor fetches it at boot, before anyone has signed in, so the route takes no token. Reach for it when you are building a client, a palette, or a graph by hand and need to know what a node type is called and what it accepts. The default response is a slim summary, because the full registry is large: curl "http://localhost:7777/api/nodes/metadata?limit=2" [ { "node_type": "fal.3d_to_3d.Hunyuan3dV31Part", "title": "Hunyuan 3d V31Part", "description": "Split 3D models into parts with Hunyuan 3D\nprocessing, 3d-to-3d, 3d, mesh, hunyuan, part", "namespace": "fal.3d_to_3d" }, { "node_type": "fal.3d_to_3d.Hunyuan3dV31SmartTopology", "title": "Hunyuan 3d V31Smart Topology", "description": "Optimize 3D mesh topology with Hunyuan 3D Smart Topology.\nprocessing, 3d-to-3d, 3d, mesh, hunyuan, smart, topology", "namespace": "fal.3d_to_3d" } ] Five query parameters narrow it: Parameter Effect node_type Exact lookup. Returns that one node’s full metadata — properties, outputs, recommended_models, layout — or 404 namespace Keep nodes whose namespace starts with this prefix query Comma-separated terms scored against title, description, node type, and namespace; unmatched nodes drop out and the best matches come first fields summary (default) or full to get complete metadata for every node in the result limit Truncate the result curl "http://localhost:7777/api/nodes/metadata?node_type=nodetool.text.Concat" { "title": "Concat", "description": "Concatenates text inputs into a single output. …", "namespace": "nodetool.text", "node_type": "nodetool.text.Concat", "layout": "default", "body": "content_card", "properties": [], "outputs": [{"name": "output", "type": {"type": "str", "type_args": []}}], "recommended_models": [] } List Workflows # List the caller's workflows (graphs included) curl "http://localhost:7777/api/workflows" \ -H "Authorization: Bearer YOUR_TOKEN" The response is {"workflows": [...], "next": <cursor or null>}. Pass limit (default 100, cap 500), cursor, run_mode, or project_id to page and filter. Create, read, save, and delete are covered in the Workflow API. Moving Workflows Between Servers A .nodetool bundle is a zip holding one or more workflow graphs plus the bytes of every asset they reference, so a workflow travels as a single file instead of a graph whose asset:// refs dangle on the far side. Three routes cover the round trip; the CLI’s workflows export-bundle / import-bundle and the editor’s command menu go through the same packer. GET /api/workflows/{id}/export-bundle returns one workflow: curl "http://localhost:7777/api/workflows/<workflow_id>/export-bundle" \ -H "Authorization: Bearer YOUR_TOKEN" \ -o my-workflow.nodetool The response is application/zip with content-disposition: attachment; filename="<workflow name>.nodetool" — the name with everything outside A-Za-z0-9._- replaced by _. Inside, each graph is a file under workflows/, asset bytes sit under their own entries, and manifest.json indexes both: { "format": "nodetool-workflow-bundle", "version": 2, "created_at": "2026-08-07T07:03:40.352Z", "workflows": [ { "file": "workflows/c3b15268a02242e682035e5e4be8a22a.json", "name": "Bundle Demo" } ], "assets": [], "thumbnail": null } POST /api/workflows/export-bundle packs several at once. workflow_ids must be a non-empty array of strings — anything else is a 400. The download is named after the single workflow when there is one, and <n>-workflows otherwise: curl -X POST "http://localhost:7777/api/workflows/export-bundle" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"workflow_ids": ["<id_a>", "<id_b>"]}' \ -o my-pack.nodetool POST /api/workflows/import-bundle takes the zip back, as a file part in a multipart form or as the raw request body: curl -X POST "http://localhost:7777/api/workflows/import-bundle" \ -H "Authorization: Bearer YOUR_TOKEN" \ -F "file=@my-workflow.nodetool" { "workflows": [ { "id": "bcec888aff5a429dae4dce4ca29fcc4c", "name": "Bundle Demo", "access": "private", "graph": { "nodes": [], "edges": [] } } ], "imported": 0, "missing": [], "checksum_mismatches": [] } Every workflow is created fresh under the caller with a new id and access: "private" — importing never overwrites an existing one. imported counts the assets stored, missing names refs the bundle did not carry, and checksum_mismatches names asset bytes that did not hash to what the manifest recorded. Neither list is fatal — the graphs still import, with those refs unresolved. A bundle that cannot be unpacked, or a graph the workflow API rejects, is a 400 ({"detail": "Invalid bundle: …"}). The import is not transactional, so a bundle that fails partway can leave the workflows created before the failure behind. Moving Timelines Between Servers A timeline zip is the same idea for a cut: manifest.json, timeline.json, and the bytes of every asset the clips point at under assets/, so a sequence arrives with its media instead of dangling asset ids. GET /api/timelines/{id}/export-zip packs one sequence: curl "http://localhost:7777/api/timelines/<timeline_id>/export-zip" \ -H "Authorization: Bearer YOUR_TOKEN" \ -o my-cut.zip The response is application/zip with content-disposition: attachment; filename="<timeline name>.zip" — the name with everything outside A-Za-z0-9._- replaced by _. A timeline the caller does not own is a 404 with {"detail": "Timeline not found"}, the same answer as an id that does not exist. The archive streams with no content-length. The server reads each local file in place while it writes, so a large asset is never held in memory. A file over 4 GiB does not fit a zip entry without Zip64 and is listed as missing. timeline.json is the sequence wire shape without its id, project, owner, and timestamps. manifest.json maps each asset id in that document to the file holding its bytes, and records what the archive could not carry: { "format": "nodetool-timeline-bundle", "version": 1, "created_at": "2026-09-06T09:12:03.114Z", "timeline": { "file": "timeline.json", "name": "My Cut" }, "assets": [ { "file": "assets/9f2c….mp4", "asset_id": "3c1a…", "name": "shot.mp4", "content_type": "video/mp4", "bytes": 4813222, "sha256": "9f2c…" } ], "missing_assets": [], "fonts": ["Inter"], "foreign_refs": { "workflow_ids": [], "storyboard_ids": [], "script_ids": [], "job_ids": [] } } Asset files are content-addressed, so two clips sharing bytes pack one file. fonts names the families the document’s titles and captions ask for — fonts ship with the app, so none are packed. foreign_refs names ids that mean something only on the exporting install (workflows, storyboards, scripts, jobs); import carries them through untouched rather than trying to resolve them. GET /api/timelines/{id}/export-premiere writes the same sequence for Adobe Premiere Pro. The zip holds <timeline name>.xml, a Final Cut Pro 7 XML sequence that Premiere imports through File > Import, the active asset of every exported clip under media/, and export-report.txt, which lists each clip that needs baking or was left out. The XML points at its media with relative media/<file> paths. The response carries x-nodetool-export-warnings with the number of report entries, and is named <timeline name>-premiere.zip by the same rule as above. GET /api/timelines/{id}/export-media packs only the media: the original file of every video, image, and audio clip under media/, named as in the Premiere export, and clips.csv with one row per clip (clip, track, media_type, start_s, end_s, in_s, file). x-nodetool-export-missing counts the files that could not be read or packed. The zip is <timeline name>-media.zip. POST /api/timelines/import-zip also accepts a zip with no manifest.json that holds a Final Cut Pro 7 XML (xmeml) sequence, such as Premiere Pro’s File > Export > Final Cut Pro XML zipped with its media. Media is matched by file name in any folder of the zip. The response adds format: "premiere_xml", missing_clips (clips whose file was not in the zip), and warnings (what had no NodeTool counterpart, such as transitions or nested sequences). POST /api/timelines/import-zip reads the zip back as a multipart form. file is required; project_id defaults to "default" (no project) and name overrides the name in the bundle: curl -X POST "http://localhost:7777/api/timelines/import-zip" \ -H "Authorization: Bearer YOUR_TOKEN" \ -F "file=@my-cut.zip" \ -F "project_id=<project_id>" { "timeline": { "id": "8f1c…", "projectId": "default", "name": "My Cut", "clips": [] }, "imported": 1, "missing": [], "checksum_mismatches": [] } Import always creates a new timeline under the caller — it never overwrites an existing one. Every asset is stored fresh and the clip fields that named the old ids (currentAssetId, thumbnailAssetId, waveformAssetId, and each versions[].assetId) are rewritten to the new ones. imported counts the assets stored, missing names ids the archive carried no bytes for — those keep their original value and stay dangling — and checksum_mismatches names files that did not hash to what the manifest recorded. Neither list is fatal. An archive that cannot be unpacked, or a document the timeline schema rejects, is a 400 ({"detail": "Invalid bundle: …"}). Fetching a Shipped Example’s Graph GET /api/workflows/examples lists the shipped templates, but every entry comes back with "graph": {"nodes": [], "edges": []} — the list is metadata, kept small because it ships hundreds of workflows. GET /api/workflows/examples/{package}/{example} is how you get one example’s actual graph. Both are unauthenticated: the templates ship with the install and belong to nobody. The two path segments are the package name and the example name, URL-encoded — the same package_name and name the list returned. curl "http://localhost:7777/api/workflows/examples/nodetool-base/Sharpen%20Footage" { "id": "", "access": "private", "name": "Sharpen Footage", "description": "Add apparent detail back after a denoise or a downscale. …", "tags": ["video", "utility", "example"], "package_name": "nodetool-base", "graph": { "nodes": [ … ], "edges": [ … ] }, "input_schema": { … }, "output_schema": { … } } The id is empty because an example is a template, not a row — POST the graph to /api/workflows to make it yours. A name the package does not ship is a 404: { "detail": "Example 'Nope' not found in package 'nodetool-base'" } Uploading an Asset On a cloud storage backend (Supabase or S3) the bytes go straight from the client to the bucket — the API only mints a target and confirms the result. assets.createUpload (tRPC) with the file’s name, content type, parent folder, and size. It creates the asset row, picks the storage key (<userId>/<assetId>.<ext> — never the client’s choice), and returns a short-lived upload target. Send the bytes to upload.url with upload.method and upload.headers. assets.finalizeUpload with the returned asset_id. The server reads the object’s real size back off the bucket, records it, and generates a thumbnail. A missing, empty, or over-cap upload is rejected and the pending row removed. upload comes back null on the local file backend, which has no direct-upload concept; fall back to the multipart POST /api/assets. The web client does this automatically. Extracting a Video’s Audio Track POST /api/assets/{id}/extract-audio copies a video asset’s audio into a new WAV asset parented to the video. The timeline calls it when a video is imported so the audio becomes an independently editable clip; call it directly when you want the same thing outside the editor. curl -X POST "http://localhost:7777/api/assets/<video_asset_id>/extract-audio" \ -H "Authorization: Bearer YOUR_TOKEN" { "has_audio": true, "asset": { "id": "b41f…", "name": "clip.mp4 (audio)", "content_type": "audio/wav", "parent_id": "<video_asset_id>", "duration": 12.5, "get_url": "/api/storage/1/b41f….wav" } } A video with no audio track returns {"has_audio": false} and creates nothing. Posting a non-video asset is a 400 ({"detail": "Asset is not a video"}), an asset you do not own is a 404, and a server with no ffmpeg runtime available answers 503. Waveform Peaks GET /api/assets/{id}/peaks?count=2000 returns the waveform the timeline draws on an audio clip: count abs-max values of channel 0 (1 to 20000, default 2000) and the decoded length. { "peaks": [0, 0.0132, 0.4821, …], "duration_ms": 184032.5 } ffmpeg decodes the asset as a stream, reading a local file in place. The result is cached under the NodeTool cache directory, keyed by the asset, the file’s size and mtime (or the row’s updated_at on a store with no local file), and count, so a file changed in place gets new peaks. The x-peaks-cache header is hit or miss. A non-media asset or a bad count is a 400, an asset you do not own is a 404, a file with no audio stream is a 422, and a server with no ffmpeg runtime answers 503. Package Assets Shipped node packs carry media alongside their code — the images and audio that example workflows reference as package://<package>/<file>. Those refs resolve over GET /api/assets/packages/{package}/{file}, which needs no token because the bytes ship with the install. curl "http://localhost:7777/api/assets/packages/nodetool-base/A%20Boolean%20Constant.jpg" \ -o thumb.jpg The file path may be nested (audio/loop.mp3); .. segments and backslashes are rejected. Responses carry cache-control: public, max-age=31536000, immutable and an ETag, since a package’s assets change only when the package version does. The two listing routes above that path are stubs. GET /api/assets/packages and GET /api/assets/packages/{package} both answer 200 with an empty page, whatever the package name: curl "http://localhost:7777/api/assets/packages/nodetool-base" { "assets": [], "next": null } The REST surface never grew a real listing, so address a package’s files by the name the workflow’s package:// ref already carries. Agents that need to enumerate them call the list_assets tool with source: "package", which walks the packages on disk and returns each file’s uri and url. Bulk Asset Download POST /api/assets/download is reserved for zipping the requested assets into one response, keeping the folder structure their parent_id relationships describe. This server does not implement it and answers 501: curl -X POST "http://localhost:7777/api/assets/download" \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"asset_ids": ["b41f…", "c02a…"]}' { "code": "SERVICE_UNAVAILABLE", "detail": "ZIP download not available in standalone mode" } The status does not depend on the body — an unknown asset id returns the same 501. Download assets one at a time through their get_url instead. Downloading a Workspace File A workspace is a directory on the server’s filesystem that agents and file tools read and write. Listing and CRUD moved to the tRPC workspace router; GET /api/workspaces/{id}/download/{path} stayed on REST because it returns bytes rather than JSON. path is relative to the workspace root and may be nested. The response is an attachment, with a content type guessed from the extension: curl "http://localhost:7777/api/workspaces/ws_abc123/download/notes/report.md" \ -H "Authorization: Bearer YOUR_TOKEN" \ -o report.md When NODETOOL_ENV=production, only the server-managed workspace is readable. A workspace row that points at some other host folder answers 403 with {"detail": "This workspace is not readable in production"}. Otherwise: Status Meaning 400 path is absolute 403 path resolves outside the workspace root, or the workspace is not readable in production 404 No such workspace for this caller, or no such file inside it 405 The method is not GET 500 The workspace has no storage configured curl "http://localhost:7777/api/workspaces/nosuchws/download/a.txt" { "detail": "Workspace not found" } Provider Credits GET /api/fal/credits and GET /api/kie/credits report what the account behind the server’s API key has left. The editor shows the number next to the provider; call them yourself to watch a budget from outside the UI. Both read the key the server holds — the stored FAL_API_KEY / KIE_API_KEY secret, falling back to the same environment variable — not anything the caller sends. With no key configured the answer is 204 No Content. curl "http://localhost:7777/api/fal/credits" \ -H "Authorization: Bearer YOUR_TOKEN" { "credit_balance": { "amount": 42.5, "currency": "USD" }, "username": "your-fal-account" } kie.ai bills in credits rather than dollars, so its currency is the literal string "credits": { "credit_balance": { "amount": 1200, "currency": "credits" } } A provider that refuses the key still answers 200, with the reason spelled out — the SPA treats a gateway status as a bug in NodeTool, so the failure is carried in the body instead: { "unavailable": true, "detail": "Invalid API key or malformed Authorization header", "credit_balance": null } Reading a fal.ai balance needs an Admin key (create one at https://fal.ai/dashboard/keys); an ordinary key gets 403 from fal.ai, which comes back as an unavailable body saying so. Provider Pricing GET /api/fal/pricing returns the unit price of one or more fal.ai endpoints. Repeat endpoint_id for each; omitting it entirely is a 400. Prices are cached per endpoint for an hour, and the route answers 204 when no FAL_API_KEY is configured. curl "http://localhost:7777/api/fal/pricing?endpoint_id=fal-ai/flux/schnell&endpoint_id=fal-ai/flux/dev" \ -H "Authorization: Bearer YOUR_TOKEN" { "byEndpointId": { "fal-ai/flux/schnell": { "unit_price": 0.003, "billing_unit": "megapixel", "currency": "USD" } }, "fetched_at": "2026-08-05T08:21:49.251Z" } GET /api/kie/pricing is the same shape over kie.ai model ids (?model_id=, repeatable, 400 when absent). It needs no API key — kie.ai publishes its pricing pages openly — and returns a per-model summary: { "byModelId": { "flux-2/pro-text-to-image": { "model_id": "flux-2/pro-text-to-image", "unit_price": 9, "billing_unit": "second", "currency": "credits", "usd_price": 0.045, "tier_count": 4, "pricing_url": "https://kie.ai/flux-2" } }, "fetched_at": "2026-08-05T08:19:54.783Z" } Most kie.ai models are priced in tiers. unit_price is the cheapest of them, tier_count says how many there were, and billing_unit is "varies" when the tiers are not billed by the same unit. An id with no published price is absent from the map rather than an error, so a request for five models can come back with three. Estimating a fal.ai Call’s Cost GET /api/fal/pricing gives the price of one unit; POST /api/fal/pricing/estimate turns that into a total for a quantity you name, so you can price a batch before running it. endpoint_id is required — without it the route answers 400 — and estimate_type picks how the quantity is read: historical_api_price (the default) counts whole calls, from call_quantity (default 1, values below 1 are ignored). unit_price counts billing units — megapixels, seconds — from unit_quantity (default 1, floor 0.000001). curl -X POST "http://localhost:7777/api/fal/pricing/estimate" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"endpoint_id": "fal-ai/flux/schnell", "call_quantity": 250}' { "endpoint_id": "fal-ai/flux/schnell", "estimate_type": "historical_api_price", "total_cost": 0.75, "currency": "USD", "fetched_at": "2026-08-10T09:14:02.118Z", "cached": false } The key comes from the secret store, falling back to the FAL_API_KEY environment variable; with neither the route answers 204 and the SPA shows no estimate. A fal.ai call that fails or answers with an unusable body is a 502. Answers are cached for an hour per endpoint and estimate type — the quantity is not part of the cache key, so a second request that changes only the quantity returns the first one’s total with cached: true. Vary the endpoint or the estimate type to get a fresh number inside that hour. Resolving a KIE Model’s Schema kie.ai adds models faster than a node can be written for each, so KieAINode takes its shape from the model’s documentation. POST /api/kie/resolve-dynamic-schema does that parse: give it the docs page as model_info, get back the node’s dynamic properties, inputs, and outputs. The editor calls it when you paste docs into the node; call it directly to check what a page would produce before wiring anything. curl -X POST "http://localhost:7777/api/kie/resolve-dynamic-schema" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"model_info": "| **Format** | `bytedance/seedance-2` |\n\n### input Object Parameters\n\n#### prompt\n- **Type**: `string`\n- **Required**: No\n- **Description**: The text prompt for the video.\n"}' { "model_id": "bytedance/seedance-2", "dynamic_properties": { "prompt": "" }, "dynamic_inputs": { "prompt": { "type": "str", "type_args": [], "optional": true, "description": "The text prompt for the video." } }, "dynamic_outputs": { "video": { "type": "video", "type_args": [], "optional": false } } } The model id comes from the docs’ Format row; parameters are read from the #### <name> headings under input Object Parameters, so an excerpt missing that section resolves to a node with no inputs. Docs with no recognizable model id are a 400 ({"code": "INVALID_INPUT", "detail": "Could not find model ID in documentation"}), as is a missing or empty model_info. What a Published App Runs Publishing a mini app freezes a snapshot: the document as it stood, plus the graph of every workflow its operations call. GET /api/applications/{id}/released-document returns that snapshot, so a runtime can serve the published app without reading the draft the author is still editing. curl "http://localhost:7777/api/applications/<application_id>/released-document" \ -H "Authorization: Bearer YOUR_TOKEN" { "id": "548770b4c014436ba8549509575e9be6", "applicationId": "76a381309a584a13b823b297cbd9b4b1", "version": 1, "document": { "schemaVersion": 3, "ui": { "root": { "props": {} }, "content": [], "zones": {} }, "operations": [], "resources": [], "variables": [] }, "capabilities": { "workflows": [], "resources": [] }, "released": true, "createdAt": "2026-08-05T08:19:40.275Z", "workflows": [ { "workflowId": "wf_abc123", "version": 4, "graphHash": "9f2c…", "graph": { "nodes": [], "edges": [] } } ] } Each entry in workflows is the graph as the release froze it. version and graph are null on a snapshot published before releases pinned anything — a runtime that meets one falls back to the live workflow. An app with nothing published answers 200 with a body of null; an app you do not own is a 404. Publishing itself is a tRPC call (applications.publish), not a REST route. Installing a Shipped Example App NodeTool ships a set of curated mini apps. They are ApplicationBundle files on disk rather than database rows, so listing and reading one touches no user data. The routes still sit behind session auth, and installing one writes into the caller’s library. GET /api/applications/examples is the catalog: curl "http://localhost:7777/api/applications/examples" [ { "slug": "dataset-builder", "name": "Dataset Builder", "description": "The smallest app in the set, and the reference for the Table widget: a dataframe reads better as rows than as a Preview node.", "workflows": ["Data Generator"], "operationCount": 1, "thumbnailUrl": "/api/workflows/examples/thumbnails/Data%20Generator.jpg?v=3f2a9c1e" } ] workflows names the workflows installing the app would create, and thumbnailUrl is the gallery art of the first of them, or null when it ships none. To read the whole thing first — the app document plus the full graph of every workflow it binds — fetch the bundle: curl "http://localhost:7777/api/applications/examples/dataset-builder" That returns {schemaVersion, name, description, app, workflows}. A slug nothing ships is a 404 ({"detail": "No example app named \"nope\""}). POST /api/applications/examples/{slug}/install goes through the normal bundle import and answers with the created application. An optional projectId in the body files it under a project: curl -X POST "http://localhost:7777/api/applications/examples/dataset-builder/install" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{}' { "id": "f85f3d79566346df90827c752eab5bcf", "projectId": "default", "name": "Dataset Builder", "description": "…", "document": { "schemaVersion": 3, "ui": { "root": { "props": {} }, "content": [], "zones": {} }, "operations": [], "resources": [], "variables": [] }, "createdAt": "2026-08-07T07:03:52.836Z", "updatedAt": "2026-08-07T07:03:52.836Z" } Omitting projectId files the app under default. Installing also creates the workflows the app binds. Workflows carrying a sourceId are created once per user, so installing two examples that share a template leaves one workflow row both apps point at. To install a bundle of your own instead of a shipped one, post it to POST /api/applications/import-bundle. POST /api/applications/examples/{slug}/scripts/{key}/run runs one of the JS scripts an example bundle carries, so the app’s page can run an example before anyone installs it. key is a scripts[].key value from the bundle returned by GET /api/applications/examples/{slug}. The body is the same as POST /api/js-scripts/{id}/run except that script_version does not apply: the run uses the document stored in the bundle. curl -X POST "http://localhost:7777/api/applications/examples/product-price-drop/scripts/plan_storyboard-v1/run" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"inputs": {"recipe": {"operations": []}}}' { "ok": false, "logs": [], "duration_ms": 277, "error": "Select a finishing model before planning." } A script that throws answers 200 with ok: false, as it does for a saved script. The run belongs to the caller’s personal project, so assets and documents the script creates land there. Send Accept: application/x-ndjson to stream the run: each line is one JSON object, {"type": "message", ...} for agent progress, {"type": "emit", "name": ..., "value": ...} for an emit call, and a last {"type": "result", "result": ...} line carrying the body the plain answer would have returned. A slug or key the bundles do not ship is a 404 ({"detail": "Example app script not found"}). An inputs value that is not an object is a 400 ({"detail": "Invalid script inputs"}). Exporting an App as a Bundle GET /api/applications/{id}/export-bundle is the other half of import-bundle: it packs one of the caller’s apps and the full graph of every workflow its operations bind into a single ApplicationBundle JSON file. Reach for it to move an app between servers, or to check into git what import-bundle will recreate. curl "http://localhost:7777/api/applications/<application_id>/export-bundle" \ -H "Authorization: Bearer YOUR_TOKEN" \ -O -J The response is application/json with a content-disposition built from the app’s name — attachment; filename="Ask_Your_Documents.app.json" — so curl -OJ writes it under that name. The body is the bundle: { "schemaVersion": 1, "name": "Ask Your Documents", "description": "Retrieval-augmented answers with citations, and a fully local fallback…", "app": { "schemaVersion": 4, "ui": { "root": {}, "content": [] }, "operations": [ { "id": "ask", "name": "Ask", "workflowId": "chat-with-your-documents", "inputs": { "question_input": { "from": "variable", "variableId": "question" } }, "outputs": {}, "policy": "replace" } ], "resources": [], "variables": [] }, "workflows": [ { "key": "chat-with-your-documents", "name": "Chat With Your Documents", "description": "Retrieval-augmented Q&A over your own documents…", "graph": { "nodes": [], "edges": [] }, "version": null, "graphHash": null } ], "scripts": [] } The operation’s workflowId is that workflow’s key in this file, not a row id — which is what lets import-bundle create the workflows on the target server and rewrite the keys to whatever ids they get there. scripts carries any JS script documents the app binds, on the same terms. By default this exports the draft document. ?released=1 (or ?released=true) exports the published snapshot instead — the document GET /api/applications/{id}/released-document returns, packed together with the graphs that release pinned: curl "http://localhost:7777/api/applications/<application_id>/export-bundle?released=1" \ -H "Authorization: Bearer YOUR_TOKEN" \ -o my.app.json The two routes disagree about an app that has never been published, so pick by which answer you want to handle. released-document calls it a 200 with a null body; export-bundle?released=1 calls it a 404 with {"detail": "Application has no released version"}. An id the caller does not own is also a 404. The CLI wraps the same call as nodetool apps export-bundle <id> [-o file] [--released]. Exporting an App as a Standalone Project GET /api/applications/{id}/export-project returns the app as a JavaScript project that runs without NodeTool, zipped under a <project-name>/ folder. ?released=1 exports the released snapshot, as for export-bundle. curl "http://localhost:7777/api/applications/<application_id>/export-project" \ -H "Authorization: Bearer YOUR_TOKEN" \ -O -J The response is application/zip with content-disposition: attachment; filename="<project-name>.zip". The project layout is described in @nodetool-ai/app-export. The CLI equivalent is nodetool apps export-project. Building an App on the Server POST /api/applications/build runs the same six-stage build the CLI’s nodetool app build runs — spec, plan, author, check, run, judge — and answers with the BuildReport. Reach for it when a caller wants a batch build without a CLI on the machine. Provider and model come from the body, falling back to NODETOOL_APP_BUILD_PROVIDER and NODETOOL_APP_BUILD_MODEL. Neither one set either way is a 400: curl -X POST "http://localhost:7777/api/applications/build" \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "prompt": "an app that drafts a note from a prompt", "provider": "anthropic", "model": "claude-sonnet-5" }' Body fields, all optional except one of prompt or spec: Field Meaning prompt What to build, in words. Either this or spec spec A pinned BuildSpec, skipping the Spec stage. Either this or prompt provider / model The builder’s provider and model judge_model provider/model for the Judge stage. Omitted, the server picks a configured model the builder did not use workflow_ids Workflows to pin, in operation declaration order, instead of planning them max_repairs Repair rounds allowed (default 3) cost_cap_usd Ceiling on build spend (default 2) timeout_ms Wall-clock cap on the build poll Return a session id immediately instead of holding the request open max_repairs must be a non-negative integer and cost_cap_usd a positive number. A present-but-invalid value is a 400 rather than silently becoming the default — cost_cap_usd: 0 turning into $2 would spend money the caller said not to. A build runs for minutes, so poll: true answers as soon as it starts: { "status": "running", "session_id": "…", "build_id": "…", "poll": "GET /api/debug/sessions/…", "cancel": "POST /api/debug/sessions/…/cancel" } Read those two paths until the session settles. A cancelled build settles as failed with reason: "cancelled". The bundle behind a green verdict is offered, never installed: turning it into an application is a separate POST /api/applications/import-bundle. Debugging an App on the Server POST /api/applications/debug simulates a mini app headlessly — it validates every widget binding, seeds input defaults, replays an interaction script, runs the workflows on the kernel, and reports what each widget ended up showing. Name a saved app with application_id, or post the live draft as document. Neither is a 400 ("An app debug run needs either an application_id or a document."). curl -X POST "http://localhost:7777/api/applications/debug" \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"application_id": "<application_id>", "run": false}' Field Meaning application_id A saved application, read from the row. Either this or document document The live draft, verbatim. Either this or application_id params Reactive values applied before the interactions, keyed by input name interact The interaction script. Omitted, the app’s natural run trigger fires run Execute workflow runs (default true); false is a static wiring check timeout_ms Per-run timeout poll Return a session id immediately, as with build above The response is the compacted report — the verdict plus what each widget shows. target.ref echoes what was named: an application id, or inline-document when the draft was posted as document. { "debug_id": "app-debug-186814b9-d387-4e67-8b8f-e7eab397575d", "status": "failed", "target": { "ref": "inline-document", "source": "application", "workflowId": null }, "app": { "title": "Ask Your Documents", "widgetCount": 25 }, "verdict": { "ok": false, "headline": "App has issues — TextInput \"in-ask-search\": bound to \"op:ask/in:search_input\" but operation \"ask\" runs a workflow with no node \"search_input\".", "issues": ["…"] } } nodetool app debug runs the same simulation locally and writes a full bundle to disk; see CLI. Exporting a Storyboard as a Zip The web app reads and writes boards over /trpc/storyboards.*. GET /api/storyboards/{id}/export-zip is the one non-tRPC door, because the body is a binary archive rather than JSON. Reach for it to hand a board to someone without a NodeTool account — a director, a client, a review thread. curl "http://localhost:7777/api/storyboards/<storyboard_id>/export-zip" \ -H "Authorization: Bearer YOUR_TOKEN" \ -O -J The response is application/zip with cache-control: no-store and a content-disposition built from the board’s name, with each run of characters outside A-Za-z0-9._- replaced by a single _ — a board named First Light — Travel Teaser becomes attachment; filename="First_Light_Travel_Teaser.zip" — so curl -OJ writes it under that name. Inside: storyboard.md the board, shot by shot, linking its media stills/01-dunes-before-sunrise.png the selected keyframe per shot clips/01-dunes-before-sunrise.mp4 the selected clip per shot Media file names are <two-digit shot number><-slug>.<ext>, where the slug is the shot’s own slug (or its action text) lowercased and hyphenated to 40 characters. The extension comes from the stored path; when the path carries none it falls back to the ref’s type, and then to .bin. storyboard.md opens with the board title, then a bullet for each of logline, brief, style, aspect ratio, and music prompt the board has set, always a shot count, and the narration when there is one. Each shot follows under a level-two heading — its number, then its slug or the first line of its action — with the still embedded when one was packed, the action text, and a bullet for each of camera, motion, dialogue, narration, duration, status, and notes it carries, plus a link to its clip. A shot’s media is resolved server-side from asset://, a /api/storage/ path, an https:// URL, or an inline data: URI — and only those. A ref in any other form is not fatal: the shot keeps its text and the Markdown carries the reason where the media would have been. - **Missing clip:** `asset://f3c1…` could not be read, so it is not in this archive. That covers a blob: handle only the browser ever held, a URL that no longer answers, and package:// — which is what the shipped example boards carry, so exporting one of those before rendering anything of your own yields storyboard.md and no media files. Unlike a .nodetool workflow bundle this is not a re-importable format — nothing rewrites the refs, and there is no import route that reads it back. A board the caller does not own is a 404 with {"detail": "Storyboard not found"}, the same answer as an id that does not exist. Exporting a Game The editor reads and writes games over /trpc/games.*. Two routes sit outside tRPC because their bodies are not JSON. Both build the game’s published revision, the same build the agent’s build_native_game capability makes (see Game Editor). GET /api/games/{id}/export-zip streams the build as a zip. Reach for it to hand a playable game to someone without a NodeTool account. curl "http://localhost:7777/api/games/<game_id>/export-zip?target=web" \ -H "Authorization: Bearer YOUR_TOKEN" \ -O -J Query Values Effect target web (default), tauri web is the static web player, tauri is the desktop app project revision 32-character hex revision id Build that revision instead of the published one Any other query parameter is rejected. The response is application/zip with cache-control: no-store and the file name game-<first 12 characters of the game id>-web.zip (-desktop.zip for tauri). Every entry sits under a web/ or desktop/ folder. A game the caller does not own, or a missing revision, is a 404 with {"detail": "Game not found"} or {"detail": "Game revision not found"}. A build that cannot run is a 422 with detail. POST /api/games/{id}/build-executable builds the desktop project and compiles it with cargo on the server’s machine. It streams cargo’s output as newline-delimited JSON (application/x-ndjson), and closing the request stops cargo. curl -N -X POST "http://localhost:7777/api/games/<game_id>/build-executable" \ -H "Authorization: Bearer YOUR_TOKEN" {"type":"status","message":"Compiling My Game with cargo","project_path":"/path/to/workspace/game-builds/..."} {"type":"log","line":" Compiling tauri v2..."} {"type":"done","binary_path":".../bin/my-game","project_path":"..."} A failed build ends the stream with {"type":"error","detail":"..."}, which adds runtime_package when a runtime package such as rust is missing. Before the stream starts, the route answers 403 when the cloud profile is active (NODETOOL_NODE_PROFILE=cloud, or NODE_ENV=production with the profile unset), 503 with runtime_package: "rust" when cargo is not installed, 404 for a game the caller does not own, and 422 when the workspace is not on this machine’s disk. All games share one cargo target directory, so only the first build compiles Tauri itself. What This Server Supports GET /api/sdk/v1/capabilities is the handshake an SDK client makes before anything else: which route families are live, which wire encodings the server speaks, what an execution request may ask for, and the numeric limits it will enforce. Read it once at startup instead of hard-coding assumptions about the server on the other end. curl "http://localhost:7777/api/sdk/v1/capabilities" \ -H "Authorization: Bearer YOUR_TOKEN" { "protocol_version": "1", "nodetool_version": "0.7.0-rc.36", "server_time": "2026-08-21T06:09:57.898Z", "supported_encodings": ["messagepack", "json-text"], "default_encoding": "messagepack", "profiles": { "discovery": "available", "execution": "available", "preflight": "available", "model_catalog": "available", "model_download": "available", "temporary_asset_upload": "available" }, "registry_revision": 2723, "python_bridge": "starting", "auth_modes": ["trusted_local"], "asset_uri_schemes": ["asset"], "execution_options": { "persistence": ["job", "session"], "event_detail": ["full", "outputs", "terminal"], "asset_persistence": ["auto", "temporary"], "defaults": { "persistence": "job", "event_detail": "full", "asset_persistence": "temporary" } }, "limits": { "max_rpc_batch": 100, "max_inline_bytes": 0, "max_upload_bytes": 1073741824, "max_queued_jobs": 0, "max_job_event_replay": 0, "request_timeout_seconds": 30 } } Each entry in profiles is available, disabled, or unavailable. On this server the map is a fixed list that always reports available, so it names the route families that exist rather than reporting which are switched on — a disabled family still shows available here and answers 503 when called. Do not gate on it; handle the 503. In limits, only max_upload_bytes is a ceiling the server enforces — it tracks NODETOOL_MAX_UPLOAD_BYTES (1 GiB by default) and is what /api/sdk/v1/assets/temporary rejects an over-size file against. The rest are advertised figures this server does not itself enforce, and the zeros are not limits of zero: max_inline_bytes is 0 because the profile carries media by asset reference rather than promising any inline payload size, and max_queued_jobs and max_job_event_replay are 0 because it states no bound for them. auth_modes is trusted_local on a server with no enforcing auth provider and bearer once one is configured, so a client can tell whether it needs a token before it sends a request without one. python_bridge is ready once the Python bridge has connected and starting until then — including on a server with no Python installed, where it stays starting indefinitely rather than reporting unavailable. A workflow using Python nodes is not runnable until it reads ready. registry_revision is the same counter the node inventory and workflow summaries report, so a client can tell whether the node registry moved under it. Two environment flags can switch parts of this family off, and they cover different routes. NODETOOL_DISABLE_SDK_LIFECYCLE_V1=1 disables this route, /preflight, and /assets/temporary, which then answer 503 with {"code": "SDK_LIFECYCLE_DISABLED", …}. NODETOOL_DISABLE_SDK_WORKFLOW_INTERFACE_V1=1 disables the discovery routes — /node-types, /workflows, /workflow-interfaces, and /api/sdk/v1/workflows/{id}/interface — which answer 503 with SDK_WORKFLOW_INTERFACE_DISABLED, except /node-types, which reports SDK_NODE_TYPE_INVENTORY_DISABLED. The model catalog at /api/sdk/v1/models and the model-download routes are unaffected by either. See Configuration. The Node Type Inventory GET /api/sdk/v1/node-types reports the pin types the loaded registry actually uses — one entry per type signature, how many nodes read or write it, and example pins to look at. Where /api/nodes/metadata is indexed by node, this is indexed by type, so it is the route to ask what a given type connects to without walking every node’s metadata. curl "http://localhost:7777/api/sdk/v1/node-types?limit=2" \ -H "Authorization: Bearer YOUR_TOKEN" { "version": 1, "registry_revision": 2723, "registry_ready": true, "python_bridge_ready": false, "node_count": 2722, "type_count": 1158, "provenance_counts": { "typescript": 2722 }, "cursor": 0, "next_cursor": 2, "types": [ { "signature": "any", "type": "any", "type_name": null, "optional": false, "type_args": [], "values": [], "values_truncated": false, "input_uses": 74, "output_uses": 54, "node_count": 67, "sources": { "typescript": 128 }, "examples": [ { "node_type": "fal.vision.ArbiterImage", "pin": "values", "direction": "output" } ] } ], "unavailable_packs": [ { "id": "elevenlabs", "name": "ElevenLabs", "reason": "disabled by built-in pack configuration" } ] } cursor (default 0) and limit (1–100) page through the list; anything outside that range is a 400 ({"code": "INVALID_INPUT", …}). next_cursor is what to pass as the next cursor. node_count appears at both levels and means different things: at the top it is every node in the registry, and inside a type entry it is the nodes using that type. type_count is how many entries exist across all pages, and provenance_counts splits the registry by where the nodes came from — typescript here, with Python nodes appearing once the bridge is up. input_uses and output_uses count pins, not nodes, so one node with two pins of a type counts twice — node_count is the number of distinct node types. examples is capped at 5 entries, and for an enum type values is capped at 64 with values_truncated set when there were more. unavailable_packs names packs whose nodes are missing from these counts and why, so a type that looks absent can be traced to a pack that never loaded rather than to a registry bug. python_bridge_ready is false until the Python bridge connects, and Python-only types are absent while it is. A Workflow’s Input and Output Pins Two routes answer what a workflow takes and returns, without fetching its graph. Use them to build a form, validate a parameter bag, or decide which outputs to subscribe to. GET /api/sdk/v1/workflows/{id}/interface?version=1 covers one workflow. The version=1 query parameter is required — omitting it is a 400 ({"code": "UNSUPPORTED_WORKFLOW_INTERFACE_VERSION", …}), not a default: curl "http://localhost:7777/api/sdk/v1/workflows/<workflow_id>/interface?version=1" \ -H "Authorization: Bearer YOUR_TOKEN" { "version": 1, "workflow_id": "23486bda2a46425fa8559f47c7c0adc9", "etag": "777f80b8651d2e56d7e1861a11f12667", "source": "server", "inputs": [ { "node_id": "in1", "name": "prompt", "description": "Text to echo", "type": { "type": "str", "optional": false, "type_args": [], "type_name": null }, "required": true, "default": "" } ], "outputs": [ { "node_id": "out1", "name": "result", "description": "The echoed text", "type": { "type": "str", "optional": false, "type_args": [], "type_name": null }, "stream": false } ], "diagnostics": [] } An input pin’s name is the key to use in a run’s params; an output pin’s name is the key its value arrives under. stream marks an output that emits repeatedly rather than once. etag changes whenever the interface changes, so a client can cache a form against it and pass it back as workflow_etag on a preflight. A graph the server can read but not interpret comes back with entries in diagnostics (severity, code, message, and the node_id / pin_name at fault) rather than an error. POST /api/sdk/v1/workflow-interfaces does the same for 1–100 workflows in one call. Both fields are required, and the ids must be unique: curl -X POST "http://localhost:7777/api/sdk/v1/workflow-interfaces" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"ids": ["23486bda2a46425fa8559f47c7c0adc9"], "version": 1}' Each entry of interfaces is exactly the object the single-workflow route returns. One bad id does not fail the batch — it lands in errors instead, and the rest still come back: { "interfaces": [ { "version": 1, "workflow_id": "23486bda2a46425fa8559f47c7c0adc9", "etag": "777f80b8651d2e56d7e1861a11f12667", "source": "server", "inputs": [], "outputs": [], "diagnostics": [] } ], "errors": [ { "workflow_id": "deadbeef", "code": "workflow_not_found", "message": "Workflow not found" } ] } code is workflow_not_found or invalid_graph. A malformed body — an empty list, over 100 ids, duplicate ids, or a missing version — is a 400 ({"code": "INVALID_INPUT", "message": "Expected 1 to 100 unique workflow ids"}) and nothing is returned. To find the ids in the first place, GET /api/sdk/v1/workflows lists the caller’s workflows in the shape this family uses: curl "http://localhost:7777/api/sdk/v1/workflows?limit=2" \ -H "Authorization: Bearer YOUR_TOKEN" { "workflows": [ { "id": "23486bda2a46425fa8559f47c7c0adc9", "name": "Interface Demo", "description": "Echoes a string input to an output", "revision": "2026-08-21T06:10:04.279Z", "registry_revision": 2723, "run_mode": "workflow" } ], "next": null } limit is 1–100, default 50; outside that range is a 400. Page by passing next back as cursor until it is null. revision is the workflow’s own last-modified stamp and registry_revision the registry’s, so a cached interface can be invalidated by either moving. This is the discovery-shaped sibling of /api/workflows — that route returns whole workflow records including graphs, this one returns only what a client needs to pick one. Checking a Workflow Before Running It POST /api/sdk/v1/preflight answers whether a workflow can run right now, what it needs first, and what it will cost — before a job row exists and before any node is paid for: curl -X POST "http://localhost:7777/api/sdk/v1/preflight" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{ "workflow_id": "23486bda2a46425fa8559f47c7c0adc9", "workspace_id": null, "workflow_etag": null, "interface_version": 1, "level": "availability", "inputs": {"prompt": "hi"} }' { "version": 1, "level": "availability", "workflow_id": "23486bda2a46425fa8559f47c7c0adc9", "workflow_etag": "777f80b8651d2e56d7e1861a11f12667", "runnable": true, "issues": [], "requirements": [ { "kind": "node_pack", "id": "base", "name": "base", "status": "available", "blocking": true, "message": null, "details": { "node_ids": ["in1", "out1"] } } ], "cost": { "amount": 0, "currency": "USD", "confidence": "exact", "unknown_cost_nodes": [], "approval_required": false } } level picks how much work the check does, and each level costs more than the one before: Level What it decides static The graph and the inputs, against the registry. Requirements are listed but their status is unknown — nothing is probed availability The same, plus a probe of each requirement: a pack becomes available or missing, a model available, downloading, or unavailable, a credential present or not. A probe that cannot decide reports unknown execution The same, plus whether there is capacity to run it now — adds worker requirements for the server itself and for execution capacity A requirement’s kind is one of provider, credential, model, node_pack, runtime, asset, worker, or approval; blocking says whether runnable turns false without it. issues carries validation findings with severity warning or error — at execution level a server that cannot read its own capacity reports the warning execution_capacity_unknown and stays runnable, because an unknown queue depth is not a broken workflow. cost.confidence separates two different kinds of doubt. exact means every node priced exactly (as it does for a graph with nothing billable in it, where amount is 0). estimate means everything priced but at least one price is itself an estimate. partial means some nodes could not be priced at all, and those are the ones unknown_cost_nodes names. unknown means none could — and that is the one case where amount is null rather than a number, so read confidence before displaying amount. approval_required compares the total against the server’s own configured threshold, so gate a spend prompt on it rather than on a threshold of the client’s. Passing the etag from the workflow’s interface as workflow_etag checks the graph has not moved since you read it. When it has, the answer is still a 200 — runnable turns false, issues gains an error with code workflow_etag_mismatch, and the response’s own workflow_etag is the current value, so a client can re-read the interface and try again. Pass null to skip the check. execution_target picks where the run would go — {"kind": "local"}, {"kind": "worker", "worker_id": …}, or {"kind": "runner", "runner_id": …} — and defaults to local. Every other field is required, including workspace_id and inputs, which take null and {} rather than being omitted. A body missing one is a 400 ({"code": "INVALID_REQUEST", "message": "Invalid preflight request"}). Uploading an Execution Input POST /api/sdk/v1/assets/temporary puts one file where a workflow run can read it, without creating an asset row, a thumbnail, or anything the user will later have to clean out of their library. Use it for a run’s inputs; use POST /api/assets when the file belongs in the asset library. The request is multipart/form-data with the file in a field named file: curl -X POST "http://localhost:7777/api/sdk/v1/assets/temporary" \ -H "Authorization: Bearer YOUR_TOKEN" \ -F "file=@sample.txt;type=text/plain" { "version": 1, "uri": "/api/storage/temp/sdk-inputs/7d0571c6-6d9c-4c3c-9cc2-292f2b0a8574.txt", "name": "sample.txt", "content_type": "text/plain", "size": 15, "expires_at": null } Pass uri as the value of the workflow input that takes the file. expires_at is null when the configured temporary store sets no expiry — retention is the store’s, not this route’s, so nothing here promises the file survives until the run starts. A body that is not multipart, or multipart without a file field, is a 400 — {"code": "INVALID_REQUEST", "message": "Expected multipart/form-data"} or "Multipart field 'file' is required". A file over the server’s upload limit — limits.max_upload_bytes from the capabilities route, 1 GiB by default — is a 413 ({"code": "UPLOAD_TOO_LARGE", …}). Running a Saved JS Script POST /api/js-scripts/{id}/run executes a stored JS script document in the QuickJS sandbox and returns what it emitted. The web editor reads and writes scripts over /trpc/jsScripts.*; this is the one plain-HTTP door, for a run console, a mini app, or a client driving a script it did not author. inputs is keyed by the script’s declared input ports. For a script declaring numbers whose body is: const total = inputs.numbers.reduce((a, b) => a + b, 0); await output("total", total); curl -X POST "http://localhost:7777/api/js-scripts/<script_id>/run" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"inputs": {"numbers": [1, 2, 3, 4]}}' { "ok": true, "logs": [], "duration_ms": 41, "outputs": { "total": 10 }, "streamed": [] } outputs is keyed by the script’s declared output ports, and streamed holds its emit calls in order — empty here because this body makes none. A body that pulls its inputs with stream rather than reading the inputs object is fed through input_streams, which stages a list of items per input handle. For a script whose body is: let total = 0; for await (const n of stream("numbers")) { total += n; await emit("running", total); } await output("total", total); curl -X POST "http://localhost:7777/api/js-scripts/<script_id>/run" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"inputs": {}, "input_streams": {"numbers": [1, 2, 3, 4]}}' { "ok": true, "logs": [], "duration_ms": 55, "outputs": { "total": 10 }, "streamed": [ { "name": "running", "value": 1 }, { "name": "running", "value": 3 }, { "name": "running", "value": 6 }, { "name": "running", "value": 10 } ] } The two are not interchangeable: staging items for a body that reads inputs.numbers leaves that name undefined, and the run fails. A handle the script does not declare as an input is a 400: { "detail": "input_streams names nope, which this script does not declare as inputs" } The script runs inside its own envelope — the packs its body imports, its declared secrets, and its own timeoutSeconds — so nothing in the request widens what it may reach. A body that throws is not an HTTP error: the response is 200 with ok: false and the message in error, because a script that failed is a result to show, not a transport failure. { "ok": false, "logs": [], "duration_ms": 29, "error": "TypeError: cannot read property 'reduce' of undefined" } Scripts are per-user. An id belonging to someone else is a 404 ({"detail": "JS script not found"}), the same answer as an id that does not exist. The response is plain JSON — there is no streaming surface here, so progress() calls do not arrive as they happen. Listing Models an SDK Client Can Use GET /api/sdk/v1/models is the model catalog behind the SDK’s discovery profile: every model the server knows about, what state it is in, and the exact value to put in a node’s model property. Reach for it when a client has to offer a model picker without hard-coding ids. curl "http://localhost:7777/api/sdk/v1/models?availability=downloadable&limit=1" \ -H "Authorization: Bearer YOUR_TOKEN" { "version": "1", "catalog_revision": "7db91c5aa30bc13588a9e4437d7ff7cabe88d82fbfc8049c8d507825326a10b6", "scope": "local", "entries": [ { "key": "llama_cpp_model||ggml-org/gemma-3-12b-it-GGUF|gemma-3-12b-it-Q4_K_M.gguf", "display_name": "Gemma 3 12B IT (GGUF)", "compatibility": "llama_cpp_model", "availability": "downloadable", "recommended": true, "scope": "local", "provider": null, "id": "ggml-org/gemma-3-12b-it-GGUF:gemma-3-12b-it-Q4_K_M.gguf", "repo_id": "ggml-org/gemma-3-12b-it-GGUF", "path": "gemma-3-12b-it-Q4_K_M.gguf", "supported_tasks": [], "size_on_disk": 7838315315, "wire_value": { "type": "llama_cpp_model", "repo_id": "ggml-org/gemma-3-12b-it-GGUF", "path": "gemma-3-12b-it-Q4_K_M.gguf" } } ], "next_cursor": "llama_cpp_model||ggml-org/gemma-3-12b-it-GGUF|gemma-3-12b-it-Q4_K_M.gguf" } wire_value is the whole point of an entry: assign it to a node’s model property and the graph runs. availability is one of five values, decided in this order: a download in flight is downloading, a model already in the local cache is ready_local, a remote provider’s model is ready_remote once that provider is configured, and a recommended repository model (Hugging Face or GGUF) you have not fetched yet is downloadable. Everything else — including a remote model whose provider has no API key — is unavailable. Query parameters, all optional: Parameter Meaning compatibility Keep only entries of one node-property type, e.g. llama_cpp_model availability One of the five availability values provider Keep only one provider’s models, e.g. ollama scope local (default) or worker cursor The previous page’s next_cursor limit 1–500, default 200 Page by passing next_cursor back as cursor; it is null on the last page. catalog_revision hashes the entries a query produced, so a client can tell whether anything moved without diffing the list. A value outside the allowed set is a 400 ({"code": "INVALID_INPUT", …}), not a silently ignored filter. scope=worker reads the models cached on the attached Python worker instead of the server’s own. With no worker attached that is a 501: { "code": "MODEL_SCOPE_UNAVAILABLE", "message": "No worker is attached to this server.", "detail": "No worker is attached to this server.", "retryable": true } Downloading a Model over the SDK Routes Three routes cover the download lifecycle without a WebSocket: start one, poll the snapshot, cancel it. They drive the same download manager the /ws/download WebSocket does, so a headless client fetches a model exactly as the editor’s model manager would — it just polls instead of subscribing. POST /api/sdk/v1/model-downloads starts a download and answers 202 with the operation’s first state: curl -X POST "http://localhost:7777/api/sdk/v1/model-downloads" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"repo_id": "hf-internal-testing/tiny-random-gpt2", "model_type": "hf.text_generation"}' { "version": "1", "operation_id": "mdl_nY06SIV5ikFSqVqIvWOHQqIKQjJtkikk0VpLuRZszTc", "scope": "local", "repo_id": "hf-internal-testing/tiny-random-gpt2", "path": null, "model_type": "hf.text_generation", "status": "start", "downloaded_bytes": 0, "total_bytes": 0, "downloaded_files": 0, "current_files": [], "total_files": 0, "error": null, "started_at": "2026-08-06T08:16:40.600Z", "updated_at": "2026-08-06T08:16:40.600Z" } repo_id and model_type are required. path fetches a single file from the repo — a .gguf weight, say — and cannot be combined with allow_patterns or ignore_patterns, which otherwise take glob lists to narrow a whole-repo fetch. scope defaults to local; worker hands the download to the attached Python worker. The operation_id is derived from the request, so starting the same download twice returns the state already in flight rather than a second run. GET /api/sdk/v1/model-downloads is the snapshot, newest update first. It takes an optional scope (default local) and operation_id to narrow to one: curl "http://localhost:7777/api/sdk/v1/model-downloads" \ -H "Authorization: Bearer YOUR_TOKEN" { "version": "1", "downloads": [ { "version": "1", "operation_id": "mdl_nY06SIV5ikFSqVqIvWOHQqIKQjJtkikk0VpLuRZszTc", "scope": "local", "repo_id": "hf-internal-testing/tiny-random-gpt2", "path": null, "model_type": "hf.text_generation", "status": "cancelled", "downloaded_bytes": 0, "total_bytes": 488544, "downloaded_files": 0, "current_files": [], "total_files": 8, "error": null, "started_at": "2026-08-06T08:16:40.600Z", "updated_at": "2026-08-06T08:16:48.638Z" } ] } status walks start → progress → completed, or ends at error or cancelled; error carries the message when it does. Finished operations stay in the snapshot so a client that reconnects can see how a download ended — the oldest terminal ones are dropped past 200 retained operations. POST /api/sdk/v1/model-downloads/cancel stops one and returns its final state: curl -X POST "http://localhost:7777/api/sdk/v1/model-downloads/cancel" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"operation_id": "mdl_nY06SIV5ikFSqVqIvWOHQqIKQjJtkikk0VpLuRZszTc"}' Cancelling an operation that already finished returns the state it settled in. An id this caller never started is a 404 ({"code": "MODEL_DOWNLOAD_NOT_FOUND", …}). Ollama models are pulled by Ollama itself, so "model_type": "llama_model" is a 501 ({"code": "MODEL_DOWNLOAD_UNAVAILABLE", …}). Downloading the Chrome Extension GET /api/extension/download zips up the browser extension build the server can find and hands it over, so you can load it unpacked without cloning the repo. curl "http://localhost:7777/api/extension/download" \ -H "Authorization: Bearer YOUR_TOKEN" \ -o nodetool-chrome-extension.zip The response is application/zip with content-disposition: attachment; filename="nodetool-chrome-extension.zip". The server looks for the build at NODETOOL_EXTENSION_DIST (set by the desktop app to its bundled copy), then walks up from its own directory and the working directory looking for chrome-extension/dist/manifest.json. When none of those holds a build, the answer is 404 with {"detail": "Extension build not found"} — build it first, per Chrome Extension. How This Server Is Configured A client has to decide whether to show a login screen before it has a token, so GET /api/config needs none. It reports the auth mode the server picked and the values a browser needs to complete a sign-in. curl "http://localhost:7777/api/config" { "authMode": "local", "supabaseUrl": null, "supabaseAnonKey": null, "authRedirectUrl": null, "googleWorkspace": false, "googleScopes": [], "version": "0.7.0-rc.38" } authMode is "supabase" when both SUPABASE_URL and SUPABASE_KEY are set, and "local" otherwise — the same choice the server’s own auth hook makes, so a client never has to guess it from the other fields. supabaseUrl, supabaseAnonKey and authRedirectUrl mirror SUPABASE_URL, SUPABASE_ANON_KEY and AUTH_REDIRECT_URL; each is null when unset. Note the anon key is the public one — the service key (SUPABASE_KEY) is never in this response. googleWorkspace says whether the Google Workspace capability is on (see NODETOOL_GOOGLE_WORKSPACE in Configuration). When it is, googleScopes lists the OAuth scopes the connect step must request; when it is not, that array is empty. Exporting or Erasing Your Account Data Two routes answer the data-subject rights the privacy policy promises: GET /api/account/export hands back everything this server holds about the caller, and DELETE /api/account erases it. Neither takes a subject — the user id comes from the identity the auth hook resolved, so a request carrying none is a 401 rather than falling back to the local user 1. There is no admin path and no way to name somebody else’s account, in the path, the query or the body. curl "http://localhost:7777/api/account/export" \ -H "Authorization: Bearer YOUR_TOKEN" \ -O -J The response is application/json with cache-control: no-store and a content-disposition of attachment; filename="nodetool-personal-data-export-2026-06-20.json", dated the day it was generated, so curl -OJ writes it under that name. Inside is one entry per table: { "format": "nodetool.personal-data-export/1", "subjectUserId": "1", "generatedAt": "2026-06-20T09:12:33.921Z", "tables": { "nodetool_workflows": { "disposition": "delete", "reach": { "kind": "direct", "column": "user_id" }, "rowCount": 12, "truncated": false, "withheldColumns": [], "rows": [{ "id": "wf_9f3c", "name": "Daily digest", "graph": "…" }] }, "nodetool_secrets": { "disposition": "delete", "reach": { "kind": "direct", "column": "user_id" }, "rowCount": 3, "truncated": false, "withheldColumns": ["encrypted_value"], "rows": [ { "key": "OPENAI_API_KEY", "encrypted_value": "[withheld: credential material]" } ] } }, "excluded": [ { "table": "mcp_oauth_clients", "reason": "A dynamically registered OAuth client (RFC 7591): name plus redirect URIs, with no owner column. …" } ] } disposition is what erasure would do with that table — delete, redact, retain, anonymize, or not-personal — and reach is how the rows are found, either a direct user-id column or an indirect join through a parent table. withheldColumns names the columns replaced with the string [withheld: credential material]: an API key, an OAuth token or a token hash is listed by name and never by value, so the export says which credentials existed without minting a fresh readable copy of a live one. Each table is capped at 50 000 rows and sets truncated: true when it hits the cap. excluded lists the tables holding nothing portable for this person, each with the reason from the registry. Which table gets which treatment is PERSONAL_DATA_REGISTRY in packages/models/src/personal-data-registry.ts. Erasure is irreversible, so it fires only on the exact phrase: curl -X DELETE "http://localhost:7777/api/account" \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"confirm": "DELETE MY ACCOUNT"}' Anything else — no body, {}, a different string, or an extra field such as user_id — is a 400 and nothing is touched, so a stray or replayed DELETE cannot erase an account: { "detail": "Erasure is irreversible. Send {\"confirm\": \"DELETE MY ACCOUNT\"} to confirm." } A confirmed call answers 200 with the request id that ties the audit rows together and a per-table report: { "requestId": "8629a9bc-052f-4af9-96fd-56779be0d94e", "report": { "userId": "1", "completedAt": "2026-06-20T09:14:02.550Z", "tables": [ { "table": "nodetool_workflows", "disposition": "delete", "deleted": 12, "redacted": 0, "retained": 0 }, { "table": "nodetool_predictions", "disposition": "redact", "deleted": 0, "redacted": 340, "retained": 340 } ], "deleted": 812, "redacted": 340, "retained": 353, "objectKeysDeleted": ["1/9f3c1a2b.png", "1/a04ef881.mp4"] } } A redact row is kept and stripped: nodetool_predictions is the billing ledger, so cost, tokens, model, provider and timestamps survive while parameters, metadata and logs are nulled. objectKeysDeleted names the stored objects removed under the caller’s <userId>/ prefix, so the bytes go with the rows rather than staying readable to anyone holding a URL. The call is idempotent — a second one re-runs the same sweep, finds nothing, and answers 200 with zeros. The data_erasure_requested and data_erasure_completed events survive the sweep as the record that the request arrived and was answered. Provider Sign-in /api/oauth/* runs the credential flows for outside accounts. Only the two browser redirect targets skip session auth, because the identity provider sends the user back with no Authorization header. Every other path reads or changes credentials stored for the caller. Methods that do not match answer 405. Provider Routes Hugging Face GET /api/oauth/hf/start (returns {"auth_url": ...}), GET /api/oauth/hf/callback (public), GET /api/oauth/hf/tokens, POST /api/oauth/hf/refresh, GET /api/oauth/hf/whoami GitHub GET /api/oauth/github/start, GET /api/oauth/github/callback (public), GET /api/oauth/github/tokens, GET /api/oauth/github/user OpenAI (Codex) GET /api/oauth/openai/start, POST /api/oauth/openai/complete, GET /api/oauth/openai/tokens, POST /api/oauth/openai/disconnect Claude subscription GET /api/oauth/claude/start, POST /api/oauth/claude/complete, GET /api/oauth/claude/tokens, POST /api/oauth/claude/disconnect Google Workspace POST /api/oauth/google/session, GET /api/oauth/google/tokens, POST /api/oauth/google/disconnect. The session route takes the Supabase Google login’s access_token and answers 404 unless NODETOOL_GOOGLE_WORKSPACE is on The Claude subscription login is one per server, shared by every user’s Claude Agent SDK calls. Only an admin may start, complete or disconnect it, and other users get 403. An admin is user "1" in Local mode, an id listed in ADMIN_USER_IDS, or an API user with the admin role. For other users, tokens reports whether the server is connected but leaves out the plan and the credentials path. Hugging Face and GitHub use PKCE with a single-use state. The start route builds the redirect URI from the request’s Host header. Token routes return metadata (scope, expiry), never the token itself. Any other path under /api/oauth/ is a 404. Streaming a Local File GET /api/files/local?path=<absolute path> streams one file from the machine the server runs on, with Range support. The editor uses it to preview file:// media. The path must resolve inside a configured root (the home directory by default), both lexically and after symlinks. A missing path or a directory is a 400, a path outside the roots is a 403, and a missing file is a 404. With NODETOOL_ENV=production every request is a 403. Indexing a File into a Collection POST /api/collections/{name}/index takes a multipart/form-data body with one file field, splits the text into chunks, and adds them to the vector collection. The response is {"path": "<file name>", "chunks": 12, "error": null}. A collection that does not exist or belongs to someone else is a 404, a body that is not multipart or has no file is a 400, and a file over NODETOOL_MAX_UPLOAD_BYTES is a 413. Create, list, query, and delete are tRPC procedures on collections. Extracting Text from a Document POST /api/documents/extract-text reads one PDF or DOCX sent as the file field of a multipart body and returns {"text": "...", "pages": 3} (pages only for PDF). Nothing is stored. The limit is 25 MB (413). The type comes from the content type, or from the file extension when the content type is application/octet-stream. Another type is a 415. A file that cannot be parsed, or a PDF or DOCX with no text, is a 422. Script and screenplay imports use it. Timeline Operations Four routes run timeline edits that need the server. The first three read the stored sequence, call a capability, save the result, and return it. The editor then reloads the document. A refusal from the capability, such as a missing clip, is a 400 with {"detail": "..."}. A body that fails validation is also a 400 and the detail names the field. Route Body What it does POST /api/timelines/{id}/track-object clip_id, initial_region (x, y, width, height), start_ms, end_ms, optional provider, model, name, direction (forward, backward, both), track_id, regenerate Tracks a subject through a clip and stores the track POST /api/timelines/{id}/bake-audio-animation audio_clip_id, target_clip_id, property (scale, opacity, offsetX, offsetY), output_range ([quiet, loud]), optional mode (envelope, beats), sensitivity, attack_ms, release_ms, offset_ms, tolerance, max_points, frame_ms, max_seconds, replace Turns an audio clip’s loudness or beats into keyframes on the target clip POST /api/timelines/{id}/isolate-subject clip_id, optional model (General Use (Light), General Use (Light 2K), General Use (Heavy), Matting, Portrait, General Use (Dynamic)), operating_resolution (1024x1024, 2048x2048, 2304x2304), refine_foreground, regenerate, background Generates a matte for the clip’s subject. A generation that fails answers 200 with status: "failed" and keeps the previous matte POST /api/timelines/animations/bake code or script_id (exactly one), role (in, out, emphasis, loop), duration_ms, clip_duration_ms, canvas (width, height), optional params, stagger_count, sample_count Runs a custom animation body once in the sandbox, with no secrets and no toolbelt, and returns {ok, curves, logs, duration_ms}. A script_id that is not the caller’s is a 404 Deployed App Routes GET /api/apps/{token} returns a deployed mini app’s released document and POST /api/apps/{token}/session mints the nda_ token its visitor runs on over /ws. The deployment token in the path is the credential, so neither route uses session auth. Both answer 404 with {"detail": "This app is not available"} for an unknown or revoked token, and for every request outside production. The GET response is gzip-compressed when the client accepts it. Public Workflow Links GET /api/shared-workflows/{token} returns the name, description, tags, and graph of the workflow behind a public share link. The owner mints the link with workflows.sharing.createLink and role: "public", and revokes it with workflows.sharing.revokeLink. The token is the credential, so the route uses no session auth. It answers 404 with {"detail": "This workflow is not available"} for an unknown, revoked, view, or edit token. A public link grants no role. A signed-in user copies the workflow into their own workflows with the workflows.sharing.duplicatePublic tRPC mutation, which creates a new private workflow. Provider Callbacks Three routes receive results from providers that call back instead of making the server poll. Route Authentication Behavior POST /api/kie/webhook None Reads taskId (or task_id, also under data) and the status, resolves or rejects the waiting run, and answers {"status": "accepted"}. A body with no task id is a 400 POST /api/providers/fal/webhook/{token} Ed25519 signature over the raw body, in x-fal-webhook-* headers Verifies the signature and request id, records the result in the durable generation inbox, and answers {"status": "accepted", "duplicate": false}. A bad signature is a 400. A transient verification or inbox failure is a 503 so fal retries POST /api/providers/atlascloud/webhook Ed25519 signature over the raw body, in x-atlascloud-webhook-* headers Same pattern for AtlasCloud predictions The fal and AtlasCloud routes allow 120 requests per minute. Sandbox Modules GET /api/sandbox-modules/{module id} serves the source of a guest module to the browser sandbox. The id is a pack entry specifier such as @acme/geo, or an internal file id such as @acme/geo::sandbox/helper.js. The response carries the module as its body with an ETag, Cache-Control: private, no-cache, and headers that name its digest, SHA-256, pack, file id, and dependency list. An unknown id is a 404, and a module the caller may not receive is a 403. MCP over HTTP /mcp is the Model Context Protocol endpoint. It binds the user the auth hook resolved, so an agent presents an ntk_ access token or an nta_ OAuth token. Without a user, initialize answers 401. In production it registers only with NODETOOL_ENABLE_MCP=1. See MCP OAuth Discovery and MCP in production. Health Check # Check if server is running (no auth required) curl "http://localhost:7777/health" Response: { "status": "ok", "timestamp": "2026-06-20T00:00:00.000Z", "uptime": 123, "turns": 0, "jobs": 0, "services": { "database": "ok", "server": "ok" } } turns and jobs count the chat turns and workflow runs this process is still executing. status is "degraded" with a 503 when the database check fails, and "draining" with a 503 after the server receives SIGUSR2. See Draining. CLI Workflow Execution You can also run workflows from the command line. nodetool run executes a TypeScript/JavaScript DSL workflow file: # Run a DSL workflow file nodetool run ./my_workflow.ts # Output results as JSON nodetool run ./my_workflow.ts --json To run a saved workflow by ID (requires a running server), use the workflows subcommands: # List workflows nodetool workflows list # Run a workflow by ID nodetool workflows run workflow_abc123 --params '{"prompt": "test"}' TypeScript / Node.js Client Example const BASE_URL = 'http://localhost:7777'; const TOKEN = 'your_token_here'; // Run a workflow (runs to completion, returns one JSON response) async function runWorkflow(workflowId, params) { const response = await fetch(`${BASE_URL}/api/workflows/${workflowId}/run`, { method: 'POST', headers: { 'Authorization': `Bearer ${TOKEN}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ params }) }); const body = await response.json(); // body: { job_id, workflow_id, status, outputs, error, message_count, background } return body.outputs; } // For real-time progress, run the workflow over the WebSocket endpoint instead. // See the WebSocket API page and examples/workflow_runner/js/workflow-runner.js. // Using OpenAI SDK (works with NodeTool!) import OpenAI from 'openai'; const openai = new OpenAI({ apiKey: TOKEN, baseURL: `${BASE_URL}/v1` }); const completion = await openai.chat.completions.create({ model: 'gpt-5.6', messages: [{ role: 'user', content: 'Hello!' }] }); console.log(completion.choices[0].message.content); Python Client Example import requests BASE_URL = "http://localhost:7777" TOKEN = "your_token_here" # Not needed for local development HEADERS = { "Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json", } # List workflows workflows = requests.get(f"{BASE_URL}/api/workflows", headers=HEADERS).json()["workflows"] # Run a workflow (runs to completion, returns one JSON response) result = requests.post( f"{BASE_URL}/api/workflows/{workflows[0]['id']}/run", headers=HEADERS, json={"params": {"prompt": "A sunset over mountains"}}, ).json() # result: {"job_id", "workflow_id", "status", "outputs", "error", "message_count", "background"} print("Outputs:", result["outputs"]) # For real-time progress, run the workflow over the WebSocket endpoint instead. # Use with OpenAI Python SDK (works with NodeTool!) from openai import OpenAI client = OpenAI(api_key=TOKEN, base_url=f"{BASE_URL}/v1") completion = client.chat.completions.create( model="gpt-5.6", messages=[{"role": "user", "content": "Hello!"}], ) print(completion.choices[0].message.content) Finding Your Workflow ID To run a workflow via API, you need its ID. Here’s how to find it: From the UI: Open a workflow in the editor — the ID appears in the browser URL bar From the API: Call GET /api/workflows to list all workflows with their IDs From the CLI: Run nodetool workflows list Error Handling API errors return standard HTTP status codes. The body shape depends on the route family: Family Body Most REST routes {"detail": "Workflow not found"}, sometimes with a code such as INVALID_INPUT /api/sdk/v1/* {"code": "INVALID_REQUEST", "message": "..."} /v1/* OpenAI shape, {"error": {"message", "type", "param", "code"}} Webhook and provider callbacks {"error": "..."} /trpc tRPC error object with data.apiCode and data.zodError Status Code Meaning Common Causes 400 Bad Request Invalid parameters, malformed JSON, a graph the server refuses to run 401 Unauthorized Missing or invalid token 403 Forbidden Production guard on local-file and workspace routes, path traversal 404 Not Found Resource missing, or owned by someone else 409 Conflict A workflow save with a stale expected_updated_at 422 Unprocessable A document or media file that cannot be read 429 Too Many Requests Rate limit, or too many live debug sessions 500 Internal Error Server-side error 501 Not Implemented Stub routes such as bulk asset download 503 Service Unavailable A feature flag is off, ffmpeg is missing, or the server is draining Related Guides Chat API — OpenAI-compatible request/response schema and WebSocket usage. Workflow API — Workflow REST paths and execution. API Server Overview — Server architecture and modules. Deployment Guide — How servers are built and exposed. CLI Reference — Commands including serve.--- # API Server Overview Source: https://docs.nodetool.ai/api-server.html NodeTool exposes a single TypeScript HTTP + WebSocket server runtime built on Node.js. The same process serves REST API routes, tRPC procedures under /trpc, WebSocket workflow execution at /ws, the /mcp endpoint, and OpenAI-compatible /v1 routes. It also serves the built web app when one is present. The server is implemented in the @nodetool-ai/websocket package (packages/websocket/src/server.ts). Key Modules server.ts – Fastify entry point. Registers CORS, rate limiting, auth, the WebSocket plugin, tRPC, and the route plugins, then starts the listener. routes/ – Fastify route plugins, one per area (health, workflows, assets, nodes, storage, files, collections, applications, timelines, storyboards, OAuth, provider webhooks, /v1). trpc/ – The tRPC router mounted at /trpc. JSON list, metadata, and delete operations live here. websocket-client-session.ts – Handles WebSocket connections for workflow execution and chat. http-api.ts – Shared request handler and options (handleApiRequest, getUserId) that route plugins call. models-api.ts – Model management and provider registration. settings-registry.ts – Reads setting values from the database. The setting definitions live in @nodetool-ai/config. storage-api.ts – Binary GET and HEAD for stored assets. mcp-server.ts – Model Context Protocol (MCP) server integration, mounted at /mcp. openai-api.ts – OpenAI-compatible /v1/chat/completions and /v1/models endpoints. Running the Server # Install the CLI globally (once) npm install -g @nodetool-ai/cli # Start the server nodetool serve --host 127.0.0.1 --port 7777 # Or run without installing globally npx --package=@nodetool-ai/cli nodetool serve --host 0.0.0.0 --port 7777 Development (from repo root): npm run build:packages npm run dev:server # scripts/dev-server.mjs: runs the server from TypeScript source under tsx watch Configuration The server is configured via environment variables: Variable Default Description PORT 7777 HTTP listen port HOST 127.0.0.1 (0.0.0.0 when NODETOOL_ENV=production) Bind address NODETOOL_ENV — Set to production for production defaults. The /mcp mount is then off unless NODETOOL_ENABLE_MCP=1. DB_PATH ~/.local/share/nodetool/nodetool.sqlite3 SQLite database path. Do not set together with DATABASE_URL. DATABASE_URL — PostgreSQL URL (postgres:// / postgresql://) or SQLite URL/path (file: / sqlite:) ANTHROPIC_API_KEY — Anthropic API key OPENAI_API_KEY — OpenAI API key GEMINI_API_KEY — Google Gemini API key OLLAMA_API_URL http://127.0.0.1:11434 Ollama server URL nodetool serve --host <host> --port <port> sets HOST and PORT for the server process and overrides any value already in the environment. See Configuration for the full list. Health Check GET /health Returns 200 OK when healthy and 503 when the database check fails ("status": "degraded") or the server is draining for a rolling deploy ("status": "draining"). The body: { "status": "ok", "timestamp": "2026-06-20T00:00:00.000Z", "uptime": 123, "turns": 0, "jobs": 0, "services": { "database": "ok", "server": "ok" } } turns and jobs count chat turns and workflow runs in flight. A rolling deploy waits for both to reach 0 before it replaces the machine. GET /ready is a simple liveness probe (always 200 with { "status": "ok" }), and GET /api/health returns { version, uptime }. None require authentication. For deployment setup, see Deployment Guide and Authentication.--- # Apify: external capabilities for the sandbox Source: https://docs.nodetool.ai/apify-integration.html Apify: external capabilities for the sandbox NodeTool’s sandbox has no network, no browser, no filesystem outside the workspace, and no way to install a package. That is the point of it. But plenty of real work needs exactly those things — read this site, drive this JavaScript-heavy page, find these businesses, pull this video’s transcript. Apify closes that gap without opening the sandbox. Actors run on Apify’s machines; NodeTool exposes a typed, permission-aware, credential-free interface to them, converts what comes back into ordinary NodeTool values and assets, and never lets the guest touch the network or the token. This is deliberately one generic capability, not a shelf of scrapers. There is no YouTubeDownloader node, no InstagramScraper node. There is run_apify_actor, plus a shipped catalog of actors that are known to work, plus the discovery calls an agent needs to find and understand an actor it has not seen before. Setting it up Get a token at https://console.apify.com/account/integrations. Settings → Secrets → APIFY_API_TOKEN. APIFY_API_KEY is still read as a fallback, so an install configured before this existed keeps working. What an agent gets Eight capabilities in the apify namespace. They are agent tools and sandbox imports at the same time — one spec behind both — so nothing can drift between what a model is told and what guest code can call. The chat belt, the MCP bridge, and the belt a nodetool.code.Code node and a JS script share all carry them through getApifyTools() (packages/agents/src/tools/external-capability-tools.ts), which is what makes nodetool.searchTools("apify") find them: the search reads the belt, and a module reachable only by import is invisible to it. The Code node belt matters as much as the chat one, because the move a chat teaches is “run it here, then put the same call in a node”. It carried the tools only in chat for a while, so that second step failed with QuickJS’s TypeError: not a function — naming neither the tool nor the belt — and read as the sandbox having no reach at all. Capability Does Class search_apify_actors Find an actor by what it does read get_apify_actor One actor’s record, pricing, and whether it may run here read get_apify_actor_schema The actor’s input contract read run_apify_actor Run it external get_apify_run Poll a run started with wait_for_finish: false read abort_apify_run Stop a run external get_apify_dataset_items Page through a run’s dataset read get_apify_key_value_record Read non-tabular output (screenshots, OUTPUT) read Only the two that act are classed external. Reading the store, an input schema, or a dataset a run already produced spends nothing and changes nothing. From sandbox code import { get_apify_actor_schema, run_apify_actor } from "@nodetool-ai/sandbox-nodetool/apify"; // Read the contract before constructing input for an unfamiliar actor. const schema = await get_apify_actor_schema({ actor_id: "apify/website-content-crawler" }); const result = await run_apify_actor({ actor_id: "apify/website-content-crawler", input: { startUrls: [{ url: "https://example.com" }], maxCrawlPages: 3 } }); return result.dataset.items.map((page) => page.markdown); Exports carry the wire name — run_apify_actor, not runActor — because the prompt, the MCP surface, tools.* and the import all name one string. No token, no fetch, no polling loop. The host does all three. The intended loop search_apify_actors("google maps businesses") → get_apify_actor_schema("compass/google-maps-extractor") → run_apify_actor({actor_id, input}) → dataset preview + ids to read the rest The middle step is not optional politeness. Every actor names its input fields differently, and a wrong key is not an error — it starts a run that succeeds and returns an empty dataset, having charged for it. The tool descriptions say so, and say not to invent actor ids. Costs differ by orders of magnitude between actors that look interchangeable, so the descriptions also steer to the cheapest thing that works: a content crawl beats an HTML scrape beats a full browser run. Keep the three ideas apart — search finds URLs, crawl traverses pages, browser interacts with one. Permissions Four modes, set with NODETOOL_APIFY_MODE: Mode Store search Runs disabled — nothing allowlist the shipped catalog only allowlisted actors discovery (default) the whole store allowlisted directly; anything else asks the user unrestricted the whole store anything NODETOOL_APIFY_ALLOWED_ACTORS is a comma-separated list added to the shipped catalog, not replacing it, because the usual intent is “the normal ones plus mine”. In allowlist mode, search_apify_actors answers from the shipped catalog instead of the store and says so. Listing actors a model cannot run reliably produces a plan built on one of them. Approval in discovery mode goes through NodeTool’s ordinary permission gate, so an actor prompt looks like every other prompt and “allow for this chat” works the way it does elsewhere. The shipped catalog The default allowlist, chosen for being first-party or the long-standing community actor for a capability with no official equivalent, with a documented input schema: Actor For apify/website-content-crawler Crawl a site to Markdown — the default for reading pages apify/cheerio-scraper Fast HTML-only extraction apify/web-scraper Browser-rendered extraction apify/playwright-scraper Full browser automation apify/google-search-scraper Search result pages compass/google-maps-extractor Businesses and places apify/instagram-scraper Public Instagram data trudax/reddit-scraper-lite Public Reddit data apify/screenshot-url Page screenshots streamers/youtube-scraper YouTube metadata and subtitles Prices move, so the catalog records only the shape of the bill (per-result, per-page, per-event, compute-time). get_apify_actor reads the live figure from the store. Cost control Runs cost money and an autonomous agent can loop, so a session carries a budget (a chat turn, a workflow execution, or a Code node action each get their own): Limit Default Env Runs per session 10 NODETOOL_APIFY_MAX_RUNS Dataset items per run 1000 NODETOOL_APIFY_MAX_ITEMS Seconds per run 300 NODETOOL_APIFY_MAX_RUN_SECONDS MB per run 4096 NODETOOL_APIFY_MAX_MEMORY_MB USD per session 5 NODETOOL_APIFY_MAX_COST_USD Caller-supplied options are clamped down to these, never up. The policy decides before the budget is charged and the budget is claimed before the run starts, so a refused actor costs nothing and a refused budget starts nothing. Every run_apify_actor result reports the session’s remaining runs and spend. Cancellation Stopping a workflow or an agent aborts the actor, it does not merely abandon the wait. That is why NodeTool starts a run and polls it rather than using Apify’s synchronous endpoint: the run id exists from the first response, so there is something concrete to abort. The abort deliberately runs on a fresh path — the obvious version, passing the caller’s signal to the abort request, cancels the cleanup with the thing it is cleaning up. Aborting a run that already finished does nothing and is not an error, so cancellation is idempotent and safe to race. Results and assets A run comes back as a preview of its dataset plus the ids needed to read the rest — never the whole thing. Datasets reach six figures of rows; a summary says total: 14392, showing 20 and how to page. Two independent limits apply, a row count and a serialized byte ceiling, because rows vary by orders of magnitude between a Maps scrape and a full-page crawl. When the byte limit bites, whole rows are dropped, never partial ones, and the result says so. Files are different. Actors return URLs into Apify’s storage and those URLs expire, so binary output is fetched by the host through NodeTool’s SSRF-screened safeFetch and written to NodeTool storage, capped at 100 MB and cut off mid- download if a server declares no length. Text and JSON come back inline — storing a copy of a string just creates a second handle on it. An import that fails reports itself and leaves the remote URL in place rather than discarding a run that has already been paid for. Two paths do the import. run_apify_actor walks its dataset preview for URLs into Apify’s key-value storage — every actor names its file field differently (downloadedFileUrl, screenshotUrl, …), so it is the URL’s shape that identifies a produced file — and imports up to five of them, returned as files with an asset_url each. get_apify_key_value_record imports a binary record on read. Both give storage URLs; keeping one in the asset library is save_asset({name, source: asset_url}), which copies host-side. The model is told so in the result, because the alternative it reaches for otherwise is read_asset → 600 KB of base64 → save_asset. Every result carries provenance: actor id, run id, retrieval time, dataset and key-value store ids, status, and cost. It carries no token and no actor input, since an input can hold something a user typed. Security The sandbox never gains what Apify has. The token lives in one file (packages/agents/src/apify/client.ts), is attached to requests and never to results, and is scrubbed from every error before it is thrown. Guest code calls run_apify_actor and receives normalized values. Actor inputs are screened for SSRF before a run starts. The walk covers the whole input at any depth rather than a known set of URL fields, because every actor names its URL field differently and a check that only knows startUrls is one a model routes around by accident. Loopback, RFC1918, link-local, CGNAT, cloud metadata, and the IPv6 forms that smuggle an IPv4 address are all refused; plain http to a public host is allowed, since actors legitimately crawl such sites. Apify running on Apify’s machines does not make this moot: a self-hosted NodeTool reachable from the internet, or an internal hostname that resolves inside a customer VPC, are both real targets. Everything an actor returns is attacker-influenced text about to enter a model’s context. Scraped pages carry whatever the page author wrote, including instructions aimed at your agent. Treat actor output as data, keep previews small, and prefer structured extraction over dumping whole pages. Errors Failures are classified rather than surfaced as raw HTTP: auth, disabled, actor_not_found, actor_not_allowed, invalid_input, run_failed, run_timed_out, run_aborted, dataset_failed, asset_download_failed, rate_limited, budget_exceeded, network, cancelled. Only rate_limited and network are retried. Retrying a rejected token or an input the actor’s schema refuses spends the same failure twice, and a run is never retried automatically, because a second start is a second charge. Example workflows Website research — crawl to Markdown, summarize: const result = await run_apify_actor({ actor_id: "apify/website-content-crawler", input: { startUrls: [{ url }], maxCrawlPages: 20 } }); Business discovery — Maps to a table: const result = await run_apify_actor({ actor_id: "compass/google-maps-extractor", input: { searchStringsArray: ["AI startups"], locationQuery: "Amsterdam" } }); Competitive research — an agent composes search → crawl → extract → compare across sites NodeTool has no integration for, which is the case the generic primitive exists to serve. Limitations No dedicated NodeTool nodes yet. The capability is reachable from agents and from sandbox code; a graph node wrapping it, with a schema-driven configuration UI generated from get_apify_actor_schema, is the obvious next step. No streaming. A run’s dataset is read after it finishes, so a long crawl reports nothing until it settles. wait_for_finish: false plus polling is the workaround. The budget is per capability run, not per user or per day. A user who starts many chat turns can spend many budgets. Actor pricing is read from the store record, which reports the pricing model rather than a predicted cost for a specific input. The SSRF screen resolves names at Apify’s end, not ours, so DNS rebinding is not covered. It is defense in depth, not a complete mitigation.--- # Design: Mini-App Harness in the App Builder Assistant Source: https://docs.nodetool.ai/app-builder-assistant-harness-integration.html Design: Mini-App Harness in the App Builder Assistant Status: Draft — for review Related: mini-app-build-harness-prd.md · mini-app-build-harness-design.md · app-builder.md · nodetool app debug (AGENTS.md) The best way to give the in-editor App Builder assistant the harness’s feedback loop, and the ways that look right but aren’t. 1. The gap The build-harness PRD closed the loop for batch builds: buildApp runs spec → plan → author → check → run → judge, and nodetool app debug gives any agent the oracle on demand. The interactive assistant — the Ask Agent panel in the App Builder — got none of it. Today it: edits the document through the 19 ui_app_* tools (web/src/lib/tools/builtin/puck.ts → PuckAgentBinder → the shared doc-ops in @nodetool-ai/app-runtime); reads its own work back only via ui_app_get_snapshot and ui_app_get_binding_targets; can verify the workflow (validate_workflow / debug_workflow are resident tools) but not the app: no tool tells it whether a binding resolves, whether a click starts a run, or what a widget ends up showing. The PRD named this in §2.1: “an agent in editor chat improvises this loop turn by turn.” The oracle exists (simulateApp in @nodetool-ai/execution/app-debug); the editor assistant just can’t reach it. There is no /api/applications/:id/debug route and no app-level tool on either the server or the frontend surface. 2. The fact that decides the design The assistant edits an unsaved draft. The document lives in React state (AppBuilderShell) plus Puck’s own state; the applications row updates only when the user saves (tRPC applications.update). Any verification keyed on application_id reads the row, so it would grade a stale document while the assistant keeps editing the live one — a verdict that lies exactly when the assistant most needs it (mid-edit). So the primary integration must carry the live document inline to the server, the way validate_workflow accepts an inline graph. Everything below follows from that. 3. Design Three pieces, smallest first. Each reuses a seam the harness already cut. 3.1 Server route: POST /api/applications/debug A sibling of POST /api/applications/build, in a new packages/websocket/src/lib/app-debug-service.ts: POST /api/applications/debug { application_id?: string, // saved app — read the row document?: unknown, // OR the live draft, verbatim params?: Record<string, unknown>, interact?: InteractionStep[], // the app-debug script language run?: boolean, // false = static wiring check only (ms, free) timeout_ms?: number } → compacted AppDebugReport (§3.4) Implementation is assembly, not new machinery: Runner: extract buildRunner from app-build-service.ts (one ExecutionSession per run over an in-memory graph, nothing persisted) into a shared helper both services use. It already satisfies AppServerRunInput → AppServerRunOutcome, the only contract simulateApp has with any kernel runner. Target resolution: the document path builds a ResolvedAppTarget from the inline document with graphs loaded per operation via Workflow.find(userId, id).getGraph(). The application_id path needs the application-row resolution that today lives in packages/cli/src/app-debug/app-target.ts (resolveApplication, hostGraphFor). Move that piece into @nodetool-ai/execution/app-debug with its loaders injected — the same relocation the build-harness design did for the simulator itself, for the same reason: CLI and server surfaces must not drift. The CLI keeps file/DSL targets and re-exports. Long runs: register with debugSessions the way a build does, honoring poll for multi-minute runs. interactive: true (agent-answered escalations) composes later for free — the session machinery is shared. 3.2 Server tool: debug_app Next to debug_workflow in packages/agents/src/tools/mcp-tools.ts: a thin apiPost to the route, permission class execute (run: false requests are read-shaped, but the tool can execute workflows, so it takes the conservative class). Make it resident in RESIDENT_TOOL_NAMES (unified-websocket-runner.ts) and teach it in CHAT_AGENT_SYSTEM_PROMPT as the app half of the existing plan → validate → create → debug loop. This serves every chat surface — global chat debugging a saved app included — not just the builder panel. 3.3 Frontend tool: ui_app_debug The piece that closes the loop on the draft. One new tool in web/src/lib/tools/builtin/puck.ts: getPuckAgentHandler(application_id) — add a document() accessor to PuckAgentHandler that assembles the current working document (Puck data + meta), mirroring the headless bridge’s existing document(). POST it as the route’s inline document. Return the compacted report. Args: { application_id, run?, params?, interact? }. With run: false it is the free static check (app debug --no-run); with run: true and no interact the simulator’s defaultInteractions click the app’s natural trigger. The assistant can script multi-step flows (set, click, run, seedResource, …) in the same InteractionStep vocabulary the CLI and the build harness use — no new language. 3.4 One compacted report shape AppDebugReport is bundle-sized; a chat turn needs the verdict. Add a summarizeAppReport(report) reducer in @nodetool-ai/execution/app-debug returning: verdict (ok, headline, issues, warnings), per-widget end state (id, type, binding, hasValue, visible, disabled, display preview), invocation outcomes (status, decision, error), value previews, and notSimulated. Both debug_app and ui_app_debug return this one shape — the “summary reducer lives in execution so surfaces cannot drift” rule debug_workflow already follows. 3.5 Prompt: teach the loop, retire the folklore APP_BUILDER_SYSTEM_PROMPT (AppBuilderAgentPanel.tsx) predates operations, variables, resources, and ui_app_get_binding_targets; its bare-name binding advice survives only through the legacy fallback that resolves against operations[0] — silently wrong on any multi-operation app. Rewrite it around the harness loop: wire through ui_app_get_binding_targets, never guessed names; after any wiring change: ui_app_debug { run: false } — free, instant; before declaring the app done: ui_app_debug { run: true } once, read the verdict, fix what it names. A run executes real workflows and spends real money — check often, run once. for “build me an app” with nothing open: the same sequence from an empty document — author the workflow, declare the operation, place the widgets, grade it with ui_app_debug. There is no one-shot build tool on the belt; the batch harness stays a CLI and HTTP surface. 4. Rejected alternatives Run buildApp’s repair loop inside the assistant. The conversation is the repair loop — the user plays judge and complaint-writer. What the assistant lacks is the oracle, not the orchestration. Wiring buildApp in would put two repair loops in one session, fighting over the document. Save-then-debug as the primary path (tool reads the row, user must save first). Simple, but grades a stale document mid-edit; verdicts that sometimes describe an older draft are worse than none. The application_id path stays for saved apps; it just isn’t what the builder panel uses. Client-side simulation. The simulator’s fold is shared code (@nodetool-ai/app-runtime), but its runs need the kernel; shipping kernel execution to the browser buys nothing over one HTTP round-trip to the same server that already runs builds. A judge stage in the interactive loop. The judge exists because batch builds have no human; here the human is present. The structural verdict is the right ceiling for a tool call. 5. Milestones M1 — Route + service. Shared runner extraction, application-target relocation into execution/app-debug, POST /api/applications/debug with inline-document support, summarizeAppReport. Tests mirror app-build-route.test.ts. M2 — debug_app. Tool + resident registration + chat system-prompt paragraph. Serves all chat surfaces immediately. M3 — ui_app_debug. PuckAgentHandler.document(), the frontend tool, headless-bridge mirror (so the app-tools eval can score the check-before-done behavior), APP_BUILDER_SYSTEM_PROMPT rewrite. M4 — Composition. poll for long runs, interactive: true escalations, and an eval case that fails an assistant which declares an app done without a green run. 6. Failure modes Failure Resolution Inline document references a workflow the user can’t read Per-operation unavailable, reported in the verdict — the simulator already models this Run exceeds timeout_ms Invocation reports timedOutMs; verdict issue; no parked session Draft diverges mid-request (user edits while the tool runs) The report names the document version it graded; the tool result says “as of this snapshot” Cost runaway from repeated run: true Prompt discipline plus the same per-run timeout; runs land in the prediction ledger like every kernel run--- # App Builder Source: https://docs.nodetool.ai/app-builder.html App Builder is the Design view of an app tab. You drag widgets onto a canvas, wire them to the inputs and outputs of the workflows the app runs, and save. This page covers the editor itself. For what Mini Apps are and how they run, see Mini Apps; for step-by-step recipes, see Building Mini Apps; for every widget and setting, see the Reference. What it does Lays out the app screen: boxes, fields, buttons, and result displays. Wires widgets to the Input and Output nodes of the workflows the app runs. Runs a workflow from a button, or when a field changes. Shows results as they stream in. Declares operations, variables, and resources in the App Data panel. Previews the screen at phone, tablet, or full width. Publishes a version that locks in the current state of every workflow it runs. Where it fits A Mini App is how your work reaches people who shouldn’t have to read a node graph. App Builder wraps one or more workflows in a screen: fields feed the workflow’s inputs, buttons start runs, and result widgets show what comes back. See Key Concepts → How everything fits together. Open it Open the Apps panel in the left sidebar. Click an app, or create one with New app or Create app from workflow. On the app tab, switch to Design. Run shows the app the way its users see it, using the released version when there is one. Preview draft runs the unpublished draft instead. Settings holds publishing, versions, the spend budget, the public link, and recent runs. Rename an app from the right-click menu in the Apps panel. Build an app Check that each workflow has the Input and Output nodes the app needs. Open one from the Linked workflows menu; it opens as its own workflow tab. Add fields: Text Input, Number Input, Slider, Switch, Select, or a Workflow Form that renders every input of an operation at once. The palette groups widgets as Inputs, Chat & AI, Actions, Display, and Layout. Wire each field to the matching Input node. Add a Button with the Run workflow action, pointing at the workflow to run. Add result widgets: Text, Markdown, Image, JSON, Progress. Wire each result widget to the matching Output node. Click Save. The header also has App Data, Back, and the phone, tablet, and fit-width preview toggle. The root settings hold the app title and a Theme (Default, Centered, or Card). Letting the agent do it The assistant sits on the right of Design, Run, and Settings. It can read the app’s workflows, add widgets, wire them up, declare operations and variables, and even edit a graph when the app needs new Input, Output, or Variable nodes. It can also check its work with the same wiring check as nodetool app debug, including unsaved edits, and run the app once to read the verdict. Good prompts describe the result you want: Build a compact app with all inputs on the left, a run button below them, and outputs on the right. App Data App Data in the Design header opens a panel with three lists. The assistant edits the same lists. Operations. Each has a name, a Runs choice of Workflow or JS script, the workflow or script to run, While one is running (Replace running, Queue behind, or Run in parallel), and Timeout (ms). Variables. Each has a name, a type (Any, Text, Number, Integer, True/false, List, or Record), a default, a scope (This session or Per user), and Remember between visits. Only per-user variables can be remembered. Resources. Each has a name, a kind (Asset, Timeline, Storyboard, or Sketch), and either one fixed document or all documents of that kind. Wiring The picker lists what each workflow actually offers. Wiring points at node ids, so renaming a node doesn’t break the app. Widget kind Wires to Fields the user edits A workflow input (op:<opId>/in:<nodeId>) or a node setting Result displays A workflow output (op:<opId>/out:<nodeId>) or a variable Toggles and selects A variable (var:<variableId>) Wiring that points at nothing is reported as an error, both in the editor and in nodetool app debug. The full list is in the Reference. Several workflows in one app Each workflow the app runs is an operation, edited under Operations. Give each one its own input and output wiring, its own rule for overlapping runs, and an optional time limit. Buttons point at an operation by name, so a two-step app is two operations and two buttons — not two apps. Publish and share Publish new version in Settings saves the draft, then takes a snapshot of the screen and of every workflow the app runs. The snapshot becomes the released version, so the published app keeps working while you keep editing the drafts. Settings also rolls back to an older version, sets the spend budget, and creates a public link on servers that offer it. See Publishing and sharing. The editor has no export button. Move an app and its workflows as one JSON file with nodetool apps export-bundle and nodetool apps import-bundle (CLI). Related topics Mini Apps — concepts and runtime Building Mini Apps — recipes per use case Mini App Reference — widgets, bindings, schema Workflow Editor Chat & Agents--- # app-views-noext.md Source: https://docs.nodetool.ai/app-views This page has moved. Please go to Desktop App Views.--- # app-views-html.md Source: https://docs.nodetool.ai/app-views.html This page has moved. Please go to Desktop App Views.--- # Architecture & Lifecycle Source: https://docs.nodetool.ai/architecture.html Two core principles: Streaming-first execution — results stream as they generate; long-running jobs can be cancelled. Unified runtime — the same workflow JSON runs in the desktop app, headless server, RunPod endpoint, or Cloud Run. System Components NodeTool is organized into distinct packages, each responsible for a specific layer of the system: Core Packages Package Purpose @nodetool-ai/kernel Workflow execution engine – graph validation, correlation analysis, node actors, inbox routing, edge counting @nodetool-ai/runtime ProcessingContext, model providers, workspace, Python bridge @nodetool-ai/protocol Shared message types and schemas (job_update, node_update, edge_update, task_update, and the rest) @nodetool-ai/node-sdk BaseNode, the @prop decorator, NodeRegistry, and the pack loader @nodetool-ai/agents CodeAct sessions, task planner and executors, agent tools and capabilities, the JavaScript sandbox @nodetool-ai/websocket HTTP, WebSocket, and MCP server @nodetool-ai/dsl TypeScript DSL for building workflows programmatically with type-safe factories @nodetool-ai/config Settings management and environment configuration Infrastructure Packages Package Purpose @nodetool-ai/deploy Docker image builds and self-hosted server provisioning over SSH @nodetool-ai/storage Asset storage backends (local filesystem, S3, Supabase, in-memory) @nodetool-ai/vectorstore Vector database integration (SQLite-vec, Pinecone, Supabase pgvector) for RAG workflows @nodetool-ai/cli Command-line interface for workflow execution, deployment, and package management @nodetool-ai/base-nodes Compatibility shell that re-exports the domain node packages and registers the built-in packs See Packages for how node packages are structured and registered. Frontend Package Purpose web React application – workflow editor, asset explorer, model manager, global chat electron Desktop app that runs the backend and the web UI mobile React Native / Expo app Execution Engine WorkflowRunner The WorkflowRunner is the DAG orchestrator that executes workflow graphs. It handles: Graph validation – Ensures all connections are valid and the graph is acyclic Node actor spawning – Creates a NodeActor for each node in the graph Input dispatch – Routes initial data to the correct input nodes Edge counting and EOS propagation – Tracks when nodes have received all inputs and propagates End-Of-Stream signals through the graph Concurrent execution – Runs independent nodes in parallel when their inputs are ready NodeActor Each node in a workflow runs as a NodeActor with one of four execution modes. Two special cases come first: a trigger node started by a trigger event emits the event payload and completes, and nodetool.control.Loop runs in a kernel-owned loop mode. See Execution Strategies. Mode Behavior When to Use Buffered Collects all inputs before processing Default for most nodes. Use when you need all data before you can produce output (e.g., image resize, text formatting). Streaming input Processes inputs as they arrive, one at a time Use for nodes that handle items in a stream (e.g., filtering, transforming individual items). Streaming output Produces outputs incrementally as they become available Use for LLMs and generators that emit tokens/chunks over time (e.g., Agent, ListGenerator). Controlled Manages its own execution lifecycle with cached input replay Use for nodes that need custom control over when and how they process (e.g., loops, conditional retry). Correlation-Aware Scheduling There is no sync_mode setting (the old zip_all/on_any/sticky modes were removed). Instead, the scheduler is correlation-aware: every value carries a correlation lineage describing which iteration/branch it came from, and the buffered actor path (_runCorrelated in NodeActor) fires a node once per matched set of correlated inputs. Inputs that share a correlation token are matched and processed together — this is what the old zip_all mode approximated, but it is now driven by the actual lineage of each value rather than a static flag. Outputs declare how they relate to their inputs via outputCorrelation (forward, iteration, chunk, aggregate, single), which tells the scheduler whether an output is a passthrough, a new per-item iteration, a chunk of one logical item, a collapse of a stream, or a one-shot value. Join nodes like Zip and Cross pair values from independent iteration sources within their common parent scope. See correlation-design.md for the full model. ProcessingContext The ProcessingContext provides the runtime environment for node execution: Message queue – Collects ProcessingMessage events for streaming to clients Cache interface – A CacheAdapter (get, set, has, delete) for intermediate results. The default is the in-memory MemoryCache. Asset storage – StorageAdapter interface supporting local filesystem, S3, or Supabase Workspace – context.workspace reads and writes the run’s files without branching on local or cloud storage Asset output modes – native (the default), data_uri, temp_url, storage_url, workspace, raw User context – Authentication tokens, user data, workspace information Job Lifecycle (run, stream, reconnect, cancel) Workflow execution uses the actor model: WorkflowRunner validates the graph, spawns one NodeActor per node, and routes values between actors over inboxes. Each actor runs in one of the four modes described above (Buffered, Streaming input, Streaming output, Controlled). The editor runs workflows over the WebSocket with run_job. POST /api/workflows/{id}/run is the HTTP route for agents and the CLI. There is no separate JobExecutionManager class or pluggable threaded/subprocess/docker “execution strategy” — actors run in-process and stream their results out. sequenceDiagram participant Client participant API as API Server participant Runner as WorkflowRunner participant Actor as NodeActor (per node) participant Msg as Messaging/WS Client->>API: run_job (WebSocket) or POST /api/workflows/{id}/run API->>Runner: Validate graph + spawn actors Runner->>Actor: Dispatch inputs, run per execution mode Actor->>Msg: Emit streaming events (node/edge updates) Msg-->>Client: token/output events Client-->>API: reconnect_job with the job id API-->>Msg: resume stream Client->>API: cancel_job over the WebSocket (or the jobs.cancel tRPC procedure) API->>Runner: cancel run Runner-->>Actor: teardown and cleanup Runner-->>Msg: end event Msg-->>Client: completion / cancelled status Message Types The protocol layer (packages/protocol/src/messages.ts) defines the messages a run emits. The main ones: Message Purpose job_update Overall job status: queued (with queue_position), running, completed, failed, cancelled node_update Per-node status: running, completed, error, warning, with the node’s result or error node_progress Progress inside a running node edge_update Data flowing through a connection, with a message counter output_update A value produced by an output node log_update Log lines from a node task_update Agent task lifecycle: task_planned, task_created, step_started, step_completed, step_failed, task_completed, task_failed, and others tool_call_update / tool_result_update Agent tool calls and their results The file also defines planning_update, chunk, notification, error, and more. Each message type has a Zod schema. Agent System One loop drives every agent. A host (a chat turn, nodetool agent run, an MCP client) builds a CodeAct session over a gated toolbelt and hands it the user’s message. The model acts by writing JavaScript that runs in the QuickJS sandbox. Decomposition is something the model can ask for, not a stage the host imposes. Components CodeActExecutor – The action loop: one step or one turn of sandboxed JavaScript over the toolbelt TaskPlanner – Breaks an objective into tasks with dependencies when the model calls create_plan ParallelTaskExecutor – Runs a plan’s independent tasks concurrently when the model calls execute_plan TaskExecutor – Walks one task’s step DAG in dependency order StepExecutor – Runs individual steps, each through a CodeActExecutor SubAgentTool – run_subtask, start_subtask, and run_search start a child loop under the parent’s budget Each task’s result goes to context.memory under task:<id>. There is no synthesis stage: the session’s next turn reads those results and writes the answer. See Agent System for the full design. Tools Tools are grouped into capability modules under packages/agents/src/capabilities/. Examples: web, browser, files, assets, workflows, jobs, models, collections, documents, email, google, memory, shared, apps, and timelines. The model calls them as imports from @nodetool-ai/sandbox-nodetool/<namespace>. External MCP servers add more through the same belt. Providers Model providers share one interface (BaseProvider in packages/runtime/src/providers/). A provider handles authentication, model listing, and inference calls. Which modalities each provider offers (text, image, video, speech, transcription, embeddings, 3D) is in the capability matrix on the Providers page. Kind Providers Cloud text and multimodal OpenAI, Anthropic, Google Gemini, xAI, DeepSeek, Mistral, Groq, Cerebras, Cohere, Alibaba Cloud, GMI Cloud, Moonshot, MiniMax, Together Media and model hosts FAL, Replicate, Hugging Face, Kie.ai, AtlasCloud, ElevenLabs, Topaz, Reve, Higgsfield Routers OpenRouter, Requesty, Opper, API Route Local and self-hosted Ollama, LM Studio, vLLM, llama.cpp, and Python-bridge providers such as MLX Agent and OAuth backends Claude Agent SDK, Codex Custom OpenAI-compatible endpoints you register A built-in cost calculator tracks usage across providers. Credentials resolve from the secret store first, then from environment variables. Storage Architecture NodeTool uses a pluggable storage system with three backends. An in-memory adapter also exists for tests: Backend Use Case Pros Cons Local filesystem Desktop app, development Zero config, fast, private Single machine only S3-compatible Production (AWS, MinIO) Scalable, durable, multi-region Requires cloud account, network latency Supabase Storage Supabase deployments Integrated auth + storage, managed Requires Supabase project NODETOOL_STORAGE_BACKEND selects the backend (file, s3, or supabase, default file). Assets and temp files use the same backend with separate buckets: ASSET_BUCKET for permanent assets and TEMP_BUCKET for intermediate results. See Storage for configuration details. Python Worker Bridge Python nodes and Python-only local providers run in a separate worker process. By default the TS backend spawns the worker with python -m nodetool.worker --stdio and communicates over a local stdio protocol. When NODETOOL_WORKER_URL is set, it connects to a running worker over WebSocket instead: binary-safe MessagePack payloads 4-byte big-endian length framing on stdio, one frame per message on WebSocket in-band discovery, execution, status, progress, chunk, and error messages structured load_errors so import failures are visible without parsing logs See Python Bridge Protocol for the full wire protocol and lifecycle. Notes All endpoints and examples use http://127.0.0.1:7777 by default; update host/port when deploying. The WebSocket uses MessagePack by default and can switch to JSON. See Chat Server for protocol details. Execution strategies are detailed in Execution Strategies. Related Key Concepts – High-level overview of workflows, nodes, and models API Reference – REST and WebSocket API documentation Python Bridge Protocol – TS ↔ Python worker transport and message schemas Developer Guide – Building custom nodes and extensions Deployment Guide – Running NodeTool in production--- # Asset Management Source: https://docs.nodetool.ai/asset-management.html NodeTool lets you store and organize files used in your workflows. Assets can be images, audio clips, videos, PDFs, 3D models, or any other resources referenced by nodes. Where this fits Assets are the shared material between NodeTool’s surfaces. Workflows read assets and write new ones; sketches open an asset and render an edited one back; timelines import assets as clips and export a finished video asset. The Asset Explorer is the one store all of them read from and write to, so a file produced in any surface is immediately usable in the others. See Key Concepts → How everything fits together for the full loop. Asset Explorer The Asset Explorer is the central hub for managing your files. Open Assets from the app menu (it opens as a tab, at /assets), or use the Assets view in the left panel. Views Grid view – Thumbnails with previews, best for visual assets like images and videos List view – Compact rows with metadata columns, best for large libraries Switch between views with the toggle in the toolbar. Toolbar Select all and Deselect Sort assets by Name, Date, or Size Filter by file size: All, Empty, < 1 MB, 1-10 MB, 10-100 MB, > 100 MB Type filter: All, Images, Videos, Audio, 3D Models, Text, Documents, Other Item Size slider for the grid Create folder and Upload files The page header shows the total size, folder count, and file count of the current folder. Navigation Folder tree – Browse your directory hierarchy in the left panel Search – Matches file names only, with a substring match. The input searches the current folder (Search current folder…). The toggle next to it switches to Search all assets…, which searches every folder and shows each result’s folder path. Global search starts at two characters. Uploading Files Drag and drop files directly into the Asset Explorer Use the Upload files button in the toolbar Drop files from your computer onto the workflow canvas. NodeTool uploads them and adds a matching constant node. The server rejects a single upload above NODETOOL_MAX_UPLOAD_BYTES (1 GiB by default). Working with Assets in Workflows Drag any asset from the Asset Explorer onto the workflow canvas. NodeTool adds a constant node for its type, such as nodetool.constant.Image, Video, Audio, Document, or Model3D. Dragging several selected assets adds one node each. An unsupported type shows an “Unsupported file type” error. Asset Viewers NodeTool includes specialized viewers for common file types: File Type Viewer Features Images Zoom up to 16x, pan Audio Waveform with zoom and minimap. To trim or fade, see Audio Editor Video Browser video controls PDF Previous and next page, zoom in, zoom out, reset zoom Text Markdown rendering, code, CSV, and plain text 3D Models Orbit and zoom, grid, axes, wireframe toggle, fullscreen Open any asset in its viewer by double-clicking it in the Asset Explorer. Collections Collections hold text chunks for semantic search in RAG (Retrieval-Augmented Generation) workflows. They are separate from the asset library: a collection stores indexed text, not asset files. Open Collections from the app menu to create one and drop text files onto it. See Collections for the upload route, its limits, and how workflows use collections. Metadata Assets track size, content_type, and duration (audio and video), plus a free-form metadata object. On the desktop app, an imported video whose longer side is at least NODETOOL_VIDEO_PROXY_MIN_SIZE_PX pixels (default 1920) gets a smaller preview proxy built in the background so timeline scrubbing seeks fast. Export reads the original. Files at or above NODETOOL_EXTERNAL_ASSET_THRESHOLD_BYTES (default 1 GiB) stay at their original path and the asset references them in place. See External assets. Document Indexing Text-based files can be indexed for vector search: Extract text from binary formats first, for example with lib.pdf.ExtractText Index the text into a Collection with an index node, or drop text files onto the collection tile Use query nodes (vector.QueryText, vector.HybridSearch) to search the indexed text Combine with language models for RAG See Indexing for detailed setup instructions. Storage Assets are stored locally under the NodeTool data directory by default: $XDG_DATA_HOME/nodetool/assets, falling back to ~/.local/share/nodetool/assets on macOS and Linux, and %APPDATA%\nodetool\assets on Windows. Set ASSET_FOLDER or STORAGE_PATH to move it. For deployed instances, assets can be stored in: S3-compatible storage – AWS S3, MinIO, or compatible services Supabase Storage – Integrated with Supabase auth See Storage for backend configuration. Next Steps Indexing – Set up document indexing for RAG workflows Storage – Configure storage backends Workflow Editor – Use assets in your workflows Example: Chat with Docs – Build a RAG workflow with your documents--- # main.scss Source: https://docs.nodetool.ai/assets/css/main.css /* ============================================ NODETOOL FUTURISTIC DARK THEME A cyberpunk-inspired documentation theme ============================================ */ /* === CSS Variables === / :root { / Colors - Matching nodetool-web paletteDark.ts / –color-bg-primary: #08090A; / background.default / –color-bg-secondary: #101113; / background.paper / c_tabs_header / –color-bg-tertiary: #1B1D21; / grey 800 / c_node_bg / –color-bg-elevated: #17181B; / Paper.overlay / c_node_menu */ –color-text-primary: #F7F8F8; –color-text-secondary: #D4D6DB; –color-text-muted: #8A8F98; –color-text-link: #93C5FD; /* Gradients - using nodetool primary/secondary palette */ –gradient-primary: linear-gradient(137deg, #3d68a8 0%, #E879F9); –gradient-secondary: linear-gradient(137deg, #22D3EE 0%, #50FA7B); –gradient-accent: linear-gradient(137deg, #FFB86C 0%, #E879F9); –gradient-header: linear-gradient(137deg, #3d68a8 0%, #E879F9); /* Accents from nodetool palette */ –color-accent-cyan: #22D3EE; –color-accent-green: #50FA7B; –color-accent-magenta: #E879F9; –color-accent-purple: #C4B5FD; –color-accent-blue: #3d68a8; –color-border: rgba(255, 255, 255, 0.08); –color-border-glow: rgba(61, 104, 168, 0.2); /* Neon Glows */ –glow-cyan: 0 0 10px rgba(34, 211, 238, 0.3), 0 0 20px rgba(34, 211, 238, 0.15), 0 0 30px rgba(34, 211, 238, 0.05); –glow-magenta: 0 0 10px rgba(232, 121, 249, 0.3), 0 0 20px rgba(232, 121, 249, 0.15), 0 0 30px rgba(232, 121, 249, 0.05); –glow-purple: 0 0 10px rgba(196, 181, 253, 0.3), 0 0 20px rgba(196, 181, 253, 0.15); /* Typography */ –font-heading: “Inter”, system-ui, -apple-system, BlinkMacSystemFont, “Segoe UI”, sans-serif; –font-body: “Inter”, system-ui, -apple-system, BlinkMacSystemFont, “Segoe UI”, sans-serif; –font-mono: “JetBrains Mono”, “Courier New”, monospace; /* Spacing */ –spacing-xs: 0.25rem; –spacing-sm: 0.5rem; –spacing-md: 1rem; –spacing-lg: 1.5rem; –spacing-xl: 2rem; –spacing-2xl: 3rem; –spacing-3xl: 4rem; /* Layout */ –max-width: 1400px; –sidebar-width: 280px; –header-height: 70px; /* Transitions */ –transition-fast: 150ms cubic-bezier(0.4, 0, 0.2, 1); –transition-base: 250ms cubic-bezier(0.4, 0, 0.2, 1); –transition-slow: 350ms cubic-bezier(0.4, 0, 0.2, 1); } /* === Global Resets & Base === */ { margin: 0; padding: 0; box-sizing: border-box; } html { font-size: 16px; scroll-behavior: smooth; } /* Lock the page horizontally. clip rather than hidden so the viewport never becomes a scroll container — hidden leaks into a side-scroll on iOS Safari, and it also turns the root into a scrollport, which breaks the sticky header. Wide content (tables, code, diagrams) scrolls inside its own box instead; see .table-scroll, pre, and .mermaid. */ html, body { overflow-x: clip; max-width: 100%; } body { font-family: var(–font-body); font-size: 1rem; line-height: 1.7; color: var(–color-text-primary); background: var(–color-bg-primary); position: relative; } /* === Cyberpunk Background Effects === */ .cyber-grid { position: fixed; top: 0; left: 0; width: 100%; height: 100%; background-image: linear-gradient(rgba(102, 144, 212, 0.03) 1px, transparent 1px), linear-gradient(90deg, rgba(102, 144, 212, 0.03) 1px, transparent 1px); background-size: 50px 50px; pointer-events: none; z-index: -1; opacity: 0.4; } @keyframes gridPulse { 0%, 100% { opacity: 0.3; } 50% { opacity: 0.6; } } /* === Typography === */ h1, h2, h3, h4, h5, h6 { font-family: var(–font-heading); font-weight: 700; line-height: 1.3; margin-top: var(–spacing-xl); margin-bottom: var(–spacing-md); color: var(–color-text-primary); letter-spacing: 0.5px; } h1 { font-size: 2.5rem; color: var(–color-text-primary); margin-top: 0; } h2 { font-size: 1.75rem; color: var(–color-text-primary); border-bottom: 1px solid var(–color-border); padding-top: var(–spacing-md); padding-bottom: var(–spacing-sm); position: relative; display: inline-block; width: 100%; } h2::after { content: “”; position: absolute; bottom: -1px; left: 0; width: 48px; height: 2px; background: var(–color-accent-blue); } h3 { font-size: 1.25rem; color: var(–color-text-primary); } h4 { font-size: 1rem; color: var(–color-text-secondary); } p { margin-bottom: var(–spacing-md); } a { color: var(–color-text-link); text-decoration: none; transition: color var(–transition-fast), text-shadow var(–transition-fast), transform var(–transition-fast); position: relative; } /* Scope the magenta glow hover to body text links only. */ .page-content a:hover, .persona-tile a:hover, .pattern-card a:hover, .mode-card a:hover, .link-grid a:hover { color: var(–color-accent-magenta); text-shadow: var(–glow-magenta); } /* Keyboard focus styling — visible ring without disturbing mouse users */ a:focus-visible, button:focus-visible, summary:focus-visible, input:focus-visible, [tabindex]:focus-visible { outline: 2px solid var(–color-accent-cyan); outline-offset: 2px; border-radius: 4px; } /* Skip-to-content link (shown when focused) */ .skip-link { position: absolute; left: 0; top: -48px; z-index: 500; background: var(–color-accent-cyan); color: var(–color-bg-primary); padding: 0.55rem 1.1rem; font-weight: 700; border-radius: 0 0 8px 0; box-shadow: var(–glow-cyan); transition: top var(–transition-fast); } .skip-link:focus, .skip-link:focus-visible { top: 0; outline: none; } /* Reading progress bar */ .reading-progress { position: fixed; top: 0; left: 0; width: 100%; height: 3px; background: transparent; z-index: 120; pointer-events: none; } .reading-progress-bar { display: block; height: 100%; width: 0; background: linear-gradient(90deg, var(–color-accent-cyan), var(–color-accent-magenta)); box-shadow: 0 0 10px rgba(102, 144, 212, 0.6); transition: width 80ms linear; } code { font-family: var(–font-mono); font-size: 0.9em; background: var(–color-bg-elevated); padding: 0.2em 0.4em; border-radius: 3px; border: 1px solid var(–color-border); color: var(–color-accent-green); } pre { background: var(–color-bg-secondary); border: 1px solid var(–color-border); border-left: 3px solid var(–color-accent-cyan); border-radius: 6px; padding: var(–spacing-md); overflow-x: auto; margin: var(–spacing-lg) 0; box-shadow: 0 4px 12px rgba(0, 0, 0, 0.3); } pre code { background: transparent; border: none; padding: 0; color: var(–color-text-primary); } blockquote { border-left: 4px solid var(–color-accent-purple); padding-left: var(–spacing-lg); margin: var(–spacing-lg) 0; font-style: italic; color: var(–color-text-secondary); background: var(–color-bg-secondary); padding: var(–spacing-md) var(–spacing-lg); border-radius: 0 6px 6px 0; box-shadow: var(–glow-purple); } .persona-callout { border: 1px solid var(–color-border-glow); border-left: 4px solid var(–color-accent-cyan); border-radius: 10px; margin: var(–spacing-lg) 0; padding: var(–spacing-md) var(–spacing-lg); background: linear-gradient(135deg, rgba(6, 147, 227, 0.08), rgba(0, 255, 163, 0.04)); box-shadow: 0 8px 30px rgba(0, 0, 0, 0.35); font-style: normal; } .persona-callout h4 { margin-top: 0; margin-bottom: var(–spacing-sm); color: var(–color-text-primary); } .persona-callout p { margin-bottom: var(–spacing-sm); color: var(–color-text-secondary); } .persona-callout ul { margin: 0; padding-left: 1.2rem; } .persona-callout li { margin-bottom: 0.25rem; } .persona-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); gap: var(–spacing-lg); margin-bottom: var(–spacing-xl); } .persona-tile { border: 1px solid var(–color-border); border-radius: 12px; padding: var(–spacing-lg); background: var(–color-bg-secondary); box-shadow: 0 8px 25px rgba(0, 0, 0, 0.4); } .persona-tile h4 { margin-top: 0; margin-bottom: var(–spacing-sm); } .persona-tile p { margin-bottom: var(–spacing-sm); color: var(–color-text-secondary); } .persona-tile ul { padding-left: 1.1rem; margin: 0; color: var(–color-text-muted); font-size: 0.95rem; } .pattern-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); gap: var(–spacing-lg); margin-bottom: var(–spacing-xl); } .pattern-card { border: 1px solid var(–color-border); border-radius: 12px; padding: var(–spacing-lg); background: var(–color-bg-secondary); display: flex; flex-direction: column; gap: var(–spacing-sm); } .pattern-card a { font-size: 0.95rem; } /* Why NodeTool (docs/comparisons.md) */ .why-lead { font-size: 1.2rem; line-height: 1.6; color: var(–color-text-secondary); max-width: 44rem; } .why-versus { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(–spacing-lg); margin: var(–spacing-xl) 0 var(–spacing-2xl); } .why-path { border: 1px solid var(–color-border); border-radius: 12px; padding: var(–spacing-lg); background: var(–color-bg-secondary); display: flex; flex-direction: column; } .why-path-new { border-color: var(–color-border-glow); background: radial-gradient(120% 90% at 0% 0%, rgba(34, 211, 238, 0.08), transparent 60%), var(–color-bg-secondary); } .why-path-label { margin: 0 0 var(–spacing-md); font-size: 0.75rem; font-weight: 600; letter-spacing: 0.14em; text-transform: uppercase; color: var(–color-text-muted); } .why-path-new .why-path-label { color: var(–color-accent-cyan); } .page-content .why-chain { list-style: none; margin: 0 0 var(–spacing-md); padding: 0; display: flex; flex-direction: column; } .page-content .why-chain li { position: relative; margin: 0; padding: 0.6rem 0.9rem; border: 1px solid var(–color-border); border-radius: 8px; background: var(–color-bg-tertiary); color: var(–color-text-primary); font-size: 0.95rem; } .page-content .why-chain li + li { margin-top: 1.6rem; } .page-content .why-chain li + li::before { content: “↓”; position: absolute; top: -1.45rem; left: 1rem; font-size: 0.9rem; color: var(–color-text-muted); } .page-content .why-path-old .why-chain li + li::before { content: “↓ download, upload”; font-size: 0.75rem; letter-spacing: 0.02em; } .page-content .why-path-new .why-chain li { border-color: var(–color-border-glow); } .page-content .why-path-new .why-chain li + li::before { color: var(–color-accent-cyan); } .why-path-foot { margin: auto 0 0; padding-top: var(–spacing-sm); font-size: 0.9rem; color: var(–color-text-muted); } .why-agent { display: grid; grid-template-columns: minmax(0, 1fr); gap: var(–spacing-lg); margin-bottom: var(–spacing-2xl); } .why-agent-media { aspect-ratio: 16 / 9; overflow: hidden; border: 1px solid var(–color-border); border-radius: 12px; background: var(–color-bg-tertiary); box-shadow: 0 20px 50px rgba(0, 0, 0, 0.45); } .page-content .why-agent-media video { display: block; margin: 0; border: none; border-radius: 0; max-width: none; width: 100%; height: 100%; object-fit: cover; } .page-content .why-points { list-style: none; margin: 0; padding: 0; display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(–spacing-md) var(–spacing-lg); } .page-content .why-points li { margin: 0; padding-left: var(–spacing-md); border-left: 2px solid var(–color-border-glow); color: var(–color-text-muted); font-size: 0.95rem; line-height: 1.55; } .why-points strong { display: block; color: var(–color-text-primary); } .why-own-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(–spacing-lg); margin-bottom: var(–spacing-xl); } .why-own { border: 1px solid var(–color-border); border-radius: 12px; padding: var(–spacing-lg); background: var(–color-bg-secondary); } .why-own h3 { margin: var(–spacing-md) 0 var(–spacing-xs); font-size: 1.05rem; } .why-own p { margin: 0; color: var(–color-text-muted); font-size: 0.95rem; line-height: 1.55; } .why-icon { display: inline-flex; align-items: center; justify-content: center; width: 2.5rem; height: 2.5rem; border-radius: 10px; border: 1px solid var(–color-border-glow); background: var(–color-bg-tertiary); color: var(–color-accent-cyan); } .why-callout { position: relative; margin: 0 0 var(–spacing-2xl); padding: var(–spacing-xl); border: 1px solid rgba(245, 158, 11, 0.25); border-radius: 12px; background: radial-gradient(60% 90% at 0% 0%, rgba(217, 119, 6, 0.12), transparent 60%), var(–color-bg-secondary); } .why-callout-kicker { margin: 0 0 var(–spacing-xs); font-size: 0.75rem; font-weight: 600; letter-spacing: 0.14em; text-transform: uppercase; color: #FBBF24; } .why-callout h3 { margin: 0 0 var(–spacing-md); font-size: 1.5rem; } .why-callout p { max-width: 44rem; color: var(–color-text-secondary); } .why-callout p:last-child { margin-bottom: 0; } .why-table { overflow-x: auto; margin-bottom: var(–spacing-2xl); border: 1px solid var(–color-border); border-radius: 12px; } .page-content .why-table .table-scroll { margin: 0; border: none; border-radius: 0; } .page-content .why-table table { width: 100%; margin: 0; border: none; border-collapse: collapse; } .page-content .why-table th, .page-content .why-table td { padding: 0.85rem 1rem; border: none; border-bottom: 1px solid var(–color-border); text-align: left; vertical-align: top; font-size: 0.95rem; } .page-content .why-table tbody tr:last-child th, .page-content .why-table tbody tr:last-child td { border-bottom: none; } .page-content .why-table tbody th { width: 26%; text-transform: none; letter-spacing: normal; color: var(–color-text-muted); font-weight: 500; } .page-content .why-table td:nth-child(2) { color: var(–color-text-primary); background: rgba(34, 211, 238, 0.04); } .page-content .why-table td:nth-child(3) { color: var(–color-text-muted); } .page-content .why-table thead th:nth-child(2) { color: var(–color-accent-cyan); } .visually-hidden { position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); white-space: nowrap; } .why-vs-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(220px, 1fr)); gap: var(–spacing-md); margin-bottom: var(–spacing-lg); } .why-vs, .why-next-card { display: flex; flex-direction: column; gap: var(–spacing-xs); padding: var(–spacing-md) var(–spacing-lg); border: 1px solid var(–color-border); border-radius: 10px; background: var(–color-bg-secondary); color: inherit; text-decoration: none; transition: border-color var(–transition-fast), transform var(–transition-fast); } .why-vs:hover, .why-next-card:hover { border-color: var(–color-border-glow); transform: translateY(-2px); } .why-vs h3 { margin: 0; font-size: 1rem; } .why-vs h3::after { content: “ →”; color: var(–color-text-link); } .why-vs p, .why-next-card span { margin: 0; color: var(–color-text-muted); font-size: 0.9rem; line-height: 1.5; } .why-next { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(–spacing-md); } .why-next-card strong { color: var(–color-text-link); } @media (max-width: 768px) { .why-versus, .why-points, .why-own-grid, .why-next { grid-template-columns: 1fr; } } @media (prefers-reduced-motion: reduce) { .why-vs:hover, .why-next-card:hover { transform: none; } } /* Hero use-case cards (docs/use-cases.md) */ .usecase-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); gap: var(–spacing-lg); margin: var(–spacing-xl) 0; } .usecase-card { border: 1px solid var(–color-border); border-radius: 12px; background: var(–color-bg-secondary); overflow: hidden; display: flex; flex-direction: column; transition: transform var(–transition-fast), box-shadow var(–transition-fast), border-color var(–transition-fast); } .usecase-card:hover { transform: translateY(-3px); border-color: var(–color-border-glow); box-shadow: 0 12px 30px rgba(0, 0, 0, 0.4); } .usecase-media { display: block; aspect-ratio: 16 / 10; overflow: hidden; background: var(–color-bg-tertiary); } .usecase-media img { width: 100%; height: 100%; object-fit: cover; } .usecase-body { padding: var(–spacing-lg); display: flex; flex-direction: column; gap: var(–spacing-sm); } .usecase-body h3 { margin: 0; font-size: 1.2rem; } .usecase-tag { align-self: flex-start; text-transform: uppercase; letter-spacing: 0.08em; font-size: 0.7rem; font-weight: 600; color: var(–color-accent-cyan); border: 1px solid var(–color-border-glow); border-radius: 999px; padding: 0.15rem 0.6rem; } .usecase-body p { margin: 0; color: var(–color-text-muted); font-size: 0.95rem; } .pipeline-chips { display: flex; flex-wrap: wrap; gap: 0.4rem; margin-top: var(–spacing-xs); } .pipeline-chips span { font-size: 0.75rem; font-family: var(–font-mono, monospace); color: var(–color-text-secondary); background: var(–color-bg-tertiary); border: 1px solid var(–color-border); border-radius: 6px; padding: 0.2rem 0.5rem; } /* Use-case detail pages */ .usecase-eyebrow { text-transform: uppercase; letter-spacing: 0.08em; font-size: 0.8rem; font-weight: 600; color: var(–color-accent-cyan); margin-bottom: var(–spacing-sm); } .usecase-hero { margin: var(–spacing-lg) 0; border: 1px solid var(–color-border); border-radius: 12px; overflow: hidden; background: var(–color-bg-tertiary); } .usecase-hero img, .usecase-hero video { display: block; width: 100%; height: auto; } .usecase-gallery { display: grid; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); gap: var(–spacing-md); margin: var(–spacing-lg) 0; } .usecase-gallery.two { grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); } .usecase-gallery figure { margin: 0; border: 1px solid var(–color-border); border-radius: 10px; overflow: hidden; background: var(–color-bg-secondary); } .usecase-gallery img, .usecase-gallery video { display: block; width: 100%; height: auto; } .usecase-gallery figcaption { padding: var(–spacing-sm) var(–spacing-md); font-size: 0.85rem; color: var(–color-text-muted); border-top: 1px solid var(–color-border); } .step-sequence { counter-reset: steps; list-style: none; margin: var(–spacing-lg) 0; padding-left: 0; } .step-sequence li { position: relative; padding-left: 2.5rem; margin-bottom: var(–spacing-md); line-height: 1.5; } .step-sequence li::before { counter-increment: steps; content: counter(steps); position: absolute; left: 0; top: 0; width: 1.75rem; height: 1.75rem; border-radius: 50%; border: 1px solid var(–color-border-glow); display: flex; align-items: center; justify-content: center; background: var(–color-bg-tertiary); color: var(–color-accent-cyan); font-weight: 600; } .step-sequence a { font-weight: 600; color: var(–color-text-primary); } .mode-split { display: grid; grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); gap: var(–spacing-lg); margin-top: var(–spacing-lg); } .mode-card { border: 1px solid var(–color-border); border-radius: 12px; padding: var(–spacing-lg); background: var(–color-bg-secondary); box-shadow: 0 8px 25px rgba(0, 0, 0, 0.35); } .mode-card ul { padding-left: 1.2rem; } .mode-card a { display: inline-block; margin-top: var(–spacing-sm); font-weight: 600; } .home-hero { text-align: left; padding: var(–spacing-xl) 0 var(–spacing-lg); } .home-hero img, .home-screenshot { display: block; width: 100%; max-width: 100%; height: auto; border-radius: 8px; border: 1px solid var(–color-border); margin: var(–spacing-lg) 0; box-shadow: 0 4px 20px rgba(102, 144, 212, 0.08); } .eyebrow { text-transform: uppercase; letter-spacing: 0.1em; font-size: 0.85rem; color: var(–color-text-muted); margin-bottom: var(–spacing-sm); } .lead { font-size: 1.1rem; color: var(–color-text-secondary); max-width: 720px; margin-bottom: var(–spacing-lg); } .cta-row { display: flex; flex-wrap: wrap; gap: var(–spacing-sm); margin-bottom: var(–spacing-lg); } .cta-button { border: 1px solid var(–color-border); border-radius: 999px; padding: 0.6rem 1.5rem; font-weight: 600; transition: transform var(–transition-fast), box-shadow var(–transition-fast); text-align: center; } .cta-button.primary { background: var(–gradient-primary); border: none; color: #fff; } .cta-button.ghost { border-color: var(–color-text-muted); color: var(–color-text-muted); } .cta-button:hover { transform: translateY(-2px); box-shadow: var(–glow-cyan); } .home-section { margin-bottom: var(–spacing-2xl); } .feature-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); gap: var(–spacing-lg); } .feature-card { border: 1px solid var(–color-border); border-radius: 12px; padding: var(–spacing-lg); background: var(–color-bg-secondary); } .link-grid { list-style: none; display: grid; grid-template-columns: repeat(auto-fit, minmax(220px, 1fr)); gap: var(–spacing-sm); padding-left: 0; } .link-grid li a { display: block; padding: 0.6rem 0.9rem; border: 1px solid var(–color-border); border-radius: 10px; background: var(–color-bg-secondary); } /* === Header === */ .site-header { position: sticky; top: 0; width: 100%; height: var(–header-height); background: rgba(15, 17, 21, 0.95); backdrop-filter: blur(10px); border-bottom: 1px solid var(–color-border); z-index: 100; box-shadow: 0 2px 20px rgba(102, 144, 212, 0.08); } .header-container { max-width: var(–max-width); margin: 0 auto; height: 100%; display: grid; grid-template-columns: auto 1fr auto; align-items: center; padding: 0 var(–spacing-lg); } .logo-section { position: relative; } .logo { font-family: var(–font-heading); font-size: 1.5rem; font-weight: 800; color: var(–color-text-primary); text-decoration: none; display: flex; align-items: center; gap: 0.25rem; transition: all var(–transition-base); } .logo-image { width: 64px; height: 64px; filter: brightness(10); } .logo-text { font-size: 1.25rem; font-weight: 700; letter-spacing: 0.1em; color: transparent; background-image: linear-gradient(to right, #60a5fa, #a78bfa, #fb7185); -webkit-background-clip: text; background-clip: text; } .logo-bracket { color: var(–color-accent-cyan); font-weight: 900; } .logo:hover { transform: translateY(-2px); text-shadow: var(–glow-cyan); } .logo:hover .logo-bracket { text-shadow: var(–glow-cyan); animation: bracket-pulse 0.6s ease-in-out; } @keyframes bracket-pulse { 0%, 100% { transform: scaleX(1); } 50% { transform: scaleX(1.2); } } .main-nav { display: flex; justify-content: center; } .nav-list { display: flex; list-style: none; gap: 0.25rem; align-items: center; padding: 0; } .nav-item { display: block; padding: 0.5rem 1rem; font-size: 0.875rem; white-space: nowrap; font-weight: 500; color: #cbd5e1; text-decoration: none; border-radius: 9999px; transition: all 0.2s; border: 1px solid transparent; } .nav-item:hover { color: #bfdbfe; background-color: rgba(30, 41, 59, 0.6); } .nav-item.active { background-color: rgba(37, 99, 235, 0.25); color: #bfdbfe; border-color: rgba(59, 130, 246, 0.4); } .nav-item-external { color: var(–color-accent-cyan, #67e8f9); border-color: rgba(103, 232, 249, 0.3); } .nav-item-external:hover { background-color: rgba(103, 232, 249, 0.1); color: var(–color-accent-cyan, #67e8f9); } /* Remove old underline effect */ .nav-item::before { display: none; } .header-right { display: flex; align-items: center; } .search-toggle { display: none; align-items: center; justify-content: center; width: 40px; height: 40px; padding: 0; background: transparent; border: 1px solid transparent; border-radius: 10px; color: var(–color-text-secondary); cursor: pointer; transition: background var(–transition-fast), color var(–transition-fast), border-color var(–transition-fast); } .search-toggle:hover, .search-toggle[aria-expanded=”true”] { background: rgba(30, 41, 59, 0.6); color: var(–color-accent-cyan); border-color: rgba(102, 144, 212, 0.4); } .search-container { position: relative; margin-right: var(–spacing-lg); width: 200px; max-width: 100%; transition: width var(–transition-base); } .search-container:focus-within { width: 360px; } .search-input-wrapper { display: grid; grid-template-columns: auto 1fr auto; align-items: center; gap: var(–spacing-sm); background: rgba(15, 23, 42, 0.6); border: 1px solid rgba(30, 41, 59, 0.7); border-radius: 14px; padding: 0.35rem 0.75rem; box-shadow: 0 10px 30px rgba(0, 0, 0, 0.25), inset 0 0 0 1px rgba(255, 255, 255, 0.03); transition: border var(–transition-fast), box-shadow var(–transition-fast), transform var(–transition-fast); } .search-input-wrapper:focus-within { border-color: rgba(102, 144, 212, 0.5); box-shadow: var(–glow-cyan); transform: translateY(-1px); } .search-icon { display: inline-flex; color: var(–color-text-muted); opacity: 0.8; } .search-input-wrapper input { width: 100%; background: transparent; border: none; color: var(–color-text-primary); font-size: 0.95rem; outline: none; padding: 0.35rem 0; } .search-input-wrapper input::placeholder { color: var(–color-text-muted); } .search-shortcut { display: inline-flex; align-items: center; justify-content: center; padding: 0.15rem 0.4rem; border: 1px solid var(–color-border); border-radius: 8px; background: rgba(255, 255, 255, 0.04); color: var(–color-text-muted); font-family: var(–font-mono); font-size: 0.75rem; letter-spacing: 0.03em; } .search-results { position: absolute; top: calc(100% + 10px); right: 0; width: min(520px, 90vw); max-height: 70vh; overflow: hidden; background: rgba(10, 12, 18, 0.98); border: 1px solid rgba(30, 41, 59, 0.9); border-radius: 14px; box-shadow: 0 20px 50px rgba(0, 0, 0, 0.5), var(–glow-purple); backdrop-filter: blur(6px); z-index: 999; } .search-results-header { display: flex; align-items: center; justify-content: space-between; padding: 0.75rem 1rem; border-bottom: 1px solid var(–color-border); color: var(–color-text-muted); font-size: 0.85rem; text-transform: uppercase; letter-spacing: 0.04em; } .search-results-list { list-style: none; max-height: 62vh; overflow-y: auto; padding: 0.35rem 0; } .search-result { padding: 0.65rem 1rem; cursor: pointer; transition: background var(–transition-fast), transform var(–transition-fast); border-left: 2px solid transparent; } .search-result:hover, .search-result.active { background: rgba(102, 144, 212, 0.08); border-left-color: var(–color-accent-cyan); transform: translateX(2px); } .search-result-title { font-weight: 700; color: var(–color-text-primary); display: block; margin-bottom: 0.2rem; } .search-result-meta { display: flex; flex-wrap: wrap; gap: 0.35rem; align-items: center; color: var(–color-text-muted); font-size: 0.85rem; } .search-badge { display: inline-flex; align-items: center; padding: 0.1rem 0.45rem; border-radius: 999px; font-size: 0.75rem; font-weight: 700; letter-spacing: 0.03em; text-transform: uppercase; border: 1px solid transparent; } .search-badge.concept { background: rgba(99, 102, 241, 0.14); border-color: rgba(99, 102, 241, 0.4); color: #c7d2fe; } .search-badge.tutorial { background: rgba(16, 185, 129, 0.14); border-color: rgba(16, 185, 129, 0.4); color: #a7f3d0; } .search-badge.reference { background: rgba(59, 130, 246, 0.12); border-color: rgba(59, 130, 246, 0.35); color: #bfdbfe; } .search-badge.api { background: rgba(236, 72, 153, 0.14); border-color: rgba(236, 72, 153, 0.4); color: #fbcfe8; } .search-badge.node { background: rgba(234, 179, 8, 0.16); border-color: rgba(234, 179, 8, 0.45); color: #fef08a; } .search-snippet { margin-top: 0.2rem; color: var(–color-text-secondary); font-size: 0.9rem; line-height: 1.4; } .search-empty { padding: 0.75rem 1rem; color: var(–color-text-muted); font-size: 0.9rem; } .social-links { display: flex; gap: var(–spacing-sm); } .social-link { padding: 0.25rem; color: var(–color-text-secondary); display: flex; align-items: center; justify-content: center; border-radius: 0.5rem; transition: all var(–transition-base); } .social-link:hover { color: var(–color-text-primary); background: rgba(30, 41, 59, 0.6); transform: none; box-shadow: none; } .mobile-menu-toggle { display: none; background: none; border: none; cursor: pointer; padding: var(–spacing-sm); position: relative; z-index: 250; } .mobile-menu-toggle.active .hamburger { background: transparent; box-shadow: none; } .mobile-menu-toggle.active .hamburger::before { top: 0; transform: rotate(45deg); } .mobile-menu-toggle.active .hamburger::after { bottom: 0; transform: rotate(-45deg); } .hamburger { display: block; width: 25px; height: 2px; background: var(–color-accent-cyan); position: relative; transition: all var(–transition-base); box-shadow: var(–glow-cyan); } .hamburger::before, .hamburger::after { content: “”; position: absolute; width: 25px; height: 2px; background: var(–color-accent-cyan); transition: all var(–transition-base); box-shadow: var(–glow-cyan); } .hamburger::before { top: -8px; } .hamburger::after { bottom: -8px; } /* === Main Container === */ .main-container { max-width: var(–max-width); margin: 0 auto; display: flex; min-height: calc(100vh - var(–header-height) - 200px); } /* === Sidebar === */ .sidebar { width: var(–sidebar-width); padding: var(–spacing-xl) var(–spacing-lg); background: var(–color-bg-secondary); border-right: 1px solid var(–color-border); position: sticky; top: var(–header-height); height: calc(100vh - var(–header-height)); overflow-y: auto; scrollbar-width: thin; scrollbar-color: var(–color-accent-cyan) var(–color-bg-tertiary); } .sidebar::-webkit-scrollbar { width: 6px; } .sidebar::-webkit-scrollbar-track { background: var(–color-bg-tertiary); } .sidebar::-webkit-scrollbar-thumb { background: var(–color-accent-cyan); border-radius: 3px; box-shadow: var(–glow-cyan); } .sidebar-header { margin-bottom: var(–spacing-xl); display: flex; align-items: center; justify-content: space-between; gap: var(–spacing-sm); } .sidebar-title { font-family: var(–font-heading); font-size: 1.1rem; color: var(–color-accent-cyan); display: flex; align-items: center; gap: var(–spacing-sm); margin: 0; } .sidebar-collapse-all { background: transparent; border: 1px solid var(–color-border); color: var(–color-text-muted); font-family: var(–font-mono); font-size: 0.7rem; text-transform: uppercase; letter-spacing: 0.08em; padding: 0.3rem 0.55rem; border-radius: 6px; cursor: pointer; transition: color var(–transition-fast), border-color var(–transition-fast), background var(–transition-fast); } .sidebar-collapse-all:hover { color: var(–color-accent-cyan); border-color: var(–color-accent-cyan); background: rgba(102, 144, 212, 0.05); } .title-icon { color: var(–color-accent-magenta); animation: pulse 2s ease-in-out infinite; } @keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.5; } } .nav-section { margin-bottom: var(–spacing-lg); border: 1px solid transparent; border-radius: 8px; transition: border-color var(–transition-fast); } .nav-section[open] { border-color: rgba(102, 144, 212, 0.08); } .nav-section-summary { list-style: none; cursor: pointer; display: flex; align-items: center; justify-content: space-between; gap: var(–spacing-sm); padding: 0.45rem 0.55rem; border-radius: 6px; transition: background var(–transition-fast); } .nav-section-summary::-webkit-details-marker { display: none; } .nav-section-summary:hover { background: rgba(102, 144, 212, 0.06); } .nav-section-title { font-family: var(–font-heading); font-size: 0.8rem; text-transform: uppercase; letter-spacing: 1px; color: var(–color-text-muted); font-weight: 600; margin: 0; } .nav-section[open] .nav-section-title { color: var(–color-accent-cyan); } .nav-section-chevron { width: 0; height: 0; border-left: 5px solid var(–color-text-muted); border-top: 4px solid transparent; border-bottom: 4px solid transparent; transition: transform var(–transition-fast); } .nav-section[open] .nav-section-chevron { transform: rotate(90deg); border-left-color: var(–color-accent-cyan); } .nav-section-list { list-style: none; margin-top: 0.3rem; padding: 0; } .sidebar-link { display: block; padding: var(–spacing-sm) var(–spacing-md); color: var(–color-text-secondary); font-size: 0.95rem; border-left: 2px solid transparent; transition: all var(–transition-fast); margin-bottom: 2px; } .sidebar-link:hover { color: var(–color-accent-cyan); background: rgba(102, 144, 212, 0.04); border-left-color: var(–color-accent-cyan); transform: translateX(4px); text-shadow: none; } .sidebar-link.active { color: var(–color-accent-cyan); background: rgba(102, 144, 212, 0.08); border-left-color: var(–color-accent-cyan); font-weight: 600; } /* === Content Area === / .content { flex: 1; / Flex items default to min-width: auto, which lets wide content push the column past the viewport. Without this the whole page grows sideways. */ min-width: 0; padding: var(–spacing-2xl) var(–spacing-xl); max-width: 900px; margin: 0 auto; width: 100%; } .doc-page { animation: fadeIn 0.5s ease-in; } @keyframes fadeIn { from { opacity: 0; transform: translateY(20px); } to { opacity: 1; transform: translateY(0); } } .page-header { position: relative; margin-bottom: var(–spacing-2xl); padding-bottom: var(–spacing-lg); border-bottom: 1px solid var(–color-border); } .page-title { margin: 0 0 var(–spacing-sm) 0; } .page-description { font-size: 1.1rem; color: var(–color-text-secondary); margin: 0; } .page-content { margin-bottom: var(–spacing-3xl); /* Long URLs and identifiers break instead of widening the page. */ overflow-wrap: break-word; } /* Inline code often holds an unbreakable path or env var name; break it rather than let it stick out. break-word and not anywhere on purpose — the latter also shrinks min-content width, which squeezes table columns into shredded paths instead of letting a wide table scroll. Code inside <pre> keeps its formatting and scrolls with the block. */ .page-content :not(pre) > code { overflow-wrap: break-word; } .page-content img { max-width: 100%; height: auto; border-radius: 8px; border: 1px solid var(–color-border); margin: var(–spacing-lg) 0; box-shadow: 0 4px 20px rgba(102, 144, 212, 0.08); } .page-content video { display: block; width: 100%; height: auto; border-radius: 8px; border: 1px solid var(–color-border); margin: var(–spacing-lg) 0; background: #0f0f17; box-shadow: 0 4px 20px rgba(102, 144, 212, 0.08); } .page-content ul, .page-content ol { margin-left: var(–spacing-lg); margin-bottom: var(–spacing-md); } @media (max-width: 768px) { .page-content ul, .page-content ol { margin-left: var(–spacing-md); } } .page-content li { margin-bottom: var(–spacing-sm); color: var(–color-text-secondary); } .page-content .icon-list li { display: flex; align-items: flex-start; } .page-content .icon-list li svg { flex-shrink: 0; vertical-align: middle; margin-right: 8px; margin-top: 2px; stroke: var(–color-accent-cyan); } .page-content li::marker { color: var(–color-accent-cyan); } /* Wide tables are the main thing that used to push docs pages sideways. They get their own horizontal scroller (added by theme.js) so the page stays put. The chrome — border, radius, background — moves to the wrapper so the rounded corners clip the scrolling table. */ .page-content .table-scroll { overflow-x: auto; max-width: 100%; margin: var(–spacing-lg) 0; background: var(–color-bg-secondary); border: 1px solid var(–color-border); border-radius: 6px; } .page-content table { width: 100%; border-collapse: collapse; margin: var(–spacing-lg) 0; background: var(–color-bg-secondary); border: 1px solid var(–color-border); border-radius: 6px; /* Fallback for the moment before theme.js wraps the table, and for readers with JS off: the table scrolls itself. display: block costs automatic column sizing, so the wrapper above is the preferred path. */ display: block; width: max-content; min-width: 100%; max-width: 100%; overflow-x: auto; } .page-content .table-scroll:focus-visible { outline: 2px solid var(–color-accent-cyan); outline-offset: 2px; } .page-content .table-scroll > table { display: table; /* Fill the column when it fits, expand (and scroll) when it doesn’t. */ width: auto; min-width: 100%; max-width: none; overflow: visible; margin: 0; border: none; border-radius: 0; background: none; } .page-content th, .page-content td { padding: var(–spacing-md); text-align: left; border-bottom: 1px solid var(–color-border); } .page-content th { background: var(–color-bg-tertiary); color: var(–color-accent-cyan); font-family: var(–font-heading); font-weight: 600; text-transform: uppercase; font-size: 0.9rem; letter-spacing: 0.5px; } .page-content tr:hover { background: rgba(102, 144, 212, 0.03); } /* === Page Navigation === */ .page-navigation { display: flex; justify-content: space-between; gap: var(–spacing-md); margin-top: var(–spacing-3xl); padding-top: var(–spacing-xl); border-top: 1px solid var(–color-border); } .nav-link { flex: 1; padding: var(–spacing-lg); background: var(–color-bg-secondary); border: 1px solid var(–color-border); border-radius: 6px; display: flex; align-items: center; gap: var(–spacing-md); transition: all var(–transition-base); text-decoration: none; } .nav-link:hover { background: var(–color-bg-tertiary); border-color: var(–color-accent-cyan); transform: translateY(-2px); box-shadow: 0 4px 12px rgba(102, 144, 212, 0.15); } .nav-link.prev { justify-content: flex-start; } .nav-link.next { justify-content: flex-end; text-align: right; } .nav-arrow { font-size: 1.5rem; color: var(–color-accent-cyan); } .nav-text { font-family: var(–font-heading); font-size: 0.9rem; color: var(–color-text-secondary); } .nav-link:hover .nav-arrow { text-shadow: var(–glow-cyan); } .nav-link:hover .nav-text { color: var(–color-accent-cyan); } /* === Footer === */ .site-footer { background: var(–color-bg-secondary); border-top: 1px solid var(–color-border); padding: var(–spacing-3xl) 0 var(–spacing-xl); margin-top: var(–spacing-3xl); position: relative; overflow: hidden; } .footer-glow { position: absolute; top: 0; left: 50%; transform: translateX(-50%); width: 100%; height: 1px; background: linear-gradient(90deg, transparent, var(–color-accent-cyan), var(–color-accent-magenta), transparent); box-shadow: var(–glow-cyan); } .footer-container { max-width: var(–max-width); margin: 0 auto; padding: 0 var(–spacing-lg); } .footer-grid { display: grid; grid-template-columns: 2fr 1fr 1fr 1fr; gap: var(–spacing-2xl); margin-bottom: var(–spacing-2xl); } .footer-title { font-family: var(–font-heading); font-size: 1.3rem; color: var(–color-accent-cyan); margin-bottom: var(–spacing-sm); } .footer-desc { color: var(–color-text-muted); font-size: 0.95rem; line-height: 1.6; } .footer-marketing-link { margin-top: var(–spacing-md); font-size: 0.95rem; } .footer-marketing-link a { color: var(–color-accent-cyan); text-decoration: none; font-weight: 500; transition: opacity 0.2s; } .footer-marketing-link a:hover { opacity: 0.8; text-decoration: underline; } .footer-heading { font-family: var(–font-heading); font-size: 0.9rem; color: var(–color-text-primary); margin-bottom: var(–spacing-md); text-transform: uppercase; letter-spacing: 1px; } .footer-links { list-style: none; } .footer-links li { margin-bottom: var(–spacing-sm); } .footer-links a { color: var(–color-text-secondary); font-size: 0.95rem; transition: all var(–transition-fast); } .footer-links a:hover { color: var(–color-accent-cyan); transform: translateX(4px); display: inline-block; } .footer-bottom { padding-top: var(–spacing-xl); border-top: 1px solid var(–color-border); text-align: center; } .copyright { color: var(–color-text-muted); font-size: 0.9rem; } .bracket { color: var(–color-accent-magenta); font-weight: bold; } /* === Responsive Design === */ @media (max-width: 1024px) { .footer-grid { grid-template-columns: 1fr 1fr; } .sidebar { width: 250px; } .header-container { grid-template-columns: auto 1fr auto; column-gap: var(–spacing-sm); padding: 0 var(–spacing-lg); } .search-toggle { display: inline-flex; } .search-container { position: absolute; top: 100%; left: 0; right: 0; margin-right: 0; padding: var(–spacing-sm) var(–spacing-lg) var(–spacing-md); width: auto; max-width: none; background: rgba(15, 17, 21, 0.98); backdrop-filter: blur(10px); border-bottom: 1px solid var(–color-border); box-shadow: 0 4px 20px rgba(0, 0, 0, 0.5); z-index: 140; display: none; } .search-container.open { display: block; } .search-container:focus-within { width: auto; } .search-results { position: static; left: auto; right: auto; width: auto; margin-top: var(–spacing-sm); box-shadow: none; } .header-right { justify-content: flex-end; gap: var(–spacing-sm); } } @media (max-width: 768px) { :root { –sidebar-width: 0; } .logo-image { width: 56px; height: 56px; } .logo-text { font-size: 1.5rem; } .sidebar { position: fixed; left: -280px; width: 280px; z-index: 200; transition: left var(–transition-base); } .sidebar.active { left: 0; } .content { padding: var(–spacing-lg); } .main-nav { position: relative; } .nav-list { display: none; position: fixed; top: var(–header-height); left: 0; right: 0; flex-direction: column; background: rgba(15, 17, 21, 0.98); backdrop-filter: blur(10px); border-bottom: 1px solid var(–color-border); padding: var(–spacing-lg); gap: var(–spacing-sm); max-height: calc(100vh - var(–header-height)); overflow-y: auto; z-index: 150; box-shadow: 0 4px 20px rgba(0, 0, 0, 0.5); } .nav-list.active { display: flex; } .mobile-menu-toggle { display: block; } .footer-grid { grid-template-columns: 1fr; gap: var(–spacing-xl); } h1 { font-size: 2rem; } h2 { font-size: 1.5rem; } .page-navigation { flex-direction: column; } .header-container { grid-template-columns: auto 1fr auto; padding: 0 var(–spacing-md); } .search-container { padding: var(–spacing-sm) var(–spacing-md) var(–spacing-md); } .search-container:focus-within { width: auto; } .social-links { gap: var(–spacing-xs); } } /* === Syntax Highlighting (Rouge/Pygments) === */ .highlight { background: var(–color-bg-secondary); border-radius: 6px; border-left: 3px solid var(–color-accent-cyan); } .highlight .c { color: var(–color-text-muted); } /* Comment */ .highlight .k { color: #ff5555; } /* Keyword - using error color */ .highlight .s { color: var(–color-accent-blue); } /* String */ .highlight .mi { color: var(–color-accent-cyan); } /* Number */ .highlight .nf { color: var(–color-accent-purple); } /* Function */ .highlight .nc { color: #ffb86c; } /* Class - using warning color */ .highlight .o { color: #ff5555; } /* Operator - using error color */ /* === Copy Buttons (moved from JS-injected styles) === */ .copy-button { position: absolute; top: 0.5rem; right: 0.5rem; padding: 0.4rem 0.8rem; background: var(–color-bg-elevated); color: var(–color-accent-cyan); border: 1px solid var(–color-border); border-radius: 4px; font-family: var(–font-mono); font-size: 0.8rem; cursor: pointer; transition: all var(–transition-fast); opacity: 0; } pre:hover .copy-button, .copy-button:focus-visible { opacity: 1; } .copy-button:hover { background: var(–color-accent-cyan); color: var(–color-bg-primary); border-color: var(–color-accent-cyan); box-shadow: var(–glow-cyan); } .copy-button.copied { background: var(–color-accent-green); color: var(–color-bg-primary); border-color: var(–color-accent-green); opacity: 1; } .copy-page-button { position: absolute; top: 0; right: 0; display: inline-flex; align-items: center; gap: 0.4rem; padding: 0.4rem 0.8rem; background: var(–color-bg-elevated); color: var(–color-accent-cyan); border: 1px solid var(–color-border); border-radius: 4px; font-family: var(–font-mono); font-size: 0.8rem; cursor: pointer; transition: all var(–transition-fast); } .copy-page-button:hover { background: var(–color-accent-cyan); color: var(–color-bg-primary); border-color: var(–color-accent-cyan); box-shadow: var(–glow-cyan); } .copy-page-button:hover svg { stroke: var(–color-bg-primary); } .copy-page-button.copied { background: var(–color-accent-green); color: var(–color-bg-primary); border-color: var(–color-accent-green); } .copy-page-button.copied svg { stroke: var(–color-bg-primary); } /* === Back to top === */ .back-to-top { position: fixed; bottom: 1.25rem; right: 1.25rem; width: 44px; height: 44px; display: inline-flex; align-items: center; justify-content: center; background: var(–color-bg-elevated); color: var(–color-accent-cyan); border: 1px solid var(–color-border); border-radius: 50%; cursor: pointer; opacity: 0; transform: translateY(8px); transition: opacity var(–transition-base), transform var(–transition-base), box-shadow var(–transition-base), background var(–transition-fast), color var(–transition-fast); z-index: 110; box-shadow: 0 6px 20px rgba(0, 0, 0, 0.35); } .back-to-top.visible { opacity: 1; transform: translateY(0); } .back-to-top:hover { background: var(–color-accent-cyan); color: var(–color-bg-primary); box-shadow: var(–glow-cyan); } @media (max-width: 768px) { .back-to-top { bottom: 0.85rem; right: 0.85rem; width: 40px; height: 40px; } } /* === Page meta (Edit on GitHub, last updated) === */ .page-meta { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: var(–spacing-sm); margin-top: var(–spacing-xl); padding-top: var(–spacing-md); border-top: 1px solid var(–color-border); color: var(–color-text-muted); font-size: 0.875rem; } .edit-on-github { display: inline-flex; align-items: center; gap: 0.4rem; color: var(–color-text-muted); border: 1px solid var(–color-border); padding: 0.35rem 0.7rem; border-radius: 6px; font-family: var(–font-mono); transition: color var(–transition-fast), border-color var(–transition-fast), background var(–transition-fast); } .edit-on-github:hover { color: var(–color-accent-cyan); border-color: var(–color-accent-cyan); background: rgba(102, 144, 212, 0.05); text-shadow: none; } .page-updated { font-family: var(–font-mono); font-size: 0.8rem; } /* === Utility Classes === */ .text-center { text-align: center; } .mt-0 { margin-top: 0; } .mb-0 { margin-bottom: 0; } /* === Mermaid Workflow Graphs === / / Restyle Mermaid diagrams to match the NodeTool editor canvas — the same look the workflow graphs have on the marketing site. The theme variables (fills, fonts, edge color) are set in _layouts/default.html; the canvas grid, node chrome, and edge polish live here. / .mermaid { / The editor canvas: dark fill with the dotted grid (NodeCanvas). */ background-color: #08090A; background-image: radial-gradient(circle, #1F2126 1px, transparent 1px); background-size: 16px 16px; border: 1px solid var(–color-border); border-radius: 12px; padding: var(–spacing-xl) var(–spacing-lg); margin: var(–spacing-xl) 0; text-align: center; overflow-x: auto; } .mermaid svg { max-width: 100%; height: auto; } /* Node chrome: dark body, rounded corners, soft drop shadow, faint edge — the NodeTool node card. */ .mermaid .node rect, .mermaid .node polygon, .mermaid .node circle, .mermaid .node ellipse, .mermaid .node path { rx: 8px; ry: 8px; fill: var(–color-bg-tertiary) !important; stroke: rgba(255, 255, 255, 0.10) !important; stroke-width: 1px !important; filter: drop-shadow(0 8px 20px rgba(0, 0, 0, 0.16)); } .mermaid .node .label, .mermaid .nodeLabel, .mermaid .node text { fill: var(–color-text-secondary) !important; color: var(–color-text-secondary) !important; font-family: var(–font-body); font-size: 14px; font-weight: 400; letter-spacing: 0.01em; } /* Edges: smooth accent-blue connectors with rounded caps, like the canvas. */ .mermaid .edgePath .path, .mermaid .flowchart-link { stroke: var(–color-accent-blue) !important; stroke-width: 1.6px !important; stroke-opacity: 0.95; stroke-linecap: round; fill: none !important; } .mermaid .arrowheadPath, .mermaid marker path { fill: var(–color-accent-blue) !important; stroke: none !important; } /* Edge labels sit on a small dark chip so they read over the grid. */ .mermaid .edgeLabel, .mermaid .edgeLabel p { background-color: var(–color-bg-secondary) !important; color: var(–color-text-muted) !important; fill: var(–color-text-muted) !important; font-size: 12px; border-radius: 4px; } /* Subgraph containers, when a graph uses them. */ .mermaid .cluster rect { fill: rgba(16, 17, 19, 0.6) !important; stroke: var(–color-border) !important; rx: 10px; ry: 10px; } .mermaid .cluster .label, .mermaid .cluster text { fill: var(–color-text-muted) !important; color: var(–color-text-muted) !important; } /* === Print Styles === */ @media print { .site-header, .sidebar, .footer, .site-footer, .page-navigation, .back-to-top, .reading-progress, .skip-link, .copy-button, .copy-page-button, .edit-on-github, .page-meta { display: none !important; } .content { max-width: 100%; } .cyber-grid { display: none; } body { background: #fff; color: #000; } h1, h2, h3, h4, h5, h6 { color: #000 !important; background: none !important; -webkit-text-fill-color: initial !important; } a { color: #0066cc !important; } } /* === Page chrome: section label, outline, pager === The section label and pager come from _data/navigation.yml through _includes/nav-position.html. The outline is filled by theme.js. */ .page-section-label { margin: 0 0 var(–spacing-xs) 0; font-size: 0.8rem; font-weight: 600; letter-spacing: 0.06em; text-transform: uppercase; color: var(–color-accent-cyan); } .doc-layout { display: block; } @media (min-width: 1280px) { .content:has(.doc-layout) { max-width: 1120px; } .has-page-toc .doc-layout { display: grid; grid-template-columns: minmax(0, 1fr) 200px; gap: var(–spacing-2xl); align-items: start; } } .page-toc { display: none; } @media (min-width: 1280px) { .page-toc:not([hidden]) { display: block; position: sticky; top: calc(var(–header-height) + var(–spacing-xl)); max-height: calc(100vh - var(–header-height) - var(–spacing-3xl)); overflow-y: auto; padding-left: var(–spacing-md); border-left: 1px solid var(–color-border); } } .page-toc-title { margin: 0 0 var(–spacing-sm) 0; font-size: 0.8rem; font-weight: 600; color: var(–color-text-primary); } .page-toc-list { list-style: none; margin: 0; padding: 0; } .page-toc-item { margin: 0 0 var(–spacing-xs) 0; font-size: 0.82rem; line-height: 1.4; } .page-toc-item.sub { padding-left: var(–spacing-md); } .page-toc-item a { color: var(–color-text-muted); text-decoration: none; } .page-toc-item a:hover, .page-toc-item a.active { color: var(–color-accent-cyan); } .page-pager { display: grid; grid-template-columns: 1fr 1fr; gap: var(–spacing-md); margin: 0 0 var(–spacing-xl) 0; } .page-pager-link { display: flex; flex-direction: column; gap: 0.2rem; padding: var(–spacing-md) var(–spacing-lg); border: 1px solid var(–color-border); border-radius: 8px; background: var(–color-bg-secondary); text-decoration: none; transition: border-color var(–transition-fast); } .page-pager-link:hover { border-color: var(–color-accent-cyan); } .page-pager-link.next { grid-column: 2; text-align: right; } .page-pager-dir { font-size: 0.75rem; color: var(–color-text-muted); } .page-pager-link.prev .page-pager-dir::before { content: “← “; } .page-pager-link.next .page-pager-dir::after { content: “ →”; } .page-pager-title { font-weight: 600; color: var(–color-text-primary); } @media (max-width: 768px) { .page-pager { grid-template-columns: 1fr; } .page-pager-link.next { grid-column: 1; } } /* === Callouts === Markdown usage: > Tip: text {: .callout-tip} Classes: callout-note, callout-tip, callout-warning. */ .page-content blockquote.callout-note, .page-content blockquote.callout-tip, .page-content blockquote.callout-warning { font-style: normal; box-shadow: none; border-radius: 6px; border: 1px solid var(–color-border); border-left-width: 4px; color: var(–color-text-secondary); } .page-content blockquote.callout-note { border-left-color: var(–color-accent-cyan); background: rgba(34, 211, 238, 0.06); } .page-content blockquote.callout-tip { border-left-color: var(–color-accent-green); background: rgba(80, 250, 123, 0.06); } .page-content blockquote.callout-warning { border-left-color: #FFB86C; background: rgba(255, 184, 108, 0.07); } .page-content blockquote[class^=”callout-“] p:last-child { margin-bottom: 0; } /* === Link cards === TitleOne line. */ .card-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(220px, 1fr)); gap: var(–spacing-md); margin: var(–spacing-lg) 0; } .doc-card { display: flex; flex-direction: column; gap: var(–spacing-xs); padding: var(–spacing-md) var(–spacing-lg); border: 1px solid var(–color-border); border-radius: 8px; background: var(–color-bg-secondary); text-decoration: none; transition: border-color var(–transition-fast), transform var(–transition-fast); } .page-content a.doc-card, .home-page a.doc-card { text-decoration: none; } .doc-card:hover { border-color: var(–color-accent-cyan); transform: translateY(-2px); } .doc-card strong { color: var(–color-text-primary); font-weight: 600; } .doc-card strong::after { content: “ →”; color: var(–color-accent-cyan); } .doc-card span { color: var(–color-text-muted); font-size: 0.9rem; line-height: 1.45; } @media (prefers-reduced-motion: reduce) { .doc-card:hover { transform: none; } } @media print { .page-toc, .page-pager { display: none !important; } } .page-content a.doc-card:hover, .page-content a.page-pager-link:hover { color: inherit; text-shadow: none; }--- # Audio Editor Source: https://docs.nodetool.ai/audio-editor The audio editor edits an audio asset in place. It decodes the file into samples in your browser, shows the waveform, and lets you cut, fade, normalize, reverse, and re-level a region. Every edit can be undone. Save to audio writes the result back to the asset as a WAV file. Quick Access: Open an audio asset in a workspace tab and click Edit in the View / Edit toggle at the top right of the tab bar. Audio tabs have two modes. View shows the waveform viewer. Edit shows the editor described here. Switching back to View with the toggle, or clicking Done, keeps your unsaved edits in the editor until you close the tab. The layout Area What it holds Toolbar Transport, zoom, region edits, undo and redo, Save to audio, Done Waveform The audio as a waveform, with the playhead and your selection Footer The selection start, end, and length on the left. The playhead time and total length on the right Times show as minutes:seconds.milliseconds. With no selection the footer reads “Drag to select · Click to seek”. If the asset cannot be fetched or decoded, the editor shows “Could not open audio editor” with the reason and a Done button. Select and play Drag across the waveform to select a region. Click to clear the selection and move the playhead to that point. The transport buttons are Play (which becomes Pause), Stop, and Loop. Situation Play does A region is selected Plays only the selection. Stop returns the playhead to the selection start No selection Plays from the playhead to the end of the audio. Stop returns the playhead to the start Loop is on and a region is selected Repeats the selection until you pause or stop Playback stops whenever the audio changes, including undo and redo. Zoom Zoom out, Zoom in, and Fit to window sit beside a logarithmic zoom slider with a percentage readout. The zoom runs from 100 percent (the whole clip fits the window) to 6400 percent. Each button click changes the zoom by a factor of 1.5. When zoomed in, scroll the waveform sideways. Edit operations Edits are destructive to the in-editor copy only. Nothing reaches the asset until you save. Some buttons need a selection and stay disabled without one. Others work on the selection when there is one and on the whole clip when there is not. Button Needs selection What it does Trim Yes Keeps the selection and discards everything outside it Delete Yes Removes the selection and joins the audio on either side. The playhead moves to where the cut was Silence Yes Replaces the selection with silence and keeps the length Fade in Yes Ramps the selection linearly from silence to its original level Fade out Yes Ramps the selection linearly from its original level to silence Normalize No Scales the selection, or the whole clip, so its loudest sample reaches full scale. A silent region is left unchanged Amplify No Multiplies the level by 1.4. Peaks above full scale are clipped Quieten No Multiplies the level by 0.7 Reverse No Reverses the selection, or the whole clip, end to end Trim and Delete clear the selection after they run. The other edits keep it, so you can chain them, for example Fade out followed by Normalize on the same region. All edits apply to every channel of a stereo or multichannel file. Undo and redo Undo and Redo step through your edits. The editor keeps the last 50 states. A new edit after an undo discards the redo branch. Save Click Save to audio. The editor encodes the clip as a 16-bit PCM WAV file and overwrites the asset’s data. The asset’s content type becomes audio/wav, whatever format it had before. The sample rate and channel count are kept. On success a “Saved edits to audio.” notification appears and the editor counts the clip as saved. On failure an error notification shows the reason and your edits stay in the editor. The save replaces the original file, so any workflow that references the asset uses the edited audio from the next run. Click Done to return to View mode. Done does not save. Related Asset Management for uploading, organizing, and viewing audio assets Workspaces for tabs and the View / Edit toggle--- # NodeTool Server Authentication Source: https://docs.nodetool.ai/authentication.html See API Reference for a matrix of endpoints, auth requirements, and streaming behavior. For environment variable defaults and precedence, see the Configuration Guide. Quick Start The server picks its mode from the environment. When both SUPABASE_URL and SUPABASE_KEY are set, it runs in Supabase mode and enforces authentication on every non-public request. Otherwise it runs in Local mode: loopback connections are trusted and run as user "1", and non-loopback requests are rejected with 401 unless they come from a network listed in NODETOOL_TRUST_LOCAL_NETWORKS or carry a valid access token. # Local mode: loopback trusted nodetool serve # Supabase mode: auth enforced on every request export SUPABASE_URL=https://your-project.supabase.co export SUPABASE_KEY=your-service-role-key nodetool serve Authentication Modes The server does not read an AUTH_PROVIDER variable. The mode comes only from the presence of both Supabase variables (packages/websocket/src/server.ts). Supabase mode. The server validates a Supabase JWT on every non-public request, over HTTP and WebSocket. Local mode. LocalAuthProvider maps requests to user "1". Loopback connections bypass auth (gated by NODETOOL_TRUST_LOCALHOST). Other sources are rejected with 401 unless a trust rule, a valid access token, or the SERVER_AUTH_TOKEN bearer token applies. AUTH_PROVIDER is only written into deployed-container environments by the @nodetool-ai/deploy tooling. The server itself never branches on it. In Supabase mode the web UI shows a sign-in screen before anything else loads: Supabase token verification results are cached in-process. The cache defaults to a 60-second TTL and at most 2000 entries. These are constructor options on SupabaseAuthProvider and are not configurable through environment variables. Sending Credentials Clients send the credential as Authorization: Bearer <token>. For WebSocket connections the server also accepts a ?api_key=<token> query parameter when the header is absent. curl -H "Authorization: Bearer $TOKEN" http://localhost:7777/v1/models curl -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -X POST http://localhost:7777/v1/chat/completions \ -d '{ "model": "llama3.2:latest", "messages": [{"role": "user", "content": "Hello"}] }' curl -H "Authorization: Bearer $TOKEN" http://localhost:7777/api/workflows/ Token Types The server recognizes these credentials. Prefixed tokens are checked before the mode’s own rules. Token Prefix Works on Notes Supabase JWT none Everything Supabase mode only. Access token ntk_ Everything, in every mode Minted by a signed-in person for an agent or MCP client. Revocable. A failed check falls through to the mode’s rules. MCP OAuth access token nta_ /mcp only Issued by the MCP OAuth flow. Presented anywhere else, or after the server was rehosted, it is refused with invalid_token. See MCP production. App session token nda_ /ws only Issued to a visitor of a deployed mini app. Confined to that one app. A failed check is refused, never retried against other providers. Delegated token none Everything, as the linked user Used by messaging bridges. Enabled only when NODETOOL_INTEGRATION_TOKEN is set to 16 or more characters. Delegated and app session tokens are signed with keys derived from the master key, so rotating SECRETS_MASTER_KEY revokes every outstanding one. Public Endpoints These paths skip session auth. They are still covered by the per-IP rate limiter. GET /health, GET /ready, GET /api/health GET /api/config GET /api/nodes/metadata GET /api/assets/packages and /api/assets/packages/* GET /api/workflows/public, /api/workflows/public/:id, /api/workflows/examples, and /api/workflows/examples/* GET /api/oauth/hf/callback and GET /api/oauth/github/callback /api/webhooks/*, /api/kie/webhook*, POST /api/providers/fal/webhook/:token, and POST /api/providers/atlascloud/webhook. These verify their own secret or signature. /api/integrations/*. Each handler requires NODETOOL_INTEGRATION_TOKEN. GET /api/apps/:token and POST /api/apps/:token/session, the deployed mini app routes (production only). GET /api/shared-workflows/:token, the read behind a workflow’s public share link. The MCP OAuth surface: /.well-known/oauth-protected-resource*, /.well-known/oauth-authorization-server*, /oauth/authorize, /oauth/token, /oauth/register, and /oauth/revoke. Static web app files, when the server serves the bundled UI. Paths under /api, /ws, /v1, /trpc, and /mcp still require auth. Source: packages/websocket/src/lib/public-routes.ts. Error Responses A rejected HTTP request receives 401 with an error field. Missing token (Supabase mode): { "error": "Unauthorized" } Token rejected by Supabase: { "error": "<reason from the auth provider>" } Non-loopback request in Local mode: { "error": "Remote access requires authentication" } A rejected WebSocket upgrade gets a bare 401 Unauthorized response with no body. /mcp responses also carry a WWW-Authenticate challenge when the MCP OAuth flow can complete. Rate Limiting Requests are limited per client IP before any token verification runs. Loopback is exempt. Excess requests receive 429. Variable Default Effect NODETOOL_RATE_LIMIT_DISABLED unset Set truthy to turn the HTTP limiter off. NODETOOL_RATE_LIMIT_MAX 1000 Requests per window per IP. NODETOOL_RATE_LIMIT_WINDOW_MS 60000 Window length in milliseconds. NODETOOL_RATE_LIMIT_TRUST_PROXY unset Key by X-Forwarded-For instead of the socket address. WebSocket messages have their own cap (NODETOOL_WS_RATE_LIMIT_DISABLED, NODETOOL_WS_RATE_LIMIT_MAX, default 200 messages per window). Source: packages/websocket/src/lib/http-rate-limit.ts, packages/websocket/src/lib/ws-rate-limit.ts. Deploy Token Helpers The @nodetool-ai/auth package defines the abstract AuthProvider base class (header and WebSocket token extraction, verifyToken), plus the AuthResult and TokenType types. The @nodetool-ai/deploy package ships a separate static-token helper set in packages/deploy/src/auth.ts: function getServerAuthToken(): string; // SERVER_AUTH_TOKEN env, then deployment.yaml, then generate and save function getTokenSource(): "environment" | "config" | "generated"; function generateSecureToken(): string; // crypto.randomBytes(32).toString("base64url") function verifyServerToken(auth: string): Promise<"authenticated">; // timing-safe compare function loadAuthConfig(): Record<string, unknown>; function saveAuthConfig(config: Record<string, unknown>): void; // file mode 0600 The token is stored as server_auth_token in ~/.config/nodetool/deployment.yaml (the path is built from the home directory on every platform, including Windows). The running server does not call these helpers. Setting SERVER_AUTH_TOKEN or creating deployment.yaml does not protect the API. The only gates are Supabase mode, the trust settings below, and ntk_ access tokens. To protect a self-hosted server, use Supabase mode. Localhost Trust and Reverse Proxies By default the server bypasses authentication for connections from loopback (127.0.0.1/::1) and runs them as user 1. This is convenient for the desktop app and local development, but dangerous behind a reverse proxy, container network, or SSH tunnel, where the proxy itself connects from loopback — a blanket bypass would silently disable auth in exactly the deployment that needs it. To make this safe: The loopback bypass is gated by NODETOOL_TRUST_LOCALHOST. It defaults off whenever auth is enforced (Supabase mode) and on otherwise. Set it explicitly (true/false) to override. X-Forwarded-For is trusted only for proxies listed in NODETOOL_TRUSTED_PROXIES (comma-separated IPs/CIDRs). When unset, the header is ignored and the unspoofable socket peer address is used to identify the client, so a remote client cannot forge a loopback origin. # Behind an nginx reverse proxy on the same host, enforcing Supabase auth: export SUPABASE_URL=... SUPABASE_KEY=... export NODETOOL_TRUSTED_PROXIES=127.0.0.1 # trust the local proxy's XFF # NODETOOL_TRUST_LOCALHOST stays off — clients must present a valid token. Local mode in Docker Docker NATs a published-port connection to the bridge gateway (e.g. 172.17.0.1), so in Local mode the request never arrives from loopback and the NODETOOL_TRUST_LOCALHOST bypass can’t fire — the UI loads (static assets and /api/config are public) but every API/WebSocket call returns 401 "Remote access requires authentication". NODETOOL_TRUST_LOCAL_NETWORKS (comma-separated source CIDRs) restores the single-local-user model by trusting those sources as user "1" without a token. It is honored only in Local mode — in Supabase mode it is ignored so every request must present a valid token. ⚠️ This bypasses authentication Every source IP in NODETOOL_TRUST_LOCAL_NETWORKS is trusted as admin user "1" with no password — full access to your workflows, files, stored secrets, and API keys. Anyone who can reach the published port from a listed range gets that access. 172.16.0.0/12 — the Docker bridge range on Linux. Host and LAN clients reach the app through the port mapping, but nothing off the bridge is trusted. 192.168.65.0/24 — Docker Desktop’s VM gateway subnet. Published-port traffic on macOS/Windows arrives from 192.168.65.1, not the bridge gateway, so the Linux range alone leaves every call 401 there. The bundled Compose file defaults to both. 0.0.0.0/0 — trusts every source, the whole internet if the port is reachable. Never use this on a public IP. Only on a network you fully control (private LAN / VPN), ideally with the port firewalled. Exposing NodeTool to the internet or untrusted users? Do not widen this list — enable Supabase auth so every request needs a real login. # Docker self-host, single user, no login (safe on a private LAN): NODETOOL_TRUST_LOCAL_NETWORKS=172.16.0.0/12,192.168.65.0/24 # bridge + Desktop VM gateway Security Hardening Production: enable Supabase mode by setting SUPABASE_URL and SUPABASE_KEY, terminate TLS in front of all non-public endpoints, and rotate Supabase service-role keys via your secrets manager. Localhost trust: keep NODETOOL_TRUST_LOCALHOST off in any reverse-proxied or containerized deployment, and list only your real proxies in NODETOOL_TRUSTED_PROXIES. Staging: keep asset buckets private or signed, and run workflows in subprocess or Docker isolation. Development: restrict Local mode to isolated machines and avoid storing real secrets in .env.development. See Security Hardening for detailed checklists across dev, staging, and production.--- # Blender Nodes Source: https://docs.nodetool.ai/blender The Blender nodes run a Blender install in the background to render and convert 3D models. They live in the nodetool.blender namespace. Every node takes a 3D model (GLB, or glTF with embedded buffers) and returns an image, a video, a set of control passes, or a model file. Use them when the preview renderer is not enough: scene cameras and lights are honored, and you can pick EEVEE or Cycles. The 3D editor and its agent use the same Blender install. See 3D Editor. Install Blender NodeTool needs Blender 5.2 or newer on the machine that runs the server. Install it from blender.org or from your system package manager. Older versions fail with an error that names the version found and the minimum. Rendering is off on cloud deployments. It works in the desktop app, on a local server, and on a self-hosted install. Find Blender: BLENDER_PATH and auto-detection NodeTool tries these candidates in order and uses the first one that answers --version: A BLENDER_PATH value saved in your settings. It is ignored on cloud deployments. The BLENDER_PATH environment variable. blender on your PATH. The usual install location for your system. System Locations checked macOS /Applications/Blender.app/Contents/MacOS/Blender Linux /usr/bin/blender and /snap/bin/blender Windows blender.exe in each Blender* folder under Blender Foundation in Program Files and Program Files (x86), newest first If Blender is installed somewhere else, set BLENDER_PATH to the executable itself, not to its folder: export BLENDER_PATH=/opt/blender-5.2/blender The result is cached while the server runs and refreshed when BLENDER_PATH changes. If nothing is found, the node fails before it renders and the error says “blender was not found”. When BLENDER_PATH points at a file that does not run, the error includes the reason. If the server has no Blender but NODETOOL_WORKER_URL names a NodeTool worker with Blender enabled, the job runs on the worker. A Blender found on the local machine always wins over a worker. Common properties RenderImage, RenderAnimation, and RenderPasses share one block of camera, lighting, and engine properties. Property Default What it does Model none The 3D model to render Camera Mode auto auto uses the scene’s first camera and falls back to an orbit camera. scene requires a camera in the model. orbit always uses the orbit camera Width, Height 1024 Output size in pixels, 16 to 4096 Azimuth 45 Horizontal orbit angle in degrees Elevation 25 Camera angle above the horizon, -89 to 89 Field of View 35 Vertical field of view in degrees Zoom 1 Above 1 moves the camera closer, below 1 farther Lighting studio studio, soft, or flat. Used only when the scene has no lights Light Intensity 1 Multiplier for the preset lights Background Color #808080 Hex color, ignored when Transparent is on Transparent Off Render with an alpha channel Engine eevee eevee is fast. cycles is slower and cleaner Samples 16 Samples per pixel Denoise On Applies to Cycles only Resolution Percentage 100 Render scale as a percent of Width by Height Timeout 600 Maximum seconds before the job stops, 1 to 3600 The default background is gray because a white model on a white background is invisible. Nodes Node Inputs Outputs Render 3D With Blender (RenderImage) Model and the common properties image Render 3D Animation With Blender (RenderAnimation) Model, the common properties, Frame Start, Frame End, FPS, Orbit Degrees video Render 3D Passes With Blender (RenderPasses) Model, the common properties, Passes, Depth Format color, depth, depth_near, depth_far, normal, mask Prepare 3D Model For Engine (PrepareForEngine) Model, Target Faces, Unwrap, Bake, Bake Resolution, LOD Count, Timeout model, lods Export 3D Model With Blender (ExportModel) Model, Format, Timeout file RenderImage Renders one frame and returns it as an image. Connect it to any image node, such as an image-to-image or upscale model. RenderAnimation Renders a frame range to an MP4 video. If the model has glTF animations, they play on the timeline. If it has none and Camera Mode is orbit, the camera sweeps around the model. Property Default What it does Frame Start 1 First frame Frame End 24 Last frame, inclusive FPS 24 Frames per second. glTF animation times map onto this timeline Orbit Degrees 360 Camera sweep across the range when the model has no animation and Camera Mode is orbit Timeout is 600 seconds by default. For a turntable clip, use a static model, orbit, and the default 360 degrees. RenderPasses Renders control passes from the same camera, which is useful for guiding video and image models. Property Default What it does Passes color, depth, normal, mask Which passes to produce. Select at least one Depth Format png16 png16 is normalized between depth_near and depth_far, with background 65535. exr is raw float depth with background infinity The passes are color (beauty), depth (distance along the view axis), normal (camera space), and mask (binary foreground). Passes you do not request come back empty. depth_near and depth_far are floats that give the real distance range of the depth image. They are 0 when depth is not requested. PrepareForEngine Cleans up a mesh for a game engine and returns it as a GLB. Property Default What it does Target Faces 5000 Face budget per mesh. Meshes already under it are left alone Unwrap On UV-unwrap every mesh before baking Bake none none, ao (occlusion multiplied into base color), normal (tangent-space normal map), or both Bake Resolution 1024 Pixel size of each baked map LOD Count 0 Extra levels of detail, 0 to 8. Each halves the face count of the one before model is the prepared model. lods is a list with one model per LOD, in order. ExportModel Converts a model to a format Blender writes and stores it as an asset. Format Notes fbx (default) For game engines obj Geometry only, with no .mtl file usd For USD pipelines The output file is an asset with the format and MIME type in its metadata. To keep a GLB, use PrepareForEngine instead. Example: turntable video and an FBX Add a nodetool.constant.Model3D node and choose a GLB asset. See 3D Editor. Connect it to Render 3D Animation With Blender. Set Engine to eevee, Width 1280, Height 720, Frame Start 1, Frame End 120, FPS 24. Keep Camera Mode on orbit. Connect the video output to a save or video node. The result is a five-second turntable. Connect the same model to Export 3D Model With Blender with Format fbx to get a file for a game engine. Limits and errors A render that exceeds its Timeout fails with a message that suggests lowering samples, using EEVEE, or raising the timeout. An empty model input fails with “model input is empty”. A node never returns an empty result silently. A missing Blender, an old version, or a missing output each raise an error that names the cause.--- # Blender Headless Integration — Design Source: https://docs.nodetool.ai/blender-headless-integration-design.html Blender Headless Integration — Design Status: Stages 0–4 implemented, revised after review. The implementation lives in packages/blender-nodes (TypeScript: binary discovery, job contract, LocalBlenderRunner, WorkerBlenderRunner, runBlenderJob, the five nodes), packages/blender-nodes/blender_ops (the Python op scripts each node runs headless), packages/protocol/src/bridge-frames.ts (blender.execute / blender.event), and packages/runtime/src/blender-executor.ts (executeBlender, the bridge client). The worker-side blender_handler.py lives in nodetool-core. Summary Blender runs as a headless processor over the glTF document NodeTool already has. A new packages/blender-nodes package ships nodetool.blender.* nodes that take a Model3DRef and return an image, a video, a model, or an exported file. Every node builds a JSON job and hands it, with its input bytes, to a BlenderRunner that works on logical file names only. The local runner owns a scratch directory and spawns blender -b --python on a vendored Python op script through the bounded host binary runner ffmpeg already uses. The worker runner, a later stage, sends the same job and bytes as bridge blobs to a GPU worker, the way the ComfyUI worker works. Nodes never know which tier ran them, and the local tier is hardened but not sandboxed: the worker is where an isolation boundary exists. Goals Render a glTF scene to a still, to render passes (depth, normal, mask), and to an animation, at a quality the three.js preview renderer cannot reach. Prepare a generated mesh for a game engine: remesh, decimate, unwrap, bake, export FBX, OBJ, USD, or GLB. Feed video models with camera-consistent frames and control passes from the same scene. Keep the existing 3D editor, ui_3d_* tools, and model3d agent capability working on the same asset before and after a Blender step. Fail with a message that names the fix when Blender is missing, too old, or the scene cannot be imported. Non-goals A Blender scene editor in the web app. The document stays glTF and the existing editor stays the editor. Arbitrary user Python inside Blender in the first release. A gated RunScript node is listed under alternatives, not in scope. Bundling Blender into the Electron installer or the Fly server image. Simulation, physics, geometry nodes, and rigging. The op catalog can grow, but none of these are in the stages below. Background jobs with a ledger row. Long renders stay synchronous under the node’s timeout until the worker tier exists. Current State The 3D document and its nodes packages/model3d is the scene document: glTF 2.0, parseModel3D, serializeModel3D, the Model3DOperation union, validateModel3D. Model3DRef is { type: "model_3d", uri, asset_id?, temp_id? } in packages/protocol/src/api-types.ts. It has no format field. Node code reads the looser Model3DRefLike with data, format, vertices, faces (packages/video-nodes/src/nodes/model3d/types.ts), and format does not survive a crossing into the protocol type. The generic stored-file type is AssetRef (type: "asset", uri, asset_id, metadata). packages/video-nodes/src/nodes/model3d/ ships 18 nodetool.model3d.* nodes: load and save, FormatConverter, Transform3D, Decimate, Boolean3D, RepairMesh, MergeMeshes, TextTo3D, ImageTo3D, and RenderToImage. They are pure TypeScript over @gltf-transform/core, manifold-3d, and meshoptimizer. RenderToImage launches headless Chromium over CDP with SwiftShader and renders through three.js (render3d-headless.ts). It produces previews: one still, preset lighting, no animation, no shadows of quality, no passes. packages/agents/src/capabilities/model3d.ts exposes list_model3ds, create_model3d, get_model3d, edit_model3d, validate_model3d. They apply Model3DOperations to the asset bytes headlessly. Host binaries packages/agents/src/host-binaries.ts owns runHostBinary(cmd, args, { cwd, timeoutMs, artifactPath?, maxArtifactBytes? }). It spawns without a shell, bounds wall clock (SIGTERM then SIGKILL), captured output (MAX_CAPTURED_BYTES), artifact size, and concurrency (maxConcurrentHostBinaries, env NODETOOL_HOST_BINARY_CONCURRENCY). A missing binary is HostBinaryMissingError. host-binary-guard.ts confines model-authored argv to the workspace. @nodetool-ai/agents depends on @nodetool-ai/video-nodes, so a node package cannot import runHostBinary without a dependency cycle. Files reach a binary through Workspace (packages/runtime/src/workspace.ts): materialize, absorb, scratchDir, and localDir, which is null on a cloud workspace. ProcessingContext.signal is the run-level AbortSignal the kernel aborts on WorkflowRunner.cancel(). Remote tools createPythonBridge (packages/runtime/src/python-bridge-factory.ts) returns a websocket bridge when NODETOOL_WORKER_URL is set and a stdio bridge otherwise. The stdio bridge refuses to run in production. The ComfyUI worker image co-locates a loopback-only ComfyUI and proxies it over comfy.execute and comfy.event frames (packages/protocol/src/bridge-frames.ts, packages/runtime/src/comfy-executor.ts). Media inputs travel as bridge blobs. packages/compute provisions and reaps such workers on RunPod and Vast. Distribution The Fly image (Dockerfile, node:22-slim) installs ffmpeg, chromium, pandoc, and a Python venv. Blender is not there. Electron installs tools on demand through CondaRuntimePackage (electron/src/runtime/packages/definitions.ts). conda-forge has no blender package, so that path is closed for the executable. PyPI ships bpy wheels pinned to one Python minor per release. Package runtime files are declared in PACKAGE_RUNTIME_ASSETS and PACKAGE_RUNTIME_ASSET_DIRS (packages/config/src/package-asset-registry.ts) so the Electron bundle stages them. Harness packages/cli/src/harness/registry.ts maps diff paths to surfaces and checks. capability-table.ts is generated by npm run capabilities:sync and fails capabilities:check on an uncovered capability. Affected components packages/agents/src/host-binaries.ts moves to packages/runtime. New packages/blender-nodes. packages/base-nodes/src/index.ts registers the new node array. packages/config/src/package-asset-registry.ts declares the op script directory. packages/agents/src/capabilities/model3d.ts gains render_model3d. packages/cli/src/harness/registry.ts gains a blender surface. packages/protocol/src/bridge-frames.ts and packages/runtime gain blender.execute in the worker stage. packages/config/src/setting-catalog.ts gains BLENDER_PATH, and start.sh doctor a Blender line, in Stage 1. Assumptions A1. Blender 5.2 LTS is the floor. The glTF importer and exporter are core add-ons enabled under --factory-startup. EEVEE Next and Cycles both render headless on CPU with no display. Revision (Stage 3): the floor moved from 4.2 LTS to 5.2 LTS. Nothing ever ran on 4.2: render_passes.py reads scene.compositing_node_group (the 5.x compositor entry point; 4.2 has only the legacy use_nodes/node_tree pair) and render_animation.py sets image_settings.media_type (the 5.x image/video split; 4.2 exposes file_format directly). On a 4.2 binary both ops raise AttributeError, so the old floor promised what the code could not do. A 4.2 backport would need the compositor tree rebuilt on the legacy Scene.node_tree API and the animation output set through file_format = "FFMPEG" with no media_type flip — untested here, since the only binary available is 5.2 LTS. A2. Desktop users who want Blender nodes install Blender themselves. The node discovers it. This holds until a first-use download exists. A3. The Python worker image is built outside this repository. Stage 4 here covers the protocol and the TypeScript client only. A4. Cycles on CPU is minutes per frame at production samples. Stills at preview samples and EEVEE animations are seconds to a minute. Unclear or conflicting requirements U1. Explainer animation needs authored camera paths and text objects. The Model3DOperation union has no camera or text primitive. This design adds camera presets on the render node and defers scene-side camera authoring. If explainers need per-shot camera keyframes soon, add_object needs a camera kind and set_transform needs keyframes, which is a change to packages/model3d, not to this package. U2. Cloud profile. On Fly there is no Blender and no worker by default. The nodes are excluded from the cloud profile until a Blender-enabled worker image ships (D8), so a cloud user never sees a node that cannot run. Proposed Design D1. One package, one namespace packages/blender-nodes (@nodetool-ai/blender-nodes), node types nodetool.blender.*. Dependencies: node-sdk, runtime, protocol, config, nodes-utils. Layout mirrors video-nodes: src/nodes/*.ts, a BLENDER_NODES array, tests/, and the ./nodes/* subpath export so base-nodes imports the array. The Python op script lives at packages/blender-nodes/blender_ops/ next to the sources. A new package rather than a subdirectory of video-nodes because the op script is a runtime asset directory with its own staging rule, and because a cloud profile allowlists by namespace. D2. runHostBinary moves to runtime packages/agents/src/host-binaries.ts moves to packages/runtime/src/host-binaries.ts with the same exports. @nodetool-ai/agents re-exports it so media.ts and its tests change only their import path. host-binary-guard.ts stays in agents. It confines argv a model wrote. Blender argv is built by the node from typed props, and every path in it is a file the node itself wrote into the scratch directory. RunHostBinaryOptions gains four optional fields. signal?: AbortSignal: on abort the runner sends SIGTERM and follows the existing SIGKILL path. env?: Record<string, string>: the child’s whole environment when set, instead of process.env. onStderrLine?: (line: string) => void: fed from the same stream the capture reads. concurrencyClass?: string: selects the semaphore. The default class keeps the existing NODETOOL_HOST_BINARY_CONCURRENCY cap for ffmpeg and yt-dlp. A render class is capped by NODETOOL_BLENDER_CONCURRENCY, default 1, so a two-minute Cycles render never holds a slot a two-second ffmpeg call is waiting for. Existing callers pass none of the four and see no change. D3. Binary discovery packages/blender-nodes/src/blender-binary.ts: export interface BlenderBinary { path: string; version: [number, number, number]; } export async function resolveBlenderBinary(): Promise<BlenderBinary>; Order: BLENDER_PATH, then blender on PATH, then well-known locations (/Applications/Blender.app/Contents/MacOS/Blender, /usr/bin/blender, /snap/bin/blender, %ProgramFiles%\Blender Foundation\Blender *\blender.exe). The first candidate that runs --version wins. Below 5.2 throws BlenderVersionError naming the found version and the floor. No candidate throws HostBinaryMissingError("blender"). The result is cached per process and invalidated when BLENDER_PATH changes. D4. The job contract Every node produces a BlenderJob. The Python side consumes it and writes a BlenderResult. Both are versioned so the TypeScript and Python halves can drift by one version during an upgrade. The job names every file on both sides. The result never names a file: it reports which declared outputs were produced, and the host reads only the paths the job itself declared. This is the invariant that keeps a buggy or compromised op script from turning result processing into an arbitrary file read, and T2 tests it. // packages/blender-nodes/src/job.ts export const BLENDER_JOB_VERSION = 1; export type BlenderOp = | { op: "render_image"; params: RenderImageParams } | { op: "render_passes"; params: RenderPassesParams } | { op: "render_animation"; params: RenderAnimationParams } | { op: "prepare_for_engine"; params: PrepareForEngineParams } | { op: "export_model"; params: ExportModelParams }; /** A bare file name: no separator, no `..`, no leading dot. */ export const jobFileNameSchema = z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]*$/); export interface BlenderJob { version: typeof BLENDER_JOB_VERSION; /** Logical input name -> bare file name. The runner writes these. */ inputs: { model: string }; /** Logical output name -> bare file name the op must write. */ outputs: Record<string, string>; job: BlenderOp; } export const blenderResultSchema = z.discriminatedUnion("ok", [ z.object({ ok: z.literal(true), /** Logical output names the op wrote. Must be a subset of job.outputs. */ produced: z.array(z.string()), stats: z.object({ blender_version: z.string(), render_seconds: z.number(), frames: z.number().int().optional(), objects: z.number().int().optional() }) }), z.object({ ok: z.literal(false), error: z.object({ code: z.enum([ "import_failed", "no_geometry", "unsupported_format", "render_failed", "export_failed", "bad_job" ]), message: z.string() }) }) ]); Camera params reuse the RenderToImage vocabulary so a user can swap the preview node for the Blender node without relearning: azimuth, elevation, fov, zoom, lighting, light_intensity, background_color, transparent. Blender-specific additions: engine (eevee | cycles), samples, denoise, resolution_percentage. camera_mode decides whose camera renders, because the orbit props always carry a default and a default cannot signal intent: camera_mode Behavior auto (default) The scene’s first camera when the glTF carries one, else an orbit camera from the props. scene The scene’s first camera. No camera is no_camera, an error. orbit Always an orbit camera from the props. The scene’s cameras are ignored. Lights follow the same rule without a switch: the scene’s lights when it has any, else the lighting preset. A test pins each branch. Output contracts the ops must honor: depth: linear distance along the camera’s view axis, in scene units, from the Z pass. depth_format is png16 (default) or exr. In png16 the value is normalized to [0, 65535] between depth_near and depth_far, the min and max finite depth in the frame, both returned as floats, and background pixels are 65535. In exr the value is the raw float, and background is +inf: the staged EXR carries the 1e10 no-hit sentinel (measured 5.2.1 on both engines), which the op rewrites to +inf before the output is written. Control-pass consumers get png16. Anything that needs precision gets exr. normal: camera-space normals from the Normal pass, mapped from [-1, 1] to 8-bit RGB. Background is (128, 128, 255). mask: 8-bit alpha of the object index pass, foreground 255. The pass is enabled before the Render Layers node is created (measured 5.2.1: Cycles then lists Image/Alpha/Depth/Object Index/Noisy Image) and every mesh renders with object index 1, so the mask keys on a positive index with background 0. Deviation (recorded, not silent, EEVEE only): EEVEE’s CompositorNodeRLayers exposes no index socket (measured 5.2.1: only Image/Alpha/Depth appear with use_pass_object_index on), so EEVEE keys the mask on finite Z instead. The gate agrees with the index pass on opaque geometry and disagrees for alpha-blended, holdout, and volume materials. EEVEE’s no-hit Z value 1e10 is a measured EEVEE behavior on 5.2.1, not a documented API value: re-measure it if the version floor moves. render_animation: the scene fps is set to fps. A glTF animation channel’s timestamp t seconds lands on frame round(t * fps). frame_start and frame_end are frames in that timeline. When the glTF has no animation and camera_mode is orbit, the orbit turns orbit_degrees across the frame range. D5. The op script packages/blender-nodes/blender_ops/run_job.py plus one module per op. Declared in PACKAGE_RUNTIME_ASSET_DIRS as { pkg: "@nodetool-ai/blender-nodes", path: "blender_ops", bundleDir: "_blender_ops", files: [...] } with every file named, so an unstaged module fails the bundle verifier instead of the product. Invocation, built by LocalBlenderRunner: blender -b --factory-startup --disable-autoexec \ --python-exit-code 64 --python <asset_dir>/run_job.py -- job.json run_job.py reads job.json, dispatches on job.op, and always writes result.json, including on exception (ok: false, code from the exception class, message from the exception). It sets sys.excepthook before importing op modules so an import error also lands in result.json. Exit code 64 is reserved for “the script raised”, distinct from Blender’s own crash codes. Scene setup is the same for every render op: bpy.ops.wm.read_factory_settings(use_empty=True), bpy.ops.import_scene.gltf(filepath=inputs.model), then the camera camera_mode selects (D4). An orbit camera is placed by the same auto-framing math render3d-core.ts uses (computeFraming, orbitOffset), ported once into blender_ops/framing.py and pinned by a fixture test on both sides (T3). Video output for render_animation uses Blender’s own FFMPEG writer (image_settings.file_format = "FFMPEG", MPEG-4 container, H.264, yuv420p), so the package needs no ffmpeg on PATH and no Mediabunny dependency. D6. The runner interface and runBlenderJob The runner works on logical files and nothing else. No scratch directory, no argv, and no path crosses the interface, so the local and worker implementations differ only in where the bytes go. // packages/blender-nodes/src/runner.ts export interface BlenderRunOptions { timeoutMs: number; signal: AbortSignal; onProgress?: (frame: number, total: number) => void; /** Per-output byte cap. Default MAX_OUTPUT_BYTES (512 MiB). */ maxOutputBytes?: number; /** Cap on the sum of all outputs. Default MAX_TOTAL_OUTPUT_BYTES (1 GiB). */ maxTotalOutputBytes?: number; } export interface BlenderRunResult { outputs: Record<string, Uint8Array>; // keyed by job.outputs name stats: BlenderStats; } export interface BlenderRunner { readonly kind: "local" | "worker"; run( job: BlenderJob, inputs: Record<string, Uint8Array>, // keyed by job.inputs name options: BlenderRunOptions ): Promise<BlenderRunResult>; } runBlenderJob(context, modelBytes, op, outputs, options) is the thin function nodes call. It validates every input and output file name with jobFileNameSchema, refuses more than MAX_OUTPUT_COUNT (32) declared outputs, builds the BlenderJob, picks a runner (D7), and returns the runner’s result. A node never touches a runner. LocalBlenderRunner.run: const cwd = await context.workspace.scratchDir(). Write each input under its declared bare file name, then job.json. Spawn through runHostBinary(binary.path, argv, { cwd, timeoutMs, signal, concurrencyClass: "render", env }). env is an allowlist: PATH, HOME, TMPDIR, LANG, SYSTEMROOT, CUDA_VISIBLE_DEVICES, plus BLENDER_USER_CONFIG, BLENDER_USER_SCRIPTS, and BLENDER_USER_EXTENSIONS pointed at an empty directory under cwd, so the user’s add-ons and startup scripts never load. No artifactPath: the host runner’s single-file watchdog cannot cover a multi-output job, so enforcement lives in step 5. Read result.json. Parse with blenderResultSchema. A missing or unparsable file is BlenderJobError("bad_result") carrying the last 4 KiB of stderr. On ok: false, throw BlenderJobError(code, message). For every name in job.outputs, stat the declared path before reading it. A declared output missing from disk or absent from produced is missing_output. A file above maxOutputBytes, or a running total above maxTotalOutputBytes, is output_too_large naming the output and the cap, thrown before the file is read into memory. A name in produced that the job did not declare is ignored and logged at warn. Delete the scratch directory in finally, including on abort and on the cap errors above. WorkerBlenderRunner.run (Stage 4) sends the same inputs as bridge blobs, the job inside a blender.execute frame, and applies the same step 5 caps to the blobs that come back, by declared size before transfer where the bridge reports one. Progress: Blender prints Fra:<n> lines on stderr during animation renders. The local runner turns them into onProgress calls through onStderrLine; the worker runner reads blender.event frames. The node turns either into node_progress messages through context.postMessage, the way the ComfyUI node does. D7. Runner selection export async function resolveBlenderRunner(): Promise<BlenderRunner>; Stage 1 ships LocalBlenderRunner only. Stage 4 adds WorkerBlenderRunner, selected when no local Blender resolves, NODETOOL_WORKER_URL is set, and the worker reports worker.status.blender.enabled, the same selector createPythonBridge uses. A local binary wins when both exist, so a desktop with Blender installed never pays for a worker. A configured worker that fails its status check is an error, never a silent fallback to local: the URL is a deployment choice. D8. Nodes All nodes: model input (model_3d), timeout prop in seconds (default 600, the ComfyUI node’s default), and @prop metadata in the video-nodes style. Outputs are inline refs (`{ type, uri: “”, asset_id: null, data: }`) like `RenderToImage`, so downstream save nodes decide persistence. | Node | Op | Inputs beyond `model` | Outputs | |---|---|---|---| | `nodetool.blender.RenderImage` | `render_image` | `camera_mode`, camera and engine params (D4) | `image` | | `nodetool.blender.RenderPasses` | `render_passes` | same, plus `passes` multi-select and `depth_format` | `color`, `depth`, `depth_near`, `depth_far`, `normal`, `mask` (contracts in D4) | | `nodetool.blender.RenderAnimation` | `render_animation` | `camera_mode`, `frame_start`, `frame_end`, `fps`, `orbit_degrees` | `video` | | `nodetool.blender.PrepareForEngine` | `prepare_for_engine` | `target_faces`, `unwrap`, `bake` (`none` \| `ao` \| `normal` \| `both`), `bake_resolution`, `lod_count` | `model` (GLB), `lods` (list of GLB) | | `nodetool.blender.ExportModel` | `export_model` | `format` (`fbx` \| `obj` \| `usd`) | `file` (`AssetRef`) | `ExportModel` does not return a `Model3DRef`. FBX, OBJ, and USD are not glTF documents, and `Model3DRef` has no `format` field, so a socket typed `model_3d` carrying an FBX would misread the moment it crossed a strict boundary. The node persists the export through `context.createAsset` and returns `{ type: "asset", uri: "asset://", asset_id, metadata: { format, mime } }`. GLB is not an `ExportModel` format because `PrepareForEngine` and the existing `FormatConverter` already produce a `Model3DRef` for it. The `model_3d` socket keeps meaning glTF everywhere. Cloud profile: the `nodetool.blender` namespace stays out of the cloud allowlist in `packages/protocol/src/cloud-profile.ts`, Stage 4 included. The worker image is built outside this repository (A3), so a cloud server cannot assume a Blender-enabled worker is reachable, and a namespace that is listed but cannot run is worse than one that is absent. `validate_workflow` reports the nodes as unavailable on a cloud server instead of failing at run time. The namespace joins the allowlist when a Blender-enabled worker image ships with the cloud deployment. A self-hosted install gets the worker tier now: `NODETOOL_NODE_PROFILE=full` keeps the namespace, and `NODETOOL_WORKER_URL` pointing at a worker whose status reports Blender selects `WorkerBlenderRunner` (D7). ### D9. Agent capability `render_model3d` joins `packages/agents/src/capabilities/model3d.ts`: input `model_id` plus the `render_image` params, output `{ image_id, url, stats }`. It reuses `runBlenderJob` and stores the PNG through `context.createAsset`. Permission category `write` (it creates an asset). Extending the existing module keeps the 3D capabilities in one place for the eval surface. Availability follows the `yt_dlp` / `browser_*` pattern exactly: the cloud profile drops `render_model3d` from the offered belt (`availableBuiltinToolNames`, via `isBlenderEnabled` in `packages/agents/src/blender-gate.ts`, the same `NODETOOL_NODE_PROFILE` switch D8 uses for the `nodetool.blender` namespace), and the implementation refuses on its own when reached by name, so a guest that imports the module still gets a deployment answer instead of a run failure. The gate stays closed on the cloud profile until a Blender-enabled worker image ships, for the reason D8 gives. A self-hosted install with `NODETOOL_NODE_PROFILE=full` serves the capability, and with `NODETOOL_WORKER_URL` it renders through the worker tier. A non-cloud server with neither binary nor worker still serves the capability and fails with the cause named ("this server has no Blender installed..."). ### D10. Configuration | Setting | Where | Default | |---|---|---| | `BLENDER_PATH` | env, settings catalog (Stage 1) | unset | | `NODETOOL_BLENDER_CONCURRENCY` | env, the `render` class cap | 1 | | `NODETOOL_HOST_BINARY_CONCURRENCY` | env, existing, the default class | 2 | | `NODETOOL_WORKER_URL`, `NODETOOL_WORKER_TOKEN` | env, existing | unset | | `MAX_OUTPUT_BYTES`, `MAX_TOTAL_OUTPUT_BYTES`, `MAX_OUTPUT_COUNT` | constants in `runner.ts`, overridable per call | 512 MiB, 1 GiB, 32 | No new database rows. No feature flag: a namespace that is absent from a profile and a node that throws a named error when Blender is missing are the gates. ## Execution Flow `RenderImage.process(context)`: 1. `resolveModelBytes(this.model, context)`, shared with `RenderToImage` (moved to `nodes-utils` so both packages import it). Empty bytes throw before Blender is touched. 2. Build `op = { op: "render_image", params }` and `outputs = { image: "render.png" }`. 3. `runBlenderJob(context, bytes, op, outputs, { timeoutMs, signal: context.signal, onProgress })`, which resolves the runner and calls `runner.run(job, { model: bytes }, options)`. 4. Return `{ image: { type: "image", uri: "", asset_id: null, data: bytesToBase64(result.outputs.image) } }`. Failure paths: - Blender absent: `HostBinaryMissingError("blender")` from step 3 before any file is written. The message names `BLENDER_PATH`. - Blender too old: `BlenderVersionError` with both versions. - glTF the importer rejects: `result.json` with `import_failed` and the importer's message. The node rethrows it with the node name prefixed. - Timeout: `runHostBinary` kills the child. The scratch directory is deleted. The error names the timeout and suggests lower samples or EEVEE. - Cancellation: `context.signal` aborts, the runner kills the child, the node rejects with the abort reason. No partial output is returned. - Blender crashes (segfault, exit code outside 0 and 64) with no `result.json`: `BlenderJobError("bad_result")` with the stderr tail. - An output over its cap: `output_too_large` from the `stat` in D6 step 5, before any byte is read. The scratch directory is deleted. - `camera_mode: scene` on a glTF with no camera: `no_camera` from the op, before any render time is spent. ## API / Interface Changes - `@nodetool-ai/runtime` exports `runHostBinary`, `HostBinaryMissingError`, `MAX_CAPTURED_BYTES`, `MAX_ARTIFACT_BYTES`, `maxConcurrentHostBinaries`. `@nodetool-ai/agents` keeps re-exporting them. - `RunHostBinaryOptions` gains `signal?`, `env?`, `onStderrLine?`, and `concurrencyClass?` (D2). All optional, so existing callers are unchanged. - `BlenderRunner`, `BlenderRunOptions`, `BlenderRunResult`, and `runBlenderJob` in `@nodetool-ai/blender-nodes` (D6). - New node types listed in D8. `ExportModel` returns an `AssetRef`. - New capability `render_model3d` (D9). - Stage 4: `blender.execute` request with `{ job, inputs: blob keys, timeout }` and `blender.event` progress frames in `packages/protocol/src/bridge-frames.ts`, `executeBlender` in `packages/runtime/src/blender-executor.ts`, and `worker.status.blender.enabled` in the status schema. Shapes mirror `comfy.execute` and `comfy.event`. ## Data Model Changes None. `Model3DRef` is unchanged and keeps meaning glTF. Rendered outputs are ordinary image and video assets. An `ExportModel` output is an ordinary stored asset with `format` and `mime` in its `metadata`. `BlenderJob` and `BlenderResult` are files in a scratch directory that is deleted after the run, never persisted. ## Alternatives Considered ### `bpy` wheel from PyPI instead of a Blender executable Advantages: pip-installable into the existing Python venv or conda env, so the Electron Package Manager could install it, and the Python worker already has Python. Disadvantages: each `bpy` release pins one Python minor (the current one requires 3.13 exactly), the wheel is a few hundred MB, and the op script would run inside NodeTool's Python process instead of a child the host runner can bound and kill. Rejected for now. The job contract in D4 is process-agnostic, so a `bpy` runner can be added as a third `BlenderRunner` without changing a node. ### Extend `RenderToImage` with an `engine: "blender"` option Advantages: no new package, one render node. Disadvantages: the node's props would fork on engine, `video-nodes` would take on the op script asset directory, and the cloud profile could not exclude Blender without excluding the preview renderer. Rejected. ### Separate worker node types, like `lib.comfy.RunWorkflowOnWorker` Advantages: explicit `worker_url` and `worker_token` props, no hidden selection. Disadvantages: every Blender node doubles, and a graph authored on a desktop stops working on a worker without editing every node. Rejected in favor of the `createPythonBridge` selector, which the repository already uses for the same question. ### Add `format` to `Model3DRef` and return exports as models Advantages: one 3D socket type, FBX and USD flow through the same edges as GLB. Disadvantages: every consumer of `model_3d` assumes glTF, from the editor to `validate_model3d` to `RenderToImage`, so the field would need a check at each of them, and a missing `format` on old refs would have to mean `glb` forever. Rejected until NodeTool decides `Model3DRef` is a general 3D asset. An `AssetRef` with `metadata.format` carries the same information without changing what an existing edge means. ### A gated `RunScript` node for user Python Advantages: unbounded flexibility, the Blender community's scripts run as is. Disadvantages: `bpy` reaches the filesystem, the network, and `subprocess`. Confining it needs a sandbox this design does not have. Deferred. If it ships, it goes in the Developer Tools group, off in the cloud profile, and runs only on a local workspace. ## Edge Cases - Two nodes render at once. The `render` class semaphore queues the second. Each has its own scratch directory, so no file collides. An ffmpeg call in the default class runs alongside either. - The same asset rendered twice with the same params produces the same bytes only when `samples` and the seed are fixed. Cycles is seeded from `scene.cycles.seed`, which the op sets to a constant. Idempotency is by construction, not by caching. - A glTF with no mesh: `no_geometry`, before any render time is spent. - A glTF with its own camera and lights under `camera_mode: auto`: both used as is. Under `orbit`: the node's camera, the scene's lights. A test pins each of the three modes with and without a scene camera. - An animation whose MP4, or a `RenderPasses` run whose four images together, pass a cap: `output_too_large` from the post-run `stat`, before the bytes are read. Blender is not killed mid-write, because the caps are checked after exit, so a single runaway frame sequence is bounded by the timeout and the scratch disk, not by the cap. That is accepted for Stage 1 and noted in R6. - `result.json` lists a `produced` name the job did not declare, or a path appears anywhere in it: the name is ignored, the path is never read, and the run is logged at warn. - `BLENDER_PATH` points at a file that is not Blender: `--version` fails, the error quotes the path and the first stderr line. - Blender writes a `.crash.txt` into its temp directory (`$TMPDIR`) on a segfault, never next to the scratch files. It is included in the `bad_result` message when present. - Process crash of the NodeTool server mid-render: the child is orphaned. Stage 1 accepts this. Stage 4's worker owns its process tree. - Cloud workspace: `scratchDir()` returns a real temp directory on the server, so the node runs without `localDir`. Nothing is written to the workspace itself. ## Performance and Scalability The hot path is Blender itself. Startup is one to three seconds. An EEVEE still at 1024 px is a few seconds on CPU. Cycles at 128 samples is tens of seconds to minutes per frame. Node overhead is one GLB write, one PNG read, and JSON parsing, all negligible against that. Concurrency is bounded by the `render` class cap, `NODETOOL_BLENDER_CONCURRENCY`, separate from the ffmpeg and yt-dlp class because a render holds a core for minutes. A render farm needs the worker tier, not a higher local cap. Output memory is bounded by `maxTotalOutputBytes`, and the check runs on file sizes before any file is read, so the peak is one job's outputs, never an unbounded record of buffers. No caching in this design. When the same scene is rendered repeatedly with the same job, the win is real, but no evidence yet says it is a bottleneck, and the hash key (bytes plus job) is a small follow-up if it becomes one. ## Security and Privacy What the local runner is: hardening. What it is not: a sandbox. Blender runs as the NodeTool OS user with that user's filesystem and network reach, and nothing in this design changes that. The claims below are the ones the implementation can keep. - Trust boundary: the glTF bytes and the props come from the workflow author. The op script and Blender come from the install. The only paths NodeTool passes to Blender are files it wrote into the scratch directory, and the only paths it reads back are the ones the job declared (D4). Neither stops `bpy` code from opening other paths. Nothing in Stage 1 runs `bpy` code NodeTool did not ship. - `--disable-autoexec` stops scripts embedded in a scene from running. `--factory-startup` and the `BLENDER_USER_*` redirects stop the user's add-ons and startup scripts from loading. A glTF carries no script, so the first of these matters only once `.blend` inputs exist. - The env allowlist keeps the secrets in `process.env` out of the child. It does not stop outbound connections. The op script opens no socket, and the job carries no URL, but Blender itself is not network-isolated here. - Blender's argv is fixed in `LocalBlenderRunner`: every param travels inside `job.json`, which the op script reads with `json.load`, so no param value can become a Blender flag and no flag-injection check exists here. The `passes` list is filtered against known constants by the node. File names in the job pass `jobFileNameSchema`, so `..` and separators never reach `job.json`. - Untrusted Blender execution, meaning a `RunScript` node or a `.blend` from someone else, needs the worker tier. The worker is a container NodeTool provisions and reaps, with a bearer token and blobs as the only file channel. That is the isolation boundary, and the cloud profile keeps the nodes off the shared server until it exists (D8). - Logs carry Blender's stderr tail. It can contain file names from the scratch directory, never user secrets. ## Observability - The kernel's `node.process` span wraps the run. `runBlenderJob` adds attributes: `blender.version`, `blender.op`, `blender.engine`, `blender.runner` (`local` | `worker`), `blender.render_seconds` from `stats`, `blender.exit_code`, and `blender.queued_ms` when the concurrency slot waited. - `node_progress` per frame on animation renders. - One structured log line per run at info with the same fields, and at warn on non-zero exit with the stderr tail. - `nodetool node run nodetool.blender.RenderImage --props ...` is the diagnostic entry point. `--no-secrets` applies because the node needs none. ## Rollout and Migration - Stage 0 is a pure move plus additive options. It ships alone and can be reverted alone. - Stages 1 to 3 add a package that no existing graph references. Absent Blender, the nodes throw a named error and nothing else changes. - Stage 4 adds bridge frames with a new `type`, ignored by older workers, and a status flag that defaults to false. It changes no profile: the cloud allowlist is untouched, so the cloud product is the same before and after. - Rollback at any stage is a revert. No migration, no persisted schema. - Hard to reverse: the `BlenderJob` version 1 shape once graphs and worker images depend on it. Version it from day one (D4) and reject an unknown version in `run_job.py` with `bad_job`. ## Testing Strategy - T1 unit (`packages/runtime/tests/host-binaries.test.ts`): abort through `signal` kills a `sleep` child, `onStderrLine` receives lines, env allowlist excludes an injected `SECRET` variable. Invert each once. - T2 unit (`packages/blender-nodes/tests/job.test.ts`): every node builds the expected `BlenderJob` from its props, `blenderResultSchema` rejects a result with an unknown error code, `jobFileNameSchema` rejects `../x.png` and `/tmp/x.png`. Against the fake Blender: a `result.json` whose `produced` names an undeclared output, or that carries a path, is ignored and the path is never opened (asserted with a sentinel file that must stay unread). A fake that writes one output over `maxOutputBytes`, and one that writes four outputs whose sum passes `maxTotalOutputBytes`, each fail with `output_too_large` before any read, with the scratch directory gone. Invert each once. - T2b unit (`camera-mode.test.ts`): the three `camera_mode` values against a fixture with a camera and one without, asserting which camera the job selects and that `scene` without a camera is `no_camera`. - T3 unit, two sides (`framing.test.ts` and `blender_ops/tests/test_framing.py`): the same fixture bounds and camera params produce the same camera position to four decimals. The Python test runs under Blender's own interpreter in CI when Blender is present and is skipped otherwise. - T4 integration (`packages/blender-nodes/tests/render-image.test.ts`): render a fixture GLB and assert the PNG decodes, has the requested size, and is not uniform. Skipped without Blender, the way `model3d-render.test.ts` skips without Chrome. CI installs Blender on the Linux leg so the test runs there. - T5 failure paths: a fake `blender` script on PATH that exits 64 without `result.json`, one that writes `ok: false`, one that never exits (timeout at 1 s), and a job aborted mid-run. Each asserts the error class and that the scratch directory is gone. - T6 concurrency: two fake renders with `NODETOOL_BLENDER_CONCURRENCY=1` assert the second starts only after the first finishes, and a fake ffmpeg call in the default class starts while a render holds the `render` slot. - T6b runner seam: a `FakeBlenderRunner` implementing `BlenderRunner` records the `job` and `inputs` it receives, and every node test runs against it. This is what proves a node never reaches past the interface. - T7 regression: `RenderToImage` tests pass unchanged after `resolveModelBytes` moves to `nodes-utils`. - T8 capability: `packages/agents/tests/capabilities-model3d.test.ts` gains `render_model3d` against the fake Blender, and `capabilities:check` passes. - T9 harness: the `blender` surface in `registry.ts` names the suites above, with a `selfcheck` running `node run nodetool.blender.RenderImage` on a fixture, cost `expensive`, so `harness gate` runs it on diffs touching `packages/blender-nodes/`. - T10 worker stage: bridge frame round-trip tests in `packages/protocol` mirroring the `comfy.execute` ones, and a fake worker in `packages/runtime/tests` that answers `blender.execute` with blobs. ## Implementation Plan 1. Stage 0, runner move. Move `host-binaries.ts` to runtime, add `signal`, `env`, `onStderrLine`, `concurrencyClass`. Update the agents import. Tests: T1, T6. 2. Stage 1, `RenderImage` local tier. New package, `blender-binary.ts`, `job.ts`, `runner.ts` with `BlenderRunner` and `LocalBlenderRunner`, `run-job.ts`, `run_job.py` with `render_image` and `framing.py`, asset dir registration, `base-nodes` registration, `resolveModelBytes` move, registry surface, CI Blender install, the `BLENDER_PATH` settings catalog entry, and a Blender line in `start.sh doctor`. Tests: T2, T2b, T3, T4, T5, T6b, T7, T9. 3. Stage 2, `RenderPasses` and `RenderAnimation`. Compositor node tree for passes with the D4 contracts, FFMPEG output, `Fra:` progress. Tests: T2 additions, T4 for each output including a depth image whose near and far match a known fixture, a progress test against the fake Blender. 4. Stage 3, `PrepareForEngine`, `ExportModel`, and `render_model3d`. Tests: T2 additions, T4 asserting exported FBX magic bytes, the `AssetRef` metadata, and GLB validity through `validateModel3D`, T8. 5. Stage 4, worker tier. Bridge frames, `blender-executor.ts`, `WorkerBlenderRunner`, status flag. The cloud allowlist stays as it is (D8). Tests: T10, plus T6b re-run with the worker runner behind the fake worker. The worker image is a separate deliverable outside this repository, and the cloud allowlist entry waits for it. ## Open Questions - Q1. Does the explainer use case need scene-side camera and text authoring (U1) in the same release? If yes, `packages/model3d` grows a `camera` and `text` primitive first, and `RenderAnimation` under `camera_mode: auto` picks them up with no further change. - Q2. Resolved. `render_model3d` supports `background: true` through the existing generation ledger and `MAX_BACKGROUND_GENERATIONS` cap; agents use `await_generation` for completion. The five `nodetool.blender.*` nodes stay synchronous because their downstream outputs remain media values and the kernel already runs independent nodes concurrently. ## Risks - R1. Blender availability. Highest. Desktop users must install it, and the cloud server has none until a Blender-enabled worker image ships. Mitigation: named errors, the `BLENDER_PATH` setting, and the profile exclusion, so no user meets a node that fails silently. - R2. Renderer parity drift. The preview node and the Blender node share camera params but not a renderer, so the same numbers frame differently. Mitigation: T3 pins the framing math on both sides. - R3. Render time versus timeouts. Mitigation: EEVEE and low samples as defaults, the `timeout` prop, progress messages, and Q2 for the long tail. - R4. Code execution in Blender. The local runner is not a sandbox. Mitigation: no user Python in scope, `--disable-autoexec`, `--factory-startup`, the env allowlist, and the rule that untrusted execution waits for the worker tier. - R5. Job contract lock-in once worker images exist. Mitigation: the version field, a `bad_job` rejection path, and a runner interface that carries no paths, so Stage 4 adds an implementation instead of rewriting `runBlenderJob`. - R6. Disk use during a run. The output caps are checked after Blender exits, so a runaway animation can fill the scratch disk up to the timeout. Mitigation: the timeout, and a follow-up that watches the scratch directory's total size while the child runs, which `runHostBinary`'s watchdog can be extended to do.--- # Blender Stage 4 — handover Source: https://docs.nodetool.ai/blender-stage4-handover.html Blender Stage 4 — handover Stages 0-3 are merged (PR #5570). Stage 4, the worker tier, is built on both sides and staged on feat/blender-headless. What remains is one measurement against a real worker, one cap on the worker side, and one allowlist decision. The design is blender-headless-integration-design.md. Read D4, D6 and D7 before touching any of this; the invariants below are its invariants, not local conventions. Where the work sits Repository Branch State nodetool main (#5570) Stages 0-3: five nodetool.blender.* nodes, render_model3d, job contract, LocalBlenderRunner, blender_ops/ nodetool feat/blender-headless Stage 4a committed: blender.execute / blender.event frames, worker.status.blender.enabled, blender-executor.ts. Stage 4b staged, uncommitted: WorkerBlenderRunner, resolveBlenderRunner, render_model3d background mode nodetool-core feat/blender-worker, PR #1050 The worker side: blender_handler.py, 26 tests, one real render Stage 4 is not part of #5570 and needs its own PR. What 4b built WorkerBlenderRunner (packages/blender-nodes/src/runner.ts) sends every file under blender_ops/ as an extra blob keyed by relative path (run_job.py, framing.py, ops/render_image.py, …). The worker writes them into its scratch directory and runs run_job.py, so a worker image carries Blender and nothing from NodeTool, and cannot drift from the release that issued the job. executeBlender keeps these blobs outside job.inputs and rejects a blob whose key collides with a declared input. BlenderExecutorError codes map to BlenderJobError one for one, and an unknown code keeps the worker’s message instead of collapsing to a bare bad_result. resolveBlenderRunner (packages/blender-nodes/src/run-job.ts, D7): A local Blender binary that resolves wins. A desktop with Blender never pays for a worker. No binary and no NODETOOL_WORKER_URL returns LocalBlenderRunner anyway. The run fails later on the missing binary, with the binary error. No binary and a URL returns WorkerBlenderRunner after assertAvailable checked worker.status. A worker that does not report blender.enabled fails here with a message that names NODETOOL_WORKER_URL: a configuration error, not a bad_result. A configured worker is never silently replaced with local after a connection or capability failure. The URL is a deployment choice. render_model3d takes background: true through the same context.runGeneration / await_generation path as media.ts. The per-run cap is MAX_BACKGROUND_GENERATIONS in packages/agents/src/capabilities/background-generation.ts, shared with the provider media calls. This resolves the design’s Q2 and is recorded there. The five nodes stay synchronous: a node that returns a job handle changes what every downstream edge means, and the kernel already runs nodes concurrently. What is left Run the round-trip test once against a real worker. packages/blender-nodes/tests/worker-roundtrip.test.ts skips when NODETOOL_WORKER_URL is empty. NODETOOL_REQUIRE_WORKER=1 turns the skip into a failure. CI sets neither, so this is a manual gate. Until it passes, the client side rests on the fake in tests/worker-runner.test.ts. # terminal 1 — the worker, from a nodetool-core checkout on PR #1050 conda activate nodetool python -m nodetool.worker --port 8787 # terminal 2 — this repo export NODETOOL_WORKER_URL=ws://127.0.0.1:8787 export NODETOOL_REQUIRE_WORKER=1 npm run build:packages npx vitest run --root packages/blender-nodes tests/worker-roundtrip.test.ts A worker-side output cap in PR #1050, mirroring MAX_OUTPUT_BYTES and MAX_TOTAL_OUTPUT_BYTES. The client refuses from sizes, but a rogue op can exhaust worker memory before any size is reported. The cloud allowlist. nodetool.blender and render_model3d stay excluded until a real worker answers a real job in an environment we provision. Flipping it is a one-line follow-up with that evidence behind it. Still unmeasured The fake in tests/worker-runner.test.ts pins the blob layout and the size-before-transfer refusal. Three client assumptions still have no measurement, and item 1 above is what settles them: the worker enforces the request’s timeout; its progress totals match what onProgress expects; an older worker answers an unknown frame type with Unknown message type rather than hanging. No old worker was available to test against. Known gaps in the worker (PR #1050) No output cap (item 2 above). Terminal error frames flatten to bad_result on the worker side, losing the detail the client now preserves. Scratch is system temp, not the workspace seam the local tier uses. Intended, since the worker has no workspace, but the two tiers clean up different places. Invariants that must survive any change here A node must not be able to tell which tier ran it. Both tiers therefore: read back only outputs the job declared. A name the op invents is ignored and logged; a path inside result.json is never opened. The FIFO sentinel in packages/blender-nodes/tests/runner.test.ts fails if that ever changes. check sizes before reading bytes, so an oversize output never enters memory. delete the scratch directory on success, op failure, timeout, cancel and crash. run Blender with --factory-startup --disable-autoexec and an allowlisted environment with the BLENDER_USER_* redirects, so neither the user’s add-ons nor process.env secrets reach the child. Pitfalls Blender 5.2 is the floor, not 4.2. scene.compositing_node_group and image_settings.media_type do not exist in 4.2. A backport would need the compositor rebuilt on the legacy Scene.node_tree API. node is tex never matches. Every Blender collection access returns a fresh Python proxy. An identity comparison silently deselected every bake target, bpy.ops.object.bake returned CANCELLED unchecked, and every baked map was black. Check operator return values the way _export_glb does. The object index pass is Cycles-only. EEVEE’s Render Layers node does not expose it, which is why the mask has two paths. The compositor stages 1e10, not +inf, on both engines. The op rewrites it before the EXR output is written. There are two DSL generators. codegen:dsl:check covers packages/dsl; the sandbox guest pack has its own gate, build:sandbox-dsl:check. A new node package needs both. git stash is not a red-proof baseline. It leaves committed work in place, so you measure your own branch. Use git checkout <base> -- <paths> or a fresh worktree. Verify per-package and you will miss suites. Two failures reached CI because only the obvious packages were run locally. npm run test:packages is what the shards run. Blender must be on PATH, at BLENDER_PATH, or in a well-known location, and must be 5.2 or newer. NODETOOL_REQUIRE_BLENDER=1 turns a missing binary into a failure instead of a skip. CI installs no Blender, so these suites skip there.--- # Bug Pattern Scan Source: https://docs.nodetool.ai/bug-pattern-scan.html Bug Pattern Scan This guide lists the bug families found in fix commits on main over four weeks, with commands that find siblings of each bug in the current tree. Use it to audit an area, or to review a diff for a known failure shape. The source set was every non-merge commit on origin/main whose message matched fix or bug (461 commits). About 240 changed behavior. The rest were checkpoints, docs, refactors, or CI tweaks. Regenerate the list with: git log origin/main --since="4 weeks ago" --no-merges -i -E --grep='fix|bug' --format='%h %s' How to Use This Guide Pick a family and run its search commands from the repository root. Treat every hit as a hypothesis. Read the code and apply the manual check. Confirm a suspect with a failing test before fixing it, as the Harness-First rules require. Scope any conclusion to the hits you read. Families are ordered by how often they recurred. P1. Races and Lost Updates Across await A writer reads state, awaits, then overwrites without re-checking. A user edit, an agent write, a running generation, or a recovery worker clobbers the other. Merge flows replace an accumulated list with the latest result. Evidence: 0ef7ee26, a7797dd4, aea51e12, 2b9d993c, 992d54ad, fb6a70bf, 3be3dafd, 799354ed, 5d7714f4, b0123af1, 72c0696b. // before: a concurrent render marked the doc dirty during the fetch const doc = await trpcClient.timeline.get.query({ id }); adopt(doc); // after: re-check after the await and route through the merge path const doc = await trpcClient.timeline.get.query({ id }); if (isDirty()) { mergeRemote(doc); } else { adopt(doc); } rg -n "await trpcClient\..*\.query" web/src/hooks web/src/stores -A4 rg -n "setConflicts\(|lastSyncedRef\.current\s*=" web/src rg -n "db\.select" packages/models/src -A6 | rg -n "insert|update" rg -n "reserve\(|commit\(|release\(" packages/runtime/src Manual check: after each await, does the code write state it captured before the await without comparing a version, dirty flag, or timestamp? Is a check-then-write pair (project exists, then insert) inside one transaction? Does a shared budget or limiter hand each caller its own reservation handle? P2. Pricing and Cost Recording Errors A per-second or per-megapixel rate is returned as the run total. A flat scalar is used where a resolution or duration grid exists. Some node types or the streaming path never record cost. Evidence: 7 to 8 commits in the pricing cluster, plus 76355344, e86d6089, d88ea441. rg -n "resolvePrice|priceScalarUnit|priceGenspendEntry" packages/model-pricing packages/node-sdk rg -n "recordPrediction|recordCost|nodetool_predictions" packages Manual check: does every per-unit rate multiply by its quantity? Are resolution and duration passed to a grid lookup? List every node and tool that calls a paid provider and confirm each one records usage on every path, including streaming and errors. P3. Scope Leaks and Wrong Default Scope State, queries, or tabs from one project appear in another. An omitted project_id falls into a default bucket. An undefined argument means “inherit the current workflow” when the caller meant “none”. A key unique only within one scope is reused across scopes. Evidence: a7ae40b8, 20c652a7, 21104cdc, cf473afb, f13e4261, f5ae0da8, 5eef8ad7, d62fad2c, plus the three cross-scope key commits in batch 1. rg -n "project_id\s*(\?\?|\|\|)\s*[\"']default" packages web/src rg -n "openTab\(" web/src | rg -v projectId rg -n "createNewThread\([^,]*,\s*undefined" web/src rg -n "projectId ===" web/src/stores web/src/components rg -n "new Map<string," packages/agents/src/capabilities Manual check: for each callee, does undefined mean “use the current X” or “no X”? For each map keyed by an entity ID, is that ID unique across the outer loop’s scope? P4. Lookup by a Non-Unique Key or Unstable Identity .find(x => x.id === value) ignores a second dimension such as provider. A preview matches “any video output” instead of its own node and output handle. Evidence: 25cbf492, 03781867. rg -n "\.find\(\(\w+\)\s*=>\s*\w+\.id\s*===" web/src packages rg -n "outputHandle|output_handle" web/src/components/node -A3 Manual check: can the collection hold two entries with the same ID? Does the selector filter by node ID and output handle together? P5. Duplicated Logic That Drifts The same value, parser, type, or renderer exists twice. One copy is edited and the other is not. Examples: a staleness hash over clipPrompt while the render sends directClipPrompt, GPU and Canvas2D renderers, a hand-written request interface beside its Zod schema, a second frontmatter parser, a hand-kept build order list that covered 21 of 49 dependencies. Evidence: c5436b22, 2d81b96a, cc1ad8fc, a7790498, e942bc39, 3d48196c, c05efcf1, e84c4de2, plus the frontmatter parser fix in batch 4. rg -n "^export interface \w+(Request|Response)" packages/protocol/src rg -n "sha256Hex|prompt_hash|isVersionStale" packages rg -l "canvas2d|gpu" packages/timeline/src/render rg -n "workspaces = \[" scripts rg -n "frontmatter" packages --type ts -l Manual check: for each interface, is there a schema with the same name? If so, derive the type with z.infer. Is the value compared in a staleness check produced by the same function that the work site calls? P6. Guards Missing on Some Sibling Operations A containment, ownership, or permission check covers most methods in a family but not all. deleteAll and mkdir skipped assertContained, which allowed a symlink escape. Approvers defaulted to allow. Capability mounts bypassed the shared gate. Evidence: 3be1f1c4, 224981b8, f6fc6466, d1be6585, 9721099e. rg -n "async (write|delete|move|copy|mkdir|deleteAll|rename)\(" packages/runtime/src rg -n "approver:\s*\(\)\s*=>\s*[\"']allow" packages rg -n "ungatedCapabilityRun" packages rg -n "requireUserId\(\)" packages/websocket/src Manual check: count guard calls per method in each storage or workspace class. A method without the guard its siblings use is a suspect. Every capability mount must go through the one shared gate. P7. Regex ReDoS and Single-Pass Sanitizing Anchored repeats like /^\/+|\/+$/g, or a repeated group around overlapping alternatives, backtrack polynomially on long input. CodeQL flagged most of them. A single .replace() strips nested tags incompletely. Evidence: 04f8d69b, 64f80c39, 0c690863, 67cd8808, 0e61f4d4, 1a538402, 3564ef7c, d8ad0317. rg -n "\.replace\(/\^?\\\\/\+" packages web/src rg -n "\+\$/" packages web/src --type ts rg -n "\(\?:[^)]*\|[^)]*\)[*+]" packages web/src --type ts rg -n "\.replace\(/<\[\^>\]\*>/" packages web/src Manual check: time each candidate on 50,000 repeated characters. Replace trailing-trim regexes with a loop, and strip markup with a parser or a loop that runs until the output stops changing. P8. Schemas Too Narrow, or Fields Accepted and Ignored MsgPack sends null where the schema allows only an optional string. A closed enum rejects a valid spelling. A stripping schema drops unknown fields during a save and reports success. A validated field is never read downstream. Evidence: d124efcf, e942bc39, 5f85fef0, 8c8a1fbc, da92dcbc, 1d4b765e, aff17ed1, d88ea441, c05efcf1. rg -n "z\.string\(\)\.optional\(\)" packages/protocol/src rg -n "\.parse\(|\.safeParse\(" packages/websocket/src packages/execution/src Manual check: can the field arrive as null over MsgPack? Does a save handler persist parsed output without reporting stripped keys? For each schema field, search its name in the consuming process or render code. Zero references outside the schema means a silent no-op. P9. Media and Asset Handling A node resolves media through a helper that handles inline bytes and the temporary store but not asset:// URIs. A chat attachment is inlined as a base64 data URI instead of uploaded. Video goes to a provider without a capability check, and the error names the wrong provider. iOS Safari never fires canplay before play(). Evidence: 465e46b6, 6c234230, 74f0cdb2, the video-input capability fix and the two iOS fixes in batch 4. rg -n "decodeImage\(|decodeAudio\(" packages --type ts rg -n "readAsDataURL|toDataURL\(" web/src rg -n "not supported by \"?openai" packages/runtime/src/providers rg -n "onCanPlay|canplay" web/src marketing Manual check: does each media input reach loadMediaRefBytes or the asset store? Is every attachment uploaded and referenced by ID? Does each gated path call the matching providerSupports* check? P10. Provider Contract Mismatches A generic spec field is never mapped to a vendor API. Numeric fields pass to an API that declares int within a range, and the API returns 422. An enum tier is written into a free-string field. context.signal is not forwarded, so a cancelled run keeps polling a paid API. Evidence: b244198d, 1a7eec1f, a59bfd06, 296476b6, the AtlasCloud size fix in batch 4. rg -n "async process\(context" -A12 packages/*-nodes/src | rg -n "fetch\(|Execute\(|poll" rg -n 'propType === "(int|float)"' packages/runtime/src/providers rg -n "spec\.\w+" packages/compute/src/providers rg -n "this\.serialize\(\)" packages/*-nodes/src Manual check: does every paid call receive context.signal? Is every WorkerSpec field read by each provider? Are numeric fields clamped and rounded like enums are coerced? Code that reads dynamic properties must read the instance, since serialize() returns only declared properties. P11. Identifier Collisions IDs built from Date.now() collide within one millisecond. Migration version strings were reused across branches. A duplicate-ID pass disqualified the rightful owner of an ID. Evidence: 9c4b24a3, plus the migration and glTF ID fixes in batch 4. rg -n 'Date\.now\(\)\.toString\(36\)|\$\{Date\.now\(\)\}' packages web/src rg -n 'version:\s*"[0-9]{8}_' packages/models/src/migrations rg -n "used\.has\(" packages Manual check: IDs need a counter or random component. Resource IDs follow the Resource ID Principles. P12. Compact Resource IDs Not Resolved at a Boundary An agent-facing tool accepts only the full 32-character ID, although agents receive 12-character prefixes. Evidence: 670a215f, cf473afb. rg -n "shortResourceId|resolveResourceId" packages/agents/src packages/protocol/src rg -ln "_id" packages/agents/src/capabilities packages/agents/src/tools Manual check: every tool input that takes an agent-visible ID must resolve an exact 12-character prefix uniquely and return an ambiguity error otherwise. P13. Failure Handling Errors One failing sub-read fails a whole aggregate through Promise.all. A cache stores a rejected or null result, so one transient error disables a resource for the rest of the run. An inner timeout equals its outer timeout, so the outer one fires with a generic message. Evidence: 5917e4fa, dbcef8c2, 91719bdc, f59c1cc1. rg -n "Promise\.all\(" packages/models/src web/src/stores rg -n "\.catch\(\(\) => null\)" packages --type ts rg -n "_TIMEOUT_MS|TIMEOUT_MS\b" packages/agents/src packages/cli/src Manual check: should independent parts degrade separately with Promise.allSettled? Does a memo cache evict failures? Is each nested timeout strictly smaller than the one enclosing it? P14. Falsy Defaults and Wholesale Replacement x ?? "image.png" keeps an empty string, and path.resolve(folder, "") returns the folder. A setX(partial) API replaced the whole default object. Evidence: 443a9187, b56d5516. rg -n '\?\?\s*"' packages --type ts rg -n "path\.resolve\(\w+,\s*(this\.)?\w+\)" packages rg -n "set[A-Z]\w*Interfaces?\(" packages/runtime/src Manual check: can the string be "" from a text field? Does the setter merge keys over the defaults? P15. UI Stacking, Focus, and Keyboard Scope A dialog opened from inside a popover renders beneath it. Opening a picker leaves a sibling menu open. Background tabs kept mounted as inert still consume key combos. A :hover reveal on a list wrapper shows every row’s button. A flex child without min-width: 0 overflows on phones. Evidence: 8af16e41, 5b09d431, e4d347d1, 98891619, 103cb769, e7d6a80f, fac29e8d, 576448dd, 814e9e3c, plus the hover fix in batch 4. rg -ln "Popover" web/src/components | xargs rg -ln "Dialog|Modal" rg -n "useCombo\(" web/src/components rg -n "\.[\w-]+:hover &" web/src/components rg -n "align-items:\s*flex-start|alignItems:\s*\"flex-start\"" web/src Manual check: use Z_INDEX tokens for nested surfaces. The combo dispatcher must check reachability when a key is pressed. Scope hover reveals to the row. P16. Unregistered Tools and Unthreaded Parameters A tool was implemented but missing from BUILTIN_TOOL_NAMES. A new callback parameter reached one loop call site but not the others. Evidence: 11d76701, e297798a. rg -n "export const \w+Tool\b" packages/agents/src/tools rg -n "BUILTIN_TOOL_NAMES" packages/agents/src/tools/builtin-tools.ts Manual check: every exported tool appears in a registry. When you add a parameter, list every call site of the function. P17. Shell Scripts and Deploy Ordering Under set -euo pipefail, a grep with no match kills the script before its empty-result branch runs. A deploy step proceeds when a request is accepted rather than when the machine is healthy. Evidence: 2f22cabe, 8177ad65, c652fe6d, dd0dd7d2, d8c7be77. rg -l "pipefail" scripts | xargs rg -n "\| *grep" rg -n "fly machine (run|update)" scripts Manual check: add || true where “no match” is valid. Wait on the health state the next step depends on. P18. Geometry and Aggregation Math An overlap test uses a value from before a clamp. An append boundary folds over kept and added but not foreign clips on the same track. A quaternion to Euler conversion had a sign error at gimbal lock. Evidence: 66f1a969, d19e962e, 60336f71. rg -n "Math\.(max|min)\(" packages/timeline/src/placement rg -n "\.reduce\(.*Math\.max" packages/timeline/src rg -n "atan2" packages/model3d/src packages/timeline/src Manual check: after a clamp, no later code should use the unclamped input. Boundary math must include every group that stays in the result. Test degenerate math on a full grid of inputs.--- # Canonical Model Intelligence — Design Source: https://docs.nodetool.ai/canonical-model-intelligence-design.html Canonical Model Intelligence — Design Status: passes 1–3 and 4a are implemented (accessors + artifact in packages/model-pricing, the rankings sync + nightly workflow, the find_model rank term, the enriched /api/models/recommended/* endpoints), and the first real artifact has landed — 168 routes, ranked across five tasks. Rankings now lead the agent’s default model choice (§4 below). The picker work in pass 4 is still open. The problem NodeTool lists fal-ai/kling-video/v3/pro (FAL) and kling/v3-pro (kie) as two unrelated models. They are two routes to one model. The picker shows the same model many times, nothing orders a list of 80 video models by quality, and find_model ranks candidates only by the static RECOMMENDED_MODELS list, provider hints, and locality. An earlier draft of this design proposed a new packages/model-catalog package with a hand-written CanonicalModel registry, a hand-written MODEL_ALIASES table, a ModelMetadata.ranking field on the wire model types, and per-route latency/reliability scores. This document keeps that draft’s one load-bearing idea — canonical model ≠ provider route — and discards most of its machinery, because the repo already ships the hard half. What already exists The GenSpend pricing sync (scripts/sync-genspend-pricing.mjs, scripts/genspend/) already solves canonicalization for the media models, and its output already ships in every release. The rankings sync no longer reads it (§1); this section describes what the pricing side provides: packages/model-pricing/src/generated/genspend-pricing.json is keyed <provider_id>:<model_id> and every entry carries model_slug — a canonical model id. Today that is 276 priced provider-native ids resolving to 104 canonical slugs, 51 of which span more than one provider (seedance-2 → atlascloud + kie, seedream-5-pro → atlascloud + kie, …). scripts/genspend/normalize.mjs is the naming bridge. Its comparison key (scripts/rankings/model-keys.mjs, re-exported there) gives one key per model (FLUX.2 [pro] ≡ flux-2-pro), vendor-prefix stripping, task suffix stripping, and the guards that keep it honest (a family name that matches too many ids is dropped as ambiguous, a capability flag that refutes a task blocks the match). scripts/genspend/match.mjs resolves an upstream model to NodeTool ids through a five-tier trust ladder (alias > variant > provider-id > receipt > catalog), and scripts/genspend/aliases.json is the hand-pin escape hatch: pin a match the comparison cannot see, or block a wrong one, without weakening the rules for everything else. The nightly GenSpend Pricing Sync workflow regenerates the artifact and opens a PR — numbers that gate spend get reviewed, not auto-merged. getGenspendPrice(provider, modelId) is how the web UI and the runner read route facts (price) today: a synchronous lookup on a shipped snapshot, no network, no key. So “introduce canonical model ≠ provider route” is not a new abstraction to build. The design below is: make the canonical id first-class (the leaderboard’s own slug, grouping provider routes), and ship quality rankings as a second generated artifact in the same shape, through the same pipeline pattern, read through the same kind of lookup. Design 1. The canonical id is the leaderboard’s own slug No new CanonicalModel registry, no new MODEL_ALIASES table in runtime code. Each Artificial Analysis row already carries a stable slug, and that slug is the canonical id: the id that groups every provider route to one model. The routes themselves come from NodeTool’s own providers (the model lists getAvailable*Models() return), so a model needs a provider route, and nothing else, to be ranked. The rankings sync reads no price catalog and shares no data with the GenSpend sync. Only the naming helper (scripts/rankings/model-keys.mjs) is shared. Three consequences, all acceptable: A model no provider lists has no route to rank. It is reported in the sync run and stays out of the artifact. A model a provider lists but the leaderboard does not carry stays what it is today: listed, runnable, unranked, ungrouped. “Unmapped” is the designed fallback — an incorrect merge is worse than an unmapped model. The vocabulary is a third party’s. A renamed slug shows up as a diff in the nightly sync PR, where a maintainer sees it. Shipped artifacts freeze the slugs they were generated with, so a rename can never break an installed app. A route that declares the tasks it serves is ranked only for those tasks, so an image editor never inherits a text-to-image rank from a shared name. Scope: media models (image, video, TTS, music) — the modalities where routes multiply and where FAL/kie/atlascloud/replicate overlap. Language models are out of scope for v1: their ids are already near-canonical per provider, and find_model already handles them adequately. 2. Rankings are a second generated artifact in packages/model-pricing scripts/sync-model-rankings.mjs # new, sibling of sync-genspend-pricing.mjs scripts/rankings/aliases.json # hand-pins: AA/Arena name → canonical slug, or null to block ↓ (Artificial Analysis data API; Arena leaderboard dataset later) packages/model-pricing/src/generated/model-rankings.json packages/model-pricing/src/model-rankings.ts # typed accessor module Not a new packages/model-catalog package. model-pricing is already the package whose job is “externally sourced model facts, shipped as a generated snapshot, read by web and runner” — rankings are the second column of the same table. If the package outgrows its name, renaming it to @nodetool-ai/model-catalog is a mechanical follow-up PR; creating a 57th package for two JSON files is not the place to start. The artifact mirrors the pricing catalog’s shape — keyed by <provider_id>:<model_id>, expanded at sync time, so runtime does zero matching: { "schemaVersion": 1, "source": "artificialanalysis.ai", "generatedAt": "2026-08-19T...", "models": { "fal_ai:fal-ai/kling-video/v3/pro": { "canonical": "kling-3-pro", "name": "Kling 3 Pro", "creator": "Kuaishou", "tasks": { "text_to_video": { "score": 1123, "normalized": 0.94, "rank": 2, "of": 41 }, "image_to_video": { "score": 1101, "normalized": 0.91, "rank": 3, "of": 38 } } }, "kie:kling/v3-pro": { "canonical": "kling-3-pro", ... } } } canonical groups routes. Each route carries the tasks it serves, with identical rankings for tasks shared by routes to the same model. tasks keys are NodeTool’s own supportedTasks vocabulary (text_to_image, image_to_video, …), which is what the AA media API’s per-task leaderboards map onto. No single overall score per model — a model mediocre at text-to-image can lead at editing. score is the source-native number (arena-style rating), normalized its 0–1 position within that task’s leaderboard, rank/of for display. No invented confidence, no stored badges — “best quality” / “best value” are derived at render time from rank + price, not persisted. How the sync matches AA/Arena rows to routes: it builds the route universe from the built runtime providers (scripts/rankings/routes.mjs) and indexes each route’s id and display name with modelKeys() from scripts/rankings/model-keys.mjs. It looks each AA row up by the same exact-key comparison — no fuzzy matching, no prefix matching. Anything unmatched is reported, never guessed. A route two rows with different slugs both reach is dropped for both and reported as ambiguous. scripts/rankings/aliases.json pins a row to routes (<provider>:<model_id> keys) or blocks it by hand. A pin chooses the model, not the task: the task filter still applies to a pinned route. Operationally it copies the pricing sync verbatim: nightly workflow, opens a PR only when something moved, sync:model-rankings:check fails CI when the artifact and a fresh measurement disagree, no API key in the shipped app, no network call at startup. NodeTool is local-first; a ranking that evaporates when a leaderboard site is down is worse than a week-stale snapshot. 3. Reading it: a lookup, not a field on the wire types The earlier draft added metadata?: { ranking?: … } to ImageModel / VideoModel and enriched inside loadImageModels(). Rejected: Those are runtime routing objects that cross the wire in every model-list response, from ~30 providers. Enriching in the loaders touches every provider path to serve a concern only three consumers have, and couples packages/runtime to ranking data it never uses to execute anything. The repo already has the right pattern in production: prices never sat on the model objects either. web/src/utils/modelUnitPricing.ts and the runner both call getGenspendPrice(provider, id) where they need it. So the accessor module is the whole runtime API: // packages/model-pricing/src/model-rankings.ts getModelRank(provider, modelId): ModelRank | null // one route's entry getCanonicalId(provider, modelId): string | null // grouping key routesFor(canonicalId): RouteEntry[] // all provider routes to one model rankedForTask(task): RankedCanonicalModel[] // leaderboard, canonical models ImageModel, VideoModel, and friends do not change. Nothing in packages/runtime’s provider layer changes. 4. Consumers find_model / nodetool.models.pick (packages/agents/src/capabilities/models.ts). The score is a ladder of tiers, each above the sum of everything below it (SCORE_TIERS): model hint 2000, provider hint 1000, prefer_local 500, a leaderboard position 200 + normalized × 80, RECOMMENDED_MODELS 100, locally served 30. Two consequences worth stating plainly: Rankings lead the default pick. With no hint from the caller, the first hit is the top of the leaderboard for the task. The first version of this term sat below the recommended bonus (≤ 80 against +100), which meant the hand-pinned list still decided every default: find_model ("text_to_video") answered with openai:sora-2, a model the artifact does not rank at all, over the rank-1 route. An agent asking only for a capability now gets the best model for the job. A ranked candidate does not also take the recommended bonus. +100 on top of a 0…80 span would let the static list reorder the leaderboard, which is the blindness the term exists to fix. RECOMMENDED_MODELS keeps ordering what the artifact says nothing about — language, embedding and ASR models, local models, anything unmatched. Results are also one row per canonical model rather than one per route: gpt-image-2 is reachable through atlascloud (two endpoints), kie (two) and openai, and a top-5 of five routes to one model shows an agent one model. The best-scoring route survives, the rest are listed under its alternate_routes. A preference the caller actually stated — a provider, a model, prefer_local — still outranks any leaderboard position. Each candidate carries rank/of/canonical in the answer. Two routes to one canonical model collapse to the better-priced one in the top results, with the alternates listed under it. The agent answer the draft asked for (“Kling 3 Pro, FAL, rank 2 of 41, $0.18/s — also via kie at $0.14/s”) falls out of routesFor + getGenspendPrice, both already-shipped lookups. This is worth more than the picker UI: model choice by agents is currently blind past the recommended set. /api/models/recommended* (packages/websocket/src/models-api.ts). The task-specific endpoints (/recommended/video/text-to-video, …) merge ranked canonical models into what they return today. RECOMMENDED_MODELS stays, demoted to what it really is: hand-pinned overrides that always surface first (and the only mechanism for modalities and providers the rankings cannot see — local Ollama models, ASR). It stops being the sole ordering. Model picker (web, then mobile). Group entries by canonical: one row per model, routes as sub-entries with per-route price from the pricing catalog. Sort ranked models by task rank above the unranked remainder (alphabetical, as today) under an “All models” divider — the full list stays complete; NodeTool’s selling point of offering everything is untouched. Badges (Best quality, Best value, Fastest where duration data exists) are computed in the picker from rank + price. Web reads the accessor module directly, as it reads prices today; mobile reads the enriched /api/models/recommended* responses. 5. What is deliberately not built No ModelRoute runtime object and no route scores. The draft’s per-route latency (42s vs 58s) and reliability (99.6% vs 98.9%) numbers have no data source — NodeTool has no fleet telemetry and should not invent numbers. Route facts NodeTool can actually stand behind are price (shipped) and, later, the user’s own prediction ledger (nodetool costs) for observed local latency. When such a source exists, it becomes another lookup keyed provider:model_id; the shape is ready for it. No weighted recommendation = 0.65·quality + 0.20·value + … formula. A stored composite hides its inputs and invites tuning debates. Sort by task quality; show price beside it; let explicit user hints dominate in find_model. The one place that blends (the find_model score term) is a single bounded addend in existing code, not a scoring service. No runtime fuzzy matching, ever. All matching happens at sync time, is exact-key, and lands in a reviewed PR. Unmatched models are reported in the sync output and fixed in aliases.json by hand. No automatic cross-provider failover. Routing a run to a different provider than the one saved on the node is a spend decision the user did not make. routesFor makes an offered alternative possible (in the picker, in an agent’s answer); nothing switches silently. Failure posture Everything fails toward today’s behavior. A missing or empty model-rankings.json means: no grouping, no rank term in find_model, the picker sorts as it does now. A model absent from the artifact is unranked, never hidden and never down-ranked below where it sits today (the rank term only adds). The sync fails closed: an AA response that does not parse, or a leaderboard whose task cannot be mapped, drops that task from the artifact and says so in the PR body, rather than shipping a number nobody can trace. Per the repo’s check discipline: the sync’s parity check must be proven able to fail (perturb one score, watch it go red) before it gates anything, and the unmatched-models report must assert it found the fixtures it plants. Rollout Accessor + artifact plumbing — model-rankings.ts, an empty-but-valid generated file, rankedForTask/routesFor/getCanonicalId with unit tests against a fixture artifact. No behavior change anywhere. Sync — sync-model-rankings.mjs against the Artificial Analysis data API (its media leaderboards map cleanly onto supportedTasks), reusing scripts/rankings/model-keys.mjs; nightly workflow + :check; first real artifact lands by PR. Arena’s dataset joins later as a second signal — averaging sources is a sync-time concern and changes nothing downstream. Agent surface — the find_model rank term and canonical/route fields in its answer; extend the find_model cases in the eval suites to pin that a ranked model outranks an unranked one and that hints still win. API + pickers — enrich /api/models/recommended*; group and sort the web picker; mobile follows on the API alone. Each pass ships alone and is useful alone; pass 1+2 already give agents and the API the data even before any UI moves.--- # Chain Editor Source: https://docs.nodetool.ai/chain-editor.html The Chain Editor is a linear alternative to the node graph. Instead of placing nodes on an infinite canvas, you compose an ordered chain of cards — “transcribe → summarize → email”, “fetch → parse → index”. Prefer the node graph? The full Workflow Editor is always one click away. Both editors read and write the same workflow format. When to Use the Chain Editor Use the Chain Editor when… Use the Workflow Editor when… Pipeline is mostly linear Pipeline branches or has loops You’re guiding a non-technical user You need fine-grained connection control You want one output per step You need multi-output nodes You’re sketching a workflow quickly You’re building a production agent Opening the Chain Editor There are two entry points: In a workflow tab. Open the ⋮ (Workflow actions) menu on the composer bar and choose Chain View. The same menu item reads Graph View while the chain is showing. The choice is the editorViewMode setting, so it applies to every workflow tab and is remembered. The graph editor stays mounted underneath, so switching back keeps your canvas position. As a page. Go to /chain/:workflowId to edit a stored workflow, or /chain to start an empty chain called “Untitled Workflow”. Anatomy of a Chain A chain is a vertical list of numbered cards, each one a node. Cards. The number on the left is the step number. Click the card header to expand or collapse it. Input mapping. Every field on an expanded card can take its value from the output of an earlier step. Output selector. A card whose node has more than one output shows a Select output menu. The chosen output is the one the next step is wired to by default. A collapsed card still shows its result preview after a run. Input nodes always sit at the top of the chain and output nodes at the bottom. Adding Steps Click the + between cards, or Add First Node on an empty chain. The Add Node dialog opens with a search box and a grid of quick actions. Search uses the same engine as the graph editor’s node menu and is not filtered by the previous step’s output type. The node is inserted at the chosen position and the editor wires one of its inputs to the nearest earlier step that produces a matching type. It prefers that step’s selected output, and prefers an input whose type matches exactly over one typed any. Insert into the middle of a chain and the step below follows: an input it took from the step now above the new card moves to the new card’s output when the types allow. Editing Card Properties Expand a card to see its properties. The fields are the same property editors the graph editor uses: strings, model pickers, sliders, asset selectors, and so on. Each card also shows the node description and, in its footer, Move up, Move down, Duplicate, and Remove buttons. In a workflow tab, every edit is mirrored into the tab’s workflow immediately, so the graph view, running, and saving all see it. On the standalone /chain page, nothing is stored until you click Save workflow next to the name field. Reordering and Removing Cards Move up and Move down change a card’s position. When a move puts a source after the step that reads it, that input mapping is dropped. Remove deletes the card and any mapping that pointed at it. Duplicate inserts a collapsed copy directly below. There is no drag handle and no keyboard delete in the chain view. Input / Output Mapping Input mapping. Each field has a link button, Use output from a previous step, that opens a menu of every earlier step’s outputs, not just the previous card’s. Compatible outputs are listed first. Incompatible ones appear greyed out with the type they produce. A wired field shows step · node title and the output name instead of an editor. Disconnect gives the editor back. A warning icon marks a wired source whose type does not match the field. Output selector. If a card has multiple outputs, pick the one that flows into the next step. Type compatibility uses the graph editor’s connection rule, plus int widening to float. Running a Chain Chain cards have no run button of their own. In a workflow tab, use the composer bar’s Run entire workflow button or Ctrl/⌘ + Enter, which run the whole graph. Cards show live state: A running card pulses and shows a progress bar. A completed card gets a green border and check icon, and its output renders on the card. A failed card gets a red border and error icon. Switching Between Editors A chain is a different view of the workflow graph. Chain → Graph. Choose Graph View in the ⋮ menu. Edits made in the chain are already in the graph, and positions of nodes that existed before are kept. Graph → Chain. The chain lists every node that has registered metadata in depth-first order from the nodes with no incoming connection, with inputs first and outputs last. Branches become extra cards, and a field wired to an earlier step shows that step as its source. Nodes the chain cannot show, such as comments and groups, are left untouched in the graph. Saving from the standalone /chain page writes only the chain’s nodes and lays them out in a single column. Next Steps Workflow Editor — the full graph editor Cookbook — linear patterns that work great in the chain editor Mobile App — running Mini Apps built from your workflows on iOS and Android--- # Changelog Source: https://docs.nodetool.ai/changelog NodeTool publishes its changes in two places. Where What it holds CHANGELOG.md One section per month, grouped into Added, Changed, and Removed, then by area (UI, agents, nodes, runtime) GitHub releases Tagged releases with the desktop installers. A tag such as v0.7.0 also produces a Docker image tag of the same version The changelog follows Keep a Changelog and Semantic Versioning. The repository’s own note says the log is generated by Claude using a script, so check the commit history when you need an exact detail. Desktop releases come in two channels, Stable and Nightly. The channel is a setting in the desktop app. See Upgrading for how to pick one and how to update. Last three months This is a summary, not the full list. The monthly sections below name the main additions and changes. September 2026 Shipped as v0.7.0. Guided creation. Guided flows start workflow, script, storyboard, timeline, and game documents. Games. A game flow exports a Godot project. Projects and chat. A project picker sits above the workspace tabs, with entities assigned to projects. A project can hold several chat threads, and memory has its own page outside chat. Storyboards. Each storyboard card has a shot editor, and reference-to-video works in chat, providers, and storyboard render. Timeline. MIDI tracks with a piano roll, 3D clips, clip crop, film grain, audio-driven motion, subject mattes, speech and music generation in the composer, and zip import and export. Models. GPT Image 2.5, Flux 3 Video Edit, and new FAL, KIE, and AtlasCloud models. Video upscale, interpolate, and outpaint task types. Runtime. External MCP servers join the agent toolbelt. Generations recover after a reload or a browser disconnect. Changed. Left-panel navigation was consolidated, the setup and connect model pickers were redesigned, and the chat model is stored per conversation. Removed. The lib.apple node library. August 2026 Projects. Projects became the workspace unit, and a new-project surface starts from a brief or a skill. Apps. Mini apps deploy to a hidden URL without login. App Builder gained form, capture, compare, pick, and Sketch Pad widgets. Editors. SVG became its own document. Sketch gained a Deform transform, and agents can place images on sketch layers. The timeline and sketch editors show cost estimates before a render. Agents. Immutable system skills, plan mode with a real planner, a live browser capability, web search through OpenAI and Gemini, and audio and video measurement tools. Runtime. QuickJS sandbox compiler and packs, 60 more AtlasCloud image and video models, and timeline JavaScript for custom clip animations. Added. A Chrome extension and a Telegram package. Changed. The dashboard was removed and onboarding moved into the new-project surface. Storyboard was restyled as a shot grid. 24 nodes already covered by the sandbox and Code node were retired. July 2026 Script editor. Author, voice, timeline bridge, SRT and VTT export, and agent tools. Storyboards. A storyboard surface with a still and clip gallery, a render queue, and an assistant panel. Entities work in @ mentions and storyboard shots. Code node. AI authoring with typed handles. Mini apps. Sketch, Timeline, chat, AI, and the remaining Puck widget types. Agents and CLI. Thread-level memory, tool-loop evals for editor surfaces, Google Workspace tools, Claude subscription login for the Claude Agent provider, and local database reads, RAG collections, and cost tracking in the nodetool CLI. Runtime. An in-process node-llama-cpp provider with native function calling, 327 FAL models and 82 Replicate models, nightly price catalog sync, and Flatpak as a Linux release target. For earlier months, read CHANGELOG.md.--- # Chat API Source: https://docs.nodetool.ai/chat-api.html NodeTool provides both OpenAI-compatible HTTP endpoints and a WebSocket endpoint for chat interactions: The single server (nodetool serve, defaults to port 7777) exposes OpenAI-compatible HTTP endpoints (/v1/chat/completions, /v1/models) for remote clients. The same server exposes a WebSocket endpoint at /ws that carries both chat and workflow messages. See the canonical matrix in API Reference for methods, auth requirements, and streaming behavior. Port Reference: Both development and production deployments use port 7777 by default. Replace localhost with your server hostname for remote connections. OpenAI-Compatible HTTP API NodeTool exposes OpenAI-compatible endpoints that allow you to use standard OpenAI client libraries and tools. When AUTH_PROVIDER is static or supabase, send Authorization: Bearer <token>; in local/none modes the token is optional for development. Chat Completions: POST /v1/chat/completions URL: http://localhost:7777/v1/chat/completions Headers: Content-Type: application/json Authorization: Bearer YOUR_TOKEN Request Body: { "model": "gpt-5.6", "messages": [ {"role": "user", "content": "Hello, how are you?"} ], "stream": true } Request fields the server reads: model, messages, stream, tools, temperature, top_p, max_tokens, presence_penalty, and frequency_penalty. Other OpenAI fields are ignored. Messages can have the roles system, user, assistant, and tool, and assistant messages can carry tool_calls. stream defaults to false. Provider selection. The server picks the provider from the model name: gpt-*, o1*, and o3* go to OpenAI, claude-* goes to Anthropic, and everything else goes to Ollama at OLLAMA_API_URL. The OpenAI and Anthropic keys come from the caller’s stored secrets (OPENAI_API_KEY, ANTHROPIC_API_KEY). A request without model uses llama3.2:latest. Responses. Without streaming, the reply is a chat.completion object whose finish_reason is stop or tool_calls. With "stream": true the reply is text/event-stream of chat.completion.chunk objects, ending with a chunk that carries finish_reason and then data: [DONE]. A failure while streaming is sent as an error event object followed by [DONE]. usage is always zeros, because the endpoint does not count tokens. Errors use the OpenAI shape, { "error": { "message", "type", "param", "code" } }. An unparsable body is a 400, and a provider that cannot start or a failed non-streaming call is a 500 with type server_error. Example using OpenAI Python client: import openai # For local development: base_url="http://localhost:7777/v1" # For production/server: base_url="http://localhost:7777/v1" or your server URL client = openai.OpenAI( api_key="YOUR_TOKEN", base_url="http://localhost:7777/v1" ) response = client.chat.completions.create( model="gpt-5.6", messages=[ {"role": "user", "content": "Hello, how are you?"} ], stream=True ) for chunk in response: if chunk.choices[0].delta.content is not None: print(chunk.choices[0].delta.content, end="") Models: GET /v1/models URL: http://localhost:7777/v1/models Returns a fixed list in the OpenAI list shape. It does not query a provider and does not reflect which keys are configured. The entries are llama3.2:latest, gpt-4, gpt-4o, gpt-4o-mini, claude-sonnet-4-20250514, and claude-opus-4-20250514. A chat request accepts any model name the three providers know, not only these. For the live catalog, use the SDK models route. # For local development and production: http://localhost:7777/v1/models curl http://localhost:7777/v1/models \ -H "Authorization: Bearer YOUR_TOKEN" WebSocket API NodeTool also exposes a /ws WebSocket endpoint for real time conversations. The server side is implemented by WebSocketClientSession which handles message parsing, tool execution and streaming responses. The connection supports both binary (MessagePack) and text (JSON) messages. Authentication can be provided via Authorization: Bearer <token> headers or an api_key query parameter. Note: The WebSocket endpoint is served on port 7777 (development) or your configured server port. WebSocket Example usage Every message is a { command, data } envelope. A chat turn is the chat_message command, and thread_id, provider, and model are required. A bare chat payload gets an invalid_message error. // For local development: ws://localhost:7777/ws // For production deployments, use your server URL const socket = new WebSocket("ws://localhost:7777/ws?api_key=YOUR_KEY"); socket.binaryType = "arraybuffer"; socket.onmessage = (event) => { const data = msgpack.decode(new Uint8Array(event.data)); if (data.type === "chunk" && data.content_type === "text") { console.log(data.content); } }; socket.onopen = () => { socket.send( msgpack.encode({ command: "chat_message", data: { thread_id: "my-thread", role: "user", content: "Hello world", provider: "openai", model: "gpt-5-mini" } }) ); }; The server saves the user message to the thread, then streams the turn. Replies are binary MessagePack by default. Send set_mode with "mode": "text" first to receive JSON. The WebSocket API lists the other chat_message fields. Server responses A turn streams these frames, each tagged with the thread_id: chunk – streamed text from the model (content_type: "text"), with done: true on the last one and thinking: true for reasoning text message – a persisted assistant or tool message tool_call_update and tool_result_update – a server-side tool call and its result tool_call – a tool the client owns. Reply with a tool_result frame tool_approval_request – a gated tool call. Reply with tool_approval_response and decision of allow, allow_for_chat, or deny secret_request – the model needs a credential. Reply with secret_request_response task_update, planning_update, todo_update – agent progress error – the turn failed Set workflow_target: "workflow" on chat_message to run a workflow as the chat agent. That path also streams job_update frames. A bare workflow_id is only context and does not start a run. Reconnecting to a turn A turn keeps running if the socket drops. After reconnecting, send list_chat_turns to find running turns, then resume_chat with the thread id and the highest chat_seq you received. The server replays the frames you missed. Send stop with the thread_id to cancel a turn.--- # Chat CLI Source: https://docs.nodetool.ai/chat-cli.html The nodetool chat command starts an interactive terminal interface for conversing with language models and running tools. Every chat session runs the unified agent loop — the assistant can call tools and decompose work on its own; there is no separate mode to toggle. Starting the Interface Run nodetool chat from your shell. The header shows the current provider and model, and the line under it shows the workspace, the connection, and the permission mode. Type /help at any time to list the commands and key bindings. nodetool chat forwards its flags to the nodetool-chat binary, so both names start the same interface. # Start with saved / auto-detected settings nodetool chat # Override provider and model nodetool chat --provider anthropic --model claude-sonnet-5 # Set the workspace directory (defaults to the current directory) nodetool chat --workspace /path/to/project # Connect to a running server instead of a local provider nodetool chat --url ws://localhost:7777/ws Other flags: --resume [id] — resume a saved conversation in this workspace. Without an ID, resume the most recent one. --permission-mode <default|auto|plan> — how tool calls are gated. Interactive chat defaults to default, piped input to auto. --cost-cap <usd> — ceiling on provider spend for one turn. 0 lifts it. Default: NODETOOL_AGENT_TURN_COST_CAP_USD. --timeout <s> — wall-clock bound on one turn, in seconds. Default: NODETOOL_AGENT_TURN_DEADLINE_MS. --no-read-only-search — remove the read-only run_search fan-out tool, which is on by default. --trace-file <path>, --trace-stdout [format], --no-trace-stdout — tracing. Fullscreen chat owns stdout, so use --trace-file. The full flag reference is in the CLI reference. The --agent flag is deprecated and has no effect — the unified chat agent always runs. Slash Commands Commands use a / prefix. Tab completes commands and arguments (provider names, model ids). /help — List the commands and key bindings. /new — Save the current session and start a fresh conversation. /clear — Clear the screen. The conversation context stays. /compact [instructions] — Summarize the conversation into a retained context message. Optional instructions focus the summary. /model <model-id> — Set the model. A bare ID that more than one provider offers is refused. Use provider/model-id then. The choice is saved. /mode <default|auto|plan> — Change the permission mode. /sessions — List saved conversations in this workspace. /resume [id] — Resume a saved session. Without an ID, resume the most recent. /export [path.md] — Save the transcript as Markdown. The path is relative to the workspace and the file must not exist. /agent [id|main] — List sub-agent threads, inspect one, or return to the main conversation. /tools — List the enabled tools. /details — Toggle tool arguments, code, and diffs. /exit, /quit — Save and exit. There is no /provider command. Pick a provider with /model provider/model-id or --provider. Enter sends. Alt+Enter, Shift+Enter in supported terminals, or Ctrl+J inserts a newline. Up and Down recall earlier prompts, Tab completes, and Page Up, Page Down, or the mouse wheel scroll the transcript. Ctrl+G jumps to the latest output and Ctrl+O toggles tool details. Esc or Ctrl+C stops a running turn. When idle, either one clears a draft, and Ctrl+C on an empty composer saves and quits. Piped input When stdin is not a terminal, chat reads one message per line and writes the answer to stdout, with tool names and status on stderr. The fullscreen commands do not apply. /new starts a fresh conversation. The commands below need --url, and without it chat prints Slash commands require --url (WebSocket mode). /run <workflow_id> [json_params] — run a workflow. /stop — stop the generation in progress. /reconnect <job_id>, /cancel <job_id>, /status [job_id] — reconnect to, cancel, or query a job. /help — list these commands. Sessions Sessions save under ~/.nodetool/chat-sessions/ after each turn and on exit. They hold the conversation context and the provider’s continuation state. /sessions and --resume list only sessions that match the current workspace and server. Tools The assistant runs with a set of enabled tools (file operations, web search, browser, NodeTool MCP tools, and more). Tools are auto-enabled based on the API keys available in your environment or the encrypted secret store. Use --tools to override the set explicitly, or /tools to see what is currently enabled. Local interactive chat also enables bash for host commands in the chat workspace. It is not available in piped stdin chat or server-connected chat (--url). Bash commands are not sandboxed: default permission mode asks before running one, plan mode blocks it, and auto mode permits it without a prompt. nodetool chat --tools read_file,write_file,grep,web_search Workspace File operations run against a workspace directory. The workspace defaults to the current working directory; override it with --workspace <path>. Settings and Logs Persisted settings (provider, model, enabledTools) are stored in ~/.nodetool/chat-settings.json. A log file is written to ~/.nodetool/chat.log so logging does not interfere with the terminal UI. Your provider and model choices are saved automatically as you change them, so they carry over between runs.--- # Serving Chat (OpenAI-Compatible API) Source: https://docs.nodetool.ai/chat-server.html There is no separate chat-server command. To serve chat over HTTP, run the NodeTool server with nodetool serve. It listens on 127.0.0.1:7777 by default and exposes an OpenAI-compatible chat API alongside the WebSocket endpoint. Quick Start # Start the server (default 127.0.0.1:7777) nodetool serve # Bind all interfaces on a custom port nodetool serve --host 0.0.0.0 --port 8080 The server provides: POST /v1/chat/completions — OpenAI-compatible chat completions (streaming and non-streaming). GET /v1/models — OpenAI-compatible model list. /ws — the WebSocket endpoint used by the editor and nodetool chat --url. /v1 passes the request straight to one provider. It does not run the NodeTool agent, so there are no tools beyond the tools you send. The agent loop is reached over /ws. Requests go through the server’s normal authentication. See Authentication. Chat Completions: POST /v1/chat/completions URL: http://localhost:7777/v1/chat/completions Request: curl http://localhost:7777/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5.6", "messages": [{"role": "user", "content": "Hello"}] }' Set "stream": true to receive Server-Sent Events with OpenAI-compatible chat.completion.chunk payloads, terminated by data: [DONE]. Without it the response is one chat.completion object. The request accepts model, messages, stream, tools, temperature, top_p, max_tokens, presence_penalty, and frequency_penalty. A response with tool calls has finish_reason tool_calls. Non-streaming usage reports 0 for every count. How the model picks the provider The server chooses the provider from the model prefix: Model starts with Provider Credential gpt-, o1, o3 OpenAI OPENAI_API_KEY claude- Anthropic ANTHROPIC_API_KEY anything else Ollama OLLAMA_API_URL (default http://127.0.0.1:11434) The key comes from the caller’s secret store, then the environment. Without model the request uses llama3.2:latest. Invalid JSON returns 400. A provider that fails to initialize returns 500 with an OpenAI-style error object. Models: GET /v1/models curl http://localhost:7777/v1/models The list is fixed, not queried from your providers: llama3.2:latest, gpt-4, gpt-4o, gpt-4o-mini, claude-sonnet-4-20250514, and claude-opus-4-20250514. Chat completions accept any model ID the prefix rules route. Use nodetool models list to see the models your install can run. Integration Example (JavaScript) const response = await fetch("http://localhost:7777/v1/chat/completions", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ model: "gpt-5.6", messages: [{ role: "user", content: "Hello, AI!" }], stream: true }) }); Because the endpoint is OpenAI-compatible, any OpenAI client SDK works by pointing its base URL at http://localhost:7777/v1. See Also Chat API — Full request/response schemas, streaming, and tool calls. NodeTool CLI — nodetool serve options. Deployment Guide — Running the server in production.--- # Chat CLI Overview Source: https://docs.nodetool.ai/chat.html Interactive CLI for chatting with models. Works with OpenAI, Anthropic, Gemini, Ollama, and others. The assistant can call tools during a session. Interactive REPL with history and completion Sandboxed workspace for file operations Provider-specific models Unified agent loop — the assistant calls tools and decomposes work on its own; no separate mode to toggle See the Chat CLI and Chat API.--- # Chrome Extension Source: https://docs.nodetool.ai/chrome-extension.html The NodeTool Chrome extension gives you two surfaces in the browser you already use. The chat side panel puts NodeTool’s agent beside the page you’re on, so you can ask about the tab without switching windows. It reads and writes the same conversation threads as the desktop and web apps. The CDP proxy relays the Chrome DevTools Protocol between your browser and your NodeTool server, so a workflow or an agent can click, type, scroll and screenshot inside a tab you’re already signed in to — with your cookies, sessions and 2FA already in place, instead of a fresh anonymous headless browser. They work together: opening the chat panel attaches the proxy to your current tab, so the agent can act on the page you’re looking at. New here? Start with Getting Started. For headless (no-login-needed) browsing, see the regular Browser node instead — you only need the proxy when a site requires your logged-in session. Overview   Chat side panel CDP proxy What it is NodeTool’s chat, in Chrome’s side panel A relay between local NodeTool clients and chrome.debugger What you get Ask an agent about the current page; the same threads as the app Workflows and agents drive your logged-in tab Transport /trpc and the /ws chat socket, msgpack frames Chrome native messaging to a local host, then a unix socket. No server involved Scope One conversation at a time, any thread on the server One tab at a time, attached explicitly — never automatic What it is not Not the full app: no media generation, no workflow editor Not an automation engine — it originates no commands, it only relays them The chat panel talks to a NodeTool server. The CDP proxy does not: the backend and the CLI reach it directly. Chrome 116 or newer is required for the side panel. Installing The extension is not published to the Chrome Web Store — build and load it unpacked from source. On a local install, the first launch with no API key stored opens a two-step onboarding. It signs in to ChatGPT and Claude, then sets up the extension with a live connection status. The app registers the native host itself, and the desktop app points Load unpacked at the extension folder it ships, so no terminal step is needed. Settings → Browser Extension has the same Connect to Chrome button. ChatGPT image generation and every Gemini request (chat, images, Veo video) run through the extension. git clone https://github.com/nodetool-ai/nodetool.git cd nodetool/chrome-extension npm install # standalone install — the extension lives outside the root npm workspace npm run build # vite build -> dist/ Then load it into Chrome: Open chrome://extensions. Enable Developer mode (top right). Click Load unpacked and select chrome-extension/dist. The extension icon appears in your toolbar. Click it to open the popup: Open chat opens the side panel, and below it the CDP proxy’s controls — a connection-status dot (disconnected / connecting / connected / error), an editable server WebSocket URL, and Attach / Detach buttons. Standalone package: chrome-extension/ is intentionally outside the root npm workspace, Turbo pipeline, and CI — it has its own package.json and is built on demand. npm run build:packages at the repo root does not build it. Downloading a prebuilt copy A server that already has the extension built will hand it to you zipped, which saves cloning the repo on the machine running Chrome — the desktop app’s install helper uses this. curl "http://localhost:7777/api/extension/download" -o nodetool-chrome-extension.zip unzip nodetool-chrome-extension.zip -d nodetool-extension Load nodetool-extension/ with Load unpacked. A server with no build to hand out answers 404 with {"detail": "Extension build not found"} — build from source as above, or point the server at an existing build with NODETOOL_EXTENSION_DIST. See API Reference. Development commands cd chrome-extension npm run typecheck # tsc --noEmit npm test # vitest run npm run dev # vite build --watch, for iterating on the extension itself npm run clean # remove dist/ Chat in the side panel Click the toolbar icon, then Open chat. Chrome’s side panel opens beside the page, and the extension attaches the CDP proxy to the current tab in the same gesture — so the agent can act on what you’re looking at without a second step. The empty state offers three starters that reflect what the panel is for: Summarize this page, Pull the key facts out of this page into a list, What can I do with this page? What’s in the panel Control Behavior Model picker Every language model from every provider your server has configured, grouped by provider, with a search box once the list is long. Nothing is picked for you — sending stays blocked until you choose one, and the composer says why. Permission mode Ask, Auto or Plan, persisted across sessions. See Permission modes below. Conversations The thread list, shared with the desktop and web apps. Start something in the panel and finish it in the app, or the reverse. Transcript Streamed replies with the agent’s tool calls listed as it works. An execute_code call renders its program as a titled code block rather than escaped JSON, and a call still running is marked at the tail of the turn. Server settings The gear icon — see Pointing the panel at a server. The panel is chat only. Image, video, audio and the other media-generation modes live in the full app, as does the workflow editor. Permission modes The picker sets how far the agent may act on its own, and the panel remembers your last choice: Mode Behavior Ask Reads run automatically; a write, execute or external call parks on an approval card. This is the default. Auto Everything runs unasked. Plan Read and propose only. No actions taken. These are the same modes the full app offers, where Ask is labelled Default. Chat & Agents explains what each one does to the agent loop. Auto points a running agent at whatever your server can reach, inside your signed-in browser session. Pick it only for a server you trust. Approvals Three requests park in the transcript as cards rather than failing the turn: A gated tool call asks Run this action?, Make this change? or Allow this action? depending on whether the call is classified execute, write or external. The card names the tool, expands to show the code or the arguments, and offers Allow, Allow for chat (stops the asking for the rest of the thread) or Deny. A proposed plan shows its title and tasks, and takes an approval or a rejection with feedback on what should change. A missing credential asks for the key by name. The value is saved to the server’s secret store from the card; it never travels over the chat socket. Pointing the panel at a server The gear icon holds the panel’s own server setting, separate from the proxy’s relay URL: Field Notes NodeTool server URL HTTP base, e.g. http://localhost:7777. The panel calls /trpc and the /ws chat socket on this host. Access token Only for a server that enforces authentication. Leave it empty for a local server — it maps loopback requests to the local user. The host has to be one Chrome lets the extension reach. localhost, 127.0.0.1 and any HTTPS host are granted in the manifest. Anything else — a LAN box over plain HTTP — prompts for permission when you save it; decline, and every request to that server is blocked by CORS even though the chat socket still connects. A green connection dot with failing requests is exactly this case: WebSockets are not CORS-gated, the HTTP calls are. Driving your logged-in browser The proxy is the half that lets a workflow or an agent act inside a tab you are signed in to. Why you’d use it Many AI product sites — Midjourney, Sora, Runway, ElevenLabs, and similar generation tools — either block headless Chrome outright or require a logged-in account with 2FA and session cookies that a fresh, server-launched browser doesn’t have. Re-implementing every such site as a dedicated API integration is slow and brittle to UI changes. The Chrome Extension solves this by reusing the browser you already use every day. You attach the extension to a tab where you’re already signed in, and a workflow’s browser-automation node drives that tab through the same action loop (click, type, scroll, extract, screenshot) it would use against a headless browser — just over a different transport. Typical scenario: You’re signed into Midjourney in a regular Chrome tab. You attach the extension to that tab, then run a workflow that submits a prompt, waits for the image grid to render, and downloads the results — all inside your real session, with no API key or cookie-jar wrangling. Recipe: generate an image on a logged-in site Install the extension (see Installing above) and pin it to your toolbar. Sign in to the target site (e.g. Midjourney) in a normal Chrome tab, as you would manually. Register the native host once: click Connect to Chrome in the app (onboarding or Settings → Browser Extension), or run nodetool extension install. Then reload the extension in chrome://extensions. Check it with nodetool extension status. No server needs to run. Open the extension popup on the signed-in tab and click Attach to this tab. Chrome shows its standard “Nodetool is debugging this browser” banner while attached. Ask the agent, run a workflow, or call a provider such as nodetool generate dreamina …. The browser_* tools drive the attached tab — browser_status first to confirm the extension is connected, then browser_restart with transport: "extension" if the process is still on its headless Chrome (or set NODETOOL_BROWSER_TRANSPORT=extension before starting it — see Transport selection). Run it. The action loop drives your attached tab: it types the prompt, submits the form, waits for the result, and captures a screenshot or the generated asset into your library. Detach from the popup (or just close the tab) when you’re done — an attachment lasts only for the current browser session. Because the site sees your real, logged-in browser rather than a bot-like headless process, you avoid the CAPTCHAs, bot-detection blocks, and re-authentication flows that a server-launched browser would hit. Example: the Dreamina provider The dreamina provider generates images, videos and music through your own logged-in Dreamina tab, so it needs no API key and exists on local installs only. Keep a signed-in dreamina.capcut.com tab open and the extension connected (nodetool extension status). The provider asks the extension for that tab by URL, so the active tab does not matter. nodetool generate dreamina --list-models nodetool generate dreamina high_aes_general_v50p_large "a red apple" --aspect-ratio 16:9 -o apple.png Images go through textToImage and imageToImage. Image-to-image uses the site’s blend mode, takes up to nine source images and a strength from 0 to 1, and works on models that list byte_edit. Video goes through the provider API: textToVideo for every model Dreamina lists, imageToVideo for models that list first_frame (the image is the first frame, and endImage is the last frame on models that list end_frame), and referenceToVideo for the Seedance 2.x models (up to nine reference images, three reference videos and three reference audios, placed in the prompt with [Image N], [Video N] and [Audio N] markers). A reference audio needs an image or video beside it. Multiframes, smart edit and long video are not supported. Music goes through textToMusic. The instrumental model needs no model key and returns a WAV track (44.1 kHz stereo) of the length you ask for in whole seconds, 10 seconds by default. Vocal models, with optional lyrics, appear in getAvailableMusicModels only when your Dreamina account lists them. The calls spend your Dreamina credits. The native host Chrome starts the host when the extension connects and ends it when the connection closes. It listens on a per-user unix socket (<tmp>/nodetool-browser-bridge-<user>/<pid>.sock, mode 0600 in a 0700 directory), so only your OS user reaches the extension. Several clients can share one extension: the host rewrites command ids per client, sends events to all, and keeps the debugger attached until the last client lets go. nodetool extension install writes the host manifest for Chrome, Chromium, Brave, Edge and Arc. The extension ID is fixed by the key in chrome-extension/manifest.json. How it works The extension is deliberately “dumb” — a pure conduit with no CDP logic of its own. All browser-automation semantics (which elements to click, how to wait for a page to settle, the action loop) live in the client process; the extension just carries the bytes. NodeTool backend / CLI / provider │ CDP commands/events, NDJSON on a unix socket ▼ native host (started by Chrome) │ native messaging, length-prefixed JSON ▼ Chrome Extension (service worker) │ chrome.debugger.sendCommand / onEvent ▼ Your real, logged-in Chrome tab Background service worker (src/background/service-worker.ts) owns the relay. It restarts the relay on chrome.runtime.onInstalled/onStartup because Manifest V3 service workers get evicted and need to reconnect, and it uses a chrome.alarms keepalive (roughly every 24 seconds) to stop the worker idling out while a debugger session is attached. CDP relay (src/lib/cdp-relay.ts) holds a native messaging port to the host with exponential backoff (1–30s) if it drops. A missing host shows up as the popup’s error text. Attach is a user gesture, with one exception: chrome.debugger.attach is never called on page load. The popup’s Attach to this tab button is the intended path, but a host attach frame on an unattached relay also attaches the currently active tab as a convenience (handleAttachRequest in src/lib/cdp-relay.ts). Any process running as your OS user can reach the host socket and so start driving whatever tab is focused. Filesystem permissions on the socket are the access control. Attaching is mutually exclusive with having Chrome DevTools open on the same tab. Wire protocol: JSON frames (native messaging between Chrome and the host, newline-delimited JSON on the socket). Five frame kinds are live: cdp / cdp_result / cdp_event (command/response/event relay), attach / attached / detach (session lifecycle), ping / pong (heartbeat), and error (fatal — e.g. the user closed the tab or DevTools banner). Three more are declared but not implemented on either side: asset_chunk (server → extension, a file-upload injection) and media_chunk / media_end (extension → server, a chrome.downloads capture). Uploads reach a file input through an in-page DataTransfer injection instead, and media is captured with Network.getResponseBody or an in-page fetch; the three frames are the shape a chrome.downloads fallback would take if a site defeats both. Transport selection One browser session exists per server process, and it is driven by either transport: a headless Chrome the server launches, or your attached tab. The same action loop, element indexing, and screenshot logic runs against both — only the CDP client differs — so nothing built for headless browsing needs different logic to use the logged-in session. Set the default before the server starts: # Use the extension-attached browser instead of a headless one NODETOOL_BROWSER_TRANSPORT=extension nodetool serve # Or pin a specific host socket (rarely needed) NODETOOL_BROWSER_BRIDGE_SOCKET=/path/to/host.sock nodetool serve Or switch a running server from the agent side, which is the same decision made later: {"tool": "browser_restart", "arguments": {"transport": "extension"}} Restarting is the only point the transport can change, because the session is a process singleton: switching tears the current one down first. On the local transport that kills Chrome and relaunches it (cookies and history gone); on the extension transport it only detaches and re-attaches the debugger, leaving your browser alone. When the extension transport is selected but no native host is running, the attach fails at once with “The NodeTool browser bridge is not running. Install the native host with nodetool extension install …”. browser_status is how an agent finds this out first. As an agent capability The browser_* capabilities are the extension’s agent-facing surface. They are registered like any other NodeTool capability, so the same fourteen names reach the chat agent, an AgentNode in a workflow, MCP clients, CodeAct, a Code node, and a JS script — and every one of them drives the same page. Capability What it does browser_status Which transport is live, whether an extension is attached, where an open session is pointed browser_view URL, title, viewport, indexed interactive elements, screenshot browser_navigate Load a URL browser_restart Restart the session; the one place transport changes browser_click, browser_input_text, browser_press_key, browser_select_option, browser_move_mouse, browser_scroll Act on the page, addressing elements by their browser_view index or by coordinates browser_console_exec, browser_console_view Evaluate an expression in the page; read recent console messages browser_capture_media Save an image/video/audio the page produced as a NodeTool asset browser_upload_asset Inject an existing asset into a page file input Element indexes are rebuilt on every browser_view, so an agent views before it acts on an index. browser_status is the one to call first when a task needs the logged-in session: { "transport": "extension", "session_open": false, "extension_connected": false, "url": null, "title": null, "hint": "No NodeTool browser bridge is running. Ask the user to run `nodetool extension install`, load the NodeTool extension in Chrome and reload it; until then every browser action fails to attach." } extension_connected is null on the local transport, where the question does not apply. Not available on nodetool.ai. The cloud profile drops all fourteen: the browser session is one page per server process, shared by every caller, which is a single-tenant shape — and the extension transport would put one user’s own Chrome behind a shared server. They are offered where the machine belongs to its user: the desktop app, a local server, or a self-hosted install (NODETOOL_NODE_PROFILE=full). Permission-wise: reading the page (browser_status, browser_view, browser_console_view) is classified read, browser_restart and browser_console_exec are execute, and everything that acts on the page — a click, a keystroke, an upload — is external, because it lands on a third-party site inside the user’s own logged-in session. The action loop itself is @nodetool-ai/browser (packages/browser/), which knows nothing about agents, nodes or assets — a screenshot comes back from it as base64. The capability module imports it directly and owns the half that needs a ProcessingContext: persisting those bytes as an asset, and resolving an asset id back to bytes for an upload. That split is why the lib.browser.Screenshot node can share the same action loop without depending on the agent layer. Server requirements The CDP proxy needs no server. It needs the native host (nodetool extension install) on the same machine and OS user as the extension and the client. The chat panel needs a running NodeTool server (nodetool serve --port 7777 or your deployed URL), reached through /trpc and /ws. Chrome 116 or newer — chrome.sidePanel is what the chat panel opens into. Limitations Some pages cannot be attached. chrome.debugger refuses chrome://, chrome-untrusted://, devtools://, edge:// and about: pages, other extensions’ pages, and the Chrome Web Store. Open chat attaches first, so on one of these pages it shows the error and does not open the panel. Switch to a normal page and retry. One tab at a time. The extension proxies a single chrome.debugger session; attach a different tab and the previous one is implicitly detached. Not published to the Chrome Web Store. Install as an unpacked extension from source. Mutually exclusive with DevTools. You can’t have Chrome DevTools open on a tab while the extension is attached to it (both use chrome.debugger). Session-only. Attaching does not persist across Chrome restarts — reattach after restarting your browser. Exclusive tab ownership. The native host queues clients across server and CLI processes. Each browser provider holds the tab through submission, response polling, and downloads, then releases it. Update the extension alongside the native host so release acknowledgments can advance the queue. The popup cannot change tabs while a client owns the debugger. Chat only in the panel. Media generation and the workflow editor are the full app’s job. Tool, plan and secret approvals do render in the panel. Manual build. The chrome-extension/ package isn’t part of npm run build:packages — build it on demand. A diff touching it does run the live-browser surface’s selfcheck through nodetool harness gate, but that covers the capability seam, not the relay: the extension → chrome.debugger → page round trip runs only in npm run test:integration --workspace=packages/browser, which needs Chrome and port 7777. Keep the two protocol definitions (chrome-extension/src/lib/protocol.ts and packages/browser/src/extension/protocol.ts) in sync by hand if you change the wire format, and likewise chrome-extension/src/lib/chat-socket.ts with packages/sdk/src/chat.ts for the chat panel. Troubleshooting The chat panel shows “Cannot reach …” Check the server URL in the panel’s settings (gear icon). If it names a host that is not localhost, 127.0.0.1 or an HTTPS address, save it again and accept Chrome’s permission prompt — without that grant the browser blocks every request to it. A green connection dot with failing requests is exactly this: the WebSocket is not CORS-gated, the HTTP calls are. Popup shows “disconnected” The popup shows the Chrome error when the native host is missing (“Specified native messaging host not found”). Click Connect to Chrome in the app (or run nodetool extension install) and reload the extension. The host is registered for one extension ID. If you loaded the extension from another folder, check that its ID matches nodetool extension install output. “No browser extension is connected” error from a workflow The extension must be attached to a tab before running a workflow that uses the extension transport — attaching is never automatic. Open the popup and click Attach to this tab. Verify NODETOOL_BROWSER_TRANSPORT=extension is set on the process running the workflow. Chrome shows “Nodetool is debugging this browser” and I can’t open DevTools This is expected while attached — chrome.debugger sessions and DevTools are mutually exclusive on the same tab. Detach from the popup, or use a different tab for DevTools. Attach fails or the tab appears frozen Close any open DevTools panel on that tab first, then retry Attach to this tab. If the tab was closed or navigated away unexpectedly, the relay reports a fatal error and tears down the session — reopen the target page and reattach. Related topics Getting Started — Desktop setup and first workflow Developer: Node Reference — Browser-automation node reference API Reference — Server API and WebSocket documentation WebSocket API — General WebSocket protocol details--- # CLI Pack System: On-Demand Node Packs for the npm CLI Source: https://docs.nodetool.ai/cli-pack-system-design.html CLI Pack System: On-Demand Node Packs for the npm CLI Status: design, not implemented. Scope: the npm package @nodetool-ai/cli. The desktop app and the Docker image keep their bundled backend. This revision answers the adversarial review of the previous draft. Finding codes F1 to F13 and question codes Q1 to Q5 refer to that review. Decisions D1 to D7 are fixed. New items use D8 and later, R1 and later, and Q6 and later. 1. Summary The core CLI stops depending on node packs and on heavy native libraries. Step 1 of the migration is this dependency cut plus a measured size gate. No installer work starts before it passes. Packs are ordinary npm packages. They install into the npm project that already holds the CLI, so packs and core resolve from one tree. A static generated catalog maps each node type to its pack. Every node type stays visible for search, validation, and the planner without its pack. A run resolves missing packs before it starts. A missing pack produces a structured missing_pack error and exit code 3. Only the CLI command layer can install. It needs its own authorization, separate from the permission gate. The server and MCP never install. After an install, the CLI re-executes itself. One process sees one fixed set of packs. 2. Problem npm install @nodetool-ai/cli@0.8.3 installs about 1.2 GB on darwin-arm64. The CLI depends on every node pack, and core packages depend on heavy native libraries. A read-only walk of the local workspace tree (method in section 8) gives 868 MB of apparent file size and 1,087 MB of disk usage for the current CLI closure, in 884 packages. The difference from 1.2 GB comes from different resolved versions in a clean install. Item Apparent size Pulled in by webgpu 67.9 MB gpu optionalDependencies (packages/gpu/package.json:22) node-av platform packages (@seydx/*) 58.8 MB agents → @mediabunny/server (packages/agents/package.json:23) @napi-rs/* (canvas) 48.6 MB agents, data-nodes, video-nodes, game-renderer node-web-audio-api 40.8 MB audio-nodes node-llama-cpp and @node-llama-cpp/* 48.0 MB cli (packages/cli/package.json:68) pdfjs-dist, pdf-parse, @hyzyla/pdfium 65.9 MB document-nodes, websocket @opentelemetry/* 23.6 MB runtime (packages/runtime/package.json:27-33) typescript (@typescript/*) 23.2 MB execution (packages/execution/package.json:29) js-tiktoken 21.4 MB runtime (packages/runtime/package.json:36) three, playwright-core 34.0 MB game-renderer, video-nodes All numbers in this document are local-tree estimates, not clean-install measurements. 3. Goals and non-goals Goals: A core install at or below the budget in section 8.4 that runs --help, validate, run and debug for core-node workflows (JSON and DSL), node run for core nodes, serve, and mcp serve. Every node type stays discoverable without its pack: search_nodes, get_node_info, validate, and the agent planner see the full catalog. A missing pack never fails silently. Every surface names the pack, its size, and the command that installs it. Packs install only with install authorization (D5). Installed packs always match the core version exactly. Non-goals: The desktop app and the Docker image. They keep @nodetool-ai/base-nodes and the full bundle from scripts/bundle-backend.mjs. Python packs. They keep the Python bridge. Third-party packs. They keep the existing loader in packages/node-sdk/src/pack-loader.ts, which discovers packages that carry a nodetool field in package.json. First-party packs carry no such field today and must not gain one (D8). pnpm, Yarn, Bun, and npx installs. Section 11 defines how the CLI detects them and what it prints. 4. Tiers Tier Contents Installed Core cli, protocol, config, security, auth, storage, models, node-sdk, kernel, runtime, execution, agents, chat, websocket, dsl, vectorstore, app-runtime, app-export, compute, model-pricing, timeline, gpu, storyboard, image-editor, model3d, browser, sandbox-compiler, huggingface, game-runtime, deploy, telegram, sdk, nodes-utils, the core node packages core-nodes, text-nodes, data-nodes, llm-nodes, code-nodes, and the new node-catalog with the CLI Node pack One npm package per pack id. Existing ids from BUILTIN_NODE_PACKS (packages/protocol/src/builtin-packs.ts:61): whisper-cpp, elevenlabs, minimax, transformers-js, fal, kie, topaz, reve, atlascloud, higgsfield, together, replicate, huggingface. New ids from the base split: image, audio, video, document, game, blender, manimgx, automation, integration on demand Runtime library Third-party packages that a core code path loads lazily: imaging (sharp, @napi-rs/canvas), node-llama-cpp, typescript, tokenizer (js-tiktoken), tracing (OpenTelemetry SDK packages), esbuild, rapier (@dimforge/rapier3d-compat). First-party libraries belong to their node pack (plan D18). on demand Rules: D16. Five libraries stay regular dependencies, because the features that use them have no fallback: webgpu (gpu), mediabunny with @mediabunny/server (agents), @hyzyla/pdfium (websocket, document-nodes), chart.js (data-nodes) and node-web-audio-api (audio-nodes, so it installs with the audio pack). The desktop app and the Docker image already require them. D8. A core package never declares a node pack in dependencies, optionalDependencies, or a non-optional peer. D9. A heavy library is a regular dependency of each pack that needs it. A core package that loads it lazily declares it as an optional peer with an exact version. This keeps the monorepo tree and the desktop bundle complete, because optional peers alone are never installed (R4). D10. A pack declares core packages as exact peerDependencies. scripts/prepare-publish.mjs:53 already pins peers. A pack declares another pack it needs (for example video needs audio) as an exact regular dependency, and the catalog records it as requires. The base pack in BUILTIN_NODE_PACKS is required today (packages/protocol/src/builtin-packs.ts:69-75) and aggregates about 60 node lists through ALL_BASE_NODES (packages/base-nodes/src/index.ts:527). The split makes each npm package its own pack. Node type strings stay unchanged. The generator in section 9 decides ownership, not this table. 5. Decisions and finding map The implementation plan defines D17 to D22, including the shared runtime registry and bundled server entry. Code Status in this design F1 Fixed by D5 and section 13. The installer checks its own authorization. F2 Fixed by section 6. Step 1 of the migration. F3 Obsolete under D1. Packs resolve core from the same tree. Section 14.6 still verifies one copy. F4 Fixed by section 7. F5 Fixed by sections 14.6 and 17. Verification runs before scripts and imports. F6 Obsolete under D1. No generations, no hard links. F7 Obsolete under D1 and D2. One lock per install root, no GC, no live switching. F8 Fixed by D3 and section 9. Exact node_type → pack map. F9 Fixed by D6 and section 10. F10 Fixed by D4 and section 12. No mid-run retry. F11 Section 14 pins npm and flags. Section 19.3 is the matrix. F12 Section 8. Measured, with a budget. F13 Fixed by D7 and section 17. D1 needs one correction. The review of npm behavior in section 11.2 shows that “the global prefix” cannot mean a separate npm install -g <pack>. It must mean the CLI’s own package directory inside the global prefix. Section 11 states the rule. 6. Core dependency cut (F2) 6.1 Manifest edges from core to packs and heavy libraries Read from the package.json files. “Cut” names the change. Edge Location Cut cli → 14 node packs (atlascloud, base, elevenlabs, fal, huggingface-nodes, minimax, replicate, reve, transformers-js-nodes, transformers-js-provider, video, whisper-cpp) and game-renderer packages/cli/package.json:24-75 Remove. Load through the pack loader (section 12). cli → reliability-harness packages/cli/package.json:46 Remove. It depends on base-nodes and websocket (reliability/harness/package.json). nodetool reliability already imports it dynamically (packages/cli/src/commands/reliability.ts:98) and reports “development checkout only” when absent. agents → blender-nodes, game-nodes, game-renderer, video-nodes packages/agents/package.json:26,30,31,44 Remove. Lazy pack import (6.2). agents → @mediabunny/server, mediabunny packages/agents/package.json:23,61 Keep as dependencies (D16). agents → @napi-rs/canvas packages/agents/package.json:24 Optional peer. Runtime library imaging. agents → office-text-extractor packages/agents/package.json:98 (optionalDependencies) Remove from optionalDependencies. It stays an optional peer (:113). websocket → 18 node packs packages/websocket/package.json:35-81 Remove. Lazy pack import. websocket → @hyzyla/pdfium, sharp packages/websocket/package.json:29,78 @hyzyla/pdfium stays a dependency (D16). sharp becomes an optional peer. Runtime library imaging. dsl → base-nodes, elevenlabs, huggingface-nodes, minimax, reve, transformers-js-nodes, whisper-cpp packages/dsl/package.json:37-50 Remove. Metadata comes from the catalog (6.2). execution → gpu → webgpu packages/execution/package.json:20, packages/gpu/package.json:22 Move webgpu from optionalDependencies to dependencies (D16). model-pricing → fal-nodes, kie-nodes packages/model-pricing/package.json:18-19 Move the generated unit-price JSON into model-pricing. Change the codegen output paths. The imports are packages/model-pricing/src/index.ts:46-47. runtime → js-tiktoken packages/runtime/package.json:36 Already an optional peer with a lazy import. Remove the remaining optional dependency from runtime and agents so clean installs omit it. Runtime id tokenizer. runtime → sharp packages/runtime/package.json:40 Optional peer. All four runtime uses are already dynamic. runtime → OpenTelemetry SDK packages packages/runtime/package.json:28-33 Keep @opentelemetry/api. Make core, resources and sdk-trace-base optional peers. They also pull in sdk-trace and semantic-conventions. Runtime library tracing. game-runtime → @dimforge/rapier3d-compat packages/game-runtime/package.json Optional peer. Regular dependency of the game pack. The browser keeps a literal dynamic import. Node can fall back to the managed rapier runtime root. Failed attempts reset the initialization cache. sandbox-compiler → esbuild packages/sandbox-compiler/package.json:25 Optional peer. Needed only by nodetool packs compile. data-nodes → @napi-rs/canvas, chart.js packages/data-nodes/package.json:18,23 chart.js stays a dependency (D16). @napi-rs/canvas becomes an optional peer. The chart node already imports it dynamically (packages/data-nodes/src/nodes/lib-charts.ts:162-167). D11. One new edge is required. packages/cli/src/run-dsl.ts:11 imports tsx/esm/api, but tsx is only a dev dependency (package.json:232, packages/dsl/package.json:54). A clean install cannot run a DSL file through that path. tsx becomes a regular dependency of cli (10.9 MB, 4 packages). The two npx tsx calls (packages/cli/src/nodetool-main.ts:586, packages/cli/src/debug/target.ts:49) are replaced by in-process tsImport. npx fetches tsx from the registry when it is absent, which is an install without consent. The release catalog also carries installPins for provided companions, npm runtime libraries and shared mediabunny packages (plan D24). Each record has the registry name, exact version and integrity. installName identifies an npm alias folder and shared: true marks shared identity checks. Runtime ids still resolve through the protocol runtime table. Runtime pins include measured installBytes for their production trees with scripts disabled. Each estimate includes dependencies, so sums can overlap. Source generation and cohort version changes clear these pins until release packing fills them. 6.2 Source import edges A static scan of src/ in every core package (tests excluded) found these imports of node packs. A load-time trace confirmed which ones run on module load: it imported each core package with a Node resolve hook and logged every pack or heavy-library specifier. Import Kind Cut packages/cli/src/nodetool-main.ts:46, packages/cli/src/node-registry.ts:15 → base-nodes/builtin-packs static Replace registerBuiltinPacks with the catalog pack loader (section 12). packages/cli/src/providers.ts:38-39 → transformers-js-provider, whisper-cpp static Register each provider only when its pack is installed. packages/cli/src/commands/timeline-render.ts:267-430 → video-nodes/nodes/timeline/* dynamic importPack("video", subpath). The type import at :12 is erased. packages/cli/src/commands/game.ts:419-637, game-smoke.ts:11 → game-renderer dynamic importRuntimeLibrary("game-renderer", subpath). packages/agents/src/capabilities/model3d.ts:28, timeline-bake.ts:22 → blender-nodes static, inside a lazily loaded capability module importPack("blender"). The capability returns missing_pack. packages/agents/src/timeline-preview/model3d.ts:32,161 → video-nodes/nodes/model3d/* static and dynamic importPack("video", subpath). packages/agents/src/capabilities/game*.ts → game-nodes, game-renderer dynamic importPack("game"), importRuntimeLibrary("game-renderer"). packages/agents/src/analysis/media-decode.ts:16, mediabunny-runtime.ts:12, sandbox-av-media.ts:22 → mediabunny static Keep. mediabunny is core (D16). @napi-rs/canvas static imports in agents (capabilities/agentic-storyboard-finish.ts:2, derive-shot.ts:13, storyboard-layout-check.ts:1, timeline-clip-frames.ts:1, evals/surfaces/sketch.ts:33, timeline-preview/compare.ts:16, frames.ts:23, model3d.ts:23, rasterize.ts:17, sheet.ts:14) and timeline (fonts/google-fonts-node.ts:15, fonts/register-node.ts:18) static One loadCanvas() helper that throws missingRuntimePackageError. The trace shows evals/surfaces/sketch.js and the timeline font modules load on import "@nodetool-ai/agents", so these block startup today. packages/websocket/src/node-registry-setup.ts:33 → base-nodes/builtin-packs static The server takes a pack source: bundled (desktop, Docker: base-nodes registrars) or installed (CLI: catalog loader). packages/websocket/src/server.ts:28-30 → transformers-js-provider, whisper-cpp, base-nodes (setCodeNodeAgentsModule) static Providers register only when installed. Import setCodeNodeAgentsModule from code-nodes, which stays core. packages/websocket/src/routes/nodes.ts:10 → base-nodes (resolveComfyWorkflow, resolveKieDynamicSchema) static importPack("integration"). The route returns missing_pack. packages/websocket/src/routes/documents.ts:17 → document-nodes/lib/pdf-text static importPack("document", "lib/pdf-text"). The route already handles missingRuntimePackageOf (:16). packages/websocket/src/triggers/file-watch.ts:35 → automation-nodes static Extract the file-watch predicates into nodes-utils, shared with websocket. Keep their Node path imports out of the browser protocol entrypoint. packages/websocket/src/trpc/routers/games.ts:9 → game-renderer/preparation3d static importRuntimeLibrary("game-renderer"). packages/websocket/src/routes/kie-pricing.ts:7 → kie-nodes/kie-pricing-api static Move kie-pricing-api.ts into model-pricing. It has no imports. packages/websocket/src/trpc/routers/models.ts:7,44 → whisper-cpp, transformers-js-nodes static Lazy. Return empty model lists with a pack hint when absent. packages/websocket/src/model-download-runtime.ts:16, routes/games.ts:110 dynamic Pack import with missing_pack. packages/dsl/src/export.ts:1, script-export.ts:22 → ALL_BASE_NODES static Read metadata from @nodetool-ai/node-catalog. buildFactoryRefs (export.ts:116) and builtinMetadataFor (script-export.ts:228) need metadata only, not classes. packages/dsl/src/registry.ts:30-69 → base-nodes and six packs dynamic Use the catalog pack loader. Catalog-only types are never Python candidates (section 10). packages/model-pricing/src/index.ts:46-47 static See 6.1. packages/websocket/src/lib/thumbnail.ts:29, mcp-agent-tools.ts:87, screenshot-docs-seed.ts:26, trpc/routers/games.ts:4 → sharp static Route through runtime/src/image-codec.ts, which already imports sharp dynamically. packages/runtime/src/run-trace-processor.ts:3 → @opentelemetry/resources static Lazy. Load only when a trace sink is configured. packages/sandbox-compiler/src/bundle.ts:17 → esbuild static Lazy. 6.3 Loader helpers Two helpers live in core. They are the only way core code reaches a pack or a runtime library. // @nodetool-ai/node-catalog export function importPack<T>(packId: string, subpath?: string): Promise<T>; export function importRuntimeLibrary<T>(libraryId: string, subpath?: string): Promise<T>; Each resolves the npm name from the catalog, calls importOptionalModule (packages/config/src/optional-modules.ts:12), and maps a module-not-found error for that package to a MissingPackError carrying the missing_pack payload (section 12.2). Other errors propagate. A lint rule forbids direct imports of node packs from core packages, and the clean-core tests in 6.4 prove it at runtime. 6.4 Clean-core tests The monorepo tree contains every pack, so in-repo Jest cannot prove the cut. scripts/smoke-packed-cli.mjs already installs packed tarballs into an empty project and runs the CLI against an empty HOME. It gains a core-only variant that installs the CLI tarball with no pack tarballs available: nodetool --help lists every command, including those that need packs. node -e 'import("@nodetool-ai/dsl")' from the install root succeeds. nodetool run examples/workflows/concat_text.ts succeeds (DSL through tsx). nodetool workflows run on a core-node JSON workflow succeeds. nodetool serve answers GET /health. nodetool mcp serve answers initialize, lists tools, and runs a core-node workflow through execute_code. nodetool validate on a fal.* workflow reports pack_not_installed and runs property validation on the fal.* node. nodetool run on that workflow exits 3 and prints the missing_pack JSON with --json. A guard asserts that no @nodetool-ai/* node pack and none of the runtime libraries in section 4 exist under the install’s node_modules, except esbuild required by tsx and TypeScript required by the non-optional tRPC peers (plan D23), including the compiler alias dependency declared by their resolved @typescript/typescript6 wrapper. The exceptions must resolve from those dependents and satisfy their declared ranges. The check fails on a deliberately added pack before it is enforced. 7. Shared third-party identity (F4) ensureCodecs (packages/agents/src/analysis/mediabunny-runtime.ts:21-32) registers the @mediabunny/server codecs into whatever mediabunny instance the server package resolves. inputFrom (:35) creates inputs with the instance that agents resolves. The installed @mediabunny/server declares mediabunny as a peer. Two instances mean codecs register on one and inputs open on the other. D12. A shared-identity library is a package whose module instance two installed packages must share. The catalog marks it (shared: true). The rules: Core and packs resolve shared-identity libraries from one tree (D1). The catalog pins each shared-identity library to one exact version and integrity. Core declares it as an exact optional peer. The installer refuses a plan that adds a second copy (section 14.5). After install, the installer resolves the library from each dependent package directory and requires one realpath (section 14.6). mediabunny and @mediabunny/server are regular dependencies of agents (D16), so one tree holds both. Test: in the packed-tarball smoke, call ensureCodecs() and decode the first audio frame and the first video frame of a checked-in fixture through inputFrom. The test asserts decoded sample counts and frame dimensions, not only a successful import. 8. Size measurement and budget (F12) 8.1 Method A read-only walker starts at the core workspace packages and follows dependencies, optionalDependencies, and non-optional peerDependencies. It resolves each name with Node’s node_modules lookup from the dependent’s real directory. Workspace edges outside the core tier are not followed. Each external package directory counts once, excluding its nested node_modules. Workspace packages count dist/, package.json, and README.md, which match their files fields. Apparent size sums st_size. Disk usage sums st_blocks × 512. The tree is the local darwin-arm64 checkout with Node 24. Step 1 commits this walker as a script so that CI can rerun it. 8.2 Results (local-tree estimates) Run npm run measure:cli-closure -- --json for the current closure and projected core. The walker rejects an incomplete dependency tree. Scenario Apparent Disk S0: CLI closure with the prerequisite changes 686 MB 838 MB S3: projected core with the section 6.1 cuts and D16 libraries 453 MB 586 MB These are local darwin-arm64 estimates. A9 sets the release gate from a clean linux-x64 core-only install, with five percent headroom. D16 cancels three cuts in the table below (@hyzyla/pdfium, mediabunny, chart.js) and keeps webgpu and node-av in core. On the local tree the five core additions use about 170 MB of disk: webgpu 68 MB, node-av with its platform package 64 MB, werift 11 MB, mediabunny 11 MB, @hyzyla/pdfium 11 MB, chart.js 6 MB. node-web-audio-api (41 MB) moves with the audio pack. The projections in 8.3 predate D16. The gate in 8.4 uses the clean installed core tree with D16 included. Savings of each cut, measured on top of S1: Cut Apparent Disk OpenTelemetry SDK packages 30.2 MB 47.3 MB @napi-rs/canvas 25.2 MB 25.3 MB typescript 23.3 MB 23.6 MB js-tiktoken 21.4 MB 21.5 MB sharp and @img/* 18.3 MB 18.4 MB @hyzyla/pdfium 10.8 MB 10.8 MB esbuild 10.3 MB 10.3 MB mediabunny 9.8 MB 10.4 MB @dimforge/rapier3d-compat 7.9 MB 8.0 MB chart.js 6.0 MB 6.3 MB 8.3 Adjustments for a clean install Adjustment Apparent Disk better-sqlite3: the local tree holds a source build (17 MB build/), a prebuilt install holds one 1.9 MB .node file −15 MB −15 MB Source maps in core dist/ (do not publish *.map) −16 MB −27 MB tsx added (D11) +11 MB +11 MB Projected clean core about 213 MB about 314 MB The largest remaining items in S3: better-sqlite3 (26.5 MB locally), drizzle-orm (9.9 MB), openai (9.3 MB), @mixmark-io/domino via turndown (7.4 MB), @anthropic-ai/sdk (6.1 MB), and NodeTool’s own dist/ (50 MB apparent: 16 MB JavaScript, 15 MB maps, 14 MB declarations). 8.4 Budget D13. The gate measures du -sk node_modules after a clean npm install <cli tarball> in an empty project on linux-x64 in CI. Hard gate: 892,097 KiB for the core-only Linux x64 install. The successful packed install measured 849,616 KiB. The limit adds five percent, rounded up to the next KiB. The same runner measured the bundled CLI at 1,075,516 KiB. Ratchet target for a later release: 250 MB disk. It needs fewer declaration files in runtime-only packages and lazy provider SDKs. The fs-safe dependency has already been removed. The previous 200 MB goal is not reachable with SQLite, the LLM provider SDKs, the MCP SDK, and Fastify in core. The check must fail on a deliberately oversized fixture before it is enforced. 9. The node catalog (D3, F8) @nodetool-ai/node-catalog is a new core package. It holds: index.json: pack entries and the exact owner map. Runtime library ids use RUNTIME_NPM_PACKAGES from protocol (D17). data/nodes.ndjson: checked-in metadata, one node per line, sorted by node_type (D21). The package build writes dist/nodes.ndjson.gz. Run npm run generate:node-catalog to rebuild the catalog and report its node count. Run npm run measure:cli-closure to measure package sizes. 9.1 Format interface NodeCatalogIndex { schemaVersion: 1; coreVersion: string; // equals the CLI version packs: CatalogPack[]; owners: Record<string, string>; // node_type -> pack id, total } interface PinnedPackage { name: string; version: string; // exact integrity: string; // sha512 of the published tarball } interface CatalogPack { id: string; // stable id, including split domain packs kind: "core" | "node-pack"; npm: PinnedPackage; // version === coreVersion registrar: string; // named export, called with the registry provides: readonly string[]; // first-party packages carried by this pack (D18) requires: readonly string[]; // other pack ids runtimeLibraries: readonly string[]; secrets: readonly string[]; // e.g. ["FAL_API_KEY"] scriptAllowlist: readonly PinnedPackage[]; // closure packages allowed to run install scripts installBytes: number; // disk, linux-x64, measured by the release job nodeCount: number; } interface CatalogNode { node_type: string; pack: string; metadata: NodeMetadata; // final registrar output declared: readonly DeclaredPropertyMetadata[]; // input to validateNodeProperties } 9.2 Generation scripts/generate-node-catalog.mjs runs in a child process with a fixed environment (an empty temporary home directory, no inherited secrets, NODE_ENV=production): For each pack, create a fresh NodeRegistry, call the pack’s registrar, and record the registered types as that pack’s ownership. Registrars are the source, not namespace prefixes. The core and audio registrars apply the compatibility metadata overrides (D19). Fail when two packs register the same type (duplicate ownership). Read getMetadata() and the class’s declared properties for each type. Fail when a declared property holds a value that JSON cannot represent. Sort packs by id, nodes by node_type, and object keys. Omit timestamps. Write data/index.json and data/nodes.ndjson. The package build gzips the node file with level 9 and a zero mtime, and copies the index into dist/. Pins (integrity, installBytes) and script allowlists come from the release job (section 17). The repository file holds empty integrity strings, zero install sizes and empty script allowlists that the release job replaces. installBytes measures the total disk allocation of a fresh pack production dependency tree on Linux x64, with scripts disabled as in the default pack installer. It includes shared dependencies and excludes artifacts that install scripts would generate. It is separate from the runnable core CLI size gate. The release job packs all catalog registrars and companion libraries before writing pins. Local core folder overrides permit measurement before the final core artifacts exist. After rebuilding the pinned catalog, it packs the remaining core packages without changing any earlier tarball. npm run check:node-catalog regenerates the catalog and diffs it. It fails on any difference and asserts that it compared more than zero node types. Add it to test:affected path checks and to the harness registry (packages/cli/src/harness/registry.ts) for changes under node packages. Namespace prefixes stay for display and search only. findBuiltinPackForNodeType (packages/protocol/src/builtin-packs.ts) picks the first matching namespace. nodetool.control.If (core-nodes), nodetool.text.Concat (text-nodes), and nodetool.game.SpriteSheet (image-nodes) all start with nodetool, and kie appears in both the base and the kie pack. The CLI uses owners instead. 10. Catalog-only nodes (D6, F9) 10.1 Origin and availability NodeMetadataSource gains "catalog". NodeRegistry exposes: type NodeAvailability = "executable" | "catalog" | "python" | "unknown"; loadCatalog(entries: Iterable<NodeCatalogEntry>, packs?: Iterable<NodeCatalogPackMetadata>): void; availability(nodeType: string): NodeAvailability; catalogEntry(nodeType: string): NodeCatalogEntry | undefined; catalogPack(nodeType: string): NodeCatalogPackMetadata | undefined; The structural entry type lives in node-sdk, so node-catalog can use it without a dependency cycle. It carries the node type, owning pack, final metadata and declared properties. Optional pack metadata supplies install sizes and install commands to graph diagnostics. Commands are null until installs are available (A2). executable: a class is registered. has() keeps its meaning. catalog: catalog metadata exists without a class. python: metadata comes from "python-package" or "python-bridge", without a class. unknown: none of these origins applies. Generic "loaded-metadata" does not authorize Python execution. A registered class wins over catalog metadata in either loading order. Catalog property validation uses the declared properties that the generator captured, including the class validator’s dynamic-slot fallback. 10.2 Call sites that change Site Catalog behavior packages/node-sdk/src/registry.ts validateNode Validate declared properties without constructing a class. packages/node-sdk/src/graph-validation.ts Emit pack_not_installed with pack id, install size and command. Continue property, choice, credential, edge and fan-in checks from metadata. packages/execution/src/service/workflow-run.ts Return catalog property and pack errors before creating a job or connecting Python. packages/execution/src/session.ts Refuse catalog execution before the bridge, including when provider preflight is disabled. hasTsExecutor retains its class-based meaning. packages/execution/src/executor-resolver.ts Throw MissingPackError before the Python executor fallback. packages/websocket/src/plugins/websocket.ts, packages/websocket/src/http-api.ts Refuse catalog types before Python executor resolution. Catalog types do not request a bridge. packages/dsl/src/registry.ts Exclude catalog types from Python candidates and throw MissingPackError when no registry has an executable class. CLI and agent graph-validation adapters Forward availability and catalog details through registry views. Metadata-only planner relaxation cannot hide a missing pack. Metadata-free types under recognized Python namespaces still request a worker. The catalog origin takes precedence over that namespace inference. The parity suite in packages/base-nodes/tests/catalog-validation-parity.test.ts enumerates every catalog type. It compares class and catalog issue lists for empty values, defaults, out-of-range values, invalid choices, unknown fields and edge-connected properties. Graph checks run the same bags and require equal issues after removing pack_not_installed. Execution tests require a graph with invalid catalog properties to report those issues without calling ensurePythonBridge or creating a job. 11. Install layout (D1) 11.1 Install modes The CLI finds its package root (cliRoot, the directory whose package.json names @nodetool-ai/cli) from the realpath of its entry file. Mode Detection Install root npm save mode global dirname(cliRoot) is <npm root -g>/@nodetool-ai cliRoot --no-save, full desired set every time local cliRoot is <X>/node_modules/@nodetool-ai/cli and <X>/package.json lists @nodetool-ai/cli <X> --save --save-exact npx cliRoot contains a _npx path segment none unsupported dev cliRoot is not under node_modules (workspace checkout) none packs are workspaces, nothing to install unsupported .pnpm in the path, a Yarn PnP file, a Bun global directory, or a local install that the project does not list none unsupported npm root -g runs only in packs commands and in the installer, never at startup. For npx, unsupported, and dev, the missing_pack payload sets install.supported: false and a reason. For npx the text is: Packs need a global or local install. Run "npm install -g @nodetool-ai/cli" or "npm install @nodetool-ai/cli" in a project. This answers Q5. The CLI never writes into the npx cache. 11.2 Why the CLI package directory, not the global prefix These experiments ran in a scratch directory against a local test registry with three tiny packages: cli depends on core, and pack-a and pack-b declare core as a peer. Each ran on npm 10.9.8 (Node 22.22.3) and npm 11.16.0 (Node 24.18.0) on macOS. The results matched. E1. npm install -g pack-a next to a global cli installs a second copy of core at <prefix>/lib/node_modules/pack-a/node_modules/core. Each global package is its own npm project. A separate global pack install breaks module identity. E2. npm install --no-save pack-a with the CLI package directory as the working directory places pack-a beside core and resolves its peer to the existing copy. No package.json or package-lock.json changes. E3. A following npm install --no-save pack-b removes pack-a (“added 1 package, and removed 1 package”). The installer must pass every desired pack in every call. E4. A reinstall with npm install -g cli removes every pack from the CLI directory. E5. In a local project, npm install --save-exact pack-a records the pack in the project’s package.json and resolves its peer to the hoisted core. E4 means a global CLI upgrade removes packs. Section 13.3 handles restoring them. 11.3 State D14. The installed set is what exists in the tree. On start the CLI reads <root>/node_modules/@nodetool-ai/ once and keeps the entries that the catalog lists with version === coreVersion. A pack with another version is ignored and reported by packs doctor. The desired set lives in <data dir>/cli-packs.json, keyed by the realpath of the install root: { root, coreVersion, packs: string[], runtimeLibraries: string[] }. The data directory is the one the CLI resolves for its database. Writes are atomic (temporary file and rename) under the install lock. The file records what the user asked for. It never authorizes an install. 12. Missing-pack resolution (D4) 12.1 Before a run At start the CLI registers core node classes, then the registrars of installed packs, then catalog entries for every other type (section 10). Commands that need no registry skip this. function findMissingPacks(graph: GraphInput, registry: NodeRegistry): MissingPackSet; It walks every node type in the graph, including nodes inside embedded subgraphs and the graphs of referenced sub-workflows loaded through the same lookup the run uses. Each catalog type maps to its pack through owners, plus the transitive requires and runtime libraries. findMissingPacks runs inside the execution preflight that already returns structured errors (packages/execution/src/service/workflow-run.ts:795-804), so every entry point gets it: run, debug, node run, workflows run, the HTTP and WebSocket run handlers, and run_workflow inside MCP. The CLI then applies the install policy (section 13). If it installs, it re-executes (section 16). Otherwise it fails with missing_pack. A DSL file builds its graph first and goes through the same check. A DSL file that imports a pack package directly fails inside tsImport with a module-not-found error for a catalog npm name. The CLI maps it to missing_pack. Calls that pick a node type at run time cannot be scanned: CodeAct invoke_node and open_node_stream, run_node, and agent control tools. They return missing_pack as a structured tool error to the agent. The run continues. Nothing retries (F10). A lazily loaded runtime library behaves the same way through importRuntimeLibrary. 12.2 The missing_pack error (Q1) { "error": "missing_pack", "packs": [ { "id": "fal", "npm": "@nodetool-ai/fal-nodes", "version": "0.8.3", "installBytes": 12582912, "requires": [] } ], "runtimeLibraries": [], "nodeTypes": ["fal.image.FluxDev"], "install": { "supported": true, "reason": null, "command": "nodetool packs add fal" }, "restartRequired": false } Surface Form CLI run commands stderr text, the JSON above with --json, exit code 3 nodetool validate pack_not_installed issues. Exit 3 when every error is pack_not_installed, else exit 1 HTTP run endpoints status 424 with a preflight detail of kind missing_pack WebSocket run a job error message with the same payload MCP a tool result with isError: true and the payload as structured content. restartRequired: true, because a running MCP server cannot load new packs No production code in packages/cli/src assigns exit code 3. It uses 0, 1, 130, and 143. packages/cli/src/child-exit.ts:15 forwards any child status, so serve and chat can pass a 3 from a child. Callers must check the missing_pack discriminator, not the exit code alone. 13. Install authorization (D5, F1, Q3) 13.1 Sources An install proceeds only with one of these: An explicit nodetool packs add <id...> or packs add --restore. --install-missing on run, debug, node run, or workflows run. NODETOOL_AUTO_INSTALL=1 in the environment of those commands. An interactive prompt, when stdin and stderr are TTYs and CI is unset. The default answer is No. The prompt names each pack and runtime library with version and estimated size, the packages that will run install scripts, and the files that change in local mode (package.json, package-lock.json). Pack size estimates include shared dependencies and omit files generated by allowlisted rebuilds. The prompt must not present them as final disk usage. serve, mcp serve, the headless job runner, chat, and every agent tool never install. They ignore NODETOOL_AUTO_INSTALL and return missing_pack. 13.2 Enforcement inside the installer The permission gate cannot carry this. decidePermission allows every category in auto mode (packages/runtime/src/permission-gate.ts:57-59), and headlessGate() returns mode auto (:197). MCP uses it (packages/websocket/src/mcp-agent-tools.ts:367). The installer lives in packages/cli. No package that websocket, agents, or runtime depends on exports it. installPacks(request, authorization) requires an InstallAuthorization value. Only the CLI command layer constructs it, with the source, the exact pack and runtime library ids, and the script allowlist the user saw. The installer rejects a request for any id outside the authorization. The serve and mcp serve entry points set a process flag. The installer refuses to run in a process with that flag. There is no packs.install capability. The agent-facing packs capability module stays limited to sandbox packages. 13.3 Upgrades and restore After a global CLI upgrade, the new CLI finds the previous desired set in cli-packs.json and packs missing from the tree (E4). It prints one stderr line: 2 packs from 0.8.3 are not installed for 0.8.4: fal, video. Run "nodetool packs add --restore". The CLI never reinstalls without authorization from 13.1. In a run with --install-missing or NODETOOL_AUTO_INSTALL=1, it installs only what that graph needs. The desired set records intent. It never authorizes new install scripts or new runtime libraries. When a restore would run scripts from packages not in the previous version’s allowlist, the prompt lists them. 14. The installer 14.1 npm Use the npm that ships with the running Node: npm-cli.js beside process.execPath (../lib/node_modules/npm/bin/npm-cli.js on Unix, node_modules/npm/bin/npm-cli.js on Windows), run with process.execPath. Fall back to npm on PATH. On Windows, never spawn npm.cmd through a shell. D15. Minimum npm 10.9.0. Supported and tested: 10.9.x and 11.x (section 19.3). The installer refuses older versions with an upgrade command. It warns on npm 12 or later. The installer rejects --preserve-symlinks and --preserve-symlinks-main in process.execArgv and NODE_OPTIONS, because they change module identity. 14.2 Flags Forced on every call as command-line flags, which take precedence over .npmrc files and npm_config_* environment variables: --ignore-scripts --omit=dev --include=prod --include=optional --strict-peer-deps --legacy-peer-deps=false --install-strategy=hoisted --prefer-dedupe --package-lock=true --global=false --workspaces=false --force=false --dry-run=false --no-audit --no-fund --update-notifier=false Plus, by mode: global: --no-save, working directory cliRoot, and every desired pack and runtime library spec in each call (E3). local: --save --save-exact --save-prod, working directory the project root. Honored user configuration: registry, scoped registries, auth tokens, proxy, https-proxy, noproxy, ca, cafile, cert, key, strict-ssl, cache, offline, prefer-offline, the fetch-* retry and timeout settings, maxsockets, userconfig, and globalconfig. A corporate mirror works because verification uses content integrity, not the registry host. 14.3 Lock One lock per install root: <root>/node_modules/.nodetool-packs.lock, created with exclusive create (wx). It holds the pid, host name, a random token, and the start time. The owner rewrites its mtime every 5 seconds. Another installer waits up to 10 minutes and prints a line every 10 seconds. A lock with an mtime older than 60 seconds is stale and is removed. Creation, stale removal and release are serialized through a short-lived sibling .guard directory. The guard is never stolen, because deleting a replacement owner’s lock can admit concurrent installers. If its owner stops while holding the guard, installation times out and names the directory to remove after confirming that owner stopped. A user’s own concurrent npm command is not covered. packs doctor detects its effects. 14.4 Flow Detect the mode (11.1). Stop for npx, dev, and unsupported. Check the authorization (13.2) and the npm version (14.1). Expand the request with requires and runtime libraries. Compute the new desired set. Take the lock. Snapshot the tree: every installed package’s path, name, version, and integrity from node_modules/.package-lock.json. A global npm install can lack this file. Only that initial global snapshot may omit integrity (plan D25). Existing malformed or stale locks still fail. Post-install verification requires a valid lock and all release pins before scripts or imports. Preserve initial identities and compare every known integrity. Plan with npm install --dry-run and the same flags and specs (14.5). Run npm install with --ignore-scripts. Verify the tree (14.6). No install script and no pack import has run yet. Run install scripts only for allowlisted packages present in the new set: npm rebuild <name> --ignore-scripts=false --foreground-scripts. Each must match its pinned version and integrity. Load-check in a child process: nodetool packs doctor --installed --json --pack <ids>. It registers each new pack in a fresh registry, compares the owned types with the catalog, and checks that the pack resolves @nodetool-ai/node-sdk to the core’s realpath. Write cli-packs.json. Release the lock. Re-execute or print the restart notice (section 16). 14.5 Plan check npm 11 prints add, change, and remove arrays with --dry-run --json. npm 10.9.8 prints only counts in JSON. Both print add <name> <version>, remove <name> <version>, and change lines in text form. The installer parses the text lines on npm 10 and the arrays on npm 11. A unit test pins both formats. The plan is refused when: It changes or removes any package that exists before the install, except the packages a packs remove request owns exclusively. It adds a core package or a second copy of a shared-identity library. It adds an @nodetool-ai package whose version differs from the core version. npm 11 can report a same-identity replay as change. Normalize only the catalog-owned pack/companion action proven by its known release integrity and exact before/after snapshot path for npm 11 JSON. npm 10 text requires a complete same-version action and a unique pinned hoisted snapshot identity (plan D26). Protected packages and unproven actions retain change refusal. Post-install checks confirm package identities and original pins before scripts or imports. Running servers retain their loaded registry and need the restart notice to see newly installed packs. 14.6 Verification (F5) Before any install script or import: Every @nodetool-ai package in the tree has the core version, and each core package exists exactly once. The integrity of each installed pack, each allowlisted script package, and each shared-identity library equals the catalog pin. The installer reads integrity from node_modules/.package-lock.json and fails when it is missing. The tree diff against the snapshot contains only additions (14.5). Each shared-identity library resolves to one realpath from every dependent (D12). Other third-party packages float within their declared ranges, as they do for the core CLI install today (R1). 14.7 Rollback A failure in steps 7 to 10 triggers rollback: global: uninstall newly added package roots with scripts disabled, restore the original manifests and replay the previous desired set. npm can retain extraneous packages after --no-save calls. If it does, remove only new, unlinked package directories inside the install root that contain no pre-existing snapshot path, then replay the previous desired set. local: npm uninstall --save for the added specs. Then the installer compares the tree with the snapshot. When it matches, the installer releases the lock and exits 1 with the original error. When it does not match, cli-packs.json records dirty: true, and every later start prints a one-line pointer to nodetool packs doctor --repair. 15. Command surface Use one nodetool packs command for installer management and sandbox compilation, as selected by the user (Q6). Keep the existing packs compile subcommand alongside list, add, remove, doctor and why. nodetool packs list [--json] id, kind, installed, version, size, node count, required by nodetool packs add <id...> [--yes] install packs or runtime libraries with their requirements nodetool packs add --restore install the desired set recorded for this root nodetool packs add --for <workflow|file> install every pack one graph needs nodetool packs remove <id...> remove, keeping packs that another installed pack requires nodetool packs doctor [--json] [--repair] mode, npm version, versions, integrity, single copies, registration, dirty state nodetool packs why <node_type|id> owner pack, requirements, runtime libraries, installed state nodetool packs compile unchanged (sandbox packs) nodetool run|debug|node run|workflows run ... --install-missing --yes on packs add skips the prompt. The explicit command is the authorization. packs remove warns that running serve and mcp serve processes may fail on a later lazy import from the removed pack. 16. Process model and caches (D2, Q4) After an install from run, debug, node run, or workflows run, the CLI re-executes: spawnSync(process.execPath, [...process.execArgv, ...process.argv.slice(1)]) with inherited stdio and NODETOOL_PACKS_REEXEC=1. It exits with childExitCode (packages/cli/src/child-exit.ts:12). A re-executed process never installs. If packs are still missing, it exits 3. packs add from a terminal prints: Restart running "nodetool serve" and "nodetool mcp serve" processes to load fal. Because of re-exec, no cache needs a refresh path. These are the snapshots that a long-running serve or mcp serve keeps, which is why they need a restart: Cache Location DSL builtin registry promise packages/dsl/src/registry.ts:13,23-26 DSL factory references and script-export metadata packages/dsl/src/export.ts:107-114, packages/dsl/src/script-export.ts:226-235. Catalog-backed after the cut, so they stop depending on installed packs. CLI validation registry packages/cli/src/validate/index.ts:45-51 Sandbox module catalog packages/websocket/src/sandbox-catalog.ts:34-40 (refreshSandboxCatalog), packages/cli/src/sandbox-catalog.ts Provider registry (packs register local providers) packages/runtime/src/providers/provider-registry.ts:39 Mediabunny codec registration, which also memoizes a rejected promise packages/agents/src/analysis/mediabunny-runtime.ts:18-31 Optional-module fallback directory packages/config/src/optional-modules.ts:39, read per call Registry metadata and search caches invalidate on mutation (packages/node-sdk/src/registry.ts revision counter). REST and tRPC metadata endpoints read the registry per request. MCP builds a fixed capability list, not one tool per node. 17. Publishing (D7, F13) Remove NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} (.github/workflows/publish.yaml:96). Configure npm trusted publishing (OIDC) for every package. The workflow already grants id-token: write (:19). Delete the NPM_TOKEN secret. Pack each package once. Publish the tarball files, not directories, so the bytes that CI pinned are the bytes on the registry. Order: build, pack the node packs, compute pins, generate node-catalog with the pins, pack the core packages, run the packed smoke tests (core-only and with packs), publish everything under the dist-tag staging. Verify for every catalog pack and runtime library pin: `npm view @ dist.integrity` equals the pin, and a provenance attestation exists whose source repository is `nodetool-ai/nodetool` and whose workflow is `publish.yaml`. Then install the CLI from the registry with `staging` and run one pack install. Promote with npm dist-tag add <name>@<version> latest for every package, the CLI last. A failure in step 4 leaves latest unchanged. unmetDeps in scripts/publish-packages.mjs:180 blocks a package only on its declared dependencies. Once packs leave the CLI’s dependencies, step 4 is the gate that keeps the catalog from advertising a missing pack. 18. Security Threat Control An agent or a headless run installs code (F1) Install authorization separate from the permission gate (13.2). No install capability. Server and MCP processes refuse. A tampered or substituted pack tarball Integrity pin from the catalog, which ships inside the provenance-attested core tarball (14.6, 17). A malicious install script in a pack’s closure --ignore-scripts for every install. Allowlisted packages run scripts only after integrity verification, pinned to version and integrity (14.4 step 9). A stolen long-lived publish token OIDC-only publishing (17). A partial release Staging tag, verification, then promotion (17). Implicit downloads Remove the npx tsx calls (D11). The CLI never invokes npx. Privilege escalation The installer never runs sudo. EACCES on a root-owned prefix fails with instructions. Third-party pack loader confusion First-party packs carry no nodetool manifest field (D8). The third-party loader keeps its allowlist and reserved namespaces (packages/node-sdk/src/pack-loader.ts:19-37). An install changes files under a running process Additive plan check (14.5). Drifting transitive dependencies Residual risk R1. Every install prints one stderr line per installed package with name, version, and whether it ran a script. 19. Test plan 19.1 Unit Catalog generator: ownership per registrar, duplicate ownership error, deterministic output, non-JSON property error, non-zero comparison. availability() for each origin. Validation parity (section 10.2). Python routing never sees catalog types. findMissingPacks over subgraphs, sub-workflows, requires, and runtime libraries. missing_pack payload and exit code per surface. Authorization matrix: each source, each refusing process, ids outside the authorization. Mode detection on Unix and Windows paths, including _npx, .pnpm, and a workspace checkout. npm flag builder, version check, and plan parser for npm 10 text and npm 11 JSON output. Lock: exclusive create, heartbeat, stale removal. 19.2 Integration A local test registry fixture serves packuments for packed tarballs, as in the section 11.2 experiments. It runs the real installer against packed NodeTool tarballs without the public registry. Clean-core tests (6.4). Real decode with media (section 7). Re-exec: a TTY-simulated run with a missing pack installs, re-executes, and finishes with the child’s exit code. Additive plan refusal: a fixture pack that requires a different version of an installed package is refused, and the tree is unchanged. Rollback: a fixture pack whose load check fails is removed, and the tree equals the snapshot. Each new gate (size budget, catalog drift, clean-core guard) fails on deliberately invalid input before it is enforced. 19.3 npm matrix (F11) Run in CI on macOS arm64, Linux x64 (glibc), and Windows x64, each with Node 22 (npm 10.9.x) and Node 24 (npm 11.x): Case Expected Global default prefix, add one pack Pack beside core, one copy of each core package Global, add a second pack First pack stays (E3 avoided by full desired set) Global custom prefix (--prefix, nvm) Same as default Global CLI reinstall Packs gone, restore line printed (E4) Local project, add and remove package.json updated, user npm ci keeps the pack npx missing_pack with install.supported: false pnpm and Yarn installs unsupported with a message .npmrc with ignore-scripts=true, legacy-peer-deps=true, global=true, install-strategy=nested, omit=optional, package-lock=false Forced flags win. Allowlisted scripts still run. Registry mirror with custom CA Install succeeds, integrity matches Offline with a warm cache Install succeeds Offline with a cold cache Clear network error, tree unchanged Two concurrent packs add Second waits, both succeed Install killed during npm install packs doctor reports, --repair restores Root-owned global prefix EACCES message, no sudo Windows long paths and the global %APPDATA%\npm layout Mode detection and installs succeed NODE_OPTIONS=--preserve-symlinks Refused 20. Migration plan Dependency cut and measurement. No installer work. Commit the walker from 8.1 and add a report-only size check to the packed smoke. Generate @nodetool-ai/node-catalog with the drift check (section 9). DSL export and validation need it without packs. Add catalog availability and the call-site changes (section 10). Apply every cut in 6.1 and 6.2, add the loader helpers (6.3), and fix D11. Add the core-only smoke variant (6.4). It installs a CI-built CLI tarball whose manifest has the node packs removed. The published CLI keeps its packs during this step, so users see no change. Exit criteria: the core-only smoke passes, the size check stays within the measured budget in section 8.4, and npm run backend:smoke passes for the desktop bundle. Publishing hardening (can run beside step 1). OIDC-only, tarball publishing, staging tag, verification, and promotion (section 17). Pack loader. Installed-pack discovery, catalog registration, findMissingPacks, missing_pack, and exit code 3. Users can install packs with plain npm in this step. Installer and packs commands behind NODETOOL_PACKS_INSTALLER=1, with the npm matrix (19.3) in CI. Switch. Remove the node packs from the CLI manifest. Enforce the size gate. Remove the installer flag. Update docs/cli.md. 21. Risks R1. Third-party transitive dependencies of packs float within their ranges. A new upstream release can change a pack’s closure after the CLI ships. Only NodeTool packages, allowlisted script packages, and shared-identity libraries are pinned. R2. A global CLI upgrade removes packs (E4). The restore line and packs add --restore cover it, but an unattended upgrade leaves runs failing with missing_pack until someone authorizes a restore. R3. packs remove can break a running server on a later lazy import. R4. Moving a library to an optional peer removes it from the monorepo tree and the desktop bundle unless a pack depends on it (D9). npm run backend:smoke and bundle-backend.mjs must cover each moved library. R5. Making sharp optional removes thumbnails and image resizing from a core-only serve until imaging is installed. 22. Open questions Q6. The user selected nodetool packs for installer management, alongside the existing packs compile subcommand. The sandbox packs capability retains its meaning. Q8. Should llm-nodes and code-nodes stay core? They have no heavy dependencies, and the Agent and Code nodes are common in core workflows. Q9. The user authorized local installer updates to the selected project’s package.json and package-lock.json. The prompt names both files, and CLI authorization binds the root and mutation paths. Q7 is resolved: chat compaction uses the existing character-count fallback when the tokenizer runtime package is absent. Q2 from the review becomes the matrix in 19.3. Q1, Q3, Q4, and Q5 are answered in sections 12.2, 13.3, 16, and 11.1.--- # CLI Pack System Implementation Plan Source: https://docs.nodetool.ai/cli-pack-system-plan.html CLI Pack System Implementation Plan Design: CLI Pack System, called “the pack spec” below. Related: Runtime Install Design. This plan turns the five migration steps of the pack spec (section 20) into assignments. Each assignment names the assignments that block it, the files it changes, the work, and its acceptance checks. Assignments A1 to A9 are the step 1 items. A10 and later are new. Decisions D17 and later continue the decision codes of the pack spec. 1. Execution rules Code Rule A0.1 Start an assignment only when every assignment that blocks it is on main, unless the user explicitly authorizes dependent work on a fork of the prerequisite branch. Keep that work in a stacked PR. A0.2 Re-read the files in section 3 before you edit them. The line numbers in the pack spec date from its review. A0.3 Land each assignment as one commit, or as a series that is green at each commit. Run the mandatory post-change verification before each commit. A0.4 Run npm run backend:smoke for each assignment that changes a manifest, the server entry, or a lazy import. The desktop bundle flattens modules, so a cut can pass in the monorepo and fail in the artifact. A0.5 Make each new gate fail on deliberately invalid input before you enforce it (pack spec 19.2). A0.6 Do not change the published CLI until A24. Before A24, the CLI manifest keeps its node packs, and the installer stays behind NODETOOL_PACKS_INSTALLER=1. A0.7 When an assignment changes a decision, update the pack spec in the same commit. 2. Plan decisions The pack spec leaves these points open, or the code has changed since the spec was written. D17. One registry for runtime libraries The runtime library tier of the pack spec uses the ids of RUNTIME_NPM_PACKAGES (packages/protocol/src/runtime-npm-packages.ts). These rules apply: A missing library throws missingRuntimePackageError. nodetool packs add installs node packs or runtime library ids through the pinned, authorized installer. The existing nodetool runtime add remains available for compatibility. The catalog has no CatalogRuntimeLibrary entries. A pack lists runtime package ids in runtimeLibraries. No importRuntimeLibrary helper is built. Core code calls importOptionalModule and throws the runtime package error. 3f9f9c2c98 loads js-tiktoken through the tokenizer id. The runtime id registry and typed runtime errors remain shared. Node-pack management adds those same ids to its pinned installation boundary. Pack spec library Runtime package id imaging (sharp, @napi-rs/canvas) imaging (new) llama node-llama-cpp (exists) typescript typescript (new) tokenizer tokenizer (exists) tracing (OpenTelemetry) tracing (new) esbuild (6.1 row, not in the tier list) esbuild (new) @dimforge/rapier3d-compat (optional physics peer) rapier (new) game-renderer none, see D18 The desktop app and the Docker image bundle the new libraries. Their registry entries name the hosts that bundle them. The desktop Package Manager shows these entries as included and never offers an install. D18. First-party libraries belong to a pack game-renderer becomes a regular dependency of game-nodes, so the game pack carries it. transformers-js-provider belongs to the transformers-js pack in the same way. CatalogPack gains provides: string[]. This is the list of first-party npm names in the pack closure other than the pack package itself. importPack(packId, specifier?) accepts a specifier under the pack’s own name or under a provides name. nodetool game, the game capabilities and the games router return missing_pack for game. A first-party package then never needs a runtime library pin that follows the core version. D19. Each node package exports its own registrar No domain node package exports a registrar today. registerBaseNodes (packages/base-nodes/src/index.ts) registers the merged ALL_BASE_NODES list. The generator in pack spec 9.2 needs one registrar per pack. Each domain node package exports register<Name>Nodes(registry) from its src/index.ts. The domain packages are core, text, llm, code, data, document, image, audio, video, integration, automation, game, blender and manimgx. The Workflow, App and Subgraph metadata overrides move into registerCoreNodes. Preview belongs to audio and its override moves into registerAudioNodes. registerBaseNodes calls the domain registrars in a staging registry and registers the resulting classes and metadata in the existing interleaved ALL_BASE_NODES order. The desktop app sees no change. D20. The bundled server entry moves to a private workspace packages/websocket/src/server.ts imports base-nodes, the local providers and setCodeNodeAgentsModule statically (:28-30). scripts/bundle-backend.mjs bundles packages/websocket/dist/server.js. After the cut, websocket cannot import packs. The bundle still needs a static import of every pack. websocket exports startServer({ packSource }). Its main block uses the catalog pack source (D22). nodetool serve starts this main block. A new private workspace packages/server (@nodetool-ai/server, "private": true) has the entry src/main.ts. It imports @nodetool-ai/base-nodes/builtin-packs and the local providers, and it calls startServer with the bundled pack source. These consumers start @nodetool-ai/server: the bundle entry (ENTRY_POINT in scripts/bundle-backend.mjs) the Docker image, which uses the same bundle script scripts/dev-server.mjs dev:server:dist @nodetool-ai/server declares each optional peer of a core package as a regular dependency. This keeps the monorepo tree, the desktop bundle and the image complete (pack spec R4). D21. The catalog stores node metadata as sorted text The repository stores packages/node-catalog/data/nodes.ndjson, with one node per line, sorted by node_type. The package build writes dist/nodes.ndjson.gz. A gzip file in git changes completely on each node edit, so its history cannot be compressed and its diff cannot be reviewed. index.json stays as specified in the pack spec. D22. Steps 1 and 2 register every pack that resolves The pack spec cuts the static pack imports in step 1, but it builds the installed-pack loader in step 3. Between these steps, the CLI must still load the packs that its manifest installs (A0.6). registerAvailablePacks(registry, policy) is part of A2. It works in four steps: Read the catalog. Call importPack for each pack that the policy enables. Call the registrar of each pack that resolves. Load catalog entries for the node types of every other pack. With all packs installed, the result is equal to registerBuiltinPacks. With no packs installed, the result is the core nodes plus catalog entries. A15 replaces the “resolves” test with the installed set and its version check. D23. Compiler dependencies required by core packages The core-only guard permits esbuild because tsx requires it (D11). It permits TypeScript because @trpc/client and @trpc/server declare a non-optional TypeScript peer. Both tRPC packages remain core dependencies. When that peer resolves to @typescript/typescript6, its declared @typescript/old compiler alias is part of the same required closure. Removing these compiler packages would require replacing the TypeScript workflow loader or the API framework. The smoke guard verifies that these packages resolve from their core dependents and satisfy the declared dependency or peer ranges. Every other runtime library and every on-demand node pack remains forbidden. Making a core package’s library peer optional does not remove a library that another dependency still requires. D24. Install identities in the release catalog index.installPins holds exact version and integrity records for provided companions, npm runtime libraries and shared mediabunny packages. An alias records its registry name in name and its installed folder in installName. The protocol runtime table remains the only mapping from runtime ids to npm specs. Shared records have shared: true. Runtime pins also carry installBytes from fresh Linux production-tree measurements with scripts disabled. Summed estimates can overlap because each includes its own dependency tree. Release packing hashes original companion tarballs and reads exact external registry metadata. Staging verifies every record. Source generation and a new cohort clear these release-specific pins. The installer refuses missing pins. D25. Global installation baseline without a hidden lock A real npm global install has no hidden lockfile. When that file is absent in global mode, the initial snapshot records package paths, names, versions and realpaths without claiming integrity. Existing malformed or stale locks still fail. Local mode still requires its hidden lock. The dry-run plan remains additive. Post-install verification requires a valid hidden lock and every release pin before scripts or imports. Existing package identities must remain unchanged, and known initial integrities must match. This trusts the already-running CLI installation. Lock metadata does not verify pre-existing file contents. D26. Verified replay actions in npm plans npm can report change when replaying a pack at the same version. The matrix requires a second global add to preserve the first pack. Normalize this action only for a catalog-owned node pack or provided companion whose known initial integrity matches its release pin and whose version matches the core cohort. For npm 11 JSON, reported before/after names, versions and paths must match the exact snapshot identity. npm 10 text must match the complete same-version change <name> <version> => <version> action. Its name must identify exactly one snapshot entry in the expected hoisted package folder, with a matching canonical realpath. Nested or ambiguous entries remain refused. Core, shared, runtime and project packages retain strict change refusal. Unproven replay actions are refused. Scripts stay disabled. Post-install checks compare every previous path, name, version, integrity and realpath before rebuilds and imports. 3. Current seams Area Starting point Consequence Pack ids packages/protocol/src/builtin-packs.ts (BUILTIN_NODE_PACKS, 14 ids, base required) The desktop keeps these ids. The catalog adds the split ids. findBuiltinPackForNodeType picks by namespace, which is ambiguous (pack spec 9.2). Registrars packages/base-nodes/src/builtin-packs.ts (BUILTIN_PACK_REGISTRARS, registerBuiltinPacks, CLI_BUILTIN_PACK_POLICY), packages/base-nodes/src/index.ts (ALL_BASE_NODES, registerBaseNodes) D19 and D22. Keep the policy semantics: the CLI excludes kie, topaz, higgsfield and together, and local runs also exclude atlascloud. Server registry packages/websocket/src/node-registry-setup.ts:28-33 It imports registerBuiltinPacks, setBuiltinPackEnabled, applyBuiltinNodePolicy and builtinPackUnavailableReason. The pack source interface (A12) covers these four calls. applyBuiltinNodePolicy uses protocol only, so it moves to core. Server entry packages/websocket/src/server.ts:28-30, scripts/bundle-backend.mjs:42-48, scripts/dev-server.mjs:79 D20. CLI serve packages/cli/src/nodetool-main.ts:452 It spawns node ../../websocket/dist/server.js. Keep the path, and use process.execPath instead of node. CLI registry packages/cli/src/node-registry.ts:12-15, packages/cli/src/nodetool-main.ts:46, packages/cli/src/providers.ts:38-42 Static pack and provider imports. Lazy modules packages/config/src/optional-modules.ts, packages/cli/src/optional-node-root.ts, packages/cli/src/commands/runtime.ts importOptionalModule and the runtime command exist. A19 extracts the npm runner from optional-node-root.ts. Image codecs packages/runtime/src/image-codec.ts sharp is lazy already. Its error text names npm install sharp. A4.1 changes it to the imaging runtime error. Packed smoke scripts/smoke-packed-cli.mjs Packs every public workspace and installs the CLI tarball into a throwaway project with overrides. A7 and A8 extend it. Publishing .github/workflows/publish.yaml:19,83-97, scripts/publish-packages.mjs, scripts/prepare-publish.mjs Uses NODE_AUTH_TOKEN and --provenance. A13. Test selection scripts/test-affected.mjs (PATH_CHECKS), packages/cli/src/harness/registry.ts A1 adds the catalog drift check. Validation packages/node-sdk/src/registry.ts (NodeMetadataSource), packages/node-sdk/src/graph-validation.ts A3. 4. Phase 0: prerequisites A10. Land the open work and correct the pack spec Blocked by: none. Commit the two design docs. Commit the runtime command and D16 in a separate commit. Correct the stale parts of the pack spec: 6.1 and 6.2 still list cuts that are done. The done cuts are typescript (aa3b120861), js-tiktoken (3f9f9c2c98) and node-llama-cpp in cli. office-text-extractor is uncommitted. The OpenTelemetry row says six packages. Three remain (core, resources, sdk-trace-base). They also pull in sdk-trace and semantic-conventions. Q7 is answered. Q10 and the fs-safe part of the 8.4 ratchet are obsolete, because fefdcafc37 removed fs-safe. Add D17 to D22 to the pack spec, or link this plan from section 5. Acceptance: the working tree has no packs or runtime work outside a commit. The pack spec names no cut that is already done. A11. Domain registrars (D19) Blocked by: none. Files: packages/*-nodes/src/index.ts for the 14 domain packages, and packages/base-nodes/src/index.ts. Add a registrar to each domain package. It registers the lists that ALL_BASE_NODES takes from that package, in the same order. Move Workflow, App and Subgraph overrides into registerCoreNodes, and Preview into its owning registerAudioNodes. Change registerBaseNodes to call the 14 registrars and replay their metadata in the original interleaved order. Acceptance: A new test in packages/base-nodes/tests/ builds a registry with the old registerBaseNodes logic and a registry with the new logic. It requires equal node type lists and deeply equal metadata for each type. Keep the snapshot of the old logic as a fixture. Do not keep the old code. Each domain registrar registers only the types of its own package. The test fails when two registrars register one type. npm run backend:smoke passes. A12. The pack source and the server entry (D20) Blocked by: A11. Files: packages/websocket/src/server.ts, packages/websocket/src/node-registry-setup.ts, the new packages/server/, scripts/bundle-backend.mjs, scripts/dev-server.mjs, the root package.json, Dockerfile and electron/ only if a path changes. Define PackSource in websocket. It has register, setEnabled, unavailableReason and registerProviders. Move applyBuiltinNodePolicy into node-sdk. Re-export it from base-nodes for existing callers. Split server.ts into startServer(options) and a main block. Create packages/server with src/main.ts and the bundled pack source. Add the optional peers of the core packages as regular dependencies. Point the bundle entry, dev-server.mjs and dev:server:dist to the new entry. Import setCodeNodeAgentsModule from code-nodes, not base-nodes. Until A2 lands, the main block in websocket uses the bundled source through a dynamic import of @nodetool-ai/server when that private workspace resolves. The packed CLI excludes private workspaces, so it falls back to the same factory exported by base-nodes/server-pack-source. Errors within a resolved source propagate. A4.6 replaces both temporary paths with the catalog source. Acceptance: npm run backend:smoke passes. npm run dev:server serves /health. The desktop node list is unchanged. Use the A11 snapshot against GET /api/nodes/metadata. A test asserts that every optional peer of a core package is a regular dependency of @nodetool-ai/server or of a pack. The test fails when you remove sharp from @nodetool-ai/server. 5. Phase 1: dependency cut and measurement (pack spec step 1) A7. The size walker (report only) Blocked by: none. Do this first, so that each cut reports its effect. Files: new scripts/measure-cli-closure.mjs, new scripts/node-catalog/tiers.mjs, scripts/smoke-packed-cli.mjs. Write the tier list in scripts/node-catalog/tiers.mjs. It contains the core packages, the node packs and the runtime libraries. The walker, the lint rule (A2), the generator (A1) and the smoke guard (A8) all read this file. Implement the walker as pack spec 8.1 describes. Print apparent size, disk usage, the package count and the 20 largest packages. --json writes the same data. In the packed smoke, run du -sk node_modules after the install and print the result. Do not fail on it yet. Acceptance: The walker prints the current S0 and S3 scenarios. Record both numbers in pack spec 8.2, because the commits since the review changed the tree. A test in scripts/__tests__/ runs the walker on a fixture tree with known sizes and asserts the totals. The test also asserts that the walker visited more than zero packages. A1. The node catalog Blocked by: A11, A7 (for the tier list). Files: new packages/node-catalog/, new scripts/generate-node-catalog.mjs, new scripts/node-catalog/packs.mjs, packages/config/src/package-asset-registry.ts, scripts/test-affected.mjs, packages/cli/src/harness/registry.ts. A1.1 Package. Create @nodetool-ai/node-catalog. Its dependencies are config, protocol and node-sdk, and it has no pack dependency. Export these items: the 9.1 types, with D17 and D18 applied readCatalogIndex() readCatalogNodes(), which loads the nodes lazily and gunzips them packForNodeType(nodeType), which uses owners Register both data files in PACKAGE_RUNTIME_ASSETS, so the desktop bundle finds them. Add a binary loader to the asset registry if it has none. A1.2 Pack definitions. scripts/node-catalog/packs.mjs lists each pack with these fields: id npm name registrar export provides runtimeLibraries secrets The generator derives requires from the pack manifests (D10). It derives nodeCount from the registrars. A1.3 Generator. Follow pack spec 9.2. Write data/index.json and data/nodes.ndjson (D21). Write placeholder pins: integrity: "" and installBytes: 0. A13.3 replaces them at release. A1.4 Drift gate. Add npm run check:node-catalog. It regenerates the catalog and fails on a difference. It also asserts that it compared more than zero node types. Add the check in three places: to PATH_CHECKS for scripts/node-catalog/** and scripts/generate-node-catalog.mjs to the harness registry for changes in node packages to the CI quality checks Acceptance: Generator unit tests cover these cases (pack spec 19.1): ownership per registrar the duplicate ownership error deterministic output for two runs the error for a property value that JSON cannot represent the non-zero comparison The generator writes equal output on linux-x64 and darwin-arm64 (R6). check:node-catalog fails after a deliberate edit of index.json. owners maps nodetool.control.If to the core pack and nodetool.game.SpriteSheet to image. It maps every kie.* type to the pack that registers it. A2. The loader helpers Blocked by: A1. Files: packages/node-catalog/src/, packages/protocol/src/ (the runtime ids), electron/src/runtime/packages/definitions.ts, tools/oxlint/ or the lint config, packages/cli/src/commands/runtime.ts (help text). A2.1 importPack(packId, specifier?) and MissingPackError follow pack spec 6.3 and 12.2, with D18 applied. Until A14 lands, the install field reports supported: false with the reason “Pack installs are not available in this version.” A2.2 registerAvailablePacks(registry, policy) follows D22. The policy semantics stay the same as in builtinPackUnavailableReason. A2.3 A lint rule blocks static and dynamic imports of node pack names from core packages. It reads the tier list. The only allowed path is importPack. A2.4 Add the runtime ids imaging, typescript, tracing and esbuild to RUNTIME_NPM_PACKAGES, RuntimePackageId and the desktop definitions (D17). Add the bundling hosts to each entry. To keep A2 and the dependent A4 cuts independently verifiable, the import rule temporarily permits existing import sites listed in tools/oxlint/pack-boundaries/migration-baseline.json. Each entry names a file, specifier, exact AST statement and occurrence count. New imports fail, including additional occurrences in a listed file. Each A4 cut removes its entries, and unused entries fail lint. Type-only imports are erased and remain allowed. Acceptance: With all packs in the tree, registerAvailablePacks produces the same types and metadata as registerBuiltinPacks with the same policy. Test it for the CLI policy and the local policy. A test hides one pack through the module resolver. Its types become catalog, and the other packs still register. The lint rule fails on a deliberate import "@nodetool-ai/fal-nodes" in websocket. A test checks each runtime registry version against the optional peer range of the core package that loads it. A3. Catalog-only nodes Blocked by: A1. Files: packages/node-sdk/src/registry.ts, packages/node-sdk/src/graph-validation.ts, packages/execution/src/service/workflow-run.ts, packages/execution/src/executor-resolver.ts, packages/websocket/src/plugins/websocket.ts, packages/websocket/src/http-api.ts, packages/dsl/src/registry.ts. Add the "catalog" source, loadCatalog, availability and catalogEntry (pack spec 10.1). Change each call site in the 10.2 table. Acceptance: availability() returns the correct value for each origin. The parity test from 10.2 runs fixture property bags through the class validator and the catalog validator for every catalog type. It requires equal issue lists. A graph with an unloaded TypeScript node and invalid properties reports the property issues. ensurePythonBridge is never called. validate reports pack_not_installed for a catalog type. A4 and A5. The manifest cuts and the source cuts A4 is the manifest cuts (pack spec 6.1), and A5 is the source cuts (6.2). Each batch below lands one package’s A4 row with its A5 rows in one change. A manifest cut without its source cut breaks a clean install. A source cut without its manifest cut saves nothing. Batch Package A4 manifest cut A5 source cut Blocked by A4.1 runtime sharp and the three OpenTelemetry packages become optional peers Make run-trace-processor.ts load OpenTelemetry only when a trace sink is set. The image-codec.ts error uses imaging. Add loadCanvas() in runtime/src/canvas-codec.ts, which throws the imaging error. A2 A4.2 timeline, data-nodes @napi-rs/canvas in data-nodes becomes an optional peer. timeline has this already. fonts/google-fonts-node.ts and fonts/register-node.ts use loadCanvas(). A4.1 A4.3 agents, game-nodes Remove blender-nodes, game-nodes, game-renderer and video-nodes from agents. @napi-rs/canvas becomes an optional peer. game-nodes gains game-renderer (D18). The 10 static canvas imports use loadCanvas(). model3d.ts and timeline-bake.ts use importPack("blender"). timeline-preview/model3d.ts uses importPack("video", …). The game capabilities use importPack("game", …). A2, A4.1 A4.4 model-pricing Remove fal-nodes and kie-nodes. Move both unit-price catalogs and kie-pricing-api.ts into model-pricing. Change the output paths in fal-codegen and kie-codegen. Do not edit prices by hand. Refresh the fixtures with scripts/provider-codegen-check.mjs --write. none A4.5 dsl Remove the 7 packs. export.ts and script-export.ts read metadata from the catalog. registry.ts uses registerAvailablePacks and never treats catalog types as Python candidates. A2, A3 A4.6 websocket Remove the 18 packs. sharp becomes an optional peer. The main block uses registerAvailablePacks, which removes the A12 temporary path. Providers register through the pack source. The other source cuts are below this table. A2, A3, A4.4, A12 A4.7 cli Remove reliability-harness. Keep the 14 node packs and game-renderer until A24. node-registry.ts and nodetool-main.ts:46 use registerAvailablePacks. providers.ts registers providers through the pack. timeline-render.ts uses importPack("video", …). game.ts and game-smoke.ts use importPack("game", …). A2, A6 A4.8 game-runtime, sandbox-compiler @dimforge/rapier3d-compat and esbuild become optional peers. Make the esbuild import in sandbox-compiler/src/bundle.ts lazy. A missing esbuild throws the esbuild runtime error. A2 These are the other A4.6 source cuts: routes/nodes.ts uses importPack("integration"). routes/documents.ts uses importPack("document", "lib/pdf-text"). The file-watch predicates move from automation-nodes into nodes-utils, shared with websocket without adding Node path imports to the browser protocol entrypoint. routes/kie-pricing.ts reads from model-pricing. trpc/routers/models.ts returns empty model lists with a pack hint. trpc/routers/games.ts, routes/games.ts and model-download-runtime.ts use importPack. The 4 static sharp imports go through image-codec.ts. Acceptance for each batch: No src/ file of the package imports a node pack outside importPack. The A2.3 lint rule checks this. The walker reports the saving of the batch. Record the number in the commit message. npm run backend:smoke passes. Each changed lazy path has a test with the dependency hidden. The test asserts the error kind and the runtime id or pack id. A6. tsx in the CLI (D11) Blocked by: none. Add tsx to the cli dependencies. Replace the npx tsx calls at nodetool-main.ts:588 and debug/target.ts:49 with an in-process tsImport. Acceptance: No npx call remains in packages/cli/src. A source scan test asserts this. nodetool run examples/workflows/concat_text.ts works in the packed smoke. A8. The core-only smoke Blocked by: A2, A3, A4.1 to A4.8, A6. Files: scripts/smoke-packed-cli.mjs, a new test fixture for the mediabunny decode. Add --core-only. It writes the CLI tarball again with the node packs and game-renderer removed from its manifest. Then it installs that tarball with no pack tarballs available. Run checks 1 to 7 and check 9 of pack spec 6.4. For check 8, this step asserts only that the run fails and names fal. A16 adds the missing_pack JSON and exit code 3. Add the pack spec 7 decode test. Call ensureCodecs(), then decode the first audio frame and the first video frame of a checked-in fixture. Assert the sample counts and the frame dimensions. Run the core-only smoke in CI beside the existing packed smoke. Acceptance: The core-only smoke passes in CI. The check 9 guard fails when a pack tarball is deliberately added. A9. The size gate Blocked by: A8. Measure du -sk node_modules of the core-only install on linux-x64 in CI (D13). Set the hard gate to that value plus 5 %. Write the value in pack spec 8.4. Make the core-only smoke fail above the gate. Acceptance: the gate fails on a deliberately oversized fixture before you enforce it. Step 1 exit criteria The core-only smoke passes, with the size gate enforced. npm run backend:smoke passes. The published CLI is unchanged. The full packed smoke still passes. 6. Phase 2: publishing (pack spec step 2) A13. Publishing hardening Blocked by: none for A13.1 and A13.2. A1 for A13.3. This phase can run beside phase 1. Files: .github/workflows/publish.yaml, scripts/publish-packages.mjs, scripts/prepare-publish.mjs, new scripts/release/pin-catalog.mjs, new scripts/release/verify-staging.mjs. A13.1 OIDC. Remove NODE_AUTH_TOKEN (publish.yaml:96). An npm organization owner must configure trusted publishing for each package on npmjs.com and then delete the NPM_TOKEN secret. The plan cannot do this step. A13.2 Tarballs. Pack each package once and publish the tarball files. A13.3 Pins. Use this order: Pack every catalog registrar package, including core registrars, and its first-party companion packages. Preserve these original tarballs. Compute the integrity and installBytes of each pack on linux-x64. Measure a fresh production install with scripts disabled, matching the default pack installer policy. Include its production dependency tree. Before final core tarballs exist, use local core folder overrides with --install-links, which respects each package’s files field. Write the pins into node-catalog/data/index.json. Rebuild the pinned catalog artifact, then pack the remaining core packages. Do not repack the earlier registrar or companion artifacts. A13.4 Staging. Publish everything under the dist-tag staging. Then verify these points: For each pin, npm view <name>@<version> dist.integrity equals the pin. A provenance attestation exists with the source nodetool-ai/nodetool and the workflow publish.yaml. Before A22, the release job installs one pack beside the CLI with plain npm. After A22, it uses packs add. A13.5 Promotion. Run npm dist-tag add <name>@<version> latest for each package, with the CLI last. A verification failure leaves latest unchanged. Acceptance: Unit tests cover the pin writer, and the verifier with a fake npm view. A dry run of the workflow on a fork or a test scope reaches the promotion step. The verifier fails on a deliberately wrong pin. 7. Phase 3: pack loader (pack spec step 3) A14. Install mode and installed set Blocked by: A2. Files: new packages/cli/src/packs/mode.ts, new packages/cli/src/packs/installed.ts. Detect the install mode from the realpath of the CLI entry (pack spec 11.1). Do not run npm root -g at startup. Read the installed set (pack spec 11.3, D14). Ignore a pack whose version differs from the core version, and record it for packs doctor. Fill the install field of MissingPackError for each mode. Use the npx text from 11.1. Acceptance: the unit tests of pack spec 19.1 for mode detection pass on Unix and Windows paths. They cover _npx, .pnpm, Yarn PnP, the Bun global directory and a workspace checkout. A15. The installed pack source Blocked by: A14, A4.6, A4.7. Replace the “resolves” test in registerAvailablePacks with the installed set. Use the same source in the CLI commands and in the websocket main block (nodetool serve). Acceptance: A pack with a wrong version stays catalog-only. packs doctor reports the wrong version. Before A22, a unit test of the report function covers this. A16. missing_pack on every surface Blocked by: A15, and runtime install step 2 for the shared exit code. Files: packages/execution/src/service/workflow-run.ts, the CLI run commands, packages/websocket run handlers, the MCP tool result path. Add findMissingPacks (pack spec 12.1). It covers subgraphs, sub-workflows, requires and runtime libraries. Call it in the execution preflight. Map the error to each surface in the 12.2 table: exit code 3 in the CLI, and the JSON with --json status 424 in HTTP a job error in WebSocket isError with restartRequired: true in MCP Make validate exit with 3 when every error is pack_not_installed. Add check 8 of pack spec 6.4 to the core-only smoke. Runtime install step 2 also adds exit code 3, for missing runtime packages. Build one mapping from error to exit code in the CLI. Make both errors use it. Acceptance: One test for each surface asserts the payload and the status or the exit code. A graph whose sub-workflow uses fal reports fal. A17. Node types that a run picks at run time Blocked by: A16. CodeAct invoke_node, open_node_stream, run_node and agent control tools return missing_pack as a structured tool error. The run continues, and nothing retries (pack spec 12.1). Acceptance: a test for each tool calls a catalog-only type and asserts the structured error. The run continues after the error. 8. Phase 4: installer (pack spec step 4) The installer stays behind NODETOOL_PACKS_INSTALLER=1 until A24. A18. The install lock Blocked by: none. If runtime install step 1 has landed, reuse its lock. Build one lock implementation for both installers (pack spec 14.3). It has exclusive create, a token, a heartbeat every 5 seconds, a 10-minute wait and stale removal after 60 seconds. packs uses <root>/node_modules/.nodetool-packs.lock. The runtime installer uses its own path inside its target. Acceptance: the unit tests of pack spec 19.1 for the lock pass. A test with two processes shows that the second process waits. A19. The npm runner Blocked by: none. Files: packages/cli/src/optional-node-root.ts, new packages/cli/src/npm-runner.ts. Extract the npm call from optional-node-root.ts into npm-runner.ts. Then runtime add and packs add use one runner. Locate npm-cli.js beside process.execPath (pack spec 14.1). Check npm 10.9.0 or later (D15). Refuse --preserve-symlinks. Build the forced flags of pack spec 14.2 for each mode. Acceptance: unit tests cover the flag builder and the version check. The runtime tests still pass. A20. The install flow Blocked by: A14, A18, A19, A13.3 (for real pins), Q9 (for local mode). Files: new packages/cli/src/packs/installer.ts. Implement pack spec 14.4 steps 1 to 11: the snapshot the dry-run plan, which parses npm 10 text and npm 11 JSON (14.5) the install with --ignore-scripts the verification (14.6) the allowlisted rebuilds the load check in a child process the write of cli-packs.json Implement rollback (pack spec 14.7) and the dirty flag. Acceptance: The plan parser tests pin both npm output formats. The pack spec 19.2 integration tests pass against a local test registry fixture: additive plan refusal, rollback, and verification failure on a wrong integrity. A21. Install authorization Blocked by: A20. Only the CLI command layer constructs InstallAuthorization (pack spec 13.2). The installer rejects any id outside the authorization. serve and mcp serve set a process flag. The installer refuses to run when the flag is set. Add the interactive prompt (13.1). Its default answer is No. Acceptance: The authorization matrix test of pack spec 19.1 covers each source, each refusing process, and ids outside the authorization. No module that websocket, agents or runtime imports exports the installer. A source scan test asserts this. A22. The commands and the re-exec Blocked by: A21. Q6 is resolved: the user selected nodetool packs. Add list, add, add --restore, add --for, remove, doctor and why (pack spec 15). Add --install-missing to run, debug, node run and workflows run, and support NODETOOL_AUTO_INSTALL=1. Re-execute after an install (pack spec 16), and print the restart notice. Print the restore line after an upgrade (pack spec 13.3). Acceptance: a TTY-simulated run with a missing pack does these steps: It installs the pack. It re-executes. It ends with the exit code of the child. A second re-executed process never installs. A23. The npm matrix Blocked by: A22. Add a CI workflow for the pack spec 19.3 matrix: macOS arm64, Linux x64 (glibc) and Windows x64 Node 22 with npm 10.9 and Node 24 with npm 11 Run it on changes under packages/cli/src/packs/, on changes to npm-runner.ts, and every night. Acceptance: every case in the 19.3 table passes on each platform. 9. Phase 5: switch (pack spec step 5) A24. Remove the node packs from the CLI Blocked by: every assignment above. Remove the 14 node packs and game-renderer from packages/cli/package.json. Make the full packed smoke install packs through packs add. Remove NODETOOL_PACKS_INSTALLER. Update docs/cli.md and docs/node-packs.md. Add a release note about packs add --restore. Acceptance: A clean npm install -g @nodetool-ai/cli@<staging> is within the size gate. packs add fal followed by a fal workflow run succeeds. 10. Dependency order Assignment Blocked by A10, A11, A7, A6, A4.4, A13.1, A13.2, A18, A19 none A12 A11 A1 A11, A7 A2, A3 A1 A4.1, A4.8 A2 A4.2, A4.3 A4.1 (A4.3 also A2) A4.5 A2, A3 A4.6 A2, A3, A4.4, A12 A4.7 A2, A6 A8 A2, A3, A4.1 to A4.8, A6 A9 A8 A13.3 to A13.5 A1 A14 A2 A15 A14, A4.6, A4.7 A16 A15, runtime install step 2 A17 A16 A20 A14, A18, A19, A13.3, Q9 A21 A20 A22 A21, Q6 A23 A22 A24 all The critical path is A11, A1, A2, the A4 and A5 batches, A8 and A9. Run A13 and the runtime install steps 1 and 2 beside it. 11. Open questions Q6. The user selected nodetool packs for installer management. The existing nodetool packs compile subcommand remains available. Q8. This plan keeps llm-nodes and code-nodes in core, as the pack spec tier table does. Q9. The user authorized local installer updates to the selected project’s package.json and package-lock.json. CLI authorization still binds the root and mutation paths and names those files in the prompt. 12. Risks of this plan R6. A registrar can register different types on different platforms. The catalog then differs between a darwin checkout and the linux release job. A1 compares the generator output on both platforms. R7. D19 changes the order of registration. A metadata override can then apply to a different class. The A11 snapshot test catches this. R8. From A2 to A15, two registration paths exist: bundled and catalog. The A2 equality test keeps them equal while both exist. R9. A dependency from node-catalog on a pack would make test:affected select the CLI and server suites for every node change. The generator therefore lives in scripts/, and node-catalog depends on no pack.--- # NodeTool CLI Source: https://docs.nodetool.ai/cli.html The nodetool CLI is the TypeScript command-line interface for the NodeTool platform. It manages servers, workflows, jobs, assets, and secrets. Run nodetool --help to see the top-level command list. Every sub-command exposes its own --help flag with detailed usage. Installation Install globally from npm to get the nodetool and nodetool-chat commands: npm install -g @nodetool-ai/cli Or run a single command without installing: npx --package=@nodetool-ai/cli nodetool --help npx --package=@nodetool-ai/cli nodetool-chat Requires Node.js 22 or later (engines in packages/cli/package.json, and the repository pins 24 in .nvmrc). Check with node --version, and install via nvm if needed. Getting Help nodetool --help — list all top-level commands. nodetool <command> --help — show command-specific options (e.g. nodetool serve --help). nodetool <group> --help — list sub-commands for grouped tooling (e.g. nodetool workflows --help). nodetool help [command] — the same output as --help, as a command (e.g. nodetool help serve). With no argument it lists the top-level commands. Global Options These flags work on any nodetool command and control OpenTelemetry tracing: --trace-file <path> — append every LLM/agent/workflow span to <path> as JSONL (analyzer-friendly). --trace-stdout [format] — stream spans to stdout: pretty (default) or json. --no-trace-stdout — disable stdout span output (overrides NODETOOL_TRACE_STDOUT). --version — print the CLI version. nodetool --trace-file trace.jsonl run workflow.ts nodetool --trace-stdout pretty workflows run <id> Core Commands nodetool info Display the CLI version, Node.js version, platform, which provider API keys are set (configured or not set, never the value), and the ENV, LOG_LEVEL and PORT environment values. Options: --json — output as JSON. Example: nodetool info nodetool info --json nodetool serve Starts the TypeScript WebSocket + HTTP backend server. This serves the REST API, WebSocket endpoints, and static assets. Options: --host (default 127.0.0.1) — bind address (use 0.0.0.0 for all interfaces). --port (default 7777) — listen port. Examples: # Start the server on the default port nodetool serve # Bind to all interfaces on a custom port nodetool serve --host 0.0.0.0 --port 8080 serve passes the two flags to the server as HOST and PORT, overwriting whatever those variables already held — use the flags, not the environment, to move the server. nodetool workflows run <workflow_id_or_file> Executes a workflow by ID (from the local database), JSON file, or TypeScript DSL file. Arguments: <workflow_id_or_file> — workflow ID, path to a .json workflow file, or path to a .ts DSL file. Options: --params <json> — JSON string of workflow parameters. --json — output result as JSON. --asset-output-mode <mode> — how media outputs are materialized: native (default), raw, data_uri, workspace, storage_url, or temp_url. --workspace <dir> — workspace directory for workspace mode output (default ./nodetool-output). --trigger-event <json> — wake a trigger node the way the scheduler or webhook adapter would, for example '{"node_id":"watch","payload":{"path":"/tmp/a.csv","event":"created"}}'. Without it a webhook, file-watch or manual trigger has no event to emit and the run stalls until its timeout. --supervise and its bounds — see Supervised runs. Examples: # Run workflow by ID nodetool workflows run workflow_abc123 # Run workflow from JSON file nodetool workflows run ./my_workflow.json # Run workflow from TypeScript DSL nodetool workflows run ./my_workflow.ts # Run with parameters as JSON nodetool workflows run workflow_abc123 --params '{"input": "hello"}' # JSON output for automation nodetool workflows run ./my_workflow.json --json --json never expands a large binary output into the JSON text. An image, audio, or video payload over 64 KiB is written to nodetool-output/<job_id>/payload-N.<ext> and appears as {"$file": "…", "bytes": N, "mimeType": "…"} in its place; the path is reported on stderr. Image nodes emit raw RGBA as their in-flight format, so a raw ref is PNG-encoded on the way out. nodetool run and nodetool node run do the same, spilling into nodetool-output/. nodetool workflows export-dsl <workflow_id_or_file> Exports a workflow as a TypeScript DSL file. Arguments: <workflow_id_or_file> — workflow ID or path to a .json workflow file. Options: --api-url <url> — fetch the workflow from a remote server instead of the local database (default NODETOOL_API_URL). -o, --output <file> — write to file instead of stdout. Examples: # Print DSL to stdout nodetool workflows export-dsl workflow_abc123 # Write to file nodetool workflows export-dsl workflow_abc123 -o workflow.ts # Export from JSON file nodetool workflows export-dsl ./my_workflow.json nodetool workflows export-script <workflow_id_or_file> Converts a workflow into a JavaScript script that calls each node in dependency order, one awaited call per node. The editor’s workflow menu and the workflows.exportScript tRPC call produce the same script. Options: -t, --target <target> — sandbox (default) writes a body for a JS script document or a Code node, calling nodes through @nodetool-ai/sandbox-nodetool/flow. node writes a standalone .mjs that calls nodes in process through @nodetool-ai/dsl/flow, takes the workflow inputs as a JSON argument, reads provider keys from the environment, and prints the outputs as JSON. -o, --output <file> — write to file instead of stdout. A node a script cannot express (a node that runs once per streamed item, a control edge, a live input stream) is listed on stderr, and the script throws before running any node. A graph with a cycle, such as a Loop, is refused. nodetool workflows export-script workflow_abc123 --target node -o flow.mjs npm install @nodetool-ai/dsl node flow.mjs '{"prompt": "a red fox"}' nodetool run <dsl-file> Shorthand for running a TypeScript DSL workflow file directly. Options: --json — output results as JSON. --supervise and its bounds — see Supervised runs. Examples: nodetool run workflow.ts nodetool run workflow.ts --json Supervised runs --supervise puts an agent on the failure path: when a node invocation throws after its own error handling is exhausted, the agent sees the failure and answers with one verdict — retry, repair the output, skip the item, or fail. Without the flag nothing changes: no escalation is constructed and the run is the run it was before. Available on nodetool run, nodetool workflows run, and nodetool debug (server surface). Options: --supervise — supervise this run. Off unless passed. --max-decisions <n> — decisions allowed in the run (default 10). --max-retries <n> — retries per node invocation (default 2). --supervisor-cost-cap <usd> — ceiling on supervisor spend (default 0.50), enforced by reservation before each model turn, not after. --supervisor-model <provider/model> — who supervises (default anthropic/claude-sonnet-4-6, or NODETOOL_SUPERVISOR_MODEL). The leading segment must be a registered provider; the rest is the model id, slashes and all (openrouter/openai/gpt-5.4-mini). Passing a bound without --supervise is an error rather than a silent unsupervised run. Output. Each decision prints a ⛨ line as it happens, and the run ends with a supervised summary: ⛨ fetch-item [3] skipped — HTTP 404 (agent, $0.0041) ⛨ supervised: 2 skipped, 1 retried, 3 decisions, +$0.0200 With --json, the decisions are in interventions — alongside the outputs in workflows run, and under {results, interventions} in nodetool run, whose bare results shape is left alone for unsupervised runs. nodetool debug puts them in server.summary.interventions with a server.supervised rollup. Each record carries the escalation the agent saw (node, item lineage, redacted inputs, allowed actions), the verdict, who decided it (agent, sticky, bounds, default, kernel), and its cost. Cost. Supervisor spend lands in the same ledger nodetool costs reads, one row per billable decision, attributed to the run and tagged supervisor in node_type — so supervision is separable from the workflow’s own spend: nodetool costs list --limit 20 # supervisor rows show node_type=supervisor Bounds are the guarantee. Every supervisor failure (timeout, an unparseable verdict, an exhausted budget, a cancelled run) resolves as fail, which is what would have happened without it. What each verdict means and why retry is opt-in per node: workflow-supervisor-design.md. Database Migrations nodetool db migrate Applies NodeTool migrations to a PostgreSQL/Supabase database. For Supabase, use the direct connection URL from Settings → Database (port 5432), not the transaction pooler URL. Options: --direct-url <url> — Supabase/PostgreSQL direct connection URL. --database-url <url> — connection URL; defaults to DIRECT_URL or DATABASE_URL. --target <version> — stop after a specific migration version. --dry-run — show pending migrations without applying them. --skip-checksums — skip checksum validation. --json — output as JSON. Examples: DIRECT_URL="postgresql://postgres:[password]@db.[project].supabase.co:5432/postgres" \ nodetool db migrate nodetool db status --direct-url "$DIRECT_URL" nodetool db migrate --direct-url "$DIRECT_URL" --dry-run Other migration commands: nodetool db status --direct-url "$DIRECT_URL" nodetool db baseline --direct-url "$DIRECT_URL" # for existing DBs nodetool db rollback --direct-url "$DIRECT_URL" --steps 1 status, baseline, and rollback take --direct-url, --database-url, and --json like migrate. baseline also takes --force to clear and recreate the migration tracking records. rollback --steps <n> defaults to 1. Chat nodetool chat Starts fullscreen terminal chat with a scrollable transcript, multiline composer, tool details, task progress, and approval prompts. The previous terminal screen is restored on exit. Piped input keeps its line-based interface. Options: -p, --provider <provider> — LLM provider (e.g., anthropic, openai, ollama). -m, --model <model> — model ID. -a, --agent — deprecated, no-op. Every chat session runs the unified agent loop; this flag has no effect. -u, --url <url> — WebSocket server URL (default: uses a local provider). With piped input, the command exits 1 when the server reports an error for any turn. -w, --workspace <path> — workspace directory for file operations (default: current directory). --no-read-only-search — disable the read-only run_search fan-out primitive (on by default). --tools <tools> — comma-separated tool names that narrow the belt. Without it, a local session offers the same belt as a server chat turn. --permission-mode <default|auto|plan> — how tool calls are gated (see Permission mode). Interactive chat defaults to default, with approval prompts. Piped input defaults to auto. --resume [id] — resume a saved conversation in this workspace. Without an ID, resume the most recently saved session. --cost-cap <usd> — ceiling on provider spend for one turn; 0 lifts it. Default: the NODETOOL_AGENT_TURN_COST_CAP_USD setting. --timeout <s> — wall-clock bound on one turn, in seconds; 0 leaves it no time at all. Default: the NODETOOL_AGENT_TURN_DEADLINE_MS setting. Both flags override two of the five NODETOOL_AGENT_* settings; the other three — concurrency, total turns, unpriced-token ceiling — come from the settings alone. The budget is one object per turn, shared by every loop the turn starts, so a ceiling bounds the turn rather than each loop separately. A turn a ceiling refuses prints the reason: [stopped] turn budget of $5 reached piped, or a Stopped: line in the TUI. The terminal also displays the local turn’s provider spend. Connected servers manage their own run budgets. Fullscreen chat reserves stdout for its screen. Use --trace-file <path> for tracing and --no-trace-stdout when stdout tracing is enabled in the environment. Enter sends a message. Alt+Enter, Shift+Enter in supported terminals, or Ctrl+J inserts a newline. Pasted lines stay in the composer. Up/Down recalls prompts, Tab completes commands. Page Up/Page Down and the mouse wheel scroll the transcript. Drag across transcript text to select and copy it to the clipboard on release. Ctrl+G returns to the latest output. Ctrl+O toggles tool details and edit diffs. Escape stops a running turn or dismisses input. Ctrl+C stops a turn, clears a draft, or saves and quits when idle with an empty composer. Use /sessions and /resume <id> to continue a conversation, /export <path.md> to save a transcript, and /mode to change permissions. /clear clears the screen while retaining context. /new starts a fresh conversation. Local /compact [instructions] retains a summary. Remote context is managed by the server. /help lists all commands. Sessions are saved under ~/.nodetool/chat-sessions/ after turns and on exit. They include conversation context and provider continuation state. The session picker filters by workspace and server. Model and provider selections persist in ~/.nodetool/chat-settings.json. Remote /model <id> accepts a server model ID without requiring a local API key. Examples: # Start interactive chat nodetool chat # Chat with a specific provider and model nodetool chat --provider anthropic --model claude-sonnet-5 # Connect to a running server nodetool chat --url ws://localhost:7777/ws Workflow Management nodetool workflows Manage workflows. Reads the local database by default; --api-url targets a remote server. Subcommands: list and get (both take --api-url <url> and --json, and list takes --limit <n>, default 100), run, export-dsl, export-script, export-example, export-bundle, import-bundle, migrate-code-inputs # List all workflows nodetool workflows list nodetool workflows list --api-url http://localhost:7777 --json # Get a workflow by ID nodetool workflows get <workflow_id> # Run a workflow (see above for full options) nodetool workflows run <workflow_id_or_file> # Export as a TypeScript DSL file (see above) nodetool workflows export-dsl <workflow_id_or_file> nodetool workflows export-example <workflow_id_or_file> Export a workflow as a shipped template: materialize its referenced assets into the package’s constant asset directory (rewriting refs to package://<pkg>/<file>) and write the example JSON. Options: --api-url <url> — fetch the workflow and asset bytes from a remote server instead of the local database (default NODETOOL_API_URL). --package <name> — owning package (default nodetool-base). --assets-dir <dir> — root holding <package>/<file> constant assets (default: the monorepo base-nodes assets directory). --examples-dir <dir> — directory the example JSON is written under, as <dir>/<package>/<name>.json. -o, --output <file> — write the example JSON to this exact path. --include-remote — also materialize http(s) and local-file refs. nodetool workflows export-example <workflow_id> nodetool workflows export-example <id> --package nodetool-base nodetool workflows export-example workflow.json -o example.json nodetool workflows export-bundle <workflow_id_or_file...> Export one or more workflows as a portable .nodetool bundle (a zip containing the graphs plus the bytes of every asset they reference), sharable as a single file. Options: --api-url <url> — fetch the workflows and asset bytes from a remote server instead of the local database (default NODETOOL_API_URL). -o, --output <file> — output path (default <name>.nodetool). --include-remote — also embed http(s) and local-file refs. nodetool workflows export-bundle <id> [<id2> ...] -o my-pack.nodetool nodetool workflows import-bundle <bundle_file> Import a .nodetool bundle into the local library: store its assets and create the workflows with refs rewritten to the imported assets. Options: --json — print the created workflows as JSON. nodetool workflows import-bundle my-pack.nodetool nodetool workflows migrate-code-inputs A one-time repair for saved nodetool.code.Code bodies. A Code node’s declared inputs used to arrive as globals of their own name; they now arrive on one inputs object, so a body written the old way throws a ReferenceError on its first input read. This walks the saved workflows and rewrites name to inputs.name for every name the node can read — its declared slots, its inline dynamic properties, and any handle an edge feeds it. The rewrite is done on the AST, so a name inside a string, a comment, an object key, or a local binding is left alone. It is safe to re-run: a body already reading inputs.* has nothing to rewrite. Options: --dry-run — report what would change and write nothing. --user-id <id> — migrate this user’s workflows instead of the local user’s (1). --json — print the report as JSON rather than the per-node lines. # See what would change first nodetool workflows migrate-code-inputs --dry-run # Then do it nodetool workflows migrate-code-inputs Each rewritten node prints one line naming the inputs it moved, followed by a count: 4 Code node(s) in 3 workflow(s) were rewritten (11 scanned across 7 workflows). A body that fails to parse is counted under a trailing N failed. line and left untouched, so one bad node does not stop the pass. nodetool validate <workflow_id_or_file> Check a workflow against the node registry without running it: unknown node types, missing required properties, dangling or mis-typed edges, model properties naming a provider or model id that does not exist, and nodetool.code.Code bodies that never return or leave an output unset. On a workflow-id target, where the secret store is reachable, it also warns about declared credentials the install cannot resolve. It finishes in well under a second, which makes it the cheap pre-flight before a run that costs money. The target is a workflow id, a workflow JSON file, or a TypeScript DSL file. File targets need no database. Options: --json — print the full validation report as JSON. --warnings-as-errors — exit non-zero on warnings, not just errors. Examples: nodetool validate <workflow_id> nodetool validate workflow.json nodetool validate workflow.ts --json nodetool validate <workflow_id> --warnings-as-errors A clean graph prints a one-line verdict and exits 0: ✅ Workflow is valid — 3 node(s), 2 edge(s). A broken one names the node and the rule it broke, then exits 1: ❌ Workflow has 1 error(s). error Unknown node type "nodetool.text.NoSuchNode" (not in the registry; Python-only nodes are not validated statically) [nodetool.text.NoSuchNode 1] (unknown_node) Codes: unknown_node — node type is not in the registry nodetool debug <workflow_id_or_file> Run a workflow end to end and collect everything it emitted into a self-contained bundle — every message, log line, node input/output, and error — then print a verdict. Reach for it when validate passes but the run still does the wrong thing. The headless server run is on by default. The expensive surfaces are opt-in: --browser starts Playwright and Chromium, --trace adds OpenTelemetry spans, and --stages screenshots the canvas at each run stage. Options: --no-server — skip the headless server surface. --browser — also run the workflow in a real browser (Playwright). --trace — capture an OpenTelemetry trace of the server run (timing, tokens, cost). --stages — screenshot the canvas at every browser run stage (implies --browser). --params <json> — JSON params keyed by input-node name. --out <dir> — bundle output directory (default nodetool-debug/<id>-<timestamp>). --timeout <ms> — per-surface run timeout. --json — print the full DebugReport as JSON to stdout. --watch — re-run on file change and print a diff of the verdict (file targets only). --supervise and its four bounds — see Supervised runs above. Examples: # Server surface only nodetool debug <workflow_id> nodetool debug workflow.json --params '{"prompt":"hi"}' # Opt into the expensive surfaces nodetool debug <workflow_id> --trace nodetool debug <workflow_id> --browser --stages # Tight edit→verify loop on a file target nodetool debug workflow.json --watch The bundle holds report.json and report.md, the resolved workflow.json, and server/messages.jsonl. --trace adds server/trace.jsonl; --browser adds browser/record.json, a canvas screenshot, and browser/console-errors.log. nodetool node run <node_type> Instantiate one node, hand it a property bag, and print what it emits — no workflow, no graph. Use it when debug says a run went wrong and you want to know whether one node is the reason. Options: --props <json> — JSON object of property values keyed by @prop name (default {}). --no-secrets — skip secret and asset resolution, so the run touches no database. --json — print the full run result as JSON. Examples: nodetool node run nodetool.text.Concat --props '{"a":"hi ","b":"there"}' nodetool node run nodetool.text.Concat --props '{"a":"hi ","b":"there"}' --no-secrets nodetool node run nodetool.text.Concat --props '{"a":"hi ","b":"there"}' --json The verdict names the node type, its title, and how long it took, then lists every record it emitted: ✅ nodetool.text.Concat (Concat) — 0ms Emitted 1 record(s): {"output":"hi there"} Mini App Management nodetool apps Move mini apps between installs. Reads and writes the local database directly — no running server. Subcommands: list, export-bundle, export-project, import-bundle # List applications with their operation count nodetool apps list nodetool apps list --json list prints id, name, operations, and updated_at. nodetool apps export-bundle <application_id> Export an application as a portable bundle: the app document plus the full graph of every workflow its operations bind, in a single JSON file. Inside the file an operation’s workflowId is a bundle-local key, not a real workflow id. Options: -o, --output <file> — output path (default <name>.app.json, with unsafe characters replaced by _). --released — export the released snapshot and the graphs it pinned, not the draft. Fails when the app has no released version. The bundle path goes to stdout and the summary line to stderr, so $(nodetool apps export-bundle <id>) captures the path alone. An operation binding a workflow that no longer exists is left unresolved in the bundle and reported as a warning. nodetool apps export-bundle <application_id> nodetool apps export-bundle <application_id> -o my.app.json --released nodetool apps export-project <application_id|bundle_file> Export an application as a standalone JavaScript project that runs without NodeTool: a Vite + React UI and a Node server that runs the app’s workflows and JS scripts in process, with no database. Takes an application id from the local database or an exported .app.json bundle. See @nodetool-ai/app-export for what the project contains. Options: -o, --output <dir> — output folder (default ./<app-name>). --released — export the released snapshot and the graphs it pinned, not the draft. --force — write into an output folder that is not empty. .env.example lists the credentials the graphs and scripts need. When a graph leaves a model for the person to pick, it also lists every provider key, commented out. Sample inputs referenced as package:// are copied under assets/. What the exported app cannot do (resource documents, editor-only widgets) is printed as warnings and listed in the project’s README. nodetool apps export-project <application_id> -o ./my-app nodetool apps export-project my.app.json cd my-app && npm install && cp .env.example .env && npm run dev nodetool apps import-bundle <bundle_file> Import an application bundle into the local library: create its workflows and the app, with operations rewired to the new ids. Options: --project <id> — project to create the app in (default default). --json — output the created application as JSON. A bundled workflow carrying a sourceId gets a row id derived from it, so importing two bundles that ship the same workflow — two example apps binding one template — reuses the existing row instead of duplicating it. Without a sourceId every import creates fresh workflows. nodetool apps import-bundle my.app.json nodetool apps import-bundle my.app.json --project my-project --json nodetool app Where nodetool apps moves finished apps around, nodetool app runs one and builds one. Both subcommands are headless — no browser, no editor — and both write a debug bundle under nodetool-debug/. nodetool app debug <application_id_or_file> Run a mini app the way the web runtime would: validate every widget binding against the workflows it binds, seed input defaults, apply params, replay a scripted interaction sequence, execute the workflows on the kernel, fold the streamed messages into the app’s reactive values, and report what each widget ends up showing. The target is an application id (read from the applications table), an ApplicationBundle JSON file (self-contained — operations reference bundle keys, so it runs without a database), or a workflow id/file carrying a legacy app_doc. Options: --params <json> — reactive values applied before interactions, keyed by input name. A resource:<binding id> key seeds that collection with an array of items. --interact <json> — scripted steps: set, change, click, run (by operation id), cancel, seedResource. --no-run — static wiring check only; never execute a workflow. --out <dir> — bundle directory (default nodetool-debug/app-<id>-<timestamp>). --timeout <ms> — per-run timeout. --json — print the full AppDebugReport to stdout. Examples: # Static wiring check — no provider calls, no workflow execution nodetool app debug my.app.json --no-run nodetool app debug <application_id> --params '{"prompt":"hi"}' nodetool app debug <application_id> --interact \ '[{"set":{"key":"prompt","value":"hi"}},{"run":"draft"}]' The verdict catches what a workflow-only run cannot: a widget bound to a token the workflow has no input, output, node, or variable for, one naming an operation the document never declares, an unknown widget type, a display widget that never received a value, an app no interaction ever ran, an operation that overran its timeout, and a click or change on a widget its own visibleWhen / disabledWhen hides or disables — that step fails and names the condition. Every declared operation runs, not just the first, and state is keyed per operation. The report lists what it did not simulate under notSimulated: layout, styling, focus, and scroll, since nothing renders a DOM; the stored resource collections, since a run reads the seeded in-memory provider rather than the database; and reactive subgraph runs, where the browser reruns one input’s downstream subgraph and the harness runs the whole workflow. nodetool app build <prompt_or_spec_file> Turn a prompt, or a hand-written spec.json, into a verified ApplicationBundle. Six stages run in order: spec pins what the app must do, plan builds one workflow per operation, author places and wires the widgets through the real ui_app_* tool contract, check validates the wiring, run replays every interaction on the kernel, and judge asks a model whether each interaction achieved what was asked. Everything still wrong at the end of a pass becomes one complaint, and the next round edits the document rather than rebuilding it. Plan, author, and judge each call a model, and run executes real workflows, so a build spends money and takes minutes. Options: -p, --provider <name> / -m, --model <id> — the builder’s provider and model. --judge-model <provider/model> — model that judges each interaction (env NODETOOL_APP_JUDGE_MODEL). Defaults to a configured model other than the builder’s, because a model grading its own work is the weakest reviewer available. --workflow <id> — pin an existing workflow instead of planning one (repeatable, in operation order). --max-repairs <n> — repair rounds after the first pass (default 3). --cost-cap <usd> — ceiling on build spend (default 2). --timeout <ms> — wall clock for the whole build (default 600000). --out <dir> — bundle directory (default nodetool-debug/app-build-<slug>-<timestamp>). --json — print the full BuildReport to stdout. --no-judge — skip the judge stage. The verdict’s notSimulated then says nothing scored the app. --watch — re-build on spec-file change and print a diff of the verdict (spec-file targets only). Every save spends money. --supervise, --max-decisions <n>, --max-retries <n>, --supervisor-cost-cap <usd>, --supervisor-model <provider/model> — the same flags described under Supervised runs. They apply to the Run stage, whose interactions execute on the kernel; buildApp itself is never supervised. Examples: nodetool app build "an app that drafts a note from a prompt" \ -p anthropic -m claude-sonnet-5 nodetool app build spec.json -p openai -m gpt-5.4-mini --json # Bind an existing workflow instead of planning one nodetool app build "..." -p anthropic -m claude-sonnet-5 --workflow <id> # Structural check only, one repair round, tight cap nodetool app build spec.json -p anthropic -m claude-sonnet-5 \ --no-judge --max-repairs 1 --cost-cap 1.00 The bundle holds report.json, report.md, spec.json, interactions/<name>/run-N.messages.jsonl per replayed run, and — only for a green build — app.bundle.json, the deliverable. The loop fails closed: an exhausted budget, an issue that reappears after being fixed, or a cancelled signal ends the build as failed with the reason named, and there is no bundle behind a failed verdict. Exit code is 0 only when verdict.ok. Editor Documents Timelines, sketches, and JS scripts each get the same three commands: validate checks the document without rendering or running it, debug replays a scripted edit session against it, and versions reads and writes its snapshot history. All of them take a JSON file or a row id — a path that exists on disk wins over an id, and file targets need no database. nodetool timeline Subcommands: validate, debug, render, score, versions nodetool timeline validate <timeline_id_or_file> Check a timeline sequence without rendering it: clips on tracks the document does not have, fields a schema round trip would strip, animation presets that do not exist, and timings nothing can render. The target is a timeline JSON file — a bare TimelineDocument or anything carrying one under document, so a timeline.get response works as-is — or a timeline_sequences row id. Options: --json — print the full TimelineValidation as JSON. --warnings-as-errors — exit non-zero on warnings, not just errors. nodetool timeline validate sequence.json nodetool timeline validate <timeline_id> --json nodetool timeline validate <timeline_id> --warnings-as-errors ✅ 0 error(s), 0 warning(s) Every finding carries a stable code. An error is a document no reading of which produces the scene its author described; a warning is a picture that renders and is not the one that was written — usually because this build could not read something a newer one wrote (forward compatibility), or because the authored motion did not fit the clip. Code Severity What it caught schema_invalid error The document does not parse; every structural check is skipped duplicate_id error Two tracks, clips or markers share an id clip_track_missing error A clip sits on a track the document does not declare negative_timing error A clip or marker starts before the origin, or a clip lasts zero fade_exceeds_duration error The in and out audio fades overlap in_out_points_invalid error The source span is negative or empty speed_multiplier_invalid error A non-positive playback rate unknown_animation_preset error A preset this build does not ship; nothing animates custom_animation_invalid error Baked curves the one gate refuses; re-bake from the script animation_link_source_missing error A clip links animation to a clip the document lacks animation_style_invalid error An animation uses glyph tracks without a stagger unit text_path_invalid error A text clip’s path cannot be read shape_path_invalid error A path shape this build cannot draw, so it draws nothing midi_clip_off_midi_track error A clip carrying notes sits on a non-midi track, so it makes no sound non_midi_clip_on_midi_track error A picture or audio clip sits on a midi track, so it is neither heard nor seen midi_clip_retimed error A midi clip carries speed or time remap, which retime samples a midi clip does not have midi_notes_invalid error A midi clip holds a note that fails its checks parent_cycle error A parentId chain loops, so the group cannot be resolved matte_source_missing error A matte names a clip the document lacks, or itself — the layer draws unmatted, showing everything the matte was hiding time_remap_not_monotonic error timeRemap keyframe t repeats or goes backwards (sourceMs may descend — that is a reverse) field_stripped warning A field the schema drops, lost on the next save in_out_duration_mismatch warning Source span and timeline duration disagree at the clip’s rate transition_exceeds_duration warning The cut is longer than the clip carrying it unknown_transition warning A transition type or direction this build cannot draw; it cross-fades left unknown_easing warning An easing outside the grammar; it eases linearly generated_matte_stale warning A generatedMatte cut from an asset the clip no longer plays, or from less source than its window now shows; regenerate it unknown_effect warning A clip effect type this build cannot apply; the layer draws ungraded mask_path_invalid warning A mask kind or path d that cannot rasterize; the layer draws unmasked unknown_shape_kind warning A shapeStyle.kind this build has no geometry for; the shape draws nothing font_not_portable warning A font family NodeTool does not ship; every host resolves it against its own installed fonts, so the editor preview and the render can differ matte_source_invalid warning A clip is matted by an adjustment clip, which has no pixels, so the layer draws unmatted crop_degenerate warning A crop whose left and right, or top and bottom, sum to 1 or more, so the clip draws its whole source bake_stale warning A bake rendered from a style the clip no longer has, so the live 3D layer draws instead reframe_stale warning Automatic framing that no longer matches the active source, so rendering falls back to centred framing source_curve_outside_window warning Animation keyframes cut from source outside the span the clip plays adjustment_no_effects, adjustment_treats_nothing, adjustment_group_empty warning An adjustment clip with no enabled effect, with no clip beneath it in its window, or in a group where nothing else overlaps it adjustment_field_ignored warning An adjustment clip carries fields it never draws layout_on_non_group, unknown_layout_display, flex_item_without_flex_parent warning A layout on a clip that is not a group, a layout.display other than flex, or a flexItem whose parent is not a flex container. Each is ignored midi_clip_silent warning A midi clip with no notes, or none starting inside its window midi_tempo_missing warning The document has midi but no tempo, so notes read at the default BPM text_backing_unproven warning Text over picture with no scrim, plate or outline, so nothing decides whether it is readable parent_missing, parent_not_group warning A parentId naming nothing, or naming a clip that is not a group; the child renders unparented layer_cap_exceeded warning More video clips overlap at an instant than the compositor draws animation_exceeds_clip warning The window does not fit the clip after its delay, so the motion is clamped — or never runs stagger_compressed warning The stagger span did not fit, so the per-unit offset was shrunk and the units overlap more than authored. At 0ms every unit moves in sync effect_animate_ignored warning animate on a generator or stylize mode whose shader never reads the clock, so it stays still, or on a stylize gradientWipe, whose map it switches effect_washes_out warning A color effect’s brightness (an offset, not a multiplier) drives the picture to white or black, or its contrast removes nearly all of it animation_link_overrides warning A link sets its channel outright and discards the clip’s own animation there, or a wiggle discards the clip’s placement typewriter_not_staggered warning A typewriter is stored with a plain durationMs and no stagger, so the whole text appears at once after that time, with no caret animation_holds_rest_before_window warning A custom out curve starts away from the channel’s rest value after clip start, so the frames before its window show the rest value and then jump replace_curves_overlap warning Two animations drive one absolute channel (positionX/Y, anchorX/Y, trimStart/End) at the same time; the last in document order wins and the other is discarded text_illegible warning Type under 2.5% of frame height, or under a 3:1 contrast ratio against its own background plate or a full-frame shape clip behind it clips_overlap, clip_shorter_than_frame, caption_out_of_range, binding_incomplete, duplicate_track_index, transcript_clip_missing, link_partner_missing warning Structural smells that still render text_illegible refuses to guess: a colour notation it cannot read, a translucent plate, gradient-filled type, or a backdrop it cannot prove sits behind the text ends the contrast check rather than producing a finding. nodetool timeline debug <timeline_id_or_file> Run the same check, then execute each scripted step against the headless ui_timeline_* bridge and validate the document the session left behind. A step names a tool with or without the ui_timeline_ prefix. A failing step is recorded and the script continues, so one bad target does not hide everything after it. Rendering, playback, decode, and generation are not simulated; the report lists that under notSimulated. Options: --interact <json> — the scripted steps, e.g. [{"tool":"add_track","input":{"type":"audio"}}]. --out <dir> — bundle output directory (default nodetool-debug/timeline-<id>-<timestamp>). --json — print the full TimelineDebugReport as JSON to stdout. nodetool timeline debug sequence.json \ --interact '[{"tool":"add_track","input":{"type":"audio","name":"Music"}}]' ✓ ui_timeline_add_track ✅ Timeline is sound — 1 interaction(s) ran clean. timeline: 2 track(s), 1 clip(s), 4000ms @ 30fps session: 1 step(s), 0 failed The bundle holds report.json, report.md, and timeline.json (the input document). The command exits 0 only when the verdict is ok. nodetool timeline render <timeline_id_or_file> Render a timeline through the GPU compositor that the Render Timeline node uses. The target is a timeline JSON file (a bare document, or a wrapper with document, fps, width, height and durationMs, such as an example bundle) or a timeline_sequences row id. Clip media resolves through the local asset store. Video exports mix audio-track clips, MIDI clips, and embedded video audio. Stills, contact sheets, PNG sequences, and video frame selections carry no audio. Use --stills to check a render frame by frame. The Canvas 2D frame preview is a second renderer, so only this command shows the exported pixels. Options: --out <path> — output file, or an output directory with loose --stills or --sheet (default nodetool-render/<name>.<ext>, <name>-frames/, or <name>-sheet.png). --format <format> — mp4 (default), webm, mov or png_sequence. --frames <spec> — render only these frames: indices and inclusive ranges, e.g. 0-89 or 175,290,380-390. Use coverage:<file> to read the Frame column of a Markdown table in a builder file’s leading comment. Non-integer rows are skipped and reported. --stills — write each selected frame as frame_<index>.png instead of encoding a video. --sheet [columns] — write one PNG contact sheet instead of loose stills. It labels each frame and defaults to three columns. Select at most 60 frames. --sheet implies still-image output. --only <clips> — render clips selected by comma-separated names or ids. Selected groups include their children; other selections retain their ancestor groups and matte sources. --scale <n> — render at this fraction of the sequence size. The layout is the same as at full size. --bitrate <bps> — video bitrate in bits per second. --json — print the result as JSON. nodetool timeline render sequence.json --out ad.mp4 --bitrate 12000000 nodetool timeline render <timeline_id> --stills --frames 175,290,376-384 --scale 0.5 nodetool timeline render sequence.json --frames coverage:build-sequence.ts --sheet --out review.png nodetool timeline render sequence.json --only title,logo --frames 60,120 --sheet 2 The command needs ffmpeg for video output and a WebGPU adapter. On a headless machine, install a Vulkan ICD such as lavapipe (headless WebGPU setup). The first frame compiles the GPU pipelines, so progress starts after a delay. nodetool timeline score <timeline_id_or_file> Score a timeline document’s craft against the shipped showcase examples (cadence, kite, prism, serein, tidewater, voltra — packages/base-nodes/nodetool/examples/timelines/*.timeline.json, excluding the footage-based t-minus-30) — a deterministic, no-LLM scorecard, not a judge call. It measures motion density (animations and keyframes per visible second), breadth (distinct animated properties, effect types, showcase-only feature flags), and structural craft (style tracks, text animators, authored transitions, scene count), each against the reference examples’ median, plus a fixed penalty per distinct showcase-tier warning code present (the showcase_* codes that the showcase validation tier adds, which timeline validate does not run) — four findings of the same code cost as much as one, so no single noisy check can dominate the score; instance counts stay visible in showcaseWarnings and showcaseWarningCodeCounts. The target is a timeline JSON file or a timeline_sequences row id, same resolution as timeline validate. Options: --json — print the full score result (every metric, its reference median and credit, the reference band, and the underlying validation) as JSON. --min-score <n> — exit non-zero when the score is below this threshold. Without it the command always exits 0 — the score is information, not a gate. --sheet <out.png> — also render a GPU contact sheet of 12 evenly spaced frames, through the same renderer timeline render --sheet uses. nodetool timeline score <timeline_id> nodetool timeline score sequence.json --json nodetool timeline score <timeline_id> --min-score 70 nodetool timeline score <timeline_id> --sheet review-sheet.png See Harness Reference for the score formula and scripts/motion-craft-eval.mjs, the eval loop that runs an agent brief and scores what it produced. nodetool timeline versions Read and write a sequence’s snapshot history against the local database: manual saves, the autosaves timeline.update writes at most every five minutes, and the pre-restore snapshot that makes a restore undoable. Subcommands: list, show, create, restore, delete. All five take --json. list <timeline_id> — newest first; --save-type <manual|autosave|restore>, --limit <n> (default 100). show <timeline_id> <version> — one version’s metadata and the document it stored. create <timeline_id> — snapshot the sequence as it stands now; --name <name> labels it. restore <timeline_id> <version> — snapshot the current state first, write the old document back, then re-validate it. delete <timeline_id> <version> — -y, --yes skips the confirmation prompt. nodetool timeline versions list <timeline_id> --save-type manual --limit 10 nodetool timeline versions show <timeline_id> 3 --json nodetool timeline versions create <timeline_id> --name "before the recut" nodetool timeline versions restore <timeline_id> 3 nodetool timeline versions delete <timeline_id> 3 --yes An old document is restored against today’s schema, so what it used to pass is not what it passes now: a restore whose document no longer validates prints the issues and exits non-zero. nodetool sketch Subcommands: validate, debug, versions nodetool sketch validate <sketch_id_or_file> Check a sketch (image document) without opening an editor: a duplicate layer id, an activeLayerId or binding pointing at a layer the document lacks, opacity or a blend mode no compositor ships, a binding with no workflow or prompt behind it, and fields a schema round trip would strip. The target is a {sketch, layerBindings} JSON file — or anything carrying one, so a sketch.get response works as-is — or an image_documents row id. Layer bitmaps stay opaque to the check. Options: --json — print the full SketchValidation as JSON. --warnings-as-errors — exit non-zero on warnings, not just errors. nodetool sketch validate sketch.json nodetool sketch validate <image_document_id> --json nodetool sketch debug <sketch_id_or_file> Run the same check, then execute each scripted step against the headless ui_sketch_* bridge (the prefix is optional) and validate the document the session left behind. Pixels, painting, rendering, generation, and asset I/O are not simulated; the report lists that under notSimulated. Options: --interact <json> — the scripted steps, e.g. [{"tool":"add_layer","input":{"name":"Glow"}}]. --out <dir> — bundle output directory (default nodetool-debug/sketch-<id>-<timestamp>). --json — print the full SketchDebugReport as JSON to stdout. nodetool sketch debug sketch.json \ --interact '[{"tool":"add_layer","input":{"name":"Shadow"}}, {"tool":"set_layer_props","input":{"target":"Shadow","opacity":0.4,"blendMode":"multiply"}}]' ✓ ui_sketch_add_layer ✓ ui_sketch_set_layer_props ✅ Sketch is sound — 2 interaction(s) ran clean. sketch: 2 layer(s), 0 binding(s), 1024x768 session: 2 step(s), 0 failed The bundle holds report.json, report.md, and sketch.json. nodetool sketch versions The same five subcommands as timeline versions, against an image document’s history. The per-layer generation takes in the editor are a different thing: those record one generated image on one layer, these snapshot the whole document. nodetool sketch versions list <image_document_id> --save-type manual --limit 10 nodetool sketch versions show <image_document_id> 3 --json nodetool sketch versions create <image_document_id> --name "before the repaint" nodetool sketch versions restore <image_document_id> 3 nodetool sketch versions delete <image_document_id> 3 --yes nodetool jsscript A JS script is a named, versioned script document: a body plus declared ports, secrets, a timeout, and saved test cases. The target of every subcommand is a script JSON file (a bare JsScriptDocument or anything carrying one under document) or a js_scripts row id. Subcommands: validate, run, test, debug, versions nodetool jsscript validate <script_id_or_file> Check the body’s syntax, its imports against the installed pack catalog, undefined names, undeclared inputs.* reads, outputs no emit/output call reaches, duplicate or non-identifier port names, and tests naming ports the script does not declare. A body that declares outputs and returns them instead of emitting them is an error — a script has no legacy return contract. Zero saved tests, and a declared secret this install lacks, are warnings. Options: --json — print the full JsScriptValidation as JSON. --warnings-as-errors — exit non-zero on warnings, not just errors. nodetool jsscript validate script.json nodetool jsscript validate <js_script_id> --warnings-as-errors nodetool jsscript run <script_id_or_file> Execute the body once in the QuickJS sandbox and print its outputs, streamed emits, logs, and error. A body that reads its inputs with stream is fed with --input-streams instead of --inputs; a staged handle the script does not declare is refused. Options: --inputs <json> — input values, e.g. '{"a":1}'. --input-streams <json> — items staged per handle, e.g. '{"nums":[1,2,3]}'. --json — print the run result as JSON. nodetool jsscript run script.json --inputs '{"numbers":[1,2,3]}' nodetool jsscript run script.json --input-streams '{"numbers":[1,2,3]}' ✅ ran in 1032ms outputs: {"total":6} streamed: [{"name":"running","value":1},{"name":"running","value":3},{"name":"running","value":6}] nodetool jsscript test <script_id_or_file> Run the document’s own saved test cases and grade each one. Exits non-zero when any case fails, which makes it the regression check after an edit. Options: --json — print the grade report as JSON. nodetool jsscript test script.json nodetool jsscript test <js_script_id> --json ✅ 2 passed, 0 failed ✓ sums three numbers ✓ sums an empty list to zero nodetool jsscript debug <script_id_or_file> Replay each scripted step against the headless ui_jsscript_* bridge (the prefix is optional) and validate the document the session left behind. The editor, persistence of a debug session, and secret values are not simulated. Options: --interact <json> — the scripted steps, e.g. [{"tool":"set_code","input":{"code":"..."}}]. --out <dir> — bundle output directory (default nodetool-debug/jsscript-<id>-<timestamp>). --json — print the full JsScriptDebugReport as JSON to stdout. nodetool jsscript debug script.json \ --interact '[{"tool":"set_code","input":{"code":"await output(\"n\", 1);"}}]' nodetool jsscript versions The same five subcommands as timeline versions, against a script’s history. nodetool jsscript versions list <js_script_id> --limit 10 nodetool jsscript versions create <js_script_id> --name "before the rewrite" nodetool jsscript versions restore <js_script_id> 3 nodetool game Validate, replay, capture, and export a native game document from a JSON file. Simulation advances exactly the requested number of 60 Hz ticks. The input recording is a JSON array of { "pressed": ["right"] } frames, one per tick. nodetool game validate game.json nodetool game simulate game.json --ticks 120 --inputs inputs.json --expect-score 1 --expect-win nodetool game simulate game.json --ticks 120 --inputs inputs.json --assertions assertions.json --verify-replay nodetool game capture game.json --ticks 120 --inputs inputs.json --out frame.png nodetool game capture game.json --ticks 120 --out frame.png --backend webgpu nodetool game build game.json --out game-build --assets-dir game-assets nodetool game build game.json --out game-desktop --assets-dir game-assets --target tauri --name "Cave Run" --compile nodetool game smoke game-build --json Subcommands and options: game validate <game_file> — --json. game simulate <game_file> — --ticks <count> (required), --seed <integer> (default 1), --inputs <file>, --assets-dir <directory> (prepared 3D collider files named <full-asset-id>.json), --expect-score <score>, --expect-win, --assertions <file>, --verify-replay, --saves <file>, --json. game capture <game_file> — --ticks <count> and --out <file> (both required), --seed <integer>, --inputs <file>, --scale <factor> (default 1), --backend <canvas2d|webgpu|webgl2>, --assets-dir <directory>, --json. A schema version 3 (3D) document defaults to webgl2. A 2D document accepts canvas2d (default) or webgpu. game build <game_file> — --out <directory> (required), --assets-dir <directory>, --target <web|tauri> (default web), --json. --target tauri writes a Tauri desktop app project: the web player in dist/ and a native shell in src-tauri/. It accepts --name <name>, --identifier <reverse-dns>, --app-version <MAJOR.MINOR.PATCH>, --icon <png|jpeg|webp>, and --compile, which runs cargo build --release and reports the executable path. Run npm install && npm run build in the project for installers. game smoke <directory> — --json. Serves an exported build in headless Chromium with outside network access blocked, renders 300 scripted frames, and checks tick progress. Console errors, page errors, failed requests, and stalled ticks return a nonzero exit status. Install Chromium with npx playwright install chromium. Schema version 2 adds sprite visualAnimation tracks and scene backgrounds. Tracks use scene or spawn age in ticks and animate rotation, scale, opacity, and tint. Backgrounds use per-axis parallax (0 screen-fixed, 1 world-fixed), scroll rate in world units per second, and none, repeat, or mirror tiling. game capture derives both at the selected tick, so a resumed simulation and an arbitrary-tick capture use the same positions. Scene lighting adds ambient color and up to 32 colored point lights. Each point has world position, radius, intensity, and falloff. Set sprite.unlit for emissive art. Lighting applies to the world before effects and leaves HUD text unchanged. game capture uses Canvas2D by default. It reports omitted optional GPU effects and fails if an effect is required. Choose --backend webgpu to run bloom, brightness/contrast, and LUT grading through the GPU modules used by the browser. The command fails when a required effect cannot run. The JSON report includes effect diagnostics. GPU capture requires the local Dawn WebGPU adapter. A 3D capture checks the frame against the document’s performance.budgets, with player defaults for any budget left out. The JSON report has budget.limits and budget.overruns, and stats.culledEntities counts the entities that distance culling hid. Text mode prints each overrun to stderr. --assertions checks selected ticks. Tick 0 is the initial state; tick 1 is after the first input frame. Each listed tick may check sceneId, entity x/y/active, and the complete ordered events array for that tick. Omitted fields are not checked. A missing entity fails the assertion. The default position tolerance is 1e-9; set tolerance at the top level to change it. Assertions beyond --ticks and duplicate tick entries are errors. { "tolerance": 0.000001, "ticks": [ { "tick": 0, "sceneId": "room", "entities": [{ "id": "player", "x": 0, "y": 0, "active": true }], "events": [] }, { "tick": 1, "events": [] } ] } --verify-replay saves the midpoint state, resumes it in a second session, and compares snapshots and ordered events after each remaining tick. The JSON report includes replay.resumeTick, replay.verified, and the first divergent tick and path when it fails. This check excludes script execution timing. Script save and load commands use in-memory storage during simulate. --saves <file> seeds it from {progress?, settings?}, each mapping a slot name to its data object. A load result reaches scripts on the next tick’s input frame, and the replay session steps the same frame, so --verify-replay covers loads. When a script saved or loaded, the JSON report lists each slot’s data under saves with any storage errors. --assets-dir supplies media files named <full-asset-id>.<extension> for capture and export. The build copies media into a content-addressed folder and includes a standalone browser player. Built-in sample art needs no media files. Schema version 2 permits up to eight ordered renderEffects. brightnessContrast sets brightness from -1 to 1 and contrast from 0 to 4. bloom sets threshold, softness, radius in pixels, and intensity. lut references an opaque image asset laid out as a packed 2D cube with dimensions size² × size. hudEffectOrder chooses beforeEffects or afterEffects; bloom defaults to HUD after effects, while a legacy one-effect document keeps its earlier output. Each effect has required: a missing GPU capability fails for required effects and reports omitted optional effects. Canvas capture reports the same omission policy; GPU capture applies the chain. HUD labels may refer to an embedded font asset by fontId. Capture loads the font bytes before drawing; a missing required font fails and an optional font reports its fallback. To compare browser and headless text metrics, run NODETOOL_BROWSER_FONT_TEST=1 npm run test --workspace=packages/game-renderer -- fonts.test.ts. Schema version 2 scenes can set music: {"assetId":"music","volume":0.5, "fadeInTicks":30,"fadeOutTicks":30} to loop an installed audio asset. Audio sources on entities remain one-shot effects. Their volume defaults to 1, and triggered effects can overlap up to the browser player’s 32-voice cap. Music position is saved as a scene start tick; loading seeks within the decoded loop. Audio starts after a browser gesture. The CLI simulation and image capture record logical audio events and state without producing audio samples. Project Files nodetool projects List projects and move whole projects between installs as .nodetool-project files. Runs against the local database, asset store, and workspaces. The server serves the same format at GET /api/projects/:id/export and POST /api/projects/import, so a file from either side imports on the other. Subcommands: list [--archived] [--json] — projects, newest first. export <project_id> [-o <file>] [--json] — write the project to a file. The default name is <project name>.nodetool-project. import <file> [--name <name>] [--json] — create a new project from a file and print its id. A project file holds every row filed under the project: workflows and their versions, storyboards, scripts, timelines, sketches, mini apps, JavaScript scripts, and games, each with its version history. It also holds chats with their messages, assets with their stored bytes, and every file in the project’s workspaces. Jobs, run traces, and spend records stay behind. Import never overwrites. It creates a new project and gives every row a new id, then rewrites each reference to an exported id: id columns, document JSON, asset:// locators, workspace file paths, and JSON files in workspaces. Imported workflows are private. Workspaces become managed folders, because the exporting machine’s paths mean nothing on another one. The Personal project imports as an ordinary named project. Examples: nodetool projects list nodetool projects export 6f1c2e... -o launch-spot.nodetool-project nodetool projects import launch-spot.nodetool-project --name "Launch spot (copy)" Job Management nodetool jobs Query job status and results. Reads the local database by default. Subcommands: list, get Options: --api-url <url> — query a remote server instead of the local database. --workflow-id <id> — filter by workflow ID (for list). --limit <n> — max results (default: 100). --json — output as JSON. --json returns the same declared row shape whether the command read the local database or a server. The local read used to dump every column, including a job’s whole graph and logs. Examples: # List all jobs nodetool jobs list # Filter by workflow nodetool jobs list --workflow-id workflow_abc123 # Get a specific job nodetool jobs get <job_id> nodetool runs Inspect durable app runs, workflow jobs and chat turns. Reads the local database as the local account by default. --api-url <url> or NODETOOL_API_URL selects a server. Every query stays within the caller’s account. Local reads apply pending SQLite migrations before opening the database. Run IDs accept their full value or an exact 12-character resource prefix. Ambiguous prefixes fail. Trace and span IDs always retain their full OTel values. show, trace and logs also accept a workflow job or chat message ID when it identifies a recorded run. Command Output and filters runs list Newest runs. --kind app\|workflow\|chat, --app-id, --instance-id, --workflow-id, --thread-id, --status running\|completed\|failed\|cancelled, --origin ui\|agent\|cli\|debug\|public, --since <ISO time>, --until <ISO time>, --limit (default 20, maximum 100), --cursor runs show <id> Record and bounded summary: failed span and root path, provider costs, slow spans, span counts, generations, documents and content flags. --include-content adds capped app run inputs and outputs runs trace <id> Span tree. The table has a detail column: <node id> (<node type>) on node spans, <provider>/<model> on LLM spans. --depth (default 4, maximum 64), --focus <full span ID>, --name, --errors-only, --limit (default 100, maximum 500) runs logs <id> Events, oldest first. With --include-content the table adds message and arguments columns. Without it, the table ends with a hint that messages are excluded. --level, --source, --span <full span ID>, --since-ms, --until-ms, --limit (default 100, maximum 500), --cursor runs tail <id> Follow until terminal. --cursor (default 0), --limit (default 100, maximum 500), --poll-interval <ms> (default 250, range 10–5000), --timeout <seconds> Each command accepts --json. Single reads return the shared runs service’s JSON result. Lists and logs return an opaque next_cursor for the next page. Content is excluded by default. show --include-content adds the app record’s capped inputs and outputs. trace --include-content requires --focus. logs --include-content and tail --include-content return capped owner content. Results report expired, absent, visitor or suppressed content and trace truncation or incompleteness. tail --json emits NDJSON, one object per line. Its frame types are resnapshot, span, event, reconnecting, terminal, interrupted, gap and error. The store retains each span’s latest complete snapshot, so a resnapshot frame warns that intermediate updates may have been coalesced. Replace cached spans by their full snapshot. Event frames are deduplicated by span ID and stable event ID during the process. After a connection loss, tail retries from its durable cursor. Save the final cursor to resume in another process. A terminal run with all spans ended drains its remaining snapshots and returns immediately. If final spans are still arriving, tail waits until they end or until one second passes without a new cursor, with a five-second maximum drain. An unfinished trace produces a gap frame and a terminal frame with trace_settled: false. Reader limits appear on resnapshot frames. Deduplication memory is bounded by the store’s span and event caps. A dedup_limit gap warns if later records exceed those bounds. Ctrl-C stops following, removes the signal handler and exits 130. It does not cancel the run. A timeout or read error exits 1. Following a failed run to its terminal record exits 0. nodetool runs list --kind app --status failed --json nodetool runs show <run_id> --json nodetool runs trace <run_id> --focus <span_id> --include-content --json nodetool runs logs <run_id> --level error --include-content --json nodetool runs tail <run_id> --cursor 42 --json Asset Management nodetool assets Manage uploaded files and workflow assets. Reads the local database by default. Subcommands: list, get Options: --api-url <url> — query a remote server instead of the local database. --query <q> — search query (for list). --content-type <type> — filter by content type (for list). --limit <n> — max results (default: 100). --json — output as JSON. Examples: # List assets nodetool assets list # Search assets nodetool assets list --query "landscape" # Get a specific asset nodetool assets get <asset_id> Storage Maintenance nodetool storage migrate-keys Move asset objects under their owner’s prefix, so an object’s key is <userId>/<assetId>.<ext> and the owner is the leading path segment — the boundary a Supabase RLS policy or an S3 bucket policy can enforce on the object itself. Run it once when upgrading an S3 or Supabase install that still holds objects written under the older flat layout. The local file backend needs no migration: it falls back to the flat key on a miss. Options: --dry-run — report what would move without writing anything. --user-id <id> — migrate only this user’s objects. --json — output the report as JSON. Examples: # See what would move first nodetool storage migrate-keys --dry-run # Move them nodetool storage migrate-keys # One user, machine-readable nodetool storage migrate-keys --user-id 1 --json A dry run reports the tally and writes nothing: scanned 0, would move 0, already migrated 0, absent 0, failed 0 Dry run — nothing was written. Re-run without --dry-run. Vector Collections nodetool collections Manage the RAG vector-store collections behind semantic search. Runs in-process against the default vector provider — sqlite-vec unless NODETOOL_VECTOR_PROVIDER points elsewhere — so no server has to be running. Subcommands: list, get, create, delete, query, index Options: --embedding-model <model> / --embedding-provider <provider> — record the embedding model and provider on a new collection (for create). -n, --n-results <n> — number of matches to return (for query, default 10). -y, --yes — skip the confirmation prompt (for delete). --json — output as JSON. Available on every subcommand. Examples: # Create a collection and record what embeds it nodetool collections create my_docs --embedding-model text-embedding-3-small # Chunk and index files, with the same splitter the server uses nodetool collections index my_docs notes.md report.txt # Semantic search nodetool collections query my_docs "how does the runner cancel a job" -n 5 # Inspect and remove nodetool collections list nodetool collections get my_docs nodetool collections delete my_docs --yes Re-indexing the same file replaces its chunks rather than adding a second copy, so an index run is safe to repeat after a document changes. Spend Tracking nodetool costs Report what NodeTool has spent on LLM and provider calls. Every call writes a row carrying its cost and token counts; these subcommands aggregate them straight from the local database, so no server has to be running. Reach for it to answer “what did that run cost” after the fact, or to find which model is eating a budget. Subcommands: summary, list, by-provider, by-model Options: --provider <name> — filter by provider (for list and by-model). --model <id> — filter by model (for list). --limit <n> — max results (for list, default 50). --json — output as JSON. Available on every subcommand. Examples: # Overall spend plus per-provider and per-model breakdowns nodetool costs summary # Recent calls, most recent first nodetool costs list --limit 20 nodetool costs list --provider anthropic # Grouped totals nodetool costs by-provider nodetool costs by-model --provider openai summary prints the overall total and both breakdowns: Overall key │ value ──────────────┼───────── total_cost │ $0.0000 total_tokens │ 0 calls │ 0 By provider (no results) By model (no results) Supervised runs and nodetool app build write to the same ledger, tagged supervisor and app-build in node_type, so costs list shows what the harnesses spent alongside the workflow’s own calls. nodetool generations Inspect the record of every media generation — image, video, audio, 3D. One row per provider call, opened before the call as running and closed with its outcome (completed, failed, cancelled, or interrupted when a restart orphaned it), its cost, the provider’s request id, and the asset ids it produced. Like costs, it reads the local database, so no server has to be running. Where costs aggregates LLM spend, this answers the per-call questions: what a render cost, which asset it wrote, and whether it is still going. Subcommands: list, get, await, cancel, reconcile, sweep, provider-list, provider-get Options: --status <status> — pending, running, recovering, completed, failed, cancelled, needs_attention, or interrupted (for list). --provider <name> / --capability <name> — filter by provider, or by capability such as text_to_video (for list). --thread-id <id> / --job-id <id> — only what a chat thread or a workflow run asked for (for list). --since <iso> — only generations created at or after this time (for list). --limit <n> — max results (for list, default 50). --timeout <seconds> — how long await waits before giving up (default 300). --provider <name> is required, with --model <id>, for provider-list and provider-get (see below). --json — output as JSON. Available on every subcommand. Examples: # What ran, newest first nodetool generations list nodetool generations list --status running nodetool generations list --provider fal --capability text_to_video nodetool generations list --since 2026-09-01T00:00:00Z --limit 20 # One row in full — cost, assets, origin, reconcile state nodetool generations get <generation_id> --json # Block until it settles, then act on the exit code nodetool generations await <generation_id> --timeout 600 # Close a running row, or ask the provider what it actually billed nodetool generations cancel <generation_id> nodetool generations reconcile <generation_id> # Close orphaned rows and drain the reconcile queue once nodetool generations sweep With nothing recorded, list prints No generations recorded. Exit codes are the verdict, so these compose in a script: await — 0 once the row reaches a terminal status, 1 if it is still running when the timeout expires. cancel — 0 if it was running and is now cancelled, 1 if it had already settled or does not exist. The provider call itself finishes on its own in whatever server process started it; only the record is closed. reconcile — 0 when the provider answered and the row was updated, 1 otherwise, with a reason naming which: no request id, or no reconciler for <provider> when that provider has no billing API, which also marks the row unavailable so it leaves the queue. get prints the whole record: status, provider, model, capability, cost with its currency, billing_unit, quantity and unit_price, asset_ids, started_at/completed_at/duration_seconds, an origin block (surface, thread_id, tool_call_id, job_id, node_id, workflow_id), the provider_request_id, and a reconcile block (reconciled_at, attempts, error, next_at). A model in no price catalog still gets a row, with cost null — list shows it as unpriced rather than as free. sweep is the startup sweep by hand: it closes rows still marked running as interrupted, then drains the reconcile queue once. The server runs it at every start, so reach for it when working against the database directly. Rows awaiting reconciliation retry with backoff at 1, 5, 30, 120, and 720 minutes. provider-list and provider-get <request_id> ask the provider instead of the local database: its own record of what this account ran, from any machine, at the price it billed. Both require --provider <name> (for example fal_ai) and accept --model <id>, the endpoint id (comma-separated for provider-list). provider-list also takes --status <running|completed|failed|cancelled|unknown>, --since <iso>, --until <iso> (exclusive), --limit <n> (default 50) and --cursor <cursor>, the next_cursor that a previous page printed. A provider without a generations API reports that instead of an empty list. nodetool generations provider-list --provider fal_ai --status running nodetool generations provider-get <request_id> --provider fal_ai --model fal-ai/flux/dev Agents reach the same record through list_generations, get_generation, await_generation, cancel_generation, and reconcile_generation. Design: Media generation tracking. Secrets Management nodetool secrets Manage encrypted secrets stored in the local database with per-user encryption. Subcommands: list, store, get Examples: # List stored secret keys nodetool secrets list # Store a secret (prompts for value) nodetool secrets store OPENAI_API_KEY # Retrieve a secret value nodetool secrets get OPENAI_API_KEY Provider Sign-In nodetool auth Sign in to providers that use an account instead of an API key. Today that is Claude: auth claude runs the same OAuth flow the claude CLI does and writes the tokens to the Claude Agent SDK’s credential file (~/.claude/.credentials.json), so a NodeTool login and a claude login are interchangeable. The Claude Agent provider picks the file up with no further configuration. No database and no server are involved. Reach for it when you want to run agents on a Claude subscription rather than store an ANTHROPIC_API_KEY. Subcommands: claude login, claude status, claude refresh, claude logout Options: --console — sign in with a Console (API-billed) account instead of a subscription (for login). --manual — skip the loopback listener and paste the code the browser shows. This is the flow for a headless or remote machine (for login). --no-browser — print the URL instead of opening a browser (for login). --force — refresh even when the current token is still valid (for refresh). --json — output as JSON. Available on login, status, and refresh. Examples: # Browser opens, loopback callback completes the flow nodetool auth claude login # Headless box: paste the code yourself nodetool auth claude login --manual # Check and refresh nodetool auth claude status nodetool auth claude refresh --force nodetool auth claude logout The same flow is available over HTTP at /api/oauth/claude/{start,complete,tokens,disconnect} and as a sign-in card on the Models & Providers settings page. Settings nodetool settings show Display the current values of a fixed list of environment variables: ENV, LOG_LEVEL, PORT, HOST, DB_PATH, NODETOOL_API_URL, the provider keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, MISTRAL_API_KEY, GROQ_API_KEY, SERPAPI_API_KEY), OLLAMA_API_URL, HF_TOKEN, VECTORSTORE_DB_PATH, ASSET_BUCKET, and S3_ENDPOINT_URL. A variable whose name ends in KEY or TOKEN prints as *** when set. Unset variables print empty. Options: --json — output as JSON. Example: nodetool settings show nodetool settings show --json Model Management nodetool models List models and providers. Queries local providers and caches by default. --api-url <url> (default NODETOOL_API_URL) targets a remote server on list, ollama, and huggingface. Every subcommand takes --json. Subcommands: list — list all models (recommended + provider + HuggingFace cached). providers — list configured providers and their capabilities. recommended — list recommended models. ollama — list Ollama models. huggingface — list HuggingFace cached models (--query, --type to filter). by-provider <provider> — list models for a provider; --kind one of llm, image, tts, asr, video, embedding (default llm). Examples: nodetool models list nodetool models providers nodetool models ollama nodetool models by-provider openai --kind image nodetool models recommended The curated shortlist — what to reach for when you want a model id that is known to work, rather than everything a provider will sell you. It reads the shipped RECOMMENDED_MODELS table in-process, so it needs no server, no API key, and no network. --category <category> — one of all (default), image, image-text-to-image, image-image-to-image, language, language-text-generation, language-embedding, asr, tts, video-text-to-video, video-image-to-video. An unknown value exits 1 listing the valid ones. --system <darwin|linux|windows> — keep only models that run on that platform. A model that declares no platforms is kept for all of them. --limit <n> — cap the results. Applied after both filters; a value that is not a positive integer exits 1. --check-servers — fetch the list from a running server instead, which also probes whether the local Ollama and llama.cpp servers can serve each entry. This is the only mode that uses --api-url (default http://localhost:7777, or NODETOOL_API_URL). --json — the full model records rather than the five-column table. # The top of the curated list, no network nodetool models recommended --limit 3 # What to embed with nodetool models recommended --category language-embedding # What a Mac can actually run, machine-readable nodetool models recommended --system darwin --json # Which of them a local Ollama has pulled nodetool models recommended --check-servers nodetool models — HuggingFace Hub and cache Five subcommands search the Hub and manage the local HuggingFace cache. They talk to the Hub’s HTTP API and the cache on disk, so they need no running NodeTool server. Reach for them when you want to find a repo id for a local model node, or pull the weights before a workflow needs them. hf-types — print the nodetool HF model types, one per line, followed by a Generic types (require --task): block. The types in that block need a pipeline tag when you search them; the ones above it do not. --json prints {types, generic} instead. list-hf <model_type> — search the Hub for models matching a nodetool model type. --task <task> supplies the HF pipeline tag (required for the generic types); --limit <n> caps the results; --json prints the raw entries. Without --task, a generic type fails with Model type 'hf.model' requires --task (e.g. 'text-to-image'). list-hf-all — search across every nodetool HF model type at once. --limit <n> caps the total; --repo-only drops file-level entries and keeps one row per repo. hf-cache — list the local cache with disk detail: repo id, path, type, whether the repo is fully downloaded, size on disk, pipeline tag. --downloaded-only keeps only complete repos; --limit <n> and --json behave as elsewhere. huggingface (above) reads the same cache but prints id/name/provider/type/repo_id and can search it (--query, --type) or target a remote server (--api-url); use hf-cache when you care about what is on disk and how big it is. download-hf --repo-id <owner/name> — download a repo into the local HuggingFace cache, printing progress to stderr. --file-path <path> fetches a single file instead of the whole repo; -a, --allow-patterns <glob> and -i, --ignore-patterns <glob> are repeatable globs that narrow the file set. Passing --cache-dir <dir> switches the download to the flat llama.cpp cache layout (~/.cache/llama.cpp on Linux, ~/Library/Caches/llama.cpp on macOS) — it does not redirect the files into <dir>. Examples: # Which model types can be searched, and which need --task nodetool models hf-types # Search one type, then every type at once nodetool models list-hf qwen3 --limit 5 nodetool models list-hf hf.text_to_image --task text-to-image --limit 5 nodetool models list-hf-all --repo-only --limit 20 # What is already on disk nodetool models hf-cache --downloaded-only # Pull weights ahead of a run — whole repo, then just the GGUF files nodetool models download-hf --repo-id Qwen/Qwen3-0.6B nodetool models download-hf --repo-id Qwen/Qwen3-0.6B-GGUF -a "*.gguf" Media Generation nodetool generate <provider> <model> <prompt...> Generate an image from any registered provider straight to a file, with no workflow in between. Use it to try a model, check that a provider’s key resolves, or produce one image from a script. Provider and model names are matched leniently against the registry and the provider’s own model manifest, so fal-ai finds fal_ai and flux-schnell finds fal-ai/flux/schnell. The API key comes from the secret store or the environment; when it is missing the error names the variable to set. Options: -o, --output <path> — output file or directory. A directory (or a path ending in a separator) keeps the generated name; a file path is used as the base name. Default is a nodetool-<model>-<timestamp>.<ext> file in the working directory. --width <n> / --height <n> / --aspect-ratio <ratio> — output size, e.g. --aspect-ratio 16:9. --negative-prompt <text> — what to avoid in the image. --seed <n> — random seed, for reproducible output. -n, --num-images <n> — number of images (default 1). Extra images get a -2, -3, … suffix. --steps <n> / --guidance <n> — inference steps and guidance scale. --image <path...> — input image(s); switches to image-to-image. --strength <n> — image-to-image strength, with --image. --list-models — list the provider’s image models and exit. --json — print the result as JSON. Examples: # Discover model ids for a provider nodetool generate fal-ai --list-models # Text to image nodetool generate fal-ai flux-schnell "a red fox in snow" -o fox.png # Four square variants into a directory nodetool generate fal-ai flux-schnell "a logo" --aspect-ratio 1:1 -n 4 -o ./out/ # Image to image nodetool generate fal-ai flux-dev "restyle this" --image in.png --strength 0.6 MCP Integration nodetool mcp Register NodeTool as an MCP server in Claude Code, Codex, and OpenCode, check the result, or run the server. See NodeTool as an MCP Server for the setup guide. Subcommands: install, uninstall, status, config, serve Examples: # Register the stdio server with every client found on this machine. # Without a global install: npx -y --package=@nodetool-ai/cli nodetool mcp install # Register for Claude Code only nodetool mcp install --claude # Connect to a running server over HTTP instead of stdio nodetool mcp install --http nodetool mcp install --url http://127.0.0.1:8000/mcp # Show the registered entries, then start each one and list its tools nodetool mcp status nodetool mcp status --check # Print an mcpServers JSON block for Cursor, Claude Desktop, or another client nodetool mcp config # Remove from all clients nodetool mcp uninstall Option Applies to Effect --claude, --codex, --opencode install, uninstall Select clients. install without one selects every client found. --npx install, config Launch through npx even when nodetool is on PATH. --http, --url <url> install, config Write an HTTP entry. The default URL is http://127.0.0.1:7777/mcp. --no-verify install Write the config without a test start of the server. --check status Start each registered server and list its tools. mcp status --check exits 1 when a registered server fails to start, and mcp uninstall exits 1 when a client config cannot be written. install starts the server once before it writes a config, so a broken command never reaches a client config. The stdio entry runs nodetool mcp serve when the Node CLI is first on PATH, else npx -y --package=@nodetool-ai/cli nodetool mcp serve. HTTP MCP (/mcp) needs the API server (nodetool serve). nodetool mcp serve uses stdio for the client and forwards to the local API server when its MCP endpoint is available on port 7777 (or PORT when set). Actions then run in the API process, so connected UI clients receive live updates. When the endpoint is unavailable, the command runs its own local MCP server. What the MCP server exposes The MCP mount offers these direct tools alongside the promoted discovery and browser tools: execute_code — the CodeAct action tool, built by the same createChatCodeActSession the in-app chat agent runs on, so the two surfaces cannot drift. Everything NodeTool can do is reached from inside an action: workflow building and debugging (create_workflow, validate_workflow, debug_workflow, debug_app, the ui_* graph editing tools), media generation (generate_image, generate_video, generate_speech, transcribe_audio, …), files (read_file, write_file, edit_file, glob, grep), web (web_search, browser, http_request), collections, documents, code execution, image critique, and memory — as nodetool.<namespace>.<method>(), by importing them, or found with await nodetool.searchTools("query"). Google Workspace tools appear only on deployments with a Google login. view_image — direct, because image content cannot ride a sandbox action’s JSON observation envelope. upload_asset — save base64 bytes in the user’s asset library, or read an absolute path on the local NodeTool server. It returns the asset ID and URI. download_asset — return an owned asset’s bytes as base64, or write them to an absolute path on the local NodeTool server. Remote MCP sessions use base64. Inside execute_code, import_asset accepts an HTTP(S) URL or an absolute server-local path on a local MCP connection. It returns an asset:// URI, byte size, and media duration when the source can be probed. Find its import line with nodetool.searchTools("import asset"). render_demo_surface renders a storyboard, script, sketch, timeline, graph, or 3D surface from a source checkout with Remotion installed. It returns a silent MP4 asset, with the baked corner label hidden by default. The tool also accepts a document cast ID. With both from_ms and to_ms, the exported clip plays that source range at normal speed and has its duration. Without a range, it renders the designed six-second loop. The 3D surface accepts ranges within its six-second camera orbit. MCP has no system prompt. The guest contract (QuickJS, not Node; nodetool.*; no finish()) is the server instructions string and the first lines of the execute_code description. The rest of the description is the CodeAct catalog. Two resources carry the machine-readable form: nodetool://capabilities (tools and modules) and nodetool://sandbox (blocked globals, unavailable bridges, worked examples). Prompts sandbox-action and sandbox-asset are complete execute_code bodies. Each skill the user can load is also a prompt with the skill’s name, and it returns the skill body. The session needs a user to run as — its tools touch that user’s secrets, assets, and files — so createMcpServer refuses one that is not bound to a user (nodetool mcp serve, the local /mcp mount, or an authenticated session). File tools read and write under a per-user workspace at <data-dir>/mcp-workspaces/<user-id>, not the host filesystem. Agents nodetool agent Run one CodeAct turn from the command line, over the default toolbelt. The objective becomes the user message and the model acts by writing sandboxed JavaScript — the same loop the chat runs. create_plan and execute_plan are on the belt, so an objective that wants decomposing gets a task DAG and the planning_update / task_update events stream with the rest of the trace. Subcommands: run, diagnose # Run an agent with an objective nodetool agent run -p anthropic -m claude-sonnet-5 --objective "Research AI trends" # Or pipe the objective echo "Research AI trends" | nodetool agent run -p anthropic -m claude-sonnet-5 # Bound the tool-calling rounds nodetool agent run -p openai -m gpt-5.4-mini -o "Summarize the README" \ --max-iterations 8 # Ask before every write, execute or external call nodetool agent run -p anthropic -m claude-sonnet-5 -o "Tidy the workflows" \ --permission-mode default # Bound what the run may spend and how long it may take nodetool agent run -p openai -m gpt-5.4-mini -o "Research AI trends" \ --cost-cap 0.50 --timeout 120 # Aggregate a failed run into one report nodetool agent diagnose <job_id> --max-steps is gone with the planner→compiler pipeline the command used to run, along with plan approval, checkpoints, the planning/reasoning model split, and skill auto-select. See Agent CLI. Run budget --cost-cap <usd> and --timeout <s> bound one run. They override two of the five NODETOOL_AGENT_* settings a chat turn reads (NODETOOL_AGENT_TURN_COST_CAP_USD, NODETOOL_AGENT_TURN_DEADLINE_MS); the other three — NODETOOL_AGENT_MAX_CONCURRENCY, NODETOOL_AGENT_MAX_TURNS, NODETOOL_AGENT_UNPRICED_TOKEN_CEILING — come from the settings alone. --cost-cap 0 lifts the dollar cap, which is what a local-only install wants; --timeout 0 leaves the run no time and stops it before its first model turn. One budget covers the whole run, not each loop: a sub-agent, an execute_plan DAG, and an AgentNode reached through run_node all reserve against it. A cap is admission, so a turn whose worst case would cross it is refused before the call rather than noticed after the money is spent — a model the price catalog does not cover has no worst case and is admitted against a prompt-token ceiling instead, never as free. A run a ceiling stops prints the reason and exits non-zero: agent stopped: turn budget of $0.01 reached With --json the same reason arrives as an error event. Every run ends with what it committed against the cap, as spent $0.0123 on stderr or a log_update in --json. Permission mode Every tool call the model makes goes through one ladder, on every host. What a mode decides: Mode Read Write, execute, external default runs asks first auto runs runs plan runs blocked, with a message telling the model to switch out --permission-mode sets it on nodetool agent run and on nodetool-chat; an unrecognized value is refused rather than falling back to a default. In a terminal, default is what an unset flag means. Each write, execute or external call waits for y (this call), n (refuse), or a (this tool for the rest of the session). Chat displays the request in its fullscreen interface. nodetool agent run prints it on stderr, leaving stdout for the run’s result. Behind a pipe, nobody is there to answer, so the run takes the headless gate: auto, printing once up front that escalated calls are denied. An explicit mode still applies — a piped --permission-mode plan blocks what plan mode blocks — but the answer to anything the ladder escalates is deny, never a silent allow. See the Agent CLI reference for full details. The nodetool db group (migrate, status, baseline, rollback) is documented under Database Migrations above. Deployment and Workers Two different things, and the CLI keeps them apart. A server is long-lived and people connect in to it — nodetool deploy manages one over Docker. A worker is a rented GPU that a NodeTool instance connects out to, bills by the minute, and is meant to be torn down — nodetool worker provisions it. See Deployment for which one you want. nodetool deploy Manage a Docker self-host server target described by a deployment.yaml. init scaffolds the file, add fills in one target, and every other subcommand names that target. The file lives in the user config directory, not the working directory: ~/Library/Application Support/nodetool/deployment.yaml on macOS, %APPDATA%\nodetool\deployment.yaml on Windows, and $XDG_CONFIG_HOME/nodetool/deployment.yaml — falling back to ~/.config/nodetool/ — elsewhere. Run nodetool deploy edit to open it rather than guessing the path. Subcommands: init, add, edit, list, show, plan, apply, status, logs, destroy, plus the remote group workflows and the users-* verbs below. Options: --type <type> — required on add. docker is the only supported value. --dry-run — print what apply would do without executing it. -f, --follow / --tail <n> / --service <service> — for logs; --tail defaults to 100. --force — skip the confirmation on destroy, workflows delete, and users-remove. --json — machine-readable output on list, workflows list, and users-list. # Scaffold, describe, and review before touching the remote host nodetool deploy init nodetool deploy add my-server --type docker nodetool deploy plan my-server # Deploy and watch it nodetool deploy apply my-server nodetool deploy logs my-server --follow --tail 200 # Open deployment.yaml in $EDITOR nodetool deploy edit list and show answer the two questions you have before naming a target in any of the other verbs, and they are the pair to reach for when you have inherited a deployment.yaml you did not write. list queries each configured target for its live state and prints one row per deployment — name, type, status, last_deployed, host, container, pod_id, project, region, service. With none configured it prints (no deployments configured); --json gives the same rows for a script. show <name> reads the file only, printing that one target’s config as YAML — no remote call, so it works while the host is down. nodetool deploy list nodetool deploy list --json nodetool deploy show my-server Both need the file to exist. Without it they exit non-zero with Deployment configuration not found at <path>. Run 'nodetool deploy init' to create it. The full server walkthrough is Deployment and Self-Hosted Deployment. Remote workflows Once a target is up, workflows acts on the deployment’s /api/workflows routes rather than on the local database: # Push a local workflow to the deployment, then run it there nodetool deploy workflows sync my-server <workflow_id> nodetool deploy workflows list my-server nodetool deploy workflows run my-server <workflow_id> -p prompt="a red fox" nodetool deploy workflows delete my-server <workflow_id> workflows run takes -p, --param <k=v>, repeatable. sync pushes the workflow row itself; the assets and models its nodes reference must already be present on the target. API users on the deployment For a multi-user server, mint and rotate the tokens its API clients authenticate with. --role is user unless you pass admin. nodetool deploy users-add my-server alice --role admin nodetool deploy users-list my-server nodetool deploy users-reset-token my-server alice nodetool deploy users-remove my-server alice Every subcommand that touches a live deployment — the users-* verbs and the workflows group above — sends an admin bearer token. --token <token> passes it explicitly and wins over NODETOOL_ADMIN_TOKEN; with neither, an interactive shell prompts and a non-interactive one exits 1. nodetool worker Provision, attach to, and tear down a rented GPU worker on RunPod or Vast. A worker bills by the minute, so stop is part of the workflow, not cleanup you get to postpone. Subcommands: profile (add, list, rm), create, list, status, token, stop, models Options: --target <runpod|vast>, --image <image>, --gpu <gpu>, --vcpu <n> — what to rent, on profile add or inline on create. --idle-timeout <minutes> / --max-lifetime <minutes> — auto-stop when idle, and a hard TTL. Both are the guard against a forgotten worker. --ssh-key <path> — OpenSSH public key file. It exposes 22/tcp so the worker can be debugged. --disk <gb> — persistent volume size, which holds the model cache (default 100). --token-policy <generate|fixed> — default generate. --profile <name> / --attach — for create. --all — for stop, stops every non-stopped worker. --json — for list and profile list. # Save a preset once, then rent from it nodetool worker profile add hf-a40 --target runpod \ --image ghcr.io/nodetool-ai/nodetool-worker:latest \ --gpu "NVIDIA A40" --idle-timeout 15 nodetool worker create --profile hf-a40 --attach # Inspect and tear down nodetool worker list nodetool worker status <instance-id> nodetool worker stop --all token <id> prints one worker’s bearer token and nothing else — no table, no label — so it pipes into the variable the bridge reads: export NODETOOL_WORKER_TOKEN=$(nodetool worker token <instance-id>) list withholds tokens on purpose; this is the one command that decrypts one, and only for the worker you name. Reach for it when a second shell, a CI step, or a --api-url client has to talk to a worker --attach did not configure. A worker that carries no token is an error rather than empty output: Worker instance '<id>' has no token (open worker). The full walkthrough, including what --attach changes locally, is Worker Deployment. nodetool worker models Manage the HuggingFace cache on the worker itself, over the WebSocket bridge — no NodeTool server has to be running. Pre-pulling weights here is what keeps the first run of a graph from paying the download on rented GPU time. The [worker-id] argument is optional: omit it and the command uses the currently attached worker, or errors telling you to attach one. Options: --repo-id <id> — HuggingFace repo, owner/name. Required for download and delete. --file-path <path> — download a single file instead of the repo. -a, --allow-patterns <pattern> / -i, --ignore-patterns <pattern> — glob filters on download, both repeatable. --json — for list. # What is already cached on the attached worker nodetool worker models list # Pull weights ahead of a run — whole repo, then just the safetensors nodetool worker models download --repo-id stabilityai/sdxl-turbo nodetool worker models download --repo-id stabilityai/sdxl-turbo -a "*.safetensors" # Reclaim disk on a named worker nodetool worker models delete <worker-id> --repo-id stabilityai/sdxl-turbo Messaging Bridges nodetool telegram Run the Telegram bridge: it turns private-chat messages into turns on the agent loop of a running NodeTool server, and streams the answers back. The bridge holds no conversation state and no user credentials — the agent loop, tools, permissions, threads, and cost tracking all stay on the server, which the bridge reaches over /ws as the linked user. Reach for it when you want NodeTool answering from a phone without opening the web UI. The server it talks to must have NODETOOL_INTEGRATION_TOKEN set, or the linking routes the bridge calls do not exist. Subcommands: serve, register-commands Options: --config <path> — path to telegram-bot.json, relative to the working directory. Both subcommands take it. Without the flag, telegram-bot.json is read if it is there and skipped if it is not; a path you asked for explicitly and that does not exist is an error. Configuration is environment variables plus that optional file. The env vars are listed in the Environment Variables Index; the file carries only the tuning knobs: // telegram-bot.json { "allowUsers": [], // Telegram user ids allowed to link; empty = anyone "editThrottleMs": 1500, // minimum gap between edits of one streamed reply "maxQueuedTurns": 3 // messages queued behind an in-flight turn } # Publish the bot's command list — a deploy step, not a boot step export TELEGRAM_BOT_TOKEN=123456:AA... export NODETOOL_INTEGRATION_TOKEN=the-same-value-the-server-has nodetool telegram register-commands # Long-poll the Bot API and serve turns nodetool telegram serve --config ./telegram-bot.json Both subcommands validate the whole configuration before doing anything and print the fields that failed rather than a stack trace: Invalid Telegram bridge environment: TELEGRAM_BOT_TOKEN: TELEGRAM_BOT_TOKEN is required NODETOOL_INTEGRATION_TOKEN: NODETOOL_INTEGRATION_TOKEN is required serve polls with getUpdates. Webhook mode is not implemented yet, and setting TELEGRAM_WEBHOOK_URL makes serve refuse to start rather than silently polling. Design notes: telegram-bot-design.md. Development Tooling nodetool affected [files...] Map changed files — or the git working tree — to the smallest set of workspaces that must be rebuilt and tested: the workspace that owns each file plus its downstream dependents, and a build:packages only when a package that loads from dist/ is in the set. It saves reaching for the full build after a one-file change. Options: --base <ref> — compare against a git ref instead of the working tree. --json — print the result as JSON. Examples: nodetool affected nodetool affected --base main nodetool affected packages/cli/src/nodetool.ts nodetool affected --json The output names each affected workspace and the commands to run: 1 affected workspace(s) from 1 changed file(s): changed @nodetool-ai/cli Suggested commands: npm run test --workspace=packages/cli nodetool harness The machine-readable inventory behind harness-first engineering: every headless harness in the repo, every product surface with the code paths it owns, and which harnesses cover which surface. Shipping a new surface means adding it to the registry with its harness — or with its debt written down. Subcommands: list, audit, capabilities, gate nodetool harness list List every harness and its capabilities. Takes --json to print the registry. nodetool harness list nodetool harness list --json Each entry names the harness, its kind, what it covers, and how to invoke it: affected meta Changed-file → workspace mapping nodetool affected [--base main] [json, no-db] packs-compile static Sandbox npm module compiler (bundle, scan, probe, cache) and the shipped bridge packs nodetool packs compile [--json] [--force] [json, no-db] nodetool harness audit Audit surface coverage. A surface with no harness must carry a written gap note; one without it fails the audit. Options: --json, and --strict to exit non-zero while any gap remains. nodetool harness audit nodetool harness audit --strict Covered surfaces list their harnesses; gaps print the note that justifies them: ok sandbox-packages packs-compile, validate GAP electron-shell Electron shell (windows, IPC, menus, auto-update) Documented gaps: electron-shell: Covered by Jest unit tests only. A harness would boot the packaged shell headlessly (Playwright's Electron driver), assert the IPC surface, and reuse backend-smoke for the server half. nodetool harness capabilities The same invariant one rung down, over the agent capabilities rather than the product surfaces: every exported capability must name the checked-in suites that exercise it, the eval cases that drive a model through it, or a written gap note. Reach for it after adding a capability — a new one with neither a check nor a note fails here, and in npm run capabilities:check. Options: --json, and --strict to exit non-zero while any gap remains. nodetool harness capabilities nodetool harness capabilities --strict The header counts covered capabilities, then each gap prints the note that justifies it: Capability coverage: 216/224 covered, 8 gap(s) GAP ui_get_graph ui GAP get_apify_actor apify Documented gaps: get_apify_actor: The Apify suites cover search, schema, run, abort, and the dataset reader; fetching one actor's record has no case. A read-only case against the recorded actor fixture would close it. A trailing Contract drift: line means a capability’s declared description, schema, category, or needsToolCallId moved without its coverage mapping being touched. Refresh the table with npm run capabilities:sync and say which case covers the new contract. nodetool harness gate [files...] Map a diff onto surfaces by path and run the selfcheck of every harness covering a touched surface. The diff selects the checks, not the author. Harnesses that need a target or a key are printed as manual work rather than silently skipped. Options: --base <ref> — diff against a git ref instead of the working tree. --all — ignore the diff and run every selfcheck. --expensive — include the expensive selfchecks (bundle staging and friends). --dry-run — print the plan without running anything. --json — print the plan, and the results unless --dry-run. --strict — exit non-zero when the diff touches a surface no harness covers. --deps — also touch surfaces whose paths sit in a workspace downstream of a changed one. --include-suites — also run suiteOnly selfchecks, which only re-run suites npm run test:affected runs for the same diff, and workspace test commands skipped when a global-file diff selects the full test suite. --jobs <n> — run cheap selfchecks n at a time (default 1). Each check’s output prints as one block when it finishes. Expensive checks run serially afterwards. --timeout <seconds> — kill a selfcheck that runs longer (default 900). A changed root manifest, lockfile, root tsconfig*.json, turbo.json, .nvmrc, or shared scripts/run-*.mjs launcher selects every selfcheck. The plan step loads no built package, so --dry-run works before npm run build:packages. For global-file diffs, commands that only run workspace test suites covered by the full test:affected selection are skipped by default. Commands with other work or unknown workspaces still run. This assumes test:affected runs for the same diff. --all alone does not enable those extra skips. CI adds --include-suites when its skip-tests input disables package tests. Check starts and completions print to stderr, including with --json. Each result includes durationMs, while stdout keeps the JSON output. CI uses --jobs 1 and NODETOOL_TEST_WORKERS=1 to run selfchecks and their test files serially. This prevents competing selfchecks from exhausting the game runtime’s wall-clock script batch budgets. Timeouts still apply. nodetool harness gate --base main nodetool harness gate --dry-run packages/base-nodes/src/index.ts nodetool harness gate --all --expensive The plan names the surfaces the diff touched before it runs anything: 1 changed file(s) touch 1 surface(s): workflow-execution (1 file(s)) Manual harnesses (need a target/key — run yourself): debug nodetool debug <id|file> [--browser --trace --watch --supervise] 3 selfcheck(s) to run nodetool package Manage TypeScript node packages — the packs that contribute node types to the registry. Not to be confused with nodetool packs, which handles the sandbox packs a Code node imports; the names are one letter apart and the subjects are unrelated. Reach for it when you are authoring a node pack: init scaffolds one, list tells you what this install has, and the three *-docs commands generate the Markdown that ships with a pack. Subcommands: list, init, docs, node-docs, workflow-docs nodetool package list List the installed packages with their node counts, or the registry’s catalog with -a, --available. --json prints the rows as JSON. nodetool package list nodetool package list --available nodetool package list --json name │ version │ description │ nodes ───────────────┼─────────────┼─────────────────────┼─────── nodetool-base │ 0.6.3-rc.41 │ Nodetool Base nodes │ 0 --available prints the Python packs in PYTHON_NODE_PACKS (packages/protocol/src/python-packs.ts). See Node Packs for what a pack is and how one is installed. nodetool package init Scaffold a new TypeScript node package in the current directory. Takes no options — it prompts for what it needs. mkdir nodetool-my-nodes && cd nodetool-my-nodes nodetool package init nodetool package docs, node-docs, workflow-docs Generate Markdown from the current package: docs writes one overview, node-docs writes a page per node, and workflow-docs documents the workflow JSONs a pack ships. Each takes -v, --verbose. Options: docs — -o, --output-dir <dir> (default docs), -c, --compact for a shorter overview. node-docs — -o, --output-dir <dir> (default docs/nodes), -p, --package-name <name> to emit only nodes under a namespace prefix. workflow-docs — -o, --output-dir <dir> (default docs/workflows), -e, --examples-dir <dir> (required) to point at the workflow JSONs, and -p, --package-name <name> to keep only those whose package_name matches. nodetool package docs --compact nodetool package node-docs --package-name nodetool.text nodetool packs Manage node packs and npm runtime libraries for a global CLI installation or an npm project. Mutation is a preview feature enabled with NODETOOL_PACKS_INSTALLER=1. Temporary npx installations and development checkouts support diagnostics but refuse installation changes. Command Behavior packs compile [--json] [--force] Compile installed sandbox pack modules. See sandbox compilation. packs list [--json] Show available packs, installation state and size estimates. packs add <ids...> [--yes] Install the pinned packs or runtime libraries and their requirements. packs add --restore [--yes] Restore the desired set recorded for this install root. packs add --for <workflow-or-file> [--yes] Install missing requirements of a saved workflow or an explicitly named local file. packs remove <ids...> Remove packages owned exclusively by the selection while preserving retained requirements. packs doctor [--json] [--installed] [--pack <ids...>] Check the mode, versions, integrity and installation state. Registration checks use a fresh child process. packs doctor --repair Repair a dirty installation after verifying its pinned cohort. packs why <node_type-or-id> Explain ownership, requirements and installation state. Add commands prompt with package versions, size estimates, approved scripts and local manifest changes. Use --yes in a non-interactive terminal. Local installation updates package.json and package-lock.json. Restart running servers after adding or removing packs. run, debug, node run and workflows run accept --install-missing. NODETOOL_AUTO_INSTALL=1 also authorizes those commands. They install missing npm requirements and re-execute once, forwarding the child exit code. Serve, MCP and agent tools do not install packages automatically. External runtimes such as ffmpeg and Python require their own setup. See the pack-system design for installation boundaries and the platform matrix. nodetool packs compile Compile every npm-backed sandbox module of every installed pack. A pack can declare a guest module by npm dependency name instead of authoring code ({"name": ".", "kind": "js", "npm": "js-yaml"}); this builds it — esbuild bundles the dependency with no externals, a scope-aware scan rejects free references to globals the guest lacks, and a capability-free QuickJS probe imports the bundle to prove it initializes. Results are cached by content digest, never by version. The server compiles during its own catalog refresh and Electron compiles after an install, so reach for this to warm the cache or to diagnose one pack. Options: --json — print the compile report as JSON. --force — recompile and re-probe even when the cache has an answer. --pack-search-path <dir> — search this node_modules root (repeatable). nodetool packs compile nodetool packs compile --json nodetool packs compile --force Each module reports its source dependency and bundled size: ok @nodetool-ai/sandbox-markdown ← marked — 66101 bytes ok @nodetool-ai/sandbox-yaml ← js-yaml — 103863 bytes 10 compiled, 0 skipped. Everything that stops a module short of admission is a named skip, not an error: npm-module-builtin-import (the dependency needs node:*), npm-module-unresolved, npm-module-too-large (1 MB cap), npm-module-forbidden-global, and npm-module-probe-failed. nodetool reliability Run a reliability journey on more than one execution surface and diff each one against the kernel oracle. A journey is a small workflow plus the invariants its run must hold — lifecycle pairing, one terminal message, no leaked cleanup — and the point is that the kernel runner and the WebSocket server must produce the same stream for it. Reach for it when a change touches execution and you need to know whether the surfaces still agree. Journeys live in reliability/journeys/. Working in the monorepo, run this one from the built CLI (npm run nodetool --) rather than from source: the journey fixtures use decorators, and the dev:nodetool transform rejects them with Decorators are not valid here. Subcommands: list, run, update-goldens nodetool reliability list List the journeys, each with its surfaces and the invariants it asserts. Takes --json for the full summaries. nodetool reliability list nodetool reliability list --json linear-text-pipeline Journey #1: input -> transform -> output. The baseline — if this diverges across surfaces, everything is suspect. surfaces: kernel, ws-server invariants: lifecycle-pairing, terminal-uniqueness nodetool reliability run <journey> Run one journey on the surfaces it declares and diff every non-oracle surface against the kernel oracle. Options: --surface <name> — run only this surface, repeatable, overriding the journey’s own list. The kernel oracle is always included. --faults <name> — inject this fault, repeatable, replacing the journey’s declared fault matrix. The provider-seam faults are implemented (provider-429, provider-500, provider-timeout, truncated-stream, malformed-sse, slow-drip, cost-omission); the ws/bridge/host/ client names parse but report as unimplemented. --diff — print the full per-channel stream diff for a diverging surface. --json — print the whole CompareReport instead of the summary. nodetool reliability run linear-text-pipeline nodetool reliability run linear-text-pipeline --surface kernel nodetool reliability run mid-run-cancel-node --faults provider-429 --diff nodetool reliability update-goldens <journey> Rewrite a journey’s expected/ fixtures from a fresh unfaulted kernel run. For a golden that legitimately moved — read the diff before committing it, since this command cannot tell a fixed bug from a new one. nodetool reliability update-goldens linear-text-pipeline nodetool eval <suite> Run one of the agent evaluation suites against a provider and model, and report its metrics — success rate, expectation score, tool calls, duration, and cost. Suites: graph-planner, graph-e2e, code-gen, task-planner, subtask, codeact, app-build, tool-loop, workflow-escalation, script-tools, jsscript-tools, sketch-tools, timeline-tools, storyboard-tools, model3d-tools, app-tools, memory-tools, creative-pipeline. Options (the same on every suite): -p, --provider <id> — provider id (anthropic, openai, claude_agent_sdk, ollama, …). -m, --model <id> — model id for that provider. --cases <ids> — comma-separated case ids to run (default: all). --list — list the suite’s cases and exit. --json — print the full report as JSON. --out <path> — write the JSON report to a file. --max-iterations <n> — turn cap per case for the loop-style suites (default 12). --timeout <ms> — per-case execution timeout for the suites that run what they plan (default 300000). --system-prompt <text> — the caller preamble the planner suites plan behind (default: the AgentNode’s own prompt). --no-agent-prompt — plan with no caller preamble, which scores the bare planner contract. --judge-model <provider/model> — the model that judges outputs in the self-judging suites (graph-e2e, app-build). Defaults to the run’s own provider and model, which grades its own work. --min-success <rate> — exit non-zero when the success rate falls below this threshold (0..1). --min-score <rate> — exit non-zero when the mean expectation score falls below this threshold (0..1). --min-cases <n> — exit non-zero when fewer than n cases ran. Skipped cases do not count. --max-cost-usd <usd> — exit non-zero when the run’s total provider spend exceeds this. --max-p95-duration-ms <ms> — exit non-zero when the p95 case duration, skipped cases excluded, exceeds this. --keyless — run only the cases that need no API key and no network (see --list). --no-find-model — run without configured model providers, skipping the model-dependent cases. Examples: # See what a suite covers before spending on it — needs no provider nodetool eval graph-planner --list nodetool eval graph-planner -p anthropic -m claude-sonnet-5 nodetool eval timeline-tools -p openai -m gpt-5.4-mini --cases cut-and-trim,titles-with-motion nodetool eval graph-e2e -p anthropic -m claude-sonnet-5 --timeout 600000 nodetool eval codeact -p anthropic -m claude-sonnet-5 --json --out report.json # Gate a run in CI nodetool eval sketch-tools -p ollama -m qwen-3.5:4b --min-success 0.8 --list prints each case id with what it checks: summarize Single LLM step with one wired input and one output branch-both-paths Conditional with both If branches wired deterministic-over-llm Pure string mechanics must not be solved with an LLM step nodetool jtbd An eval suite scores a model. A job asks whether the product let the agent finish something, and keeps the transcript so you can say what to change. A job is one objective taken end to end across whatever surfaces it needs, stated the way a user would state it, handed to the agent in the user’s own words, and graded on the world it left behind. No job names a tool — which tool the agent reaches for, and in what order, is what is under test. The worlds are the same headless bridges the tool-loop eval suites drive. Subcommands: list, run, optimize. # What the jobs are, before spending anything — needs no provider nodetool jtbd list nodetool jtbd list --json Each entry prints the job id, its difficulty tier, the surfaces it crosses, the statement, and the outcomes it grades: storyboard-a-scene [standard] storyboard When I have a scene to shoot, I want it broken into shots with the action described, so I can see the coverage before I spend on renders. outcomes: five-shots, action-written, savable nodetool jtbd run Drives a model through the jobs and writes one bundle per job. -p, --provider <id> — provider id (anthropic, openai, ollama, …). Required. -m, --model <id> — model id for that provider. Required. --jobs <ids> — comma-separated job ids (default: all; see list). --out <dir> — bundle directory (default nodetool-debug/jtbd). --max-iterations <n> — turn cap per job, for jobs that declare none. --min-achieved <rate> — exit non-zero when the achievement rate falls below this threshold (0..1). --no-find-model — run without configured model providers. --json — print the full report as JSON. nodetool jtbd run -p anthropic -m claude-sonnet-5 nodetool jtbd run --jobs workflow-from-prompt,timeline-assemble-cut -p openai -m gpt-5.4-mini # Gate a run in CI nodetool jtbd run -p anthropic -m claude-sonnet-5 --min-achieved 0.8 run writes nodetool-debug/jtbd/<job>/ holding report.json (transcript, tool calls, outcomes, friction) and review.md, the same run rendered for a person to read. The transcript is the point: it records what the model was told and what it said between calls, not only which tools fired. Before any model reviews a run, a pure pass derives what the transcript decides on its own, and each finding names an owner. A tool that errored repeatedly, or answered the same call identically three times, is a harness finding — the fix is a schema or an error string in NodeTool’s own code. A run that called no tool at all is a prompt finding. A run that took far more calls than the job needs is unattributed, because whether that is a prompt failing to describe the short path or a tool surface forcing the long one is the judgement a pure pass cannot make. nodetool jtbd optimize Hands one recorded run — the system prompt verbatim, every assistant turn, every call with its arguments and result — to a different model and asks what one change would make the next run go better. -p, --provider <id> — reviewing provider; use a different one than ran the jobs. -m, --model <id> — reviewing model id. --bundle <dir> — bundle directory to review (default nodetool-debug/jtbd). --all — review clean runs too, not just failures and friction. --json — print the proposals as JSON. nodetool jtbd optimize -p openai -m gpt-5.4-mini nodetool jtbd optimize -p openai -m gpt-5.4-mini --all --bundle ./my-runs A proposal must name a target and a change; “improve the prompt” is rejected by the parser. Proposals land in proposals.json next to the run. It proposes; it never applies. Nothing in the loop edits a prompt or a tool. Run and review are separate commands because the bundle is the handoff: a bundle can be re-reviewed with a better optimizer without paying for the runs again, and a person can read the transcript before any model proposes anything. Tips Use --json flags for machine-readable output suitable for scripting. Set NODETOOL_API_URL environment variable to avoid specifying --api-url on every command. Use nodetool serve to start the local backend server before running API commands. See Environment Variables for a complete list of configurable variables.--- # Code Node Output Interface: emit and output Source: https://docs.nodetool.ai/code-node-emit-design.html Code Node Output Interface: emit and output Status: implemented (sandbox bridge, CodeNode pump, validation, harness; the migration codemod remains open). Owner: code-nodes / node-sdk. 2026-08-12. Two implementation choices differ from the text below, both deliberate: the legacy deprecation warning lives only in the validate_code harness — graph validation would red the shipped-examples gate (it treats warnings as errors and every shipped Code body is legacy), and a run-log warning changed the message stream the Ring 0 reliability goldens pin — and the pristine default body return {}; is exempt from the warning, because an empty node is not a legacy body. Problem The Code node’s output contract is a return value: “return an object — its keys become output handles” (normalizeCodeOutput in packages/node-sdk/src/code-body.ts). That shape has three costs, and agents pay all of them: Streaming is replay, not streaming. A body with yield runs through genProcess, but the guest collects every yielded value into an array and the host replays them after the run completes (packages/code-nodes/src/nodes/code-node.ts, the yield_ rewrite). A loop that fetches 50 pages delivers nothing downstream until page 50. The rewrite itself is a regex (code.replace(/\byield\b/g, "yield_")), so the body the node runs is not the body the author wrote, and top-level yield is not valid JavaScript — real parsers, including our own code-analysis.ts, must special-case it. The contract is implicit. Nothing in the body names an output; the shape of every return path does. The validator must prove each declared output is set on each return path, and its own error message tells authors to avoid branching in code and use nodetool.control.If nodes instead — the check is fighting normal code shape. On top of the return-path analysis sit two more inference layers: implicit-return wrapping of the last expression (wrapImplicitReturn) and shape normalization of whatever came back (normalizeCodeOutput). There is no call an agent can reason about. CodePlanner, validate_code and repair loops infer outputs from data shape. A function call — emit("x", value) — is a fact an AST query answers directly. Guest contract Outputs leave the body only through two host-bridged calls. The body’s return value is ignored — return is what it is in JavaScript, control flow, and carries no output semantics. await emit(name, value) // stream one value to output handle `name`, now await output(name, value) // set the final value of output handle `name` emit(name, value) delivers { [name]: value } downstream immediately, while the body keeps running. Call it any number of times per handle. output(name, value) records the handle’s final value. It is delivered when the body completes, all final values as one bag — so a downstream node that consumes one value per handle sees exactly one. A second output call for the same handle throws: a final value that changes is a bug, and a deterministic error beats a silent last-write-wins. Both are await-able; awaiting emit applies backpressure (see Limits). name must be a string naming a declared output handle; values go through the same marshaling as today (JSON-safe data, media refs from media.toImage etc.). return does nothing to outputs. Use it to exit early. A body may end without calling either function only when the node declares no outputs. The symmetry is the point: one verb for “here is an item” and one for “here is the answer”, both explicit, both statically visible, nothing inferred from data shape. Deleted with the return contract: hasReturnStatement routing, wrapImplicitReturn (implicit return of the last expression), and normalizeCodeOutput (shape normalization). A body that today reads inputs.a + inputs.b becomes await output("output", inputs.a + inputs.b) — one line longer, zero inference. Semantics Situation Behavior emit("out", v) {out: v} streams downstream in call order, live output("sum", v) then body ends final bag {sum: v} is the last message several output calls, distinct handles one final bag carrying all of them output twice on one handle throws in the guest return v with any value v is discarded; control flow only body ends, declared output neither emitted nor output validation error before run (see below); at runtime the handle stays empty emit/output to an undeclared name validation error before run; at runtime the call throws non-serializable value same marshal rules as today (JSON deep-copy; media via media.to*) error thrown after emits already-emitted values were delivered; recorded output values are not posted; the node fails timeout mid-stream same as error: delivered emits stand, finals dropped, node fails Host mechanics emit and output join EXPOSED_BRIDGE_NAMES in packages/agents/src/js-sandbox.ts as awaitable host calls (the asyncified path, not the fire-and-forget progress path — values cannot be rate-limited or dropped). process() disappears from CodeNode; every body runs through one genProcess pump: start runInSandbox(...) → promise P while P pending or queue non-empty: yield next bag from queue (each emit pushed one) await P (failure → throw, finals discarded) yield the collected output() bag (unless empty) The queue is a bounded async channel owned by the node invocation. The host side of emit resolves the guest’s promise only after its bag is accepted by the channel, so a fast producer awaiting emit blocks until the kernel has consumed — backpressure with no new kernel concept. output never queues; the host stores the value and posts the bag after P resolves. A body that calls emit without await still works; the channel bound (below) caps unacknowledged values. Limits Channel capacity: 64 pending bags. emit awaits drain beyond that. Max emits per run: 10 000 (same spirit as MAX_PROGRESS_CALLS; exceeding throws in the guest, naming the cap). Per-value size: bounded by the existing sandbox marshal limits; no new cap. Validation and analysis The return-path analysis in code-node-validation.ts — “every declared output set on every return path” — is deleted, not amended. Its replacement is one reachability rule over two call names: Every declared output must have ≥1 reachable emit or output call with that literal name. Branching freely is fine; the rule is per-handle existence, not per-path coverage. A call with a non-literal first argument downgrades that body to a warning (“cannot check output names statically”) instead of failing it. A call naming an undeclared handle is an error, symmetric with the existing undeclared-inputs.<name> read check. emit and output count as known globals; a bare read of either stops being a ReferenceError candidate. A top-level return <value> (non-undefined argument) is a warning naming the new contract, since under the old one it meant “these are the outputs”. The analysis lives where it does today (code-analysis.ts / code-node-validation.ts in node-sdk), shared by the graph validator, the submit_code planner, and the editor. Surface parity run_code / test_code (packages/agents/src/capabilities/code.ts): emitted values land in the existing streamed array as {name, value} entries, in order; final values land in outputs. test_code cases may assert on both. CodePlanner / code-gen eval: prompts and cases rewritten to the emit/output contract; the code-gen suite gains one streaming case. Editor: the Code node description and the assistant dialog prompt teach the two calls; no UI change — streamed values render through the same messages every streaming node already produces. Docs: docs/javascript-sandbox.md § Outputs and § Streaming rewritten. Backwards compatibility and migration Removing the return contract breaks every existing body, so the old path stays executable for one release behind a per-body probe: A body that calls emit or output runs on the new contract; its return value is ignored. A body that calls neither runs on the legacy path (return bag, implicit return, yield replay), and validate_code emits a deprecation warning naming the two calls. Saving such a node in the editor surfaces the same warning. The workflow migration is mechanical and shippable as a codemod on the stored code string: return {a, b} → await output("a", a); await output("b", b); return;, yield x → await emit("output", x). The codemod runs through validate_code on its own result and refuses any body it cannot prove equivalent, leaving those for hand migration. After the window, hasReturnStatement, hasYieldStatement, wrapImplicitReturn, normalizeCodeOutput, the yield_ rewrite, and the return-path validator all delete. Testing Unit (packages/code-nodes): emit ordering; emit+output interleave; finals post as one bag; double output throws; return value discarded; backpressure (producer faster than consumer); cap exceeded; error after emit drops finals; undeclared handle throws; legacy body still runs and warns. Analysis (packages/node-sdk): reachable-call rule satisfies declared outputs; non-literal name warns; undeclared name errors; top-level valued return warns. Each new check is proven failable with an inverted fixture before landing. Codemod: round-trips the shipped example workflows’ Code bodies and the code-gen eval fixtures; every migrated body passes validate_code and test_code with unchanged expectations. End-to-end: a workflow where a Code node emits into a downstream collector runs under nodetool debug and shows per-item messages arriving before the node completes. Out of scope CodeAct (execute_code) — its scripts act on the toolbelt and return one result; this contract is for node code. New kernel message types — emits and the final bag ride the existing genProcess semantics. Per-output typing of emitted values beyond the handle’s declared type.--- # Code Node Input Streaming: stream Source: https://docs.nodetool.ai/code-node-input-streaming-design.html Code Node Input Streaming: stream Status: implemented, 2026-08-12. Owner: code-nodes / node-sdk / kernel. Companion: code-node-emit-design.md — the output side of the same contract, already implemented. Problem A Code node consumes its inputs buffered: the kernel invokes the body once per upstream item, with inputs holding that invocation’s snapshot. Streaming out of a body is solved (emit), but streaming in is not — the body never sees the stream, only one frame of it. That shape has four costs: Cross-item logic needs the state escape hatch. A running total, a window, a dedupe set — anything spanning items lives in the mutable state object that survives invocations. The logic of one loop is split across N invocations of a body that reads like a single step, and the author must reason about which variables reset and which persist. End-of-stream is invisible. A body cannot say “when the stream ends, emit the summary”. The workarounds are a downstream Collect node plus a second Code node, or nothing. Two streams cannot be merged by arrival. Each invocation is one correlated frame. Interleaving two upstream streams as they arrive — a chat stream plus a control stream, audio chunks plus parameter changes — is not expressible in a body at all. The kernel mode exists; the Code node cannot reach it. Streaming-input execution is shipped machinery: is_streaming_input nodes implement run(inputs, outputs) and drain their inbox via NodeInputs.stream() / .any() (packages/kernel/src/io.ts, actor path in packages/kernel/src/actor.ts). Collect, the audio synth chain, and FrameToVideo all run this way. The Code node cannot, because the flag is resolved per node type: hydrateGraphNodeFlags and Graph.loadFromDict let the class static (cls.isStreamingInput, false for CodeNode) override anything saved on the node, deliberately, so stale saved flags cannot survive a type migration. Whether a Code node streams is a property of its body, and no per-type flag can carry that. Guest contract One new global, stream, mirroring the kernel’s NodeInputs verbs. It is the input-side twin of emit/output: explicit calls, statically visible names, nothing inferred from data shape. for await (const item of stream(name)) // items for handle `name`, until EOS for await (const [handle, item] of stream.any()) // all handles, arrival order const item = await stream.first(name) // next item, or undefined at EOS stream.open(name) // true while upstream can still produce stream(name) returns an async iterable over the values arriving on input handle name, in order, completing at end-of-stream. name must be a string literal naming a connected input handle (see Validation). stream.any() interleaves every connected handle by arrival, yielding [handle, value] pairs, completing when all upstreams end. stream.first(name) takes the next value for one handle, undefined when the stream already ended — for a config value that arrives once. stream.open(name) answers “could more arrive?” without consuming. Values are marshaled exactly like buffered inputs today (JSON deep copy, media as refs readable via media.*). Outputs from a streaming body leave through emit/output only. The legacy return contract cannot apply — there is one invocation and its return value is control flow, per the emit design. inputs.<name> still exists in a streaming body but carries the node’s configured property values (what is typed on the node), never per-item edge data — connected handles are only reachable through stream. Reading a connected handle via inputs.<name> is a validation error naming stream(name) (see Validation). state remains defined but is pointless here: one invocation, so plain local variables already persist across items. Examples: // Running total, live, with a final summary — today: state + Collect + a second node. let sum = 0; for await (const n of stream("numbers")) { sum += n; await emit("running", sum); } await output("total", sum); // Merge two streams by arrival — today: not expressible. for await (const [handle, value] of stream.any()) { await emit("merged", { from: handle, value }); } Mode selection Automatic from the body, exactly like the emit contract: a textual probe usesStreamInputContract(code) in packages/node-sdk/src/code-body.ts, matching a free call/member of stream outside strings and comments — the sibling of usesEmitOutputContract and shaped the same way, textual because it must answer before the parser, on bodies that may not parse. Probe true → the node executes in streaming-input mode (run()). Probe false → today’s buffered per-item invocation, byte-for-byte unchanged. No checkbox, no new node type. The body is the declaration. Flag hydration: per-instance resolution The kernel actor trusts node.is_streaming_input to pick run() over per-item invocation, and both hydration paths currently stamp it per type. The change is one optional class hook: // BaseNode static, node-sdk static resolveStreamingInput?(node: { properties?: Record<string, unknown>; }): boolean; CodeNode implements it as usesStreamInputContract(String(node.properties?.code)). Consulted at both stamp sites: hydrateGraphNodeFlags (packages/node-sdk/src/registry.ts): is_streaming_input: (cls ? cls.resolveStreamingInput?.(node) ?? cls.isStreamingInput : meta?.is_streaming_input) ?? node.is_streaming_input ?? false. Graph.loadFromDict (packages/kernel/src/graph.ts): the resolver’s per-type descriptorDefaults cannot carry a per-instance answer, so createGraphNodeTypeResolver’s resolved shape gains an optional resolveInstanceFlags(node) => { is_streaming_input?: boolean } closure the merge site calls with the raw node; its answer takes the slot descriptorDefaults.is_streaming_input holds today. Kernel stays registry-agnostic — it calls a function it was handed, as it already does for type resolution. Every existing per-type node is untouched: no hook means today’s exact resolution. The saved node.is_streaming_input keeps its current role (applies only when the registry has no opinion), so a stale saved flag still cannot survive an edit that removes the last stream call — the hook re-reads the body every hydration. is_streaming_output needs nothing: CodeNode overrides genProcess, so hasStreamingOutput already answers true. Host mechanics CodeNode gains run(inputs, outputs, context), which the actor calls when the hydrated flag is true. It starts runInSandbox once and bridges both directions: Guest → host (outputs): unchanged — onEmit routes each bag through outputs.emit(name, value) live; recorded output finals post as one bag after the run succeeds. The bounded emit channel and its backpressure carry over from the genProcess pump. Host → guest (inputs): one new awaitable bridge call, __takeInput(handle | null), behind the stream prelude. The host keeps one inputs.stream(handle) generator per handle (created lazily) plus one inputs.any() generator for stream.any(), and answers each call with {done} or {value} ({handle, value} for any). Concurrent takes on one generator serialize on a promise chain; distinct handles proceed in parallel, same as concurrent fetch calls today. stream.open(name) maps to inputs.hasStream(name) || inputs.hasBuffered(name). Backpressure is free. The guest pulls; an item the body has not asked for stays in the kernel inbox, which already buffers for slow consumers. No channel, no cap. Timeout must not count waiting. A streaming body legitimately lives as long as its upstream. The sandbox already has the mechanism: a SandboxClock the host suspends during round-trips it cannot hurry. The input bridge suspends the clock while a take is parked on an empty, still-open inbox and resumes it when the value (or EOS) arrives — the body’s compute budget (timeout, default 30 s) is spent only on its own execution. The suspend allowance for input waits is unbounded: the run’s own cancellation is the lifetime bound, and a stream that never ends is the upstream’s bug, surfaced by the runner, not a reason to kill a correct consumer. Cancellation: run cancel aborts the existing sandbox signal; the inbox iterators observe NodeInputs.signal and end; parked takes resolve as done. Lineage: NodeInputs records the most recently consumed envelope per handle, and the actor’s existing streaming-input rule gives outputs.emit() that inherited lineage — the same treatment the unmigrated stream filters (Take, Drop, Filter) get today. The guest API stays data-only; per-envelope forward() semantics are out of scope. Semantics Situation Behavior stream("a") on a connected handle yields each arriving value in order, completes at EOS stream("a") on an unconnected handle validation warning; at runtime completes immediately (empty) stream.any() [handle, value] pairs by arrival; completes when all upstreams end stream.first("a") after EOS undefined two concurrent stream("a") iterations items are distributed, not duplicated (one inbox iterator per handle); validation warns emit between takes delivered downstream immediately, as today body ends with items unread remaining items are dropped with the invocation, as for any streaming-input node error thrown mid-stream already-emitted values stand, output finals dropped, node fails — same as emit design timeout counts guest execution only; time parked on input is clock-suspended run cancelled while parked take resolves done, body unwinds, sandbox aborts streaming body under --supervise unchanged: the kernel already stamps streamingInput on the escalation context and computes the allowed-verdict set accordingly Validation and analysis All in code-analysis.ts / code-node-validation.ts (node-sdk), shared by the graph validator, submit_code, validate_code, and the editor: stream joins SANDBOX_GLOBALS; a bare read stops being a ReferenceError candidate. stream(<literal>) / stream.first(<literal>) / stream.open(<literal>) naming a handle the node does not declare is an error, symmetric with the undeclared-inputs.<name> read check. A non-literal argument downgrades to a warning (“cannot check stream names statically”). stream(name) on a declared but unconnected handle is a warning (it runs, but yields nothing) — decidable because validateCodeNodeBody runs inside validateGraph with the edges in hand. In a streaming body, inputs.<name> where name has an incoming edge is an error naming stream(name) — the one footgun the split contract creates, closed statically. A streaming body on the legacy return/yield contract is an error: a body that calls stream must send outputs through emit/output. (Probe order: usesStreamInputContract implies the emit contract is required, whatever usesEmitOutputContract says.) Buffered bodies validate exactly as today. Each new check lands with an inverted fixture proving it can fail, per the repo rule. Surface parity run_code / test_code (packages/agents/src/capabilities/code.ts): a case gains optional inputStreams: { [handle]: unknown[] }; the harness feeds them through an in-memory inbox, and streamed outputs land in the existing streamed array. validate_code gains the checks above. A streaming body that runs in the harness runs the same way in a workflow — the body-shaping rules stay in node-sdk, shared. CodePlanner / code-gen eval: the prompt teaches stream; the suite gains a windowing case and a merge-by-arrival case. Editor: the Code node description documents stream; the assistant dialog prompt teaches it. The editor’s streaming badge and edge rendering read the hydrated flag; the web bundle computes it with the same usesStreamInputContract probe (code-body.ts is dependency-free and already browser-bundled). DSL (sandbox-dsl and workflows export-dsl): the code(...) helper stamps is_streaming_input from the probe so withExplicitNodeFlags graphs run correctly without registry hydration. Docs: docs/javascript-sandbox.md gains § Input streams. Limits No new caps. Input volume is bounded by upstream producers; take rate is bounded by the body’s own timeout-metered execution; emit keeps its existing channel bound and MAX_EMIT_CALLS. Backwards compatibility Purely additive. A body that never mentions stream hydrates, validates, and executes byte-for-byte as today — buffered per-item invocation, state, legacy return contract included. There is no migration, no codemod, and no deprecation: buffered mode remains the right contract for per-item transforms, which most Code nodes are. stream is for the bodies that today need state, a Collect detour, or are not expressible at all. Testing Unit (packages/code-nodes): ordered delivery per handle; any() interleave across two handles; EOS completes the loop; first after EOS; emit-while-parked interleave; unread items dropped at body end; error mid-stream drops finals; cancellation unparks; clock suspension (a body with a 1 s timeout survives a 5 s input gap, and still dies after 1 s of its own compute). Hydration (packages/node-sdk, packages/kernel): a graph node whose code streams hydrates is_streaming_input: true through both hydrateGraphNodeFlags and loadFromDict; editing the stream call out flips it back; saved stale flags still lose; every hook-less node type hydrates exactly as before (pinned against current fixtures). Analysis (packages/node-sdk): each new rule red on an inverted fixture — undeclared stream name, connected-handle inputs. read, streaming body on the legacy contract, unconnected-handle warning. End-to-end: a workflow where a generator streams into a Code node running a windowed aggregate into a Preview runs under nodetool debug and shows per-item emit messages arriving while upstream is still producing; the same body passes test_code with inputStreams. Deviations from the design as built Finals post through outputs.emitGroup(finals), not slot-by-slot emit. One invocation’s finals are one frame, so sibling handles share minted lineage instead of arriving as N independent items. run() called for a body that does not stream throws. It can only happen when a stale saved flag was trusted without hydration, and the path never delivers per-item invocations — a buffered body would silently see none of its connected inputs, so failing names the misconfiguration instead. stream.open reads hasBuffered through an optional method. StreamingInputs (packages/runtime/src/node-executor.ts) gained hasBuffered?(name); the kernel’s NodeInputs already had it, and a hand-rolled test double need not. run_code / test_code name the field input_streams (wire names in this surface are snake_case) and define stream.any() order as round-robin by index across the handles in declaration order — a live inbox is observed, pre-staged items have to be ordered by rule. The DSL probe lives in createNode (packages/dsl/src/core.ts), gated on the Code node’s type, rather than in the generated helper: @nodetool-ai/dsl runs graphs through withExplicitNodeFlags with no registry hydration, and the generator stamps flags per type. Every other surface — the server run path, Graph.loadFromDict, ExecutionSession — hydrates against the registry and needs nothing. sandbox-dsl likewise needs nothing: its graphs reach execution through create_workflow / run_workflow, which hydrate. Out of scope CodeAct (execute_code) — acts on the toolbelt, returns one result; no inbox exists there. Per-envelope lineage in the guest — forward()/drop() equivalents and Zip/Cross-style scope reads stay host-side; the guest sees data. Windowing/batching helpers (stream.window(n), debounce) — expressible in plain JavaScript over stream(); sugar can come later without contract changes. Python Code nodes — the worker’s execution model is separate; this contract is the QuickJS guest’s.--- # CodeAct Execution — Design Source: https://docs.nodetool.ai/codeact-design.html CodeAct Execution — Design Status: the only agent execution mode. There is no switch. Code: packages/agents/src/codeact/. The engine every action runs on: JavaScript Sandbox. What this is The action space of the agent step loop. The older loop had the model emit one JSON tool call per action, the host execute it, and the result come back as a tool message — one round trip per tool. Under CodeAct the model acts by writing a JavaScript program; the program runs in the QuickJS sandbox where the same toolbelt is imported as async functions (import { web_search } from "@nodetool-ai/sandbox-nodetool/web"), and one round trip can chain, loop over, branch on, and post-process any number of tool calls. The program’s output — return value, console logs, thrown error — comes back as the observation for the next turn. The research this follows: CodeAct (Wang et al., ICML 2024, arXiv:2402.01030) — executable code as the action space beats JSON/text tool calling: up to +20% success, ~30% fewer turns. The core loop here (code action → execution observation → repair) is that paper’s. CaveAgent (arXiv:2601.01569) — a persistent runtime across turns (stateful objects survive between actions) adds +5–13.5% success and ~28% fewer tokens on data-heavy tasks. An in-sandbox state object played this role at launch; it is gone now — process-local, LRU-capped per thread on the chat runner, and silently lossy across restarts. Durable carry is thread memory (nodetool.memory.*) plus the asset:// refs generation calls already return. MCP design-choice study (arXiv:2602.15945) and Anthropic’s Code execution with MCP (Nov 2025) — decoupling tool results from model context is where the token savings come from (their headline example: 98.7% reduction). A CodeAct program can fetch a large payload, reduce it in the sandbox, and surface only the reduction; in tool mode the whole payload transits the transcript. To Run or Not to Run (arXiv:2606.26978) — execution isn’t free; there are regimes where restricting it saves cost with little accuracy loss. The codeact eval suite is where that cost/benefit stays measurable. What already exists (and is reused unchanged) Piece Where Role here QuickJS WASM sandbox packages/agents/src/js-sandbox.ts (runInSandbox) Executes every action. All its limits (30 s timeout, 64 MB heap, fetch caps, output truncation, SSRF guard, workspace containment) apply per action. Tool base class + registry src/tools/base-tool.ts, tool-registry.ts The toolbelt is the same Tool[] a step has always been given — codeact adds no capability of its own. Provider loop BaseProvider.generateLoop Drives the turn loop; codeact presents exactly one provider tool. Result-schema validation src/utils/json-schema-validate.ts finish(result) validates host-side with the same checker finish_step uses. Never-reject bridge convention js-sandbox.ts Host bridges resolve {ok, ...} envelopes; a guest prelude re-throws. Required by the QuickJS handle-leak workaround. Agent memory context.memory Step/task results land under the same keys; memory tools are in the toolbelt as functions like everything else. CodeAct replaced script mode. ScriptRunner had an LLM-authored orchestration script spawn sub-agents through agent(); an action does the same thing with ordinary control flow and nodetool.agents.run(prompt), in the same sandbox, without a second planner or a second set of budget knobs. The action protocol The model sees one provider tool: execute_code({ code: string }) Code actions arrive through a tool call rather than fenced text because every provider adapter already delivers tool calls reliably; scraping code blocks out of prose is exactly the fragility tool calling was invented to avoid. The CodeAct paper’s gains come from the action space being code, not from the transport being free text. Inside the sandbox, on top of the standard surface (console, fetch, workspace, crypto, data, format, …), the action gets: import { <name> } from "@nodetool-ai/sandbox-nodetool/<namespace>" — one async function per tool in the step’s toolbelt, reached by import; there is no tools.<name>() global. A name a capability module owns comes from that namespace, and a tool the session added at its call site comes from .../session. Calls bridge to Tool.executeTool on the host. A tool that returns an {error} payload throws in the guest, so try/catch is the error-handling idiom. Per-action tool-call cap (maxToolCallsPerAction, default 50) so a runaway loop can’t drain budgets silently. Memory — the durable carry between actions and turns. There is no cross-action variable bag: local variables die with the action, return is the observation only. Generation calls save their result as an asset and return its asset:// uri; an action records what later actions or turns need with nodetool.memory.save (shown back to the model at turn start). Fetch or generate once, reuse every turn; never re-fetch or re-generate to re-look at something. finish(result) — completes the step. For schema’d steps the host validates against the declared schema; an invalid result throws in the guest with the violation list, so the same action can repair and retry, or the failure becomes the observation for the next action. Valid finish ends the provider loop (AbortController, same mechanism as finish_step). The action’s return value — becomes part of the observation. Returning a small summary of big intermediate data is the context-decoupling move; the prompt says so explicitly. The observation sent back as the tool result is a JSON envelope: { ok, result?, error?, stack?, logs?, finished?, toolCalls } truncated by the same truncateToolResult cap as any tool result (20 000 chars). toolCalls is the count consumed, so the model can see budget burn. Completion semantics (identical contract to StepExecutor) Schema’d step: only a schema-valid finish(result) completes. Iterations exhausted → explicit failed step, never a silent guess. Unschema’d step: finish(...) works, and a plain assistant message with no tool call also finalizes (its text is the result) — the same prose-mode rule the tool-mode executor has. Failure reporting, memory writes (step:<id>, task:<id> with useFinishTask), and the ProcessingMessage stream (task_update, step_result, tool_call_update, chunk) are byte-compatible with StepExecutor, so every consumer — CLI tree, web ExecutionTree, script runner, supervisor — works unchanged. Each host-bridged tool invocation is surfaced as a tool_call_update (id codeact_<n>), so observability keeps per-tool granularity even though the provider transcript only carries execute_code. Prompting buildCodeActSystemPrompt renders: The action contract (write code, observe, repair; memory carry; keep observations small — return summaries, save payloads as assets or in memory). The tool catalog as typed signatures, generated from each tool’s JSON schema (await browse({url: string, timeout?: number}) + first sentence of the description), grouped under the import statement that makes each group callable — and only for the resident set. The high-traffic tools nearly every step reaches for (the whole search family — web_search, search_nodes, run_search, asset_search, grep, glob — plus the Claude-agent file set (read_file, write_file, edit_file, list_directory), browser, HTTP, memory, run_subtask — CODEACT_RESIDENT_TOOL_NAMES, overridable per executor) stay fully documented; once the belt exceeds CODEACT_DEFER_THRESHOLD (16), everything else is listed by name only and discovered in-sandbox via await nodetool.searchTools("query"), which reuses the ToolSearch query grammar (select:, keywords, +substr) and returns each match’s signature and description. Deferred tools remain callable — the split spends prompt tokens, not capability. This is the progressive-disclosure half of the Anthropic MCP result. A condensed sandbox API reference (what exists beyond the imports, what is blocked, the key limits) derived from the same manifest the Code-node prompt uses, so it cannot advertise an API the sandbox doesn’t marshal. The output-schema section for schema’d steps. Caller-supplied system prompts remain preambles, exactly as in StepExecutor.buildSystemPrompt — they cannot override the execution contract. Security posture The action executes with the same privileges tool mode already grants: Every imported function is a tool the model could have called directly; the bridge adds no capability. Per-step tools allow-lists stay a privilege boundary — a codeact step only sees its allowed subset. The sandbox’s own limits bound the new part (arbitrary computation): CPU via interrupt handler, heap, fetch count/size/SSRF guard, workspace containment, no eval/Function, no module loader. The genuinely new risk (per the MCP design-choice study) is composition: one action can chain tool calls without per-call visibility in the provider transcript. Mitigations: per-action tool-call cap, per-invocation tool_call_update events (nothing becomes invisible to the host), and the 30 s default action timeout. finish validation is host-side; the guest cannot forge a completed step. Integration surface Every step executor is a CodeActExecutor: Agent → ParallelTaskExecutor → TaskExecutor construct one per step, and run_subtask / run_search spawn their children the same way. There is no option, setting, or flag selecting an action space. Two callers stay on StepExecutor, the older JSON-tool-call loop, and it is no longer exported from the package: SupervisorAgent and the app-build spec stage. Both are one-shot structured verdicts on a fail-closed path, where a sandbox error would add a failure mode and code actions buy nothing. Chat turns (websocket runner) A chat turn with tools presents execute_code (plus view_image, the one channel that puts pixels into context and so cannot ride the JSON observation envelope) instead of the toolbelt; discovery is the in-sandbox nodetool.searchTools(). The adapter is createChatCodeActSession (packages/agents/src/codeact/chat-codeact.ts): a chat toolbelt mixes server tools with client (ui_*) tools that exist server-side only as schemas, so instead of buildToolBridge the session routes every imported belt name to the chat runner’s own executeTool router — permission gating, client round-trips over the ToolBridge, and asset materialization all stay where they are. There is no finish() — a plain assistant message ends the turn, and the prompt says so (variant: "chat" of buildCodeActSystemPrompt). A gated tool call parks the running program on the user’s answer, so the turn carries a sandbox clock (createSandboxClock): the runner suspends it for the length of every tool- and plan-approval round trip, and the action resumes with the budget it had when it asked. Without it the action’s wall clock ran through the prompt and killed the program mid-wait — the dialog was still on screen, and answering it resolved nothing. Tests: packages/websocket/tests/chat-codeact-approval.test.ts. Auto mode reads the action’s declared risk Auto mode used to mean “every call runs, no prompts”, which made a code action the one place a user could lose work they never agreed to lose: the per-call ladder answers allow for every category, so a program that deletes a workflow ran exactly like one that counted rows. So execute_code carries one more required option, risk ("low" | "high"), next to title and code. The model declares it; the host reads it before a line of the program runs (admitCodeAction, packages/agents/src/codeact/execute-code-contract.ts). In auto mode a low action runs unattended and a high one asks once — for the whole action, because the action, not the bridged call, is what the user is being shown. A denied action never runs, and the refusal is the observation. “Allow for this chat” is keyed on execute_code itself, so it stops the asking for the rest of the thread. It fails closed: a call with no risk, or one carrying anything outside the enum, reads as high. The program’s own imports set a floor under the declaration (importedActionRisk): a static import of an execute or external capability — the calls that spend money or leave the account — is high whatever the call said, and a namespace import of a capability module takes that module’s highest class. A write import stays at the declared risk, because a note the user asked for and a delete share the class and only the model can tell them apart. Plan and default modes are untouched — their per-call ladder already blocks or asks, and a second question per action would only double the prompts. Hosts with no one to ask (the MCP mount, the security monitor’s pass in Agent) carry an always-allow requestApproval, so nothing there can deadlock. Tests: packages/agents/tests/codeact-action-risk.test.ts. Workflow graph editing: the JS object model When the belt carries the ui_* workflow document tools, code actions get an object model instead of one bridged call per mutation (packages/agents/src/codeact/graph-model.ts): const wf = await openWorkflow(workflowId); const input = wf.addNode("in1", "nodetool.input.StringInput", { name: "prompt" }); wf.addNode("llm1", "nodetool.agents.Agent", {}, { x: 400, y: 120 }); wf.connect("in1", "output", "llm1", "prompt"); wf.node("llm1").setTitle("Draft").set({ system: "be brief" }); await wf.commit(); Mutators are synchronous: they update a local mirror (wf.nodes, wf.edges, wf.node(id)) and queue the equivalent ui_* operation. commit() replays the queue through the bridged tools — the same contract the renderer and the headless document tools implement, so routing, validation, and live-editor sync are untouched. A failed commit names the failing operation and keeps it (and everything after it) queued for a retry; removing a never-committed node cancels its queued ops instead of issuing a delete. Tests: packages/agents/tests/chat-codeact.test.ts. The platform as objects: nodetool.* Alongside the graph model, an action gets the nodetool object model (packages/agents/src/codeact/nodetool-api.ts): namespaces wrapping belt tools, so gating and routing stay untouched and a method whose backing tool is absent throws naming it. The namespaces are workflows, nodes, agents, models, providers, media, documents, web, memory, style, email, assets, jobs, collections, apps, timelines, sketches, scripts, storyboards, plus batch() for bounded fan-out. Authoring a graph is not one of them. It is a sandbox package, @nodetool-ai/sandbox-dsl: one generated function per node type, carrying that node’s real inputs, one importable module per namespace. A node type the catalog does not have has no export, so a hallucinated type fails at import rather than at validation — the failure mode the string-typed nodetool.graph() builder could not close. workflow(...terminals) returns the {nodes, edges} shape validate_workflow and create_workflow already take. The pack reaches an action through the session allowlist like any other: withGraphDslPackage (src/codeact/graph-dsl-package.ts) adds it when the belt carries create_workflow, validate_workflow and run_workflow and the catalog serves the pack. A machine without it installed gets neither the specifier nor the prompt section naming it. web is the outside world behind one surface: search(query, {provider}) picks whichever search backend the belt carries ("default", "openai", "google", "dataforseo" pin one), with news, images, browse(url), fetch(url), download and screenshot alongside. memory is the conversation’s durable notes (memory_*), style the user’s accumulated taste, email the Gmail three. Every wrapped tool is filtered out of the prompt’s tool catalog (nodetoolApiCoveredToolNames) — it stays callable through the bridge and findable via nodetool.searchTools(), but nodetool.* is its one documented form. Workspace files are the exception on purpose: they go through the sandbox’s own in-process workspace.* API, which costs no tool call, and the prompt’s action contract says so. What the belt does not carry Two shrinks keep the belt to capabilities a model cannot write itself: The pure-computation tools (calculate, geometry, trigonometry, statistics, unit_conversion) are gone everywhere, MCP included. Anything a model can do by writing code is not a tool. The code tools run_code and js went with them: execute_code is the code path, and a second one only offered the model a sandbox without the nodetool.* API. The provider-specific media duplicates — image_generation, openai_image_generation, google_image_generation, openai_text_to_speech — are deleted, because nodetool.media already covers them through the provider-agnostic generate_image / generate_speech. The provider-specific search names went the same way, but their implementations stayed: they are the backends web_search / the single web_search capability routes to, now plain functions rather than tools. Evaluation eval codeact (registered next to subtask): objectives with instrumented tools where the interesting metric is rounds and tool routing, scored structurally (required tools invoked, forbidden ones not, action count within bounds, final result correct): npm run dev:nodetool -- eval codeact -p anthropic -m claude-sonnet-5 Harness tests (tests/codeact-executor.test.ts) drive the executor with a ScriptedProvider — tool chaining in one action, the absence of a state global, schema repair after an invalid finish, error observations, prose finalization — no network, no model. Non-goals (now) Python actions. The sandbox is JS; the CodeAct result is about code as the action space, not about Python specifically. Replacing planners. GraphPlanner/CodePlanner already use code-shaped artifacts; this changes the step execution loop only.--- # Collections Source: https://docs.nodetool.ai/collections.html Collections bundle related documents into a single searchable unit. Index text into a collection, then query it from your workflows and agents. Opening Collections Open Collections from the app menu, or go directly to /collections. The explorer shows the collections you own, and any collection that has no owner recorded, with each one’s document count, embedding model, and linked workflow. Another user’s collection does not appear, and asking for it by name answers “not found”. Creating a Collection Enter a name in Collection Name. Choose an Embedding Model. The Create button stays disabled until both are set. Click Create. Drag files from your computer onto the collection tile to index them. A name is unique across the store. It can be at most 128 characters and cannot start or end with whitespace, contain / or \, or contain control characters. Dropping a file sends it to POST /api/collections/:name/index. The server reads the file as UTF-8 text, splits it into chunks of 2000 characters with 1000 of overlap, and stores each chunk as <file name>#<n>. Use text formats such as TXT, Markdown, HTML source, CSV, or transcripts. The route does not extract text from PDF, DOCX, EPUB, or images, so extract their text in a workflow first (for example with lib.pdf.ExtractText) and index it with an index node. A file over NODETOOL_MAX_UPLOAD_BYTES (1 GiB by default) is rejected. The route does not embed the chunks with the collection’s embedding model. Use the vector.Index* nodes when you need embeddings. See Indexing. Managing Documents Each collection is a tile in the explorer. From the tile you can: Index documents by dropping files onto it. Progress and any per-file errors are reported inline. Link a workflow by clicking the workflow name. The tile shows the workflow’s name, or “No workflow”. The link is stored in the collection’s workflow metadata. Dropping files does not run it. Delete the collection and everything indexed in it. Using Collections in Workflows Use collections in RAG pipelines: Add an index node (vector.IndexString, vector.IndexTextChunk, vector.IndexAggregatedText, vector.IndexImage, vector.IndexEmbedding) or a query node (vector.QueryText, vector.QueryImage, vector.HybridSearch) to your workflow. Set its collection input to a collection. The vector.Collection node gets or creates one by name. Run the workflow. Index nodes write into the collection; query nodes read from it. See the Chat with Docs example for the full query → format → answer wiring. Indexing Options A collection’s embedding model is chosen when you create it and is recorded in the collection’s embedding_model and embedding_provider metadata. The linked workflow can be changed from the collection tile. Chunking is not a collection setting. splitDocument() defaults to 2000 characters with 1000 characters of overlap, and the upload route uses those defaults. See Indexing for deeper tuning notes. Storage By default index data lives in a vectorstore.db SQLite file (via sqlite-vec) in the NodeTool data directory, separate from the workflow database. Set VECTORSTORE_DB_PATH to move it. Nothing leaves your machine unless you use a cloud provider for the embedding model. For multi-user deployments, set NODETOOL_VECTOR_PROVIDER=supabase. See Vector Storage and Supabase Deployment. Related Docs Indexing — the upload route, index nodes, and vector store settings Vector Storage — backends and filters Chat with Docs example — end-to-end RAG workflow--- # ComfyUI Recipes Source: https://docs.nodetool.ai/comfyui-recipes.html A ComfyUI workflow loaded into a NodeTool node keeps doing exactly what it did in ComfyUI. What changes is everything on either side of it: where the prompt comes from, how many times it runs, where the files go, and who presses the button. This page is that second half. For the three nodes and their properties, see ComfyUI. For getting a server the node can reach, see ComfyUI Setup. The shape of every recipe A loaded workflow becomes a node with typed handles. Every dynamic input is keyed <comfyNodeId>:<field> and takes the value ComfyUI would otherwise read from the prompt JSON. Every dynamic output is keyed <comfyNodeId>:<kind> and carries a NodeTool media ref, one per file. The ids are the keys of your API-format export, so a CLIPTextEncode saved as node 6 exposes 6:text, a LoadImage saved as node 10 exposes 10:image, and a SaveImage saved as node 9 emits on 9:image. Which fields are offered, and which are never offered because ComfyUI wires them internally, is What becomes an input or an output. Two facts shape every recipe below: A handle only exists if you asked for it. Media inputs, prompt text, and save outputs are derived automatically. Seeds, steps, CFG, and sizes are offered as a checkbox list in the Load Workflow dialog. Until you tick one there is no handle, and the prompt keeps its exported value. See Loading a workflow. The node runs on the server, not in your browser. All three ComfyUI nodes are server-side, so endpoint has to resolve from wherever the NodeTool backend runs. A 127.0.0.1:8188 that works on your laptop is not the same address to a cloud install. Recipe 1 · A model writes the prompt What it does. The positive prompt is generated per run instead of typed into the ComfyUI graph, so one saved workflow covers a brief rather than a sentence. Nodes. nodetool.input.StringInput, nodetool.agents.EnhancePrompt, lib.comfy.RunWorkflow, nodetool.output.Output graph LR brief["StringInput (Brief)"] enhance["EnhancePrompt (target: image)"] comfy["RunWorkflow"] out["Output"] brief --> enhance -->|text| comfy comfy -->|"9:image"| out Wiring. EnhancePrompt’s text output goes to the comfy node’s 6:text handle. nodetool.agents.Agent works the same way when you want tools or a system prompt, and its text output is the one to wire. Watch out. A negative prompt is a second CLIPTextEncode with its own id and its own handle. Leaving it unconnected keeps its exported text, so driving only the positive one is the usual arrangement. Recipe 2 · One workflow, many runs What it does. The same diffusion graph runs once per item, unattended. Nodes. nodetool.input.StringInput, nodetool.generators.ListGenerator, lib.comfy.RunWorkflow, nodetool.control.Collect, nodetool.output.Output graph LR brief["StringInput (Brief)"] list["ListGenerator (streams item)"] comfy["RunWorkflow"] collect["Collect"] out["Output"] brief --> list -->|item| comfy comfy -->|"9:image"| collect --> out Wiring. ListGenerator emits item (a string) once per generated line, and each item drives one run of the comfy node. nodetool.control.ForEach does the same for a list you already have, and nodetool.image.LoadImageAssets does it for a folder of image assets, streaming image and name per file. nodetool.control.Collect gathers the results back into one list. Watch out. Every run gets its own timeout, not a shared one, so a fifty-item batch on a 600 second timeout can take hours before anything fails. On Comfy Cloud, the plan’s concurrent-job limit (1, 3, or 5) is the real width of the fan-out: a full queue answers 429, the SDK retries for 60 seconds, and then the node fails with Comfy queue is full. Recipe 3 · A stored asset in, a stored asset out What it does. Feeds a NodeTool image asset into a LoadImage node and puts what comes back into an asset folder. Nodes. nodetool.input.ImageInput, lib.comfy.RunWorkflow, nodetool.image.SaveImage graph LR input["ImageInput"] comfy["RunWorkflow"] save["SaveImage (folder, name)"] input -->|"10:image"| comfy comfy -->|"9:image"| save Wiring. A connected image, audio, or video ref is uploaded to the ComfyUI server before the prompt is submitted (POST /upload/image, which ComfyUI accepts for any input file) and the stored filename is substituted into the prompt. On Comfy Cloud the same ref goes through the SDK’s asset API and is deduplicated by a blake3 hash of the bytes, so re-running with the same file uploads nothing the second time. Watch out. Outputs arrive as media refs carrying the bytes, not as saved assets. Nothing is written to your asset library until a node writes it, which is what nodetool.image.SaveImage is for (nodetool.video.SaveVideo for video). And a LoadImage field you leave as a literal filename still resolves against the ComfyUI server’s own input directory, which is why an unwired workflow keeps working and a wired one replaces the file. Recipe 4 · Frames into a cut What it does. A batch of stills out of ComfyUI becomes a video without a second tool. Nodes. lib.comfy.RunWorkflow, nodetool.video.FrameToVideo, nodetool.output.Output graph LR comfy["RunWorkflow (SaveImage batch)"] frames["FrameToVideo (fps)"] out["Output (Video)"] comfy -->|"9:image"| frames --> out Wiring. The comfy node emits one media ref per output file on the same slot, so a batch of four images is four frames rather than one list. nodetool.video.FrameToVideo takes a streamed input and collects them into a video at the fps you set. For a timeline instead of a flat encode, send the slot through nodetool.control.Collect into nodetool.timeline.AddClips (which has an image_duration_ms property) and render it with nodetool.timeline.RenderTimeline. Watch out. Nodes ComfyUI serves from its cache never emit an executed event, so their files are reconciled from /history/<prompt_id> after the run rather than streamed during it. A fully cached batch therefore arrives all at once at the end. Frame order follows emission order, so a graph with several save nodes gives you several slots to wire separately rather than one ordered stream. Recipe 5 · Behind a form, in a mini app What it does. Puts the workflow in front of someone who should not have to read a node graph, or install ComfyUI. How. Save the graph, open its app tab’s Design view, add fields wired to the workflow’s Input nodes, add a Button whose On click action is Run workflow, and wire an Image result widget to the Output node carrying the comfy slot. App Builder covers the editor, and Mini Apps covers how one runs. Watch out. Widgets wire to Input and Output nodes, not to comfy handles, so every parameter the form should expose needs its own nodetool.input.* node in front of the handle. Publishing locks in the current state of every workflow the app runs, so a later edit to the graph does not reach the published app until you publish again. Recipe 6 · Ask for it in chat What it does. Reaches a saved ComfyUI workflow by name instead of by tab. How. Save the workflow in the editor, then select it in chat or ask for it by name. The results land in the thread. See Chat & Agents. Watch out. The agent runs the graph as saved, so whatever the ComfyUI node holds in its workflow property is what runs. This is the same server-side run as every other path, which makes it the quickest check that endpoint is reachable from the backend rather than only from your machine. Running it without the editor The CLI harnesses validate first, because it costs nothing: npm run dev:nodetool -- validate <workflow_id> npm run dev:nodetool -- validate comfy-graph.json --json It checks the NodeTool graph: unknown node types, missing required properties, dangling or mis-typed edges, and model properties naming a provider or a model id that does not exist (which is what catches a bad model on the EnhancePrompt node in Recipe 1). It never opens a socket to ComfyUI, so everything inside the workflow string is invisible to it. A missing checkpoint, an uninstalled custom node, and a class_type that server does not have all pass validation and fail at submit with Submit failed (400). An empty workflow property passes too, and the node raises ComfyUI workflow is required at run time. Then debug, which runs the graph and keeps everything it emitted: npm run dev:nodetool -- debug <workflow_id> npm run dev:nodetool -- debug comfy-graph.json --params '{"brief":"a red bicycle"}' npm run dev:nodetool -- debug <workflow_id> --trace The bundle’s server/messages.jsonl holds the ComfyUI lifecycle the node forwards into the run log: execution start, the count of reused nodes, the class name of each executing node, sampler progress, and the error text ComfyUI returned. That is the ComfyUI console without a ComfyUI browser tab. --watch re-runs a file target on every save and prints a diff of the verdict. To just run it: npm run dev:nodetool -- workflows run <workflow_id> --params '{"brief":"a red bicycle"}' npm run dev:nodetool -- run comfy-graph.ts --json With --json, an image or video payload over 64 KiB is written to nodetool-output/<job_id>/payload-N.<ext> and appears as a {"$file": …} reference instead of being inlined. Flag reference: CLI › validate, CLI › debug, Harness reference. The DSL packages/dsl/src/generated/lib.comfy.ts exports one function per node: runWorkflow, runWorkflowOnWorker, and runWorkflowOnCloud. A file like this runs with nodetool run: import { libComfy, output, workflow } from "@nodetool-ai/dsl"; const prompt = { "3": { class_type: "KSampler", inputs: { seed: 42, steps: 20, cfg: 7, model: ["4", 0] } } }; const comfy = libComfy.runWorkflow({ endpoint: "127.0.0.1:8188", workflow: JSON.stringify(prompt), timeout: 600 }); export const comfyRun = workflow( output.output({ name: "result", value: comfy.output() }) ); Dynamic inputs type-check, dynamic outputs do not resolve. Because the node supports dynamic inputs, RunWorkflowInputs names the four static properties (endpoint, api, workflow, timeout) and also has a [name: string]: unknown index signature, so a "6:text" key compiles. A literal under that key is stored as a dynamic property and injected into prompt["6"].inputs.text at run time. An upstream node’s output handle under a key such as "10:image" becomes an edge into that handle, which is how a NodeTool media ref reaches a Load* node. The generated function declares one output slot, output, which carries the raw ComfyUI history (the job manifest when api is v2). It declares none of the <comfyNodeId>:<kind> slots, because they come from a workflow the generator has never seen, and comfy.output("9:image") throws Unknown output slot '9:image'. A DSL script therefore cannot select the per-file image, audio or video slots. Two ways around it: Read output when the script only needs ComfyUI’s history payload. Author the graph in the editor when downstream nodes need the media on a save slot. Run the saved graph by id from the CLI or the API. The API A saved workflow containing a ComfyUI node runs like any other: curl -X POST "http://localhost:7777/api/workflows/YOUR_WORKFLOW_ID/run" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{"params": {"brief": "a red bicycle"}}' The response is one JSON object with job_id, workflow_id, status, outputs, error, message_count, and background. outputs is keyed by output-node name, so the comfy slot reaches the caller under whatever you named the nodetool.output.Output node wired to it. params is keyed by input-node name, not by comfy handle. A workflow whose 6:text is fed by a literal takes no parameter at all. Put a nodetool.input.StringInput in front of the handle and its name becomes the key. The route does not stream, so per-node progress and the ComfyUI lifecycle lines need the /ws WebSocket instead. Full shapes: API Reference, WebSocket API. Cost, time, and failure Cost. No ComfyUI run writes a provider cost record, on any of the three nodes, so nodetool costs shows nothing for one. A direct or worker run is billed by whoever owns the GPU. A Comfy Cloud run is billed per GPU second by plan, and Comfy’s v2 job response carries no cost field, so the node records the job id and no charge amount. The run log line Comfy job <id> submitted is the join key against Comfy’s own billing. See Known limitations. Time. All three nodes default timeout to 600 seconds and each bounds the NodeTool side of the run. Raise it for video and upscale graphs. The message names which side gave up: Timeout waiting for ComfyUI result on the direct native path, ComfyUI workflow did not finish within <n>s on a v2 path, and Comfy Cloud job did not finish within <n>s on Cloud, whose own cap is 30 minutes per job (60 on Pro) regardless of what you set here. Cancellation does not reach ComfyUI the same way everywhere: Path What a cancelled NodeTool run does RunWorkflow, api: native Posts /interrupt and closes the WebSocket, then fails with ComfyUI execution was canceled RunWorkflow, api: v2 Calls the job’s cancel endpoint, so the job stops server-side RunWorkflowOnCloud Calls the job’s cancel endpoint, so Comfy stops billing the run RunWorkflowOnWorker, v2 path Calls the job’s cancel endpoint RunWorkflowOnWorker, bridge path Makes no cancel call. The bridge connection closes when the call settles, and the prompt runs on inside the worker’s ComfyUI The bridge path is the one every shipped worker takes today, so treat a long-running worker run as uncancellable and set timeout accordingly. The error message table for everything else is Troubleshooting. Related ComfyUI: the three nodes, their properties, handles, outputs, and limitations ComfyUI Setup: getting a server the node can reach Creative Cookbook: the same fan-out, collect, and render shapes, without ComfyUI in the middle CLI Reference: every flag on validate, debug, run, and workflows run Harness Reference: what each harness simulates and what it does not App Builder: the Design view Recipe 5 uses API Reference: endpoint matrix, auth, and the run payload--- # ComfyUI Setup Source: https://docs.nodetool.ai/comfyui-setup.html You have a ComfyUI workflow in API format and a NodeTool graph to put it in. What is missing is a server to run it on. This page covers getting that endpoint and what has to be installed on it. The nodes themselves, their properties, the workflow loader and the output slots are in ComfyUI. Choosing a path Path Node Who holds the models Cost Reachable by ComfyUI on your own machine lib.comfy.RunWorkflow You, on your disk Electricity Nothing outside your machine, when it stays on loopback ComfyUI on a rented GPU you expose lib.comfy.RunWorkflow You, on the pod’s volume Per minute while the pod is up Anyone with the URL NodeTool ComfyUI worker on a rented GPU lib.comfy.RunWorkflowOnWorker You, on the pod’s volume Per minute while the pod is up The bearer token holder Comfy Cloud lib.comfy.RunWorkflowOnCloud Comfy Per job, from your Comfy account Comfy’s API key holder Three questions usually settle it: Do you have a GPU that fits the workflow? If yes, run ComfyUI locally and point the direct node at it. Nothing leaves the machine and there is no start-up wait. Do the weights have to stay yours? A rented GPU keeps them on a volume you control. Comfy Cloud does not: you submit a graph and Comfy supplies the models. Is the endpoint allowed to be public? A pod’s proxy URL is reachable by anyone who has it, since the direct node sends no credential. The NodeTool ComfyUI worker keeps ComfyUI on loopback inside the container and puts a bearer token in front of it. Cold starts differ. A local ComfyUI is already warm. A rented pod pays for boot plus whatever model downloads its volume still needs. Comfy Cloud has no boot, but a full queue answers 429 and the plan’s concurrent-job limit is 1, 3, or 5. A ComfyUI on your own machine lib.comfy.RunWorkflow has an endpoint property, defaulting to 127.0.0.1:8188. A bare host:port gets http:// prepended. A full http:// or https:// URL is taken as given. The process at that address has to serve ComfyUI’s HTTP API and its WebSocket on the same port. NodeTool submits to /prompt, follows the run on /ws, downloads finished files from /view, reconciles anything it missed from /history/<prompt_id>, uploads connected media through /upload/image, and posts /interrupt when you cancel. A stock ComfyUI serves all of it on one port. See ComfyUI’s documentation for how to start it. No credential is sent on this path, so the endpoint has to accept an unauthenticated submit. Where 127.0.0.1 is resolved from The ComfyUI nodes are server-tagged (node, workers, edge, never browser), so the submit is made by the NodeTool backend, not by your browser tab. 127.0.0.1:8188 means “port 8188 on the machine running the NodeTool server”. In the desktop app the bundled backend runs on your machine, so a local ComfyUI is reachable with the default. If your NodeTool server runs somewhere else, a Docker container or a self-hosted box, then that host is what must reach ComfyUI, and loopback there is its loopback. ComfyUI on another machine on the LAN A ComfyUI bound to loopback is unreachable from any other machine, including the one running NodeTool. ComfyUI’s flag for binding a wider interface is --listen, and --port moves it off 8188. Both belong to ComfyUI, not to NodeTool: see ComfyUI’s documentation for the exact syntax and for install steps. Once it listens, set endpoint to that machine’s host:port. Anyone who can reach the address can submit prompts to it, so keep it on a network you trust. A rented GPU Expose ComfyUI yourself Run ComfyUI on a pod and reach the port from outside: a RunPod pod’s HTTP proxy URL, a Cloudflare tunnel, an SSH tunnel. Put the resulting https:// URL in endpoint and use the direct node. The scheme is derived from the endpoint, so a https:// address upgrades the WebSocket to wss:// on its own. This publishes ComfyUI to whoever holds the URL. There is no Authorization header on the direct node’s requests, so the proxy URL is the only secret, and anyone with it can queue prompts, read /history, and download outputs. Put a tunnel with its own auth in front of it, or use the worker below. The NodeTool ComfyUI worker Deploy ghcr.io/nodetool-ai/nodetool-worker-comfy:latest, which runs a co-located ComfyUI that stays loopback-only inside the container, and drive it with lib.comfy.RunWorkflowOnWorker. The node talks to the worker’s bridge at worker_url (default ws://127.0.0.1:7777/ws, use wss:// for a proxied pod), authenticated with the worker’s bearer token in the worker_token property. ComfyUI’s own ports are never published. Provisioning, the profile and instance model, and the idle-stop and TTL guards that stop a forgotten pod billing are in Worker Deployment. Pick the ComfyUI image from the Worker image preset dropdown, or pass --image ghcr.io/nodetool-ai/nodetool-worker-comfy:latest to nodetool worker create or nodetool worker profile add. The worker node has no workflow loader in the editor, so its workflow property and dynamic handles are filled in by hand. That and the bridge path’s slot naming are in Known limitations. Comfy Cloud No GPU and no ComfyUI install. lib.comfy.RunWorkflowOnCloud submits the same API-format prompt to https://cloud.comfy.org. Get a key at platform.comfy.org. Store it as COMFY_API_KEY, either on the Comfy Cloud card in Settings → Models & Providers or with nodetool secrets store COMFY_API_KEY, which prompts for the value and writes it to the encrypted secret store. The node declares COMFY_API_KEY in requiredSettings, so the editor badges it as unconfigured until the key is there, and a run without it fails with COMFY_API_KEY is required to run a workflow on Comfy Cloud. The same key is sent with the workflow as extra_data.api_key_comfy_org, which is what authenticates any partner (API) node inside the graph. One key covers both the job and the API nodes in it. Comfy’s own limits apply to the run: 1, 3, or 5 concurrent jobs by plan, and 30 minutes per job, 60 on Pro. Provider details are in Providers › Comfy Cloud. comfy-api-proxy for the Comfy API v2 lib.comfy.RunWorkflow’s api property selects the transport. native is ComfyUI’s own protocol. v2 sends the workflow through the Comfy API v2 client the Cloud node uses, pointed at endpoint. ComfyUI core does not serve /api/v2 yet, so a local ComfyUI needs Comfy-Org’s own adapter in front of it. It translates v2 onto the native protocol: pip install comfy-api-proxy comfy-api-proxy --comfyui http://127.0.0.1:8188 --port 8189 Then set endpoint to 127.0.0.1:8189 and api to v2. Leaving api on v2 while endpoint points at a plain ComfyUI gets a 404 on /api/v2/jobs. No API key is sent on this path, so the proxy has to accept an unauthenticated submit. What changes about the run, including the different output slot naming, is in Using the v2 API. What the server must already have A workflow in API format names its checkpoints, LoRAs, VAEs, and node classes by string. Those strings are resolved by the ComfyUI that runs the prompt, not by NodeTool. NodeTool clones the prompt, writes the connected input values into prompt[nodeId].inputs[field], and posts it. It downloads no weights and installs no extensions. So the server needs, before the first run: Every model file the workflow names, under the filename the prompt uses. sd_xl_base_1.0.safetensors in the prompt is the name ComfyUI looks for in its own model folders, subfolder included. Every custom node class the workflow uses. A class_type that is not registered on that ComfyUI is rejected at submit, exactly as it would be in ComfyUI’s own UI. Media inputs are the exception: a connected image, audio, or video ref is uploaded to the server before the prompt goes out, so input files do not have to pre-exist. A missing model or an unknown class_type fails the submit rather than the run. The direct node reports Submit failed (400) with ComfyUI’s own response body included, which names the offending node. The full message list is in Troubleshooting. Checking what is there On a worker, the Python bridge exposes comfyObjectInfo() for the node catalog and comfyModelsList(folder?) for the model files on the volume. No shipped node calls either one. They are reachable from code against a connected bridge, which is what Known limitations records. The message shapes are in Python Bridge Protocol. For a ComfyUI you reach directly, NodeTool offers no enumeration at all. Open ComfyUI’s own UI on that endpoint and check the loader dropdowns. For Comfy Cloud there is nothing to check from NodeTool: what is installed is Comfy’s business, and the Comfy API enumerates no models. That is why the provider contributes none to the model pickers, as Providers › Comfy Cloud notes. A first run to prove the wiring Use the smallest workflow you have, a text-to-image with one checkpoint and one SaveImage, before anything ambitious. Export it from ComfyUI with Save (API Format). Add Run ComfyUI Workflow to a graph and load the JSON with the node header’s Load Workflow icon. See Loading a workflow. Set endpoint if it is not 127.0.0.1:8188. Run it. A healthy run writes this sequence to the node’s log: Running ComfyUI workflow (7 nodes) on 127.0.0.1:8188 Execution started Executing CheckpointLoaderSimple (#4) Executing KSampler (#3) Output from #9 A sampler progress bar appears while KSampler runs, and the image arrives on the 9:image slot, keyed by the save node’s ComfyUI id. A second run of the same prompt logs Reused cache for N node(s) instead of re-executing the unchanged nodes. If the log stops at WebSocket connection failed, the endpoint is wrong or ComfyUI is not listening. If it stops at Submit failed (400), the server is reachable but missing something the workflow names. Related ComfyUI - the three nodes, their properties, the loader, and the outputs ComfyUI Recipes - worked examples of ComfyUI nodes inside NodeTool graphs Worker Deployment - renting a GPU and provisioning the ComfyUI worker image Providers - Comfy Cloud alongside every other provider, and where keys go--- # ComfyUI Source: https://docs.nodetool.ai/comfyui.html NodeTool runs ComfyUI workflows as nodes in a NodeTool graph. You export a workflow from ComfyUI in API (prompt) format, load it into a node, and its Load* nodes become typed inputs while its Save* nodes become typed outputs. Everything around it — asset handling, LLM steps, mini apps — stays NodeTool’s. Three nodes cover three topologies:   lib.comfy.RunWorkflow lib.comfy.RunWorkflowOnWorker lib.comfy.RunWorkflowOnCloud Title Run ComfyUI Workflow Run ComfyUI Workflow (Worker) Run ComfyUI Workflow (Comfy Cloud) Talks to Any ComfyUI HTTP/WebSocket endpoint A NodeTool worker that fronts ComfyUI Comfy Cloud, https://cloud.comfy.org Transport ComfyUI’s own /prompt, /ws, /view, /history, or the Comfy API v2 The worker bridge’s comfy.* messages, or the Comfy API v2 @comfyorg/sdk over the Comfy API v2 API native (default) or v2, from the api property Bridge, or v2 when the worker reports comfy.api_v2 v2 Auth None Worker bearer token in a node property The COMFY_API_KEY secret GPU Yours The rented worker’s Comfy’s ComfyUI reachable from NodeTool? Yes, directly No, loopback-only inside the worker Not applicable Workflow loader in the editor Yes No (see limitations) Yes Outputs Streamed, one frame per file Buffered on the bridge path, streamed on the v2 path Streamed, one frame per file All three live in the lib.comfy namespace and ship in the built-in base node pack, so every install registers them — including a server running the curated cloud profile (NODETOOL_NODE_PROFILE=cloud, the production default), which allowlists the three runners by name while dropping the rest of the namespace. If they don’t appear in the node menu, that is the menu’s own decluttering, not a gate: lib.comfy sits in the Developer Tools group of web/src/config/optionalNodePacks.ts, hidden from the browsable namespace tree until you reveal that group. Search, paste, and saved workflows resolve the nodes either way. Quickstart From a workflow you already have to a run inside NodeTool. Export in API format. In ComfyUI use Save (API Format). The regular save writes the UI format, which every node here rejects with a message saying exactly that. Get an endpoint. A ComfyUI already listening on 127.0.0.1:8188 needs nothing further. With no local install and no GPU, use the Cloud node and a COMFY_API_KEY instead. Both paths, and the rented-GPU one between them, are in ComfyUI Setup. Add the node. Search the node menu for Run ComfyUI Workflow. Searching finds it even though lib.comfy is collapsed out of the browsable tree. Load the workflow. Use the Load Workflow button in the node header: paste the JSON, drop the .json, or drop a .png that ComfyUI wrote. The inputs picked in ComfyUI App Mode become typed input handles, or the workflow’s Load* nodes and prompt text when it has no App Mode inputs. Its Save* nodes become typed output handles. Run it. Outputs arrive one file at a time as each save node finishes. If the submit fails, the cause is usually a model or custom node the server doesn’t have rather than anything in NodeTool. Troubleshooting lists the messages. When to reach for a ComfyUI node NodeTool generates images and video with its own nodes. nodetool.image.TextToImage and its siblings take a model property and route to whichever provider owns that model, so switching models is a dropdown change and there is no graph to maintain. That is the shorter path for most generation. A ComfyUI node earns its place when the graph itself is the point: The workflow already exists. A tuned ControlNet chain, a multi-pass upscale, a regional-prompting setup runs as it is, with no porting. The technique lives in custom nodes. Anything a ComfyUI extension does that no NodeTool node covers stays available, because the workflow executes on ComfyUI. The weights are yours. Checkpoints and LoRAs on your own disk never leave the machine that runs them. The two compose rather than compete: a ComfyUI node is one node in a NodeTool graph, so an agent can write its prompt, a dataframe can drive a batch of runs, and its stills can feed a video assembly. Those patterns are in ComfyUI Recipes. Run ComfyUI Workflow Point the node at a ComfyUI server you can reach. It submits the prompt over HTTP, follows the run on ComfyUI’s WebSocket, and downloads each output file as the node that produced it finishes. That is the default native API; setting api to v2 swaps the transport, see Using the v2 API. Properties Property Default Meaning endpoint 127.0.0.1:8188 Host:port or an http:// or https:// URL. A bare host:port gets http://, and the WebSocket scheme follows the HTTP one, so https:// proxies (RunPod pods, Cloudflare tunnels) work. A ws:// or wss:// URL is not accepted. workflow empty The API-format prompt as a JSON string: a map of node id to { class_type, inputs }. timeout 600 Seconds to wait for the run, minimum 1. Also bounds the WebSocket handshake and the submit request. api native Which API the endpoint speaks. native is ComfyUI’s own /prompt plus WebSocket. v2 treats endpoint as a Comfy API v2 origin. Loading a workflow In the editor, the node header has an upload icon with the tooltip Load Workflow. It opens a Load ComfyUI Workflow dialog that accepts: Pasted JSON in API format. In ComfyUI, use Save (API Format) — not the regular save, which writes the UI format with a nodes array. Pasting the UI format gives you an error saying exactly that. A dropped .json file in the same format. A dropped .png exported by ComfyUI, whose prompt metadata chunk is read out of the PNG’s tEXt/iTXt chunks. Its workflow chunk, the UI graph, supplies the App Mode inputs. A wrapper object with the prompt nested under a prompt key is unwrapped automatically. A UI graph next to it, under workflow or under extra_data.extra_pnginfo.workflow as in ComfyUI’s /prompt request body, supplies the App Mode inputs. The dialog summarizes what it found (node count, typed inputs, outputs). Choose Apply to write three things onto the node: the normalized prompt into the workflow property, the derived dynamic_inputs/dynamic_outputs, and default values for the exposed inputs. Values you had already edited on handles that still exist are preserved. The server does the parsing: the dialog sends the JSON or the PNG to POST /api/comfy/resolve-workflow and shows what comes back. Setting the workflow without the editor A graph created over the API, by an agent, or from a file can set workflow directly and leave dynamic_inputs and dynamic_outputs empty. Before every run the server derives the same handles from workflow and merges them under any saved on the node, so an edge from 15:image or into 4:text works without the workflow ever being opened in the editor. When the editor opens such a node, it writes the derived handles onto it so they show on the canvas. What becomes an input or an output The parser (packages/integration-nodes/src/nodes/comfy-schema.ts) reads the prompt and derives handles. Every dynamic handle is keyed <comfyNodeId>:<field>. App Mode inputs come first. When the PNG or the JSON wrapper carries the UI graph, and the author picked inputs in ComfyUI App Mode, those widgets become the input handles and nothing else is exposed automatically. ComfyUI stores the selection in the UI graph’s extra.linearData.inputs as [widgetId, widgetName, config?] entries, where widgetId is graphId:nodeId:widgetName. A file field of a Load* node gets its media type and any other widget gets its value’s type. An input’s App Mode description, when set, becomes the handle’s description. A widget promoted from a subgraph, or one converted to a linked input, has no literal in the API prompt and is skipped. The rules below apply when no App Mode input maps onto the prompt. A plain API-format export has no UI graph, so it always uses them. Typed media inputs come from Load* classes. Curated classes map to a known field and type — LoadImage, LoadImageMask, LoadImageOutput (image), LoadAudio, VHS_LoadAudioUpload (audio), LoadVideo, VHS_LoadVideo (video). Any other class whose name starts with Load falls back to a prefix guess: the media kind comes from the class name (Audio/Video, else image) and the field is its first literal string input. Typed outputs come from Save* and Preview* classes. Curated: SaveImage, PreviewImage, SaveAnimatedWEBP, SaveAnimatedPNG (image), SaveAudio, SaveAudioMP3, SaveAudioOpus, PreviewAudio (audio), SaveVideo, VHS_VideoCombine (video). Other Save*/Preview* classes fall back to the same name-based guess. Each output slot is keyed <comfyNodeId>:<image|audio|video> and carries a singular media type, because outputs stream one item per file. Prompt text inputs come from text-encoder classes, any class whose name contains TextEncode (CLIPTextEncode, CLIPTextEncodeSDXL, TextEncodeQwenImageEdit, …). Each literal string field becomes an optional str input, so a prompt from another node can be wired in. Left unconnected, it keeps the exported text. Everything else literal, with or without App Mode, is offered in the loader dialog as a checkbox list under Expose additional parameters as inputs — seeds, steps, CFG, sizes. Ticking one adds an optional input handle with the value’s inferred type (bool, int, float, str). Inputs that are ComfyUI connections ([sourceNodeId, slot]) are never offered; they’re wired inside the workflow. Connected values are injected into prompt[nodeId].inputs[field] before submission. A connected image, audio, or video ref is uploaded to the ComfyUI server first (POST /upload/image, which ComfyUI accepts for any input file) and the stored filename is substituted. The stored workflow property is never mutated — the node deep-clones it per run. Outputs This is a streaming-output node. When a save node finishes, its files are downloaded from /view immediately and emitted on that node’s slot, one media ref per file, so a batch of four images produces four frames rather than one list. Cached nodes never emit an executed event, so after the run the node reconciles anything it missed from /history/<prompt_id>. A final frame carries ComfyUI’s raw history payload on the static output slot (dict[str, any]). Non-streaming consumers get the same data buffered: process() drains the generator and merges frames, collapsing repeated slots into arrays. Progress and logs The node forwards ComfyUI’s WebSocket lifecycle to the NodeTool run log: execution start, execution_cached (as a count of reused nodes), the class name of each executing node, and errors. Sampler progress events become node_progress messages, so the node shows a progress bar. Cancelling the run posts /interrupt to the ComfyUI server and closes the WebSocket, so the prompt stops there instead of running on with nothing listening. The node then fails with ComfyUI execution was canceled. Using the v2 API Set api to v2 and the node sends the workflow through the same Comfy API v2 client the Cloud node uses, pointed at endpoint instead of https://cloud.comfy.org. Two things serve that API in front of a local ComfyUI: comfy-api-proxy, Comfy-Org’s Python adapter, which translates v2 onto the native protocol. pip install comfy-api-proxy, then comfy-api-proxy --comfyui http://127.0.0.1:8188 --port 8189, and point endpoint at port 8189. ComfyUI itself, once core serves /api/v2. Then the proxy is unnecessary and v2 becomes the default. No API key is sent on this path, so the endpoint has to accept an unauthenticated submit. Everything else matches the Cloud node’s run: the submit carries an idempotency key and retries a 429 for up to 60 seconds, media inputs are uploaded through the asset API and deduplicated by a blake3 hash of the bytes, the run is followed over SSE with polling as the fallback when the stream drops, and cancelling the NodeTool run calls the job’s cancel endpoint. Outputs use the same <comfyNodeId>:<kind> convention as the Cloud node rather than ComfyUI’s history payload: image, audio and video carry a media ref, text carries the decoded string, and file and latent carry a document ref. The static output slot carries the same job manifest the Cloud node returns: job_id, status, and one entry per output file. Run ComfyUI Workflow (Worker) Use this when ComfyUI runs on a rented GPU and should not be exposed. Deploy the ghcr.io/nodetool-ai/nodetool-worker-comfy:latest image (selectable as NodeTool Worker + ComfyUI in the worker profile dialog — see Worker Deployment). That worker runs a co-located, loopback-only ComfyUI and proxies it over the bridge; ComfyUI’s own ports are never published. Properties Property Default Meaning worker_url ws://127.0.0.1:7777/ws WebSocket URL of the worker. worker_token empty Bearer token, when the worker requires one. workflow empty Same API-format prompt JSON as the direct node. timeout 600 Seconds, passed through to the worker. previews false Stream ComfyUI preview images during the run. Costs bandwidth. How a run works The node opens a one-shot bridge connection (no auto-reconnect), because the bridge is how it reads worker.status.comfy, and that status picks the path. The v2 path A worker whose status reports comfy.api_v2: true serves the Comfy API v2 on its own port 7777, behind the bearer token it already requires, with ComfyUI still loopback-only. The node closes the bridge, turns worker_url into an HTTP origin (wss://host/ws becomes https://host), and runs the workflow through the same v2 client the Cloud node uses, with worker_token as the key. Media inputs go through the asset API and are deduplicated by a blake3 hash of the bytes. The submit carries an idempotency key and retries a 429 for up to 60 seconds, the run is followed over SSE with polling as the fallback, and cancelling calls the job’s cancel endpoint. The worker image that reports the flag is a nodetool-core change and has not shipped, so today every worker takes the bridge path. The bridge path Without the flag the node calls comfy.execute over the same connection, and closes it whether the run succeeds or throws. This path also requires supportsComfy(): bridge protocol v3+ and worker.status.comfy.enabled: true. A worker built from the plain image fails that check with a message naming the ComfyUI image. Media inputs are not uploaded over HTTP here. Each one is sent as a bridge blob and referenced from the workflow JSON as "blob:<key>"; the worker uploads it into ComfyUI’s input directory and splices in the real filename before submitting. Scalar inputs are written into the prompt directly, same as the direct node. Lifecycle events arrive as dedicated comfy.event frames rather than generic progress frames, in emission order: queued → queue → started/cached → executing → progress → node_output → preview → completed/cancelled The node logs started, cached, and executing, and turns progress (value/max) into node_progress. Outputs On the v2 path each finished file is emitted on its own slot, keyed <comfyNodeId>:<kind>: image, audio and video carry a media ref, text carries the decoded string, and file and latent carry a document ref. The static output slot carries the job manifest, job_id and status and one entry per output file. That is the convention the direct and Cloud nodes use. On the bridge path output files come back as blobs on the terminal result. Each blob’s media kind is sniffed from its leading bytes (PNG, JPEG, GIF, RIFF for WEBP/WAVE/AVI, ftyp boxes for MP4/MOV, OGG, and ID3) and emitted as a base64 media ref on a slot named after the worker’s blob key. ComfyUI’s raw outputs land on the static output slot. Run ComfyUI Workflow (Comfy Cloud) Use this when you have no GPU and no ComfyUI install. The node submits the same API-format prompt to Comfy Cloud through the official @comfyorg/sdk, which speaks the Comfy API v2: POST /api/v2/jobs to submit, /api/v2/assets for media inputs, and an SSE stream at /api/v2/jobs/{id}/events for the run. Base URL is https://cloud.comfy.org (the SDK reads COMFY_BASE_URL from the server’s environment if you point it somewhere else) and the key travels as a Bearer token. Add the key first. Get one at https://platform.comfy.org (Comfy Cloud keys are prefixed comfyui-) and store it as COMFY_API_KEY, either on the Comfy Cloud card in Settings → Models & Providers or with nodetool secrets store COMFY_API_KEY. The node declares it in requiredSettings, so the editor badges it as unconfigured until it is set. The same key is sent alongside the workflow as extra_data.api_key_comfy_org, which is what authenticates any partner (API) node inside the graph. Properties Property Default Meaning workflow empty The API-format prompt as a JSON string, same shape as the other two nodes. Required. timeout 600 Seconds to wait for the job. Comfy caps a run at 30 minutes, 60 on Pro. previews false Write a line to the run log for each preview frame Comfy sends. No slot carries the bitmaps. The node has the same Load Workflow button and the same schema-derived handles as the direct node, so a pasted API-format JSON, a dropped .json, or a ComfyUI .png all fill it in. See Loading a workflow and What becomes an input or an output. How a run works The prompt is deep-cloned per run, so the stored workflow property is never mutated. Each dynamic input keyed <comfyNodeId>:<field> is written into prompt[id].inputs[field]: a scalar goes in as it is, and a connected image, audio, or video ref is uploaded through the SDK’s asset API and referenced as a core/ASSET object. That upload is deduplicated by a blake3 hash of the bytes, so re-running with the same input file uploads nothing the second time. UI-format JSON is rejected before any request is made. Re-export from ComfyUI with Save (API Format). While the job runs, the SSE stream drives the node: sampler progress becomes node_progress, and logs, status changes and queue position go to the run log. When the stream ends the node refreshes the job over HTTP and decides on that status, so a deployment with no live SSE still finishes correctly. Cancelling the NodeTool run calls the job’s cancel, so the job stops on Comfy’s side too rather than running on unwatched. Outputs This is a streaming-output node. Each finished file is fetched while the job is still running, because the signed URL Comfy hands out expires, and emitted on its own slot, keyed <comfyNodeId>:<output type>: image, audio and video carry a media ref, text carries the decoded string, and anything else (file, latent) carries a document ref. A final frame on the static output slot (dict[str, any]) describes the finished job: job_id, status, and one entry per output file with its node_id, name, type, content_type, size_bytes and asset_id. The bytes already went out on the dynamic slots, so this frame is metadata. Limits Comfy’s own, not NodeTool’s: Concurrent jobs: 1, 3 or 5 at a time depending on the plan. A full queue answers 429; the SDK waits the Retry-After and resubmits until a 60 second budget runs out, then the node fails with Comfy queue is full: …. Runtime: 30 minutes per job, 60 on Pro. The node’s timeout bounds the NodeTool side of the same run. Credit: an account without enough credit gets 402 from the submit and the node fails with Comfy account has insufficient credits: …. The comfy.* bridge surface comfy.execute is the only call a shipped node makes, but the bridge exposes the whole proxy family. The bridge does not check supportsComfy() for you, so callers test it first. Bridge method Wire message Returns comfyExecute(workflow, options, onEvent, requestId) comfy.execute {prompt_id, status, outputs, blobs} cancelComfyExecute(requestId) cancel — (rejects the local promise itself) comfyQueue() comfy.queue {queue_running, queue_pending} comfyInterrupt() comfy.interrupt — (global stop; admin-only) comfyCancelPrompt(promptId) comfy.cancel — (per-prompt, the user-facing cancel) comfyUpload(filename, bytes, options) comfy.upload stored file info comfyView(filename, options) comfy.view file bytes comfyObjectInfo() comfy.object_info the node catalog, unwrapped comfySystemStats() comfy.system_stats ComfyUI system/VRAM stats comfyStatus() comfy.status {enabled, url, reachable, system_stats, queue_remaining, error} comfyFree(options) comfy.free — (unload models from VRAM) comfyModelsList(folder?) comfy.models.list model files; folder filters client-side comfyModelsDownload(req, onProgress, requestId) comfy.models.download — (streams progress frames) comfyModelsDelete(folder, filename) comfy.models.delete boolean Two shapes are easy to get wrong. comfy.execute takes its prompt under workflow, not prompt. comfy.models.download takes a nested source: { type: "huggingface", repo_id, path, revision? } or source: { type: "url", url } — a flat {url} or {repo_id} is rejected by the worker. comfy.models.list has no server-side folder filter, so the bridge fetches the whole volume and narrows it locally. Cancel a run with cancelComfyExecute(requestId) rather than the bare cancel: it sends the cancel frame and rejects the local promise, because the worker may never emit a terminal frame after a cancel. Wire-level details live in Python Bridge Protocol; the authoritative field reference is docs/comfy-proxy.md in nodetool-core. Known limitations The worker node has no workflow loader in the editor. The Load Workflow button and the schema-derived handles belong to lib.comfy.RunWorkflow and lib.comfy.RunWorkflowOnCloud, whose slot names the parser can predict. For lib.comfy.RunWorkflowOnWorker you paste the API-format JSON into the workflow property and add dynamic input handles yourself, keyed <comfyNodeId>:<field>. What keeps it out is the bridge path’s slot naming below, which the v2 path fixes. On the bridge path, the worker node’s output slot names come from the worker, not from the <comfyNodeId>:<kind> convention the direct and Cloud nodes and the schema parser use. Check the emitted keys against a real run before wiring downstream nodes. The v2 path uses the convention. On the bridge path the worker node does not stream outputs. It returns everything on completion. The v2 path yields each file as it lands, like the direct and Cloud nodes. Cancelling a NodeTool run does not stop a worker run on the bridge path. The direct node posts /interrupt, and the Cloud node and both v2 paths call the job’s cancel. The bridge path calls comfy.execute without a request id and wires no abort listener, so cancelComfyExecute is never sent: the prompt runs to completion on the worker while NodeTool stops listening. Since every shipped worker image takes the bridge path, treat a worker run as uncancellable and use the timeout property to bound it. include_temp is not exposed on either node. The bridge supports it (ComfyExecuteOptions.includeTemp), so preview-node outputs can be fetched from code, but no node property surfaces it. The Cloud node reports no cost. Comfy’s v2 job response carries no cost field, so no provider cost record is written for a Cloud run and the Comfy Cloud column in a run’s cost breakdown stays empty. The Comfy job id is written to the run log, so a run can be matched against Comfy’s own billing. The worker node’s v2 path is built; the worker image that reports comfy.api_v2 is a nodetool-core change and ships separately, so the comfy.* bridge family stays until it does. The direct node’s v2 stays opt-in while a local ComfyUI needs comfy-api-proxy in front of it. The design is docs/superpowers/specs/2026-09-05-comfy-sdk-integration-design.md in the repository (engineering specs are not part of the published site). Troubleshooting Message Cause This looks like a ComfyUI UI workflow… Exported with the regular save. Re-export with Save (API Format). No ComfyUI prompt found in this PNG The PNG was written without metadata, or re-encoded by another tool. Node "<id>" is not in API format The JSON isn’t a map of node id to { class_type, inputs }. ComfyUI workflow is not valid JSON The workflow property holds malformed JSON. WebSocket connection failed Wrong host/port, or ComfyUI isn’t listening. Check endpoint. Submit failed (400) ComfyUI rejected the prompt — usually a missing model or an unknown class_type on that server. The response body is included. Timeout waiting for ComfyUI result The run exceeded timeout seconds. Raise it for large video or upscale graphs. ComfyUI workflow did not finish within <n>s The same timeout on a v2 path of the direct or worker node. Comfy job <id> failed in <class_type> (#<node>): … The job ran and a ComfyUI node raised. The text after the colon is ComfyUI’s own message and error code. The connected worker does not front a ComfyUI server The worker isn’t running the ComfyUI image, or reports comfy.enabled: false. A submit that answers 404 on /api/v2/jobs api is v2 but the endpoint is a plain ComfyUI, which serves no /api/v2. Put comfy-api-proxy in front of it, or set api back to native. Comfy account has insufficient credits Comfy Cloud answered 402. Top the account up at platform.comfy.org. Comfy queue is full Your plan’s concurrent-job limit was still full 60 seconds after the first 429. Comfy Cloud job did not finish within <n>s The job outlived the Cloud node’s timeout. Raise it, up to Comfy’s own 30 or 60 minute cap. COMFY_API_KEY is required to run a workflow on Comfy Cloud No key stored. Add it in Settings → Models & Providers. ComfyUI workflow is in UI-export format The v2 client rejected the prompt before sending it, on the Cloud node or on either v2 path. Re-export with Save (API Format). Where the code lives Concern File Direct and worker nodes packages/integration-nodes/src/nodes/comfy.ts Cloud node packages/integration-nodes/src/nodes/comfy-cloud.ts Comfy API v2 transport and runner packages/integration-nodes/src/nodes/comfy-sdk.ts — cloudTransport for Comfy Cloud, v2Transport(baseUrl, apiKey?) for the worker and direct v2 paths Direct HTTP/WS executor packages/runtime/src/comfy-executor.ts comfy.* bridge methods packages/runtime/src/python-bridge-base.ts Bridge types packages/runtime/src/python-bridge-types.ts Schema parser (browser) web/src/utils/comfyDynamicSchema.ts Editor node + loader web/src/components/node/DynamicComfySchemaNode/ Related ComfyUI Setup — getting an endpoint these nodes can reach, and what has to be installed on it ComfyUI Recipes — composing a ComfyUI node with the rest of a graph, and running one headless Worker Deployment — renting a GPU and picking the ComfyUI worker image Provider Reference — the Comfy Cloud provider and its key Python Bridge Protocol — the comfy.* message family on the wire Comparisons — how NodeTool and ComfyUI differ as tools--- # Why NodeTool Source: https://docs.nodetool.ai/comparisons.html NodeTool is an open-source, agent-first creative workspace. Tell the agent what to make. It builds the project with image, video, audio, and text models, and you edit every part of the result. The usual way Write the idea in a chat app Generate stills in an image app Animate them in a video app Cut them with music in an editor Four subscriptions, a downloads folder, and no record of the steps. One change means doing it all again. With NodeTool Write one brief The agent drafts, boards, renders, and cuts Open the project and change any part One project on your own keys. Every step stays editable and re-runnable. The agent builds. You stay in control. No programming needed. The agent picks the models, connects them, runs them, and fixes errors it finds. Nothing is hidden. What it builds is an ordinary storyboard, timeline, or graph. Open it, swap a model, rewire a step, and run again. Send a note, not a restart. Ask for a change to one shot and only that shot renders again. Agents inside workflows. Put an agent node in your own graph to plan, decide, and call tools. What you keep Bring your own keys Connect OpenAI, Anthropic, Gemini, FAL, KIE, Replicate, and the video specialists. Keys are stored encrypted, with the encryption key in your operating system's keychain in Studio. No markups If an image costs $0.03 at the provider, you pay $0.03 to the provider. No credit packs. NodeTool takes no cut. A project file that opens anywhere The board, the script with its takes, and the multi-track cut are documents you can open and edit again. Workflows and mini apps export as a .nodetool bundle you can import anywhere. Open source, end to end Studio and Cloud are built from the same AGPL-3.0 source, with no paywalled features. Read it, fork it, or host it yourself. Where NodeTool fits Pick the model. Pick the price. Take Seedance, one of today's best video models. FAL, Replicate, and KIE each sell it at a different price, and NodeTool lets you pick the cheapest. When the next Veo or Kling arrives, you switch in one click. That is what holding the keys buys you: the best model at the best price each week, and nothing to lose if a tool gets bought, repriced, or shut down. An open studio and a hosted platform TopicNodeToolHosted platforms ModelsEvery major provider, switched in one clickThe list they picked When a better model shipsAdd it the day it shipsWait for them to add it What you payProvider list prices, on your own keysTheir credits What you keepThe board, the takes, and the multi-track cut as an editable projectAn exported video. The project stays in their app. SourceOpen, AGPL-3.0Closed Where it runsDesktop app and browser, self-host any timeTheir servers only Local modelsRun on your own GPU in Studio, offlineNot available Head-to-head Comparing NodeTool with a specific tool? Each page covers features, pricing, and when to pick which. ComfyUI Deep local image control. NodeTool adds video, audio, and text, and runs an exported ComfyUI graph as a node. Figma Weave (Weavy) A hosted canvas on credits. NodeTool is open source and runs on your own keys. n8n Connects business apps. NodeTool is built for creative AI and media generation. Dify Mostly text apps. NodeTool adds native image, video, and music generation. Flowise Chatbots on a canvas. NodeTool builds those too, next to media generation. Langflow Text pipelines. NodeTool adds image, video, and music on the same canvas. Already have a tuned ComfyUI graph? It keeps working inside NodeTool. See ComfyUI. Next steps Quick Start → Install NodeTool and run your first project in minutes. Models & Providers → Every model NodeTool supports, and how to connect a key.--- # Configuration Guide Source: https://docs.nodetool.ai/configuration.html NodeTool uses a layered configuration system so local development, automated deployments, and production environments can share sensible defaults with minimal duplication. Settings come from environment variables and .env files, while secrets are stored encrypted at rest in a local SQLite database (managed via the CLI). The configuration helpers live in the @nodetool-ai/config package (environment.ts). They are plain functions — loadEnvironment(), getEnv(), and requireEnv() — there is no Environment class and no settings.yaml/secrets.yaml loading. Configuration Layers loadEnvironment() loads .env files in this order, with later files overriding earlier ones; system environment variables always win over file values: .env .env.<NODE_ENV> .env.<NODE_ENV>.local NODE_ENV defaults to development when unset. Use getEnv("KEY") to read a value (returns undefined when unset) and requireEnv("KEY") to read a value that must be present (throws a descriptive error otherwise). This hierarchy allows committed defaults, per-environment overrides, and developer-specific overrides to co-exist without file conflicts. Managing Settings & Secrets The in-app Settings page is the easiest way to manage everything. It has four tabs, each with its own sidebar of sections: Tab What it covers General Editor, Updates and Vaults (desktop app only), Appearance, Execution, Canvas, Default Models, Autosave Models & Providers Provider API keys, stored as encrypted secrets Integrations Configuration groups (including Folders), Connected Accounts, MCP Servers and Browser Extension (local server only), Nodetool API Token (hosted only) About Application, operating system, features and versions, installation paths (development builds), links Stored settings are defined in setting-catalog.ts in @nodetool-ai/config. A setting read through the server checks the database first, then the environment variable of the same name (getSetting in packages/websocket/src/settings-registry.ts). From the command line: nodetool settings show [--json] – print a fixed set of variables read from the process environment (ENV, LOG_LEVEL, PORT, HOST, DB_PATH, NODETOOL_API_URL, the main provider keys, HF_TOKEN, VECTORSTORE_DB_PATH, ASSET_BUCKET, S3_ENDPOINT_URL). Names ending in KEY or TOKEN print as ***, and unset variables print empty. It does not read .env files or stored secrets. nodetool secrets list – list stored secret keys (values are never shown). nodetool secrets store <key> [--description <text>] – store or update a secret (prompts for the value). nodetool secrets get <key> – print a stored secret value. The three secrets commands take --user-id <id> (default 1). Secrets are encrypted and persisted in a local SQLite database, not in YAML files. There is no nodetool settings edit command and no --secrets option. Secret Storage and Master Key Secrets saved through the CLI are encrypted with AES-256-GCM, using a per-user key derived from the master key via PBKDF2-SHA256 (100,000 iterations). initMasterKey() in @nodetool-ai/security resolves the master key in this order: SECRETS_MASTER_KEY environment variable. Local system keyring (macOS Keychain, Windows Credential Manager, or Secret Service via keytar). Generates a new key and persists it to the keyring. For shared deployments you must pre-provision the master key (via the SECRETS_MASTER_KEY environment variable) so every server can decrypt secrets generated locally. On a headless host with no keychain and no provisioned key, master-key initialization (and therefore startup) fails because there is no place to persist a generated key. Moving Secrets to a Server Export the master key once and set it on every server instance using the value from your deployment pipeline or secrets manager: export SECRETS_MASTER_KEY="<your-base64-master-key>" Enter the secrets on the server itself, through Settings or the nodetool secrets store command run against that server’s database. nodetool deploy apply does not copy local secrets. The POST /admin/secrets/import route exists but answers 501 (“Secrets import not available in standalone mode”). Ciphertext is encrypted per user with the master key, so a database copied between machines decrypts only where SECRETS_MASTER_KEY is the same value. For a container deployment, nodetool deploy add and the deploy config generate a SECRETS_MASTER_KEY (32 random bytes, base64) in container.environment when none is set. Storage Backend Selection The storage backend is selected explicitly by NODETOOL_STORAGE_BACKEND (one of file, s3, or supabase; defaults to file). It is not auto-detected from the presence of S3 or Supabase credentials. file (default) — assets are written to the local assets directory. s3 — requires ASSET_BUCKET (and, for the temp store, TEMP_BUCKET); reads S3_REGION and S3_ENDPOINT (or S3_ENDPOINT_URL) as needed. supabase — requires SUPABASE_URL, SUPABASE_KEY, and the relevant bucket var. A missing required variable throws Missing required env var <name> for NODETOOL_STORAGE_BACKEND=<backend>, and an unknown backend value throws. The asset store uses ASSET_BUCKET; the temp store uses TEMP_BUCKET. See storage-config.ts in @nodetool-ai/config. Using Environment Variables in Code When adding a feature that reads configuration: Read the variable with getEnv("YOUR_ENV_VAR") from @nodetool-ai/config so .env load order is respected. If the value is required, use requireEnv("YOUR_ENV_VAR") so a missing key raises a descriptive error. Document the new entry in .env.example. Recommended Workflow cp .env.example .env.development.local vim .env.development.local # add OpenAI/Anthropic/HF tokens, S3 credentials, etc. nodetool secrets store OPENAI_API_KEY # encrypt a provider key into the secrets DB (optional) nodetool settings show # verify resolved configuration Use .env.<env>.local for machine-specific overrides and keep secrets out of version control. When deploying, provide environment variables via your orchestrator or the container.environment map of a deployment in deployment.yaml. Supabase Settings NodeTool integrates with Supabase for both user authentication and asset storage. Set the following to enable Supabase: NODETOOL_STORAGE_BACKEND=supabase – select the Supabase storage backend SUPABASE_URL – your project URL, e.g. https://<ref>.supabase.co SUPABASE_KEY – a service role key (server-side only) ASSET_BUCKET – Supabase Storage bucket used for assets (e.g. assets) TEMP_BUCKET – bucket for temporary assets (e.g. assets-temp) Use a separate Supabase for user-provided nodes: NODE_SUPABASE_URL – user/node project URL (kept distinct from core SUPABASE_URL) NODE_SUPABASE_KEY – service role key for user/node data (kept distinct from core SUPABASE_KEY) NODE_SUPABASE_SCHEMA – optional schema for node tables (defaults to public) NODE_SUPABASE_TABLE_PREFIX – optional prefix applied to node tables to avoid collisions with core tables Behavior: Storage is used for Supabase buckets only when NODETOOL_STORAGE_BACKEND=supabase; the backend is chosen explicitly, not auto-detected from the presence of credentials. Authentication enters Supabase mode (enforced auth) when both SUPABASE_URL and SUPABASE_KEY are set — storage and auth are configured independently. See Authentication. Security notes: Use the service role key only in server environments. Do not expose it to clients. Public buckets make generated URLs directly accessible. For private buckets, add a signing step. Serving the Web UI and TLS The API server can also serve the built web app. STATIC_FOLDER points at the directory holding index.html; unset, or naming a directory that is not there, no static handler is registered and anything outside the API routes answers 404. STATIC_FOLDER=/app/web/dist nodetool serve With it set the server serves that directory at /, sends index.html for / and /apps/index.html, and falls back to index.html for any other extension-less GET that is not under /api, /ws, /trpc, /mcp, /health, /v1, /.well-known/, or /oauth (except /oauth/consent, which is the SPA’s MCP consent page) — so client-side routing survives a reload. The Docker image and the desktop app set it to their own bundled copy. TLS is on when both a certificate and a key resolve: TLS_CERT=/etc/ssl/nodetool/cert.pem \ TLS_KEY=/etc/ssl/nodetool/key.pem \ REDIRECT_PORT=8080 \ nodetool serve --host 0.0.0.0 --port 8443 A variable naming a path that does not exist is ignored. The server then looks for cert.pem and key.pem by walking up to five directories from its working directory, so a stray pair beside the process turns TLS on with neither variable set. With TLS active every URL the server logs uses https / wss, and a plain HTTP listener on REDIRECT_PORT (default 80) answers 301 to https://<request Host>:<server port><path>. Ports below 1024 need elevated privileges. When that bind fails with EACCES the server logs that the redirect listener was skipped and keeps serving HTTPS. Python Nodes Python nodes run in a separate worker process that NodeTool spawns and talks to over stdio. The server picks the interpreter in this order: NODETOOL_PYTHON, if set — an absolute path to the executable. Nothing else is tried, so a wrong path is a hard failure rather than a silent fallback. An active CONDA_PREFIX whose environment is named nodetool or conda_env, or whose path contains nodetool/conda_env. The first of these that exists: Windows: the desktop app’s nodetool\conda_env under %ALLUSERSPROFILE% (or %APPDATA% when that is unset), then envs\nodetool under Miniconda3, miniconda3, Anaconda3, or anaconda3 in your home folder, then C:\ProgramData\nodetool\conda_env. macOS: ~/nodetool_env, then ~/miniconda3/envs/nodetool and ~/anaconda3/envs/nodetool. Linux: ~/.local/share/nodetool/conda_env, /opt/nodetool/conda_env, then ~/miniconda3/envs/nodetool and ~/anaconda3/envs/nodetool. With none of them available the server logs Python not found — Python nodes will not be available at startup and runs everything else normally. NODETOOL_PYTHON=/opt/conda/envs/nodetool/bin/python nodetool serve The interpreter needs nodetool-core and the packs installed from PyPI. Setup steps are in Python nodes without the desktop app. The worker starts with the server. With NODETOOL_PYTHON_ON_DEMAND=true it starts when a workflow first needs a Python node instead, and Python-only providers such as huggingface-local and MLX appear in model menus only after that. Settings shows no field for it. Set it in the environment, or store it under the same name with the agent’s set_setting tool. The server reads it at startup, so restart after changing it. NODETOOL_PYTHON_ON_DEMAND=true nodetool serve Python nodes run on the device PyTorch finds: Apple Metal (MPS), then CUDA, then the CPU. NODETOOL_TORCH_DEVICE overrides that with cuda, cuda:<N> for the Nth card counting from 0, mps, or cpu. The worker reads it at start, so restart after changing it. Hardware limits are in GPU requirements. The bridge is a local-only feature: when NODETOOL_ENV=production it refuses to connect, and a workflow reaching a Python node fails with “Python bridge is disabled in production”. Set NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 to override that on a host where you do want the worker. The flag alone is not enough on the published Docker image: it ships no Python worker, so derive an image that installs nodetool-core and point NODETOOL_PYTHON at that interpreter. See Self-hosted deployment. The three NODETOOL_PYTHON_*_TIMEOUT_MS variables bound how long the server waits on the worker. Raise NODETOOL_PYTHON_EXECUTE_TIMEOUT_MS past its 12-minute default for nodes that legitimately run longer. NODETOOL_WORKER_NAMESPACES narrows which Python node namespaces the worker loads. Its value is passed through unchanged as --namespaces <value> when the worker is spawned, and the worker parses it; unset, the flag is not passed and the worker loads everything installed. NODETOOL_WORKER_NAMESPACES=nodetool.image nodetool serve Protocol Validation Two settings schema-check messages in flight. Both accept 1/true to force validation on and 0/false to force it off; unset, both default to on under NODE_ENV=test or Vitest and off everywhere else — a malformed frame should fail the test that produced it rather than throw mid-stream on a production connection. NODETOOL_VALIDATE_OUTBOUND_WS — every server→client WebSocket frame whose type matches a known message schema is parsed against it before it goes on the wire. Frames with an unrecognized or absent type are left alone. A failure throws where the frame was sent. NODETOOL_VALIDATE_BRIDGE_FRAMES — the same check on frames arriving from the Python worker. A frame that fails is rejected with a structured, non-fatal error instead of being dispatched as malformed data. Turn one on outside tests when you are chasing a protocol bug and want it to surface at its source: NODETOOL_VALIDATE_OUTBOUND_WS=1 nodetool serve JavaScript Sandbox Threading The QuickJS sandbox behind Code nodes, CodeAct, and JS scripts runs on a worker thread whenever it can. A CPU-bound guest blocks whichever thread runs it, and on the server’s main thread that freeze takes the event loop with it — including the WebSocket stop frame that would have cancelled the run. On a worker, cancelling is terminate(), immediate for a spinning guest and a parked one alike. Two settings change that choice: NODETOOL_SANDBOX_INPROC=1 runs every guest on the calling thread. It is treated as a chosen fallback, so it warns about nothing. NODETOOL_SANDBOX_WORKER=require refuses to fall back at all. A run that cannot reach a worker fails with the sandbox worker path is required (NODETOOL_SANDBOX_WORKER=require) but unavailable: <reason> instead of running in-process. Some runs stay in-process with no setting involved. A run that streams its inputs does, because the synchronous stream.open probe is served from a worker-local mirror that must be seeded with the handle names; those runs park on takes and yield constantly, so the freeze the worker exists for cannot build up. Globals that cannot be structured-cloned keep a run on this thread too. The by-design cases are quiet; an environmental one warns once per process: sandbox: running in-process (<reason>); a CPU-bound guest will block this thread until its timeout NODETOOL_SANDBOX_INPROC=1 nodetool serve Video Input on Vision Models Most chat APIs are OpenAI-compatible and have no video content part. A clip sent to one used to be refused outright, even when the model behind the endpoint reads images perfectly well. The runtime now samples the clip into stills with ffmpeg and sends those instead, labelled with their timestamps, under a header naming the frame count and the sample rate — so a model that needs the audio or the motion between frames can say it is missing rather than invent it. A provider that reads video natively (Gemini) still gets the whole clip and never goes through this path. ffmpeg and ffprobe must be on PATH; without them the call fails naming the binary. The sample is bounded by four settings: NODETOOL_VIDEO_FRAME_FALLBACK=0 turns the whole thing off and restores the refusal, for a caller that would rather fail than pay for a lossy read. NODETOOL_VIDEO_FRAME_MAX_FRAMES — frames per clip. Default 16. NODETOOL_VIDEO_FRAME_MAX_FPS — ceiling on the sample rate. Default 1. A clip long enough that the frame budget cannot reach this rate is sampled more sparsely so the frames still span the whole thing. NODETOOL_VIDEO_FRAME_MAX_DIMENSION — longest edge of each frame, in pixels. Default 768. Frames are never upscaled. NODETOOL_VIDEO_FRAME_MAX_FRAMES=8 NODETOOL_VIDEO_FRAME_MAX_FPS=0.5 nodetool serve Development-Only Settings Two settings exist for local development and are off unless set: NODETOOL_ENABLE_TEST_TOPUP=1 allows the credits top-up mutation to mint credits with no payment behind it. Without it the mutation is refused. Minted credits unlock spend on platform-owned keys, so leave this unset on anything reachable by someone else. NODETOOL_SEED_DEMO_COSTS=1 seeds ~90 days of plausible spend, plus the demo workflows whose names it references, so the Costs dashboard has something to show. The seed is idempotent — a marker row stops it re-running. NODETOOL_SEED_DEMO_COSTS=1 nodetool serve Test Harness Settings These configure the harnesses, not the server. nodetool debug --browser sets the two NODETOOL_DEBUG_* variables below from its own flags; you set them by hand only when driving the Playwright spec directly. NODETOOL_DEBUG_STAGES — 1 or true captures a canvas screenshot at every stage of the run into stages/ under the output directory, up to 16 intermediate frames plus the final one. This is what nodetool debug --stages turns on, and it is off otherwise because each frame costs a screenshot. NODETOOL_DEBUG_TIMEOUT — per-run timeout in milliseconds for the in-page run, which is how nodetool debug --timeout reaches the browser surface. Read as a positive integer; anything else is ignored. The spec waits 30s longer than the value, and never less than its 5-minute floor, so polling outlives the run it is watching. NODETOOL_E2E_EXAMPLES_DIR — the examples directory the e2e test server serves at /api/examples. A path that does not exist is ignored rather than fatal; the server then looks for packages/base-nodes/nodetool/examples/nodetool-base and examples/workflows under the repo root. The resolved path is printed in the server’s readiness line. NODETOOL_TEST_CHROME — the Chrome binary the packages/browser integration suite drives. Branded Google Chrome silently ignores --load-extension, so the suite needs Chrome for Testing; test:integration installs one under <repo>/chrome and the harness finds it there. Point this at a binary you already have to skip that download. CHROME_PATH is read as a fallback, and a path that does not exist is ignored rather than fatal. NODETOOL_TEST_CHROME=/opt/chrome-for-testing/chrome \ npm run test:integration --workspace=packages/browser NODETOOL_QA_STATE — read by the test-suite backend (screenshot-server.ts) only. empty starts with no workflows, threads, assets, or secrets, as a new account does. failing-run is that account plus one workflow that fails until fixed. Any other value, or none, seeds the usual demo data. First-time-user QA (the agentic-qa skill) uses it. The graph, output directory, and run params come from NODETOOL_DEBUG_GRAPH, NODETOOL_DEBUG_OUT, and NODETOOL_DEBUG_PARAMS: cd web NODETOOL_DEBUG_GRAPH=/tmp/graph.json \ NODETOOL_DEBUG_OUT=/tmp/debug-out \ NODETOOL_DEBUG_STAGES=1 \ NODETOOL_DEBUG_TIMEOUT=120000 \ npm run test:debug-harness The run writes record.json, screenshot.png, and — with stages on — stages/ and stages.json into the output directory. It also starts its own hermetic backend on 127.0.0.1:7777, so stop any server already on that port first; the harness refuses to run against one rather than exercise a real database with real providers. Hermetic providers The user-journey suite (web/tests/journeys) sends chat messages and runs whole workflows with no API keys and no network. NODETOOL_FAKE_PROVIDERS=1 puts the suite’s backend (screenshot-server.ts) in hermetic mode: every registered LLM provider is re-registered as a deterministic fake, and external or media-generating nodes — fal, Replicate, search, HTTP, image/video/audio generation — resolve to an executor that returns type-correct placeholder outputs derived from the node’s output metadata. Structural nodes (input/output/control) and pure-compute nodes (text, data, math) still run for real, so the assertions mean something. Every faked LLM call returns the string deterministic e2e response. npm run test:journeys sets the variable itself — tests/journeys/globalSetup.ts does process.env.NODETOOL_FAKE_PROVIDERS ??= "1". Set it by hand only when starting that backend yourself. The screenshot and visual suites leave it off, because they only render pages and never run a node. NODETOOL_FAKE_DEBUG=1 logs each node’s REAL/FAKE resolution to stderr, which is how you find out why a node you expected to be faked ran for real. NODETOOL_ENABLE_FAKE_PROVIDER=1 registers the separate fake provider id as a builtin, so a workflow can select it the way it selects any other provider. Ignored when NODETOOL_ENV=production. cd web NODETOOL_FAKE_PROVIDERS=1 NODETOOL_FAKE_DEBUG=1 npm run test:journeys Chat Turn Replay A chat or agent turn outlives the WebSocket connection that started it. Every frame the turn emits is stamped with an increasing chat_seq and appended to a bounded buffer; a client that reconnects sends {command: "resume_chat", data: {thread_id, last_seq}} and gets the missed tail replayed. Three variables size that machinery: NODETOOL_CHAT_DETACH_GRACE_MS — a running turn nobody is attached to is aborted after this long when configured as a positive integer. Disabled by default (0). Detached agents retain their normal turn budget and deadline. NODETOOL_CHAT_REPLAY_RETENTION_MS — a finished turn is kept this long so a client reconnecting just after it ended still gets the tail (default 5 minutes). NODETOOL_CHAT_REPLAY_BUFFER_EVENTS — frames buffered per turn (default 2000). Positive integers override these defaults. Zero or invalid values use the default. Assistant and tool messages are persisted independently of the buffer, so an expired or truncated replay costs only unpersisted stream chunks — the client refetches thread history over REST. Job Run Replay A workflow run outlives the WebSocket connection that started it, the same way a chat turn does. Every frame the run emits is stamped with an increasing job_seq and appended to a bounded buffer; a client that reconnects sends {command: "reconnect_job", data: {job_id, last_seq}} and gets the missed tail replayed, followed by live frames if the run is still going. Three variables size that machinery: NODETOOL_JOB_DETACH_GRACE_MS — a running job nobody is attached to is cancelled after this long (default 10 minutes), so an abandoned client cannot leave a workflow burning provider spend forever. NODETOOL_JOB_REPLAY_RETENTION_MS — a finished session is kept this long so a client reconnecting just after the run ended still gets the tail (default 5 minutes). NODETOOL_JOB_REPLAY_BUFFER_EVENTS — frames buffered per run (default 2000). NODETOOL_JOB_DETACH_GRACE_MS=1800000 nodetool serve Each is read as a positive integer; a value that is not one is ignored and the default used. Terminal state is persisted to the jobs table independently of the buffer, so an expired or truncated replay degrades to reconnect_job’s persisted-row fallback — the run’s real status, just without its events. These size one process’s buffer. Getting a reconnect to the process that holds the run is a separate concern; see Multi-instance deployments. Backend Bundle Targeting These configure the build, not the server. The packaged Electron backend is one bundled server.mjs plus a flat _modules/ directory staged by scripts/bundle-backend.mjs. Some staged dependencies ship prebuilt binaries for every OS and architecture in a single package; staging all of them wastes disk in a single-target artifact, so the staging step prunes them to one target. NODETOOL_BUNDLE_PLATFORM — the platform to keep prebuilds for. Defaults to process.platform (darwin, linux, win32). NODETOOL_BUNDLE_ARCH — the architecture to keep prebuilds for. Defaults to process.arch (x64, arm64). Set both only when cross-building — staging on one machine for another target. A build for the host needs neither. NODETOOL_BUNDLE_PLATFORM=win32 NODETOOL_BUNDLE_ARCH=x64 node scripts/bundle-backend.mjs scripts/verify-backend-bundle.mjs reads the same two variables and resolves the @seydx/node-av-<platform>-<arch> binary it expects to find staged (node-av-win32-<arch>-msvc on Windows), so a verification run must be given the target the staging run used. When no prebuild matches the target, staging warns and leaves every prebuild in place rather than pruning the artifact down to nothing — a mis-set variable costs size, never a missing binary. Environment Variables Index Variable Purpose Secret Notes NODE_ENV Environment name (development, test, production) no Defaults to development; selects which .env.<NODE_ENV> files load. It does not switch production behavior. See NODETOOL_ENV NODETOOL_ENV Production mode switch no production turns off local-only features: the file browser and local-file previews, the Python bridge, the /mcp mount, the fake provider, Transformers.js, and unmanaged workspaces. Public app deployment routes are available only in production. It also makes the default bind address 0.0.0.0, makes cloud the default workspace storage, and tightens the node-pack allowlist default. The Docker image sets it. The desktop app does not, because it needs the Python bridge and Transformers.js, and sets NODETOOL_PACKS_REQUIRE_ALLOWLIST=1 instead. Separate from NODE_ENV PORT / HOST Port and address the server binds no Default 7777. HOST defaults to 127.0.0.1, or 0.0.0.0 when NODETOOL_ENV=production. nodetool serve --port and --host set both variables, and --host defaults to 127.0.0.1 STATIC_FOLDER Directory the server serves the built web app from no Unset, or naming a directory that is not there, no static handler is registered and anything outside the API routes answers 404. Set, the directory is served at /, / and /apps/index.html send index.html, and other extension-less GETs fall back to it so client-side routing survives a reload. See Serving the Web UI and TLS TLS_CERT / TLS_KEY PEM certificate and private key that put the server on HTTPS/WSS no Both are paths, and both must resolve or TLS stays off. A path that does not exist is ignored — the server then walks up to five directories from its working directory looking for cert.pem and key.pem, so a stray pair beside the process turns TLS on with neither variable set REDIRECT_PORT Port of the plain-HTTP listener that redirects to the HTTPS one no Default 80, and only bound when TLS is active. Answers 301 to https://<request Host>:<server port><path>. A port below 1024 needs elevated privileges; on EACCES the server logs that the redirect listener was skipped and keeps serving HTTPS SUPABASE_URL / SUPABASE_KEY Enable Supabase auth mode (both required) SUPABASE_KEY When both are set, the server enforces auth and validates Supabase JWTs. See Authentication SUPABASE_ANON_KEY The project’s public anon key, handed to the web app by GET /api/config no Designed to reach a browser — set it to the anon key, never SUPABASE_KEY (the service-role key), which this endpoint never returns. Required whenever auth is enforced: with SUPABASE_URL and SUPABASE_KEY set but this unset, /api/config still answers 200 with supabaseAnonKey: null, the web app falls back to a placeholder, and every login 401s. The server names that case at boot (describeMissingAnonKey in packages/websocket/src/routes/config.ts) AUTH_REDIRECT_URL URL Supabase sends the user back to after email or OAuth sign-in no Returned by GET /api/config as authRedirectUrl, and it wins over the client’s own resolution. Unset, the web app falls back to the build-time VITE_AUTH_REDIRECT_URL, then to window.location.origin + "/". Set it when the public URL is not the origin the browser sees — behind a proxy, on a custom domain, or in the Electron shell — and add the same value to the Supabase project’s redirect allow list. See Self-hosted deployment SERVER_AUTH_TOKEN Static bearer token for a Local-mode server no nodetool deploy generates one per deployment and passes it to the container. In Local mode a request with Authorization: Bearer <token> (or ?api_key= on a WebSocket) runs as user 1. Supabase mode ignores it and logs a warning. It selects no auth mode USERS_FILE Path to the JSON registry of API users and hashed bearer tokens no Default ~/.config/nodetool/users.json, or %APPDATA%\nodetool\users.json on Windows. The Docker image sets /workspace/users.json. In Local mode the server accepts each user’s token as a bearer credential for that user’s id. It selects no auth mode — the server picks that from the Supabase credentials (Authentication) — it is the file the admin user routes and nodetool deploy users-add \| users-list \| users-reset-token \| users-remove read and write. A file that is missing or unparseable reads as no users. See CLI › API users on the deployment ADMIN_USER_IDS User ids allowed to call the admin user-management routes no Comma-separated, whitespace trimmed. User 1 — the loopback user in local auth mode — is always admin, so a single-user install never needs this. An API user created with --role admin is admin too. Every other caller gets FORBIDDEN. A Supabase-mode server refuses to create API users or reset their tokens, because they authenticate only in Local mode NODETOOL_TRUST_LOCALHOST Allow loopback connections to bypass auth as user 1 no Defaults off when auth is enforced (Supabase), on otherwise. Leave off behind a reverse proxy/SSH tunnel where the proxy connects from loopback. NODETOOL_TRUST_LOCAL_NETWORKS ⚠️ Source CIDRs trusted as user 1 without a password (Local mode only) no Comma-separated IPs/CIDRs; ignored in Supabase mode. Needed so Docker’s NAT’d bridge traffic isn’t rejected — scope to the bridge (172.16.0.0/12), never 0.0.0.0/0 on a public IP. See Authentication → Local mode in Docker. NODETOOL_TRUSTED_PROXIES Reverse proxies whose X-Forwarded-For is trusted no Comma-separated IPs/CIDRs. When unset, X-Forwarded-For is ignored and the socket peer address is used. NODETOOL_ALLOWED_ORIGINS Extra browser origins allowed to make cross-origin requests no Comma-separated exact origins, added to the built-in list rather than replacing it: localhost, 127.0.0.1, and [::1] on any port and either scheme, the Electron renderer’s file://, and https://nodetool.ai and its subdomains. A single * entry restores allow-all, for a deployment fronted by its own gateway. A request with no Origin is allowed either way — there is no browser enforcing CORS in that case. The list is parsed once and cached, so a change needs a restart. It backs both the global CORS plugin and the hand-written headers on the storage and /mcp endpoints (packages/websocket/src/cors.ts) NODETOOL_LOCAL_FILE_ROOTS Directories the file browser and local-file previews may read no Platform-delimited (: on POSIX, ; on Windows), ~ expands. Defaults to the user’s home directory. Both surfaces are disabled entirely when NODETOOL_ENV=production. See Security hardening. NODETOOL_WORKSPACES_DIR Root for the workspace folders NodeTool manages itself no Defaults to <data dir>/workspaces. Every user gets a default workspace under here on first use, so a chat turn or a workflow that names no workspace still reads and writes somewhere bounded. In production it is the only readable workspace — a row pointing at another host folder is refused by listFiles and the download route. Point it at a mounted volume on a server deployment. NODETOOL_WORKSPACE_STORAGE Whether workspaces are folders on disk or objects in the asset bucket no local or cloud. Defaults to cloud when NODETOOL_ENV=production, local otherwise. A cloud workspace is a key prefix (workspaces/<user>/) in the same storage assets use, so it survives the machine being replaced; every node and agent tool reads and writes it through the same Workspace interface either way. Set local on a self-hosted server with a mounted volume, or cloud on a local install pointed at S3. OPENAI_API_KEY / ANTHROPIC_API_KEY / GEMINI_API_KEY Provider access yes Set only the providers you use HF_TOKEN / FAL_API_KEY / REPLICATE_API_TOKEN HuggingFace-family providers yes Optional per workflow HF_API_TOKEN / HUGGING_FACE_HUB_TOKEN Alternate spellings of HF_TOKEN for Hub requests yes The Hub client takes the first non-empty of HF_TOKEN, HF_API_TOKEN, HUGGING_FACE_HUB_TOKEN, trimmed. They exist so an environment already configured for huggingface_hub or the older huggingface-cli works unchanged — set HF_TOKEN on a fresh install. The resolved token is cached in-process after the first read, so changing it needs a restart HF_TOKEN_PATH Token file read when none of the three variables is set yes A leading ~ expands. Default <HF_HOME>/token, and with HF_HOME unset that is $XDG_CACHE_HOME/huggingface/token, falling back to ~/.cache/huggingface/token — the file huggingface-cli login writes. A path that is missing or unreadable reads as no token rather than an error, so an anonymous Hub request is what a typo produces HUGGINGFACE_API_KEY Key the huggingface inference nodes send yes Resolution order is stored secret HF_TOKEN, stored secret HUGGINGFACE_API_KEY, then the same two names from the environment. The token needs the Inference Providers permission, which a Hub read token does not carry; without any of the four the node throws HF_TOKEN is not configured. Separate from the Hub client above, which never reads this name HF_HUB_CACHE Directory holding the models--* folders of the HuggingFace cache no Used verbatim — it names the hub directory itself, not its parent. A leading ~ expands. Unset, NodeTool uses the legacy alias HUGGINGFACE_HUB_CACHE (also verbatim), then $HF_HOME/hub, then $XDG_CACHE_HOME/huggingface/hub, then ~/.cache/huggingface/hub, the same order as the Python huggingface_hub library. The server resolves it once for Model Manager downloads, the installed list and downloaded badges, nodetool models hf-cache and download-hf, node-llama-cpp, and whisper.cpp. The Python worker takes the location from huggingface_hub itself, which applies the same order to the same variables, so both see one cache. The desktop app sets HF_HOME for the server and the worker: your own HF_HOME, else $XDG_CACHE_HOME/huggingface, else ~/.cache/huggingface. When XDG_CACHE_HOME is set (Flatpak sets it per app) but ~/.cache/huggingface/hub already holds models, the app keeps using ~/.cache/huggingface, so an update does not hide models downloaded earlier NODETOOL_HF_DOWNLOAD_CONCURRENCY Files of one HuggingFace repo downloaded at once no Default 8. A positive integer, anything else falls back to the default. The server and the Python worker read the same variable. Lower it (NODETOOL_HF_DOWNLOAD_CONCURRENCY=2) when a repo with many shards draws HTTP 429 from the Hub OLLAMA_API_URL Ollama base URL no Default http://127.0.0.1:11434. Set it in Settings → Integrations → Local Model Servers or the environment. A stored value wins over the environment. Ollama is a separate program that must be running; NodeTool does not start it OLLAMA_CONTEXT_LENGTH Context window, in tokens, sent to Ollama as num_ctx with every chat request no Unset, NodeTool uses the model’s Modelfile num_ctx, else the model’s trained length capped at 32768. Ollama’s own default is a few thousand tokens and drops the oldest part of a long prompt OLLAMA_KEEP_ALIVE How long Ollama keeps a model loaded after a request no Default 10m, so large models are not evicted between turns. Accepts Ollama’s duration syntax: 30m, -1 to keep forever, 0 to unload immediately. See Models & Providers LMSTUDIO_API_URL Base URL of the local server LM Studio’s desktop app exposes no Default http://127.0.0.1:1234; trailing slashes are stripped. A value stored under the same name in Settings → Integrations → Local Model Servers wins over the environment variable. See Providers › LM Studio LMSTUDIO_API_KEY Bearer key sent to the LM Studio server yes Default lm-studio. Set it only when LM Studio’s server requires authentication VLLM_BASE_URL Base URL of a self-hosted, OpenAI-compatible vLLM server no Required to use the provider — there is no default, and constructing it without one throws VLLM_BASE_URL is required (options.baseURL, secret, or env). Same secret-over-environment precedence; trailing slashes are stripped. Models appear from the server’s /v1/models endpoint. See Providers › vLLM DASHSCOPE_BASE_URL Region endpoint for Alibaba Cloud Model Studio (the Qwen models) no Default https://dashscope-intl.aliyuncs.com/compatible-mode/v1 — the international (Singapore) region. Model Studio keys are region-scoped, so set this to your region’s /compatible-mode/v1 endpoint when the key was created elsewhere. A stored setting of the same name wins over the environment variable, and key verification probes whichever endpoint this resolves to. See Providers › Alibaba Cloud DATA_FOR_SEO_LOGIN / DATA_FOR_SEO_PASSWORD DataForSEO credentials behind the web_search capability DATA_FOR_SEO_PASSWORD Both required — one alone leaves the backend unconfigured. Read from the stored secret first, then the environment. web_search runs the first configured backend of serpapi, dataforseo, openai, gemini unless the call pins one with provider, so these take effect when SERPAPI_API_KEY is unset. DataForSEO serves all three search types (web, news, images) against https://api.dataforseo.com, defaulting to location code 2840 (United States) and language en. Once a backend runs, its failure is the call’s failure — nothing falls through to the next one GOOGLE_MAIL_USER / GOOGLE_APP_PASSWORD Gmail account the email capability reads over IMAP GOOGLE_APP_PASSWORD Both required; read from the stored secret first, then the environment. The password is a Google app password, not the account password — 2-step verification has to be on to mint one. The capability connects to imap.gmail.com:993 over TLS and backs search_email, archive_email, and add_label_to_email. Unrelated to GOOGLE_CLIENT_ID/GOOGLE_CLIENT_SECRET, which serve the Google Workspace integration KIE_WEBHOOK_URL Public base URL kie.ai calls back when a task finishes no Unset, kie tasks are polled. Set, a submission carries callBackUrl: <value>/api/kie/webhook and the run waits for that request instead of polling — so it has to be an address kie.ai can reach from the internet, e.g. https://nodetool.example.com. Trailing slashes are stripped GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET OAuth App credentials for the GitHub sign-in flow at /api/oauth/github/* GITHUB_CLIENT_SECRET Without the id, /api/oauth/github/start answers 500 naming it. The flow builds its redirect URI from the request’s own Host — http://<host>/api/oauth/github/callback for a localhost host, https://… otherwise — so register exactly that on the OAuth App. Read from the process environment, not the encrypted secret store: a value entered only in Settings never reaches this code path GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET OAuth client the server refreshes an expired Google access token with GOOGLE_CLIENT_SECRET The same pair configured for the Google provider in the Supabase dashboard. Supabase does not refresh provider tokens, so without them a Google credential stops working an hour after sign-in and the server logs Google token expired but GOOGLE_CLIENT_ID/GOOGLE_CLIENT_SECRET are unset. Set them wherever the Google Workspace capability is on — see NODETOOL_GOOGLE_WORKSPACE below NODE_LLAMA_CPP_MODELS_DIR Directory the node_llama_cpp provider loads GGUF models from no Unset, the llama.cpp cache (see LLAMA_CACHE). A secret stored under the same name wins over the environment variable LLAMA_CPP_CONTEXT_LENGTH Context size, in tokens, llama-server was started with (-c) no The server does not report it, so the llama_cpp provider budgets prompts with this value. Unset, the model’s published context window is used WAN2GP_MCP_URL MCP endpoint of a Wan2GP server for the Wan2GP pack’s nodes no Default http://127.0.0.1:7866/mcp. A node’s non-blank server_url wins. The Python worker receives it when it starts, so restart after changing it. See Wan2GP NODETOOL_TORCH_DEVICE Device Python nodes run PyTorch on no cuda, cuda:<N>, mps, or cpu. Unset, MPS, then CUDA, then CPU. See Python Nodes WHISPER_CPP_MODELS_DIR Extra directory for whisper.cpp GGML and VAD models no The Hugging Face hub cache is always scanned WHISPER_CPP_GPU_BACKEND whisper.cpp backend: auto, metal, cuda, vulkan, or cpu no auto uses Metal on macOS. On Windows and Linux it tries the CUDA build, then the Vulkan build, then the default CPU build. Restart after changing WHISPER_CPP_SERVER_URL Base URL of a user-run whisper-server no Required for whisper_cpp_server NODE_LLAMA_CPP_GPU_BACKEND GPU backend node-llama-cpp runs against no auto, metal, cuda, vulkan, or cpu, matched case-insensitively. Any other value is ignored and the library chooses for itself. Same secret-over-environment precedence LLAMA_CACHE Cache root checked for GGUF files a separate llama.cpp already downloaded no Same variable llama.cpp reads. Default ~/Library/Caches/llama.cpp on macOS, %LOCALAPPDATA%\llama.cpp on Windows, $XDG_CACHE_HOME/llama.cpp (or ~/.cache/llama.cpp) on Linux. Consulted only when the file is not already in the HuggingFace cache. On Linux with XDG_CACHE_HOME set and LLAMA_CACHE unset, the desktop app sets LLAMA_CACHE to ~/.cache/llama.cpp while only that directory holds files, so models downloaded before a Flatpak update stay visible. Replaces LLAMA_CPP_CACHE_DIR, which is no longer read: rename the variable if you set it TRANSFORMERS_JS_CACHE_DIR Cache directory for the Transformers.js runtime no Default <data dir>/transformers-js-cache. Set it in Settings → Integrations → Local Model Servers or the environment. Deliberately outside ~/.cache/huggingface: Transformers.js uses a flat {cacheDir}/{repo_id}/{file_path} layout the Python huggingface_hub cache cannot share NODETOOL_INTEGRATION_TOKEN Service token for messaging-bridge integrations (Telegram bot) yes ≥16 chars. Enables /api/integrations/:provider/* (account linking + delegated tokens); unset, those routes do not exist. Set the same value on the bridge process. See telegram-bot-design.md §5 NODETOOL_PUBLIC_URL HTTPS base URL reachable by browsers and hosted provider callbacks no Trailing slashes are stripped. Used for integration link URLs, for FAL callback URLs at /api/providers/fal/webhook/:token, and for AtlasCloud callback URLs at /api/providers/atlascloud/webhook. Both providers’ callbacks are included only when this is a valid https URL on a public host that the provider can reach. An http, loopback, or private address is ignored rather than sent. Unset, local FAL requests use queue polling. AtlasCloud predictions are polled either way: a callback only shortens the wait, and the prediction endpoint is still read every 15s while one is outstanding. Integration links still fall back to the request’s Host header, which may be useful when a bridge reaches the server at an address the user’s browser cannot, such as http://nodetool:7777 inside a compose network. See telegram-bot-design.md §5 TELEGRAM_BOT_USERNAME Bot username for the Telegram link deep link no Without the @. When set, Settings → Integrations renders t.me/<username>?start=<code> links; unset, the UI shows the bare code for manual /start entry NODETOOL_API_URL Server the CLI and the Telegram bridge talk to no Default for the --api-url flag on the read commands (workflows, jobs, assets, models list/ollama/huggingface), which otherwise read the local database; an explicit --api-url wins. The Telegram bridge reads it directly and defaults to http://127.0.0.1:7777. See CLI TELEGRAM_BOT_TOKEN Bot token from @BotFather, read by the nodetool telegram bridge process yes Required. Read on the bridge, not the server. With it unset, nodetool telegram serve refuses to start and names the field. See CLI › nodetool telegram TELEGRAM_WEBHOOK_URL Absolute https:// URL Telegram should deliver updates to no Unset = getUpdates long polling, which is the only mode implemented. Setting it makes nodetool telegram serve refuse to start rather than silently poll TELEGRAM_WEBHOOK_SECRET Value verified against the X-Telegram-Bot-Api-Secret-Token header yes Required whenever TELEGRAM_WEBHOOK_URL is set — config validation rejects the pair otherwise NODETOOL_GOOGLE_WORKSPACE Force the Google Workspace integration (Drive, Gmail, Docs, Sheets, Calendar) on or off no 1/true on, 0/false off. Unset follows Supabase auth mode — the google capability signs in with the token a Google login returns, so a local install with no login hides it instead of offering an integration that can only error. Set 1 on a local server pointed at a hosted Supabase project NODETOOL_SYSTEM_STATS Force the system_stats WebSocket broadcast (the editor’s CPU/RAM readout) on or off no 1/true on, 0/false off. Unset follows Supabase auth mode: a shared server sends nothing, because the figures describe a container the user does not own; a local install sends them. Set 1 on a local server pointed at a hosted Supabase project. NODE_ENV is not consulted — the desktop app and the Docker image both set it to production while serving one user DB_PATH / DATABASE_URL Database connection no Set only one. DB_PATH configures SQLite; DATABASE_URL supports PostgreSQL (postgres://, postgresql://) and SQLite (file:, sqlite:) DATABASE_POOL_MAX Maximum PostgreSQL connections one server process opens no Default 10. A Supabase session-mode pooler (port 5432 on pooler.supabase.com) holds one server connection per client and rejects new clients with EMAXCONNSESSION at its pool size, so this value times the number of server processes must stay below that size. A transaction-mode URL (port 6543) shares server connections, and the server turns off prepared statements for it DIRECT_URL Dedicated PostgreSQL URL used by migration commands and the release-phase migration runner no Resolution order is --direct-url, --database-url, DIRECT_URL, then DATABASE_URL; with none of them the command fails naming all four. On Supabase this is the project’s direct connection URL, which is separate from the pooled DATABASE_URL used by the server. Set it on deployments so migrations do not exhaust the pooler’s session limit. See CLI › Database Migrations NODETOOL_STORAGE_AUTO_CLEANUP Expire database history on a schedule instead of when someone asks no 1 or true, trimmed and case-insensitive; unset, 0, or anything else keeps the sweep manual. It only supplies the default for the per-user storage.retention.automaticCleanup setting, so a value saved from the UI wins over it. Turned on, the server sweeps at startup and every 6 hours, and each sweep does work only if the last one finished more than 24 hours ago: workflow autosaves past 10 per workflow or 7 days, manual versions past 90 days, terminal jobs past 30 days, run events past 30 days, and the parameters/metadata/logs columns of predictions past 400 days (prediction rows themselves are billing records and survive). Current workflows, assets, messages, memories, and authored documents are never candidates, and each window is overridable per user under storage.retention.*. The startup sweep runs for user 1 only, so on a multi-user deployment it covers that account and no other. Set it where a published retention promise has to hold without anyone pressing a button NODETOOL_ERROR_TRACES Server-side error trace capture no On by default. 0, false or off stops it. See Error Tracing NODETOOL_ERROR_TRACE_RETENTION_DAYS Days an error trace is kept no Default 30. Pruned at startup and once a day NODETOOL_ERROR_TRACE_SYNC_URL / NODETOOL_ERROR_TRACE_SYNC_TOKEN Push redacted error traces to another NodeTool server yes Both required, or sync stays off. The URL must be https except for localhost. The token is an access token for that server NODETOOL_STORAGE_BACKEND Storage backend (file, s3, supabase) no Default file. Selected explicitly — not auto-detected from credentials ASSET_FOLDER / STORAGE_PATH Directory NodeTool keeps assets in on the local filesystem no ASSET_FOLDER wins; STORAGE_PATH is the fallback, and with neither set the path is <data dir>/assets — ~/.local/share/nodetool/assets, or %APPDATA%\nodetool\assets on Windows. Both are used verbatim, so a relative path resolves against the process’s working directory. This is the file backend’s root, and also what the CLI harnesses (nodetool run, debug, node run, eval) read and write directly, whatever NODETOOL_STORAGE_BACKEND is set to. Point it at the mounted volume when running in Docker, or job containers cannot reach the assets. See Storage S3_* S3-compatible storage settings yes Includes access keys and region ASSET_BUCKET / TEMP_BUCKET Asset and temp buckets (s3 / supabase backends) no Use signed URLs for private buckets NODETOOL_VECTOR_PROVIDER / VECTORSTORE_DB_PATH Vector store config no Default backend is local SQLite-vec; switch to pinecone or supabase for remote. See Indexing. NODETOOL_VECTOR_SCHEMA PostgreSQL schema the supabase vector provider reads and writes no Default public. Read only when NODETOOL_VECTOR_PROVIDER=supabase; the SQLite-vec and Pinecone backends ignore it. Set it when the migration in packages/vectorstore/sql/supabase-migration.sql was installed into a schema of its own rather than public. See Vector storage NODE_SUPABASE_URL / NODE_SUPABASE_KEY / NODE_SUPABASE_SCHEMA / NODE_SUPABASE_TABLE_PREFIX User/node Supabase config NODE_SUPABASE_KEY Kept separate from core Supabase credentials and tables NODETOOL_RATE_LIMIT_DISABLED Disable per-IP HTTP rate limiting no Limiter is on by default; localhost is always exempt NODETOOL_RATE_LIMIT_MAX Max HTTP requests per window per IP no Default 1000 NODETOOL_RATE_LIMIT_WINDOW_MS Rate-limit window length (ms) no Default 60000 (1 minute) NODETOOL_RATE_LIMIT_TRUST_PROXY Key the limiter by X-Forwarded-For (req.ip) instead of the socket address no Enable only behind a trusted proxy that sets the header NODETOOL_WS_RATE_LIMIT_DISABLED Disable the per-connection WebSocket inbound message cap no Cap is on by default NODETOOL_WS_RATE_LIMIT_MAX Max inbound WS messages per window per connection no Default 200; over-cap clients are closed with code 1008 NODETOOL_WS_RATE_LIMIT_WINDOW_MS WebSocket rate-limit window length (ms) no Default 1000 (1 second) NODETOOL_WS_HEALTH_DISABLED Disable the per-connection ping / idle-timeout watchdog no Watchdog is on by default; it terminates half-open peers that never sent a close frame NODETOOL_WS_PING_INTERVAL_MS How often each WebSocket peer is pinged no Default 20000 NODETOOL_WS_IDLE_TIMEOUT_MS Peer silence before the connection is terminated no Default 70000; keep above the client’s own 45s liveness threshold NODETOOL_WS_MAX_BUFFERED_BYTES Outbound buffer per connection before sends wait for drain no Default 8388608 (8 MiB) NODETOOL_WS_DRAIN_TIMEOUT_MS How long a send waits for a slow reader before it is dropped no Default 30000; the drop uses code 1001 so clients reconnect NODETOOL_WS_MAX_QUEUED_FRAMES Undelivered inbound frames per connection before it is closed no Default 2000; closes with code 1008 NODETOOL_WS_MAX_MESSAGE_BYTES Largest inbound WebSocket frame accepted before it is deserialized no Default 268435456 (256 MiB). MsgPack can expand a small frame into a huge structure, so the raw byte length is checked first. A non-numeric or non-positive value falls back to the default rather than turning the cap off NODETOOL_MAX_UPLOAD_BYTES Largest payload a single storage upload may write no Default 1073741824 (1 GiB) for cloud and in-memory storage, 2147483648 (2 GiB) for local file storage reads and writes, including media staging. An explicit value applies to both paths. Same strict parsing as the frame cap NODETOOL_DISABLE_TRIGGERS Skip trigger ingestion on this process (no dispatcher, scheduler, file watcher, or webhook route) no Ingestion is on by default. Set to 1 when a second server shares one database, or for an embedded server that must not start background work NODETOOL_DISABLE_SDK_LIFECYCLE_V1 Turn off the SDK lifecycle routes: /api/sdk/v1/capabilities, /preflight, and /assets/temporary no 1 only; on by default. Disabled, each answers 503 with {"code": "SDK_LIFECYCLE_DISABLED"} rather than 404, so a client can tell “switched off here” from “wrong URL”. /api/sdk/v1/models and the model-download routes are not covered — they stay available. See API Reference › What This Server Supports NODETOOL_DISABLE_SDK_WORKFLOW_INTERFACE_V1 Turn off the SDK discovery routes: GET /api/sdk/v1/workflows/:id/interface, POST /api/sdk/v1/workflow-interfaces, GET /api/sdk/v1/workflows, and GET /api/sdk/v1/node-types no 1 only; on by default. Disabled, they answer 503 with {"code": "SDK_WORKFLOW_INTERFACE_DISABLED"} — except node-types, which reports SDK_NODE_TYPE_INVENTORY_DISABLED. Clients then have no way to read a workflow’s pins without fetching its graph NODETOOL_REQUIRE_SDK_AUTH_V1 Require a token on the SDK discovery routes even when the server does not enforce auth no 1 only. Those routes (capabilities, models, node-types, workflows, workflow-interfaces, workflows/:id/interface) are otherwise auth-exempt in local trust mode, since they only describe the server. Set it when a local-mode server is reachable beyond loopback. A server that already enforces auth requires the token regardless NODETOOL_EXTENSION_DIST Directory holding the built Chrome extension served by /api/extension/download no Set by the desktop app to its bundled copy. When unset (or pointing at a directory with no manifest.json), the server walks up from its own directory and the working directory looking for chrome-extension/dist. See Chrome Extension NODETOOL_BROWSER_TRANSPORT Which browser the browser_* capabilities drive no extension drives the user’s own signed-in Chrome through the native host’s socket; anything else, or unset, launches a headless Chrome in the process (local). A browser_restart call pins the transport for the rest of the process and wins over it. See Chrome Extension NODETOOL_BROWSER_BRIDGE_SOCKET Socket of the native host to use no Default: the newest <pid>.sock in <tmp>/nodetool-browser-bridge-<user>/. The host also listens here when the variable is set in the environment that starts it. Set it to pin one extension when several browsers run the host NODETOOL_ENABLE_MCP Keep the /mcp streamable-HTTP mount registered when NODETOOL_ENV=production no Off in production unless set to exactly 1; on everywhere else. The mount inherits the server’s auth mode — it is not a second door — and binds the user the auth hook resolved, so a request it cannot authenticate gets 401 at initialize instead of an anonymous session. A session id belongs to the user who opened it: another user presenting it gets 404. When disabled the server logs that at startup and the route is not registered. An agent authenticates with a token minted in Settings → MCP → Connect an agent remotely; see MCP on a production server NODETOOL_DISABLE_MCP_OAUTH Turn off the MCP OAuth 2.1 authorization-server surface no 1 only. Disabled: /mcp never sends a WWW-Authenticate challenge, and /.well-known/oauth-*, /oauth/authorize, /oauth/token, /oauth/register, and /oauth/revoke all 404. Pasting an ntk_ token (Settings → MCP → Connect an agent remotely) stays the only way to connect. Setting the flag also refuses outstanding nta_ access tokens immediately. The flow needs no enable flag of its own — it activates wherever /mcp is mounted (in production that requires NODETOOL_ENABLE_MCP; in dev the mount is on by default) once NODETOOL_PUBLIC_URL is set to an HTTPS or loopback URL. See MCP OAuth design NODETOOL_MCP_URL /mcp endpoint the .mcpb bundle’s stdio bridge connects to no Read by the bridge process the MCP bundle launches (scripts/mcpb/bridge.mjs), not by the server. Default http://127.0.0.1:7777/mcp. Claude Desktop writes it from the bundle’s NodeTool server URL config field; set it by hand only when launching the bridge yourself. Point it at a deployed server’s /mcp and pair it with a token — see MCP on a production server NODETOOL_MCP_TOKEN Bearer token the .mcpb bridge sends with every forwarded request yes Same process, same source — the bundle’s Auth token (optional) config field. Optional: unset, the bridge connects with no Authorization header, which is what a loopback server in local auth mode expects. A remote server needs an ntk_ token minted in Settings → MCP → Connect an agent remotely NODETOOL_MCP_RETRY_MS How often the .mcpb bridge retries while the server is unreachable no Default 5000. Unlike the two above there is no config field for it, so it only takes effect when you set it in the client’s own environment. The bridge starts even with nothing to connect to, serving a single nodetool_status tool, and attaches with list_changed notifications once the server appears NODETOOL_APP_BUILD_PROVIDER / NODETOOL_APP_BUILD_MODEL Provider and model POST /api/applications/build falls back to no Read only when the request body omits provider / model; the body wins. With neither the body nor both variables set, the build is refused with invalid_input naming them. The judge model is chosen separately — see NODETOOL_APP_JUDGE_MODEL under CLI › nodetool app. Set them on a server that builds apps for callers who should not have to pick a model NODETOOL_DESKTOP_USER_DATA The desktop app’s userData directory, used as the data root when NODETOOL_HOST=desktop no Electron sets it (with NODETOOL_HOST=desktop) in the backend environment, and a value already set wins. The runtime-install directories (optional-node, uv-tools, micromamba) resolve under it. With NODETOOL_HOST=desktop and no value, runtime-location lookup throws The desktop host must supply its userData directory. Other hosts ignore it. See Runtime install design NODETOOL_HOST_BINARY_CONCURRENCY Host binaries (ffmpeg, yt-dlp) the media capabilities may run at once no Default 2; the rest queue, so one run cannot take every core from the request handlers. Read per spawn, as a whole number ≥ 1 — anything unparseable or non-positive keeps the default. Raise it on a machine with cores to spare and lower it on a shared one NODETOOL_APIFY_MODE Which Apify actors the apify capabilities may search and run no One of disabled, allowlist, discovery, unrestricted; default discovery, and an unrecognized value falls back to it rather than failing. discovery searches the whole store freely, runs an allowlisted actor directly, and sends anything else through the permission gate first. allowlist refuses store search and runs only allowlisted actors; unrestricted runs any actor, for a trusted environment. See Apify integration NODETOOL_APIFY_ALLOWED_ACTORS Actor ids added to the allowlist no Comma-separated, canonicalized, blanks dropped. Added to the shipped catalog, never replacing it — a replacing list would silently turn off the wrapper capabilities. Takes effect in allowlist mode, and in discovery mode decides which actors run without asking the user. The per-session ceilings are the separate NODETOOL_APIFY_MAX_RUNS / MAX_ITEMS / MAX_RUN_SECONDS / MAX_MEMORY_MB / MAX_COST_USD NODETOOL_APIFY_MAX_RUNS Actor runs one session may start no Default 10. A value that is not a positive number falls back to the default NODETOOL_APIFY_MAX_ITEMS Result items one actor run may return no Default 1000. Same fallback rule NODETOOL_APIFY_MAX_RUN_SECONDS Seconds one actor run may take no Default 300. Same fallback rule NODETOOL_APIFY_MAX_MEMORY_MB Memory in MB one actor run may be given no Default 4096. Same fallback rule NODETOOL_APIFY_MAX_COST_USD Total Apify spend one session may incur, in USD no Default 5. Same fallback rule APIFY_API_TOKEN Apify API token for the apify capabilities and Apify-backed web search yes Get it at console.apify.com/account/integrations. APIFY_API_KEY is the deprecated alias and is still read as a fallback. See Apify integration SERP_PROVIDER Search provider behind web search no One of serpapi, dataforseo, brave, apify, openai, gemini. Read from the stored secret first, then the environment. Default serpapi. See SerpAPI integration BRAVE_API_KEY Brave Search API key yes Required when SERP_PROVIDER is brave. Get it at api-dashboard.search.brave.com BLENDER_PATH Path to the Blender executable for the nodetool.blender nodes and 3D renders no Checked first. Unset, NodeTool looks for blender on PATH, then in the usual install locations for macOS, Linux, and Windows. Blender 5.2 or newer is required. See Blender NODETOOL_CREDIT_MODELS Managed models hosted credits may be spent on no Separated by commas, whitespace, or newlines. Unset means the whole nodetool provider catalog. Any model not listed is refused before a platform key is used. See Costs and credits NODETOOL_SIGNUP_CREDITS One-time credits granted to a user seen for the first time no Default 500. Fractions round down. 0, a negative number, or a non-number grants nothing NODETOOL_PLATFORM_FAL_KEY / NODETOOL_PLATFORM_ANTHROPIC_KEY Platform-owned fal and Anthropic keys the nodetool provider runs managed models on yes A managed model is offered only when the key for its delegate provider is set. Cloud only. See Costs and credits NODETOOL_SUPERVISOR_PROVIDER / NODETOOL_SUPERVISOR_MODEL Provider and model that supervise a run when nothing nearer names one no Only consulted for a run that asked to be supervised; neither turns supervision on. On the server the order is the request, then the connection’s configured defaults, then these — they exist for hosts that have no per-connection defaults at all, such as the trigger dispatcher’s headless runs. A supervised run that resolves neither logs that and runs unsupervised rather than failing. The CLI reads NODETOOL_SUPERVISOR_MODEL alone, as a single provider/model string behind --supervisor-model, defaulting to anthropic/claude-sonnet-4-6. See CLI › Supervised runs NODETOOL_CHAT_COMPACTION_TOKENS Prompt size at which a chat turn summarizes the earlier part of its thread no Unset, the threshold is 90% of the model’s input window. The window comes from the provider’s Models API (Anthropic max_input_tokens, with the provider’s beta headers), then models.dev by provider and model, then a 128000-token fallback that logs a warning. On a small window the threshold drops further so the summarizer call still fits. The prompt size is the larger of the provider-reported size of the thread’s last call and a local estimate of the messages. A provider that holds the conversation itself (a resumed session, the Claude Agent SDK) is not measured this way and compacts only when it reports that the transcript no longer fits NODETOOL_CHAT_COMPACTION_KEEP_TURNS User turns a compaction leaves verbatim no Default 4. Everything before them becomes the summary, and the cut lands on a user message so a tool call is never separated from its result NODETOOL_COMPACTION_MODEL Model that writes the compaction summary no provider/model, or a bare model id on the turn’s own provider. Unset, the turn summarizes with the model it is already running. A summarizer that fails leaves the thread uncompacted and the turn runs against the full history NODETOOL_INSTANCE_ID Which server instance this process is, for multi-instance routing no Overrides FLY_MACHINE_ID, which Fly sets and whose value the proxy’s fly-replay: instance=<id> header also addresses — so one string both stamps a job row and routes a handshake back to the instance that owns the run. Letters, digits, _ and - only: the value is written into a response header verbatim, so anything else is refused with a one-time warning. Unset or refused means single-machine, and every feature keyed off it is inert. Set it on a non-Fly multi-instance deployment NODETOOL_JOB_CANCEL_POLL_MS How often an instance re-reads its own running jobs to notice a remote cancel no Default 15000; 0 disables the poll. Anything unparseable or negative keeps the default. It only matters alongside NODETOOL_INSTANCE_ID / FLY_MACHINE_ID — a cancel arriving at the instance that holds the run is immediate and never waits for this NODETOOL_SHUTDOWN_GRACE_MS How long SIGTERM waits for aborted turns and runs to settle before the process exits no Default 240000; anything unparseable or negative keeps it. Under Fly’s 300 s cap between the signal and the kill. It bounds only the fallback path: a deploy drains a machine with SIGUSR2 first and waits on /health instead, which has no deadline. Turns and runs still executing when it expires are logged and dropped, so their transcripts stay incomplete. See WebSocket API › Draining NODETOOL_VIDEO_FRAME_FALLBACK 0 refuses a video sent to a provider with no video content part instead of sampling frames from it no Any other value (or unset) keeps the fallback on. See Video Input on Vision Models NODETOOL_VIDEO_FRAME_MAX_FRAMES Frames sampled from one clip no Default 16. Anything unparseable or non-positive keeps the default NODETOOL_VIDEO_FRAME_MAX_FPS Ceiling on the frame sample rate no Default 1. A clip too long for the frame budget to reach this rate is sampled more sparsely, so the frames still span it NODETOOL_VIDEO_FRAME_MAX_DIMENSION Longest edge of a sampled frame, in pixels no Default 768. Frames are never upscaled NODETOOL_SHIPPED_PACKS_DIR Roots the sandbox packs that ship with NodeTool are read from no Comma-, semicolon-, or PATH-separator-delimited, same as NODETOOL_PACK_SEARCH_PATHS. Candidates that do not exist are dropped, so a bad path yields no packs rather than an error. Unset, the loader looks for _sandbox/ beside the bundled server.mjs (packaged desktop app, Docker image), then walks up to packages/sandbox-packs (a checkout). Set it only for a host that stages the packs somewhere else. See Sandbox package design NODETOOL_SYSTEM_SKILLS_DIR Root the system skills that ship with NodeTool are read from no One path, not a list. Unset, the loader looks for _skills/ beside the bundled server.mjs (packaged desktop app, Docker image), then walks up to packages/system-skills (a checkout). A path that does not exist means no system skills rather than an error, and so does an unset variable in a build that ships none. Each skill is a SKILL.md under a directory that names it — a frontmatter name disagreeing with the directory is skipped, as is a file that cannot be parsed. Read once per process. Set it only for a host that stages the skills somewhere else. See Agent Architecture › Skills NODETOOL_SANDBOX_INPROC Run every QuickJS guest on the calling thread instead of a worker no 1 only. A chosen fallback, so it warns about nothing. A CPU-bound guest then blocks the thread — on the server’s main thread that freezes the event loop, including the frame that would have cancelled the run. See JavaScript sandbox threading NODETOOL_SANDBOX_WORKER Require the sandbox worker path no require only. A run that cannot reach a worker fails with the sandbox worker path is required (NODETOOL_SANDBOX_WORKER=require) but unavailable: <reason> rather than falling back in-process. Runs that stream their inputs never reach a worker, so this fails them NODETOOL_GPU_VALIDATE Escape hatch for the WGSL linearity validator no off disables it. The validator rejects a shader module whose WGSL contradicts its declared premultiplied-alpha contract, at module load. Use it to ship a hotfix while the shader is corrected, not as a standing setting; read once per process NODETOOL_GPU_DEBUG Comma-separated GPU debug passes to enable no premul scans every premultiplied output texture after dispatch and logs texels that break the invariant (rgb ≤ a, rgb ≥ 0, no NaN): NODETOOL_GPU_DEBUG=premul. Off by default and zero cost when off — the pass is never encoded. Read once per process NODETOOL_CACHE_DIR Per-user cache root for derived artifacts NodeTool can always rebuild no Everything under it is safe to delete — it is deliberately separate from the data directory. Unset, it is %LOCALAPPDATA%\nodetool\cache on Windows and $XDG_CACHE_HOME/nodetool (falling back to ~/.cache/nodetool) elsewhere. The compiled sandbox guest modules live in sandbox-modules/ under it, cached by content digest; see Sandbox package design NODETOOL_PACKAGE_ASSETS_DIR Directory package://<pkg>/<file> refs are resolved from on disk no Set by the server at startup to the first package-assets root it finds, so in-process workflow execution reads the bytes directly instead of an HTTP round-trip to its own /api/assets/packages/… route. Set it yourself only when embedding the runtime with no server in the process. Refs are confined to the root — a path escaping it is refused NODETOOL_BASE_EXAMPLES_DIR Directory the shipped example workflows are read from no Overrides detection when it exists on disk; a path that does not exist is ignored rather than fatal. Unset, the server looks beside its own entry point (the packaged layout), then at packages/base-nodes/nodetool/examples/nodetool-base (the monorepo layout). With none found it logs Examples directory not found and template workflows are unavailable. The resolved path is logged at startup NODETOOL_EXAMPLE_COMPOSITIONS_DIR Directory the shipped timeline compositions are read from no Unlike NODETOOL_BASE_EXAMPLES_DIR above, a path that does not exist is not ignored: it yields no shipped compositions at all rather than falling back to detection, so list_compositions returns only the user’s own. Unset, the loader looks for examples/compositions beside the bundled server.mjs (packaged desktop app, Docker image), then walks up to packages/base-nodes/nodetool/examples/compositions (a checkout). Set it only for a host that stages them somewhere else NODETOOL_EXAMPLE_GAMES_DIR Directory the shipped example games (*.game.json) are read from no Like NODETOOL_EXAMPLE_COMPOSITIONS_DIR, a path that does not exist is not ignored: it yields no example games rather than falling back to detection. Unset, the loader walks up to eight directories from the module looking for examples/games, then packages/base-nodes/nodetool/examples/games (a checkout). Set it only for a host that stages the games somewhere else NODETOOL_EXAMPLE_TIMELINES_DIR Directory the shipped example timelines (*.timeline.json) are read from no Same resolution as NODETOOL_EXAMPLE_GAMES_DIR: a path that does not exist yields no example timelines, and unset the loader searches examples/timelines and then packages/base-nodes/nodetool/examples/timelines upward from the module. Set it only for a host that stages the timelines somewhere else NODETOOL_DISABLE_DURABLE_GENERATIONS 1 turns off durable queue tracking for fal.ai media generations started in a chat session no Any other value, or unset, keeps it on. By default a fal.ai generation is recorded in the database as it is accepted, so a restart can recover it. With 1 the call runs without that record. Other providers never use it. The code reserves this for an ephemeral test or local host. See Media generation tracking NODETOOL_PYTHON Python interpreter the Python bridge spawns no An absolute path to the executable. When unset, an active CONDA_PREFIX that looks like a NodeTool env is tried, then the desktop app’s env and nodetool conda envs in the usual places. See Python Nodes NODETOOL_PYTHON_ON_DEMAND Start the Python worker on first use instead of at server start no true or false, default false. With true, huggingface-local and MLX models appear only after a workflow has started the worker. A stored value wins over the environment. Read at startup. See Python Nodes NODETOOL_ALLOW_PRIVATE_MEDIA_FETCH Let a media ref be fetched from a private address or over plain http no Off unless set to exactly 1. Media refs are otherwise fetched under NodeTool’s default egress policy — https, to a public host, every redirect hop re-checked. Turn it on for a self-hosted install that serves media off its own LAN. It applies to media-ref fetches only; every other screened surface is unaffected. See URL egress inventory NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION Let the Python bridge connect when NODETOOL_ENV=production no Off unless set to exactly 1. Otherwise a production server refuses to spawn the worker: Python nodes are a local-only feature NODETOOL_PYTHON_EXECUTE_TIMEOUT_MS How long one Python node invocation may run no Default 720000 (12 minutes) NODETOOL_PYTHON_STATUS_TIMEOUT_MS How long a worker status request waits no Default 30000 NODETOOL_PYTHON_DOWNLOAD_IDLE_TIMEOUT_MS Silence from a worker-side model download before it is abandoned no Default 300000 (5 minutes). Idle time, not total — a slow download that keeps reporting progress is not cut off NODETOOL_WORKER_NAMESPACES Narrow which Python node namespaces the worker loads no Passed through unchanged as --namespaces <value>. Unset, the flag is not passed and the worker loads everything installed. See Python Nodes NODETOOL_VALIDATE_OUTBOUND_WS Schema-check every server→client WebSocket frame before sending no 1/true on, 0/false off. Unset, on under NODE_ENV=test/Vitest and off elsewhere. See Protocol validation NODETOOL_VALIDATE_BRIDGE_FRAMES Schema-check every frame arriving from the Python worker no Same values and default as NODETOOL_VALIDATE_OUTBOUND_WS. A failing frame is rejected, not dispatched NODETOOL_ENABLE_TEST_TOPUP Allow the credits top-up that mints credits with no payment no 1/true only; off otherwise and the mutation is refused. Development servers only. See Development-only settings NODETOOL_SEED_DEMO_COSTS Seed demo spend for the Costs dashboard at startup no 1 only. Idempotent — a marker row stops it re-running NODETOOL_DEBUG_STAGES Capture a canvas screenshot at every stage of a browser debug run no 1 or true. Set by nodetool debug --stages; off otherwise. Up to 16 intermediate frames land in stages/ under the output directory. See Test harness settings NODETOOL_DEBUG_TIMEOUT Per-run timeout (ms) for the in-page run of the browser debug harness no How nodetool debug --timeout reaches the browser surface. Read as a positive integer; anything else is ignored and the harness’s 5-minute floor applies NODETOOL_E2E_EXAMPLES_DIR Examples directory the e2e test server serves at /api/examples no A path that does not exist is ignored; the server then tries packages/base-nodes/nodetool/examples/nodetool-base and examples/workflows under the repo root NODETOOL_FAKE_PROVIDERS Run the user-journey backend hermetically — every LLM provider and every external/media node is a deterministic fake no 1 only. npm run test:journeys sets it itself; the screenshot and visual suites leave it off. Structural and pure-compute nodes still run for real. See Hermetic providers NODETOOL_FAKE_DEBUG Log each node’s REAL/FAKE resolution in hermetic mode no 1 only, written to stderr. Use it when a node you expected to be faked ran for real NODETOOL_ENABLE_FAKE_PROVIDER Register the fake provider id as a builtin, so a workflow can select it no 1 only, and ignored when NODETOOL_ENV=production. Separate from NODETOOL_FAKE_PROVIDERS, which fakes the providers that are already registered NODETOOL_PACK_SEARCH_PATHS Extra node_modules directories to load node packs from no Comma-, semicolon-, or PATH-separator-delimited (: is not a separator on Windows, so drive letters survive). Paths that do not exist are dropped. Searched before the walk up from the working directory. See Node Packs NODETOOL_OPTIONAL_NODE_MODULES A single extra node_modules directory for pack loading no The one-path form of NODETOOL_PACK_SEARCH_PATHS; both are read, and the desktop app uses this to point the loader at its bundled install root NODETOOL_PACKS_REQUIRE_ALLOWLIST Default allowUnlisted to false without production mode no 1 only. Same trust default NODETOOL_ENV=production gives, without disabling the local-only features production mode turns off. The packaged desktop app sets it — its optional-node directory holds user-installed code, but it needs the Python bridge, file browser, and the rest of the local surface. An explicit allowUnlisted in packs.json still wins NODETOOL_PACKS_ALLOWLIST Node packs trusted to load, as a comma-separated list of pack names no Whitespace around each name is trimmed and empty entries dropped. It is the environment layer of the same allowlist packs.json holds under allow, and it wins over the file — so it is the ephemeral override (NODETOOL_PACKS_ALLOWLIST=nodetool-base,my-pack nodetool serve), never a persisted setting: a partial trust update written back to disk deliberately ignores it. Unset, the file’s allow applies, then an empty list. It only decides which listed packs are trusted; whether unlisted packs load at all is NODETOOL_PACKS_REQUIRE_ALLOWLIST and NODETOOL_ENV. See Node Packs NODETOOL_PACKS_CONFIG Path to the node-pack trust file no Default ~/.config/nodetool/packs.json on every platform (the loader anchors it at the OS home directory, not at NODETOOL_CACHE_DIR). The file is JSON with four optional keys — allow (array of pack names), allowUnlisted (boolean), enabledBuiltins and disabledBuiltins (arrays) — and a file that is missing or unparseable reads as no configuration rather than an error. Point it elsewhere to run a server against a trust set that is not the calling user’s NODETOOL_CHAT_DETACH_GRACE_MS How long a running chat turn survives with no client attached no Default 0 (disabled). Positive values add a disconnect timeout. Agent execution budgets still apply. See Chat turn replay NODETOOL_CHAT_REPLAY_RETENTION_MS How long a finished turn is kept for a late reconnect no Default 300000 (5 minutes) NODETOOL_CHAT_REPLAY_BUFFER_EVENTS Frames buffered per turn for replay no Default 2000. A client whose last_seq predates the buffer is told the replay is incomplete and refetches thread history over REST NODETOOL_JOB_DETACH_GRACE_MS How long a running workflow job survives with no client attached no Default 600000 (10 minutes), then the run is cancelled so an abandoned client cannot leave a workflow spending forever. See Job run replay NODETOOL_JOB_REPLAY_RETENTION_MS How long a finished run is kept for a late reconnect no Default 300000 (5 minutes) NODETOOL_JOB_REPLAY_BUFFER_EVENTS Frames buffered per run for replay no Default 2000. Beyond the buffer, reconnect_job falls back to the persisted jobs row — the run’s status without its events LOG_LEVEL / NODETOOL_LOG_LEVEL Logging level no Defaults to info (NODETOOL_LOG_LEVEL takes precedence) NODETOOL_LOG_FILE File logs are appended to instead of stderr no Opened once, in append mode; a path that cannot be opened falls back to stderr silently, so a bad value loses the logs rather than the process. It exists because a TUI owns the terminal: nodetool-chat sets it to ~/.nodetool/chat.log (and the level to error) when you have not, so Ink’s rendering is not interleaved with log lines. Set it yourself to keep a server’s logs off stderr, or to send a chat session’s logs somewhere else NODETOOL_TRACE_FILE Append every workflow/agent/LLM span as JSONL to this path no One JSON span per line, in the same TraceRecord shape the stdout sink emits, so a downstream analyzer ingests either. Parent directories are created. Equivalent to the --trace-file flag in CLI › Global Options, which wins when both are set. It is a sink, not a switch: with no sink configured at all, telemetry is skipped and no SDK is loaded NODETOOL_TRACE_STDOUT Write the same spans to stdout no pretty for the reader-friendly tree, json for JSONL; 1 and true both mean pretty, and any other value turns it off. --trace-stdout sets it per run and --no-trace-stdout overrides it. Distinct from OTEL_TRACES_EXPORTER=console, which is the OTel SDK’s own coarser exporter OTEL_EXPORTER_OTLP_ENDPOINT OTLP backend the workflow/agent/LLM spans are exported to no A base URL — /v1/traces is appended, so Jaeger or a local collector is http://localhost:4318. Sends OTLP/HTTP JSON with Content-Type: application/json. The spans carry gen_ai.usage.input_tokens, output_tokens, and cost_usd, so a backend can chart spend per model. The per-run file and stdout sinks are the --trace-file / --trace-stdout flags in CLI › Global Options OTEL_EXPORTER_OTLP_TRACES_ENDPOINT Full OTLP traces URL no Used as given, with no /v1/traces appended. Wins over OTEL_EXPORTER_OTLP_ENDPOINT when both are set TRACELOOP_API_KEY Export the same spans to Traceloop cloud yes Sent as Authorization: Bearer <key> to https://api.traceloop.com/v1/traces. OTEL_EXPORTER_OTLP_ENDPOINT wins over the Traceloop URL when both are set, and the key is still sent as the header OTEL_SERVICE_NAME service.name tag on every exported span no Default nodetool. Set it when several NodeTool installs report to one backend OTEL_TRACES_EXPORTER Print spans through the OTel SDK’s console exporter no console exactly; any other value is ignored. Independent of the sinks above, and coarser than NODETOOL_TRACE_STDOUT=pretty, which is NodeTool’s own reader-friendly stdout sink TRACELOOP_DISABLE_BATCH Flush each span as it ends instead of batching no true exactly. Applies to the OTLP exporter only. Spans are otherwise batched, so a short-lived CLI run can exit before they are sent — set it when a local run shows nothing in the backend SECRETS_MASTER_KEY Master key for secret encryption yes See Secret Storage and Master Key AWS_SECRETS_MASTER_KEY_NAME ⚠️ Removed — setting it makes secret resolution throw at startup no AWS Secrets Manager sourcing of the master key is gone. The variable is still checked so an upgrade fails loudly instead of falling through to a freshly generated key, which would orphan every secret encrypted under the old one. Read the key value out of AWS Secrets Manager, set it as SECRETS_MASTER_KEY, and unset this RUNPOD_API_KEY RunPod deployments yes Used by CLI and providers NODETOOL_CONTAINER_RUNTIME Container runtime the self-hosted deploy tooling drives no docker or podman; any other value is ignored. Unset, a local target probes for docker then podman and falls back to docker, and a remote target resolves the same way in the shell it runs. Set NODETOOL_CONTAINER_RUNTIME=podman on a host with both installed. See Deployment NODETOOL_WORKER_URL WebSocket URL of a running Python worker to attach to no Selects the Python bridge transport. Set to a ws:// or wss:// address (e.g. ws://127.0.0.1:7777) and workflows with Python nodes attach to that worker, which is the only transport allowed in production. Unset or blank, NodeTool spawns python -m nodetool.worker locally, and that path is gated to non-production. nodetool worker create --attach prints the export line for you. See Worker deployment NODETOOL_WORKER_TOKEN Worker bearer token for admin endpoints yes Rotate regularly. Also the shared secret the WebSocket bridge sends as Authorization: Bearer <token> on connect and every reconnect; unset or empty, no header is sent. Read only when NODETOOL_WORKER_URL selects the remote transport. Print one with nodetool worker token <instance-id> NODETOOL_ADMIN_TOKEN Admin bearer token the nodetool deploy user and database subcommands send to a remote deployment yes Equivalent to their --token flag, which wins when both are set. Without either, an interactive shell prompts for it and a non-interactive one exits 1. See Deployment NODETOOL_TIMELINE_RENDERER Skip headless Chromium when RenderTimeline renders a timeline no Set to node (case-insensitive) to render with the Node compositor, which decodes and encodes through ffmpeg and needs a WebGPU adapter. Any other value, or unset, tries Chromium first. A host with no Chromium or no adapter falls back on its own, to the Node compositor and then to an ffmpeg rough cut. metadata.render_mode on the output reports which path ran: browser, composited or rough_cut. The packages/video-nodes test config sets it to node NODETOOL_TSC_VERSION Select the compiler used by the shared TypeScript launcher no 7 by default for the native TypeScript 7 CLI. Set to 6 for the TypeScript 6 JavaScript compiler when using a rollback or comparing compiler diagnostics. The value must be 6 or 7; invalid values fail before compilation. Compiler-API consumers continue to import the pinned TypeScript 6 compatibility package regardless of this setting NODETOOL_TSC_HEAP_MB V8 heap (MiB) given to the TypeScript 6 JavaScript compiler child process no Build-time only, not read by the server. Default 8192, passed as --max-old-space-size only when NODETOOL_TSC_VERSION=6. Digits only — any other value falls back to the default. Ignored entirely when NODE_OPTIONS already carries a --max-old-space-size. The TypeScript 7 native compiler does not receive this Node heap option NODETOOL_BUNDLE_PLATFORM / NODETOOL_BUNDLE_ARCH Target the backend-bundle staging and verification scripts prune prebuilt binaries to no Build-time only, not read by the server. Default to process.platform / process.arch; set both only when cross-building. See Backend bundle targeting NODETOOL_FETCH_ALL_NODE_RUNTIMES Download the Node binaries for every packaging target, not just the host’s no Build-time only, not read by the server. 1 exactly — any other value is ignored. Turns electron/scripts/fetch-node-runtime.mjs from the host target into all four (darwin-arm64, darwin-x64, win32-x64, linux-x64). A target named on the command line wins over both. The fetch is idempotent, so a cached binary is skipped NODETOOL_PROBE_OPENAI_MODEL / NODETOOL_PROBE_GEMINI_MODEL Model the live provider contract probe sends no Read only by npm run probe:providers. Default gpt-5.4-mini and gemini-3-flash. The per-provider budget is charged at the manifest’s declared estimate, not what the override actually costs. See Provider contract probes NODETOOL_TEST_CHROME Chrome binary the packages/browser integration suite drives no Test-only. Must be Chromium or Chrome for Testing — branded Google Chrome ignores --load-extension. CHROME_PATH is the fallback; a path that does not exist is ignored, and the harness then looks under <repo>/chrome. See Test harness settings Use nodetool settings show to view resolved values and verify the merge order. Related Documentation Storage Guide – how asset storage backends are selected. Deployment Guide – passing environment variables in deployment.yaml. CLI Reference – settings-related commands.--- # NodeTool Cookbook Source: https://docs.nodetool.ai/cookbook.html Each recipe starts from the thing you want at the end, such as a launch film, a set of ads, or a playable game. It names where to start, what the agent does, and which editor you finish in. Most recipes start the same way. You write a brief in chat or pick a guided flow, the agent builds the project, and you review it in an editor. You can change any part by hand or send the agent a note about one shot. When the same job comes back every week, the last section turns it into a workflow that runs without you. How a recipe runs Start. Open New project and type the brief, or pick a guided flow. In chat, type / to pick the skill a recipe names. The agent then follows that skill’s steps for the request. Attach what is real. Type @ to attach a product photo, a logo, or an entity such as a recurring character. Generations use its reference image, so the cast and product stay the same across shots. Watch and approve. The plan, each tool call, and each render appear as they run. The thread’s permission mode decides which actions ask first. Plan mode proposes the steps without running them. Finish in the editor. The result is a storyboard, timeline, sketch, 3D model, game, or app that you can open and edit, not a single exported file. Recipes run on your own provider keys, local models, or hosted credits. Storyboards render cheap stills before they spend on video, and NodeTool records what each provider call cost. See Costs & Credits. Video recipes You want Start with You finish in A launch film from a product page /launch-commercial and the page URL Storyboard, then timeline A polished product film or brand spot /product-commercial, or the Storyboard flow Storyboard, then timeline A phone-shot creator ad or testimonial /ugc-video Timeline A 15-second sound-off ad from real screenshots /motion-ad Timeline A narrated explainer Script flow, or /script-video Script, then timeline A short film, scene, or trailer Storyboard flow, or /short-film Storyboard, then timeline Your version of an ad you saw /video-clone with the reference video Storyboard, then timeline A campaign: one cast, stills, several cuts /launch-kit Entities, storyboards, timelines A fix to a shot that is already cut /nodetool-video-post Timeline Launch film from a product page Paste the product URL after /launch-commercial. The agent reads the page, writes a beat sheet, boards the shots with the product as a reference, renders stills and clips, adds voice and music, and assembles the cut. The project reports what it cost. Open the storyboard to swap a shot before the clips render, or the timeline to trim the finished cut. Short film or trailer Pick the Storyboard flow and describe the story in one sentence. You review the screenplay, assign entities to the shots that need a fixed character or location, choose the look, and render stills. Approve the framing you want, then generate clips for those shots alone. AI Video Production walks through each step, and the Creative Agent page covers redoing a single shot by note. Narrated explainer Pick the Script flow when the words drive the timing. Write or import the script, assign a voice per speaker, and generate the takes. Create a linked storyboard from the script for the visuals. When you edit a line later, its take is marked stale so you know which lines to voice again. See Script: write and cast the words first. Sound-off social ad /motion-ad builds a 4:5 or 9:16 motion-graphics ad from the product’s real screenshots and photos, and checks each claim against the page it links to. The ad library shows the formats with a beat sheet, an asset list, and a sample cut for each. For a beat-by-beat example, read a 15-second vertical product ad. Fix one shot in a finished cut Open the timeline and ask the agent for the fix, or type /nodetool-video-post. It covers cutouts, tracking, outpainting, upscaling, generated sound, voice replacement, and lip sync. Each repair lands on the timeline as an inactive candidate next to the original clip, and you choose whether to use it. The rest of the cut stays as it was. AI Timeline Editing lists the edits the agent can make on a timeline. Image, 3D, and game recipes You want Start with You finish in One image with variations to choose from Image flow Sketch editor A composition where you control each layer A sketch, or /nodetool-sketch Sketch editor A recurring character, product, or location Entity flow Entities A set blocked out in 3D as a shot reference Ask the agent, or /nodetool-3d-scene 3D editor A playable 2D or 3D game Game flow, or /native-game Game editor Key art. The Image flow turns an idea into a brief, then a contact sheet of variations. The pick opens as a sketch. Bind a layer to a prompt to regenerate that layer alone, or paint a mask to change one region. A character that stays the same. Make the character once with the Entity flow. After that, @ the entity in chat or in a Prompt node, or assign it to storyboard shots, and generations use its reference image. A game. The Game flow starts a 2D room game or a 3D exploration. Play it in the editor, ask the agent for a new level or new assets, then export a web player. Use /game-direction first when the game plays correctly but feels flat. Tools for a team You want Start with Read A simple screen a coworker can run without the graph /nodetool-app-builder Mini Apps Answers grounded in your own documents /nodetool-rag-indexing Collections An agent that browses, extracts, and fills forms /nodetool-browser-agent Chat & Agents A run on a schedule, a file drop, or a webhook A trigger node in a workflow Triggers A procedure the agent should follow every time Your own skill Skills NodeTool driven by another agent or your code The MCP server or the API MCP Server, API When you repeat the same instructions across chats, save them as a skill. It then appears in the / list with the shipped ones. Facts the agent should keep across chats, such as your brand rules, belong in memory. Repeat it with a workflow An editor is where you make one piece and judge it. A workflow graph is where you make the same piece again for the next brief, SKU, language, or aspect ratio. Ask the agent to build a workflow for the job, type /video-workflow, or use the Workflow flow. Storyboards, scripts, timelines, and sketches are typed values a graph can read and write, so an approved document can feed the graph directly. What you are doing Where it belongs One film, judged shot by shot Storyboard The same film for forty briefs Brief to cut Trimming, mixing, and captioning one cut Timeline Every delivery format of that cut Derivatives Painting a mask, composing a frame Sketch That composition in twelve styles Sketch as control Re-voicing a script after each copy edit Script to voiced cut Thirty on-brand assets with one cast Entities The workflow patterns give each graph with its nodes and the shipped template closest to it. Workflow concepts covers typed edges, documents as values, fan-out, and how to check a graph before it spends. I want to… Pattern Key nodes Turn a brief into a rendered film, unattended 1 · Brief to cut Director, ShotBatch, ShotChain, RenderTimeline Gate each shot on a cheap still first 2 · Shot fan-out ScreenplayShots, TextToImage, ImageToVideo Hold one cast and look across a batch 3 · Entities ApplyEntities, ListGenerator, TextToImage Voice a script and caption the cut 4 · Script to voiced cut WriteScript, VoiceScript, ScriptToTimeline, ScriptToSubtitles Drive generation from a drawn composition 5 · Sketch as control RenderSketch, SketchLayers, ImageToImage Get a gallery of variants from one brief 6 · Variant fan-out Agent, ListGenerator, TextToImage, Collect Ship one cut in every required shape 7 · Derivatives Transcript, RetargetTimeline, RenderTimeline, AddSubtitles Handle naming, packaging, and subtitle math 8 · Code node Code Start from a template Open Examples in the app menu and load one. NodeTool opens a private copy, so the original stays unchanged. The Templates Gallery lists them all. Direct a Short Film: a brief in, a cut film out, with no editor step. Directed Film to Timeline: the same trip with a still for each shot. Concept Art Iteration Board: one brief, a gallery of directions. Localized Explainer: one brief, a voiced and captioned explainer per language. Three Ratios: one approved cut, delivered at 9:16, 1:1, and 16:9. Podcast Repurposing Studio: one recording, a whole content pack. Before an expensive run, nodetool validate checks the graph without spending. After a failed run, nodetool debug returns each node’s output and error. See Debugging. More examples Use CasesFlagship projects end to end: trailers, product videos, ad batches, posters. ExamplesEvery shipped example: apps, recipes, workflows, storyboards, timelines, sketches, 3D models, and games. Shipped SkillsEvery skill the agent can load, grouped by the work it covers. Ad LibraryVertical ad formats with beat sheets and sample cuts.--- # Workflow Concepts Source: https://docs.nodetool.ai/cookbook/core-concepts.html What a workflow is A workflow is a graph. Nodes are operations, edges carry typed values, and a node runs as soon as its inputs are ready. You never order the steps yourself. Brief → Direct → Still → Clip → Cut Four rules follow from that: Edges are typed. An image output does not connect to a text input. The editor refuses the connection instead of failing at run time. Execution follows the data. Independent branches run at the same time; a Collect node is how you wait for a stream to finish. Streaming is normal. ListGenerator emits prompts one at a time, and a downstream image node starts on the first one while the last is still being written. Failures are local. A red node stops its own branch. Other branches finish, and the run then ends as failed with each node’s error. Documents are values The creative surfaces are documents, and a document travels through a graph as one typed value. This is what connects the surface you work in to the graph that repeats the work. Document Into a graph What reads it What writes it Sketch nodetool.constant.Sketch RenderSketch (image + mask), SketchLayers CreateSketch Script nodetool.constant.Script LoadScript, VoiceScript, ScriptToTimeline, ScriptToSubtitles, FillScript WriteScript, FillScript, VoiceScript (saves takes) Timeline nodetool.constant.Timeline Transcript, RenderTimeline, RetargetTimeline, FillTimelineText AddClips, ScriptToTimeline, RetargetTimeline, FillTimelineText Storyboard nodetool.constant.Storyboard LoadStoryboard, StoryboardShots, RenderStills, RenderClips, AssembleTimeline, RecastStoryboard RecastStoryboard, RenderStills, RenderClips nodetool.creative.Director writes the same screenplay shape as a board, so a graph can start from a brief instead of from a board you already approved. RecastStoryboard copies an approved board onto a new cast and keeps every frame whose prompt did not change, so a re-run pays only for what moved. Data types Type Carries Typical nodes str Text StringInput, Prompt, Agent image One image TextToImage, ImageToImage, RenderSketch video One clip TextToVideo, ImageToVideo, RenderTimeline audio One track TextToSpeech, TextToMusic list[T] Many of T ListGenerator, Collect, SketchLayers dict A record Director (screenplay), ScreenplayShots (shot) sketch script timeline storyboard A document the constants above image_model video_model tts_model A model choice ImageModelInput, VideoModelInput Models are values too. Wire a VideoModelInput into several nodes and one selection changes the whole graph. That is how a cheap draft run and an expensive final run stay the same graph. Fan-out and collection Most creative graphs are one of these two shapes. Stream, then gather. A generator emits N items, each one runs the same pipeline, Collect turns the results back into a list. graph LR brief["Brief"] --> gen["ListGenerator"] --> img["TextToImage"] --> collect["Collect"] --> out["Output"] Sequence with carry-over. Each step depends on the last, such as a shot seeded by the previous shot’s final frame. ShotChain does this internally; ForEach plus a Code node does it when the rule is your own. graph LR shots["ShotBatch"] --> chain["ShotChain (frame → next shot)"] --> cut["AddClips"] The difference matters for cost. Fan-out spends N times in parallel and fails independently; a chain spends in order and a failure halfway leaves the tail unrendered. Check before you spend A video run costs real money, so check the graph statically first: npm run dev:nodetool -- validate workflow.json It reports unknown node types, missing required properties, unselected models, models naming a provider or model id that does not exist, dangling or mis-typed edges, and Code bodies that will not run. It does this without executing any node. Against a saved workflow id, it also warns about credentials this install cannot resolve. The workflow editor’s estimate panel prices the graph from per-node unit pricing in the same spirit. When a run does fail, nodetool debug <id> runs it and hands back every message, log, output, and error in one bundle. Details in Workflow Debugging.--- # Creative Workflow Patterns Source: https://docs.nodetool.ai/cookbook/patterns.html Eight patterns for creative work worth running more than once. Each one names the graph, the nodes, and the shipped template closest to it. To build any of them: press Space to add a node, drag an output handle to an input handle, press Ctrl/⌘+Enter to run, and drop Preview nodes wherever you want to see an intermediate value. Pattern 1 · Brief to cut, unattended A line of brief becomes a rendered film: the Director writes the screenplay, ShotChain films the shots in order, and the clips land on a timeline that gets rendered. {% mermaid %} graph LR brief[“StringInput (Brief)”] dir[“Director”] batch[“ShotBatch”] chain[“ShotChain”] cut[“AddClips”] render[“RenderTimeline”] out[“Output (Film)”] brief –> dir –> batch –> chain –> cut –> render –> out {% endmermaid %} ShotChain seeds each shot from the previous clip’s last frame, which is what holds continuity across a cut nobody is watching. Shot 1 has nothing to seed from unless its screenplay shot carries a keyframe, so it runs text-to-video while the rest run image-to-video. On providers that publish those as separate model ids (kie does), fill in Continuation Model too. Leave it empty to reuse Model. Nodes: nodetool.creative.Director, nodetool.creative.ShotBatch, nodetool.creative.ShotChain, nodetool.timeline.AddClips, nodetool.timeline.RenderTimeline Automate it when you have many briefs, or one brief that gets re-cut on a schedule. Use the storyboard surface instead when you want to approve each still before paying for its clip. Template: Direct a Short Film Pattern 2 · Shot fan-out through stills The same trip from brief to cut, but with a keyframe in the middle: ScreenplayShots streams one prompt per shot (shot_prompt), each prompt becomes a still, and each still is animated. {% mermaid %} graph LR brief[“StringInput (Brief)”] dir[“Director”] shots[“ScreenplayShots (streams)”] still[“TextToImage”] clip[“ImageToVideo”] collect[“Collect”] cut[“AddClips”] render[“RenderTimeline”] brief –> dir –> shots –> still –> clip –> collect –> cut –> render {% endmermaid %} A still costs cents and a clip costs dollars, so the keyframe is both a control point and a cheap thing to inspect after an unattended run. Anchoring every keyframe to one style frame keeps thirty shots looking like one film. Generate the style frame first, then run the shot stills through ImageToImage against it. Nodes: nodetool.creative.ScreenplayShots, nodetool.image.TextToImage, nodetool.image.ImageToImage, nodetool.video.ImageToVideo, nodetool.control.Collect Automate it when the shot count is large enough that clicking through the board is the slow part. Templates: Directed Film to Timeline (ends in AddClips and RenderTimeline), Movie Trailer Generator (ends in Concat), Script to Screen (an Agent writes the direction document instead of Director) Pattern 3 · One cast across many assets Entities are named characters, locations, styles, and props whose canonical descriptor is pasted verbatim into every prompt that uses them. That verbatim text is what holds a face or a palette steady across a batch. {% mermaid %} graph LR brief[“StringInput (Brief)”] prompts[“ListGenerator (one prompt per asset)”] cast[“Entity (constant)”] apply[“ApplyEntities”] image[“TextToImage”] collect[“Collect”] out[“Output”] brief –> prompts –> apply cast –>|entities| apply apply –>|prompt| image image –> collect –> out {% endmermaid %} TextToImage, ImageToImage, TextToVideo, and ImageToVideo take an entities property directly, so the graph above also works with the entities set on TextToImage and no ApplyEntities. ImageToImage also appends each entity’s reference images to its input images. Reach for nodetool.creative.ApplyEntities when something else needs the seasoned text. It returns the composed prompt and the reference_images as separate outputs, and an empty text applies every entity. Nodes: nodetool.constant.Entity, nodetool.creative.ApplyEntities, nodetool.generators.ListGenerator, nodetool.image.TextToImage Automate it when a campaign needs the same cast in thirty frames. Do it by hand for a single hero image. The picker in the Prompt node is one @ away. Pattern 4 · Script to voiced cut and captions A script is a document with cast voices attached to its lines. WriteScript writes one from a brief, or a Script constant loads one you already have. VoiceScript synthesizes every line that is draft or stale, using each line’s own voice, and saves the takes back onto the script. {% mermaid %} graph LR script[“Script (constant) or WriteScript”] voice[“VoiceScript”] tl[“ScriptToTimeline”] render[“RenderTimeline”] subs[“ScriptToSubtitles”] out[“Output (Cut)”] srt[“Output (SRT)”] script –> voice –> tl –> render –> out voice –> subs –> srt {% endmermaid %} Lines already up to date are skipped, so a re-run after a copy edit pays for the changed lines only. That is what makes the whole chain worth wiring: the script is the source of truth, and the cut plus the subtitle file are both derived from it. ScriptToTimeline raises an error when no line has a take, so voice first. ScriptToSubtitles writes SRT or WebVTT (format) with one cue per line or per word (granularity), from the takes’ word timings. Nodes: nodetool.constant.Script, nodetool.script.WriteScript, nodetool.script.VoiceScript, nodetool.script.ScriptToTimeline, nodetool.script.ScriptToSubtitles, nodetool.timeline.RenderTimeline Automate it when copy changes often, or when the same script ships in several languages. WriteScript takes a language, so a list of languages in front of it gives one voiced master per language. FillScript fills a template script’s {% raw %}{{key}}{% endraw %} placeholders from a value bag when the copy is fixed and the values change. Template: Localized Explainer implements this chain. Narrate a Script and Localise a Script and Revoice It are string-to-speech chains without a script document. Pattern 5 · Sketch as the control input A sketch document carries layers and a mask. RenderSketch flattens it to an image and returns the mask alongside when the sketch has a visible mask layer. That gives you the composition and the region to change in one node. {% mermaid %} graph LR sketch[“Sketch (constant)”] render[“RenderSketch”] styles[“StringListInput (Styles)”] each[“ForEach”] img[“ImageToImage”] collect[“Collect”] out[“Output (Variants)”] sketch –> render –>|image| img styles –> each –> img –> collect –> out {% endmermaid %} That is the whole-frame version. For a masked edit, send the same node’s mask output alongside its image to openai.image.EditImage, which repaints the white areas and leaves the rest alone — the mask is painted once and every variant reuses it. SketchLayers is the other half. It returns layers and names for each visible layer, so foreground and background can go through different pipelines and be composited back together with Compositor. Nodes: nodetool.constant.Sketch, nodetool.sketch.RenderSketch, nodetool.sketch.SketchLayers, nodetool.image.ImageToImage, openai.image.EditImage, nodetool.image.Compositor Automate it when the composition is settled and you want it in twelve styles or six aspect ratios. Paint in the sketch editor while the composition itself is still the question. Pattern 6 · Variant fan-out from one brief An agent turns a brief into a direction, a list generator writes N distinct prompts against it, and each prompt renders. The fan-out is the point: one input, a gallery out. {% mermaid %} graph LR brief[“StringInput (Brief)”] count[“IntegerInput (Count)”] direction[“Agent (Art Director)”] ask[“Prompt (direction + count)”] prompts[“ListGenerator”] image[“TextToImage”] preview[“Preview”] collect[“Collect”] out[“Output”] brief –> direction –> ask –> prompts –> image –> collect –> out count –> ask prompts –> preview {% endmermaid %} ListGenerator has no count property. State the count in the prompt that feeds it, which is why a Prompt node sits in front. It streams its item output, so the first prompts render while the last are still being written, and a Preview on the prompt stream shows you what is coming before the images arrive. Nodes: nodetool.agents.Agent, nodetool.text.Prompt, nodetool.generators.ListGenerator, nodetool.image.TextToImage, nodetool.control.Collect Automate it when you want variety to pick from — concepts, thumbnails, poster directions, a social kit. Templates: Concept Art Iteration Board, Hook & Thumbnail Factory, Movie Posters, Brand Asset Generator Pattern 7 · Derivatives from a finished cut One master, many deliverables. Transcript reads the timeline’s own transcript lines, so titles, show notes, and social copy come from the cut rather than from a second transcription pass. RetargetTimeline derives a new timeline on another aspect ratio from the approved cut and keeps every trim and placement. {% mermaid %} graph LR tl[“Timeline (constant)”] transcript[“Transcript”] agent[“Agent (titles, notes, posts)”] retarget[“RetargetTimeline (9:16)”] render[“RenderTimeline”] audio[“ExtractAudio”] asr[“Transcribe (segments)”] subs[“AddSubtitles”] copy[“Output (Copy)”] post[“Output (Vertical cut)”] tl –> transcript –> agent –> copy tl –> retarget –> render –> subs –> post render –> audio –> asr –> subs {% endmermaid %} RetargetTimeline takes aspect_ratio (the short edge is 1080 px) and fit. cover fills the frame and crops, and its cropped output lists the clips it cut into. contain letterboxes. The source timeline is never written. nodetool.video.Resize is different: it scales to the exact width and height you give it, so a 16:9 clip resized to 1080 by 1920 is squashed, not cropped. Burned-in captions need timings, not prose: AddSubtitles takes list[audio_chunk], which openai.audio.Transcribe returns as segments or words. Transcript is the text path — it feeds the copy, not the caption burn. Nodes: nodetool.constant.Timeline, nodetool.timeline.Transcript, nodetool.timeline.RetargetTimeline, nodetool.timeline.RenderTimeline, nodetool.video.ExtractAudio, openai.audio.Transcribe, nodetool.video.AddSubtitles, nodetool.agents.Agent Automate it when every cut ships in more than one shape. Re-running the graph after an edit rebuilds every derivative from the same master. Templates: Three Ratios (one cut to 9:16, 1:1, and 16:9), Podcast Repurposing Studio (one recording to titles, notes, a newsletter, posts, and quote cards), Subtitle Text from a Recording. Cut a Landscape Clip for Vertical uses Resize on a clip, so it stretches the frame. Pattern 8 · Code node for the parts a model should not do Naming, ordering, deduping, packaging: deterministic work that a model does expensively and inconsistently. One Code node does it in JavaScript. {% mermaid %} graph LR clip[“VideoInput”] audio[“ExtractAudio”] asr[“AutomaticSpeechRecognition”] code[“Code (slugify)”] out[“Output (Filename)”] clip –> audio –> asr –> code –> out {% endmermaid %} The body runs in the QuickJS sandbox and imports what it declares. Sandbox packs cover the file formats a delivery step usually needs — @nodetool-ai/sandbox-subtitle for SRT and VTT, -csv, -zip, -yaml, -xlsx. Validate a body before running the graph with nodetool validate <file>, or author it against validate_code / run_code / test_code. Nodes: nodetool.code.Code Automate it when the step has one right answer: a slug, a manifest, a per-shot cost table, a subtitle file assembled from timings. Template: Name a File from Its Narration (one Code node truncates the transcript, a second slugifies it) Leave it on the surface A graph earns its place by repeating. These do not repeat, and the surface is faster: Job Where it belongs Judging a film shot by shot, re-rolling the weak ones Storyboard Trimming, mixing, and captioning one cut Timeline editor Painting a mask or composing a frame Sketch editor Rewriting a line and hearing it back Script editor One hero image, one question about a document Chat Each surface is also drivable by the chat agent through its ui_* tools, and from outside over MCP — so “do it on the surface” does not mean “do it by hand”.--- # Costs and credits Source: https://docs.nodetool.ai/costs-and-credits.html NodeTool prices a run before you start it, records what each provider call cost afterward, and lets you cap spend in the places where a run can get expensive. This page covers all three, and the one case where NodeTool itself meters usage: hosted credits. Your own keys or hosted credits There are two ways a generation gets paid for.   Your own provider keys Hosted credits Who bills you The provider (fal, OpenAI, Anthropic, and so on) NodeTool, against a credit balance NodeTool’s role Estimates the cost and records the spend Also enforces the balance before a run starts Where it exists Every install Only NodeTool’s hosted cloud Provider Each provider you configure The nodetool provider With your own keys, NodeTool does not mark up a provider’s price and never charges you. The estimates and the Costs page are for your planning. The provider invoices you. Local models (Ollama, vLLM, LM Studio, llama.cpp) need no key and have no catalog price. See Providers. Hosted credits exist only where the nodetool provider is registered. That provider runs a curated model catalog on NodeTool’s own platform keys, and it is cloud-only. Desktop and self-hosted installs do not register it, so none of the credit features below apply to them. Credits meter only calls through the nodetool provider. Calls on your own keys never reduce a credit balance, even on a hosted server where both kinds coexist. Estimating a run before you start The estimate panel In the workflow editor, the Cost estimate section sits under the Inspector in the right panel. It is collapsed by default. Its header shows the total, for example $0.42 / run, and expanding it opens a table with one row per node that uses an AI model: Column Meaning Node The node’s title. If you renamed the node, your name is used. Hover for the full node type. Provider / model The provider and model the node has selected. Units What one run buys, such as 4 images or 2 × 5 s @ 720p. Cost The estimated cost for that node. The estimate updates as you edit the graph. Nodes that do not use an AI model (plain data and utility nodes) are not listed. A node’s fan-out counts: if it has a num_images, num_outputs, num_samples, or batch_size value above one, the cost is multiplied by it. With none of those set, the estimate assumes one output. Hover a row to see how the figure was reached, such as 5 s × $0.205/s at 720p, plus any warnings. Reading the markers An estimate states how sure it is. You see It means ~$0.40 The price rests on an assumed default, such as a 1 s duration or the base resolution. The real run can land on either side. ≥ $0.40 The figure leaves out a cost the catalog says exists, so the real cost is at least this. A question mark and — NodeTool has no price for this node’s model. Hover the icon for the reason. N nodes without a known price are excluded from the total The total adds up only the priced rows. incomplete in the collapsed header At least one node is unpriced. An unpriced node is listed, never hidden, and counts as zero in the total. Treat a total with unpriced nodes as a floor, not a quote. Local models are a common case, because no catalog prices them. Where prices come from Prices are list prices that ship with the app and are refreshed by a nightly sync. A line at the bottom of the table names the source (genspend.io and provider catalogs) and the date the prices were last updated. For a model on a given provider, NodeTool looks up that provider’s own published price. It uses a published price grid (resolution, duration, audio) when one exists, and otherwise the provider’s single catalog rate, converted to a per-run figure. It never substitutes another provider’s price for the same model, because a reseller’s rate differs from the vendor’s. If the catalog prices the model only on other providers, the row stays unpriced and the tooltip names them. A price quoted in a unit with no fixed value, such as bare “credits” or “units”, is also left unpriced instead of added to the total. Managed nodetool models are priced at the rate of the model behind them. Other places an estimate appears The same price lookup feeds several other surfaces, so a figure shown in one place matches the others: Template cards in the Templates gallery show an estimated cost computed from published node prices. A graph with unpriced nodes shows “at least” or “unknown” with the count of unpriced nodes. New project in the user interface shows an estimate read from what your own past projects of the same kind cost. It appears once you have two finished projects with fully priced spend. The chat composer’s media mode, and the clip inspectors and prompt bar in the timeline, show a single ≈ or ≥ figure for the generation about to run. Hover it for the breakdown. The tooltip notes that the provider bills at its own rates. The server runs the same lookup before a run starts, to apply the limits described below. Tracking what was spent Every provider call writes a record with its cost and token counts. Two views read those records. The Costs page Open Costs from the More tab of the left panel, under the app pages (see Editor Panels). It opens as a page tab. It is not in the logo menu. The page shows: Four summary cards: Total spend (with the change against the prior period of the same length), Node executions (with failed count), Avg / execution, and Top cost driver. A daily spend chart, stacked by provider. A table you can group by Execution, Node type, Workflow, Provider, or Model, with a filter box. Range buttons for 7d, 14d (the default), 30d, and 90d. All providers and All workflows filters. Export CSV, which saves the rows currently in the table as nodetool-costs-<range>.csv. If the data cannot be loaded, the page says so instead of showing zeros. From the terminal nodetool costs reads the same records straight from the local database, with no server running. It has four subcommands: summary, list, by-provider, and by-model, and each takes --json. See nodetool costs for the options. Supervisor spend lands in the same records, tagged supervisor in node_type, so it can be separated from the workflow’s own spend. In traces Runs emit OpenTelemetry spans. Every llm.chat and llm.stream span carries gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.total_tokens, and gen_ai.usage.cost_usd. Use --trace-file or --trace-stdout on CLI runs to see them. See observing agent execution. Limiting spend Limits apply in different places. Each one is described below with where it is set. Agent runs An agent run (a chat turn, its sub-agents, and any agent node it starts) shares one budget: Setting Default What it bounds NODETOOL_AGENT_TURN_COST_CAP_USD 5.00 Provider spend for the whole run, in US dollars. 0 means no cap. NODETOOL_AGENT_TURN_DEADLINE_MS 1800000 (30 minutes) Wall-clock time for the run. NODETOOL_AGENT_MAX_TURNS 200 Model turns in total. This still holds for a local model with no price. NODETOOL_AGENT_UNPRICED_TOKEN_CEILING 400000 Prompt tokens for a turn on a model the catalog does not price. The cost cap works as admission. A turn whose worst case would cross the cap is refused before the call, so the money is not spent first. A model without a catalog price has no worst case, so it is bounded by the token ceiling and counted as unpriced rather than free. On the command line, --cost-cap <usd> overrides the dollar cap for one run, and 0 lifts it, which suits a local-only install. See Run budget and the Agent CLI. Mini apps A published mini app has a Spend budget in its Settings: a period (per day, per month, or lifetime), a maximum spend in USD, and a maximum number of runs. An empty field means no limit. Runs of the released app are checked before they reach a provider, and Settings shows what has been used. A public link requires a budget with at least one finite limit, because visitors run the app on your account. Recent invocations list their cost, marked “est.” until the actual cost is known. Storyboards In the creative agent, spend is gated shot by shot. You approve a still before the clip is generated from it, and the cheap stage (stills) comes before the expensive one (clips). Hosted credits Where the nodetool provider is available, the server checks the credit balance before a run that uses a managed model starts. A run is refused in two cases: It names a managed model the operator did not open for spend. This is refused regardless of balance. The balance is empty, or does not cover the run’s estimate. The message reads like This run needs about N credits but M remain. Spend already admitted to runs still in flight counts against the balance, so several simultaneous runs cannot each pass against the same balance. Estimates are a floor, because an unpriced node estimates as zero. For that reason an empty balance blocks a managed run even when its estimate is zero. If the credit check itself fails, the run is allowed, so a metering fault never stops the runner. How credits work One credit is one US cent (USD 0.01) of provider spend at the delegate’s price. The balance is every grant ever recorded, minus the spend recorded for the nodetool provider, rounded up to whole credits. Spend on your own provider keys is excluded. Grants arrive lazily, the first time the balance is read: A one-time welcome grant when you are first seen. The default is 500 credits. A monthly plan grant, keyed to the calendar month in UTC. The plans are defined in code: Plan Credits per month Displayed price Free 300 $0 Creator 3,000 $12 per month Pro 10,000 $40 per month The price is for display. Payment processing is not implemented, so switching plans takes effect immediately and no card is charged. You see the balance in the Studio header as a credits chip, and on the Plan & credits page at /studio/account, which also lists the plans and switches between them. The page notes that credits meter only NodeTool’s managed models. The same data is available from the credits tRPC router (status, setPlan, and topup). Operator settings A server operator controls hosted credits with these environment variables: Variable Effect NODETOOL_CREDIT_MODELS The managed models credits may be spent on, separated by commas, spaces, or newlines. Unset means the whole catalog. Any other model is refused before a key is used. NODETOOL_SIGNUP_CREDITS The one-time welcome grant. The default is 500. 0, a negative value, or an unparseable value means no welcome grant. NODETOOL_PLATFORM_FAL_KEY The platform-owned fal key that funds managed image and video models. NODETOOL_PLATFORM_ANTHROPIC_KEY The platform-owned Anthropic key that funds managed language models. NODETOOL_ENABLE_TEST_TOPUP Set to 1 or true to allow the test top-up on a development server. A managed model is offered only when its delegate provider has a platform key and the model is on the allowlist. The top-up is for development only. It adds credits with no payment behind it, and the server refuses it unless NODETOOL_ENABLE_TEST_TOPUP is set, because minted credits unlock spend on platform-owned keys. When the variable is set, the account page shows an Add 1000 credits (test) button. Known gaps The estimate cannot price a model the catalogs do not carry for the selected provider. Those rows show as unknown and stay out of the total. Credits have no payment provider behind them yet. Plan prices are displayed only. For provider keys and setup, see Providers.--- # Creative Agent Source: https://docs.nodetool.ai/creative-agent NodeTool turns a one-line brief into a finished film through a pipeline you can inspect and steer at every step: a Director agent writes a typed screenplay, a storyboard renders cheap stills you pick from before video spend, each shot becomes a clip you can revise in place, and one click assembles the cut into the timeline editor for finishing and export. The whole chain is drivable three ways: from the UI, from chat (the agent uses the same ui_* tools), or from an external agent such as Claude Code over MCP (nodetool mcp serve). The workflow, end to end graph LR brief["Brief"] direct["Direct(Director agent)"] stills["Stills(cents)"] pick["Pick a still"] clips["Clips(dollars)"] revise["Revise(video-to-video)"] assemble["Assembletimeline"] nle["Timeline editor(trim, mix, captions)"] export["Export"] brief --> direct --> stills --> pick --> clips --> assemble --> nle --> export clips --> revise --> clips revise -. "round-trips into the cut" .-> nle Cheap stages come first. A still usually costs far less than a clip, so you pick the still you like before paying for the clip. Each render dialog shows an estimate before you confirm. When you select a new clip take for a shot after the cut exists, the linked timeline clips pick up the new asset. Start from a shipped board New → New storyboard… (under Blank documents) opens a list with Blank storyboard and the example boards that ship with the install. The Storyboard entry under Guided flows in the same menu walks you through four steps (Idea, Story, Entities, Look) and ends on the rendered board. An example arrives finished: every shot already carries its action text, its still, and its clip, so the surface shows a directed sequence before you spend anything. The media are constant package assets, so installing one writes a single row and copies no bytes — regenerate any shot and the new render replaces the shipped one on that card alone. Direct: brief in, screenplay out Open a storyboard (New → New storyboard… → Blank storyboard), open Board settings, write a brief and a style, pick an aspect ratio and a shot count (3, 4, 5, 6, 8, 10, or 12), and press Direct. Once the board has shots the button reads Re-direct, and it asks for confirmation because it rewrites the screenplay and replaces every shot. The screenplay is typed: title, logline, style bible, narration, music direction, and one structured shot per card with action, camera (framing, lens, angle, movement, equipment), motion, and duration. The board uses the same Director authoring prompt and screenplay schema as the nodetool.creative.Director node, which you can also place in any workflow graph (shot count 1 to 20, default 5). Two sibling nodes make the screenplay usable inside any workflow graph: nodetool.creative.ScreenplayShots streams each shot with a composed image-generation prompt, and nodetool.creative.ApplyEntities injects entity descriptors for consistency (below). The storyboard: plan, pick a still, then spend The board is a grid of shot cards. A card carries what reads at a glance — the still or the clip, a shot label such as SH 03, a status pill, and the action line. The toolbar has Render stills and Render clips buttons. Each opens a dialog where you pick the model and see an estimated cost, then confirm. Select a card and a footer docks under the grid with Edit, Iterate, Regenerate, and Delete, plus an “Appears in” row of links to the shot’s clip in the cut and its script line. Edit opens the shot’s fields, still and clip takes, and script lines under its card, where you set the current still or take. Iterate re-renders the current clip with a text note (“make it darker, add rain”) and is disabled until the shot has a clip. The status pill reads planned, still, still · clip queued, rendering still, rendering clip, failed, or covered, with · stale added when the style or models changed after the render. A shot with a clip shows no pill. Fixing shot 3 never means re-rendering shots 1 to 5. Agents drive the same surface through the ui_storyboard_* tools, among them: get_state, direct, set_screenplay, set_entities, add_shot, update_shot, generate_keyframe, generate_clip, revise_shot, assemble_timeline, select_shot, extract_script, relink_script, reproject_shots, and set_duration_source. The full set is registered in web/src/lib/tools/builtin/storyboard.ts. Assemble: from storyboard to timeline Create timeline (below the shot grid, enabled once a shot has a still or clip) turns the shots into a persisted timeline sequence and opens the editor. Shots lay end to end on a Shots video track at the length of the footage that came back, and a shot with only a still becomes an image clip. Each clip’s own sound goes on a linked Shot Audio track, and the screenplay’s narration and music become draft text-to-audio clips on Narration and Music tracks. A board linked to a script is cut jointly, so each voiced line gets its own voiceover clip and the editor shows a Script tab. Shots without an accepted clip are skipped. Timeline playback and export mix audio clips, never a video element’s track, so the shot-audio clips are how a video model’s dialogue and room tone reach the cut — mute one when a shot should play silent under narration. From here it’s a normal edit — trim, transitions, audio mix, captions — and export. Every assembled clip stays linked to its shot. Select a new clip take for a shot afterward and the new asset replaces the one in the persisted cut. Once a timeline is linked, the button reads Rebuild linked timeline… and asks for confirmation, because rebuilding replaces every clip the storyboard owns, including trims made to those clips. Tracks and clips you added yourself are kept. The storyboard plans, the timeline finishes, and revisions flow forward. Entities: reusable ingredients Characters, locations, styles, and props are named, reusable objects. The Entities page covers creating and editing them. Tag any image asset with a kind, a name, and a canonical descriptor (voice id and tags are optional) — the exact sentence pasted into every prompt that names the entity. That verbatim descriptor is what holds a character’s look steady across shots. Entities live in the asset library (a metadata marker, no migration), appear in the Entities panel in the left sidebar and the full-page library (Add entity), and reach prompts through the ApplyEntities node or the ui_entity_apply tool. On a storyboard, the Entities field — under Board settings — pins a cast to the board: styles and locations season every shot’s still and clip prompt, while characters and props activate on the shots that mention them by name. Open a shot with Edit and the board’s entities show as chips. Click one to include or exclude it for that shot. The Direct run also hands the cast to the screenplay model so shots reference entities by their exact names. Entities are also one @ away wherever prompts are written: the mention picker in the chat composer and the workflow editor’s Prompt node lists them first, and a picked entity carries its descriptor and reference image into the generation. Cost governance Spend is shown before you commit it. The storyboard’s Render stills and Render clips dialogs, and the per-shot cost line, price the request before you confirm and say how many requests could not be priced. In the workflow editor, the Cost estimate section under the Inspector prices the graph before it runs. It uses per-node fal and kie unit pricing and model price lookups, counts fan-out properties such as num_images, num_outputs, num_samples, and batch_size, and lists nodes with no known price as unknown instead of hiding them. Caps are per agent run (a USD cap on a chat turn and its sub-agents) and per published mini app (a spend budget). How NodeTool estimates, records, and limits spend is described in Costs and credits. Graph templates The same pipeline ships as workflow templates for batch runs with no surface interaction: Script to Screen — brief → direction document → style-frame-anchored keyframes → per-shot animation → cut with voiceover and score (Concat assembly). Directed Film to Timeline — the same direction assembled onto the timeline with nodetool.timeline.AddClips → RenderTimeline. For continuity across cuts, nodetool.creative.ShotChain generates clips sequentially, extracting each clip’s last frame to seed the next shot. The approved document as a graph value Those templates direct from a brief. The nodes below start from a document a person already approved and re-run it — a board per SKU, a script per language, a cut per aspect ratio — without re-directing anything. A ref is read-only unless the run derived it, so a batch cannot draw over the template. Node Takes Gives nodetool.constant.Storyboard / .Entity a picked board or entity storyboard / entity nodetool.entity.LoadEntity entity, or a name and kind entity, descriptor, name, kind, reference_image, voice_id nodetool.entity.ListEntities kind, tags, name_contains, project entities; streams one entity per row nodetool.entity.CreateEntity image, kind, name, descriptor, key entity, created — upserts on key, so a re-run reuses the row nodetool.storyboard.LoadStoryboard storyboard shots, entities, style, aspect_ratio, name, both models, shot_count nodetool.storyboard.StoryboardShots storyboard streams shot, index, slug, keyframe, clip nodetool.storyboard.CreateStoryboard screenplay (from Director), name, cast, both models, reuse_existing a new writable storyboard, shots, created — a re-run returns the board it made nodetool.storyboard.RecastStoryboard storyboard, cast, replaces, reuse_existing the copy, invalidated, kept — keeps every frame whose prompt did not move nodetool.storyboard.RenderStills the derived storyboard, targets, max_shots, only_stale keyframes, rendered, skipped, failed nodetool.storyboard.RenderClips the same, plus require_keyframe clips, rendered, skipped, failed nodetool.storyboard.AssembleTimeline the derived storyboard timeline, skipped_shots, retimed — a copy inherits the template’s cut, titles and music nodetool.script.WriteScript model, brief, format, cast, language a new script, line_count nodetool.script.FillScript script, values a new script, filled, unresolved nodetool.timeline.FillTimelineText timeline, values a new timeline, filled, unresolved nodetool.timeline.RetargetTimeline timeline, aspect_ratio, fit a new timeline, cropped nodetool.game.LoadGameTemplate template manifest, slots; streams one slot nodetool.game.SlotPrompt slot, style, cast prompt, width, height, kind, checker, seconds nodetool.game.StageGameAssets template, game_id, fills bindings, paths — candidates for a native game revision RenderStills, RenderClips and AssembleTimeline refuse a board this run did not derive; allow_writes is the per-node override for a graph whose whole purpose is to render the board a person picked. Four shipped examples wire them: Per-SKU Ad Factory, Localized Explainer, Top-down Native Asset Pack, Three Ratios. Driving it from outside nodetool mcp serve exposes NodeTool over MCP (stdio) as an execute_code action tool plus a small set of direct tools. Inside an action, the same capabilities the chat agent has are available, including the storyboard, script, entity, and timeline tools, so an external agent can search nodes, build and run workflows, and generate media. nodetool mcp install writes the config for Claude Code, Codex, or OpenCode. See NodeTool as an MCP Server. Inside NodeTool, the chat agent reaches every surface through the same frontend tools the buttons use.--- # Custom Nodes: user-saved node types in the library Source: https://docs.nodetool.ai/custom-node-library-design.html Custom Nodes: user-saved node types in the library Status: design. Owner: web / code-nodes / agents. 2026-08-18. Problem A user who has shaped a Code node into something useful — a scraper for one API’s response shape, a formatter for their invoice numbers, a validator for their CSV layout — has no way to keep it as a node. The shipped snippets (web/src/config/codeSnippets.ts) show the right interaction: open the node menu, find “Extract URLs”, drop it, and a pre-filled Code node appears. But snippets are hardcoded into the build. A user’s own work has exactly one durable home today, the JS script document (js-script-document-design.md) — and a script is reachable only through the Code node’s Link script picker, three clicks deep inside a node that already exists. Nothing a user saves ever shows up where they look for nodes: the node menu. This design closes that gap. A custom node is a JS script document the user has chosen to expose in the node menu. It appears under a My Nodes section of the palette with its declared inputs and outputs, drops onto the canvas like any node, and the graph it lands in stays portable: the placed node is a plain nodetool.code.Code node with the script materialized onto it and a provenance link back to the library. No new document type, no new storage, no new execution path. The feature is a palette flag on the script document plus the wiring that makes flagged scripts show up and drop correctly. What already exists The three mechanisms this design composes, none of them modified in kind: JS script documents are the library. js_scripts rows own a versioned document with code, declared inputs/outputs ports, secrets, timeoutSeconds, and saved tests (packages/models/src/js-script.ts, tRPC router packages/websocket/src/trpc/routers/js-scripts.ts). The snippet palette path already puts virtual node types in the menu. generateSnippetMetadata() (web/src/config/snippetMetadata.ts) fabricates NodeMetadata records that useMetadata.ts merges into the store, and instantiatePaletteNode (web/src/utils/instantiatePaletteNode.ts) expands a virtual type into a real nodetool.code.Code node on drop. The registry never hears about these types, and the saved graph never contains them. The Code node script link already defines what a placed custom node is. useCodeNodeScriptLink.materialize() (web/src/hooks/nodes/useCodeNodeScriptLink.ts) copies a script’s code, packages, secrets and timeout onto the node, derives dynamic_inputs / dynamic_outputs from the declared ports, and records properties.script = { id, version } as provenance. Graph validation checks link freshness through jsScriptPortMismatches and collectJsScriptLinks (packages/node-sdk/src/js-script-link.ts, graph-validation.ts), and the editor already offers update to latest and detach. A custom node is these three composed: a script (1) surfaced like a snippet (2) that drops as a linked Code node (3). Document shape One optional field on JsScriptDocument (packages/protocol/src/api-schemas/js-scripts.ts): interface JsScriptDocument { // ...existing fields unchanged... palette?: { category: string; // menu grouping, e.g. "Text", "My API" — free text }; } Absence means not exposed. Every existing script stays a plain library script until its owner opts it in; schemaVersion stays 1 and old saves parse unchanged. category is the only knob. The node’s title is the script’s name, its description is the document’s description, its ports are the declared inputs/outputs — all of which the script already maintains for its other consumers. Duplicating any of them into palette would create a second copy to drift. Validation (validate_js_script and the document-level rules in packages/node-sdk): a palette on a script with zero declared outputs is a warning (the node would have no handles to connect), and an empty or whitespace category is an error. Palette surfacing Client-side synthetic metadata, the snippet path — not registry registration. The registry (NodeRegistry.global, packages/node-sdk/src/registry.ts) is process-wide with no per-user scoping, and it does not need any: the virtual node type exists only between the menu and the drop, and the saved graph never references it. Registering per-user types server-side is deferred (see Out of scope) until something server-side actually consumes them. New module web/src/config/customNodeMetadata.ts: export const CUSTOM_NODE_PREFIX = "user."; export function customNodeType(scriptId: string): string { return `${CUSTOM_NODE_PREFIX}${categorySlug}.${scriptId}`; } export function generateCustomNodeMetadata( scripts: JsScriptSummary[] ): Record<string, NodeMetadata> { /* ... */ } Node type is keyed on the script id, not the name: ids are stable and unique, names collide and get renamed. The type string is transient (menu → drop), so its looks cost nothing. Title and search text come from name and description, which is what the menu’s search index actually ranks (NodeSearchIndex weights title 6, description 1). Namespace is user.<categorySlug>, slugified the way snippetMetadata.ts slugifies snippet categories. useNamespaceTree builds the tree from namespace strings alone, so My Nodes appears as a user top-level group with the user’s categories under it, no tree changes needed. NamespaceIcon.tsx gets one entry for the user root. Ports map directly. A declared input becomes a Property with its TypeMetadata and the type’s default value; a declared output becomes an output entry. No AST inference — scripts declare their ports, which is the precision the snippet generator never had. supports_dynamic_inputs / supports_dynamic_outputs stay true, since the placed node is a Code node. A script whose body streams its inputs (usesStreamInputContract) gets is_streaming_input: true, so the menu and the drop both say what the node will do. Loading and refresh Custom-node metadata is user data, so it cannot ride the unauthenticated GET /api/nodes/metadata fetch. A hook beside useMetadata.ts — useCustomNodeMetadata — queries jsScripts.list over tRPC (TanStack Query, key ['jsScripts', 'palette']), generates the records, and merges them into MetadataStore after the base metadata load. Refresh rides the existing cross-client sync: js_scripts changes already broadcast resource_change (registerDocumentSync), which invalidates the query; saving a script, toggling its palette flag, or deleting it updates the menu without a reload. Signed-out and error states merge nothing — the menu degrades to the shipped catalog. Drop semantics: materialize, with provenance instantiatePaletteNode gains a third branch, before the snippet branch: const script = findCustomNodeScript(metadata.node_type); // from the query cache if (script) { /* materialize */ } The branch does what useCodeNodeScriptLink.materialize() does today — the port→slot mapping (portsToSlots / portsToOutputs) and property copying move into a shared helper both call sites use, so drop and link cannot drift: properties: code, secrets, timeout from the document, and script: { id, version } pinning the version that was dropped. data.title: the script’s name. data.codeNodeMode = "custom" — a third mode beside "snippet", so codeNodeUi.ts renders the custom title and keeps it editable, and the property panel can show the link affordances (update to latest, detach, open script) without the user hunting for them. afterAdd: dynamic_inputs / dynamic_properties / dynamic_outputs from the declared ports, merged once the node exists in the store — the snippet pattern. Everything after the drop is the already-shipped link behavior: The graph is self-contained. Execution reads only the node’s own properties (CodeNode.envelope() never resolves the link), so the workflow runs on the server, in the packaged app, and after export exactly as a hand-written Code node does. A deleted or moved script costs freshness warnings, never the run. Staleness is visible, not silent. validateGraph with a jsScriptLookup reports js_script_missing and port mismatches through jsScriptPortMismatches; the editor’s update to latest re-materializes and re-pins. Editing the library script does not mutate placed nodes — the same pin-and-update contract the link feature already made, now stated as the custom-node contract too: your graphs keep the version you dropped until you update them. Detach turns the node back into an anonymous Code node; the code is already inline. Save path: where “save my customized stuff” happens Two entry points, both writing through the existing script machinery: From a Code node — “Save to My Nodes”. The Code node’s context menu and property panel gain the action next to the existing Extract to script; it is extractToScript (which already lifts body, ports, packages, secrets into a new script and links the node) plus a small dialog for name and category that sets palette on the created document. One action, and the node the user customized is now in the menu. A node already linked to a script offers Add to My Nodes instead, which just sets palette on the linked script. From the script editor. The jsscript surface’s header gains a Show in node menu toggle plus the category field — the same palette write over the existing autosave/CAS sync. ui_jsscript_set_meta gains the field, so the script assistant can do it too. Snippets stay as they are: a shipped snippet the user modifies is an anonymous Code node until they save it, at which point path 1 makes it theirs. No migration of codeNodeMode: "snippet" nodes. Agents and headless surfaces Nothing headless ever sees a user.* node type — graphs carry nodetool.code.Code — so validate, debug, the kernel, mini apps, and mobile need no changes. What agents need is authoring parity with the user: Discovery is list_js_scripts, which already returns id, name, description, and ports. Its result gains the palette field so an agent can distinguish “the user’s nodes” from utility scripts, and the graph tools’ system prompt tells the agent to check it before writing a Code node body from scratch. Placement: the ui_add_node path in the web graph tools accepts the virtual user.* type like any palette type (it goes through instantiatePaletteNode already). Headless graph authoring (create_workflow, the DSL) materializes the same way a drop does; a shared helper in @nodetool-ai/node-sdk (materializeJsScriptNode(document, link) → properties + dynamic slots) keeps the web drop, the link hook, and headless authoring on one code path. Validation and safety The materialized body is validated exactly like any Code node body — the link adds freshness checks, it never relaxes anything. Secrets: the placed node carries the script’s declared secrets list, and the Code node’s existing secret gating applies. Dropping a custom node grants nothing the same node hand-written would not have. Scripts are per-user (loadOwned in the tRPC router); the palette is too. Sharing a custom node with another user is sharing the workflow (the code travels materialized) or exporting the script — a marketplace is out of scope. Phases Schema + save path. palette on the document (Zod, validation rules), ui_jsscript_set_meta support, the script-editor toggle, and the Code node’s Save to My Nodes action on top of extractToScript. Palette. customNodeMetadata.ts, useCustomNodeMetadata, the instantiatePaletteNode branch, codeNodeMode: "custom", the shared materialize helper refactor of useCodeNodeScriptLink, NamespaceIcon entry. A user can save a node and drop it from the menu. Agent parity. palette in list_js_scripts, headless materialization through the shared helper, prompt guidance, and an eval case in jsscript-tools covering “save this Code node as a custom node, then use it in a second workflow”. Each phase lands green alone; phase 2 reads what phase 1 wrote and nothing else. Testing Schema (packages/protocol / packages/node-sdk): palette parses and round-trips; the no-outputs warning and empty-category error each go red on a fixture built to violate them. Web (Jest/RTL): generateCustomNodeMetadata maps ports to properties/outputs (typed, defaults, streaming flag); the instantiatePaletteNode branch produces a node whose properties and dynamic slots equal what materialize() produces for the same script — asserted by comparing the two outputs, so the shared helper is proven shared; menu refresh on resource_change. Link freshness: dropping version N, bumping the script to N+1 with a changed port, and validating reports the mismatch — the existing jsScriptPortMismatches suite gains the drop-created fixture. Harness: nodetool jsscript validate on a fixture with palette passes; the eval case in phase 3 gates on the agent using the saved script rather than re-writing the body. Out of scope Server-side registry entries for custom nodes. Metadata-only registration exists (loadMetadata), but the global registry has no per-user scoping and nothing server-side consumes the virtual types today. If search_nodes should someday return them, that needs a per-request registry view — its own design. Sharing and marketplace. Custom nodes travel inside workflows (materialized) and as exported scripts; a catalog of other users’ nodes is a distribution problem this design does not touch. Custom icons and colors. Metadata has no icon field; the user namespace gets one icon. Per-node art waits until metadata grows it for shipped nodes too. Non-Code custom nodes (saving a configured LLM node, a subgraph). A subgraph-as-node is a different feature with its own execution semantics; this design deliberately stops at what the Code node can express.--- # Custom Providers Source: https://docs.nodetool.ai/custom-providers A custom provider is an endpoint that speaks the OpenAI API, added by URL instead of by code. Use it for a proxy, a gateway, a self-hosted router, or any vendor that NodeTool has no built-in provider for. Its chat, image and video models then appear in the matching model pickers. For the built-in providers see Providers. Add an endpoint Open Settings and go to Models & providers. Scroll to OpenAI-compatible endpoints and click Add endpoint. Fill in the dialog and click Add. Field Required What it does Name No Display name shown in the model menu and settings. Defaults to the slug Slug Yes Unique id for this endpoint. Starts with a lowercase letter, then lowercase letters, digits or underscores, 1 to 32 characters. It follows the name as you type, and you can edit it before saving Base URL Yes The endpoint address, http or https, including the version path it serves, usually /v1. Trailing slashes are removed API key No Sent to the endpoint. Leave it empty for endpoints that accept anonymous requests, such as a local router Models No Comma-separated chat model ids. Leave it empty to read GET <base URL>/models Image models No Image model ids to add. Calls go to /images/generations Video models No Video model ids to add. Calls go to /videos The provider id is custom_<slug>. A slug myproxy becomes custom_myproxy. That id is written into saved workflows and model references, so the slug cannot be changed after you create the endpoint. After you save, NodeTool runs a test automatically. Model discovery NodeTool finds models in this order: If you typed ids into Models, it offers those and does not call /models. Otherwise it sends GET <base URL>/models. The request times out after 5 seconds so an unreachable endpoint cannot stall the model menu. It sorts each listed model into chat, image or video. It uses the endpoint’s own metadata where there is any and the model id otherwise. Ids you entered under Image models and Video models are always offered in those pickers, on top of what the listing detects. A model with only an image-like or video-like id stays in the chat list, because the id alone is a guess and hiding a chat model would leave no way to pick it. Tool calling is not detected. Every chat model is offered with tools, and a model that does not support them fails at call time with the endpoint’s own error. The provider declares no image sizes or video durations. The node’s own size options apply and are passed to the endpoint as given. Test an endpoint Click Test on an endpoint’s card. The server requests the model list and reports how many models landed in each picker, for example “12 chat models, 3 image models and 1 video model available.” The card shows Chat, Image and Video counts. The test passes when it finds at least one model. If the endpoint answers with no models, the card says to enter model ids by hand, which is what to do for an endpoint with no /models route. If the request fails, the card shows the error. If an image or video model is missing from the counts, edit the endpoint and add its id to the matching field. API key handling The base URL and the API key are stored as encrypted secrets under your account. The card shows a Key set chip when a key exists and never shows the key itself. On a server you can skip the UI and set both values as environment variables. For slug myproxy: CUSTOM_MYPROXY_BASE_URL=https://proxy.example.com/v1 CUSTOM_MYPROXY_API_KEY=sk-... The variable names are CUSTOM_<SLUG>_BASE_URL and CUSTOM_<SLUG>_API_KEY with the slug in upper case. A stored value takes precedence over the environment variable. The endpoint’s name and model lists still live in the settings catalog, so the environment variables replace only the URL and key. A provider with no base URL counts as unconfigured and its models do not appear. Edit an endpoint Click Edit on the card. All fields except the slug can change. The API key field starts empty with the hint “Leave blank to keep the stored key”. Leaving it blank keeps the stored key. The dialog has no control to clear a stored key. Remove the endpoint and add it again to drop the key. Remove an endpoint Click Remove and confirm. NodeTool deletes the catalog entry and both secrets. Workflows that reference its models stop resolving, so switch their nodes to another model first. Where the models appear Chat, image and video models appear in the matching model pickers, listed under the endpoint’s name. The CLI loads the same endpoints for the default local user, so they work from nodetool commands as well. See Providers for the provider list and Configuration for environment variables.--- # Data Types and Connections Source: https://docs.nodetool.ai/data-types.html Every handle on a node has a data type. The editor uses it to color the handle, to decide whether two nodes can connect, and to label what flows along a line. For the canvas itself, see the Workflow Editor. For the ideas behind nodes and connections, see Key Concepts. Data types The table groups types that share a handle color. Hover a handle to see its type name and description. Type What it holds Handle color Example nodes Text (str, text) Labels, prompts, and multi-line text amber #FFA808 String Input, String Message, Thread A chat message with a role, or a conversation thread amber #FFA808 Create Thread Integer (int) A whole number #0891B2 Integer Input, Integer Float (float) A decimal number cyan #18E0F8 Float Input, Float Boolean (bool) True or false emerald #0DD49A Boolean Input, Bool Image A picture (PNG, JPEG, WebP, and others) fuchsia #E838FF Image Input, Image Audio Sound for playback, processing, or generation sky #08B8FF Audio Input, Audio Video A clip (MP4, WebM, and others) violet #9460FF Video Input, Video Model 3D A 3D model (GLB or glTF) blue #3888FF Model 3D Input, Model 3D, Transform 3D Document Text content with metadata, such as a PDF or DOCX blue #3888FF Document Input, Document List An ordered collection of items yellow #FFD612 Text List Input, List, Collect Dictionary (dict), Object, JSON Key-value data and nested structures yellow #FFD612 Dict Dataframe A table of rows and columns yellow #FFD612 Dataframe Input, Data Frame, For Each Row Chunk One piece of a streaming response yellow #FFD612 Summarizer CV A control-voltage stream for audio synthesis rose #FF3060 LFO Any Accepts or produces any type slate #6880A0 For Each The color names come from the palette comments in the source. The Integer color has no name there. Other types follow the same scheme. Asset, file, folder, bytes, font, and model references (language, embedding, image, video, and so on) are blue #3888FF. A sketch is lime #B0F030, a timeline and storyboard are violet #9460FF, a script is sky #08B8FF, and an entity is fuchsia #E838FF. The full list is in web/src/config/data_types.ts. A type the editor does not know gets a color derived from its name. A handle with no declared type uses grey #A7B1BF. Tip: Hold the pointer over a handle to see its type and a short description. When two ends have different colors, the line between them blends from the source color to the target color. Making connections To connect two nodes, drag from an output handle on the right of a node to an input handle on the left of another. Release on the handle to connect. The Workflow Editor covers releasing on empty canvas, which opens a menu of nodes that match the type you dragged. One output can feed many inputs. An input takes one line, so connecting a second line to the same input replaces the first. Collect inputs An input typed list[T] is a collect input. It accepts several lines at once, as long as each source produces T (or a list[T]). Use it to gather several images into one image list, for example. The exception is list[any], which takes a single line. Which connections are accepted The editor checks every connection against these rules. A connection that fails does not attach. Same type connects. An image output connects to an image input. Any connects to anything. A handle of type any on either end is always accepted. int and float are separate types. There is no automatic conversion between them, so an integer output does not connect to a float input. A single value connects to a list of that type. An image output connects to a list[image] input. Typed lists connect when their element types are compatible. Lists with no stated element type connect without a check. A list does not connect to a single value. To turn a list into items, use For Each. Dictionaries compare keys and values. dict connects to dict when both the key type and the value type are compatible. Unions match any member. A union source connects if any of its members fits the target, and a union target accepts any source that fits one of its members. Text and enum. Text connects to an enum input and an enum connects to a text input. Two enums connect only when they are the same enum. Object accepts structured types. An object input takes any source except text, numbers, booleans, null, enums, lists, dictionaries, unions, and tuples. CV and chunk interchange. A cv output connects to a chunk input and the reverse, so an LFO can drive a streaming audio input. CV does not connect to audio or float directly. The editor also refuses a connection that would form a cycle, and it refuses an exact duplicate of an existing line. The one exception is the feedback input of a Loop node. See the Loop node reference. Note: A node whose type metadata is not loaded, such as a placeholder for a missing node pack, accepts connections without a type check. Streaming Some nodes produce their output in pieces. A text model emits chunks as it generates, and For Each emits one list item at a time. The node keeps running while the pieces travel down the line, so downstream nodes can start before the upstream node finishes. How a downstream node reacts depends on the node: Per-item nodes run once for each value that arrives. Everything between For Each and the end of its branch runs once per item. Streaming-input nodes read values as they arrive instead of waiting for all of them. The LFO is one. Collect waits for the stream to end and returns every item as one list. Use For Each to split a list into items, and Collect to join them back. To repeat a step until a condition is met, use the Loop node. Edge states While a workflow runs, the editor shows what each line is doing. Moving dashes: a line animates with flowing dashes while its source node is starting or running, and while a message is crossing it. Item count: when more than one value has crossed a line, a small badge shows the count. Hover it to see how many items streamed. Selected: a selected line gets a thicker stroke and a soft glow in the source type color. Control lines, which let an agent trigger other nodes, are drawn with a separate style. Next steps Workflow EditorPlace, connect, and run nodes on the canvas. Your First WorkflowBuild a four-node workflow from an empty canvas. Control NodesLoop, For Each, Collect, If, and the other flow nodes. Key ConceptsThe ideas behind nodes, workflows, and assets.--- # End-to-End Deployment Guide Source: https://docs.nodetool.ai/deployment-e2e-guide.html This guide is a practical runbook for deploying NodeTool end-to-end with the unified server entrypoint (nodetool serve). It covers two production scenarios: Local / desktop runtime (no remote auth) Supabase-backed server (multi-user auth) What Runs in Production The server entrypoint is: nodetool serve --host 0.0.0.0 --port 7777 nodetool serve accepts only --host (default 127.0.0.1) and --port (default 7777). It sets HOST and PORT and starts the server from the built @nodetool-ai/websocket package. Use environment variables to control auth and runtime behavior. Run node backend/server.mjs in the Docker image, where HOST defaults to 0.0.0.0 in production mode. Important runtime behavior: Auth is enabled automatically when both SUPABASE_URL and SUPABASE_KEY are set; otherwise the server uses a local auth provider. There is no AUTH_PROVIDER switch read by serve. Production mode is selected by NODETOOL_ENV=production (the container also sets NODE_ENV=production). Production mode also turns off the /mcp mount and the Python bridge until you opt in, see Self-Hosted Deployment. The server needs a secrets master key. It reads SECRETS_MASTER_KEY first, then the system keychain, and generates a key into the keychain when neither exists. A host with no keychain (a headless Linux server) must set SECRETS_MASTER_KEY. The Docker image generates one into /workspace/.secrets_master_key when you leave it unset. /health (503 when the database check fails or the server is draining), /ready (liveness), and /api/health (version and uptime) require no authentication. Prerequisites NodeTool CLI installed (@nodetool-ai/cli). Container runtime: Docker or Podman Image for local deployment tests. The image refuses to start unless DB_PATH or DATABASE_URL is set. Build it from the repository root: docker build -t nodetool:local . If you use Podman: podman build -t nodetool:local . 1) Local / Desktop Runtime (no remote auth) Use this for local runtime where the app talks to a local API server. With no SUPABASE_* vars set, the server uses the local auth provider. export DB_PATH=/path/to/workspace/nodetool.db export HF_HOME=/path/to/hf-cache nodetool serve --host 127.0.0.1 --port 7777 Verify: curl -s http://127.0.0.1:7777/health curl -s http://127.0.0.1:7777/ready 2) Supabase-Backed Server Use this for internet-facing deployments where user auth is Supabase-backed. Setting both SUPABASE_URL and SUPABASE_KEY enables and enforces Supabase auth. export NODETOOL_ENV=production export SUPABASE_URL=https://<project>.supabase.co export SUPABASE_KEY=<service-role-or-server-key> export SECRETS_MASTER_KEY=<strong-random-secret> export DB_PATH=/workspace/nodetool.db export HF_HOME=/hf-cache nodetool serve --host 0.0.0.0 --port 7777 Verify: curl -i http://<host>:7777/health curl -i http://<host>:7777/api/workflows Expected: /health returns 200 (no auth required). /api/workflows without auth returns 401/403. Containerized End-to-End Run Docker docker run --rm -p 7777:7777 \ -e NODETOOL_ENV=production \ -e SECRETS_MASTER_KEY=<secret> \ -e DB_PATH=/workspace/nodetool.db \ -e HF_HOME=/hf-cache \ -v $(pwd)/workspace:/workspace \ -v $(pwd)/hf-cache:/hf-cache \ nodetool:local To enable Supabase auth, add -e SUPABASE_URL=… -e SUPABASE_KEY=… and -e SUPABASE_ANON_KEY=… for the login screen. Without Supabase, the server runs in Local mode and rejects requests that do not come from loopback or a network in NODETOOL_TRUST_LOCAL_NETWORKS, which docker run does not set. See Authentication. Podman podman run --rm -p 7777:7777 \ -e NODETOOL_ENV=production \ -e SECRETS_MASTER_KEY=<secret> \ -e DB_PATH=/workspace/nodetool.db \ -e HF_HOME=/hf-cache \ -v $(pwd)/workspace:/workspace \ -v $(pwd)/hf-cache:/hf-cache \ nodetool:local Workflow Sync to a Deployed Server Create the deployment config with nodetool deploy init and nodetool deploy add <name> --type docker (the file is deployment.yaml in the NodeTool config directory, ~/.config/nodetool/ on Linux). The deployment’s host and container.port give the server URL. Pass an admin bearer token with --token or NODETOOL_ADMIN_TOKEN, then sync: export NODETOOL_ADMIN_TOKEN=<admin token> nodetool deploy workflows sync <deployment-name> <workflow-id> Sync pushes the workflow definition only. Assets and models are not copied. List remote workflows: nodetool deploy workflows list <deployment-name> Run synced workflow via REST: curl -s -X POST http://<host>:7777/api/workflows/<workflow-id>/run \ -H "Authorization: Bearer <token>" \ -H "Content-Type: application/json" \ -d '{}' Production Checklist NODETOOL_ENV=production SECRETS_MASTER_KEY set, or a persistent /workspace volume so the image can keep its generated key Supabase auth (SUPABASE_URL + SUPABASE_KEY) configured if you need remote auth /health and /ready are green Unauthorized access to protected endpoints returns 401/403 Workflow sync and workflow run both succeed Troubleshooting Server exits on startup The Docker image exits at once when neither DB_PATH nor DATABASE_URL is set. Outside the image, a host with no keychain exits when SECRETS_MASTER_KEY is missing. Generate a key: openssl rand -base64 32 docker run or podman run reports “image not found” Docker and Podman keep separate image stores. Check the one you run with: docker image ls | grep nodetool podman image ls | grep nodetool Then rebuild/tag in that same runtime context.--- # Deployment Guide Source: https://docs.nodetool.ai/deployment.html NodeTool has two orthogonal deployment concerns. Don’t conflate them: Server (publish) — long-lived NodeTool infrastructure that humans and the UI connect into. NodeTool self-hosts as a single Docker server. See Self-Hosted Deployment. Worker (attach) — an ephemeral, billing-sensitive GPU box that one NodeTool instance connects out to, runs Python nodes on, and tears down. See Worker Deployment.   Server Worker Direction humans/UI connect in a NodeTool instance connects out Lifetime long-lived, always-on ephemeral — spin up, attach, tear down Identity a URL handed to people a {wsUrl, token} an instance adopts Cost flat bills while it exists, so teardown matters How nodetool deploy … (Docker) nodetool worker … (RunPod, Vast, Verda) For a walkthrough across desktop, public, private, and Docker/Podman self-hosting, see the End-to-End Deployment Guide. Quick Reference: What do you want to do? I want to… Go to Run the NodeTool server on my own machine/host Self-Hosted Deployment Understand how api.nodetool.ai itself is deployed Production Deploy (Docker) Rent a GPU to run Python nodes (RunPod / Vast / Verda) Worker Deployment Tune GPU/memory/volumes for the server container Docker Resource Management Use Supabase for auth/storage Supabase Deployment Integration Set up TLS/HTTPS Serving the Web UI and TLS Connect Claude Code or another MCP client to a deployed server MCP on a Production Server Server: self-host with Docker The quickest path is the reference docker-compose.yml at the repo root — cp .env.example .env && docker compose up -d. See Self-Hosted Deployment › Docker Compose. For a managed flow (remote hosts over SSH, image transfer, workflow sync), the nodetool deploy commands manage a single Docker server target driven by a deployment.yaml file. Pull the base image: docker pull ghcr.io/nodetool-ai/nodetool:latest Initialize and add a target: nodetool deploy init nodetool deploy add my-server --type docker deploy init creates deployment.yaml in the NodeTool config directory (~/.config/nodetool/ on Linux, ~/Library/Application Support/nodetool/ on macOS, %APPDATA%\nodetool on Windows). deploy add prompts for the host, SSH user and key (remote hosts only), image name and tag, container name, and port. The single target type is docker. Add container.environment, container.gpu, or paths afterwards with nodetool deploy edit, which opens the file in $EDITOR. Review & plan (no remote mutation): nodetool deploy list nodetool deploy show my-server nodetool deploy plan my-server # prints the plan as JSON nodetool deploy apply my-server --dry-run Apply & monitor: nodetool deploy apply my-server nodetool deploy status my-server nodetool deploy logs my-server --follow nodetool deploy destroy my-server deploy apply does not pull images. A local host needs the image already present. For a remote host, apply pushes the image from your local Docker daemon only when the remote has no copy. See Self-Hosted Deployment for the full server walkthrough, deploy workflows and deploy users-* commands, and Supabase Deployment Integration to add hosted auth and storage. Heads up: NodeTool used to ship server-deploy targets for RunPod serverless, Google Cloud Run, Fly, Railway, and HuggingFace Spaces. Those are gone. Server self-hosting is Docker only; GPU access is now a worker concern, not a server target. To run on a GPU, rent a worker (below). Worker: rent a GPU (RunPod, Vast, Verda) When a graph needs a GPU you don’t have, provision a remote worker, attach to it, run your Python nodes there, and tear it down: nodetool secrets store RUNPOD_API_KEY nodetool worker profile add hf-a40 --target runpod \ --image ghcr.io/nodetool-ai/nodetool-worker:latest \ --gpu "NVIDIA A40" --idle-timeout 15 nodetool worker create --profile hf-a40 --attach nodetool worker list # what's live, and what it's costing nodetool worker stop --all # pause every live worker GPU workers bill while they exist, so the worker subsystem ships a cost guard: idle auto-pause, a hard TTL that destroys the worker, and orphan reconcile. worker stop pauses a worker and keeps its volume, which still bills a little. To destroy a worker and its volume, use Terminate in the Workers panel. See Worker Deployment for profiles, attaching from the UI or CLI, supported targets, and the cost guard in full. Server configuration deployment.yaml accepts these top-level keys per deployment (see DockerDeployment in @nodetool-ai/deploy deployment-config.ts): type – always docker enabled – whether the deployment is active host – Docker host (IP/hostname, or localhost) ssh – SSH connection details for remote hosts (omit for local) paths – host directories for the workspace (default ~/nodetool_data/workspace, mounted at /workspace) and the HF cache (default ~/nodetool_data/hf-cache, mounted at /hf-cache, read-only unless persistent_paths is set) persistent_paths – optional container paths for the users file, database, vector store (vectorstore_db_path), HF home, assets, and logs. Setting it changes the injected environment, see below image – container image name/tag/registry container – name, port, gpu (device ids such as 0 or 0,1, passed as --gpus "device=…"), and environment (env vars injected into the container) server_auth_token – auto-generated bearer token for admin/sync calls. The server accepts it in Local mode (see Authentication) state – deployment state tracked by the deployer The container is named nodetool-<container.name> and runs with --restart unless-stopped. The server listens on 7777 inside the container and the deployer publishes it on host port container.port. A container.port of 7777 is published as 8000. The deployer also sets PORT, NODETOOL_API_URL, DB_PATH (/workspace/nodetool.db), VECTORSTORE_DB_PATH (/workspace/vectorstore.db), ASSET_FOLDER (/workspace/assets), HF_HUB_CACHE (/hf-cache), and SERVER_AUTH_TOKEN in the container, on top of container.environment. With persistent_paths set, the first three and HF_HOME come from it instead. Environment variables live under container.environment. When a deployment is loaded, deploy generates SECRETS_MASTER_KEY in container.environment and a server_auth_token if they are missing, and saves them to deployment.yaml. Keep that file private. Monitoring & health checks # Health endpoint (no auth required). Use the published host port: 17777 for # the bundled Compose file, container.port (default 8000) for `deploy apply`. curl http://your-server:17777/health # 200: {"status": "ok", "services": {...}, ...} # 503: status "degraded" (database check failed) or "draining" (shutting down) curl http://your-server:17777/ready # liveness only, always 200 nodetool deploy status <name> nodetool deploy logs <name> --follow Indicator What to watch Action Health endpoint Should return 200. /api/health returns the version and uptime Restart service if unhealthy Memory usage Models consume significant RAM/VRAM Scale up or use smaller models Disk space Model cache and assets grow over time Periodic cleanup or larger volumes Response time First request after cold start is slow (model loading) Warm up via health check Troubleshooting Problem Likely cause Fix Container exits immediately Missing env vars or invalid config Check nodetool deploy logs <name> deploy apply fails its health check The server did not answer /health after 10 probes, 2 seconds apart Read nodetool deploy logs <name>, fix the cause, and run apply again 503 Service Unavailable Overloaded or out of memory Scale up resources or reduce concurrency Port already in use Another service on the same port Change container.port in deployment.yaml “Image not found” Docker image not present docker pull ghcr.io/nodetool-ai/nodetool:latest Permission denied on volumes Container user lacks access Fix host directory permissions For more, see the Troubleshooting Guide. Upgrading docker pull ghcr.io/nodetool-ai/nodetool:latest # on the Docker host nodetool deploy apply <name> nodetool deploy status <name> apply stops and removes the existing container, then starts a new one from the image the host holds. It never pulls. For a remote host, pull on that host or change image.tag, because an image already on the remote is not replaced. Workflows, assets, and settings are preserved because /workspace is a host directory. Related Self-Hosted Deployment Worker Deployment Docker Resource Management Supabase Deployment Integration End-to-End Deployment Guide </content>--- # Desktop App Source: https://docs.nodetool.ai/desktop-app The desktop app is NodeTool packaged for Windows, macOS, and Linux. It opens the same editor as the web version in its own window and runs the NodeTool server on your machine, so your workflows, assets, and database stay local. This page is the hub for desktop-specific questions. Installing and connecting a provider is covered in Installation. The windows, tray, and menus are cataloged in Desktop App Views. Platforms and downloads Download the installer from nodetool.ai. The release build produces: OS Package Architectures macOS .dmg (and a .zip) Apple Silicon (arm64) and Intel (x64) Windows NSIS installer (.exe) x64 Linux AppImage, plus a Flatpak build See Installation Step-by-step install instructions for each OS are in Installation. First launch NodeTool shows a boot splash while the local server starts. The first start can take longer than later ones, and the splash shows the current step. If the server does not start, the splash offers Retry start, Open logs, and Reinstall environment. On macOS and Linux, NodeTool shows a one-time dialog explaining that it will use the system keychain to protect your stored API keys. Windows does not show it. What is optional: A Python environment. Only nodes written in Python need it, such as HuggingFace and MLX nodes, and those nodes also need their pack from Package Manager → Python packs. NodeTool sets the environment up on demand, from Tools → Package Manager or when you run a workflow that needs it. See What downloads later. Local model runners and models. llama.cpp, whisper.cpp, and model files download only when you install them. Ollama is a separate download from ollama.com. A cloud provider needs none of this. A provider. You do need one model source to run agents or generate media. See Connect an AI provider. The local server The app starts its own backend on 127.0.0.1, so it is reachable only from your machine. It tries port 7777 first. If that port is taken, it uses the next free one, up to 50 ports higher. The tray menu shows the port in use (“NodeTool: Running on "). The server starts when the app launches and stops when you quit. If the app cannot reach its server on Windows, approve the firewall prompt. See If installing goes wrong. Where your data lives Locations come from the app’s code. Replace ~ with your home folder. What macOS and Linux Windows Database, assets, and vector store ~/.local/share/nodetool/ (or $XDG_DATA_HOME/nodetool/) %APPDATA%\nodetool\ App settings (settings.yaml) ~/.config/nodetool/ %APPDATA%\nodetool\ Log file ~/.local/share/nodetool/logs/nodetool.log %LOCALAPPDATA%\nodetool\logs\nodetool.log Vaults ~/.local/share/nodetool/vaults/<id>/ %LOCALAPPDATA%\nodetool\vaults\<id>\ Python environment macOS ~/nodetool_env, Linux ~/.local/share/nodetool/conda_env %ALLUSERSPROFILE%\nodetool\conda_env The database file is nodetool.sqlite3, next to assets/ and vectorstore.db. Provider keys are stored encrypted in that database. See Connect an AI provider for where the encryption key lives. Other settings and environment variables are listed in Configuration. Desktop-only settings Open Settings and look under General → Workspace. Three sections appear only in the desktop app. Updates Automatic Updates is off by default. Turn it on to have the app check for and download updates in the background. It only runs in installed (packaged) builds. Update Channel is Stable (full releases) or Nightly (prerelease builds). Nightly builds default to the Nightly channel. On Close Behavior sets what closing the main window does: Ask Every Time, Quit Application, or Keep Running in Background. The same choice is in the tray menu. When an update is ready, a notice appears at the top right of the main window. See Update Notification. Vaults A vault is a separate data store with its own database, assets, and RAG collections. Use vaults to keep different sets of workflows and data apart. In Settings → Vaults you can create a vault, Switch to it, Rename it, or Delete it. The built-in default vault cannot be renamed or deleted, and you cannot delete the active vault. Switching restarts the backend and reloads the window. You can also switch from the Vaults menu in the menu bar. Vaults use SQLite, so they are skipped when an external DATABASE_URL is set. Package Manager Open Tools → Package Manager to install system runtimes (such as Python and the model runners) and to enable, disable, install, or update node packs. See Desktop App Views for the screen and Node Packs for where packs come from. Tray and menus NodeTool keeps a tray icon while it runs. From it you can start or stop the service, show the window, open chat, mini apps, the Log Viewer, and Settings, and quit. The full item lists for the tray and the menu bar are in Desktop App Views. Logs Use Tools → Log Viewer to read the server log in the app, or Open Log File in the tray menu to open the raw file at the path in the table above. Help → System Information lists the app version, OS details, and the data, conda, and log paths, with a Copy to Clipboard button. Include it when you ask for help. For common problems see Troubleshooting. Uninstalling Uninstalling removes the application files only. Your data, settings, logs, and Python environment stay on disk, and the Windows uninstaller keeps them on purpose. Steps for each OS are in Uninstalling. To remove everything, also delete the folders in Where your data lives. Related Installation Desktop App Views Self-Hosted Deployment for running the server without the desktop app Configuration--- # Development environment gotchas Source: https://docs.nodetool.ai/dev-environment.html Development environment gotchas Two setups where a plain npm install or a plain test run does not work: locked-down containers, and machines without a Vulkan driver. The everyday setup is AGENTS.md § Prerequisites. TypeScript compiler roles Normal repository builds and type-checks use the TypeScript 7 native compiler. Compiler watch commands use TypeScript 6 because TypeScript 7.0.2 can miss filesystem changes on Linux and in containers (upstream issue). The repository also installs TypeScript 6 as the compatibility package for tools that use the TypeScript compiler API, including Jest and documentation tooling. The shared launcher keeps these roles explicit and avoids npm bin link order deciding which compiler a script runs. To select a compiler for a command, set NODETOOL_TSC_VERSION: NODETOOL_TSC_VERSION=7 npm run typecheck NODETOOL_TSC_VERSION=6 npm run typecheck The named :ts6 and :ts7 package commands are the preferred rollback and comparison entry points. NODETOOL_TSC_HEAP_MB controls the V8 heap only for the TypeScript 6 JavaScript compiler. The TypeScript 7 native compiler does not receive that Node heap option. If a compiler-role check fails after install, run npm run check:typescript before attempting a build. It reports missing or mismatched compiler packages, native platform packages, and direct compiler-path drift. Local test and build memory Local Turbo scripts run one task at a time, and local Jest and Vitest suites use two workers. This leaves memory for macOS and the editor while checking the same code. CI keeps its existing parallelism. To tune a local run, set NODETOOL_TURBO_CONCURRENCY for Turbo tasks or NODETOOL_TEST_WORKERS for test workers, for example: NODETOOL_TEST_WORKERS=1 npm run test:affected NODETOOL_TURBO_CONCURRENCY=2 npm run build:packages TypeScript 7 is a native process, so Node’s heap flags do not limit its memory. The root npm run typecheck checks web, Electron, and mobile sequentially. libc in the lockfile npm selects a prebuilt native package by the os, cpu and libc recorded in package-lock.json. npm 10 reads libc from the registry but never writes it, which makes npm ci reify both the glibc and the musl build of every such package — npm install filters them, npm ci does not, so it lands in CI, the Docker image and every fresh clone rather than on a developer’s machine. npm run fix:lockfile-libc backfills the field from the registry, and npm run check:lockfile-libc (part of npm run check) fails when it goes missing again. Rerun the fix after any npm install under npm 10 that rewrites the lock, and commit the result. Worktrees with cloned dependencies (macOS) A full npm install in a new worktree takes minutes and several GB. On APFS, clone the dependency and build folders from an existing checkout instead. cp -c makes a copy-on-write clone: it takes seconds, uses almost no disk, and a later write in one tree does not change the other. Run scripts/setup-agent-env.sh inside a new worktree to do this automatically. It finds the main checkout through git, clones with cp -cR on macOS or cp -R --reflink=always on Linux, skips worktrees nested in the main checkout, and runs npm install when the file system cannot clone. The manual steps: MAIN=~/workspace/nodetool # checkout with installed deps WT=~/workspace/nodetool-wt/<name> git -C "$MAIN" fetch origin git -C "$MAIN" worktree add -b <branch> "$WT" origin/main cd "$MAIN" { find . -name node_modules -type d -prune | grep -v '/node_modules/.*node_modules' find . -path ./node_modules -prune -o -name dist -type d -prune -print | grep -v node_modules } | sort -u | while read -r d; do mkdir -p "$WT/$(dirname "$d")" && cp -cR "$d" "$WT/$d" done cd "$WT" && npm run build:packages # the cloned dist/ is from the source branch The workspace links in node_modules/@nodetool-ai/ are relative, so they resolve to the new worktree’s packages/. The cloned dist/ folders match the source checkout’s branch. Run npm run build:packages before you trust a test or harness result. du counts a clone at full size. Measure the space you get back with df. On Linux, cp --reflink=auto does the same on btrfs and XFS. On other file systems it makes a full copy, so run npm ci instead. Change dependencies in a worktree An install in a worktree writes only to that worktree’s clones. The source checkout does not change. A failed root postinstall deletes the better-sqlite3 binary it tried to rebuild. Clone it back with cp -cR "$MAIN/node_modules/better-sqlite3/build" node_modules/better-sqlite3/, or install with --ignore-scripts and run npm run rebuild:native. After npm install changes a version range, run npm explain <package>. npm can place the new version under packages/<name>/node_modules/ instead of the root. A package that the runtime loads by name, such as node-llama-cpp, must stay at the root. Record the root placement in the lockfile. A later npm install keeps a placement that already satisfies the range. npm 10 removes libc on every install that rewrites the lockfile. Run npm run fix:lockfile-libc last, and confirm with npm run check:lockfile-libc. mobile/node_modules is cloned with the rest but can be incomplete. Run npm --prefix mobile ci before mobile typecheck or tests. Remove worktrees git worktree remove --force deletes uncommitted work. First list the work that only the worktree holds: git -C "$WT" status --porcelain # uncommitted git -C "$WT" rev-list --count HEAD --not --remotes # unpushed commits Commit anything you keep to a branch, then run git worktree remove --force "$WT" and git worktree prune. Install in sandboxed / proxied environments Three postinstall steps break npm install in locked-down containers (CI sandboxes, Claude Code on the web, proxied networks). A failed postinstall makes npm roll back the entire node_modules tree, so one bad package means no dependencies at all — including ESLint and the design-lint gate. keytar needs libsecret headers on Linux. Without them node-gyp fails with Package libsecret-1 was not found. Fix first: apt-get install -y libsecret-1-dev. electron downloads its binary in postinstall. Proxies that block the download (HTTP 403) fail the install. Skip it when you don’t need to launch Electron: ELECTRON_SKIP_BINARY_DOWNLOAD=1 npm install. onnxruntime-node downloads CUDA binaries from GitHub releases in postinstall (it assumes CUDA when nvcc is absent). Same proxy failure mode, and there is no skip env var in our pinned version. When only the JS toolchain matters (typecheck, lint, unit tests that don’t hit native modules), bypass all of the above in one step: npm install --ignore-scripts --no-audit --no-fund This skips every postinstall — including the root better-sqlite3 rebuild — so anything touching the database needs npm run rebuild:native afterwards (which will still require the downloads above to have succeeded). WebGPU on a headless machine The image nodes are shader-backed: every lib.image.* generator and every nodetool.image transform reaches WebGPU through Dawn. On a machine with no Vulkan driver they fail with: No WebGPU adapter available (Node/Dawn). On headless Linux this usually means no Vulkan driver (ICD) is installed — Dawn has no software fallback of its own. This is an environment gap, not a broken test and not an unsupported platform. CI already solves it: the test-packages-* shards of .github/workflows/quality-checks.yml and the browser job in test.yml both install mesa-vulkan-drivers, which ships lavapipe — a CPU Vulkan ICD. Do not conclude from this error that shader-backed nodes cannot be tested, and do not skip a test because your box hits it; the same test passes in CI. With root: apt-get install -y mesa-vulkan-drivers Without root (sandboxes, dev containers), extract the driver and point the Vulkan loader at it. libvulkan1 — the loader — is usually already present; only the ICD is missing: apt-get download mesa-vulkan-drivers dpkg-deb -x mesa-vulkan-drivers_*.deb /tmp/vk # The shipped manifest names the library relatively, so rewrite it absolutely: python3 - <<'PY' import json p = "/tmp/vk/usr/share/vulkan/icd.d/lvp_icd.json" d = json.load(open(p)) d["ICD"]["library_path"] = "/tmp/vk/usr/lib/x86_64-linux-gnu/libvulkan_lvp.so" json.dump(d, open("/tmp/vk/lvp_icd.json", "w")) PY export VK_DRIVER_FILES=/tmp/vk/lvp_icd.json Then run the tests as usual. Lavapipe is a software rasterizer, so it is slow but exact — pixel comparisons (nodetool.compare.CompareImages) are reproducible under it, which is what packages/base-nodes/tests/image-examples-run.test.ts relies on.--- # Developer Guide Source: https://docs.nodetool.ai/developer/ Resources for building custom nodes, extending NodeTool, and integrating workflows programmatically. Quick Start: Custom Nodes Creating custom nodes in TypeScript: import { BaseNode, prop } from "@nodetool-ai/node-sdk"; export class UppercaseTextNode extends BaseNode { static readonly nodeType = "mypackage.text.Upper"; static readonly title = "Uppercase Text"; static readonly description = "Convert text to uppercase."; static readonly metadataOutputTypes = { output: "str" }; @prop({ type: "str", default: "", title: "Input Text" }) declare inputText: string; async process(): Promise<Record<string, unknown>> { return { output: String(this.inputText ?? "").toUpperCase() }; } } Export a register(registry) function and add a nodetool field to package.json ({ "apiVersion": 1, "register": "register" }). At startup the server scans installed packages for that field and calls the export. In production, only packs on the allowlist load. -> Full Custom Nodes Guide (TypeScript) – End-to-end walkthrough: packaging, governance, streaming, testing, distribution. Guides Provider & Model Guides Provider Guides – Add models & nodes to any provider. Per-provider runbooks (OpenAI, Anthropic, Gemini, xAI, FAL, Replicate, KIE, Together, AtlasCloud, Topaz, MiniMax, Reve, ElevenLabs, HuggingFace, Ollama, local inference, OpenAI-compatible) with exact files, commands, and how past PRs did it. Written for coding agents and contributors. Custom Node Development (TypeScript) Custom Nodes Guide – Start here! End-to-end guide for authoring, packaging, and distributing TypeScript node packs. TypeScript DSL Guide – Type-safe workflow definitions with auto-generated factory functions. Custom Node Reference (TypeScript) Node Implementation Quick Reference – Templates and common @prop / process() patterns. Node Implementation Patterns – Architectural patterns: multi-output, streaming, stateful, secrets. Python Nodes Node Implementation Examples – Python node examples for the Python bridge. Advanced Suspendable Nodes – Why there is no pause/resume node API, and what WaitNode actually does. JavaScript Sandbox – The QuickJS guest behind the Code node and agent code actions: capabilities, limits, imports, security model. Programmatic Workflows TypeScript DSL Guide – Type-safe workflow definitions with auto-generated factory functions Portable Workflow Runner – Run a workflow graph from a (Request) => Response handler on Node, Bun, Deno, Vercel, Cloudflare Workers, or in the browser Architecture Architecture – Packages, execution engine, message types, providers, storage Execution Strategies – How the kernel schedules actors and how the Code node is sandboxed Packages – How node packages are structured, registered, and managed from the CLI Node Packs – Built-in packs, the Package Manager, and installing third-party packs Python Bridge Protocol – Wire protocol between the TypeScript runtime and the Python worker API Integration API Reference – REST API endpoints and authentication Headless Mode – Run workflows via CLI and HTTP API Chat API – OpenAI-compatible chat endpoints Workflow API – Execute workflows programmatically Contributing Contribute to NodeTool on GitHub. Share Custom Nodes Options: Publish as a separate npm package Contribute to the core node library in the repository Share workflow examples on Discord--- # Custom Nodes Guide (TypeScript) Source: https://docs.nodetool.ai/developer/custom-nodes-guide.html This is the full guide to writing TypeScript custom nodes for NodeTool. Node Patterns and Node Reference cover the same TypeScript API in less depth; Node Examples covers the Python nodes reached through the Python bridge. A custom node lives in a standalone npm package — a pack — that the server discovers and loads at startup. There is nothing to register by hand: drop a nodetool field in package.json, install the package, and the loader does the rest. Related: Node Packs (user-facing intro), Package Registry Guide (first-party package conventions), TypeScript DSL Guide (using nodes in code-defined workflows). Contents Quick start Package layout package.json and the pack manifest Trust model and governance tsconfig.json Anatomy of a node The @prop decorator reference The type system Declaring outputs ProcessingContext — the runtime surface Streaming nodes (genProcess) Lifecycle hooks Registering nodes Building the pack Testing nodes Installing and running the pack Versioning the pack API Common pitfalls 1. Quick start Scaffold a pack, write one node, install it, run the server: mkdir nodetool-mypack && cd nodetool-mypack npm init -y npm install --save @nodetool-ai/node-sdk @nodetool-ai/runtime @nodetool-ai/protocol npm install --save-dev typescript @types/node vitest nodetool package init scaffolds a similar package. It writes the nodetool field and an empty export function register(...), but its tsconfig.json lacks experimentalDecorators. Add the compiler flags from §5 before you write nodes. src/nodes/reverse.ts: import { BaseNode, prop } from "@nodetool-ai/node-sdk"; export class ReverseTextNode extends BaseNode { static readonly nodeType = "mypack.text.Reverse"; static readonly title = "Reverse Text"; static readonly description = "Reverse a string character by character."; static readonly metadataOutputTypes = { output: "str" }; @prop({ type: "str", default: "", title: "Text" }) declare text: string; async process(): Promise<Record<string, unknown>> { return { output: [...(this.text ?? "")].reverse().join("") }; } } src/index.ts: import type { NodeRegistry } from "@nodetool-ai/node-sdk"; import { ReverseTextNode } from "./nodes/reverse.js"; export function register(registry: NodeRegistry): void { registry.register(ReverseTextNode); } Add a nodetool field to package.json (see §3), build with tsc, then npm link (or npm install) into your NodeTool workspace and restart the server. The node appears in the menu as Reverse Text under mypack.text. 2. Package layout A pack is a normal npm package. Keep one node (or a small group) per file: nodetool-mypack/ ├── package.json ├── tsconfig.json ├── README.md ├── src/ │ ├── index.ts # entry — exports `register(registry)` │ └── nodes/ │ ├── reverse.ts │ └── math.ts └── tests/ └── reverse.test.ts Keep the public surface in src/index.ts minimal: re-export node classes and the register function. Everything else stays internal. 3. package.json and the pack manifest { "name": "@acme/cool-nodes", "version": "0.1.0", "description": "Cool custom nodes for NodeTool", "type": "module", "main": "dist/index.js", "types": "dist/index.d.ts", "exports": { ".": { "import": "./dist/index.js", "types": "./dist/index.d.ts" } }, "scripts": { "build": "tsc", "test": "vitest run", "lint": "tsc --noEmit" }, "dependencies": { "@nodetool-ai/node-sdk": "latest" }, "devDependencies": { "@types/node": "^22.0.0", "typescript": "^5.4.0", "vitest": "^1.6.1" }, "nodetool": { "apiVersion": 1, "register": "register" } } The nodetool field is what makes the package a pack. At startup the loader scans installed packages, picks up any with this field, imports the resolved entry, and calls the named export with the registry. Field Default Meaning apiVersion 1 Pack API version you built against. Packs declaring a version newer than the host supports are skipped with a warning. register "register" Named export the loader calls with the registry. Can be async. The loader also looks for it on a default export object. sandboxModules none Declares guest modules for the JavaScript sandbox. A pack that sets this field without register is a sandbox-only pack, and the node loader ignores it. The loader uses the "." entry of exports (a string, or the import condition falling back to default, which may be nested), or main, or index.js — in that order. A nodetool field that fails schema validation logs a warning and the package is skipped. Where the loader looks. It scans every node_modules directory from the working directory up to the filesystem root, plus the directories listed in NODETOOL_OPTIONAL_NODE_MODULES (one path) and NODETOOL_PACK_SEARCH_PATHS (a list separated by commas, semicolons, or the platform path separator). It then scans the root holding the sandbox packs that ship with NodeTool (NODETOOL_SHIPPED_PACKS_DIR overrides that root). When two directories hold a package of the same name, the first one found wins. Packs load once at server start. The packs.reload tRPC mutation re-runs the scan, but it cannot unload nodes that are already registered. 4. Trust model and governance Custom nodes run in the server process as the server user. They have full filesystem, network, secret-store, and process.env access, with no sandbox. A pack is exactly as trusted as any dependency you npm install. Only install packs you trust. To avoid silently running whatever happens to be in node_modules in production, the loader is gated: Allowlist — a list of trusted pack names; "*" allows everything. Set via: The env var NODETOOL_PACKS_ALLOWLIST (comma-separated names), which takes precedence, or The allow field of ~/.config/nodetool/packs.json (path overridable with the NODETOOL_PACKS_CONFIG env var). allowUnlisted — whether packs not on the allowlist load anyway. Defaults to true in development (so installing a pack just works) and false when either NODETOOL_ENV=production or NODETOOL_PACKS_REQUIRE_ALLOWLIST=1 is set. The packaged desktop app sets the latter — it needs the allowlist without production mode, which would disable local-only features. Override via the allowUnlisted field of the config file. The packs.getTrust and packs.setTrust tRPC procedures read and write the same file, and setTrust does not copy an env override into it. Two further guards protect the registry regardless of trust: Reserved namespaces — packs cannot register node types whose first dot-separated segment is one of: nodetool, lib, comfy, default, huggingface, hf, mlx, transformers, openai, gemini, anthropic, mistral, groq, ollama, replicate, fal, elevenlabs, kie, vector, apify, search, messaging. Such nodes are skipped with a warning. Collision protection — a pack cannot shadow an already-registered node type (a built-in, or a node registered earlier by another pack). The conflicting node is skipped with a warning; the original wins. Example allowlist file: { "allow": ["@acme/cool-nodes", "@other-org/audio-pack"], "allowUnlisted": false } 5. tsconfig.json The SDK uses legacy (experimental) decorators without runtime metadata emission. Match those flags or builds will produce unusable output: { "compilerOptions": { "target": "ES2022", "module": "Node16", "moduleResolution": "Node16", "lib": ["ES2022"], "strict": true, "esModuleInterop": true, "skipLibCheck": true, "declaration": true, "sourceMap": true, "outDir": "dist", "rootDir": "src", "experimentalDecorators": true, "emitDecoratorMetadata": false }, "include": ["src"] } Keep "emitDecoratorMetadata" set to false. The SDK does not read decorator metadata. Property declarations must use declare (see §6), which emits no field initializer and so works whatever useDefineForClassFields resolves to. 6. Anatomy of a node Every node extends BaseNode and declares its inputs with @prop. The runtime assigns properties on the instance before calling process() — your method reads inputs from this.<field>, not from a parameter. import { BaseNode, prop } from "@nodetool-ai/node-sdk"; import type { ProcessingContext } from "@nodetool-ai/runtime"; export class AddOffsetNode extends BaseNode { // ── Identity ──────────────────────────────────────────────── static readonly nodeType = "mypack.math.AddOffset"; static readonly title = "Add Offset"; static readonly description = "Add a constant offset to a number."; // ── Output type ──────────────────────────────────────────── static readonly metadataOutputTypes = { output: "float" }; // ── Inputs ───────────────────────────────────────────────── @prop({ type: "float", default: 0.0, title: "Value" }) declare value: number; @prop({ type: "float", default: 1.0, title: "Offset" }) declare offset: number; // ── Execution ────────────────────────────────────────────── async process(_context?: ProcessingContext): Promise<Record<string, unknown>> { return { output: (this.value ?? 0) + (this.offset ?? 1) }; } } Required static members Member Type Purpose nodeType string Unique dotted identifier. By convention: <namespace>.<category>.<Name>. title string Display name in the node menu. description string One-line tooltip. Common optional static members Member Type Purpose metadataOutputTypes { [name]: typeString } Maps output handle name → NodeTool type string. Declare every output the node returns: a node with no declared outputs has no output handles. isStreamingInput boolean Marks the node as consuming a stream via the run(...) hook (pair with inputMode = "stream"). supportsDynamicInputs boolean Allows users to add/remove input handles in the UI; read/write them with getDynamic / setDynamic. inputMode "buffered" \| "stream" \| "controlled" How inputs are consumed. Default (undefined) is buffered. outputCorrelation { [output]: OutputCorrelation } Controls per-iteration / per-chunk fan-out semantics. Streaming output is inferred from this (forward, iteration, or chunk kinds) or from overriding genProcess. There is no isStreamingOutput flag. requiredSettings string[] Secret names resolved from the secret store before the node runs. Read them from this._secrets. inlineFields / inputFields string[] Which properties render compactly on the node body, and which render as expanded inputs. effect "pure" \| "read" \| "write" \| "external" What running the node does to the world. Reactive runs (a slider drag, a mini app input change) execute only pure and read nodes. Default "external". cacheTtl number \| "forever" How long a partial run (“Run Node”, “Run from here”) may reuse the result. "forever" for pure deterministic nodes, seconds for time-sensitive ones. Default: never reuse. retrySafe boolean Opt in when re-running with identical inputs is safe. The workflow supervisor offers retry only for nodes that declare it. Default false. deprecated / hidden / replacedBy boolean / boolean / string Lifecycle flags. hidden keeps the node runnable but out of discovery UIs. platforms Platform[] Deployment targets the node supports. Default ["node"]. The process method abstract process(context?: ProcessingContext): Promise<Record<string, unknown>> Returns an object whose keys are output handle names declared in metadataOutputTypes. Reads inputs from this.<field>. The runtime has already populated them via assign(). The context parameter is optional. Many pure-compute nodes ignore it. Anything that touches secrets, storage, HTTP, or providers will use it — see §10. Throwing an Error fails the node and surfaces the message to the UI. Throw Error objects, not strings. Do not add an inputs parameter — that pattern is from an older draft of this guide and does not match the runtime contract. Property assignment happens before process() runs. 7. The @prop decorator reference @prop is the only decorator pack authors call directly. Outputs are declared via static class members, not decorators. @prop(options: PropOptions) Option Type Purpose type string (required) NodeTool type string — see §8. default unknown Default value when no upstream input is connected. title string Display name in the UI. Defaults to the field name. description string Tooltip text. min / max number Bounds for numeric types — used by sliders and validation. required boolean Whether the input must be connected or set. values (string \| number)[] Allowed values; renders as a dropdown / enum selector. json_schema_extra Record<string, unknown> Custom UI metadata (renderer hints, layout, etc.). Always pair @prop with a declare field — the runtime owns assignment, so emitted initializers would just be overwritten. @prop({ type: "str", default: "", title: "Prompt", description: "What to generate" }) declare prompt: string; @prop({ type: "float", default: 0.7, min: 0, max: 2, title: "Temperature" }) declare temperature: number; @prop({ type: "str", default: "auto", values: ["auto", "fast", "best"], title: "Mode" }) declare mode: string; 8. The type system The type string in @prop and the values in metadataOutputTypes come from the same vocabulary — the one shared with Python nodes. Scalars String TS type "str" string "int" number "float" number "bool" boolean "json" unknown "any" unknown Collections String TS type "list[T]" T[] (e.g. list[str], list[any]) "dict[str, any]" Record<string, unknown> "dict[str, str]" Record<string, string> Media references Media flows through the graph as small reference objects, not as raw bytes. Import the types from @nodetool-ai/node-sdk: @nodetool-ai/node-sdk re-exports only ImageRef, AudioRef, VideoRef, TextRef, and DataframeRef. DocumentRef, Model3DRef, and FolderRef are not re-exported by the SDK — import those from @nodetool-ai/protocol: import type { ImageRef, AudioRef, VideoRef, TextRef, DataframeRef } from "@nodetool-ai/node-sdk"; import type { DocumentRef, Model3DRef } from "@nodetool-ai/protocol"; String TS type Use for "image" ImageRef Images (URI or inline bytes) "audio" AudioRef Audio clips "video" VideoRef Video files "document" DocumentRef PDFs, Word, plain-text files "text" TextRef Large text blobs by reference "dataframe" DataframeRef Tabular data "model_3d" Model3DRef 3D meshes / glTF "folder" FolderRef Asset folders "collection" { type: "collection", name } Vector database collections (see vector.Collection) DocumentRef, Model3DRef, and FolderRef come from @nodetool-ai/protocol. Model selectors These render as model pickers in the UI: String Meaning "language_model" An LLM (provider + model id pair). "image_model" An image-generation model. "video_model" A video-generation model. "tts_model" A text-to-speech model. "asr_model" A speech-recognition model. "embedding_model" An embedding model. Domain types "date", "datetime", "image_size", "enum" — render with specialised inputs. The vocabulary is open: asset and node-defined types are allowed. Use NodeTool’s spellings, not JSON Schema or TypeScript ones. integer, string, boolean, object, and array are not types here, and a dynamic slot declared with one of them is rejected because the handle would never connect to an int or str input. 9. Declaring outputs Single output, default name output: static readonly metadataOutputTypes = { output: "str" }; async process(): Promise<Record<string, unknown>> { return { output: "hello" }; } Multiple outputs: static readonly metadataOutputTypes = { text: "str", tokenCount: "int" }; async process(): Promise<Record<string, unknown>> { const text = this.input ?? ""; return { text, tokenCount: text.split(/\s+/).length }; } Every key returned by process() must be declared in metadataOutputTypes. The editor exposes only declared outputs as handles, so an undeclared key is unreachable downstream. A key you leave out (or set to undefined) sends no message on that output, and nodes wired only to it do not run. nodetool.control.If relies on this: it returns only if_true or only if_false. If you do not declare metadataOutputTypes, the class falls back to a static outputTypes map, and a node with neither has no outputs. 10. ProcessingContext — the runtime surface The context passed to process() and genProcess() is your gateway to everything the runtime owns: secrets, storage, cache, HTTP, providers, messages. You only need it when a node has side effects. import type { ProcessingContext } from "@nodetool-ai/runtime"; Identity context.jobId; // string — unique per workflow run context.workflowId; // string | null context.userId; // string context.signal; // AbortSignal — pass it to fetch and other cancellable calls context.workspace; // Workspace | null — the run's files (preferred) context.workspaceDir; // string | null — deprecated, null for cloud workspaces Read and write run files through context.workspace (read, readText, write, exists, stat, list, copy, move, delete) with workspace-relative paths. It works on local and cloud runs alike. context.workspace.localDir is null for a virtual (cloud) workspace. To run a host binary on a workspace file, use materialize and absorb, and scratchDir() for outputs. Secrets const apiKey = await context.getSecret("OPENAI_API_KEY"); // string | null const required = await context.getSecretRequired("STRIPE_KEY"); // throws if missing Always prefer getSecret over process.env — secrets are user-scoped and may come from an encrypted store, not the process environment. HTTP helpers const resp = await context.httpGet("https://api.example.com/data", { headers: { Authorization: `Bearer ${apiKey}` } }); await context.httpRequestWithRetries("POST", url, { headers: { "Content-Type": "application/json" }, body: JSON.stringify({ foo: 1 }), retry: { maxRetries: 5 } }); httpGet is the GET shorthand. Every other verb goes through httpRequestWithRetries, which carries the shared retry and backoff policy. Options are RequestInit plus retry (maxRetries default 3, backoffMs default 500 with exponential backoff, retryStatuses default 408, 425, 429, 500, 502, 503, 504). A non-retryable error status throws without retrying. These helpers do not apply SSRF protection. For a URL that a user, provider, or model chose, use safeFetch or fetchExternalMedia from @nodetool-ai/runtime. Cache The cache is exposed at context.cache and is in-memory unless the host supplies another adapter. All methods are async. set takes an optional TTL in seconds, and get takes only the key (it returns undefined on a miss — there is no default-value argument): // CacheAdapter signatures: // get<T>(key: string): Promise<T | undefined> // set<T>(key: string, value: T, ttlSeconds?: number): Promise<void> // has(key: string): Promise<boolean> // delete(key: string): Promise<void> const hit = await context.cache.get<Record<string, unknown>>("my-key"); if (hit !== undefined) return hit; const result = await expensive(); await context.cache.set("my-key", result, 3600); // expires after 1 hour For node-result memoization, the convenience helpers wrap the same cache. The key combines the user id, the node type, and a deterministic serialization of the properties. cacheResult defaults to a one-hour TTL: const cached = await context.getCachedResult(this.nodeType, this.serialize()); if (cached) return cached; const result = await expensive(); await context.cacheResult(this.nodeType, this.serialize(), result, 3600); return result; Storage and workspace files const uri = await context.storage.store("output.png", bytes, "image/png"); const data = await context.storage.retrieve(uri); await context.workspace?.write("notes.txt", "hello", "text/plain"); const notes = await context.workspace?.readText("notes.txt"); context.storage is null when the host wires no storage adapter. context.workspaceStorage still exists but is deprecated in favor of context.workspace. To read the bytes of an image, audio, video, or model reference, call loadMediaRefBytes(ref, context) from @nodetool-ai/runtime. It handles inline data, asset:// references, storage-backed URIs, and http(s) URLs (through the media egress policy), and returns null when it finds no bytes. LLM providers if (await context.isProviderConfigured("openai")) { const provider = await context.getProvider("openai"); const stream = provider.streamChat({ messages, model: "gpt-5.4-mini" }); } Variables (per-job key/value scratch) context.set("counter", 0); const n = context.get<number>("counter", 0); context.hasVariable("counter"); // true Messages context.emit(msg) posts a ProcessingMessage (log line, status update, tool call, …) onto the run’s event stream. The UI uses these to surface progress. 11. Streaming nodes (genProcess) For nodes that produce results incrementally, override genProcess and declare an iteration outputCorrelation. The base class detects that the subclass overrides genProcess and iterates it — there is no isStreamingOutput flag to set: import { BaseNode, prop } from "@nodetool-ai/node-sdk"; import type { ProcessingContext } from "@nodetool-ai/runtime"; import type { OutputCorrelation } from "@nodetool-ai/protocol"; export class WordStreamNode extends BaseNode { static readonly nodeType = "mypack.text.WordStream"; static readonly title = "Word Stream"; static readonly description = "Emit each word of the input as a separate event."; static readonly metadataOutputTypes = { word: "str" }; static readonly outputCorrelation: Record<string, OutputCorrelation> = { word: { kind: "iteration", source: "__execution__", group: "items" }, }; @prop({ type: "str", default: "", title: "Text" }) declare text: string; async process(): Promise<Record<string, unknown>> { // Fallback for non-streaming consumers. return { word: this.text }; } async *genProcess( _context?: ProcessingContext ): AsyncGenerator<Record<string, unknown>> { for (const word of String(this.text ?? "").split(/\s+/)) { if (word) yield { word }; } } } Output correlation outputCorrelation tells the runtime how each output relates to the node’s inputs, so downstream nodes line up items correctly: // metadataOutputTypes declares both outputs: { word: "str", count: "int" } static readonly outputCorrelation = { word: { kind: "iteration", source: "__execution__", group: "items" }, count: { kind: "single", source: "__execution__" } }; kind Meaning single One logical output per invocation. It inherits the invocation’s lineage. iteration Each emitted value is a new logical item with its own correlation token. Outputs that share a group share one token per yield. chunk A piece of a streamed value, such as text deltas. It inherits the base lineage. forward Emits per input item and copies that item’s lineage. source names the input handle. aggregate A stream-mode node that consumes child items and emits at a collapsed scope. Requires collapse: "innermost". Grouped iteration outputs must be emitted together, so yield one object per item from genProcess (as ForEach does). Once you declare outputCorrelation, it needs one entry for every declared output and no entry for an undeclared one. Every entry needs a source, forward cannot use __execution__, and aggregate needs collapse and a non-buffered inputMode. registry.register throws a CorrelationMetadataError on any violation, and inside a pack that makes the whole pack fail to load. See Correlation Design for the rules. Streaming inputs (run) A node that consumes a stream sets isStreamingInput = true and inputMode = "stream", and implements run(inputs, outputs, context?) instead of relying on process(). It still needs a process() stub because the base class requires one. nodetool.control.Take is a small example: import type { StreamingInputs, StreamingOutputs } from "@nodetool-ai/node-sdk"; async run(inputs: StreamingInputs, outputs: StreamingOutputs): Promise<void> { for await (const item of inputs.stream("input_item")) { await outputs.emit("output", item); } } inputs offers stream(name), any() (all handles in arrival order), first(name, default?), and hasStream(name). outputs offers emit(slot, value), emitGroup(values) for grouped iteration outputs, forward, drop, and complete(slot) for early end-of-stream. Default behaviour If you don’t override genProcess, the base implementation yields the single result of process(). So a non-streaming node needs nothing extra; only streaming nodes override genProcess. 12. Lifecycle hooks Override these on your class to run setup and teardown: async initialize(): Promise<void> { /* once per node, before the run starts */ } async preProcess(): Promise<void> { /* once, when the node's actor starts, before its first invocation */ } async finalize(): Promise<void> { /* once when the node finishes, even after an error */ } initialize runs for every node in the graph before any node executes. If one node’s initialize throws, the run fails and the nodes already initialized are finalized. Use it for expensive one-time setup (e.g. opening a connection) and reset of per-run state, and use finalize to release it. preProcess does not run before each process() call, and it is rarely needed. A finalize error is swallowed so it cannot hide the original failure. 13. Registering nodes Your pack’s register function — the export named in the manifest — receives the registry and adds each node class: import type { NodeRegistry } from "@nodetool-ai/node-sdk"; import { AddOffsetNode } from "./nodes/math.js"; import { ReverseTextNode } from "./nodes/reverse.js"; import { WordStreamNode } from "./nodes/stream.js"; const ALL_NODES = [AddOffsetNode, ReverseTextNode, WordStreamNode] as const; export function register(registry: NodeRegistry): void { for (const cls of ALL_NODES) { registry.register(cls); } } Things the loader will refuse — silently dropping the offending node and warning in the log: A nodeType under a reserved namespace. A nodeType already registered by a built-in or an earlier pack (no shadowing). The register function may be async — useful if you build node classes from a manifest at load time. Calling registry.register directly (outside the loader) skips both guards and replaces an existing registration of the same type with a warning. What NodeRegistry exposes Pack authors only need: Method Purpose register(nodeClass, opts?) Add a node class. has(nodeType) Already registered? list() All registered node types. getClass(nodeType) The class for a node type. listMetadata() UI metadata for every registered node. register throws if the class has no nodeType. Everything else on the registry is for the runtime. 14. Building the pack npm run build # tsc → dist/ npm run lint # tsc --noEmit npm run test # vitest run Ship the dist/ directory plus package.json and a README.md. Don’t ship src/ — it won’t be loaded. Recommended package.json “files” allowlist "files": ["dist", "README.md", "LICENSE"] 15. Testing nodes Use Vitest (same harness the SDK uses internally). The pattern for an end-to-end test: import { describe, it, expect } from "vitest"; import { NodeRegistry } from "@nodetool-ai/node-sdk"; import { register } from "../src/index.js"; import { ReverseTextNode } from "../src/nodes/reverse.js"; describe("ReverseTextNode", () => { it("registers under its nodeType", () => { const registry = new NodeRegistry(); register(registry); expect(registry.has(ReverseTextNode.nodeType)).toBe(true); }); it("reverses a string", async () => { const node = new ReverseTextNode({ text: "hello" }); const result = await node.process(); expect(result).toEqual({ output: "olleh" }); }); }); For nodes that need a ProcessingContext, use createFakeContext from @nodetool-ai/runtime. It returns { context, workspaceDir, providers, cleanup } with in-memory storage and cache, in-memory asset and timeline saving, a stubbed fetch, and a fake provider. Pass providers, secretResolver, fetchFn, or variables to control what the node sees, and call cleanup() in afterEach: import { createFakeContext } from "@nodetool-ai/runtime"; const handle = createFakeContext({ secretResolver: (k) => (k === "API_KEY" ? "test" : null) }); const result = await node.process(handle.context); handle.cleanup(); To run one node against the real registry from the command line, use nodetool node run <type> --props '{"text":"hello"}'. Add --no-secrets for a hermetic run and --json for the full result. See CLI reference. Testing streaming nodes it("yields one event per word", async () => { const node = new WordStreamNode({ text: "foo bar baz" }); const chunks: unknown[] = []; for await (const chunk of node.genProcess()) chunks.push(chunk); expect(chunks).toEqual([{ word: "foo" }, { word: "bar" }, { word: "baz" }]); }); 16. Installing and running the pack For a published pack: cd /path/to/nodetool npm install @acme/cool-nodes # in production, allowlist it: echo '{ "allow": ["@acme/cool-nodes"] }' > ~/.config/nodetool/packs.json npm run dev:server For local development, npm link the pack so edits land without re-publishing: cd nodetool-mypack && npm run build && npm link cd /path/to/nodetool && npm link @acme/cool-nodes npm run dev:server The server logs which packs were discovered: Loaded node pack @acme/cool-nodes@0.1.0 (3 node(s)) Skipped node pack @other/blocked@1.0.0: not on pack allowlist Pack @evil/shadowy: skipped node nodetool.text.Override (reserved-namespace) Failed to load node pack @acme/broken@0.2.0: entry "…/dist/index.js" has no callable export "register" The node-level skip reasons are reserved-namespace and collision. A pack-level skip reports not on pack allowlist or requires pack API v2, host supports v1. A pack that throws while loading is reported as failed and does not stop other packs. packs.list (tRPC) returns the same startup snapshot. 17. Versioning the pack API The pack API is versioned with a single integer (apiVersion, currently 1). Forward compatibility: If your pack declares apiVersion: 2 and runs on a host that only knows 1, the host skips it cleanly rather than crashing. If the host knows 2 and your pack still declares 1, everything keeps working — the host is responsible for backward compatibility. When the host bumps the version, the changelog will name exactly what changed and what (if anything) authors need to migrate. 18. Common pitfalls Property declarations without declare. TypeScript will emit initializers that overwrite the values the runtime just assigned. Always: @prop({ type: "str", default: "" }) declare text: string; // good @prop({ type: "str", default: "" }) text: string = ""; // bad — runs after assign() ESM .js extensions in imports. With module: "Node16", internal imports must use the .js extension (matching the compiled output), even from .ts source: import { ReverseTextNode } from "./nodes/reverse.js"; // good import { ReverseTextNode } from "./nodes/reverse"; // bad emitDecoratorMetadata. Leave this false. The SDK does not consume runtime metadata and enabling it can produce conflicting decorator output. Returning an undeclared output key. The editor offers only the handles in metadataOutputTypes. A key outside that map is unreachable downstream. Loading from src/ instead of dist/. The loader resolves the package’s exports/main, which points at dist/. Forgetting to build before installing means the pack loads stale code (or nothing). Reserved-namespace nodeTypes. A nodeType of nodetool.foo.MyNode will be silently rejected. Use your own namespace (mypack.foo.MyNode, @acme.foo.MyNode, …). Collision with built-ins. If a built-in already owns the nodeType you picked, your node is dropped. Pick a unique name and check the server log for skipped node … (collision). Throwing strings instead of Error. The runtime surfaces .message of the error it catches, so a thrown string has no message to show. Long-running process blocking the event loop. If your work is CPU-bound, do it in a Worker. The server runs on a single Node.js event loop. Reading from process.env for user-scoped secrets. Use requiredSettings with this._secrets, or context.getSecret(key). Both consult the user’s secret store first (which may be encrypted and is not the process environment) and then fall back to the environment. Related documentation Node Packs — user-facing intro to packs. Package Registry Guide — first-party package conventions. Node Implementation Examples (Python) — annotated Python node examples. Node Patterns — TypeScript design patterns: multi-output, streaming, stateful, media refs, secrets. Node Reference — TypeScript @prop / process() / genProcess() templates. TypeScript DSL Guide — use your nodes in code-defined workflows. Suspendable Nodes — why there is no pause/resume node API.--- # developer-dsl-guide.md Source: https://docs.nodetool.ai/developer/dsl-guide This page has moved. Please go to TypeScript DSL Guide.--- # Node Graph Editor Performance Audit Source: https://docs.nodetool.ai/developer/node-editor-performance-audit.html Node Graph Editor Performance Audit Audited: 2026-07-02. Sources verified against web/src at commit 8ff1421. ReactFlow (@xyflow/react) 12.10.2, React 19, Zustand 5. Scope: the canvas render path — ReactFlowWrapper, BaseNode and everything rendered per node, the Zustand stores the editor subscribes to, edge/handle/ pointer interaction paths, and the editor CSS. All 27 existing performance guard tests pass; the issues below live in paths those tests do not cover. The costs concentrate on three hot paths: Node drag / pan — per-frame work that is O(all nodes), not O(moved nodes). Workflow execution — per-websocket-message work that sweeps the whole canvas, including other tabs’ runs. Paint — CSS filter chains on every node (and, in light mode, every edge and handle) that force filter-pipeline rasterization during drags. High severity H1. onNodesChange clones every node on every change batch NodeStore.ts:58-65, 504-508: const rawNodes = applyNodeChanges(filteredChanges, currentNodes); const nodes = syncAllReactFlowNodeChromeClass(rawNodes ?? currentNodes); // = nodes.map((node) => ({ ...node, className: reactFlowNodeChromeClassName(node.data) })) applyNodeChanges preserves object identity for untouched nodes so ReactFlow can skip them; the unconditional .map() + spread then gives every node a fresh identity on every change batch — every drag pointermove frame, every selection click, every dimension change. Verified against the installed @xyflow/system (adoptUserNodes): RF12 skips re-adoption only when userNode === internals.userNode, so every frame re-runs position clamping, handle parsing, and z-calculation for all N nodes, and every NodeWrapper re-renders (the inner BaseNode is saved only by its memo comparator). It also poisons every downstream cache keyed on node identity. Fix (small): only clone when the class actually changed, and return the original array when nothing did: const syncReactFlowNodeChromeClass = (node) => { const cls = reactFlowNodeChromeClassName(node.data); return cls === node.className ? node : { ...node, className: cls }; }; bypassed/collapsed only change in toggleBypass/updateNodeData, which already set className on the touched node — the hot path can be fully identity-preserving. H2. useViewport() re-renders the whole canvas wrapper every pan/zoom frame ReactFlowWrapper.tsx:362, consumed only at :718-733 to derive a zoomed-out class. useViewport fires on every transform change (x, y, and zoom — every pointermove of a pan). The wrapper is the heaviest component in the editor (~10 store subscriptions, ~15 memos, a ~50-prop <ReactFlow> element) and re-runs per frame to compute a boolean that only changes when zoom crosses ZOOMED_OUT. Fix: useStore((s) => s.transform[2] <= ZOOMED_OUT) — re-renders only on threshold crossings. Same fix for ViewportStatusIndicator.tsx:39 (full MUI toolbar re-render per pan frame; it only needs zoom) and NodeInfoPanel.tsx:201 (re-renders per frame even with nothing inspected). AxisMarker already does this right (useOnViewportChange + ref writes). H3. The memoized node chrome components are dead code — every consumer imports the unmemoized export NodeHeader.tsx, NodeErrors.tsx, NodeInputs.tsx, NodeOutputs.tsx each export a memo(..., isEqual) default export (NodeHeader.tsx:394, NodeErrors.tsx:223, NodeInputs.tsx:321, NodeOutputs.tsx:102) — and every canvas consumer (BaseNode.tsx:30-31, NodeContent.tsx:3-4, all bespoke node types) imports the named, unmemoized component. Verified by grep: the memo wrappers have zero canvas imports. BaseNode re-renders on every streaming chunk, status flip, and hover enter/leave; each re-render re-runs NodeHeader (~350 lines, a large emotion css object, an always-mounted logs dialog) and NodeErrors (nodeErrorToDisplayString runs twice per render) for every affected node. Fix: memoize the named exports themselves (or switch imports to the defaults), and give NodeHeader a comparator narrower than deep-equal on the whole data blob — it only reads data.bypassed. H4. findNode is O(N), and per-node selectors call it on every store write → O(N²) per drag frame NodeStore.ts:724-725 (get().nodes.find(...)). Callers that run per node on every NodeStore update: useNodeGenerations.ts:40-47 — two subscriptions, mounted by every BaseNode via useNodeArtifacts (BaseNode.tsx:534), plus PreviewNode. NodeOutputs.tsx:58-61, PreviewNode.tsx:268-282. NodeOutput.tsx connect-drag fallback (per output handle). Zustand runs all listeners on every set(); dragging updates nodes at ~60fps. At 300 nodes that is ~270k id comparisons per frame in selector work alone, before React renders anything. Same cost per property keystroke. Fix: maintain a nodesById index in NodeStore (updated wherever nodes is set) and make findNode O(1) — one change fixes all call sites. For useNodeGenerations specifically, selected_generation(s) are already on the data prop RF passes to the node — thread them in as parameters. H5. The canvas subscribes to the global status maps — every execution message re-renders it ReactFlowWrapper.tsx:589-590: const edgeStatuses = useResultsStore((state) => state.edges); const nodeStatuses = useStatusStore((state) => state.statuses); Both maps are rebuilt with a full spread on every write (StatusStore.ts:121, ResultsStore.ts:335-341) and are keyed globally (workflowId:jobId:nodeId). Every node_update/edge_update from any workflow and any job — including runs in other tabs — re-renders the wrapper and re-runs the useProcessedEdges status pass, an O(E) map that produces fresh edge objects for active edges each time (useProcessedEdges.ts: 387-424), so RF re-diffs edges per message while anything is running. Fix: key the stores per run (statusesByJob[jobKey][nodeId]) so the canvas selects only its focused run’s sub-map (identity stable when other jobs write), and make the status pass return prior edge objects when that edge’s status/counter did not change. H6. CSS filter chains on every node — and on every edge and handle in light mode styles/nodes.states.css:6-27: every node carries a filter: drop-shadow(...) at all times; selected nodes get two stacked drop-shadows (one with 24px blur) on top of the box-shadow the node body already draws (selectionStyles.ts:97-101). Since clicking a node selects it, every drag is a selected-node drag: the filter region re-rasterizes per frame. nodes.base.css also transitions filter on all nodes. The code already knows filters are costly (CRISP_NO_BLUR_STYLES sets filter: none on the inner body) — the outer rule re-adds the cost. styles/handle_edge_tooltip.css:310-317: light mode applies filter: saturate(...) brightness(...) to every handle and every edge path, permanently. Filters on SVG paths disable cheap stroke repaint; node drags repaint every attached edge through the filter pipeline. CustomEdge.tsx:88-101: selected edges get an inline SVG drop-shadow while their path changes every drag frame (node-driven edge selection makes this the common case). handle_edge_tooltip.css:656-665: the run-time edgeFlow dash animation adds filter: drop-shadow(...) per animated edge and will-change: stroke-dashoffset (buys nothing — dash offset is main-thread paint), running for offscreen edges too. Fix: replace node drop-shadows with the existing body box-shadow; bake the light-mode desaturation into the palette instead of filtering at paint time; approximate the selected-edge glow with a second wider translucent stroke; drop filter/will-change from the animated edge rule. Medium severity M1. Streaming amplification: per-token store writes, whole-node re-renders, O(k²) accumulation ResultsStore.addChunk (:738-749) spreads the full chunks map per token; setOutputResult with append (:606-654) copies the whole accumulated array per item — O(k²) over a k-item stream. Audio already has coalescing (200ms flush, workflowUpdates.ts:49-93); text and other streams do not. BaseNode.tsx:534 subscribes to the accumulated chunk string at the top of the node, so each token re-renders the entire node (amplified by H3), and ChunkDisplay.tsx:78 re-parses the full markdown per token — O(n²) per stream. NodeLogsDialog is mounted inside every NodeHeader (NodeHeader.tsx:384-389) and its memos re-slice/join the full log history on every appended line even while closed (NodeLogs.tsx:64-91); NodeHeader itself subscribes to the log array just to show a count. workflowUpdates.ts:587-599 JSON.stringifys every non-string output into a log line — a multi-MB image ref serializes before the 20k-char truncation runs. Fix: reuse the audio coalescing for text chunks/appends; move chunk/terminal subscriptions into leaf components (NodeProgress already models this); mount the logs dialog only when open and select logsByNode[key]?.length for the count; summarize rich types before stringifying. M2. nodeTypes identity is tied to MetadataStore — map replacement remounts all nodes ReactFlowWrapper.tsx:399-422 memoizes on useMetadataStore((s) => s.nodeTypes), whose identity changes on every setNodeTypes/addNodeType. Loading a workflow with unknown types fires one addNodeType per type (NodeStore.ts:257-261); each replaces the map, and a changed nodeTypes identity makes ReactFlow remount every node — full DOM teardown and handle re-measure, possibly several times in a row. Related: NodeStore’s metadata subscriber (NodeStore.ts:267-281) re-runs sanitizeGraph and replaces the edges array unconditionally on every MetadataStore write (including fal/kie pricing merges), invalidating the expensive structural edge pass in every open tab. Fix: unknown types already fall through to default: PlaceholderNode, so the per-type placeholder registration can likely be dropped; otherwise batch into one store write and bail when keys are unchanged. Make the sanitize subscriber a no-op when the sanitized edges are unchanged. M3. Every Ctrl/Meta press re-renders every property editor on the canvas PropertyField.tsx:94-99 subscribes each field to isKeyPressed("Control")/isKeyPressed("Meta") and threads the result into PropertyInput’s memo comparator. Every modifier press and release (copy/paste, shortcuts, Cmd-click) re-renders hundreds of property editors. The only consumer is a mouse handler that already receives event.ctrlKey/event.metaKey (PropertyInput.tsx:392). Fix: delete both subscriptions; read the event modifiers (or useKeyPressedStore.getState() inside the handler). M4. Per-frame O(N) bookkeeping in the wrapper, doubled by a redundant second store write The layout-fingerprint effect (ReactFlowWrapper.tsx:295-359) rebuilds a Map of signature strings for all nodes on every nodes change — every drag frame. nodesStructureKey (useProcessedEdges.ts:76-94) joins an O(N) string per frame (multi-KB allocation on large graphs). setWorkflowDirty(true) is called unconditionally after graph mutations (NodeStore.ts:511-513 and ~15 other sites) with no change guard, so every drag frame performs two set() calls — every mounted selector in the graph runs twice per frame. Fix: guard setWorkflowDirty on actual change (one line); skip the fingerprint/structure-key passes while a drag is in progress (the isSelecting trick already exists for marquee drags), or key them on a structural version counter bumped by the store. M5. updateNodeInternals over all nodes on edge changes, six refresh passes per layout change ReactFlowWrapper.tsx:276-290 refreshes handle bounds for all node ids on every edge add/remove — O(N × handles) getBoundingClientRect calls for a change touching two nodes. scheduleNodeInternalsRefresh.ts then runs each refresh six times over ~160ms (raf, raf, timeout 0/24/72/160). Fix: refresh only the affected edge’s endpoints (the withEdgeNeighborNodeIds helper already exists for this); trim the scheduler to raf-raf plus one late fallback. M6. Connection-drag costs: per-pointermove BFS and a whole-graph handle sweep isValidConnection (useConnectionEvents.ts:30-40 → graphCycle.ts) rebuilds a full adjacency map over all edges and BFSes per call, and RF calls it from pointermove whenever the wire is near a handle. Grabbing a wire re-renders every handle in the graph (they subscribe to ConnectionStore connect state); each handle re-runs hashType (typeFilterUtils.ts:6-13 — recursive string building, uncached) and NodeOutput falls back to O(N) findNode per output handle. Cost lands twice per wire drag (grab and release) — a visible hitch at 100+ nodes. Fix: precompute reachable-set once in onConnectStart (O(1) checks per move); cache hashType per TypeMetadata in a WeakMap; both shrink further once H4’s nodesById index exists. M7. Undo history: full-graph equality scan per tracked write, one entry per keystroke, and phantom entries from runtime echoes zundo is configured with equality: customEquality (NodeStore.ts:1423-1431), which iterates every node with three shallow() compares each (customEquality.ts:31-63) — on every tracked set(). Property typing is undebounced (updateNodeProperties per keystroke), so each keystroke pays the scan and pushes a history entry (undo steps back one character). Drags are correctly pause()/resume()d. Worse: node_update websocket echoes call updateNodeData while tracking is active (workflowUpdates.ts:1146-1173) — running a workflow creates undo entries the user never authored and marks the workflow dirty (autosave version churn). Related: BaseNode’s overlay re-measure (BaseNode.tsx:767-778) fires updateNode per node at run completion, each dirtying the workflow. Fix: pause temporal (and skip the dirty flag) around runtime echoes and re-measures; group/debounce property-edit history. M8. onlyRenderVisibleElements={false} — every cost above scales with total nodes, not visible nodes ReactFlowWrapper.tsx:770. All nodes, property editors, handles, and tooltips stay mounted regardless of viewport; collapsed nodes also keep their full subtree mounted (CSS-only collapse via display: none). RF12’s visibility culling is much cheaper than v11’s. Worth a benchmark, at least above a node-count threshold, and worth skipping field-editor rendering for collapsed nodes (handles must stay mounted for edge anchoring — HandleColumn/ HandleOnlyField already provide that). Low severity (grouped) Stale-edge correctness bug in PreviewNode: PreviewNode.tsx:237-243 memoizes the incoming edge on [getEdges, props.id] — both stable — so it computes once at mount and never updates when the preview is re-wired. Also uses fast-deep-equal as a subscriber equality over potentially huge property values (:268-282); property values are replaced immutably, so reference equality suffices. FindInWorkflowDialog subscribes to nodes while closed (useFindInWorkflow.ts:85) — re-renders per drag frame to return null. Latent footgun: useEdgeHandlers.ts:77-121 (onEdgeMouseEnter/Leave) mutates store edges in place and would trigger a full O(E) structural reprocess per hover — currently unwired; delete or rewrite before wiring. Every mounted OutputRenderer keeps a permanent document-level mousemove listener (OutputRenderer.tsx:325-336); attach on mousedown instead. transition: all on every handle (handle_edge_tooltip.css:298) plus layout-affecting width/height transitions when is-connecting flips — violates the repo’s own MOTION rules; list the animated properties. :has() selectors in per-node CSS (nodes.states.css:30, nodes.zoomed.css) make style invalidation walk node subtrees during runs; set a class from JS instead. getNodeCategoryColor rebuilds its color table per node per minimap frame (ColorUtils.ts:151-165); hoist to module scope. panOnDrag array recreated per render (useDragHandlers.ts:291-294); whole-settings subscription in the same hook; inline 30-line onNodeDoubleClick arrow (ReactFlowWrapper.tsx:823-851). NodeEditor subscribes to the whole upload store (NodeEditor.tsx:65) — re-renders per upload progress tick. Marquee-end edge hit-testing does one document.querySelector per edge (useSelectionEvents.ts:139-156); use one querySelectorAll. PropertyInput.tsx:228 subscribes to the entire metadata map for a reset-to-default handler; read getState() in the callback. KieCreditsFooter subscribes every node to two hot stores before its “not a KIE node” early return (KieCreditsFooter.tsx:96). useAutoEnableNodePacks allocates an O(N) types array on every store update (useAutoEnableNodePacks.ts:31-33); cache by nodes identity. What is already done well Preserve these patterns when fixing the above: useNodes defaults to shallow equality (NodeContext.tsx:64); useShallow is used consistently for multi-value selections; no whole-store no-selector subscriptions in the canvas path. Per-node execution state is correctly isolated: status/progress/error/ duration lookups are O(1) keyed maps returning stable references — a progress tick on node A does not re-render node B, and NodeProgress is a leaf subscriber mounted only while running. useProcessedEdges two-stage split (heavy structural pass keyed by a structure fingerprint; light status pass with whole-result identity bail), O(1) lookup maps, and the incomingByTargetHandle index that removed a prior O(E²). Identity-cached selectors: selectedNodeIdsSelector (ReactFlowWrapper.tsx:629-651), getSelectedNodeCount/getSelection closures, BaseNode’s hasConnectedInput/hasControlEdge, GroupNode’s children-status selector. History is paused during drags, slider scrubs, and group operations; partialize keeps viewport/hover out of history; selection changes don’t create undo entries. Audio streaming is fully coalesced (200ms batches, single-copy append, rolling caps, React-free worklet fast path) — the template for text streams. Connection drag writes no store state per pointermove; the connectability matrix is precomputed at metadata load; EdgeGradientDefinitions renders one gradient per active type pair with stabilized keys; CustomEdge/ControlEdge have explicit memo comparators. AxisMarker and GhostNode handle per-frame updates via refs/rAF with zero React re-renders; viewport is persisted only on onMoveEnd; drag intersection checks are throttled to 50ms. LogStore is pre-keyed by node with caps and truncation; autosave is interval-based, not per-change; no structuredClone/JSON round-trips on hot paths. Suggested fix order # Fix Effort Pays off on 1 H1: identity-preserving syncReactFlowNodeChromeClass tiny every drag frame 2 H2: boolean zoom selector instead of useViewport (3 components) tiny every pan/zoom frame 3 H3: use the memoized node chrome exports; narrow NodeHeader comparator small every streaming chunk/hover 4 M3: drop Ctrl/Meta subscriptions in PropertyField tiny every shortcut press 5 M4: setWorkflowDirty change guard tiny every graph mutation 6 H4: nodesById index for findNode small drag frames, keystrokes, connect drags 7 H5: per-run keying for status/results maps medium execution messages 8 H6: remove node/edge/handle CSS filters small paint during drag, light mode 9 M1: coalesce text-stream writes; leaf-subscribe chunks; lazy logs dialog medium streaming runs 10 M2: stabilize nodeTypes; conditional sanitize small workflow load 11 M8: benchmark onlyRenderVisibleElements on large graphs experiment large graphs Test coverage gaps The existing suites (web/src/__tests__/performance/) guard LogStore bucket identity, selection-count memoization, and selector patterns — none cover the top findings. Worth adding once fixed: untouched node objects keep identity through onNodesChange (locks in H1); ReactFlowWrapper does not re-render on another job’s setStatus (H5); a streamed chunk re-renders the chunk display leaf, not BaseNode (M1); modifier keypress does not re-render PropertyInput (M3). The pattern-level assertions in componentPerformance.test.tsx are gated behind PERF_TESTS=true and do not enforce in CI.--- # Node Implementation Examples Source: https://docs.nodetool.ai/developer/node-examples.html Real Examples from Codebase This document provides annotated examples of Python nodes. Each example highlights specific patterns and implementation details to help you build your own nodes. To write a TypeScript node, use the Custom Nodes Guide, Node Patterns, and Node Reference instead. The Python base classes (BaseNode, HuggingFacePipelineNode, ProcessingContext) live in the separate nodetool-core and nodetool-huggingface repositories, which this repository does not contain. This repository runs those nodes through the Python bridge: it reads the metadata the worker reports (is_dynamic, is_streaming_output, is_streaming_input, required_settings, properties, outputs) and sends execute and execute.stream calls. The Python API shown below is not checked against source here. 1. Simple Processing Nodes These nodes take one or more inputs, perform a calculation, and return a single result. They are the building blocks of most workflows. Example: Text Concatenation Pattern: BaseNode with simple inputs and a single return value. class Concat(BaseNode): """ Concatenates two text inputs into a single output. text, concatenation, combine, + """ # Define inputs using Pydantic fields a: str = Field(default="") b: str = Field(default="") @classmethod def get_title(cls): return "Concatenate Text" # The return type annotation (str) tells the UI what this node outputs async def process(self, context: ProcessingContext) -> str: return self.a + self.b Key Takeaways: Use Field(default="") to define inputs. The process method must be async. The return type annotation is crucial for the UI to validate connections. 2. Structured Output Nodes Sometimes a node needs to return multiple values (e.g., an audio transcription that returns both the text and the detected language). Example: Automatic Speech Recognition Pattern: Using TypedDict to define multiple named outputs. class AutomaticSpeechRecognition(BaseNode): """ Automatic speech recognition node. audio, speech, recognition """ # Define the output structure class OutputType(TypedDict): text: str language: str model: ASRModel = Field(...) audio: AudioRef = Field(...) async def process(self, context: ProcessingContext) -> OutputType: # ... logic to transcribe audio ... return { "text": "Hello world", "language": "en" } Key Takeaways: Define a TypedDict named OutputType inside your class. Set the return annotation of process to OutputType. Return a dictionary matching the structure. 3. Streaming Nodes For operations that take a long time or produce results incrementally (like reading a large folder of images), use a generator. Example: Load Image Folder Pattern: gen_process with AsyncGenerator. class LoadImageFolder(BaseNode): """ Load all images from a folder. image, load, folder """ folder: str = Field(default="") class OutputType(TypedDict): image: ImageRef path: str # Use gen_process instead of process async def gen_process( self, context: ProcessingContext ) -> AsyncGenerator[OutputType, None]: # Iterate over files and yield results one by one for path in self.iter_files(self.folder): image = await context.image_from_bytes(...) yield {"image": image, "path": path} Key Takeaways: Use gen_process instead of process. Return type is AsyncGenerator[OutputType, None]. yield results as they become available. This allows downstream nodes to start processing immediately. 4. Dynamic Nodes Some nodes need to adapt their inputs based on user configuration. For example, a template node might need different inputs depending on the variables in the template string. Example: Format Text (Jinja2) Pattern: _is_dynamic flag and _dynamic_properties. {% raw %} class FormatText(BaseNode): """ Replaces placeholders in a string with dynamic inputs. """ # 1. Flag this node as dynamic _is_dynamic: ClassVar[bool] = True template: str = Field(default="Hello {{ name }}!") async def process(self, context: ProcessingContext) -> str: # 2. Access the dynamic inputs provided by the user # These are inputs that were added to the node at runtime dynamic_inputs = self.get_dynamic_properties() # Render the template using these inputs return self.render_template(self.template, **dynamic_inputs) {% endraw %} Key Takeaways: Set _is_dynamic = True. The UI will allow users to add arbitrary inputs to this node. Access these inputs via self.get_dynamic_properties() or self._dynamic_properties. 5. Working with Assets Nodes often need to load or save heavy assets like images or audio. Example: Save Text to File Pattern: Using ProcessingContext to create assets. class SaveText(BaseNode): """ Saves input text to a file. """ text: str = Field(default="") filename: str = Field(default="output.txt") async def process(self, context: ProcessingContext) -> TextRef: # 1. Create the asset using the context asset = await context.create_asset( filename=self.filename, mime_type="text/plain", data=self.text.encode("utf-8") ) # 2. Create a reference to return result = TextRef(uri=asset.uri, asset_id=asset.id) # 3. Notify the UI that a file was saved (optional but recommended) context.post_message(SaveUpdate( node_id=self.id, name=self.filename, value=result, output_type="text" )) return result Key Takeaways: Never write directly to disk if you can avoid it. Use context.create_asset. Return a Ref object (like TextRef, ImageRef) so other nodes can use the asset. Use SaveUpdate to show the saved file in the UI’s “Outputs” tab. 6. Control Flow Nodes can control the execution flow of the graph. Example: If Node Pattern: Conditional logic in gen_process. class If(BaseNode): """ Conditionally executes branches. """ condition: bool = Field(default=False) value: Any = Field(default=None) class OutputType(TypedDict): if_true: Any if_false: Any async def gen_process(self, context: Any) -> AsyncGenerator[OutputType, None]: if self.condition: # Only emit to the 'if_true' output yield {"if_true": self.value, "if_false": None} else: # Only emit to the 'if_false' output yield {"if_true": None, "if_false": self.value} Key Takeaways: Control flow nodes often use gen_process to conditionally yield results. The TypeScript nodetool.control.If returns only the taken branch’s key. A key that is absent sends no message, so nodes wired only to the untaken branch do not run. In TypeScript, null is a value and does send a message. Check how the Python worker treats None before relying on it to stop a branch. 7. Model Nodes (HuggingFace / model-backed) Nodes that load a machine-learning model (HuggingFace, diffusers, …) extend HuggingFacePipelineNode — a BaseNode subclass with extra hooks for loading, device placement, and surfacing downloadable checkpoints. This is the pattern the nodetool-huggingface package uses for every node. Example: A text-to-audio model node class MyAudioGen(HuggingFacePipelineNode): """ One-line description shown in the UI. audio, generation, text-to-audio """ model: HFTextToAudio = Field( default=HFTextToAudio(repo_id="org/my-model"), description="The model to use.", ) prompt: str = Field(default="", description="Text prompt.") seed: int = Field(default=-1, ge=-1) _pipeline: Any = None @classmethod def get_title(cls) -> str: return "My Audio Gen" @classmethod def get_basic_fields(cls) -> list[str]: # Fields rendered on the node body; the rest live in the expanded view. return ["model", "prompt"] @classmethod def get_recommended_models(cls) -> list[HuggingFaceModel]: # Checkpoints offered for one-click download in the Model Manager. return [ HFTextToAudio( repo_id="org/my-model", allow_patterns=["**/*.safetensors", "**/*.json", "**/*.txt"], ), ] def get_model_id(self) -> str: return self.model.repo_id async def preload_model(self, context: ProcessingContext): # Called once before process(); load the model here. from diffusers import MyPipeline self._pipeline = await self.load_model( context=context, model_class=MyPipeline, model_id=self.get_model_id(), torch_dtype=available_torch_dtype(), device="cpu", # the runner moves it to the GPU/MPS via move_to_device ) async def move_to_device(self, device: str): if self._pipeline is not None: self._pipeline.to(device) async def process(self, context: ProcessingContext) -> AudioRef: # run_pipeline_in_thread keeps the asyncio event loop responsive. output = await self.run_pipeline_in_thread(prompt=self.prompt) return await context.audio_from_numpy(output.audios[0], 24000) Key Takeaways: Extend HuggingFacePipelineNode and declare a model: field of the matching HF type (HFTextToImage, HFTextToVideo, HFTextToAudio, HFTextGeneration, …). get_recommended_models() lists checkpoints the Model Manager can download; get_basic_fields() selects which fields show on the node body; get_title() sets the display name. preload_model() loads the model, move_to_device() places it on the accelerator, and process() (or gen_process()) runs inference. Use run_pipeline_in_thread() so long-running inference doesn’t block the event loop. To make a model usable outside the node graph (e.g. via an agent or the generation API), implement the matching method on the package’s local provider — e.g. text_to_audio() for the TEXT_TO_AUDIO capability.--- # Node Implementation Patterns Source: https://docs.nodetool.ai/developer/node-patterns.html Overview This guide covers the key implementation patterns you will encounter when building custom nodes for NodeTool. Every node extends BaseNode from @nodetool-ai/node-sdk, declares its inputs with the @prop decorator, and implements a process() or genProcess() method that reads input values from this.<field> and returns an outputs record. The optional argument is a ProcessingContext from @nodetool-ai/runtime. Simple Single-Output The most common pattern. The node declares one or more @prop inputs and returns a single keyed output. Use metadataOutputTypes to tell the UI the output’s type. This pattern is adapted from ConstantStringNode in packages/core-nodes/src/nodes/constant.ts: import { BaseNode, prop } from "@nodetool-ai/node-sdk"; export class ConstantStringNode extends BaseNode { static readonly nodeType = "nodetool.constant.String"; static readonly title = "String"; static readonly description = "Represents a string constant in the workflow.\n text, string, characters"; static readonly metadataOutputTypes = { output: "str", }; @prop({ type: "str", default: "", title: "Value" }) declare value: string; async process(): Promise<Record<string, unknown>> { return { output: this.value ?? "" }; } } Key points: metadataOutputTypes maps each output key to its type string ("str", "int", "float", "bool", "image", "audio", etc.). The engine assigns connected input values onto the instance before calling process(), so read them as this.field. The return value is a plain object whose keys match the declared output names. Type the field with its real type (string, number, a ref type) instead of any. The runtime assigns the value, so declare carries only the compile-time type. Multi-Output When a node produces more than one output, list every key in metadataOutputTypes. Each key becomes a separate output connector in the UI. This pattern is adapted from IfNode in packages/core-nodes/src/nodes/control.ts. It reads its inputs as a buffered set (inputMode = "buffered") and routes the value input to whichever branch matches the condition. The outputCorrelation map declares that both outputs forward the value input — so each output carries the same correlation lineage as its source, which is how the scheduler knows the branch is a passthrough rather than a new value: import type { InputMode, OutputCorrelation } from "@nodetool-ai/protocol"; export class IfNode extends BaseNode { static readonly nodeType = "nodetool.control.If"; static readonly title = "If"; static readonly description = "Conditionally executes one of two branches based on a condition.\n" + " control, flow, condition, logic"; static readonly metadataOutputTypes = { if_true: "any", if_false: "any", }; static readonly inputMode: InputMode = "buffered"; static readonly outputCorrelation: Record<string, OutputCorrelation> = { if_true: { kind: "forward", source: "value" }, if_false: { kind: "forward", source: "value" }, }; @prop({ type: "bool", default: false, title: "Condition" }) declare condition: boolean; @prop({ type: "any", default: [], title: "Value" }) declare value: unknown; async process(): Promise<Record<string, unknown>> { const condition = Boolean(this.condition ?? false); const value = this.value ?? null; // Return only the taken branch. A missing key sends no message on that // output, so the untaken branch's downstream nodes are skipped. if (condition) { return { if_true: value }; } return { if_false: value }; } } Key points: Every key you return must be declared in metadataOutputTypes. Declared outputs you leave out of the returned object (or set to undefined) send nothing, and nodes wired only to them do not run. Returning null for an output does send a message. inputMode is one of "buffered" (collect a matched set of inputs, then call process() once), "stream" (consume inputs as an async stream via run()), or "controlled" (the node is driven by control edges). Most nodes leave it undefined (the default buffered behavior). outputCorrelation maps each output to how it relates to its source input. A { kind: "forward", source: "<input>" } entry means the output forwards that input’s correlation token unchanged. There is no separate isStreamingOutput flag, and streaming behavior is inferred from forward, iteration, or chunk correlation, or from overriding genProcess(). Once you declare it, every declared output needs an entry. See Custom Nodes Guide §11. Streaming with genProcess() For nodes that emit multiple results over time, implement genProcess() as an async generator. The engine calls genProcess() instead of process() when it is defined. This pattern is adapted from ForEachNode in packages/core-nodes/src/nodes/control.ts (the real node also has a limit property). It buffers its input list, then emits each element as a separate iteration — outputCorrelation declares the outputs as kind: "iteration", which mints a fresh correlation token per emitted item so downstream nodes treat each one as a distinct logical value: import type { InputMode, OutputCorrelation } from "@nodetool-ai/protocol"; export class ForEachNode extends BaseNode { static readonly nodeType = "nodetool.control.ForEach"; static readonly title = "For Each"; static readonly description = "Iterate over a list and emit each item sequentially.\n" + " iterator, loop, list, sequence"; static readonly metadataOutputTypes = { output: "any", index: "int", }; static readonly inputMode: InputMode = "buffered"; static readonly outputCorrelation: Record<string, OutputCorrelation> = { output: { kind: "iteration", source: "__execution__", group: "items" }, index: { kind: "iteration", source: "__execution__", group: "items" }, }; @prop({ type: "list[any]", default: [], title: "Input List" }) declare input_list: unknown[]; async process(): Promise<Record<string, unknown>> { return {}; } async *genProcess(): AsyncGenerator<Record<string, unknown>> { const values = this.input_list ?? []; const list = Array.isArray(values) ? values : [values]; for (const [index, item] of list.entries()) { yield { output: item, index }; } } } Key points: You must still provide a process() stub (it can return {}) — the base class requires it. Each yield sends one batch of outputs to downstream nodes. Yield objects. A value returned from the generator is discarded. Declared iteration outputs that share a group are emitted together, so yield one object carrying all of them, as above. Nodes that consume a stream implement run(inputs, outputs) instead. See Custom Nodes Guide §11. genProcess() is detected automatically (the base class checks whether the subclass overrides it) — there is no isStreamingOutput flag to set. The outputCorrelation iteration kind tells the scheduler each yielded value is a new correlated item. Stateful Collector Some nodes accumulate values across multiple invocations within a single workflow run. Hold the state in a private instance field and use initialize() to reset it at the start of each run. This pattern is adapted from CollectTextNode in packages/text-nodes/src/nodes/text-extra.ts: export class CollectTextNode extends BaseNode { static readonly nodeType = "nodetool.text.Collect"; static readonly title = "Collect Text"; static readonly description = "Collects streaming text inputs into a single concatenated string.\n" + " text, collect, stream, aggregate"; static readonly metadataOutputTypes = { output: "str", }; private _items: string[] = []; @prop({ type: "str", default: "", title: "Input Item" }) declare input_item: string; @prop({ type: "str", default: "", title: "Separator" }) declare separator: string; async initialize(): Promise<void> { this._items = []; } async process(): Promise<Record<string, unknown>> { this._items.push(this.input_item); return { output: this._items.join(this.separator) }; } } Key points: Private instance fields (like _items) hold state between invocations within a single run. initialize() runs once per node before the workflow run starts – use it to clear accumulated state. finalize() runs when the node finishes, even after an error. The node instance is reused across the items it receives, so process() is called once per incoming value and appends to _items. Media Refs Images, audio, video, and 3D models are passed between nodes as ref objects: plain objects with a type discriminator ("image", "audio", "video", "model_3d") plus uri, data, and metadata fields. Nodes load bytes from the ref and return new refs after processing. Read input bytes with loadMediaRefBytes(ref, context) from @nodetool-ai/runtime. It resolves inline data, asset:// references, storage-backed URIs, and http(s) URLs (through the media egress policy), so a node that reads ref.data or calls fetch(ref.uri) itself silently drops media supplied as an asset. It returns null when it finds no bytes. Build output refs with a helper. imageRefFromBytes(bytes) from @nodetool-ai/runtime sets type, raw base64 data, the MIME type, and the dimensions when sharp is present. import sharp from "sharp"; import type { ImageRef } from "@nodetool-ai/node-sdk"; import type { ProcessingContext } from "@nodetool-ai/runtime"; import { imageRefFromBytes, loadMediaRefBytes } from "@nodetool-ai/runtime"; export class ResizeNode extends BaseNode { // ... @prop({ type: "image", default: { type: "image", uri: "", data: null }, title: "Image" }) declare image: ImageRef; @prop({ type: "int", default: 512, title: "Width", min: 1, max: 4096 }) declare width: number; @prop({ type: "int", default: 512, title: "Height", min: 1, max: 4096 }) declare height: number; async process(context?: ProcessingContext): Promise<Record<string, unknown>> { const bytes = await loadMediaRefBytes(this.image, context); if (!bytes) throw new Error("Resize: no image data"); const outputBytes = await sharp(bytes).resize(this.width, this.height).toBuffer(); return { output: await imageRefFromBytes(outputBytes) }; } } Rules for refs (the repository’s packages/AGENTS.md lists more): data is raw base64 or a Uint8Array, never a data: URI. Put the MIME type in a field such as mimeType. Always include type on a ref you emit. A bare Uint8Array output is an untyped value. Check data.length > 0 instead of truthiness when choosing between data and uri. Audio and video have equivalent helpers in their packages, such as audioBytesAsync in packages/audio-nodes/src/nodes/audio.ts. Enum Inputs Use @prop with type: "enum" and a values array to present a dropdown in the UI. This pattern is adapted from FilterStringNode in packages/text-nodes/src/nodes/text-extra.ts, which offers six filter types: @prop({ type: "enum", default: "contains", title: "Filter Type", description: "The type of filter to apply", values: ["contains", "starts_with", "ends_with"], }) declare filter_type: "contains" | "starts_with" | "ends_with"; The value arrives unchecked, so validate it before relying on it. FilterStringNode guards the prop with a type predicate and matches nothing when the value is not one it implements. Secret Access Nodes that call external APIs declare the keys they need in requiredSettings. Before calling process(), the base class resolves each key through ProcessingContext.getSecret() and exposes them as this._secrets. A key that does not resolve is logged as a warning and is absent from this._secrets. This pattern is adapted from the OpenAI nodes in packages/llm-nodes/src/nodes/openai.ts (e.g. openai.text.Embedding, whose output type is "list"): export class EmbeddingNode extends BaseNode { static readonly nodeType = "openai.text.Embedding"; static readonly title = "Embedding"; static readonly description = "Generate vector representations of text."; static readonly requiredSettings = ["OPENAI_API_KEY"]; @prop({ type: "str", default: "", title: "Input" }) declare input: string; static readonly metadataOutputTypes = { output: "list[float]" }; async process(): Promise<Record<string, unknown>> { const apiKey = this._secrets.OPENAI_API_KEY; if (!apiKey) throw new Error("OPENAI_API_KEY is not configured"); const response = await fetch("https://api.openai.com/v1/embeddings", { method: "POST", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json", }, body: JSON.stringify({ input: this.input, model: "text-embedding-3-small" }), }); const data = await response.json(); return { output: data.data[0].embedding }; } } Key points: requiredSettings is a static string array of secret key names. The base class resolves each key from the active ProcessingContext and assigns them to this._secrets before process() runs. The server’s secret resolver checks the user’s secret store first and then the process environment, so a node does not need its own process.env fallback. Inside run(), this._secrets is populated the same way. Outbound requests to a URL a caller or model chose belong behind safeFetch from @nodetool-ai/runtime. A fixed provider endpoint like the one above does not. Best Practices Input Resolution Property values for @prop fields are populated on the instance before process() is called. Resolve with: const value = this.field ?? defaultValue; The engine handles connected edges, manually set values, and declared defaults — by the time process() runs, this.field already holds the effective value. Async / Await All process() and genProcess() methods are async. Use await for any I/O – file reads, HTTP calls, or image processing – to keep the workflow engine responsive. metadataOutputTypes Always declare metadataOutputTypes so the UI can render the correct output connectors and validate connections between nodes. If you omit it (and the static outputTypes fallback), the node has no outputs. Error Handling Let exceptions propagate or throw Error with a clear message. The engine catches them and displays the error on the node in the UI. Docstring Format The description static field serves double duty: the first line is the summary other tools show, and the node search indexes the whole text, so subsequent indented lines work as keywords. static readonly description = "Short description of what the node does.\n" + " keyword1, keyword2, keyword3"; File Structure Nodes are organized by category across the node packages — packages/core-nodes/ (constants, control flow, math), packages/text-nodes/, packages/image-nodes/, packages/llm-nodes/, packages/automation-nodes/, packages/integration-nodes/, and others. (The old packages/base-nodes/src/nodes/ directory is gone — packages/base-nodes/src/index.ts only re-exports and aggregates, into ALL_BASE_NODES.) Each file exports an array of node classes (e.g., CONTROL_NODES, TEXT_EXTRA_NODES) and the package index registers them all. A class is only available once something registers it with a NodeRegistry. Third-party packs do that in their register function. See the Custom Nodes Guide.--- # Node Implementation Quick Reference Source: https://docs.nodetool.ai/developer/node-reference.html Essential Node Templates import { BaseNode, prop } from "@nodetool-ai/node-sdk"; import type { InputMode, OutputCorrelation } from "@nodetool-ai/protocol"; // SIMPLE PROCESSING NODE export class SimpleNode extends BaseNode { static readonly nodeType = "mypackage.example.Simple"; static readonly title = "Simple Node"; static readonly description = "Clear description of what this node does.\n" + " keyword1, keyword2, keyword3"; static readonly metadataOutputTypes = { output: "str", }; @prop({ type: "str", default: "", title: "Input Value", description: "Help text" }) declare input_value: any; @prop({ type: "int", default: 100, title: "Threshold", min: 0, max: 255 }) declare threshold: any; async process(): Promise<Record<string, unknown>> { return { output: `Result: ${String(this.input_value ?? "")}` }; } } // MULTI-OUTPUT NODE export class MultiOutputNode extends BaseNode { static readonly nodeType = "mypackage.example.MultiOutput"; static readonly title = "Multi Output"; static readonly description = "Produces multiple outputs.\n multi, output"; static readonly metadataOutputTypes = { if_true: "any", if_false: "any", }; // Forward correlation: both outputs carry the `value` input's correlation // token unchanged. Streaming behavior is inferred from this — there is no // isStreamingOutput flag. static readonly inputMode: InputMode = "buffered"; static readonly outputCorrelation: Record<string, OutputCorrelation> = { if_true: { kind: "forward", source: "value" }, if_false: { kind: "forward", source: "value" }, }; @prop({ type: "bool", default: false, title: "Condition" }) declare condition: any; @prop({ type: "any", default: [], title: "Value" }) declare value: any; // Return only the taken branch. A missing key sends nothing on that // output, so nodes wired only to it do not run. async process(): Promise<Record<string, unknown>> { if (this.condition) { return { if_true: this.value }; } return { if_false: this.value }; } } // STREAMING / GENERATOR NODE export class StreamingNode extends BaseNode { static readonly nodeType = "mypackage.example.Streaming"; static readonly title = "Streaming Node"; static readonly description = "Emit multiple items.\n stream, iterate"; static readonly metadataOutputTypes = { output: "any", index: "int", }; // Each yielded item is a new correlated value (iteration). genProcess() // is detected automatically — no isStreamingOutput flag. static readonly inputMode: InputMode = "buffered"; static readonly outputCorrelation: Record<string, OutputCorrelation> = { output: { kind: "iteration", source: "__execution__", group: "items" }, index: { kind: "iteration", source: "__execution__", group: "items" }, }; @prop({ type: "list[any]", default: [], title: "Input List" }) declare input_list: any; async process(): Promise<Record<string, unknown>> { return {}; } async *genProcess(): AsyncGenerator<Record<string, unknown>> { const values = (this.input_list ?? []) as unknown[]; const list = Array.isArray(values) ? values : [values]; for (const [index, item] of list.entries()) { yield { output: item, index }; } } } // STATEFUL COLLECTOR NODE export class CollectorNode extends BaseNode { static readonly nodeType = "mypackage.example.Collector"; static readonly title = "Collector"; static readonly description = "Collect streamed items.\n collect, aggregate"; static readonly metadataOutputTypes = { output: "list[any]", }; private _items: unknown[] = []; @prop({ type: "any", default: [], title: "Input Item" }) declare input_item: any; async initialize(): Promise<void> { this._items = []; } async process(): Promise<Record<string, unknown>> { this._items.push(this.input_item); return { output: [...this._items] }; } } Common @prop Patterns // Text input @prop({ type: "str", default: "", title: "Text" }) declare text: any; @prop({ type: "str", default: "", title: "Text", description: "Help text" }) declare text: any; // Number with constraints @prop({ type: "int", default: 0, title: "Count", min: 0, max: 100 }) declare count: any; @prop({ type: "float", default: 0.5, title: "Threshold", min: 0.0, max: 1.0 }) declare threshold: any; // Boolean @prop({ type: "bool", default: false, title: "Enabled" }) declare enabled: any; // Optional (nullable) @prop({ type: "str", default: null, title: "Label" }) declare label: any; // List @prop({ type: "list[str]", default: [], title: "Tags", description: "List of tags" }) declare tags: any; @prop({ type: "list[any]", default: [], title: "Items" }) declare items: any; // Enum choices (dropdown in UI) @prop({ type: "enum", default: "option_a", title: "Choice", values: ["option_a", "option_b", "option_c"], }) declare choice: any; // Model selections @prop({ type: "language_model", default: null, title: "Model", required: true }) declare model: any; @prop({ type: "image_model", default: null, title: "Image Model", required: true }) declare image_model: any; @prop({ type: "tts_model", default: null, title: "TTS Model", required: true }) declare tts_model: any; // Asset references @prop({ type: "image", default: { type: "image", uri: "", data: null }, title: "Image" }) declare image: any; @prop({ type: "audio", default: { type: "audio", uri: "", data: null }, title: "Audio" }) declare audio: any; @prop({ type: "video", default: { type: "video", uri: "", data: null }, title: "Video" }) declare video: any; @prop({ type: "document", default: { type: "document", uri: "", data: null }, title: "Document" }) declare document: any; @prop({ type: "folder", default: { type: "folder", uri: "" }, title: "Folder" }) declare folder: any; // Data structures @prop({ type: "dataframe", default: { type: "dataframe", uri: "", data: null }, title: "Data" }) declare dataframe: any; @prop({ type: "dict[str, any]", default: {}, title: "Config" }) declare config: any; ProcessingContext Essentials The optional argument to process() and genProcess() is a ProcessingContext from @nodetool-ai/runtime. It provides access to provider predictions, secrets, storage, the workspace, the cache, and HTTP helpers. See the Custom Nodes Guide §10 for the full surface. Property values for declared @prop fields are assigned to this before process() is called — read them directly from this.<field>. import type { ProcessingContext } from "@nodetool-ai/runtime"; async process(context?: ProcessingContext): Promise<Record<string, unknown>> { // Access injected secrets (requires static requiredSettings). // The base class resolves keys from context and exposes them on this._secrets. const apiKey = this._secrets.MY_API_KEY ?? process.env.MY_API_KEY ?? ""; // Run a provider prediction (image generation, TTS, etc.) if (context) { const output = await context.runProviderPrediction({ provider: "openai", capability: "text_to_image", model: "gpt-image-1", params: { prompt: "a cat" }, }); } // Stream a provider prediction (e.g., TTS chunks) if (context) { for await (const chunk of context.streamProviderPrediction({ provider: "openai", capability: "text_to_speech", model: "tts-1", params: { text: "hello" }, })) { // process each chunk } } // Resolve a secret manually (returns null when missing) if (context) { const secret = await context.getSecret("SOME_KEY"); } return { output: "result" }; } Working with Media Bytes Nodes handle media as ref objects. Extract bytes, process them, and return a new ref: import { imageRefFromBytes, loadMediaRefBytes } from "@nodetool-ai/runtime"; // Load bytes from any ref: inline data, asset://, storage URIs, http(s) const bytes = await loadMediaRefBytes(this.image, context); // Uint8Array | null // Create an image ref from encoded bytes (sets type, base64 data, MIME, size) const ref = await imageRefFromBytes(outputBytes); loadMediaRefBytes(ref, context?) returns null when it finds no bytes, so check the result. Reading ref.data or calling fetch(ref.uri) directly misses asset:// references. Audio and video have helpers in their packages, such as audioBytesAsync and audioRefFromBytes in packages/audio-nodes. Static Class Properties // Declare output types (required for UI connectors). Every key a node // returns must be declared here. static readonly metadataOutputTypes = { output: "str", count: "int" }; // Enable dynamic (user-added) input connectors. Read/write extra inputs at // runtime with this.getDynamic(key) / this.setDynamic(key, value). static readonly supportsDynamicInputs = true; // Support dynamic output slots static readonly supportsDynamicOutputs = true; // Restrict which types a user may pick for a dynamic input slot static readonly allowedDynamicSlotTypes = [{ type: "str", type_args: [] }, { type: "int", type_args: [] }]; // Field split for the UI: inlineFields render compactly on the node body; // inputFields render as the larger expanded inputs. static readonly inlineFields = ["prompt"]; static readonly inputFields = ["model"]; // Input consumption mode (default is undefined → buffered): // "buffered" — collect a matched set of inputs, call process() once // "stream" — consume inputs as an async stream via run() // "controlled" — node runs when control events arrive (see isControlled) static readonly inputMode = "buffered"; // Accept streaming input (used together with inputMode = "stream") static readonly isStreamingInput = true; // Per-output correlation. Drives scheduling and whether an output streams. // forward — output carries the source input's correlation token unchanged // iteration — each emitted value is a fresh correlated item // aggregate — collapse a stream into one value // single — one value per invocation, not correlated to a source static readonly outputCorrelation = { output: { kind: "forward", source: "value" }, }; // Declare required secrets — injected onto this._secrets static readonly requiredSettings = ["OPENAI_API_KEY"]; // Run-planning hints static readonly effect = "pure"; // "pure" | "read" | "write" | "external" (default "external") static readonly cacheTtl = "forever"; // reuse results of partial runs: "forever" or seconds static readonly retrySafe = true; // re-running with identical inputs is safe static readonly platforms = ["node"]; // deployment targets (default ["node"]) There is no isStreamingOutput, syncMode, isDynamic, or basicFields static field. Streaming is inferred from outputCorrelation (and from defining genProcess()/run()); dynamic inputs use supportsDynamicInputs; the field split uses inlineFields/inputFields. Lifecycle Hooks // Called once per node before the run starts -- reset state here async initialize(): Promise<void> { this._items = []; } // Called once when the node's actor starts, before its first invocation // (not before every process() call) async preProcess(): Promise<void> {} // Called when the node finishes, including after an error async finalize(): Promise<void> {} Return Type Patterns // Single output async process(): Promise<Record<string, unknown>> { return { output: "result" }; } // Multiple outputs async process(): Promise<Record<string, unknown>> { return { text: "hello", score: 0.95 }; } // Streaming (generator) async *genProcess(): AsyncGenerator<Record<string, unknown>> { for (const [i, item] of items.entries()) { yield { output: item, index: i }; } } Input Node Quick List Types in the nodetool.input namespace (packages/core-nodes/src/nodes/input.ts). The generated catalog under docs/nodes/nodetool/input/ documents each one. StringInput - Text value IntegerInput - Whole number (min/max) FloatInput - Decimal (min/max) BooleanInput - True/False toggle SelectInput - Choice from a fixed set of options ColorInput - Color picker ImageSizeInput - Image width and height ValueInput - Any value, passed through unconverted StringListInput - List of strings TextListInput - List of text values ImageListInput - List of images AudioListInput - List of audio clips VideoListInput - List of videos LanguageModelInput - Select LLM ImageModelInput - Select image model VideoModelInput - Select video model TTSModelInput - Select text-to-speech model ASRModelInput - Select speech-recognition model EmbeddingModelInput - Select embedding model HuggingFaceModelInput - Select a Hugging Face model ImageInput - Image asset reference AudioInput - Audio asset reference VideoInput - Video asset reference DocumentInput - Document asset reference Model3DInput - 3D model asset reference DataframeInput - Tabular data AssetFolderInput - Folder asset reference FolderPathInput - Local folder path FilePathInput - Local file path DocumentFileInput - Load document from file MessageInput - Chat message MessageListInput - List of chat messages MessageDeconstructor - Split a message into its fields RealtimeAudioInput - Live audio stream Vector collections come from vector.Collection, not an input node. Output Node Quick List nodetool.output.Output - Named output for any data type nodetool.workflows.base_node.Preview - Show values inside the graph Docstring Keywords by Category Data Types text, string, number, integer, float, boolean, list, array, dict, object, document, file Operations extract, filter, map, reduce, merge, split, join, sort, group, aggregate, transform, analyze Media image, picture, visual, video, audio, sound, document, file, folder, asset AI/ML model, embedding, classification, clustering, generation, language, agent, tool Control flow, condition, loop, iterator, generator, stream, branch, switch I/O input, output, load, save, read, write, import, export, download, upload Testing Pattern // packages/<your-pkg>/src/nodes/my-nodes.ts export class MyNode extends BaseNode { static readonly nodeType = "mypackage.MyNode"; static readonly title = "My Node"; static readonly description = "My node.\n keywords"; static readonly metadataOutputTypes = { output: "str" }; @prop({ type: "str", default: "", title: "Value" }) declare value: any; async process(): Promise<Record<string, unknown>> { return { output: String(this.value ?? "").toUpperCase() }; } } // tests/my-nodes.test.ts import { describe, it, expect } from "vitest"; import { MyNode } from "../src/nodes/my-nodes.js"; describe("MyNode", () => { it("uppercases input", async () => { const node = new MyNode({ value: "hello" }); const result = await node.process(); expect(result.output).toBe("HELLO"); }); }); For nodes that read the context, build one with createFakeContext() from @nodetool-ai/runtime and pass handle.context to process(). To run a single registered node from the shell, use nodetool node run mypackage.MyNode --props '{"value":"hello"}'. Key Reminders All process() and genProcess() methods must be async Always declare metadataOutputTypes – it drives the UI output connectors Use @prop for every input – it provides validation, defaults, and UI hints Read input values from this.<field> – the engine assigns them before process() runs Return a plain object whose keys are declared in metadataOutputTypes (omit a key to send nothing on that output) Use genProcess() with yield for streaming outputs Use initialize() to reset state in stateful / collector nodes Declare requiredSettings for API keys; read them from this._secrets A class exists to the runtime only after something registers it. Packs do that in their register function Test with vitest – nodes are plain classes, easy to instantiate and call--- # Provider Guides: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/ One runbook per provider for adding new models and nodes. Each guide names the exact files, the command to run, a copy-pasteable snippet, and how the same change looked in past PRs. Audience: coding agents and contributors keeping NodeTool current — a new image model shipped, a new endpoint went live, a new provider needs wiring. Start with the provider you’re updating; each guide is self-contained. NodeTool spans many providers but only a few distinct mechanisms. Find your provider’s mechanism first — it tells you whether a new model needs zero code, a manifest entry, or a codegen run. How adding a model works, by mechanism Mechanism What “add a model” means Providers Dynamic fetch Nothing — models appear from the provider’s live /models (or local daemon) endpoint. Code only for capability/cost overrides. OpenAI (chat), Anthropic, Gemini (text), xAI, OpenAI-compatible (Groq, Mistral, DeepSeek, Moonshot, Cerebras, Alibaba Cloud, OpenRouter, Requesty), Ollama, Local inference (LM Studio, llama.cpp, vLLM, in-process llama.cpp) Static model list A code edit to a hardcoded array in the provider (plus cost entry). OpenAI (image), Gemini (image, Veo, Omni, Lyria, TTS, ASR, embeddings) Codegen from upstream schemas Add an endpoint id to a config, run the generator — the manifest is regenerated, never hand-edited. FAL, Replicate, KIE Manifest Add a JSON entry to the package manifest. AtlasCloud adds a sync script for fields and Together generates its manifest from a script. AtlasCloud, Topaz, Together Static node package Add a model id to the provider/base arrays, or a new BaseNode subclass. MiniMax, Reve, ElevenLabs, HuggingFace All guides OpenAI — chat models auto-fetch; image models (gpt-image-*) are a static list. Anthropic (Claude) — models fetched live from the Anthropic API. Google Gemini — text auto-fetches; image, Veo, Omni, and Lyria models are static lists. xAI (Grok) — a static catalog plus chat, image, and video rows classified from /v1/models. OpenAI-compatible providers — Groq, Mistral, DeepSeek, Moonshot, Cerebras, Alibaba Cloud, Cohere, OpenRouter, Requesty, Opper, API Route, and how to add a new one. FAL — nodes generated from FAL OpenAPI schemas via codegen. Replicate — nodes generated from Replicate model schemas via codegen. KIE — configs and manifest generated from KIE’s published API docs. AtlasCloud — hand-maintained manifest drives nodes and the model picker. Topaz — manifest-driven image/video enhancement nodes. Together AI — generated image/video manifest plus live-fetched chat models. MiniMax — chat, image, video, and audio via provider arrays and a node pack. Reve — image create/edit/remix nodes. ElevenLabs — TTS models and voices as static arrays (decorator package). HuggingFace — Hub-backed model discovery plus a hand-written node pack. Ollama — local models discovered from the running daemon (ollama pull). Local inference — LM Studio, llama.cpp, and vLLM over OpenAI-compatible localhost, plus in-process llama.cpp. Before you open a PR Whatever the mechanism, run these from the repo root: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main For decorator/codegen packages (FAL, Replicate, KIE, ElevenLabs, and other dist/-loaded packs), run npm run build:packages after regenerating so the runtime picks up the change. Each guide lists its exact verify commands. npm run check is the full aggregate and runs only when explicitly needed. Contributions are welcome — open a PR on GitHub or say hi on Discord.--- # Anthropic (Claude): Provider Guide Source: https://docs.nodetool.ai/developer/providers/anthropic.html Anthropic provider NodeTool calls Anthropic’s Messages API through AnthropicProvider. The provider supports text, image input, client tools, native web search, streaming, prompt cache accounting, and extended thinking. It uses ANTHROPIC_API_KEY by default through the built-in provider registry. Direct integrations can construct the provider with an auth token, structured SDK credentials, or a custom base URL; those alternatives are not resolved by the built-in registry. Anthropic changes model capabilities independently of the Models API. A model appearing in the selector does not prove that every NodeTool request option works with it. Check the model’s thinking mode, sampling parameters, stop reasons, and tool behavior before making it a default or recommendation. Files Concern Path Provider packages/runtime/src/providers/anthropic-provider.ts Recommended models packages/runtime/src/recommended-models.ts Provider registration packages/runtime/src/providers/index.ts Provider tests packages/runtime/tests/providers/anthropic-provider*.test.ts Catalog tests packages/runtime/tests/recommended-models.test.ts Cost calculation packages/runtime/src/providers/cost-calculator.ts ClaudeAgentProvider is a separate integration backed by the Claude Agent SDK and a local Claude subscription. Changes to AnthropicProvider do not affect it. Model discovery getAvailableLanguageModels() calls /v1/models on the configured API base URL. It follows the Models API pagination cursor and uses each model’s display_name. Discovery retries transient errors, including Anthropic’s 529 overload response, and honors Retry-After as described in Anthropic API errors. There is no static fallback list. If discovery fails, the provider returns no Anthropic models for that fetch. The curated entry in recommended-models.ts is independent of discovery and must name an active model. Model lifecycle Use Anthropic’s model deprecation table before adding or changing a recommended model. Do not recommend a -latest alias: it can move between snapshots and does not communicate a lifecycle date or version. Anthropic’s dateless ids from Claude 4.6 onward are pinned model versions, not moving aliases. When changing the recommendation: Confirm the model is active and generally available. Check that @pydantic/genai-prices recognizes its exact id. Verify text, streaming, tool use, and configured thinking behavior. Update the catalog test anchor in the same change. Compatibility checks Thinking NodeTool uses manual thinking on models that accept budget_tokens: { "thinking": { "type": "enabled", "budget_tokens": 4096 } } This shape is not valid for every current model. The provider uses adaptive thinking for models that require it. Anthropic’s extended thinking guide requires adaptive configuration for Claude Fable 5, Claude Mythos 5 and Preview, Claude Opus 5, Claude Opus 4.8 and 4.7, and Claude Sonnet 5. Opus 5 and Sonnet 5 enable adaptive thinking by default; Opus 5 only accepts disabled thinking at effort high or lower, and Fable and Mythos do not allow thinking to be disabled at all. Manual budget_tokens is deprecated but still accepted for Claude Opus 4.6 and Claude Sonnet 4.6. Claude Haiku 4.5 also accepts manual thinking. Manual budgets must be at least 1,024 tokens and less than max_tokens. Tests must cover both streaming and non-streaming request shapes. Thinking with tools must preserve signed thinking blocks unchanged across the tool-result turn. Sampling parameters Anthropic rejects non-default temperature, top_p, and top_k on current adaptive-only/default model families, including Claude Opus 4.7 and later, Sonnet 5, Fable 5, and Mythos. The provider omits incompatible sampling parameters according to Anthropic’s parameter deprecation table. Stop reasons Every successful Messages response has a stop_reason. Anthropic documents end_turn, max_tokens, stop_sequence, tool_use, pause_turn, refusal, and model_context_window_exceeded in its stop-reason guide. pause_turn requires another request with the assistant content preserved; the provider continues these turns internally. A refusal is an HTTP 200 response and may contain no content. Claude Fable 5 also returns stop_details. The provider surfaces refusals and truncated output instead of treating them as a normal completed answer. Tools and content The provider maps NodeTool client tools to Anthropic input_schema definitions without dropping supported schema constraints and maps web_search to web_search_20250305. Anthropic still lists that web search version as GA, alongside newer versions in the tool reference. The adapter also passes strict, input_examples, cache_control, defer_loading, and allowed_callers when configured. Tool execution failures are returned with Anthropic’s is_error marker. Image input supports JPEG, PNG, GIF, and WebP through inline base64 blocks. Document input supports inline PDF data, HTTPS PDF URLs, and plain text, including document titles, context, and citation configuration. See Anthropic’s PDF support for API limits. Uploaded Anthropic Files API ids are not part of NodeTool’s shared message contract. Citation ranges and source metadata remain structured on text content; the provider does not append citation Markdown to Claude’s text. Cost tracking Claude token prices come from @pydantic/genai-prices. Check a new model id before recommending it: node -e "const {calcPrice}=require('@pydantic/genai-prices'); \ console.log(calcPrice({input_tokens:1000,output_tokens:500},'claude-sonnet-5','anthropic'))" Do not add Claude token prices to the local non-token modality table. Update the upstream catalog or bump the dependency when it already contains the model. Anthropic reports uncached input separately from cache reads and writes. The provider adds those values before sending usage to the shared cost calculator. Verify Run the focused catalog and provider tests first: npm test --workspace=packages/runtime -- \ --run tests/recommended-models.test.ts \ tests/providers/anthropic-provider.test.ts \ tests/providers/anthropic-provider-coverage.test.ts Then run the required repository checks: npm run typecheck npm run lint npm run test With ANTHROPIC_API_KEY configured, inspect discovery and run a small request: npm run dev:nodetool -- models by-provider anthropic --kind llm --json npm run dev:nodetool -- node run nodetool.agents.Agent \ --props '{"prompt":"hello","model":{"type":"language_model","provider":"anthropic","id":"claude-sonnet-5"},"max_tokens":50}'--- # AtlasCloud: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/atlascloud.html Navigation: Developer Index Providers Overview → AtlasCloud Audience: contributors and coding agents adding AtlasCloud models or node classes. No familiarity with the runtime internals is assumed; you need to edit JSON and run two commands. TL;DR Add an entry to packages/atlascloud-nodes/src/atlascloud-manifest.json. Run node scripts/sync-atlascloud-manifest.mjs to fill the entry’s fields from AtlasCloud’s published schema, then --check to confirm no drift. Run npm run build:packages to copy the manifest into dist/. Run npm run typecheck && npm run test --workspace=packages/atlascloud-nodes. That’s it. No TypeScript code is required for most new models. Skip to Add a new model/node for the exact steps. Where things live What Path Manifest (JSON, single source of truth. Entries are added by hand and fields are reconciled by the sync script) packages/atlascloud-nodes/src/atlascloud-manifest.json Manifest copy in dist/ (generated by build) packages/atlascloud-nodes/dist/atlascloud-manifest.json Sync script (reconciles manifest fields with AtlasCloud’s schemas) scripts/sync-atlascloud-manifest.mjs Node class factory (reads manifest at runtime) packages/atlascloud-nodes/src/atlascloud-factory.ts API key lookup, re-exports of the runtime transport packages/atlascloud-nodes/src/atlascloud-base.ts Submit, wait, download (shared by the pack and the provider) packages/runtime/src/providers/atlascloud-transport.ts Pending-prediction registry for callbacks packages/runtime/src/providers/atlascloud-webhook-registry.ts Signed callback route packages/websocket/src/routes/atlascloud-webhook.ts Package entry point (exports ATLASCLOUD_NODES) packages/atlascloud-nodes/src/index.ts Runtime provider (chat via the OpenAI-compatible API, image/video model lists, textToImage etc.) packages/runtime/src/providers/atlascloud-provider.ts Provider registration packages/runtime/src/providers/index.ts (registerBuiltinProvider(PROVIDER_IDS.ATLASCLOUD, ...)) Manifest-to-model-list helper (shared with KIE, FAL) packages/runtime/src/providers/manifest-models.ts How AtlasCloud models/nodes are defined atlascloud-manifest.json is the single file you edit to add a model. You write each entry’s identity (className, moduleName, modelId, title, description) by hand. scripts/sync-atlascloud-manifest.mjs keeps the fields of shipped models in line with the Input schema that AtlasCloud publishes for each model, and it never adds models. Manifest → node class packages/atlascloud-nodes/src/index.ts loads the manifest with loadPackageAssetJson at module load time and passes it to loadAtlasNodesFromManifest. That function calls createAtlasNodeClass once per manifest entry. createAtlasNodeClass (in atlascloud-factory.ts) produces a BaseNode subclass whose nodeType is atlascloud.<moduleName>.<className>, whose input props match the fields array, and whose process() method calls atlasSubmit → atlasAwaitResult → atlasDownload. The transport functions live in packages/runtime/src/providers/atlascloud-transport.ts and are shared with the provider. The build script in package.json compiles TypeScript then copies src/atlascloud-manifest.json to dist/: "build": "node ../../scripts/build-typescript-workspace.mjs && node -e \"require('node:fs').cpSync('src/atlascloud-manifest.json', 'dist/atlascloud-manifest.json')\"" The manifest export declared in package.json is: "./atlascloud-manifest.json": "./dist/atlascloud-manifest.json" Manifest → model list (provider) AtlasCloudProvider.getAvailableImageModels() and getAvailableVideoModels() call loadImageModels / loadVideoModels from manifest-models.ts, which load the manifest through loadPackageAssetJson from @nodetool-ai/atlascloud-nodes/atlascloud-manifest.json. The model-list functions read outputType, modelId, and title from each entry. buildModelMap() in the provider reads the same manifest to know which fields each model accepts for the generic textToImage / imageToVideo etc. methods. Key point: the manifest drives both the node UI (fields, defaults, enum values) and the model picker (image/video model lists). Editing the manifest is sufficient for both. Wire protocol Submit: POST https://api.atlascloud.ai/api/v1/model/generateImage or .../generateVideo Body: flat JSON — { model: "<modelId>", ...fields } (not nested under input) Poll: GET https://api.atlascloud.ai/api/v1/model/prediction/<id> Terminal success statuses: completed, complete, succeeded, success, done. Terminal failure statuses: failed, fail, error, cancelled, canceled (http-transport.ts) Chat models use AtlasCloud’s OpenAI-compatible API at https://api.atlascloud.ai/v1 through AtlasCloudProvider and need no manifest entry. Submit is never retried — a 5xx may have already created and billed the job upstream. Add a new model/node Step 1: Identify the model From the AtlasCloud docs or API, collect: The model id string, e.g. "bytedance/seedance-3.0/text-to-video" The modality: "image" or "video" The field names and types the endpoint accepts Step 2: Choose a className and moduleName className: PascalCase identifier used as the TypeScript class name and the last segment of the node type. Example: "Seedance3TextToVideo". moduleName: "image" or "video" (matches the output type). Used in the node type: atlascloud.video.Seedance3TextToVideo. Step 3: Append an entry to the manifest Open packages/atlascloud-nodes/src/atlascloud-manifest.json. Append an object following this shape (copy the closest existing entry and adjust): { "className": "Seedance3TextToVideo", "moduleName": "video", "modality": "video", "modelId": "bytedance/seedance-3.0/text-to-video", "outputType": "video", "title": "Seedance 3.0 — Text to Video", "description": "AtlasCloud / ByteDance Seedance 3.0 — text-to-video with native audio.", "pollInterval": 5000, "maxAttempts": 600, "fields": [ { "name": "prompt", "type": "str", "default": "", "title": "Prompt", "description": "Text prompt describing the desired video.", "required": true }, { "name": "duration", "type": "int", "default": 5, "title": "Duration (seconds)", "values": [-1, 4, 5, 6, 7, 8, 9, 10] }, { "name": "resolution", "type": "enum", "default": "720p", "title": "Resolution", "values": ["480p", "720p", "1080p"] }, { "name": "ratio", "type": "enum", "default": "16:9", "title": "Aspect Ratio", "values": ["16:9", "9:16", "1:1", "adaptive"] } ] } Field type reference: type Description "str" Text input "int" Integer (coerced from string dropdowns) "float" Float "bool" Checkbox "enum" Dropdown; requires "values" array "image" Single image ref "video" Single video ref "audio" Single audio ref "list[image]" Multiple image refs "list[video]" Multiple video refs "list[audio]" Multiple audio refs Use "list[image]" (not image with "array": true) for multi-image inputs. Use "null" as the default for optional enum fields without a sensible preselected value (e.g. aspect_ratio on Nano Banana models). pollInterval: milliseconds between poll requests. Existing entries use 3000 for image models and 5000 for video models. maxAttempts: maximum polls before timing out. Existing entries use 300 for image and 600 for video. wrapInto: set on a typed asset field (reference_images, reference_videos, reference_audios) when the API takes one mixed array such as refers: [{url, type}]. The factory appends each field’s resolved URLs into that array in field order. The provider’s generic imageToVideo reads the same flag. Step 4: Reconcile fields with the published schema node scripts/sync-atlascloud-manifest.mjs --check # exit 1 if the manifest is out of date node scripts/sync-atlascloud-manifest.mjs # rewrite the manifest The script fetches the catalog at GET https://api.atlascloud.ai/api/v1/models (no auth) and each model’s Input schema. It updates enum options, defaults, numeric bounds, scalar types, and required flags. It skips asset fields (image, video, audio, list[...]) and the transport switches enable_base64_output, enable_sync_mode, return_last_frame, and output_dir, which break the poll-then-download flow or cannot surface. A schema property typed anyOf needs a hand-written field. After you add an entry, --check must report the manifest up to date. A drift line against the new model means the entry disagrees with its schema. Step 5: Build npm run build:packages This compiles TypeScript and copies the manifest into dist/. The factory picks it up at module load time — no TypeScript changes needed. Step 6: Smoke-test the node ATLASCLOUD_API_KEY=<your_key> \ npm run dev:nodetool -- node run \ atlascloud.video.Seedance3TextToVideo \ --props '{"prompt":"a red balloon floating upward","duration":5,"resolution":"720p","ratio":"16:9"}' For a hermetic check (no secrets, no API call — verifies the node can be instantiated): npm run dev:nodetool -- node run atlascloud.video.Seedance3TextToVideo \ --props '{"prompt":"test"}' --no-secrets 2>&1 | grep -E "error|ATLASCLOUD" Expected: ATLASCLOUD_API_KEY is not configured (meaning the node loaded and reached the key check). Step 7: Validate (no API call) npm run dev:nodetool -- validate workflow.json If you have a workflow JSON referencing the new node, this verifies the node type is registered and all edges type-check. Prediction callbacks (webhooks) AtlasCloud can POST a prediction’s terminal state back instead of making the run poll for it (AtlasCloud webhooks). Nothing in the manifest turns this on. It follows from one setting. Set NODETOOL_PUBLIC_URL to an https address on a public host that AtlasCloud can reach, and atlasSubmit adds webhook_url: <NODETOOL_PUBLIC_URL>/api/providers/atlascloud/webhook to every submission. An http, loopback, or private address is ignored rather than sent, because AtlasCloud refuses those. What happens then: AtlasCloud POSTs the callback with an Ed25519 signature over <timestamp>.<raw body>, the signing key id, and the delivery timestamp. The route verifies the signature against the provider’s JWKS (https://api.atlascloud.ai/api/v1/webhooks/jwks.json), which it caches for a day and refreshes once when a key id is unknown. Deliveries older than five minutes are rejected. The route skips session auth because the signature is the credential. A verified callback resolves the pending prediction in the registry, and the run that submitted it continues with the callback’s payload.outputs. Delivery is at-least-once and best effort, so the callback is an optimization, never the only way a run finishes. atlasAwaitResult races it against a poll of the prediction endpoint every 15s, and the run takes whichever reports a terminal state first. A callback that never arrives costs latency, not the run. A callback this process cannot match, such as a duplicate or one for a prediction another replica submitted, is acknowledged with matched: false so AtlasCloud stops retrying it. Run npm run test --workspace=packages/runtime -- atlascloud-webhook and npm run test --workspace=packages/websocket -- atlascloud-webhook after changing any of this. nodetool harness gate selects both from the atlascloud-callbacks surface. Verify Run these in order after any manifest change: # 1. Rebuild (copies manifest to dist) npm run build:packages # 2. Type check the whole repo npm run typecheck # 3. Test atlascloud-nodes npm run test --workspace=packages/atlascloud-nodes # 4. Test runtime providers (checks manifest-models and the provider) npm run test --workspace=packages/runtime # 5. Affected tests, lint, and the harness gate npm run test:affected npm run lint npm run dev:nodetool -- harness gate --base origin/main None of these make live API calls. Tests use vitest with mocked HTTP. Contributing Source: https://github.com/nodetool-ai/nodetool Discord: https://discord.gg/WmQTWZRcYE Before opening a PR, run the Verify steps above. The PR description should include the AtlasCloud model id, the node type string, and a brief note on which fields were included and why any fields from the API docs were omitted.--- # ElevenLabs: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/elevenlabs.html Audience: Contributors and coding agents adding ElevenLabs models, voices, or nodes to NodeTool. TL;DR Add the model id to the MODELS array in elevenlabs-provider.ts and the @prop values array in the relevant node file. To add a voice, add one entry to VOICE_ID_MAP in both elevenlabs-base.ts and elevenlabs-provider.ts. To add a node, create packages/elevenlabs-nodes/src/nodes/<name>.ts, export a readonly NodeClass[], and add it to src/index.ts. Run npm run build:packages (required — this package loads from dist/), then npm run test:affected, npm run typecheck, and npm run lint. Where things live What Path Provider (TTS, voice list, model list) packages/runtime/src/providers/elevenlabs-provider.ts Shared voice/key helpers packages/elevenlabs-nodes/src/elevenlabs-base.ts Node package entry packages/elevenlabs-nodes/src/index.ts TTS node packages/elevenlabs-nodes/src/nodes/text-to-speech.ts STT node packages/elevenlabs-nodes/src/nodes/speech-to-text.ts Realtime TTS node (WebSocket) packages/elevenlabs-nodes/src/nodes/realtime-tts.ts Realtime STT node (WebSocket) packages/elevenlabs-nodes/src/nodes/realtime-stt.ts Standard voice picker node packages/elevenlabs-nodes/src/nodes/standard-voice.ts Provider registration packages/runtime/src/providers/index.ts (registerBuiltinProvider(PROVIDER_IDS.ELEVENLABS, ...)) Node tests packages/elevenlabs-nodes/tests/ How ElevenLabs models and nodes are defined Models Models are static arrays, not fetched at runtime. Two separate places hold them: Provider (elevenlabs-provider.ts): the MODELS array drives what getAvailableTTSModels() returns to the unified TTS picker: const MODELS: Array<{ id: string; name: string }> = [ { id: "eleven_v3", name: "Eleven v3" }, { id: "eleven_multilingual_v2", name: "Multilingual v2" }, // … ]; Node (text-to-speech.ts / realtime-tts.ts): each node’s model_id prop carries its own values enum that controls what appears in the workflow UI: @prop({ type: "enum", default: "eleven_monolingual_v1", title: "Model", values: ["eleven_v3", "eleven_multilingual_v2", "eleven_turbo_v2_5", /* … */] }) declare model_id: any; The two lists are independent and not identical. The node’s model_id enum also carries the speech-to-speech models eleven_multilingual_sts_v2 and eleven_english_sts_v2, which the provider omits. Keep them in sync when adding a TTS model. Voices Voices are also static. The canonical map lives in two files that mirror each other: packages/elevenlabs-nodes/src/elevenlabs-base.ts — used by all nodes (voice ID resolution, the StandardVoice node enum) packages/runtime/src/providers/elevenlabs-provider.ts — used by the provider to surface voices in the unified TTS picker VOICE_ID_MAP maps display name to voice id. VOICE_NAMES = Object.keys(VOICE_ID_MAP) is passed to StandardVoiceNode’s enum values and to getAvailableTTSModels() voice lists. The TextToSpeech node’s voice_id prop is a plain string that defaults to the Aria id, so it accepts any ElevenLabs voice id. The provider resolves a voice name through the map and passes an unknown value through as an id. Nodes Each node file exports a readonly NodeClass[] named <CATEGORY>_NODES. src/index.ts spreads them all into ELEVENLABS_NODES and registerElevenLabsNodes() calls registry.register() on each. The package loads from dist/ (see "main": "dist/index.js" in package.json). TypeScript decorators (@prop) require a build step before any changes take effect. Add a new model or node Add a model to an existing node Step 1. Add the model id to MODELS in packages/runtime/src/providers/elevenlabs-provider.ts: const MODELS: Array<{ id: string; name: string }> = [ { id: "eleven_v3", name: "Eleven v3" }, { id: "eleven_multilingual_v3", name: "Multilingual v3" }, // new // … ]; Step 2. Add the same id to the values array of the model_id prop in the relevant node(s). For TextToSpeechNode in packages/elevenlabs-nodes/src/nodes/text-to-speech.ts: @prop({ type: "enum", default: "eleven_monolingual_v1", title: "Model", values: [ "eleven_v3", "eleven_multilingual_v3", // new "eleven_multilingual_v2", // … ] }) declare model_id: any; Repeat for realtime-tts.ts if the model supports streaming. Add a voice Step 1. Add the entry to VOICE_ID_MAP in packages/elevenlabs-nodes/src/elevenlabs-base.ts: export const VOICE_ID_MAP: Record<string, string> = { // existing entries … Matilda: "XrExE9yKIg1WjnnlVkGX", // new }; Step 2. Mirror the same entry in packages/runtime/src/providers/elevenlabs-provider.ts: const VOICE_ID_MAP: Record<string, string> = { // existing entries … Matilda: "XrExE9yKIg1WjnnlVkGX", // new }; VOICE_NAMES derives from Object.keys(VOICE_ID_MAP) in both files, so no other change is needed. Add a node Step 1. Create packages/elevenlabs-nodes/src/nodes/<name>.ts. Follow the shape of an existing node — extend BaseNode, set the required static fields, use @prop for inputs, implement process(): import { BaseNode, prop } from "@nodetool-ai/node-sdk"; import type { NodeClass } from "@nodetool-ai/node-sdk"; import { getElevenLabsApiKey } from "../elevenlabs-base.js"; export class SoundEffectNode extends BaseNode { static readonly nodeType = "elevenlabs.SoundEffect"; static readonly body = "content_card"; static readonly title = "Sound Effect"; static readonly description = "Generate a sound effect from a text prompt."; static readonly metadataOutputTypes = { output: "audio" }; static readonly requiredSettings = ["ELEVENLABS_API_KEY"]; @prop({ type: "str", default: "", title: "Prompt", description: "Text description of the sound to generate." }) declare prompt: any; async process(): Promise<Record<string, unknown>> { const apiKey = getElevenLabsApiKey(this._secrets); const prompt = String(this.prompt ?? ""); if (!prompt) throw new Error("Prompt is required"); const response = await fetch("https://api.elevenlabs.io/v1/sound-generation", { method: "POST", headers: { "xi-api-key": apiKey, "Content-Type": "application/json" }, body: JSON.stringify({ text: prompt }) }); if (!response.ok) { throw new Error(`ElevenLabs API error: ${await response.text()}`); } const data = Buffer.from(await response.arrayBuffer()).toString("base64"); return { output: { type: "audio", data: `data:audio/mpeg;base64,${data}` } }; } } export const SOUND_EFFECT_NODES: readonly NodeClass[] = [SoundEffectNode]; Step 2. Register it in packages/elevenlabs-nodes/src/index.ts: import { SOUND_EFFECT_NODES } from "./nodes/sound-effect.js"; export const ELEVENLABS_NODES: readonly NodeClass[] = [ ...TEXT_TO_SPEECH_NODES, ...SPEECH_TO_TEXT_NODES, ...REALTIME_TTS_NODES, ...REALTIME_STT_NODES, ...STANDARD_VOICE_NODES, ...SOUND_EFFECT_NODES, // new ]; Step 3. Add a test in packages/elevenlabs-nodes/tests/<name>.test.ts and add a registry.has("elevenlabs.SoundEffect") assertion in tests/registration.test.ts. Verify # Build first — required because this package loads from dist/ npm run build:packages # Type check all packages npm run typecheck # Run the elevenlabs-nodes test suite npm run test --workspace=packages/elevenlabs-nodes # Smoke-test a node in isolation (no workflow needed) npm run dev:nodetool -- node run elevenlabs.TextToSpeech \ --props '{"text":"hello","voice_id":"9BWtsMINqrJLrRacOk9x"}' \ --no-secrets # Validate a workflow that uses the node (if you have one) npm run dev:nodetool -- validate workflow.json # Affected tests and the harness gate before committing npm run test:affected npm run lint npm run dev:nodetool -- harness gate --base origin/main The tests/ files assert enum contents through getDeclaredProperties(), for example TextToSpeechNode.getDeclaredProperties().find(p => p.name === "model_id"). Follow that pattern in new tests. Contributing Source and issues: https://github.com/nodetool-ai/nodetool Community discussion: https://discord.gg/WmQTWZRcYE Run the Verify steps and make sure they pass before opening a PR. Two things catch most mistakes before review: npm run build:packages (catches missing exports, wrong import paths) and npm run test --workspace=packages/elevenlabs-nodes (catches registration gaps and API contract regressions).--- # FAL: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/fal.html FAL nodes in NodeTool are generated, not hand-written. A codegen pipeline fetches each endpoint’s OpenAPI schema from fal.ai, converts it to a manifest, and the runtime loads node classes from that manifest. Audience: coding agents and contributors adding new FAL endpoints or fixing generated node behavior. TL;DR Add the endpoint ID + NodeConfig to the right src/configs/<category>.ts file in packages/fal-codegen/. Run npm run generate:fal from the repo root. To fetch only endpoints the manifest lacks, run npx tsx scripts/append-new-endpoints.ts from packages/fal-codegen/. Run npm run build:packages (fal-nodes loads from dist/). Run npm run lint --workspace=packages/fal-codegen, npm run test --workspace=packages/fal-nodes, and npm run generate:fal:check. Open a PR — never commit manual edits to fal-manifest.json. Where things live Concern Path Endpoint configs (source of truth) packages/fal-codegen/src/configs/<category>.ts Config type definitions packages/fal-codegen/src/types.ts Schema fetcher (caches to .codegen-cache/) packages/fal-codegen/src/schema-fetcher.ts Schema → NodeSpec parser packages/fal-codegen/src/schema-parser.ts Config overrides applier packages/fal-codegen/src/node-generator.ts Codegen entry point packages/fal-codegen/src/generate.ts Generated manifest (do not edit) packages/fal-nodes/src/fal-manifest.json Runtime node class factory packages/fal-nodes/src/fal-factory.ts FAL API call utilities packages/fal-nodes/src/fal-base.ts Package entry point packages/fal-nodes/src/index.ts Pricing bundles (generated) packages/fal-nodes/src/generated/ High-level provider (text→image, TTS, etc.) packages/runtime/src/providers/fal-provider.ts Manifest model loading packages/runtime/src/providers/manifest-models.ts How FAL nodes are generated src/configs/<category>.ts ← you edit this ↓ schema-fetcher.ts fetches https://fal.ai/api/openapi/queue/openapi.json?endpoint_id=... caches each schema in packages/fal-codegen/.codegen-cache/<sha>.json ↓ schema-parser.ts converts OpenAPI → NodeSpec (inputFields, outputType, enums) ↓ node-generator.ts merges NodeConfig overrides (className, fieldOverrides, enumOverrides…) ↓ generate.ts writes packages/fal-nodes/src/fal-manifest.json ↓ fal-factory.ts creates runtime node classes from manifest at startup The root generate:fal script runs in strict mode by default: if any configured endpoint cannot be fetched or parsed, generation fails. Remove stale endpoints from the config or fix them before regenerating. Never edit fal-manifest.json directly. Changes are overwritten on the next npm run generate:fal. Add a new FAL endpoint 1. Find the right config file Pick the file in packages/fal-codegen/src/configs/ that matches the endpoint’s modality: Endpoint type Config file text → image text-to-image.ts image → video image-to-video.ts text → video text-to-video.ts text → speech text-to-speech.ts image → image image-to-image.ts other matching <input>-to-<output>.ts, or unknown.ts 2. Add a NodeConfig entry Open the chosen config file and add a key inside configs whose value is a NodeConfig: // packages/fal-codegen/src/configs/text-to-image.ts import type { ModuleConfig } from "../types.js"; export const config: ModuleConfig = { configs: { // ... existing entries ... "fal-ai/my-new-model": { className: "MyNewModel", docstring: "One sentence describing what the model does.", tags: ["image", "generation", "text-to-image", "txt2img"], useCases: [ "Generate images from text descriptions", "Create concept art from prompts" ], fieldOverrides: { prompt: { description: "The text prompt to generate an image from" }, image_size: { propType: "enum", enumRef: "ImageSizePreset", // reuse a sharedEnum if available default: "landscape_4_3", description: "Output image size preset" }, seed: { propType: "int", default: -1, description: "Seed for reproducible results. Use -1 for random" } } } } }; NodeConfig fields (all optional — only add what needs overriding): Field Type Purpose className string PascalCase class name for the node docstring string Node description shown in the UI tags string[] Used for node search useCases string[] Shown in node detail panel fieldOverrides Record<string, Partial<FieldDef>> Override parsed field properties (type, default, description…) enumOverrides Record<string, string> Rename enums: { ImageSize: "ImageSizePreset" } enumValueOverrides Record<string, Record<string, string>> Rename enum values FieldDef override keys: propType, default, description, enumRef, min, max. Valid propType values: "str" "float" "int" "bool" "image" "video" "audio" "enum" "list[image]" "list[video]" "list[audio]" "dict[str, any]". 3. (Optional) Add a shared enum If the endpoint introduces a new enum that other nodes in the same module will reuse, add it to the sharedEnums block: export const config: ModuleConfig = { sharedEnums: { MyAspectRatio: { name: "MyAspectRatio", values: [ ["SQUARE", "1:1"], ["WIDESCREEN", "16:9"], ["PORTRAIT", "9:16"] ], description: "Aspect ratio for output" } }, configs: { /* ... */ } }; Then reference it via enumRef: "MyAspectRatio" in a fieldOverrides entry. 4. Run codegen # From the repo root: npm run generate:fal This fetches the schema (or reads from .codegen-cache/ if cached), writes packages/fal-nodes/src/fal-manifest.json, and updates pricing bundles under packages/fal-nodes/src/generated/ when FAL_API_KEY is set. Without the key, existing pricing bundles stay untouched. Pass --no-pricing to silence the warning. To force a fresh schema fetch (bypass cache): npm run generate --workspace=packages/fal-codegen -- --strict --no-cache Verify Run these in order after any config change: # 1. Regenerate the manifest npm run generate:fal # 2. Check for TypeScript errors in codegen npm run lint --workspace=packages/fal-codegen # 3. Run codegen tests and the fixture drift gate npm run test --workspace=packages/fal-codegen npm run generate:fal:check # 4. Build fal-nodes (loads manifest from dist/) npm run build:packages # 5. Run fal-nodes tests (exercises the generated node classes) npm run test --workspace=packages/fal-nodes # 6. Smoke-test one generated node (replace type and props as appropriate) npm run dev:nodetool -- node run fal.text_to_image.FluxDev \ --props '{"prompt": "a red apple on a white table"}' \ --no-secrets The node type follows the pattern fal.<module_name>.<ClassName>, where module_name comes from the config file key in src/configs/index.ts (e.g. text_to_image). Before committing, inspect the manifest diff: git diff packages/fal-nodes/src/fal-manifest.json | head -80 Confirm the new entry appears with the expected className, inputFields, and outputType. Parser and codegen bugs go in packages/fal-codegen/. Runtime bugs go in packages/fal-nodes/. Never patch fal-manifest.json. Fixing a wrong input or output on a generated node Problem Where to fix Field parsed with wrong type packages/fal-codegen/src/schema-parser.ts Field needs a different default or description fieldOverrides in the endpoint’s config entry Enum name conflicts across nodes enumOverrides in the config entry Asset input defaults to "" instead of AssetRef defaultForPropType() in packages/node-sdk/src/manifest-node-values.ts Audio/video output missing preview set metadataOutputTypes in the factory, not outputTypes API call behavior (retry, upload, mapping) packages/fal-nodes/src/fal-base.ts Contributing PRs are welcome. Open one at https://github.com/nodetool-ai/nodetool. Before pushing, run: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main Join the discussion on Discord.--- # Google Gemini: Add Models Source: https://docs.nodetool.ai/developer/providers/gemini.html The Gemini provider supports seven modalities: text/chat, image generation, video generation, music generation, TTS, ASR, and embeddings. Language models are auto-discovered from the API; every other modality uses a hand-maintained array in the provider file. Audience: coding agents and contributors who need to add a new Gemini, Veo, or Lyria model to NodeTool. TL;DR Text/chat models: nothing to do — they are fetched live from GET /v1beta/models. Image models (gemini-* image generation): add one entry to getAvailableImageModels() in the provider. Video models (Veo, Gemini Omni): add one entry to GEMINI_VIDEO_MODELS. Music models (Lyria): add one entry to getAvailableMusicModels(). TTS / ASR / embedding models: add one entry to the matching method. Run npm run test:affected, npm run typecheck, and npm run lint before committing. Where things live Concern Path Provider (all Gemini logic) packages/runtime/src/providers/gemini-provider.ts Language model listing (dynamic) GeminiProvider.getAvailableLanguageModels() Image model listing (static) GeminiProvider.getAvailableImageModels() TTS model listing (static) GeminiProvider.getAvailableTTSModels() ASR model listing (static) GeminiProvider.getAvailableASRModels() Video model listing (static) GEMINI_VIDEO_MODELS, returned by getAvailableVideoModels() Music model listing (static) GeminiProvider.getAvailableMusicModels() Embedding model listing (static) GeminiProvider.getAvailableEmbeddingModels() Token/chat cost @pydantic/genai-prices catalog (automatic — no edit needed) Non-token cost tiers packages/runtime/src/providers/cost-calculator.ts — PRICING_TIERS / MODEL_TO_TIER Provider registration packages/runtime/src/providers/index.ts (registerBuiltinProvider(PROVIDER_IDS.GEMINI, ...)) How Gemini models are defined Language models — dynamic getAvailableLanguageModels() calls GET https://generativelanguage.googleapis.com/v1beta/models?key=<GEMINI_API_KEY> with pageSize=1000 and follows nextPageToken. It keeps entries whose supportedGenerationMethods includes "generateContent", drops names that match embedding, aqa, imagen, veo, image, or tts, removes duplicate ids, and maps each to { id, name, provider: "gemini" }. A failed request returns an empty list. A new text/chat model becomes available when Google adds it to that endpoint and its name passes the filter. Image models — static array getAvailableImageModels() returns a hardcoded array. textToImage() and imageToImage() accept only "gemini-" IDs, which call POST /models/<id>:generateContent with responseModalities: ["IMAGE", "TEXT"]. Google shut down Imagen in the Gemini API, so imagen-* IDs fail before any request. Video models — static array GEMINI_VIDEO_MODELS lists each video model with its supportedTasks, resolutions, and aspect ratios. referenceToVideo(), extendVideo(), and videoToVideo() refuse a model whose entry lacks the task. veo-* IDs use the async predictLongRunning endpoint with polling. Veo 3.1 and Veo 3.1 Fast also take up to three reference images, a last frame (endImage), and extension. Reference images and a last frame force an 8-second clip. Extension appends exactly 7 seconds to the end of a 720p input. Veo 3.1 Lite takes none of these. gemini-omni-* IDs call POST /v1beta/interactions with response_format: { type: "video" }. Omni covers text-to-video, image-to-video, reference-to-video, and video editing (videoToVideo). Input videos above the inline limit go through the Files API. Music models — static array getAvailableMusicModels() lists the Lyria models. textToMusic() calls POST /v1beta/interactions. Lyria has no duration or lyrics field, so the provider appends both to the prompt. Output is MP3 unless the caller asks Lyria 3.5 for WAV. TTS / ASR / Embedding — static arrays Each is a simple array returned from the matching getAvailable* method. TTS models carry a voices field. Embedding models carry a dimensions field. Add a new model 1. Text/chat model Nothing to do. The model appears automatically once Google adds it to the list API. Verify it shows up: curl "https://generativelanguage.googleapis.com/v1beta/models?key=$GEMINI_API_KEY" \ | jq '[.models[] | select(.supportedGenerationMethods[] | contains("generateContent")) | .name]' Token/chat cost is priced through @pydantic/genai-prices. NodeTool maps the "gemini" provider to "google" in GENAI_PROVIDER_MAP. 2. Image model Open packages/runtime/src/providers/gemini-provider.ts and add an entry to getAvailableImageModels(). The ID must start with "gemini-": async getAvailableImageModels(): Promise<ImageModel[]> { return [ { id: "gemini-3.1-flash-image", name: "Gemini 3.1 Flash Image", provider: "gemini" }, { id: "gemini-3-pro-image", name: "Gemini 3 Pro Image", provider: "gemini" } // plus gemini-3.1-flash-lite-image and any other existing entries ]; } Use the exact ID returned by Google. Do not add guessed future IDs. 3. Video model Add an entry to GEMINI_VIDEO_MODELS and list only the tasks Google documents for that variant: { id: "veo-3.1-generate-preview", name: "Veo 3.1 Preview", provider: "gemini", supportedTasks: ["text_to_video", "image_to_video", "reference_to_video", "extend_video", "extend_video_end"], resolutions: ["720p", "1080p", "4k"], aspectRatios: ["16:9", "9:16"] } Veo IDs must start with "veo-" and Omni IDs with "gemini-omni-". The prefix selects the endpoint. Leave durations off a model that supports extend_video unless its extension lengths match its generation lengths: the timeline checks the extension length against that list. 4. Music model override async getAvailableMusicModels(): Promise<MusicModel[]> { return [ { id: "lyria-3.5", name: "Lyria 3.5", provider: "gemini", supportedTasks: ["text_to_music"] }, { id: "lyria-3-clip-preview", name: "Lyria 3 Clip Preview", provider: "gemini", supportedTasks: ["text_to_music"] } ]; } Lyria IDs must start with "lyria-". 5. ASR model async getAvailableASRModels(): Promise<ASRModel[]> { return [ { id: "gemini-3.5-transcribe", name: "Gemini 3.5 Transcribe", provider: "gemini" }, { id: "gemini-3.5-flash", name: "Gemini 3.5 Flash", provider: "gemini" }, { id: "gemini-3.1-flash-lite", name: "Gemini 3.1 Flash-Lite", provider: "gemini" } ]; } 6. TTS model Add the model and its supported voices: async getAvailableTTSModels(): Promise<TTSModel[]> { const voices = ["Zephyr", "Puck" /*, ... existing voices ... */]; return [ { id: "gemini-3.8-flash-tts", name: "Gemini 3.8 Flash TTS", provider: "gemini", voices }, { id: "gemini-3.8-flash-lite-tts", name: "Gemini 3.8 Flash-Lite TTS", provider: "gemini", voices } // plus gemini-3.1-flash-tts-preview ]; } 7. Embedding model async getAvailableEmbeddingModels(): Promise<EmbeddingModel[]> { return [ { id: "gemini-embedding-2", name: "Gemini Embedding 2", provider: "gemini", dimensions: 3072 }, { id: "gemini-embedding-001", name: "Gemini Embedding 001", provider: "gemini", dimensions: 3072 } ]; } Verify # 1. Type-check all packages npm run typecheck # 2. Lint npm run lint # 3. Run the affected tests npm run test:affected # 4. Smoke-test a new image node (requires GEMINI_API_KEY in env or DB) npm run dev:nodetool -- node run nodetool.image.TextToImage \ --props '{"prompt": "a red apple", "model": {"type": "image_model", "id": "gemini-3.1-flash-image", "provider": "gemini", "name": "Gemini 3.1 Flash Image"}}' # 5. Smoke-test via chat agent (text model, auto-discovered, no list change needed) npm run dev:chat -- --provider gemini --model gemini-3.5-flash # 6. Harness gate npm run dev:nodetool -- harness gate --base origin/main Contributing Open a PR at https://github.com/nodetool-ai/nodetool. Run the checks in the Verify section before pushing. Join the discussion on Discord.--- # HuggingFace: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/huggingface.html Audience: coding agents and contributors who need to add new HuggingFace models or node types to NodeTool. For user-facing docs (setup, authentication, example workflows) see docs/huggingface.md. TL;DR New model in an existing task — nothing to do. Models are discovered live from the HF Hub API; any warm inference model for a given pipeline tag appears automatically. New node type — add a BaseNode subclass to packages/huggingface-nodes/src/nodes/<modality>.ts, export it in the modality’s array and src/index.ts, add its type to registration.test.ts, run npm run build:packages && npm run test --workspace=packages/huggingface-nodes. Where things live Concern Path Provider class (model discovery + API calls) packages/runtime/src/providers/huggingface-provider.ts Provider registration + secret key packages/runtime/src/providers/index.ts (registerBuiltinProvider(PROVIDER_IDS.HUGGINGFACE, ...)) Provider ID constant packages/protocol/src/api-types.ts — PROVIDER_IDS.HUGGINGFACE = "huggingface" Shared HTTP helpers (fetch, auth, binary handling) packages/huggingface-nodes/src/huggingface-base.ts Node implementations — text/NLP packages/huggingface-nodes/src/nodes/text.ts Node implementations — image/vision packages/huggingface-nodes/src/nodes/image.ts Node implementations — audio packages/huggingface-nodes/src/nodes/audio.ts Node implementations — video packages/huggingface-nodes/src/nodes/video.ts Package entry + registration helper packages/huggingface-nodes/src/index.ts Node tests (unit, registration) packages/huggingface-nodes/tests/ How HuggingFace models and nodes work Two separate concerns NodeTool has two different HuggingFace integration points that serve different purposes. HuggingFaceProvider (in packages/runtime/) is the AI-provider abstraction used by agent nodes, chat, and workflow LLM nodes. It wraps the @huggingface/inference SDK, which is an optional dependency loaded on first use, and implements the standard BaseProvider interface — generateMessage, textToImage, textToSpeech, etc. packages/huggingface-nodes/ is a separate node package that talks to the HuggingFace Inference Providers API directly over fetch (no SDK dependency). These nodes cover the full set of HF Inference pipeline tasks, including NLP tasks the provider abstraction does not expose. Both use HF_TOKEN from the NodeTool secrets store. Model discovery (provider side) HuggingFaceProvider does not use a static model list. It queries https://huggingface.co/api/models at runtime, filtered to inference=warm models sorted by likes, with a limit of 100 per pipeline tag: getAvailable*() method HF pipeline tag queried getAvailableLanguageModels text-generation getAvailableImageModels text-to-image getAvailableVideoModels text-to-video getAvailableTTSModels text-to-speech getAvailableASRModels automatic-speech-recognition getAvailableEmbeddingModels feature-extraction Results are cached for 10 minutes (CACHE_TTL_MS = 10 * 60 * 1000). Any model that becomes warm on the Hub appears automatically in the next request. Node transport (node-package side) All node API calls go to https://router.huggingface.co: Chat/LLM tasks: POST /v1/chat/completions (OpenAI-compatible) All other pipeline tasks: POST /hf-inference/models/{model} with a { inputs, parameters } body Helpers in huggingface-base.ts: hfChatCompletion(token, body) — chat completions hfPipelineJson(token, model, body) — pipeline tasks returning JSON as an HfJsonValue. Read it with jsonObject, jsonArray, jsonString, or jsonNumber hfPipelineBinary(token, model, body) — pipeline tasks returning raw media bytes (image/video) refToBase64(ref, context) / refToBytes(ref, context) — resolve an image/audio/video MediaRef to base64 or bytes for API input imageRefFromBytes(bytes, mimeType) / videoRefFromBytes(bytes, mimeType) — wrap output bytes into a MediaRef-shaped object cleanParams(params) — strip null/undefined entries from a parameters object Add a new model If the model runs a pipeline task that already has a node (or that the provider already supports), nothing needs changing. The model can be typed into the node’s model field directly, or it will appear in provider model lists once it goes warm on the Hub. If the model needs a new node type (new task or new output shape), follow the steps below. Add a new node 1. Choose the right modality file Task type File Text / NLP packages/huggingface-nodes/src/nodes/text.ts Image / vision packages/huggingface-nodes/src/nodes/image.ts Audio packages/huggingface-nodes/src/nodes/audio.ts Video packages/huggingface-nodes/src/nodes/video.ts 2. Implement the node class Copy the shape of an existing node in the same file. Below is an annotated skeleton for a JSON-output pipeline task: // packages/huggingface-nodes/src/nodes/text.ts (example addition) import { BaseNode, prop } from "@nodetool-ai/node-sdk"; import type { NodeClass } from "@nodetool-ai/node-sdk"; import { getHfToken, hfPipelineJson, jsonArray } from "../huggingface-base.js"; export class MySentenceSimilarityNode extends BaseNode { // Required statics ————————————————————————————————————————————— static readonly nodeType = "huggingface.MySentenceSimilarity"; // unique ID static readonly title = "My Sentence Similarity"; static readonly description = "Compute similarity between a query and a list of sentences.\n" + "text, similarity, sentence-transformers, embedding, huggingface\n\n" + "Use cases:\n" + "- Semantic search\n" + "- Duplicate detection"; static readonly inlineFields = ["source_sentence"]; // shown in node body static readonly requiredSettings = ["HF_TOKEN"]; // always include this static readonly metadataOutputTypes = { output: "list" }; // declare output shape // Properties (one @prop per input) ———————————————————————————— @prop({ type: "str", default: "sentence-transformers/all-MiniLM-L6-v2", title: "Model", description: "Sentence-similarity model repo id." }) declare model: string; @prop({ type: "str", default: "", title: "Source Sentence", description: "Query sentence." }) declare source_sentence: string; @prop({ type: "str", default: "", title: "Sentences", description: "Newline-separated sentences to compare." }) declare sentences: string; // process() calls the HF Inference API ———————————————————————— async process(): Promise<Record<string, unknown>> { const token = getHfToken(this._secrets); const source = String(this.source_sentence ?? ""); if (!source) throw new Error("Source sentence cannot be empty"); const items = String(this.sentences ?? "") .split("\n") .map((s) => s.trim()) .filter(Boolean); const result = jsonArray( await hfPipelineJson( token, String(this.model ?? "sentence-transformers/all-MiniLM-L6-v2"), { inputs: { source_sentence: source, sentences: items } } ) ); return { output: result }; } } For binary output (image or video), use hfPipelineBinary and wrap the result with imageRefFromBytes or videoRefFromBytes. See TextToImageNode in image.ts for the full pattern. For audio input, resolve the MediaRef with refToBase64(ref, context) — the process() method accepts context as its first argument and forwards it. See AutomaticSpeechRecognitionNode in audio.ts. @prop type values: "str" "int" "float" "bool" "enum" "image" "audio" "video" "dict" "list" 3. Export the new class In the same modality file, add the class to the exported array: // packages/huggingface-nodes/src/nodes/text.ts — at the bottom export const HUGGINGFACE_TEXT_NODES: readonly NodeClass[] = [ ChatCompletionNode, // ... existing nodes ... MySentenceSimilarityNode // ← add here ]; No changes needed to src/index.ts. It spreads the modality arrays into HUGGINGFACE_NODES through tagContentCardBodies, which gives nodes with displayable output a content-card body. 4. Update the registration test Open packages/huggingface-nodes/tests/registration.test.ts and add the new type to EXPECTED_NODE_TYPES: const EXPECTED_NODE_TYPES = [ // ... existing types ... "huggingface.MySentenceSimilarity" // ← add here ]; 5. Add a unit test (recommended) Add a describe block in packages/huggingface-nodes/tests/nodes.test.ts. Mock fetch to return a fixture, construct the node via new MySentenceSimilarityNode({...}), call process(), and assert on the output. See existing tests in that file for the pattern. Verify # 1. Build the package (huggingface-nodes loads from dist/) npm run build:packages # 2. Run the package tests npm run test --workspace=packages/huggingface-nodes # 3. Typecheck + lint across all packages npm run typecheck npm run lint # 4. Smoke-test the new node (replace type and props) npm run dev:nodetool -- node run huggingface.MySentenceSimilarity \ --props '{"source_sentence":"hello","sentences":"world\nhi there"}' \ --no-secrets # 5. Affected tests and the harness gate npm run test:affected npm run dev:nodetool -- harness gate --base origin/main If the smoke-test hits a real HF endpoint (i.e., HF_TOKEN is set), it calls the router at https://router.huggingface.co/hf-inference/models/<model>. For a hermetic run, pass --no-secrets and mock the network in tests. Contributing PRs are welcome at https://github.com/nodetool-ai/nodetool. Join the discussion on Discord.--- # KIE: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/kie.html Audience: Coding agents and contributors adding new KIE models or modifying KIE node behavior. This page covers the full cycle from manifest entry to verified node. TL;DR kie-manifest.json and packages/kie-codegen/src/configs/{image,audio,video}.ts are both generated from KIE’s published docs. Do not edit either by hand. npm run generate:kie re-fetches the docs, rewrites the configs, and rewrites the manifest. A model that KIE lists in https://docs.kie.ai/llms.txt appears after a regeneration. Persistent behavior changes go in the fetcher, parser, writer, or generator code. Then build and verify. Where things live Path Purpose packages/kie-codegen/src/schema-fetcher.ts Fetches llms.txt and the linked English docs pages packages/kie-codegen/src/schema-parser.ts Converts each page’s embedded OpenAPI YAML to a NodeConfig packages/kie-codegen/src/config-writer.ts Writes the three module configs packages/kie-codegen/src/configs/{image,audio,video}.ts Generated module configs packages/kie-codegen/src/generate-configs.ts Fetches KIE docs, re-generates configs packages/kie-codegen/src/generate.ts Reads configs, writes manifest + pricing packages/kie-codegen/src/fixture-generate.ts Offline generation from fixtures/, used by the drift gate packages/kie-nodes/src/kie-manifest.json Generated — do not edit packages/kie-nodes/src/kie-factory.ts Creates node classes from manifest at runtime packages/kie-nodes/src/kie-base.ts API submission, polling, upload, result conversion packages/kie-nodes/src/index.ts Loads manifest, exports KIE_NODES registry packages/runtime/src/providers/kie-provider.ts KieProvider — chat + media generation packages/runtime/src/providers/manifest-models.ts Reads manifest to build model lists How KIE nodes and models are defined The manifest packages/kie-nodes/src/kie-manifest.json is an array of KieManifestEntry objects. Each entry becomes one node class at runtime via kie-factory.ts. The node type is kie.<moduleName>.<className>. A minimal image-generation entry looks like this: { "className": "BytedanceSeedream", "moduleName": "image", "modelId": "bytedance/seedream", "title": "Seedream3.0 - Text to Image", "description": "...", "outputType": "image", "pollInterval": 1500, "maxAttempts": 400, "fields": [ { "name": "prompt", "type": "str", "default": "", "title": "Prompt", "description": "...", "required": true }, { "name": "guidance_scale", "type": "float", "default": 2.5, "title": "Guidance Scale", "min": 1, "max": 10 } ], "validation": [ { "field": "prompt", "rule": "not_empty", "message": "Prompt is required" } ] } Field types type value UI / wire type str text input int / float number input bool checkbox enum dropdown; requires values: string[] image single image AssetRef audio / video single audio/video AssetRef list[image] / list[video] / list[audio] list of AssetRefs list[str] / list[int] / list[float] / list[dict] list of scalars or objects Upload descriptors When a field holds an AssetRef that the KIE API expects as a URL, declare an uploads entry. The factory uploads the asset and injects the URL under the correct API parameter name before submitting the task. "uploads": [ { "field": "images", "kind": "image", "isList": true, "paramName": "image_urls" } ] field — the node property name (must match a field in fields) kind — "image", "audio", or "video" isList — set true when the API receives an array of URLs paramName — API request body key; defaults to <field>_url (single) or <field>_urls (list) groupKey — group multiple fields into one array parameter (e.g., image1 + image2 → image_urls) isVideoClip — builds {url, start, ends} clip payloads instead of plain URLs Conditional fields conditionalFields controls whether a scalar field is included in the request: condition Behavior gte_zero Include only when Number(value) >= 0 truthy Include only when the value is truthy not_default / (anything else) Always include Model surfacing manifest-models.ts reads the manifest to build the provider’s model lists: outputType === "image" → getAvailableImageModels() outputType === "video" → getAvailableVideoModels() outputType === "audio" and name/id contains a TTS signal → getAvailableTTSModels() outputType === "audio" and name/id contains a music signal → getAvailableMusicModels() Task classification (text_to_image, image_to_image, text_to_video, etc.) is inferred from the model id and title. Add supportedTasks to the manifest entry to override inference. Add a new KIE model 1. Find the KIE API model id and endpoint shape Check https://docs.kie.ai or the KIE dashboard. Note: modelId (e.g. vendor/model-name) Output type (image, video, audio) Input fields and their types/defaults/constraints Any media upload fields (where the API wants a URL, not raw bytes) 2. Check that the KIE docs list the model schema-fetcher.ts reads https://docs.kie.ai/llms.txt and each linked page. schema-parser.ts turns the page’s embedded OpenAPI YAML into a node: class name, modelId, title, fields, uploads, validation, and conditionalFields. A Suno page that links an old-model version is replaced by that page. A page that fails to fetch keeps the node’s previous config. If the model’s page is missing from llms.txt, or the parser mishandles it, fix the fetcher or parser rather than adding a config entry. Rules the parser follows: URL media inputs become AssetRef fields: image, video, audio, list[image], list[video], list[audio]. Each needs an uploads entry. List fields set isList: true and the API paramName. Other arrays take their list type from the item schema (list[dict], list[int], list[float], list[str]). They never fall back to list[image], because the factory skips an asset-typed field that has no upload config. A _url or _urls parameter is media even when KIE declares it type: object. A _file_urls or _link_urls parameter stays list[str]. Poll tuning: pollInterval (ms between status checks) and maxAttempts inherit from the module’s defaults (defaultPollInterval, defaultMaxAttempts) unless overridden on the node entry. Image defaults are 1500 ms / 400 attempts, audio defaults are 4000 ms / 120 attempts, and video defaults are 8000 ms / 450 attempts. 3. Regenerate the configs and manifest npm run generate:kie This runs generate --all --refresh-configs in packages/kie-codegen. It rewrites src/configs/*.ts, writes packages/kie-nodes/src/kie-manifest.json, and updates the pricing bundles in packages/kie-nodes/src/generated/. If the KIE pricing API is unreachable, the run writes empty pricing bundles. To skip the pricing fetch on purpose, pass --no-pricing: npm run generate --workspace=packages/kie-codegen -- --all --refresh-configs --no-pricing Without --refresh-configs, the run regenerates only the manifest from the checked-in configs. 4. Rebuild kie-nodes loads from dist/, so a build is required before the new node appears at runtime: npm run build:packages Verify # Type-check codegen and runtime packages npm run typecheck # Run kie-nodes and kie-codegen tests and the fixture drift gate npm run test --workspace=packages/kie-nodes npm run test --workspace=packages/kie-codegen npm run generate:kie:check # Run a single node in isolation (replace with the new node type) npm run dev:nodetool -- node run kie.image.VendorNewModel \ --props '{"prompt":"a red apple"}' --no-secrets # Static validation of a workflow that uses the node npm run dev:nodetool -- validate workflow.json --json Check git diff packages/kie-nodes/src/kie-manifest.json to confirm only the expected entry was added or changed. Contributing Source: https://github.com/nodetool-ai/nodetool Discord: https://discord.gg/WmQTWZRcYE Before opening a PR, run npm run test:affected, npm run typecheck, and npm run lint. Patches to kie-manifest.json or src/configs/*.ts alone will be rejected. Changes must come from the codegen pipeline. After an intended generator change, refresh the drift fixtures with node scripts/provider-codegen-check.mjs --provider kie --write.--- # Local Inference (LM Studio, llama.cpp, vLLM): Add Models Source: https://docs.nodetool.ai/developer/providers/local-inference.html NodeTool ships three providers that talk to a local OpenAI-compatible HTTP server (a fourth, node_llama_cpp, runs in-process and is covered below): LM Studio, llama.cpp (llama_cpp), and vLLM. All three follow the same pattern: point NodeTool at the server’s base URL, load a model in the server, and the model appears in the UI automatically via a /v1/models fetch. For local speech recognition, whisper_cpp runs GGML models in the backend and whisper_cpp_server calls a user-run whisper-server. See provider configuration for their settings. Audience: coding agents and contributors adding local models to NodeTool, or changing these providers’ code. TL;DR Start the local server and load a model. Set the server’s base URL (env var or Settings → Integrations → Local Model Servers). Models appear in NodeTool automatically — no code change needed. The only time you touch code is when you are changing provider behavior (URL resolution, message normalization, tool-call handling) — see Provider-level changes. Where things live Concern Path LM Studio provider packages/runtime/src/providers/lmstudio-provider.ts llama.cpp provider packages/runtime/src/providers/llama-provider.ts vLLM provider packages/runtime/src/providers/vllm-provider.ts Default base URLs packages/runtime/src/providers/defaults.ts Provider registration block packages/runtime/src/providers/index.ts, the local-only if (!_cloudProfile) block How models are discovered Each provider implements getAvailableLanguageModels(), which calls GET <baseURL>/v1/models at runtime. The response shape is the standard OpenAI list: { "data": [{ "id": "model-name" }, ...] } llama.cpp also accepts a models top-level key as a fallback (payload.data ?? payload.models ?? []). The provider maps each id to a LanguageModel record; no static list exists in the codebase. Loading a new model in the server makes it visible in NodeTool on the next model-list refresh. URL resolution order (all three providers): options.baseURL (constructor arg — tests only) Secret store / env var (LMSTUDIO_API_URL / LLAMA_CPP_URL / VLLM_BASE_URL) Hard-coded default (LM Studio only — http://127.0.0.1:1234; llama.cpp and vLLM throw if unset) Trailing slashes are stripped before /v1 is appended. Default ports: Server Default URL LM Studio http://127.0.0.1:1234 (defined in defaults.ts) llama.cpp none — must be set vLLM none — must be set Registration All three are registered in packages/runtime/src/providers/index.ts inside a guard that skips the cloud profile — isCloudProfileActive() reads NODETOOL_NODE_PROFILE and NODETOOL_ENV, so a self-hosted deployment setting NODETOOL_NODE_PROFILE=full keeps them registered while the commercial cloud does not: if (!_cloudProfile) { registerBuiltinProvider( PROVIDER_IDS.LMSTUDIO, LMStudioProvider, {}, { LMSTUDIO_API_URL: LMSTUDIO_DEFAULT_URL, LMSTUDIO_API_KEY: "lm-studio" }, { access: "local_service", displayName: "LM Studio" } ); registerBuiltinProvider( PROVIDER_IDS.LLAMA_CPP, LlamaProvider, { LLAMA_CPP_URL: "" }, {}, { access: "local_service", displayName: "llama.cpp server" } ); registerBuiltinProvider( PROVIDER_IDS.VLLM, VLLMProvider, { VLLM_BASE_URL: "" }, { VLLM_API_KEY: "sk-no-key-required" }, { access: "local_service", displayName: "vLLM" } ); } The third argument is the required kwargs. An empty value for a required key (LLAMA_CPP_URL, VLLM_BASE_URL) keeps the provider unavailable until the user sets it. The fourth argument is optionalKwargs, settings re-resolved from the secret store and then the environment on every getProvider() call without blocking isProviderConfigured(). That is why LM Studio works by default (the URL has a fallback). The fifth argument sets the provider’s access kind and display name. Per-server setup LM Studio Env var / settings key: LMSTUDIO_API_URL Default URL: http://127.0.0.1:1234 API key: lm-studio (hard-coded default; override with LMSTUDIO_API_KEY) LM Studio’s OpenAI-compatible server starts when you enable it in LM Studio → Local Server. Once running, NodeTool connects automatically with no URL config needed. To use a different port: # In your shell environment, or via Settings → Integrations → Local Model Servers export LMSTUDIO_API_URL=http://127.0.0.1:8080 Or in the chat CLI: npm run dev:chat -- --provider lmstudio --model <model-id> Tool calls are always reported as supported (hasToolSupport returns true); whether a specific model actually handles them depends on the model. llama.cpp Env var / settings key: LLAMA_CPP_URL (required — no default) API key: none (sk-no-key-required is sent automatically) Start the llama.cpp HTTP server: ./llama-server --model /path/to/model.gguf --port 8080 --ctx-size 4096 Then set the URL: export LLAMA_CPP_URL=http://127.0.0.1:8080 llama-server constrains generation with a grammar built from the tool schemas, so tool calling is native for any model it serves. LlamaProvider returns true from hasToolSupport and does no prompt-level emulation. LlamaProvider extends OpenAICompatProvider, which supplies the chat path, sampling parameters, usage tracking, and message normalization — the same code LM Studio and vLLM ride. The provider itself only handles URL resolution, model listing, and context-length error detection. llama.cpp in-process (node_llama_cpp) NodeLlamaCppProvider (node-llama-cpp-provider.ts) runs GGUF models inside the NodeTool process through the optional node-llama-cpp native binding. It needs no server and no secret. It lists GGUF files from the models directory, and from the Hugging Face hub cache. The directory comes from NODE_LLAMA_CPP_MODELS_DIR and defaults to the shared llama.cpp cache (~/.cache/llama.cpp, ~/Library/Caches/llama.cpp on macOS, %LOCALAPPDATA%\llama.cpp on Windows). NODE_LLAMA_CPP_GPU_BACKEND accepts auto, metal, cuda, vulkan, or cpu. It also serves embeddings and reports tool support as true. vLLM Env var / settings key: VLLM_BASE_URL (required — no default) API key: optional via VLLM_API_KEY; defaults to sk-no-key-required Start a vLLM server: vllm serve meta-llama/Llama-3-8B-Instruct --port 8000 Then set the URL: export VLLM_BASE_URL=http://127.0.0.1:8000 If your vLLM deployment requires an API key: export VLLM_API_KEY=your-key VLLMProvider extends OpenAICompatProvider and reports tool support as true. Provider-level changes (dev path) You need to edit source only when changing how a provider works, not when adding models. Change File to edit URL resolution or default port packages/runtime/src/providers/defaults.ts and the constructor in the provider file Tool-call handling lmstudio-provider.ts / llama-provider.ts / vllm-provider.ts Message normalization (all three) packages/runtime/src/providers/openai-compat-provider.ts Registration kwargs (env var name, default value) packages/runtime/src/providers/index.ts registration block Container env propagation getContainerEnv() in the provider class These providers do not load from dist/ (unlike base-nodes, node-sdk), so no build step is needed — changes take effect on the next npm run dev. Verify # 1. Check types and lint npm run typecheck npm run lint # 2. Start the local server (example: llama.cpp on port 8080) ./llama-server --model /path/to/model.gguf --port 8080 # 3. Set the URL and run a single node against it export LLAMA_CPP_URL=http://127.0.0.1:8080 npm run dev:nodetool -- node run nodetool.agents.Agent \ --props '{"prompt": "What is 2+2?", "model": {"type": "language_model", "provider": "llama_cpp", "id": "your-model-id"}}' # Or use the chat CLI npm run dev:chat -- --provider llama_cpp --model your-model-id # 4. Run the repository checks npm run test:affected npm run dev:nodetool -- harness gate --base origin/main For LM Studio and vLLM, substitute the provider name (lmstudio, vllm) and the corresponding env var. Contributing PRs are welcome at https://github.com/nodetool-ai/nodetool. Before pushing: npm run test:affected npm run typecheck npm run lint Join the discussion on Discord.--- # MiniMax: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/minimax.html Audience: contributors and coding agents adding new MiniMax models or workflow nodes. TL;DR Add a model ID to the relevant catalogue array in minimax-provider.ts (for the generic provider) and to the matching constant in minimax-base.ts (for the node pack’s @prop enum). For a new node type, copy an existing node file in packages/minimax-nodes/src/nodes/, register it in src/index.ts, add a test under tests/. Build: npm run build:packages. Verify: npm run typecheck, then npm run dev:nodetool -- node run minimax.<YourNodeType> --props '{...}'. Where things live What Path Provider class (chat/image/video/TTS/music) packages/runtime/src/providers/minimax-provider.ts Shared node constants and helpers packages/minimax-nodes/src/minimax-base.ts Node implementations packages/minimax-nodes/src/nodes/*.ts Node pack entry point packages/minimax-nodes/src/index.ts Node pack tests packages/minimax-nodes/tests/*.ts Provider ID constant packages/protocol/src/api-types.ts (PROVIDER_IDS.MINIMAX) Provider registration packages/runtime/src/providers/index.ts (registerBuiltinProvider(PROVIDER_IDS.MINIMAX, ...)) API docs https://platform.minimax.io/docs/api-reference/api-overview How MiniMax models and nodes are defined Provider catalogue (static arrays) MinimaxProvider in minimax-provider.ts overrides five getAvailable* methods (language, image, video, TTS, music). Each returns a hardcoded array. There is no live API discovery. The arrays are the source of truth for the generic nodes (chat, generic TTS, generic video). // packages/runtime/src/providers/minimax-provider.ts override async getAvailableVideoModels(): Promise<VideoModel[]> { return [ { id: "MiniMax-Hailuo-2.3", name: "MiniMax Hailuo 2.3", provider: "minimax", supportedTasks: ["text_to_video", "image_to_video"] }, // ... ]; } Node pack constants (in minimax-base.ts) packages/minimax-nodes/src/minimax-base.ts duplicates the model IDs as plain string arrays. These feed the values field in each node’s @prop decorator, which drives UI drop-downs and validation. // packages/minimax-nodes/src/minimax-base.ts export const MINIMAX_T2V_MODELS: string[] = [ "MiniMax-Hailuo-2.3", "MiniMax-Hailuo-02", "T2V-01-Director" ]; Why two places? Nodes call MiniMax’s REST API directly (not through the provider) so they can expose MiniMax-specific fields (emotion, volume, pitch, camera direction, lyrics) that the generic BaseProvider interface does not carry. Node registration src/index.ts exports MINIMAX_NODES (the full array) and registerMinimaxNodes (the registry helper). Every new node class must be imported and added to both. Add a new model or node Case A: new model ID in an existing node type Add the ID to minimax-provider.ts in the matching getAvailable* method: { id: "MiniMax-Hailuo-3.0", name: "MiniMax Hailuo 3.0", provider: "minimax", supportedTasks: ["text_to_video", "image_to_video"] }, Add the same ID to minimax-base.ts in the matching constant: export const MINIMAX_T2V_MODELS: string[] = [ "MiniMax-Hailuo-3.0", // new "MiniMax-Hailuo-2.3", // ... ]; Do the same for MINIMAX_I2V_MODELS if the model supports image-to-video. TTS, music, and image models have matching constants (MINIMAX_TTS_MODELS, MINIMAX_MUSIC_MODELS). Chat models exist only in the provider, because the pack has no chat node. The node’s @prop enum picks up the new entry automatically — no change to the node file needed. Case B: new node type Say MiniMax releases a new endpoint, for example POST /v1/text_to_3d. The 3D node below is an illustrative sketch. The pack has no 3D node today. Create packages/minimax-nodes/src/nodes/text-to-3d.ts, modelled on an existing node: import { BaseNode, prop } from "@nodetool-ai/node-sdk"; import type { NodeClass } from "@nodetool-ai/node-sdk"; import { assertBaseResp, getMinimaxApiKey, MINIMAX_BASE_URL, minimaxHeaders } from "../minimax-base.js"; const TEXT_TO_3D_MODELS = ["3d-01"]; export class MinimaxTextTo3DNode extends BaseNode { static readonly nodeType = "minimax.TextTo3D"; static readonly body = "content_card"; static readonly title = "MiniMax Text to 3D"; static readonly description = "Generate a 3D model from a text prompt using MiniMax 3D-01.\n" + "3d, generation, text-to-3d, minimax\n\n" + "Use cases:\n" + "- Prototype 3D assets from descriptions\n" + "- Generate objects for scenes"; static readonly metadataOutputTypes = { output: "model3d" }; static readonly inlineFields: string[] = []; static readonly inputFields: string[] = ["prompt"]; static readonly requiredSettings = ["MINIMAX_API_KEY"]; static readonly autoSaveAsset = true; @prop({ type: "enum", default: "3d-01", title: "Model", description: "The MiniMax 3D model to use.", values: TEXT_TO_3D_MODELS }) declare model: any; @prop({ type: "str", default: "A ceramic vase with floral patterns", title: "Prompt", description: "Text prompt describing the 3D object." }) declare prompt: any; async process(): Promise<Record<string, unknown>> { const apiKey = getMinimaxApiKey(this._secrets); const prompt = String(this.prompt ?? ""); if (!prompt) throw new Error("Prompt is required"); const res = await fetch(`${MINIMAX_BASE_URL}/v1/text_to_3d`, { method: "POST", headers: minimaxHeaders(apiKey), body: JSON.stringify({ model: String(this.model ?? "3d-01"), prompt }) }); if (!res.ok) { throw new Error(`MiniMax text_to_3d failed: ${res.status} ${await res.text()}`); } const data = (await res.json()) as Record<string, unknown>; assertBaseResp(data, "text_to_3d"); // parse response and return output ref return { output: { type: "model3d", data: "" } }; } } export const TEXT_TO_3D_NODES: readonly NodeClass[] = [MinimaxTextTo3DNode]; Register in src/index.ts: import { TEXT_TO_3D_NODES } from "./nodes/text-to-3d.js"; export { MinimaxTextTo3DNode } from "./nodes/text-to-3d.js"; export const MINIMAX_NODES: readonly NodeClass[] = [ ...VOICE_NODES, ...TEXT_TO_SPEECH_NODES, ...MUSIC_NODES, ...TEXT_TO_IMAGE_NODES, ...TEXT_TO_VIDEO_NODES, ...IMAGE_TO_VIDEO_NODES, ...TEXT_TO_3D_NODES // add here ]; Add a test in packages/minimax-nodes/tests/text-to-3d.test.ts. Stub fetch with vi.stubGlobal (see any existing test for the pattern). At minimum assert the node calls the right endpoint and returns an output. Update tests/registration.test.ts — add "minimax.TextTo3D" to both registry.has(...) checks and the types array. Verify Run these in order after any change: # 1. Type check all packages npm run typecheck # 2. Build (minimax-nodes loads from dist/) npm run build:packages # 3. Run a single node in isolation (no full server needed) npm run dev:nodetool -- node run minimax.TextToSpeech \ --props '{"text":"hello","voice_id":"English_Trustworth_Man","model":"speech-2.6-hd"}' \ --no-secrets # 4. Run the minimax-nodes test suite npm run test --workspace=packages/minimax-nodes # 5. Validate a workflow that uses the new node type npm run dev:nodetool -- validate workflow.json The --no-secrets flag lets the node runner skip the database. The node will fail at the network call (no real API key), but a type error or missing-field panic surfaces before that. Contributing Source: https://github.com/nodetool-ai/nodetool Discord: https://discord.gg/WmQTWZRcYE Before opening a PR: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main All four must pass.--- # Ollama: Add Models Source: https://docs.nodetool.ai/developer/providers/ollama.html The Ollama provider connects NodeTool to a local Ollama daemon. Models are discovered dynamically from the running daemon — no code change is needed to add a new model. Provider-level changes (tool detection, embedding behavior, URL configuration) require editing the provider source. Audience: coding agents and contributors — both users adding local models and devs modifying the provider. TL;DR ollama pull llama3.2 # makes the model appear in NodeTool automatically ollama pull nomic-embed-text # embedding models work the same way No code change. No restart required (NodeTool queries the daemon on every model-list call). If you need to change tool-support detection, the default URL, or keep_alive behavior, see Provider-level changes below. Where things live Concern Path Provider implementation packages/runtime/src/providers/ollama-provider.ts Default daemon URL packages/runtime/src/providers/defaults.ts (OLLAMA_DEFAULT_URL) Registration + optionalKwargs packages/runtime/src/providers/index.ts (the PROVIDER_IDS.OLLAMA block) Provider registry resolution logic packages/runtime/src/providers/provider-registry.ts How Ollama models are discovered OllamaProvider.getAvailableLanguageModels() fetches GET {apiUrl}/api/tags from the local daemon on every call. It maps each entry’s model field (falling back to name) to a LanguageModel record: // packages/runtime/src/providers/ollama-provider.ts async getAvailableLanguageModels(): Promise<LanguageModel[]> { const response = await this._fetch(`${this.apiUrl}/api/tags`); if (!response.ok) return []; const payload = (await response.json()) as { models?: Array<{ name?: string; model?: string }>; }; ... } getAvailableEmbeddingModels() calls getAvailableLanguageModels() and re-wraps every result as an EmbeddingModel with dimensions: 0. Ollama does not advertise context size or dimensions over /api/tags, so both methods surface all pulled models regardless of type. The caller (RAG pipeline, agent config) is responsible for picking an appropriate model. URL resolution order (highest priority first): Secret store, key OLLAMA_API_URL (set via nodetool secrets store OLLAMA_API_URL or Settings → Integrations → Local Model Servers) Environment variable — OLLAMA_API_URL Registered default — http://127.0.0.1:11434 (OLLAMA_DEFAULT_URL in defaults.ts) The registration in index.ts uses an optionalKwarg for OLLAMA_API_URL so the registry always considers Ollama “configured” (no key required) while still resolving a user-set URL on every getProvider() call without a restart. // packages/runtime/src/providers/index.ts registerBuiltinProvider( PROVIDER_IDS.OLLAMA, OllamaProvider, {}, { OLLAMA_API_URL: OLLAMA_DEFAULT_URL }, { access: "local_service", displayName: "Ollama" } ); Ollama is registered only when the cloud profile is inactive. The cloud profile is active when NODETOOL_NODE_PROFILE=cloud, or when NODETOOL_NODE_PROFILE is unset and NODETOOL_ENV=production. Self-hosted production servers that set NODETOOL_NODE_PROFILE=full keep Ollama, so they can point it at a sidecar container. The constructor also reads OLLAMA_KEEP_ALIVE from process.env. Default is "10m". Set it to "-1" to keep models resident indefinitely, or "0" to unload immediately after each request. Add a model (user path) Pull the model with the Ollama CLI: # Language models ollama pull llama3.2 ollama pull gemma3:27b ollama pull phi4 ollama pull deepseek-r1:8b # Embedding models ollama pull nomic-embed-text ollama pull mxbai-embed-large # Vision models (multimodal) ollama pull llava:13b ollama pull minicpm-v NodeTool picks up the model on the next call to getAvailableLanguageModels() or getAvailableEmbeddingModels() — no restart needed. The model ID passed to the provider must match the tag exactly as shown by ollama list (e.g. llama3.2:latest, gemma3:27b). To point NodeTool at a non-default Ollama instance (remote host, different port): npm run dev:nodetool -- secrets store OLLAMA_API_URL # Paste the URL when prompted, e.g.: http://192.168.1.10:11434 Or set the environment variable before starting the server: OLLAMA_API_URL=http://192.168.1.10:11434 npm run dev Provider-level changes (dev path) Edit packages/runtime/src/providers/ollama-provider.ts. The main extension points: Tool-support detection hasToolSupport(model) queries POST {apiUrl}/api/show and checks for "tools" in the capabilities array. It caches results in _modelInfoCache (a Map<string, Record<string, unknown>>). When the /api/show call fails or the model has no capabilities field, it defaults to true. If a model reports no tool support, generateMessage and generateMessages fall back to _injectToolEmulationPrompt + _parseEmulatedToolCalls, which inject tool descriptions into the system prompt and parse function_name(param='value') patterns from the output. To override detection for a specific model (e.g. a model that advertises tools but calls them incorrectly), add a guard at the top of hasToolSupport: if (model.startsWith("my-broken-model")) return false; Embedding dimensions getAvailableEmbeddingModels() returns dimensions: 0 for every model because Ollama’s /api/tags response does not include that field. If you need accurate dimensions, query POST /api/show for the pulled model and read model_info["embedding_length"]. The _modelInfoCache already stores the full /api/show response, so the call is free on the second access. Keep-alive Change the default (currently "10m") by editing the fallback in the constructor: // packages/runtime/src/providers/ollama-provider.ts const keepAlive = process.env.OLLAMA_KEEP_ALIVE?.trim(); this.keepAlive = keepAlive && keepAlive.length > 0 ? keepAlive : "10m"; Default URL Change OLLAMA_DEFAULT_URL in packages/runtime/src/providers/defaults.ts. The constant is re-exported from packages/runtime/src/providers/index.ts so callers import it as @nodetool-ai/runtime. Verify # 1. Confirm the daemon is running and a model is present curl http://127.0.0.1:11434/api/tags # 2. List the models NodeTool sees npm run dev:nodetool -- models by-provider ollama --kind llm # 3. Run a single Ollama-backed node (replace model with one you have pulled) npm run dev:nodetool -- node run nodetool.agents.Agent \ --props '{"prompt":"hello","model":{"type":"language_model","provider":"ollama","id":"llama3.2:latest"}}' # 4. Affected tests npm run test:affected If the daemon is not running, getAvailableLanguageModels() returns [] silently (the provider guards if (!response.ok) return []). Check that ollama serve (or the Ollama desktop app) is running before debugging further. Contributing PRs welcome at https://github.com/nodetool-ai/nodetool. Before pushing: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main Join the discussion on Discord.--- # OpenAI-Compatible Providers: Add a Provider or Models Source: https://docs.nodetool.ai/developer/providers/openai-compatible.html Audience: Coding agents and contributors who want to add a new OpenAI-compatible cloud provider or change how an existing one exposes models. This guide covers ten providers that share one implementation pattern: Groq, Mistral, DeepSeek, Moonshot (Kimi), Cerebras, Alibaba Cloud (Qwen), Cohere, OpenRouter, Requesty, Opper, and API Route. No code needed: add an endpoint from Settings Most OpenAI-compatible endpoints — a proxy, a company gateway, a self-hosted router — need nothing from this guide. Settings → Models & Providers → OpenAI-compatible endpoints → Add endpoint takes a name, a slug, a base URL and an optional API key, and the provider appears in the model menu straight away. Add as many as you like. The slug becomes the wire id: myproxy → custom_myproxy, which is what lands on model objects, spans and saved workflows. It is fixed once created, because renaming it would orphan every model reference in a saved graph. The endpoint and key are stored per account and encrypted, and they also read from the environment under the same names — so a server can be configured with no database writes at all: CUSTOM_MYPROXY_BASE_URL=https://proxy.example.com/v1 CUSTOM_MYPROXY_API_KEY=sk-… Include the version path the endpoint serves (usually /v1) in the URL, the same way you would set baseURL on an OpenAI client. Models come from GET <base_url>/models. Endpoints without that route take a hand-written list of model ids instead. An aggregator often lists chat, image and video models side by side. Each listed model goes to the matching node pickers: A vendor field decides first. A type, model_type, task, category or mode naming image or video counts, as does OpenRouter-style output_modalities. A model whose output is image and text stays a chat model. Otherwise the id decides. Family names such as flux, dall-e, seedream and imagen mean image, and kling, veo, sora and seedance mean video. A model classified only by its id also stays in the chat picker, so a misread chat model can still be picked. The Image models and Video models fields in the endpoint dialog override both, for ids the detection misses. They are stored in the catalog as image_models and video_models. Image models call POST <base_url>/images/generations and /images/edits with the requested size passed through unchanged. Video models call the OpenAI video API: POST <base_url>/videos, then poll until the clip is done. No resolution or duration limits are declared, so the node’s own options apply. Test reports how many chat, image and video models the endpoint serves, and runs on its own after each save. What Path Ids, secret names, validation packages/protocol/src/custom-providers.ts Chat, image and video classification packages/runtime/src/providers/custom-model-kinds.ts The provider packages/runtime/src/providers/custom-openai-provider.ts Registration packages/runtime/src/providers/custom-provider-registry.ts Storage + registry sync packages/websocket/src/custom-providers.ts API packages/websocket/src/trpc/routers/custom-providers.ts (customProviders.list\|save\|delete\|test) UI web/src/components/menus/CustomProvidersSection.tsx Write a provider file instead when the endpoint needs more than the OpenAI routes over a base URL: its own media API, non-standard request fields, per-model tool support or size limits, or a curated model list that ships with NodeTool. That is the rest of this guide. TL;DR Models appear automatically: each provider fetches its /models endpoint at runtime, so new upstream models show up in the model picker without a code change. Adding a whole new provider is three small edits: one constant in @nodetool-ai/protocol, one file of about 45 lines in packages/runtime/src/providers/, one registerBuiltinProvider call in the runtime index. Cohere is the only exception: it subclasses BaseProvider directly (embeddings only, no chat). Every other provider here subclasses OpenAICompatProvider, which owns the chat path. Moonshot (Kimi) uses the same pattern against its OpenAI-compatible endpoint, with its own /models fetch. Where things live What Path Provider ID constants packages/protocol/src/api-types.ts (PROVIDER_IDS const) Provider source files packages/runtime/src/providers/<name>-provider.ts Registration + exports packages/runtime/src/providers/index.ts Compat base class (chat, model listing helpers) packages/runtime/src/providers/openai-compat-provider.ts OpenAI base class (images, ASR, TTS, embeddings) packages/runtime/src/providers/openai-provider.ts Base class for all providers packages/runtime/src/providers/base-provider.ts Provider types (LanguageModel, etc.) packages/runtime/src/providers/types.ts The shared pattern DeepSeek is the canonical example. The whole file is about 45 lines. // packages/runtime/src/providers/deepseek-provider.ts import { OpenAICompatProvider, type OpenAICompatProviderOptions } from "./openai-compat-provider.js"; // (1) inherit chat, streaming, tools import type { LanguageModel } from "./types.js"; const DEEPSEEK_BASE_URL = "https://api.deepseek.com/v1"; // (2) provider's OpenAI-compatible base export class DeepSeekProvider extends OpenAICompatProvider { // (3) requiredSecrets() names the settings key the registry resolves at call time static override requiredSecrets(): string[] { return ["DEEPSEEK_API_KEY"]; } constructor( secrets: { DEEPSEEK_API_KEY?: string }, options: OpenAICompatProviderOptions = {} // (4) fetchFn, client, compatClient for tests ) { const apiKey = secrets.DEEPSEEK_API_KEY; if (!apiKey) { throw new Error("DEEPSEEK_API_KEY is required"); } // (5) providerId labels spans and model objects; baseURL is the chat endpoint root super( { providerId: "deepseek", apiKey, baseURL: DEEPSEEK_BASE_URL }, options ); } // (6) export the provider-specific key to Docker/subprocess environments override getContainerEnv() { return { DEEPSEEK_API_KEY: this.apiKey }; } // (7) true when the API supports function/tool calling; override per-model if mixed override async hasToolSupport(_model: string): Promise<boolean> { return true; } // (8) GET <baseURL>/models, mapped to LanguageModel rows tagged with providerId. // A failed request is logged and answers [] so one broken provider does not block the model menu. override async getAvailableLanguageModels(): Promise<LanguageModel[]> { return this.listCompatModels(); } } The config object also takes defaultHeaders, sent on every chat request and /models fetch. OpenAICompatProvider extends OpenAIProvider, so image, ASR, TTS, and embedding surfaces are inherited. Those return empty lists unless the subclass declares its own catalog (see servesOpenAICatalog() in the OpenAI guide). Groq, Mistral, Cerebras, Moonshot, and Alibaba Cloud follow this pattern. Differences: Alibaba Cloud reads DASHSCOPE_API_KEY. The base URL defaults to the international DashScope endpoint, https://dashscope-intl.aliyuncs.com/compatible-mode/v1. A key from another region needs DASHSCOPE_BASE_URL set to that region’s /compatible-mode/v1 URL. hasToolSupport returns false for families DashScope excludes from function calling (qwen-math, qvq, qwen-vl-ocr, qwen-mt, audio, TTS, and embedding ids). The provider sends max_tokens in place of max_completion_tokens. Groq annotates request failures with a token estimate and detects context-exceeded errors (groq-request.ts). Moonshot reads KIMI_API_KEY, targets https://api.moonshot.ai/v1, and fetches /models itself. OpenRouter sends attribution headers (HTTP-Referer, X-Title) through defaultHeaders. It adds image, video, TTS, ASR, and embedding methods, and hasToolSupport returns false for ids containing o1 or o3. Requesty sends the same attribution headers and builds its model list from /models/managed followed by /models, keeping only rows whose api is "chat". It reads supports_tool_calling per model, and only an explicit false disables tools. Opper builds its model list from /models?type=pool followed by /models, keeping only rows whose opper.type is "llm". It reads opper.capabilities per model, and only a listed model without tools disables tools. API Route reads /models once and keeps rows whose supported_endpoint_types include openai, plus rows without endpoint metadata. The shared classifyListedModel helper excludes image/video entries by metadata or ID, even when their endpoint types also include openai. Bare chat IDs are preserved. Add a brand-new OpenAI-compatible provider Step 1 — Add the provider ID to @nodetool-ai/protocol Open packages/protocol/src/api-types.ts and add a key to PROVIDER_IDS: // packages/protocol/src/api-types.ts export const PROVIDER_IDS = { // ... existing entries ... ACME: "acme", // wire id — used in model objects, spans, settings } as const; The comment above the const says: “Adding a provider? Add it here first, then register it in @nodetool-ai/runtime’s provider index.” Follow that order. Build the protocol package so downstream packages see the new constant: cd packages/protocol && npm run build Step 2 — Write acme-provider.ts Create packages/runtime/src/providers/acme-provider.ts. Copy the DeepSeek template above and substitute: Placeholder Replace with DeepSeek Acme DEEPSEEK_API_KEY ACME_API_KEY DEEPSEEK_BASE_URL / https://api.deepseek.com/v1 ACME’s base URL "deepseek" (the wire id string) "acme" If the provider’s /models response uses a different shape (not { data: [{ id, name }] }), replace listCompatModels() with your own parser. fetchCompatModelRows(url) returns the raw rows when you need vendor fields. If the provider does not support tool calling for some or all models, implement hasToolSupport to return false where appropriate (see the OpenRouter implementation for a model-name heuristic). Step 3 — Export and register in packages/runtime/src/providers/index.ts Two lines: // near the other thin-provider imports import { AcmeProvider } from "./acme-provider.js"; // near the other exports export { AcmeProvider }; And one registerBuiltinProvider call in the registration block, before the local-only if (!_cloudProfile) section: registerBuiltinProvider(PROVIDER_IDS.ACME, AcmeProvider, { ACME_API_KEY: "" }); The empty string for ACME_API_KEY is intentional — it forces the registry to resolve the key from the DB or env on every getProvider() call rather than baking a stale value at module load. Adjust an existing provider’s models Chat models are fetched dynamically — no change needed If Groq, DeepSeek, Mistral, Cerebras, Alibaba Cloud, OpenRouter, Requesty, Opper, or API Route adds a new chat model, it appears in the model picker automatically on the next call to getAvailableLanguageModels(). No code change is required. Tool-support overrides hasToolSupport(model: string) controls whether the UI offers tool/function calling for a given model. Groq, Mistral, Cerebras, DeepSeek, Moonshot, and API Route return true unconditionally. OpenRouter, Alibaba Cloud, Requesty, and Opper decide per model. OpenRouter checks the model id: // packages/runtime/src/providers/openrouter-provider.ts override async hasToolSupport(model: string): Promise<boolean> { const lower = model.toLowerCase(); if (lower.includes("o1") || lower.includes("o3")) return false; return true; } Apply the same pattern to any other provider where some models lack tool support. Embedding models (Mistral) Mistral is the only provider here that subclasses OpenAICompatProvider and also lists embeddings (OpenRouter lists embeddings from its own catalog). The model list is static (one entry, mistral-embed) and lives in getAvailableEmbeddingModels() in packages/runtime/src/providers/mistral-provider.ts. To add an embedding model, append to that array. Cohere — embeddings-only, different base class Cohere subclasses BaseProvider directly, not OpenAICompatProvider. It has no chat support. Its embedding model list is the static COHERE_EMBEDDING_MODELS array at the top of packages/runtime/src/providers/cohere-provider.ts. Append there to add a new Cohere embedding model. Moonshot Moonshot fetches GET https://api.moonshot.ai/v1/models live, so new Kimi models need no code change. Verify After any change: 1. Build protocol if you changed api-types.ts. cd packages/protocol && npm run build 2. Type-check. npm run typecheck 3. Confirm the provider returns models (requires the API key in secrets or env). npm run dev:nodetool -- models by-provider acme --kind llm Or run a single chat to exercise the full path: npm run dev:chat -- --provider acme --model <model-id> 4. Single-node smoke test (chat node via the provider). npm run dev:nodetool -- node run nodetool.agents.Agent \ --props '{"prompt":"hello","model":{"type":"language_model","provider":"acme","id":"<model-id>"}}' 5. Affected tests, lint, and the harness gate. npm run test:affected npm run lint npm run dev:nodetool -- harness gate --base origin/main Contributing Repository: https://github.com/nodetool-ai/nodetool Discord: https://discord.gg/WmQTWZRcYE Run the Verify steps above before opening a PR. Follow docs/WRITING_STYLE.md for any Markdown you touch.--- # OpenAI: Add Models Source: https://docs.nodetool.ai/developer/providers/openai.html NodeTool’s OpenAI integration lives in one file: packages/runtime/src/providers/openai-provider.ts. The provider is registered under PROVIDER_IDS.OPENAI = "openai" in packages/protocol/src/api-types.ts. Audience: coding agents and contributors adding new OpenAI models. TL;DR Model type What to do Chat / LLM (e.g. gpt-5.6) Nothing. The list is fetched live from https://api.openai.com/v1/models and filtered to the gpt-5 family. Add the id to OPENAI_FALLBACK_MODELS if it should show offline Image (e.g. gpt-image-3) Add one entry to getAvailableImageModels() and a pricing rule in cost-calculator.ts Video (e.g. sora-3) Add one entry to getAvailableVideoModels() TTS / ASR / Embedding Add one entry to the matching static getter Where things live Concern Path Notes Provider class packages/runtime/src/providers/openai-provider.ts All model lists and API calls Provider ID constant packages/protocol/src/api-types.ts PROVIDER_IDS.OPENAI = "openai" Provider registration packages/runtime/src/providers/index.ts Imports and re-exports OpenAIProvider Non-token pricing tiers packages/runtime/src/providers/cost-calculator.ts PRICING_TIERS object Per-model tier mapping packages/runtime/src/providers/cost-calculator.ts MODEL_TO_TIER object Quality-based image cost packages/runtime/src/providers/cost-calculator.ts gptImageQualityTier(), gptImage25TokenCost(), calculateImageCost() How OpenAI models are defined Chat / LLM models — dynamic getAvailableLanguageModels() makes a live GET request to https://api.openai.com/v1/models using the stored API key, then maps the returned ids to LanguageModel objects: async getAvailableLanguageModels(): Promise<LanguageModel[]> { const isOpenAI = this.provider === PROVIDER_IDS.OPENAI; const response = await this._fetch("https://api.openai.com/v1/models", { headers: { Authorization: `Bearer ${this.apiKey}` } }); if (!response.ok) return isOpenAI ? OPENAI_FALLBACK_MODELS : []; let models: LanguageModel[]; try { models = decodeOpenAIModelList(await response.json(), { provider: this.provider, onlyResponsesModels: isOpenAI }); } catch { return isOpenAI ? OPENAI_FALLBACK_MODELS : []; } return models.length === 0 && isOpenAI ? OPENAI_FALLBACK_MODELS : models; } Two filters shape the result. isOpenAIResponsesModel() keeps only the gpt-5 family — everything older (gpt-4o, gpt-4.1, the o-series) is retired on the openai provider, and audio/realtime/transcribe variants are excluded because the Responses API doesn’t serve them. OpenAI-compatible subclasses set their own provider id, skip the filter, and keep their full catalog. When the request fails or returns nothing, the provider falls back to OPENAI_FALLBACK_MODELS: gpt-6-astra, gpt-5.6 (plus -sol, -terra, -luna), gpt-5.5, gpt-5.5-pro, gpt-5.4 with its -pro/-mini/-nano tiers, and gpt-5, gpt-5-mini, gpt-5-nano. Add new releases there so they show up before a successful live fetch. A response body without a data array also triggers the fallback. A new gpt-5-family chat model released by OpenAI appears in NodeTool automatically the next time the model list is refreshed — no code change needed. Tool support is controlled by hasToolSupport(): async hasToolSupport(model: string): Promise<boolean> { if (this.usesResponsesApi(model)) return true; return !(model.startsWith("o1") || model.startsWith("o3")); } Responses-API models always support tools. If a Chat Completions model on a compatible subclass needs to opt out, add its prefix here. Image, video, TTS, ASR, and embedding models — static These model types are returned from hardcoded lists inside the provider because OpenAI’s /v1/models endpoint does not distinguish modalities. Each getter returns an array of typed objects. getAvailableImageModels() lists gpt-image-2.5-flare, gpt-image-2.5-sunburst, gpt-image-2, gpt-image-1.5, gpt-image-1, and gpt-image-1-mini. getAvailableVideoModels() lists sora-2 and sora-2-pro, with the durations, resolutions, and aspect ratios each accepts. getAvailableTTSModels() (gpt-4o-mini-tts, tts-1, tts-1-hd), getAvailableASRModels() (gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize, whisper-1), and getAvailableEmbeddingModels() (text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002) follow the same pattern. The OpenAI-compatible subclasses inherit this class but not its lineup. Every static getter returns an empty list unless servesOpenAICatalog() is true, which holds only for the openai provider id. Subclasses that serve media (Together, MiniMax, xAI, and others) override the getters with their own catalog. Add a new image model 1. Add the entry to getAvailableImageModels() Open packages/runtime/src/providers/openai-provider.ts and add to the array returned by the method. Match the shape of existing entries exactly: // packages/runtime/src/providers/openai-provider.ts async getAvailableImageModels(): Promise<ImageModel[]> { if (!this.servesOpenAICatalog()) return []; return [ { id: "gpt-image-3", // OpenAI API model ID name: "GPT Image 3", // display name shown in the UI provider: "openai", supportedTasks: ["text_to_image", "image_to_image"] }, // ... existing entries ... ]; } supportedTasks is a free string array. Existing image entries use "text_to_image" and "image_to_image". use whichever the model actually supports. 2. Add pricing if quality-based CostCalculator.calculate() resolves image models before the generic tier lookup. gptImageQualityTier() matches the gpt-image-1 family (gpt-image-1 and gpt-image-1-mini, via /^gpt-image-1(?:$|-)/) when an image count is reported and routes it through quality tiers in PRICING_TIERS. The default quality is medium. gptImage25TokenCost() prices gpt-image-2.5-flare and gpt-image-2.5-sunburst per token with rates written inline. calculateImageCost() is a thin wrapper over calculate(). // cost-calculator.ts PRICING_TIERS imageGptLow: { costType: CostType.IMAGE_BASED, perImage: 0.011 }, imageGptMedium: { costType: CostType.IMAGE_BASED, perImage: 0.042 }, imageGptHigh: { costType: CostType.IMAGE_BASED, perImage: 0.167 }, If gpt-image-3 uses the same three-tier structure at different prices, add new tiers and extend the qualityMap inside gptImageQualityTier(): // cost-calculator.ts PRICING_TIERS — add new tiers imageGpt3Low: { costType: CostType.IMAGE_BASED, perImage: 0.015 }, imageGpt3Medium: { costType: CostType.IMAGE_BASED, perImage: 0.060 }, imageGpt3High: { costType: CostType.IMAGE_BASED, perImage: 0.200 }, Then branch on the model id before the existing regex test: // cost-calculator.ts gptImageQualityTier() if (lower.includes("gpt-image-3")) { const qualityMap = { low: "imageGpt3Low", medium: "imageGpt3Medium", high: "imageGpt3High" }; // ... } If the model is flat-rate (no quality levels), add it to MODEL_TO_TIER instead: // cost-calculator.ts MODEL_TO_TIER "openai:gpt-image-3": "imageGptMedium", // or a new tier 3. Add pricing for non-image modalities (TTS / ASR) New TTS or ASR models follow the same pattern as existing entries in MODEL_TO_TIER. Add "openai:<model-id>": "<tierName>" and, if needed, a new tier object in PRICING_TIERS. Add a new chat / LLM model Usually you do not need to do anything. The live fetch covers all models in your account. Check the model is available in your tier at https://platform.openai.com/docs/models. A code change is needed only for these cases: Disable tool use for a new reasoning model prefix — extend hasToolSupport() in openai-provider.ts: async hasToolSupport(model: string): Promise<boolean> { if (this.usesResponsesApi(model)) return true; return !( model.startsWith("o1") || model.startsWith("o3") // add new prefix here ); } Token-based pricing — chat models are priced via @pydantic/genai-prices (imported in cost-calculator.ts). That package is community-maintained and tracks OpenAI pricing. If a brand-new model is missing from the catalog, wait for a @pydantic/genai-prices release or pin an interim entry by adding a dummy MODEL_TO_TIER key that maps to an existing tier. Verify Run these in order after any edit: # 1. Type-check the runtime package (and all packages) npm run typecheck # 2. Smoke-test the model list (requires OPENAI_API_KEY in environment) npm run dev:nodetool -- models by-provider openai --kind llm npm run dev:nodetool -- models by-provider openai --kind image # 3. Smoke-test a single image node (no secrets needed for type check) npm run dev:nodetool -- node run nodetool.image.TextToImage \ --props '{"prompt": "a red circle", "model": {"type": "image_model", "id": "gpt-image-1", "provider": "openai"}}' \ --no-secrets # 4. Affected tests, lint, and the harness gate npm run test:affected npm run lint npm run dev:nodetool -- harness gate --base origin/main All of these must pass before committing. Contributing Open a PR at https://github.com/nodetool-ai/nodetool. Before pushing: npm run test:affected npm run typecheck npm run lint Join the discussion on Discord.--- # Replicate: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/replicate.html Audience: Coding agents and contributors adding Replicate models to NodeTool. This guide assumes familiarity with the monorepo structure described in AGENTS.md. TL;DR Add the model’s "owner/name" entry to the matching config file in packages/replicate-codegen/src/configs/. Export REPLICATE_API_TOKEN and run npm run generate:replicate. Run npm run build:packages (replicate-nodes loads from dist/). Verify with npm run dev:nodetool -- node run replicate.<module>.<ClassName> --props '{...}'. Do not hand-edit packages/replicate-nodes/src/replicate-manifest.json — it is generated output. Where things live Path Purpose packages/replicate-codegen/src/configs/ One TypeScript file per module (e.g. image-generate.ts). Edit these to add models. packages/replicate-codegen/src/configs/index.ts Imports every module config and exports allConfigs. Register new config files here. packages/replicate-codegen/src/generate.ts CLI entry point for the generator. Fetches Replicate schemas and writes the manifest. packages/replicate-codegen/src/schema-fetcher.ts Fetches/caches OpenAPI schemas from api.replicate.com. Cache lives in .schema-cache/replicate/. packages/replicate-codegen/src/schema-parser.ts Parses the OpenAPI schema into a NodeSpec. packages/replicate-codegen/src/node-generator.ts Applies NodeConfig overrides (className, returnType, fieldOverrides) onto a NodeSpec. packages/replicate-codegen/src/types.ts ModuleConfig, NodeConfig, NodeSpec, FieldDef, EnumDef. packages/replicate-nodes/src/replicate-manifest.json Generated. Node definitions consumed at runtime. Never edit by hand. packages/replicate-nodes/src/replicate-factory.ts Builds NodeClass instances from manifest entries at runtime. packages/replicate-nodes/src/replicate-base.ts Shared Replicate API utilities: replicateSubmit, assetToUrl, output converters. packages/replicate-nodes/src/index.ts Exports REPLICATE_NODES (array of NodeClass) and registerReplicateNodes. packages/runtime/src/providers/replicate-provider.ts ReplicateProvider — chat, image, video, audio, TTS, ASR, embeddings. packages/runtime/src/providers/manifest-models.ts Reads the manifest to surface image/video/music model lists to the UI. packages/runtime/src/providers/index.ts Registers ReplicateProvider under PROVIDER_IDS.REPLICATE. How Replicate nodes are generated The codegen pipeline runs in four steps: Config lookup — generate.ts reads allConfigs from configs/index.ts. Each config maps a Replicate model id ("owner/name") to a NodeConfig (class name, return type, field overrides). Schema fetch — SchemaFetcher calls https://api.replicate.com/v1/models/<owner>/<name> to get the model’s OpenAPI input/output schema. Results are cached under .schema-cache/replicate/ (keyed by SHA-256 of the model id). Pass --no-cache to force a fresh fetch. Parsing and override — SchemaParser converts the OpenAPI schema to a NodeSpec. NodeGenerator.applyConfig() then merges the NodeConfig overrides onto it (renames class, sets outputType, fixes propType for image/audio/video fields that Replicate types as string). Manifest write — The generator writes every NodeSpec as a ManifestEntry to packages/replicate-nodes/src/replicate-manifest.json. At runtime, replicate-factory.ts reads this file and produces NodeClass objects via createReplicateNodeClass(). The node type for a manifest entry follows the pattern replicate.<moduleName>.<className> where moduleName uses dots instead of dashes (e.g. replicate.image.generate.Flux_Schnell). Add a new Replicate model Step 1 — Choose the right config file Pick the config whose module matches the model’s task: File Module key Covers image-generate.ts image.generate Text-to-image, image-to-image image-analyze.ts image.analyze Captioning and image analysis image-ocr.ts image.ocr OCR image-face.ts image.face Face restoration and swap image-process.ts image.process Other image processing image-3d.ts image.3d Image to 3D (returnType: "str") image-enhance.ts image.enhance Enhancement, retouching image-upscale.ts image.upscale Upscaling image-background.ts image.background Background removal video-generate.ts video.generate Text/image to video video-enhance.ts video.enhance Video enhancement video-face.ts video.face Video face swap and restoration video-process.ts video.process Other video processing audio-generate.ts audio.generate Music, sound generation audio-enhance.ts audio.enhance Audio enhancement audio-separate.ts audio.separate Stem separation audio-speech.ts audio.speech TTS audio-transcribe.ts audio.transcribe ASR text-generate.ts text.generate Text LLMs via Replicate embedding.ts embedding Embedding models If none fit, create a new file (see Step 1b below). Step 2 — Add the model entry Add one key to the configs object in the chosen file. The key is the Replicate model id exactly as it appears in the URL (replicate.com/<owner>/<name>). Minimal entry (no image inputs): "acme-org/my-flux-variant": { className: "MyFluxVariant", returnType: "image" } Entry with image inputs (Replicate types these as string; NodeTool needs image/video/audio): "acme-org/my-img2img": { className: "MyImg2Img", returnType: "image", fieldOverrides: { image: { propType: "image" }, mask: { propType: "image" } } } Full NodeConfig shape (all fields optional): interface NodeConfig { className?: string; // PascalCase; if omitted the parser derives one docstring?: string; // Override the model description tags?: string[]; useCases?: string[]; returnType?: string; // "image" | "video" | "audio" | "str" fieldOverrides?: Record<string, Partial<FieldDef>>; // propType corrections enumOverrides?: Record<string, string>; // rename enum classes enumValueOverrides?: Record<string, Record<string, string>>; } returnType is required for media-producing models. Without it the factory cannot map the output to the right AssetRef type. Use "str" for outputs that are not an asset, such as a 3D mesh URL. Asset input propType rules — Replicate schemas type all URLs as string. Override these manually: Field carries Set propType to Single image "image" Single video "video" Single audio "audio" List of images "list[image]" Step 1b (optional) — Create a new config file If no existing module fits, create packages/replicate-codegen/src/configs/<domain>-<task>.ts: // packages/replicate-codegen/src/configs/image-depth.ts import type { ModuleConfig } from "../types.js"; export const imageDepthConfig: ModuleConfig = { configs: { "acme-org/depth-map": { className: "DepthMap", returnType: "image" } } }; Then register it in packages/replicate-codegen/src/configs/index.ts: import { imageDepthConfig } from "./image-depth.js"; // ... export const allConfigs: Record<string, ModuleConfig> = { // ...existing entries... "image.depth": imageDepthConfig, }; The module key ("image.depth") becomes the moduleName in the manifest (stored as "image-depth") and the middle segment of every node’s type string (replicate.image.depth.<ClassName>). To regenerate one module, run npm run generate --workspace=packages/replicate-codegen -- --module image-depth --strict. Verify Run these in order after adding the config entry: # 1. Generate the manifest (fetches from Replicate API; uses cache by default) export REPLICATE_API_TOKEN="r8_..." npm run generate:replicate # 2. Regenerate without cache if the model schema was recently updated REPLICATE_API_TOKEN="r8_..." npm run generate --workspace=packages/replicate-codegen -- --all --strict --no-cache # 3. Build — replicate-nodes loads from dist/, so rebuild is required npm run build:packages # 4. Type-check the codegen package npm run lint --workspace=packages/replicate-codegen # 5. Run codegen tests npm run test --workspace=packages/replicate-codegen # 6. Smoke-test the node (confirm it appears in the registry and properties look right) npm run dev:nodetool -- node run replicate.image.generate.MyFluxVariant \ --props '{"prompt": "a red fox"}' --no-secrets --json # 7. Affected tests, lint, and the harness gate npm run test:affected npm run lint npm run dev:nodetool -- harness gate --base origin/main If the model id is wrong or the model is private/unavailable, the generator exits with an error listing the failures. Fix the id or remove the entry. Contributing Open a PR against main at github.com/nodetool-ai/nodetool. Before pushing: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main Questions? Join the Discord.--- # Reve: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/reve.html Navigation: docs/developer/index.md → Reve provider guide Audience: coding agents and contributors adding new Reve image nodes or model version strings to the @nodetool-ai/reve-nodes package. TL;DR Add a new class in packages/reve-nodes/src/nodes/ extending BaseNode. Export it from packages/reve-nodes/src/index.ts and push the class into REVE_NODES. Call reveGenerate from reve-base.ts with one of the three API endpoints: "create", "edit", or "remix". npm run build:packages — compiles from src/ to dist/ (the package loads from dist/). npm run typecheck && npm run lint — must pass before committing. Where things live What Path Shared API utilities packages/reve-nodes/src/reve-base.ts Create Image node packages/reve-nodes/src/nodes/create-image.ts Edit Image node packages/reve-nodes/src/nodes/edit-image.ts Remix Image node packages/reve-nodes/src/nodes/remix-image.ts Package entry point packages/reve-nodes/src/index.ts Runtime provider (generic image picker) packages/runtime/src/providers/reve-provider.ts Provider registration packages/runtime/src/providers/index.ts (registerBuiltinProvider(PROVIDER_IDS.REVE, ...)) PROVIDER_IDS.REVE packages/protocol/src/api-types.ts Unit tests packages/reve-nodes/tests/nodes.test.ts, reve-base.test.ts Registration test packages/reve-nodes/tests/registration.test.ts How Reve models and nodes are defined Reve exposes three POST endpoints, each mapping to one TypeScript class: Endpoint Class Node type POST /v1/image/create CreateImageNode reve.CreateImage POST /v1/image/edit EditImageNode reve.EditImage POST /v1/image/remix RemixImageNode reve.RemixImage There is no manifest or code-generation script. Each node is a hand-written BaseNode subclass in its own file under packages/reve-nodes/src/nodes/. Fields are declared with the @prop decorator from @nodetool-ai/node-sdk. Shared utilities in reve-base.ts: REVE_ASPECT_RATIOS — the aspect ratios all endpoints accept. REVE_POSTPROCESSING — optional post-generation operations (upscale, remove_background, fit_image, effect). getReveApiKey(secrets) — reads REVE_API_KEY from this._secrets then process.env, throws if absent. reveGenerate(apiKey, endpoint, body) — POSTs to https://api.reve.com/v1/image/{endpoint} with Authorization: Bearer, returns the parsed ReveImageResponse. refToBase64(ref, context) — converts an ImageRef-like value to a base64 string; handles inline data, data URIs, asset://, file://, and remote URLs. reveImageToRef(base64) — wraps the response image field into a NodeTool ImageRef object; attaches dimensions via sharp when available. The runtime provider (reve-provider.ts) separately implements textToImage and imageToImage for the generic image-generation picker in the UI (the two hardcoded model ids reve-create and reve-edit). This is a thinner path — new Reve API endpoints should be added as full nodes in reve-nodes, not as branches in the provider. Add a new model version To add a new dated model version pin (e.g. reve-create@20260101), open the relevant node file and extend the version prop’s values array: // packages/reve-nodes/src/nodes/create-image.ts @prop({ type: "enum", default: "latest", title: "Version", description: "Model version to use.", values: ["latest", "reve-create@20250915", "reve-create@20260101"] // ← add here }) declare version: any; That is the entire change. Rebuild and verify (see Verify below). Add a new node Use this pattern when Reve releases a new endpoint (or a sufficiently different variant that warrants its own node type). Step 1. Create packages/reve-nodes/src/nodes/your-endpoint.ts: import { BaseNode, prop } from "@nodetool-ai/node-sdk"; import type { NodeClass } from "@nodetool-ai/node-sdk"; import { getReveApiKey, postprocessingArray, reveGenerate, reveImageToRef, REVE_ASPECT_RATIOS, REVE_POSTPROCESSING } from "../reve-base.js"; export class YourEndpointNode extends BaseNode { static readonly nodeType = "reve.YourEndpoint"; static readonly body = "content_card"; static readonly title = "Your Endpoint"; static readonly description = "One-line description of what this node does.\n" + "image, reve, keyword1, keyword2\n\n" + "Use cases:\n" + "- First use case\n" + "- Second use case"; static readonly metadataOutputTypes = { output: "image" }; static readonly inlineFields: string[] = ["prompt"]; static readonly requiredSettings = ["REVE_API_KEY"]; static readonly autoSaveAsset = true; @prop({ type: "str", default: "", title: "Prompt", description: "Text prompt (max 2560 chars)." }) declare prompt: any; @prop({ type: "enum", default: "3:2", title: "Aspect Ratio", description: "Proportions of the generated image.", values: [...REVE_ASPECT_RATIOS] }) declare aspect_ratio: any; @prop({ type: "enum", default: "latest", title: "Version", description: "Model version to use.", values: ["latest"] }) declare version: any; @prop({ type: "enum", default: "none", title: "Postprocessing", description: "Optional postprocessing operation applied to the result.", values: [...REVE_POSTPROCESSING] }) declare postprocessing: any; @prop({ type: "int", default: 1, title: "Test Time Scaling", description: "Effort multiplier for quality (1-15). Higher costs more.", min: 1, max: 15 }) declare test_time_scaling: any; async process(): Promise<Record<string, unknown>> { const apiKey = getReveApiKey(this._secrets); const prompt = String(this.prompt ?? ""); if (!prompt) throw new Error("Prompt is required"); const result = await reveGenerate(apiKey, "create", { // ← swap endpoint name prompt, aspect_ratio: String(this.aspect_ratio ?? "3:2"), version: String(this.version ?? "latest"), postprocessing: postprocessingArray(this.postprocessing), test_time_scaling: Number(this.test_time_scaling ?? 1) }); return { output: await reveImageToRef(result.image) }; } } export const YOUR_ENDPOINT_NODES: readonly NodeClass[] = [YourEndpointNode]; Nodes that accept a reference image (like EditImageNode) must take a context argument in process and pass it to refToBase64: async process( context?: Parameters<BaseNode["process"]>[0] ): Promise<Record<string, unknown>> { const referenceImage = await refToBase64(this.image, context); // ... } Step 2. Wire the new class into packages/reve-nodes/src/index.ts: import { YOUR_ENDPOINT_NODES } from "./nodes/your-endpoint.js"; export { YourEndpointNode } from "./nodes/your-endpoint.js"; export const REVE_NODES: readonly NodeClass[] = [ ...CREATE_IMAGE_NODES, ...EDIT_IMAGE_NODES, ...REMIX_IMAGE_NODES, ...YOUR_ENDPOINT_NODES // ← add here ]; No other files need to change. The node automatically appears in the registry because registerReveNodes iterates REVE_NODES. Step 3. Add tests in packages/reve-nodes/tests/nodes.test.ts. Follow the existing pattern: stub fetch, construct the node with props, call process(), assert the fetch URL and body fields. Verify Run these commands in order after any change: # 1. Compile src/ → dist/ (required — the package loads from dist/) npm run build:packages # 2. Type-check the whole repo npm run typecheck # 3. Lint npm run lint # 4. Run the reve-nodes unit tests npm run test --workspace=packages/reve-nodes # 5. Confirm the new node type resolves in the CLI registry # Expected: a REVE_API_KEY error, not "unknown node type" npm run dev:nodetool -- node run reve.YourEndpoint \ --props '{"prompt":"a red fox"}' \ --no-secrets # 6. Static validation (no API key required) npm run dev:nodetool -- validate workflow.json # 7. Affected tests and the harness gate npm run test:affected npm run dev:nodetool -- harness gate --base origin/main A REVE_API_KEY is not configured error from step 5 means the node registered correctly — it loaded from dist/ and resolved the type. An unknown node type error means the build or export wiring is broken. Contributing Source: https://github.com/nodetool-ai/nodetool Discord: https://discord.gg/WmQTWZRcYE Before opening a PR, run the Verify steps. Add a test for every new node; the existing nodes.test.ts stubs fetch globally, so new test cases fit cleanly in the same file.--- # Together AI: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/together.html Audience: Contributors and coding agents adding Together AI models or nodes to this repo. TL;DR Edit IMAGE_MODELS, VIDEO_MODELS, TTS_MODELS, or ASR_MODELS in scripts/generate-manifest.mjs. Run npm run gen:manifest — rewrites src/together-manifest.json. Run npm run build in the package, then npm run lint && npm run test (lint is tsc --noEmit). For TTS and ASR models, also update the matching static array in packages/runtime/src/providers/together-provider.ts. Embedding models exist only there. Verify with nodetool node run together.<module>.<ClassName>. Where things live What Path Manifest generator (source of truth) packages/together-nodes/scripts/generate-manifest.mjs Generated manifest (do not hand-edit) packages/together-nodes/src/together-manifest.json Node factory (manifest → BaseNode subclasses) packages/together-nodes/src/together-factory.ts HTTP helpers / API executors packages/together-nodes/src/together-base.ts Node registration entry point packages/together-nodes/src/index.ts Provider class (chat, image, video, TTS, ASR, embeddings) packages/runtime/src/providers/together-provider.ts Manifest→model-list bridge packages/runtime/src/providers/manifest-models.ts Provider registration packages/runtime/src/providers/index.ts (registerBuiltinProvider(PROVIDER_IDS.TOGETHER, ...)) Pack registration (registerTogetherNodes) packages/base-nodes/src/builtin-packs.ts Pack catalog entry packages/protocol/src/builtin-packs.ts PROVIDER_IDS.TOGETHER constant packages/protocol/src/api-types.ts How Together models and nodes are defined Image and video nodes — generated from the manifest src/together-manifest.json is generated code. The source of truth is scripts/generate-manifest.mjs, which defines four model catalogs (IMAGE_MODELS, VIDEO_MODELS, TTS_MODELS, ASR_MODELS) and expands each (model × task) pair into a manifest entry (image, video, audio/text_to_speech, and transcription/automatic_speech_recognition entries). Each entry looks like this: { "className": "FLUX1SchnellTextToImage", "moduleName": "image", "modality": "text_to_image", "modelId": "black-forest-labs/FLUX.1-schnell", "outputType": "image", "title": "FLUX.1 Schnell — Text to Image", "description": "FLUX.1 Schnell — text to image via Together AI serverless.", "fields": [ ... ], "supportedTasks": ["text_to_image"] } The factory (together-factory.ts) turns every entry into a BaseNode subclass at runtime. The node type is together.<moduleName>.<className>. Fields declared in "fields" become node properties; the "modality" value routes execution to the matching HTTP executor in together-base.ts. The same manifest is read by manifest-models.ts (via loadImageModels / loadVideoModels) to populate TogetherProvider.getAvailableImageModels() and getAvailableVideoModels(). This means the node catalog and the provider’s model catalog stay in sync automatically — there is no second list to update for image and video models. Chat / language models — fetched dynamically TogetherProvider.getAvailableLanguageModels() calls https://api.together.xyz/v1/models at runtime and filters entries where type === "chat" || type === "language" || type === "". No static list exists for chat models; Together’s live catalog drives what appears in the model picker. No code change is required when Together adds a new chat model. TTS, ASR, and embedding models — static arrays in the provider These are hardcoded in packages/runtime/src/providers/together-provider.ts: TOGETHER_TTS_MODELS — 3 entries (Orpheus 3B, Kokoro 82M, Cartesia Sonic), each with a voices list. TOGETHER_ASR_MODELS — 3 entries (Whisper Large v3, Voxtral Mini 3B, Parakeet TDT 0.6B v3). TOGETHER_EMBEDDING_MODELS — 1 entry (Multilingual E5 Large, 1024 dims). TTS and ASR models are also in the manifest, so they get workflow nodes (together.audio.<Model>TextToSpeech and together.transcription.<Model>Transcribe). Embedding models appear only in the provider array and have no workflow node. Add a new model or node The steps differ slightly by modality. Image or video model 1. Open the generator and add an entry to the right catalog. // packages/together-nodes/scripts/generate-manifest.mjs const IMAGE_MODELS = [ // existing entries ... { id: "black-forest-labs/FLUX.3-ultra", // Together API model id name: "FLUX.3 Ultra", // display name → drives className tasks: ["text_to_image", "image_to_image"] // which node(s) to generate } ]; For video, append to VIDEO_MODELS with tasks: ["text_to_video", "image_to_video"] (or just one if the model is task-specific). 2. Regenerate and build. cd packages/together-nodes npm run gen:manifest # rewrites src/together-manifest.json npm run build # tsc + copies manifest into dist/ npm test That’s all. The factory auto-generates the node class, and manifest-models.ts picks up the new model for the provider’s model lists. TTS model TTS models need entries in both the manifest generator and the provider array. 1. Add to the generator’s TTS_MODELS. // packages/together-nodes/scripts/generate-manifest.mjs const TTS_MODELS = [ // existing entries ... { id: "vendor/new-tts-model", name: "New TTS Model", voices: ["voice_a", "voice_b"] } ]; 2. Regenerate and build (same as above). 3. Add the provider-side entry. // packages/runtime/src/providers/together-provider.ts const TOGETHER_TTS_MODELS: TTSModel[] = [ // existing entries ... { id: "vendor/new-tts-model", name: "New TTS Model", provider: "together", voices: ["voice_a", "voice_b"] } ]; ASR model ASR models also need entries in both places. Add { id, name } to ASR_MODELS in the generator, regenerate and build, then add the provider entry: // packages/runtime/src/providers/together-provider.ts const TOGETHER_ASR_MODELS: ASRModel[] = [ // existing entries ... { id: "vendor/new-asr-model", name: "New ASR Model", provider: "together" } ]; Embedding model Update only the provider array. The generator has no embedding catalog: // packages/runtime/src/providers/together-provider.ts const TOGETHER_EMBEDDING_MODELS: EmbeddingModel[] = [ // existing entries ... { id: "vendor/new-embedding-model", name: "New Embedding Model", provider: "together", dimensions: 1024 } ]; Verify 1. Build and type-check. npm run build --workspace=packages/together-nodes npm run typecheck 2. Run package tests. npm run test --workspace=packages/together-nodes 3. Exercise the node in isolation (requires TOGETHER_API_KEY in secrets or env). For a text-to-image node the type is together.image.<ClassName>: npm run dev:nodetool -- node run together.image.FLUX1SchnellTextToImage \ --props '{"prompt":"a red panda on a skateboard","width":512,"height":512}' For a video node: npm run dev:nodetool -- node run together.video.Veo30TextToVideo \ --props '{"prompt":"timelapse of clouds","aspect_ratio":"16:9","resolution":"720p","duration":6}' For a TTS node: npm run dev:nodetool -- node run together.audio.Orpheus3BTextToSpeech \ --props '{"text":"Hello from Together AI","voice":"tara","format":"mp3"}' 4. Static workflow check (no API call). npm run dev:nodetool -- validate workflow.json 5. Affected tests, lint, and the harness gate. npm run test:affected npm run lint npm run dev:nodetool -- harness gate --base origin/main Contributing Repository: https://github.com/nodetool-ai/nodetool Discord: https://discord.gg/WmQTWZRcYE Before opening a PR, run the Verify steps above. Follow the writing style in docs/WRITING_STYLE.md for any Markdown you touch.--- # Topaz: Add Models & Nodes Source: https://docs.nodetool.ai/developer/providers/topaz.html Navigation: docs/developer/index.md → Topaz provider guide Audience: coding agents and contributors adding new Topaz image or video endpoints, or new model variants within existing endpoints. TL;DR Edit packages/topaz-nodes/src/topaz-manifest.json — one JSON object per endpoint (node). npm run build:packages — copies the manifest to dist/ and compiles the factory. npm run typecheck && npm run lint — must pass before committing. No new TypeScript files needed for standard endpoints; the factory generates the node class at runtime. Where things live What Path Manifest (all nodes) packages/topaz-nodes/src/topaz-manifest.json API + HTTP utilities packages/topaz-nodes/src/topaz-base.ts Runtime node factory packages/topaz-nodes/src/topaz-factory.ts Package entry point packages/topaz-nodes/src/index.ts Runtime provider (upscale only) packages/runtime/src/providers/topaz-provider.ts Manifest loader (for ImageModel list) packages/runtime/src/providers/manifest-models.ts Provider registration packages/runtime/src/providers/index.ts Unit tests packages/topaz-nodes/tests/topaz-base.test.ts, topaz-factory.test.ts, packages/runtime/tests/providers/topaz-provider.test.ts How Topaz models and nodes are defined The manifest is the single source of truth. Every Topaz node is described by one JSON object in packages/topaz-nodes/src/topaz-manifest.json. There is no code-generation script. The manifest is hand-maintained. At package load time src/index.ts loads the JSON with loadPackageAssetJson and calls loadTopazNodesFromManifest, which feeds each entry to createTopazNodeClass in topaz-factory.ts. The factory builds a full BaseNode subclass at runtime — no decorators, no separate .ts file per node. The build script (package.json build) compiles the TypeScript and then copies src/topaz-manifest.json to dist/topaz-manifest.json so the package.json exports entry "./topaz-manifest.json" resolves correctly. The runtime provider (topaz-provider.ts) loads the same file through loadManifest in manifest-models.ts to build its list of ImageModel objects, but only exposes the enhance and enhance-gen endpoints as upscale-task models. All other endpoints (sharpen, sharpen-gen, denoise, denoise-gen, lighting, matting, restore-gen) live solely as workflow nodes. The manifest holds nine image endpoints and two video endpoints (EnhanceVideo, InterpolateVideo). Image nodes call a single multipart POST → poll status → download flow implemented in topazExecuteImageTask (topaz-base.ts). Video nodes drive a five-step pipeline: create job → accept (get presigned upload URLs) → PUT bytes → complete-upload → poll status → download. This requires ffprobe installed locally to probe the source video’s resolution, frame rate, and duration before the job is created. Add a new model or node Case A — new model variant in an existing endpoint Open packages/topaz-nodes/src/topaz-manifest.json and add the variant name to the values array of the relevant entry’s model field: { "name": "model", "type": "enum", "default": "Standard V2", "title": "Model", "required": false, "values": [ "Standard V2", "High Fidelity V2", "Upscale High Fidelity V3", "Low Resolution V2", "CGI", "Transparency Upscale", "Detail", "Faces", "Text Refine", "Standard V3" ] } That is the entire change. The factory picks up the new variant automatically. If the new variant is in the enhance or enhance-gen endpoint, the runtime provider will also surface it as a new ImageModel entry named topaz/image/enhance/Standard V3 (the modelId/variant pattern used in buildVariantMap). Case B — new image endpoint (new node) Add one object to the top-level array in topaz-manifest.json. Copy the shape of an existing image entry: { "className": "RemoveBackground", "moduleName": "image", "modelId": "topaz/image/remove-bg", "title": "Topaz Remove Background", "description": "Topaz Image API — background removal.\n\n topaz, image, background, removal\n\n Multipart POST /image/v1/remove-bg/async.", "outputType": "image", "submitEndpoint": "https://api.topazlabs.com/image/v1/remove-bg/async", "statusEndpoint": "https://api.topazlabs.com/image/v1/status/{process_id}", "downloadEndpoint": "https://api.topazlabs.com/image/v1/download/{process_id}", "pollInterval": 3000, "maxAttempts": 400, "fields": [ { "name": "image", "type": "image", "title": "Image", "description": "Source image.", "required": true, "uploadField": true }, { "name": "output_format", "type": "enum", "default": "png", "title": "Output Format", "required": false, "values": ["jpeg", "png", "tiff"] } ], "validation": [] } Key rules: className — PascalCase, unique across the manifest. Becomes the node class name and part of the node type: topaz.<moduleName>.<className>. modelId — slash-separated string, unique across the manifest. Used as the primary key in the provider’s variant map. Exactly one field must set "uploadField": true. This is the asset (image or video) the factory reads and sends to Topaz. submitEndpoint must be an absolute URL. statusEndpoint and downloadEndpoint use {process_id} as a placeholder. pollInterval is in milliseconds. maxAttempts × pollInterval sets the total timeout budget. Image entries use 3000 ms and 400 attempts. Video entries use 15000 ms and 1000 attempts. Case C — new video endpoint Copy either EnhanceVideo or InterpolateVideo from the manifest. Video entries add: "outputType": "video" "videoKind": "upscale" or "videoKind": "interpolate" "requiredRuntimes": ["ffmpeg"] — tells the UI the node needs ffmpeg/ffprobe installed "acceptEndpoint" and "completeEndpoint" for the two extra pipeline steps A "video" typed field with "uploadField": true The factory detects outputType === "video" and routes through topazExecuteVideoTask, which calls ffprobe to probe the source before creating the job. Verify After editing the manifest, run these commands in order: # 1. Build the package (compiles TS and copies manifest to dist/) npm run build:packages # 2. Type-check the whole repo npm run typecheck # 3. Lint npm run lint # 4. Run the topaz-nodes and provider unit tests npm run test --workspace=packages/topaz-nodes npm run test --workspace=packages/runtime -- topaz-provider # 5. Run a node in isolation to confirm it loads from the manifest # (replace 'topaz.image.SharpenImage' with your new className) npm run dev:nodetool -- node run topaz.image.SharpenImage \ --props '{"image":{"type":"image","uri":"https://example.com/photo.jpg"}}' \ --no-secrets # 6. Static validation: confirm the manifest-derived node type is known npm run dev:nodetool -- validate workflow.json Step 5 will hit a missing-API-key error in --no-secrets mode, which is expected — the goal is to confirm the node type resolves and all fields load without a panic. A TOPAZ_API_KEY not configured error means the node registered correctly. Contributing Source: https://github.com/nodetool-ai/nodetool Discord: https://discord.gg/WmQTWZRcYE Before opening a PR, run npm run test:affected, npm run typecheck, npm run lint, and npm run dev:nodetool -- harness gate --base origin/main. The manifest is the right place to document API-level details (endpoint URL, poll interval, field constraints) — keep the description field useful for future readers.--- # xAI (Grok): Add Models Source: https://docs.nodetool.ai/developer/providers/xai.html XAIProvider sits at packages/runtime/src/providers/xai-provider.ts. It extends OpenAICompatProvider, so chat uses xAI’s OpenAI-compatible Chat Completions endpoint. It overrides model discovery and the four image and video generation methods to call xAI’s REST API directly. Each modality lists the static XAI_KNOWN_MODELS catalog first, then any further models from xAI’s /v1/models listing. Audience: coding agents and contributors adding new xAI Grok models (chat, image generation, video generation). TL;DR To support a new model: Add it to the matching modality in XAI_KNOWN_MODELS with a display name. The catalog keeps the model listed when /v1/models omits it or cannot be reached. Update the KNOWN_*_IDS constants in packages/runtime/tests/providers/xai-provider.test.ts. If the id is ambiguous, check that classifyModel() assigns the live row the same modality. Otherwise the model appears twice, once under each modality. Run npm run test:affected, npm run typecheck, and npm run lint. xAI’s /v1/models rows carry no modality fields, so classifyModel() usually falls back to the model id. Models the listing returns that the catalog lacks are still offered, after the catalog entries. Where things live Concern Path Provider class packages/runtime/src/providers/xai-provider.ts Provider registration packages/runtime/src/providers/index.ts (registerBuiltinProvider(PROVIDER_IDS.XAI, XAIProvider, …)) Provider ID constant packages/protocol/src/api-types.ts (PROVIDER_IDS.XAI = "xai") Cloud profile node namespaces packages/protocol/src/cloud-profile.ts (CLOUD_NODE_NAMESPACES lists xai. NON_CLOUD_PROVIDER_IDS does not) Provider tests packages/runtime/tests/providers/xai-provider.test.ts Type definitions packages/runtime/src/providers/types.ts How xAI models are defined Each modality starts from its XAI_KNOWN_MODELS entries. The provider then adds rows from a single call to xAI’s /v1/models endpoint (fetchModelRows()) whose ids the catalog does not already contain. A failed listing yields the catalog alone. The private classifyModel() function sorts models into "language" | "image" | "video" by inspecting output_modalities first, then falling back to the model’s id string: // packages/runtime/src/providers/xai-provider.ts function classifyModel(row: XAIModelRow): ModelModality { const out = (row.output_modalities ?? []).map((m) => m.toLowerCase()); if (out.includes("video")) return "video"; if (out.includes("image")) return "image"; if (out.includes("text")) return "language"; const id = row.id.toLowerCase(); if (id.includes("video")) return "video"; if (id.includes("image")) return "image"; return "language"; // fallback } getAvailableLanguageModels(), getAvailableImageModels(), and getAvailableVideoModels() each call listModels(), which merges the catalog with the classified live rows. Image models are tagged supportedTasks: ["text_to_image", "image_to_image"]. Video models are tagged supportedTasks: ["text_to_video", "image_to_video"]. Both sets are returned with provider: "xai". Generation calls go directly to xAI’s REST API: Task Endpoint textToImage POST /v1/images/generations imageToImage POST /v1/images/edits textToVideo / imageToVideo POST /v1/videos/generations (async, polls /v1/videos/{request_id} until done, failed, or expired, with a 600 second default timeout) Image inputs are converted to base64 data URIs before sending — xAI’s JSON API rejects multipart uploads. Add a new model Chat (language) model Add the model to XAI_KNOWN_MODELS.language. A chat model that /v1/models returns is also picked up without a code change, after the catalog entries and under its raw id. To check what the live listing returns: curl -s https://api.x.ai/v1/models \ -H "Authorization: Bearer $XAI_API_KEY" | jq '.data[].id' Image model Add the model to XAI_KNOWN_MODELS.image. A live row reaches the image list only when output_modalities contains "image" or the id contains "image". For any other id, add a guard in classifyModel(): // packages/runtime/src/providers/xai-provider.ts — inside classifyModel() const id = row.id.toLowerCase(); if (id.includes("video")) return "video"; if (id.includes("image")) return "image"; if (id.includes("grok-imagine")) return "image"; // ← add this if needed return "language"; getAvailableImageModels() will then include the model with supportedTasks: ["text_to_image", "image_to_image"] and route calls through textToImage / imageToImage already in the provider. No supportedTasks override is needed unless xAI adds a task the model does not support (e.g., image editing); in that case narrow the array: // hypothetical future: image-only model (no editing) supportedTasks: ["text_to_image"] That change goes in getAvailableImageModels() after the classifyModel() filter, keyed on row.id. Video model Add the model to XAI_KNOWN_MODELS.video. A live row reaches the video list when its id contains "video" or output_modalities contains "video", it surfaces in getAvailableVideoModels() tagged supportedTasks: ["text_to_video", "image_to_video"]. The async polling loop in generateVideo() handles all video models uniformly. xAI video parameters: duration: 1–15 whole seconds (from params.durationSeconds, or numFrames at an assumed 24 fps, rounded and clamped) aspect_ratio: passed through if set resolution: passed through if set No new code is needed unless the model requires a parameter xAI did not previously support. Add it in textToVideo or imageToVideo: // packages/runtime/src/providers/xai-provider.ts override async textToVideo(params: TextToVideoParams): Promise<Uint8Array> { const request: Record<string, unknown> = { model: params.model.id, prompt: params.prompt }; const duration = XAIProvider.resolveVideoDuration(params); if (duration !== undefined) request.duration = duration; if (params.aspectRatio) request.aspect_ratio = params.aspectRatio; if (params.resolution) request.resolution = params.resolution; // Add new xAI-specific param here, e.g.: // if (params.style) request.style = params.style; return this.generateVideo(request, params.timeoutSeconds); } Verify # 1. Confirm the model appears in the live listing curl -s https://api.x.ai/v1/models \ -H "Authorization: Bearer $XAI_API_KEY" | jq '.data[].id' # 2. Type check and lint npm run typecheck npm run lint # 3. Run the xAI provider tests npm run test --workspace=packages/runtime # 4. Smoke-test: list available models via the CLI (requires XAI_API_KEY set) npm run dev:nodetool -- models by-provider xai --kind video # 5. Affected tests and the harness gate npm run test:affected npm run dev:nodetool -- harness gate --base origin/main If you changed classifyModel() or any generation method, also run the provider unit tests directly: npm run test --workspace=packages/runtime -- --reporter=verbose xai-provider Contributing Open PRs at https://github.com/nodetool-ai/nodetool. Before pushing: npm run test:affected npm run typecheck npm run lint Join the discussion on Discord.--- # Suspendable Nodes Source: https://docs.nodetool.ai/developer/suspendable-nodes.html Suspendable Nodes NodeTool has no suspend/resume API. There is no SuspendableState, no SuspendableNode interface, no suspendWorkflow(), and no WorkflowSuspendedError. RunResult.status (packages/kernel/src/runner.ts) is "completed" | "failed" | "cancelled", and JobStatus (packages/protocol/src/messages.ts) adds only "pending", "running", and "error" — none of them is "suspended". A node cannot pause a run and be resumed later. packages/models/src/run-event.ts does declare RunSuspended, NodeSuspended, and NodeResumed event types, but nothing in the runtime emits them. For a run that needs to wait on something external, the shapes that do exist are trigger nodes (packages/kernel/src/trigger-wakeup.ts delivers an event to a registered trigger and starts a run), the webhook route (POST /api/webhooks/:token), and the interactive escalation path — a run started with interactive: true parks on a failing node and waits for the caller’s verdict (see Workflow Supervisor). The Built-in WaitNode (a delay, not a suspension) Important: WaitNode (nodetool.triggers.Wait, in packages/automation-nodes/src/nodes/triggers.ts) does not suspend anything. It is a simple in-process delay: process() sleeps for timeout_seconds via setTimeout, then passes its input through. The run stays alive and in memory for the whole delay. Use it for rate-limiting and fixed delays. import { WaitNode } from "@nodetool-ai/automation-nodes"; // A node that delays for a fixed number of seconds, then forwards `input`. const waitNode = new WaitNode(); waitNode.timeout_seconds = 5; // Seconds to wait (0 = no wait, pass through immediately) waitNode.input = { request_id: "REQ-123" }; WaitNode Properties Property Type Default Description timeout_seconds int (minimum 0) 0 Seconds to wait before continuing. 0 = no wait (pass through immediately). input any "" Input data passed through to the output after the delay WaitNode Output After the delay, the WaitNode outputs: { data: { /* input data passed through */ }, resumed_at: "2026-03-16T12:00:00.000Z", // ISO timestamp after the delay waited_seconds: 5.0 // Actual seconds slept } The output field names (resumed_at, waited_seconds) read like suspension semantics, but they only reflect the setTimeout delay — nothing was suspended. Writing a trigger node A trigger node is how a node “waits” for an outside event. Set static readonly isTrigger = true on the class. When a run starts because an event arrived (the run request carries a trigger_event that targets the node), the kernel calls emitTriggerEvent(event, outputs) instead of the live-listening genProcess() loop. Without a trigger event, the node keeps its streaming behavior, which is what an in-editor live test uses. import type { StreamingOutputs, TriggerEvent } from "@nodetool-ai/node-sdk"; async emitTriggerEvent(event: TriggerEvent, outputs: StreamingOutputs): Promise<void> { await outputs.emit("data", event.payload); } TriggerEvent has node_id, payload, and input_id (the idempotency key of the stored trigger input). The default implementation on BaseNode copies each key of an object payload to the declared output of the same name and drops other keys. Override it to shape adapter payloads onto your outputs. nodetool.triggers.ManualTrigger in packages/automation-nodes/src/nodes/triggers.ts is a complete example. To fire a trigger from the CLI, pass --trigger-event to nodetool workflows run (see the CLI reference). See Also Triggers design - How triggers register, fire, and start runs Workflow API - API endpoints for workflow control--- # TypeScript workflow authoring Source: https://docs.nodetool.ai/developer/ts-dsl-authoring-design.html TypeScript workflow authoring The DSL defines operations in the existing NodeTool graph. The visual editor, validator, kernel, and canonical source exporter share that graph. A factory call registers an operation and does not execute a provider request. Runtime DSL Connectable<T> admits a literal, OutputHandle<T>, or a node whose single output has type T. Generated factories still declare their real output slots. resolveConnection() normalizes direct inputs centrally, so traversal and serialization continue to operate on explicit handles. Multi-output nodes must select a slot, including when a low-level node declares a default. Factories accept a second NodeOptions argument containing an optional id. Metadata options remain generated and cannot be overridden by this argument. Explicit IDs must be non-empty and unique in the current build. Weak references tie nodes and handles to their original descriptors so reusing an ID in a later build cannot reconnect an old reference to a different operation. Host automatic IDs remain UUIDs. Sandbox automatic IDs remain readable names. Those automatic IDs do not promise stable source identity across edits. workflowToDsl() emits explicit graph IDs for known factories and custom-node fallbacks. Canonical regeneration preserves executable data-edge graph semantics, not arbitrary source text or editor layout. The two DSLs retain their existing graph envelopes: host nodes use data and execution flags, while sandbox nodes use kernel properties. The host runner and CLI adapter convert the former. Sandbox list fan-in remains supported, including single-output nodes in place of handles. Host nested connections remain rejected. Neither path adds an execution engine. Capability classification Feature Classification Delivery Implicit single-output connections and normalization PURE TYPESCRIPT / RUNTIME DSL Implemented Explicit generated-factory IDs PURE TYPESCRIPT / RUNTIME DSL Implemented Canonical regeneration with explicit IDs PURE TYPESCRIPT / RUNTIME DSL Implemented for existing exportable graph shapes Typed workflow inputs and returned outputs PURE TYPESCRIPT / RUNTIME DSL Implemented Callable workflow composition PURE TYPESCRIPT / RUNTIME DSL Implemented Named output properties PURE TYPESCRIPT / RUNTIME DSL Implemented with reserved names and node.outputs Runtime branches REQUIRES EXPLICIT GRAPH CONSTRUCT Implemented with If gates and isolated Subgraphs Runtime iteration and streaming joins REQUIRES EXPLICIT GRAPH CONSTRUCT Map uses ForEach, Subgraph, and Collect with kernel correlation Tagged template interpolation REQUIRES EXPLICIT GRAPH CONSTRUCT Implemented through Template dynamic inputs Automatic stable source identity REQUIRES COMPILER / AST SUPPORT FOR FULL EXPERIENCE Deferred Preserving arbitrary handwritten TypeScript across visual edits REQUIRES COMPILER / AST SUPPORT FOR FULL EXPERIENCE Outside canonical export guarantees Converting ordinary if and for into runtime control flow REQUIRES COMPILER / AST SUPPORT FOR FULL EXPERIENCE Deferred Existing boundaries Typed schemas lower to existing scalar/media Input nodes. The new ValueInput preserves optional, list, and arbitrary values, including null, without scalar conversion. .optional() means nullable with a null fallback. .optional(value) supplies a typed fallback. Named return fields lower to Output nodes. Definitions are callable graph values recognized by the CLI. Composition uses scoped Reroute nodes for its symbolic arguments and inlines body operations. Explicit invocation scopes cannot be reused within a build. The public authoring facade and lower-level core form an acyclic dependency. The sandbox pack transforms the same authoring source rather than duplicating helper implementations. Host and guest cores retain their existing envelopes. Generated factories carry input modes and output correlation from metadata. Canonical export preserves those fields and dynamic output declarations, using low-level factories when generated defaults would change behavior. choose builds isolated branch graphs. If gates every captured input and an invocation gate, preventing untaken provider operations from running. A Code node joins the selected output. Producers constructed outside callbacks remain upstream dependencies. map builds a ForEach/Subgraph/Collect graph. Captures are gated once per item with the item’s correlation. Producer descriptor identity distinguishes captures from nested scopes with identical local IDs. Empty lists collect to empty lists. The existing actors govern ordering, cancellation, errors, and concurrency. No helper adds a scheduler. The agent run_workflow capability calls packages/execution/src/service/workflow-run.ts. That service loads a saved workflow, hydrates its graph, performs preflight checks, records a job, and executes through WorkflowRunner. validate_workflow accepts inline graphs. Transient graph execution should share the service path while omitting the Workflow row. The existing host runGraph() helper is not that MCP capability. Authored operation identity, run identity, and asset identity remain separate. Graph source contains configuration and connections, not generated results. Existing job, asset, spend, and provenance systems remain the execution boundary. Compiler and source-map proposal Automatic IDs currently identify a build, not a stable source location. Full source identity requires a compiler boundary with an explicit supported syntax: A TypeScript AST pass identifies factory and composition call sites. A sidecar stores opaque authored operation IDs, module paths, source spans, and enclosing composition scopes. Existing explicit IDs take precedence. An incremental matcher reuses identities across edits only when the old and new call sites match uniquely. Ambiguous matches allocate new IDs and report the ambiguity. Renaming a variable does not change identity. A loop index and a content hash do not become authored operation IDs. The transform injects IDs into factory calls and composes instance scopes. Source maps associate the lowered branch/map nodes and kernel diagnostics with their authored expressions. Invocation and asset IDs remain separate. Visual edits initially regenerate canonical source. An AST-preserving editor may patch only supported call expressions and connections with a unique sidecar match. Unsupported syntax and ambiguous edits fall back to canonical regeneration, with an explicit diagnostic. Compiler tests must cover insertion, deletion, movement, renaming, duplicate expressions, module moves, nested composition, and source-span diagnostics. Compiler output must execute and export through the same graph path as the runtime DSL. This proposal does not convert ordinary JavaScript control flow automatically. That requires a further restricted syntax and compiler analysis. Arbitrary handwritten TypeScript is outside the canonical export guarantee. Remaining service phase P1.5 is separate from P1 and P2: expose run_graph for validated inline graphs through the persisted execution service, retaining permission/cost gates, job records, logs, cancellation, assets, and escalation while omitting the Workflow row. Remote graph execution accepts graph data. Arbitrary TypeScript evaluation requires its own sandbox compiler boundary.--- # TypeScript DSL Guide Source: https://docs.nodetool.ai/developer/ts-dsl-guide.html The TypeScript DSL (@nodetool-ai/dsl) provides type-safe factory functions for building NodeTool workflows in code. Define workflows programmatically with full IDE autocompletion, then serialize them to the same JSON format used by the visual editor. Table of Contents Installation Core Concepts Basic Workflow Connecting Nodes Multi-Output Nodes Typed workflows and composition Runtime branches, maps, and text Building the Workflow Graph Namespaces Code Generation Best Practices Installation Install from npm: npm install @nodetool-ai/dsl Or inside the NodeTool monorepo, all workspace packages are available after npm install at the repo root. Import namespaces directly: import { constant, text, image } from "@nodetool-ai/dsl"; import { workflow } from "@nodetool-ai/dsl"; The package root is the only public entry point. package.json also exports ./flow, but that is the internal native-flow backend, not a supported API (see the native flow design). A subpath import of a generated namespace file does not resolve. Run npm run codegen --workspace=packages/dsl and read packages/dsl/src/generated/index.ts for the namespace names this build ships. Core Concepts OutputHandle When you create a node, you get back a DslNode object. Its output() method returns an OutputHandle — a symbolic reference to one of the node’s output slots. You pass handles as inputs to other nodes to create connections. const a = constant.integer({ value: 5 }); a.output() // → OutputHandle<number> — reference, not the value itself Connectable Every input field accepts a literal value, an output handle, or a compatible single-output node: const greeting = constant.string({ value: "hi" }); const joined = text.collect({ input_item: "hi", separator: ", " }); // literal values const joined2 = text.collect({ input_item: greeting.output(), separator: ", " }); // connection + literal DslNode The frozen object returned by every factory function: const node = constant.integer({ value: 42 }); node.nodeId // explicit id when supplied, otherwise a UUID node.nodeType // "nodetool.constant.Integer" node.inputs // { value: 42 } node.output() // OutputHandle for the node's default output slot Typed workflows and composition The terminal-node form workflow(...nodes) remains available. A schema and callback provide named parameters and results: import { workflow, run, t, template, text } from "@nodetool-ai/dsl"; const greeter = workflow( { name: t.string(), suffix: t.string().optional("!") }, ({ name, suffix }) => ({ greeting: template`Hello ${name}${suffix}` }) ); const result = await run(greeter, { params: { name: "Ada" } }); // { greeting: "Hello Ada!" } const pair = workflow({}, () => ({ first: greeter({ name: "Ada" }, { id: "first" }).greeting, second: greeter({ name: "Grace" }, { id: "second" }).greeting })); t provides t.string(), t.int(), t.float(), t.boolean(), t.image(), t.audio(), t.video(), t.list(item), and t.value() (an arbitrary value). For example, t.list(t.image()) carries image handles into a map body. Required scalar and media parameters become the existing specialized Input nodes (StringInput, IntegerInput, FloatInput, BooleanInput, ImageInput, AudioInput, VideoInput). Lists, arbitrary values, and optional parameters use ValueInput, which preserves their values without conversion. .optional(defaultValue) supplies a fallback. .optional() defaults to null and includes null in the inferred type. Missing required parameters fail before execution. Callback inputs are symbolic handles. Return a non-empty object whose keys name the workflow’s Output nodes. A definition exposes .nodes and .edges. Use toKernelGraph(definition) to obtain kernel properties and execution flags for validation, saving, or visual editing. A definition is callable during another build. Each invocation uses a namespace for its operations and explicit IDs. Pass { id } to give that namespace a name. Duplicate explicit invocation names fail. Automatic invocation names are local to the build. Builders run synchronously during authoring and composition. Keep host side effects outside them. Runtime values cannot drive ordinary JavaScript if, for, string coercion, or property access on media handles. Runtime branches, maps, and text import { workflow, t, choose, map, template } from "@nodetool-ai/dsl"; const greetings = workflow( { names: t.list(t.string()), formal: t.boolean() }, ({ names, formal }) => ({ greetings: map(names, (name, index) => choose(formal, { then: () => template`${index}: Hello ${name}`, else: () => template`${index}: Hi ${name}` })) }) ); choose(condition, { then, else }) builds both branch callbacks as isolated Subgraphs. At runtime, If nodes gate every branch input so only the selected Subgraph executes. Operations must be constructed inside those callbacks. A producer created before choose remains an ordinary upstream dependency. Branch output types must match, except that int and float may mix. A mismatch throws Branch output types do not match. Callbacks must return synchronously. map(list, (item, index) => result) uses ForEach, a Subgraph body, and Collect. Item/index correlation and captured values remain aligned. Nested maps are supported. An empty list produces []. Execution order and concurrency follow the kernel’s existing actors. There is no separate scheduler or parallelMap API. The template tag connects interpolated nodes and handles to the existing Template node’s dynamic inputs. Literal text stays in its template property. Use this tag instead of coercing symbolic values with a JavaScript template literal. Generated nodes expose node.outputs.slot for every declared slot, including names that collide with the node API. Safe names also support node.slot, such as branch.if_true. node.output("slot") remains available. Reserved names include output, outputs, nodeId, nodeType, inputs, defaultOutputHandle, then, constructor, and __proto__. Canonical export preserves lowered graph nodes, IDs, dynamic output metadata, input modes, correlations, and connections. It regenerates executable source rather than preserving the original helper calls or handwritten text. Basic Workflow A workflow follows three steps: create nodes, connect them, build the graph. import { constant, text } from "@nodetool-ai/dsl"; import { workflow } from "@nodetool-ai/dsl"; // 1. Create nodes const x = constant.string({ value: "Hello" }); const y = constant.string({ value: ", " }); // 2. Connect nodes by passing output handles const joined = text.collect({ input_item: x.output(), separator: y.output() }); // 3. Build the workflow graph const wf = workflow(joined); console.log(wf.nodes); // 3 nodes console.log(wf.edges); // 2 edges (x→joined, y→joined) The workflow() function traces all connections from the terminal nodes back to their sources, producing a serializable Workflow object with nodes and edges. Connecting Nodes Implicit single-output connections Before: const greeting = constant.string({ value: "Hello" }); const joined = text.concat({ a: greeting.output(), b: "!" }); After: const greeting = constant.string({ value: "Hello" }, { id: "greeting" }); const joined = text.concat({ a: greeting, b: "!" }, { id: "joined" }); const graph = workflow(joined); Both forms produce the same connection. resolveConnection(value) recognizes an explicit handle or a single-output node. It returns undefined for literals. The factory normalizes direct input values before graph traversal. Nodes with multiple outputs require an explicit selection even when the low-level factory has a default slot. A node buried inside an object or array is rejected by the host DSL because that shape cannot wire a direct input. { id } is available as the second argument on every generated factory. IDs must be non-empty strings and unique within one build. Omit the argument to retain automatic IDs. Calling workflow() clears the build registry, so a new build can reuse explicit IDs. Old nodes and handles remain spent. A node is a symbolic operation. Factory calls build the graph, and run() executes it through the existing kernel. Interpolating a node into a string throws. Build runtime text with the template tag or the Template node’s dynamic inputs. The sandbox pack accepts the same node and ID syntax. It retains readable automatic IDs and returns kernel nodes with properties. It also retains its existing list fan-in support for arrays of handles or single-output nodes. The host returns data, streaming, and streamingInput, which its runner and the CLI translate to kernel descriptors. Linear Chain const a = constant.string({ value: "five" }); const b = text.collect({ input_item: a.output(), separator: "," }); const c = text.collect({ input_item: b.output(), separator: " " }); const wf = workflow(c); // 3 nodes, 2 edges: a→b→c Diamond (Shared Dependencies) A node’s output can be connected to multiple downstream nodes. The graph builder deduplicates automatically. const shared = constant.string({ value: "ten" }); const left = text.collect({ input_item: shared.output(), separator: "," }); const right = text.collect({ input_item: shared.output(), separator: " " }); const final = text.collect({ input_item: left.output(), separator: right.output() }); const wf = workflow(final); // 4 nodes, 4 edges — `shared` appears only once Multiple Terminal Nodes Pass multiple nodes to workflow() to trace all branches: const branch1 = text.collect({ input_item: x.output(), separator: "," }); const branch2 = text.collect({ input_item: x.output(), separator: " " }); const wf = workflow(branch1, branch2); Multi-Output Nodes Some nodes produce multiple outputs (e.g., If has if_true and if_false). Use output("slotName") to select the slot you want: import { control } from "@nodetool-ai/dsl"; const branch = control.if_({ condition: true, value: "hello" }); // Access named outputs branch.output("if_true") // → OutputHandle branch.output("if_false") // → OutputHandle // Calling output() without a slot throws when there is no default output Each output slot is individually typed, so TypeScript catches type mismatches at compile time. Building the Workflow Graph workflow() function workflow(...terminals: WorkflowTerminal[]): Workflow; function workflow(schema, builder): WorkflowDefinition; A terminal is any node (or object) with a nodeId. The first form traces from the terminals via BFS, discovers all connected nodes and edges, performs a topological sort, and returns a frozen Workflow object. It throws if no terminal is given, if a terminal belongs to an earlier build, or if the graph has a cycle. A node input that holds a handle nested inside an array or object also throws, because only a handle assigned directly to an input becomes an edge. Give each source its own input instead. The second form is the typed schema form described above. The result can be serialized to JSON: const wf = workflow(outputNode); const json = JSON.stringify(wf, null, 2); The host runner and CLI convert this host DSL shape to kernel descriptors. The sandbox DSL returns the kernel graph shape directly for agent validation and persistence. run() / runGraph() async function run(wf: Workflow, opts?: RunOptions): Promise<WorkflowResult>; async function runGraph(...terminals: DslNode<never>[]): Promise<WorkflowResult>; run() executes the graph locally and returns the last value each result node produced. An Output node is keyed by its name property, and any other node with no outgoing data edge is keyed by its node id. It throws if the run fails or a node reports an error. runGraph(...terminals) is run(workflow(...terminals)). run() resolves each node in this order: RunOptions.registry, then NodeRegistry.global, then the built-in packs (base-nodes plus the ElevenLabs, MiniMax, Transformers.js, Hugging Face, and Reve packs, each skipped if it cannot load), then the Python bridge. An unknown type throws Unknown node type. RunOptions field Meaning params Values for the workflow’s typed parameters (the schema form), or Input node values. userId User whose secrets resolve. Defaults to "1", the CLI’s local user. authToken Bearer token for calls back to the owning NodeTool API. registry A NodeRegistry consulted first. Use it for custom pack nodes. secretResolver (key, userId) => string \| null. Without it, secrets come from environment variables only and a node with requiredSettings warns that its secret is missing. bridgeOptions Python worker bridge options (wsUrl, workerToken). Otherwise NODETOOL_WORKER_URL and NODETOOL_WORKER_TOKEN apply. Graphs with Python nodes connect the bridge automatically. Running from the CLI nodetool run <file.ts> imports a TypeScript or JavaScript file, finds every exported Workflow (or workflow definition), runs each, and prints the outputs per export name. It fails with No Workflow exports found when the file exports none. nodetool validate <file.ts> checks the graph without running it, and nodetool workflows export-dsl <id_or_file> [-o file] writes a saved workflow as DSL source. See the CLI reference. Using nodes from a custom pack Packs are not loaded by run(), and they have no generated factory. Build the node with createNode, register the pack on a registry, and pass it as RunOptions.registry: import { createNode, output, workflow, run } from "@nodetool-ai/dsl"; import { NodeRegistry } from "@nodetool-ai/node-sdk"; import { register } from "nodetool-mypack"; const registry = new NodeRegistry(); register(registry); const reversed = createNode<{ output: string }, "output">( "mypack.text.Reverse", { text: "hello" }, { outputNames: ["output"], defaultOutput: "output" } ); const result = await run(workflow(output.output({ name: "reversed", value: reversed })), { registry }); // { reversed: "olleh" } createNode(nodeType, inputs, opts) takes id, outputNames, defaultOutput, outputTypes, multiOutput, streaming, streamingInput, dynamicOutputs, inputMode, and outputCorrelation. The generated factories fill these from node metadata. For a node that streams, copy the flags from its class (streaming: true when it overrides genProcess or declares iteration, forward, or chunk correlation). Namespaces The generated barrel lists the available namespaces. Run npm run codegen --workspace=packages/dsl to enumerate the current factories. Import the namespace object and call factory functions: Import Description Example constant Fixed-value nodes constant.integer({ value: 5 }) text Text processing {% raw %}text.template({ string: "Hello, {{ name }}" }){% endraw %} image Image I/O image.loadImageFile({ path: "..." }) audio Audio processing audio.sliceAudio({ start: 0, end: 5 }) video Video processing video.trim({ ... }) control Flow control control.if_({ condition: true, value: x }) agents AI agents agents.agent({ prompt: "..." }) geminiText Google Gemini geminiText.groundedSearch({ ... }) openaiText OpenAI text openaiText.webSearch({ ... }) Namespaces under nodetool. drop the prefix (nodetool.text becomes text). Other dotted namespaces become camelCase (gemini.text becomes geminiText, lib.image.color becomes libImageColor, vector stays vector). Factory names are the last segment of the node type in camelCase (nodetool.text.FilterString becomes text.filterString), with a trailing underscore for JavaScript reserved words (control.if_) and for repeated names. See the full list in packages/dsl/src/generated/index.ts. Code Generation The factory functions are auto-generated from node metadata. To regenerate after adding or modifying nodes: npm run codegen --workspace=packages/dsl This reads all nodes in ALL_BASE_NODES from @nodetool-ai/base-nodes, introspects their metadata (inputs, outputs, types, defaults), and emits typed factory functions into packages/dsl/src/generated/. The same pass also writes the guest-side native flow modules into packages/dsl/src/flow/generated/. Pass --graph or --flow to regenerate one tree. Generated files are committed to git. npm run codegen:check --workspace=packages/dsl writes nothing and exits 1 when either tree differs from the registry, and CI runs it. The codegen script is at packages/dsl/scripts/codegen.ts. Type Mapping Node Type TypeScript Type str string int, float number bool boolean image ImageRef audio AudioRef video VideoRef text TextRef dataframe DataframeRef folder FolderRef storyboard StoryboardRef list[T] T[] dict[K,V] Record<K, V> enum (with values) string literal union optional types T \| undefined union[...] union of the mapped types any and any unmapped type (models, messages, 3D refs) unknown Best Practices Use namespace imports — import { text } from "@nodetool-ai/dsl" gives you autocompletion for all text nodes. Let TypeScript catch errors — the DSL is fully typed. If you pass a string where a number is expected, the compiler tells you. Don’t reuse handles across builds — after calling workflow(), the internal registry is cleared. Handles from previous builds are stale and will throw if used in a new workflow() call. Build workflows linearly — create source nodes first, then processing nodes. The immutable API prevents cycles by construction. Choose the graph boundary: host DSL graphs run through run() or the CLI adapter. Sandbox DSL graphs go directly to agent workflow tools. Canonical source and visual editing workflowToDsl(graph) generates canonical TypeScript from a graph. It emits explicit IDs through generated factories or the low-level createNode() fallback so operation identity survives regeneration. The guarantee covers representable data-edge graphs and their executable semantics. Control edges are rejected. Editor layout, comments, variable names, constants, and spreads from the original source are not preserved. The initial authoring subset uses imports, const bindings, generated factory calls, literal options, references to earlier bindings, explicit output selection, and workflow(...terminals). Literal arrays and objects are allowed where the input contract supports them. Symbolic nested objects are rejected. The exporter emits only the terminal form workflow(...terminals). It does not emit typed schema callbacks, returned output objects, or choose and map. A saved graph exports as lowered nodes, and Input and Output nodes export as ordinary nodes. Arbitrary if, loops, mutation, dynamic property access, reflection, unknown spreads, and filesystem/network behavior fall outside the guaranteed visual editing subset. They may run during graph construction but do not become runtime graph control flow. Canonical export preserves graph semantics rather than arbitrary handwritten TypeScript. See the authoring design and follow-up plan.--- # Portable Workflow Runner Source: https://docs.nodetool.ai/developer/workflow-runner @nodetool-ai/workflow-runner runs a workflow graph without the NodeTool server. It wraps the kernel WorkflowRunner in two things: an async generator that yields live messages, and a (Request) => Response handler that streams those messages as Server-Sent Events. Both use only Web-standard APIs, so the same code runs on serverless and edge platforms. A separate ./browser entry runs graphs inside a web page. The runner has no file system, subprocess, or transport assumptions. You supply the graph, a node registry, and optionally a processing context. Storage, secrets, and the cache come in through the context. Install npm install @nodetool-ai/workflow-runner The package needs Node 22 or newer when it runs on Node. It exports two entries: Entry Use @nodetool-ai/workflow-runner runWorkflow, createWorkflowHandler, envSecretResolver, and the browser helpers @nodetool-ai/workflow-runner/browser The browser helpers, built for page bundles Exported symbols Symbol What it does runWorkflow(options) Runs a graph and yields each ProcessingMessage live. The generator’s return value is the final RunResult createWorkflowHandler(options) Returns a (Request) => Promise<Response> handler that streams a run as Server-Sent Events envSecretResolver(env) Turns an environment-variable object into a secret resolver for a context createBrowserRegistry(nodeClasses) Builds a registry that keeps only classes that support the browser platform runBrowserWorkflow(options) Like runWorkflow, checked against the browser platform, with every message stamped with job_id and workflow_id graphRunsInRegistry(graph, registry) True when every node in the graph is in the registry, so the graph can run client-side Build a registry A registry decides which nodes the deployment can run. Register the node classes your graphs use, then optionally filter for a platform: import { NodeRegistry } from "@nodetool-ai/node-sdk"; import { CONSTANT_NODES, CONTROL_NODES } from "@nodetool-ai/core-nodes"; const registry = new NodeRegistry(); for (const nodeClass of [...CONSTANT_NODES, ...CONTROL_NODES]) { registry.register(nodeClass); } const edgeRegistry = registry.forPlatform("edge"); A graph is a list of nodes and edges. Each node has an id, a type, and properties. A name on a node names its entry in the result outputs. Run a workflow in Node Call runWorkflow and read the messages: import { runWorkflow } from "@nodetool-ai/workflow-runner"; const graph = { nodes: [ { id: "s", type: "nodetool.constant.String", name: "text", properties: { value: "hello" } } ], edges: [] }; const run = runWorkflow({ graph, registry }); let next = await run.next(); while (!next.done) { console.log(next.value.type); next = await run.next(); } console.log(next.value.status, next.value.outputs); for await also works, but it discards the final RunResult. Pass params to set the values of input nodes, signal to cancel the run, and workflowId or jobId to label it. runWorkflow builds a minimal context unless you pass one. To supply your own pieces without building a context, pass storage, workspaceStorage, cache, environment, secretResolver, or sandboxModuleCatalog. These are ignored when you pass a full context. Serve a workflow over HTTP createWorkflowHandler takes a registry and returns a handler: import { createWorkflowHandler } from "@nodetool-ai/workflow-runner"; export const handler = createWorkflowHandler({ registry, platform: "edge" }); The handler accepts a POST with a JSON body: { "graph": { "nodes": [], "edges": [] }, "params": {}, "workflow_id": "optional", "job_id": "optional" } The response is text/event-stream. Each message is a data: event, and the run ends with an event: result event that carries the RunResult. Request problems return JSON with a 4xx status: 405 for a method other than POST, 400 for invalid JSON, a missing graph, or a failing beforeRun hook. After the stream starts, a failure arrives as an event: error event with the message “Workflow execution failed” and the HTTP status stays 200. Option What it does registry The nodes available in this deployment createContext(req) Builds a ProcessingContext per request, for example with secrets and storage. Return nothing for a minimal context beforeRun(body, req) Authorizes or rewrites the parsed body. Throw to reject with a 400 platform Rejects graphs that contain nodes that do not support this platform, before any node starts Secrets come from the context. This builds one from environment variables: import { ProcessingContext } from "@nodetool-ai/runtime/context"; import { createWorkflowHandler, envSecretResolver } from "@nodetool-ai/workflow-runner"; const handler = createWorkflowHandler({ registry, createContext: () => new ProcessingContext({ jobId: crypto.randomUUID(), secretResolver: envSecretResolver(process.env) }) }); createContext must return a ProcessingContext instance. Any other value is ignored and the run uses a minimal context. Run on each platform The README for the package lists Vercel (Node and Edge), Cloudflare Workers, Bun, and Deno. The handler is a plain fetch handler, so each platform wraps it in its own entry point. The platform wrappers below are standard for each runtime and are not shipped by the package. Platform Entry point Vercel export const POST = handler; in a route file Cloudflare Workers export default { fetch: handler }; Bun Bun.serve({ fetch: handler }); Deno Deno.serve(handler); Node Call runWorkflow directly, or adapt the handler with a Web-standard server of your choice On Cloudflare Workers, secrets arrive in the env argument of fetch, which the handler does not receive. Build the handler inside fetch, so createContext can use that env: export default { fetch(req: Request, env: Record<string, string>) { return createWorkflowHandler({ registry, platform: "workers", createContext: () => new ProcessingContext({ jobId: crypto.randomUUID(), secretResolver: envSecretResolver(env) }) })(req); } }; Run in the browser runBrowserWorkflow runs a graph of browser-safe nodes in a web page, with no server round trip. It checks the graph against the browser platform first, so a server-only node fails before any node starts. Every message carries job_id and workflow_id, so a client can route it through the same code as messages from the server. import { createBrowserRegistry, graphRunsInRegistry, runBrowserWorkflow } from "@nodetool-ai/workflow-runner/browser"; import { ALL_BROWSER_NODES } from "@nodetool-ai/base-nodes/platforms/browser"; const registry = createBrowserRegistry(ALL_BROWSER_NODES); if (graphRunsInRegistry(graph, registry)) { const run = runBrowserWorkflow({ graph, registry, workflowId, jobId }); let next = await run.next(); while (!next.done) { deliver(next.value); next = await run.next(); } const result = next.value; } When graphRunsInRegistry returns false, send the graph to a server instead. A Code node in the browser needs a sandboxModuleCatalog, because a browser has no process-wide catalog to resolve imports from. Which nodes run where Each node class lists the platforms it supports. A node that lists none supports only node. Platforms are: Platform Runtime node Full Node.js: native modules, subprocesses, file system. Electron, RunPod, Vercel Node functions, plain servers workers Cloudflare Workers and Durable Objects, with the Node compatibility layer. No native modules or subprocesses edge Web APIs only. Vercel Edge Runtime and Deno Deploy browser A browser page with WebGPU, the DOM, canvas, and IndexedDB In the repository, node groups are tagged like this: All four platforms. The constant, input, control, variable, compare, subgraph, app, workflow, and vector nodes in the core node package, the fake media nodes, and the Code node. Node, Workers, and Edge. Server-oriented groups such as the LLM provider nodes, document, text extras, data, sketch, storyboard, script, timeline, entity, and messaging nodes. Node only. Any node that declares nothing, and the nodes that say so. An image Scale node, for example, reports “not supported on platform ‘browser’ (supports: node)”. The curated browser set is ALL_BROWSER_NODES, exported from @nodetool-ai/base-nodes/platforms/browser. It holds the app, compare, constant, control, input, subgraph, vector, and workflow nodes, the placeholder and fake media nodes, and the Code node. Check a node’s class for its exact platforms value before you rely on it. Limitations Only the registry you build is available. The package does not bundle any nodes, so a graph that names an unregistered node type is rejected. Nodes that need native modules, subprocesses, or the file system run only on node. The Blender nodes are an example. See Blender Nodes. Browser runs cover only browser-tagged nodes. Graphs that use other nodes must run on a server. The handler sends no authentication and does not limit request size or run time. Add checks in beforeRun and in your platform. The request body is JSON, and the response is Server-Sent Events as text. It does not use the MsgPack WebSocket protocol of the NodeTool server. There is no persistence. Jobs, assets, and run history are not stored unless you pass a context with storage. Request cancellation passes through req.signal. A client that disconnects stops the run. See also TypeScript DSL Guide for building graphs in code Custom Nodes Guide for writing nodes and the platforms setting Architecture for the kernel that does the execution--- # Production Deploy (Docker) Source: https://docs.nodetool.ai/docker-production-deploy Production deploy api.nodetool.ai runs on the Docker host 46.224.131.110. Caddy routes traffic to app-1 and app-2 with cookie affinity. Both replicas use the same Supabase database, auth project, and storage buckets. Deployment configuration and the rolling release script live in the adjacent nodetool-deploy repository, checked out at /home/claude/nodetool-deploy on that host. Release chain A push to main starts the Docker image build, User Journeys, and Test independently. The Deploy to Docker workflow runs when any of them completes. It deploys only when Docker and User Journeys have succeeded for that exact commit and Test has succeeded or has no run, because Test filters its push trigger by path. It counts only push runs. It does not wait: the last workflow to finish triggers the release. It rejects a commit superseded on main. It retains the historical filename fly-deploy.yml, but does not invoke Fly or use its credentials. Manual dispatch follows the same gates and uses the dispatch commit, never a mutable image tag. The Deploy server job connects to the host over SSH and sends only the full commit SHA. A host-side forced command validates the SHA, rechecks the build gates and current main, and invokes the rolling script with ghcr.io/nodetool-ai/nodetool:main-<shortsha>. The host resolves this to an immutable digest. GitHub serializes release runs without cancelling a rollout in progress. The host script also takes a deployment lock. web-deploy.yml follows Deploy to Docker and checks that its Deploy server job succeeded before releasing the same commit to Cloudflare Pages. A skipped or failed server release cannot promote the frontend. Cloudflare Pages credentials remain in web-production. GitHub connection settings Repository variables: DOCKER_DEPLOY_HOST: the Docker host IP or hostname. DOCKER_DEPLOY_USER: the SSH account that owns the deployment checkout. Repository secrets: DOCKER_DEPLOY_SSH_KEY: a dedicated Ed25519 private key. DOCKER_DEPLOY_KNOWN_HOSTS: the host’s verified SSH public key entry. The matching public key in authorized_keys uses restrict and forces /home/claude/nodetool-deploy/bootstrap/github-ssh-deploy.sh. It permits only a full release SHA or the read-only check probe, not shell commands, forwarding, or a terminal. The SSH client requires the pinned host key. Database credentials remain in the host’s mode-0600 .env and are not copied into Actions. Rolling release The script pulls the image and runs its database migrations once before changing either replica. It then verifies the peer is healthy, signals SIGUSR2 to drain the selected replica, waits for its turns and jobs to reach zero, and recreates it. Health must pass before the peer is drained. The saved image digest changes only after both replacements succeed. A failed migration or replacement stops the release. Migrations must remain compatible with the previous image during the handover. Inspect the running deployment on the host: cd /home/claude/nodetool-deploy docker compose ps node bootstrap/verify.mjs --public docker compose logs --tail 100 app-1 app-2 For a manual rollback, review schema compatibility and run the rolling script on the host with a previously deployed immutable image digest. Rollbacks are not accepted by the restricted GitHub key, which only releases current main. Check game draft storage compatibility before choosing an image whose game reader predates the dotted draft version identifiers. Unchanged database schemas do not make those readers compatible. The deployment setup guide covers DNS, trigger-dispatch handover, pool sizing, and affinity checks. The retained Fly guide covers legacy rollback operations, not the active GitHub release destination. Run history retention The production image sets NODETOOL_STORAGE_AUTO_CLEANUP=1. The server sweeps run trace owners on startup and on its maintenance timer, using each account’s retention settings. Run content expires under runTraceRetentionDays and finished records under terminalJobRetentionDays. Keep automatic cleanup enabled on the hosted service to meet the privacy policy. Self-hosted operators can override the image setting when configuring their container. Local desktop installs keep the existing manual cleanup default.--- # Docker Resource Management Source: https://docs.nodetool.ai/docker-resource-management.html Use these guidelines to keep multi-tenant or long-running NodeTool deployments stable. The examples use Docker’s standard --memory / --cpus flags and volume mounts. nodetool deploy apply does not set memory or CPU limits. Its deployment.yaml has no field for them, and it passes only --gpus "device=<container.gpu>" when container.gpu is set. To cap a deployer-managed container, update it after apply with docker update --memory 8g --cpus 4 nodetool-<container.name> (this does not survive the next apply), or run the container yourself with docker run or Compose. Baseline Settings Set memory and CPU caps for every container to avoid noisy-neighbor issues. Start with --memory 8g and --cpus 4 for GPU hosts, then tune from docker stats. Mount a Hugging Face cache read-only only when models are already downloaded. The server cannot add models to a read-only cache. nodetool deploy mounts paths.hf_cache at /hf-cache read-only unless persistent_paths is set. Mount the workspace read/write. The image runs as the node user, so the host directory must be writable by it. Keep /tmp and workspace volumes on fast disks. Avoid sharing /tmp across unrelated services. Local Docker Runs For standalone containers or quick tests: docker run --gpus all \ --memory 8g --cpus 4 \ -v /data/nodetool/workspace:/workspace \ -v /data/hf-cache:/hf-cache:ro \ -e DB_PATH=/workspace/nodetool.db \ -e HF_HOME=/hf-cache \ -p 7777:7777 \ ghcr.io/nodetool-ai/nodetool:latest The image exits at start unless DB_PATH or DATABASE_URL is set. In Compose, use mem_limit and cpus on the service, or deploy.resources.limits. The reference docker-compose.yml sets none. Use --memory-reservation to set soft limits when co-locating multiple servers. Without a keychain the server needs SECRETS_MASTER_KEY. The image generates one under /workspace/.secrets_master_key, so keep /workspace on a persistent volume. Keep per-run tmp data on the workspace volume (e.g., TMPDIR=/workspace/tmp) so it persists through restarts. Mount shared caches read-only (:ro) to prevent accidental mutation. Monitoring and Tuning Watch docker stats to catch containers hitting limits or restarting. If runs are OOM-killed, lower batch sizes in your workflows or increase --memory incrementally. For storage-heavy jobs, pair limits with the guidance in Storage to ensure caches and outputs have enough space.--- # Document sync: merge external changes into a dirty draft Source: https://docs.nodetool.ai/document-sync-plan.html Document sync: merge external changes into a dirty draft Decisions: ADR 0001. Terms: CONTEXT.md. Today documentSync.ts reloads a clean editor and warns a dirty one. A dirty editor’s next autosave fails the CAS and the only recovery is a refresh. This plan replaces the warning with a per-merge-unit merge, draft wins, one conflict banner. Eight slices. S0 is shared and ships first. S1 to S7 are independent after S0 and can go in any order; the ADR names storyboard first. S0 Foundation S0.1 Ops on the write ModelObserverCallback (packages/models/src/base-model.ts:22) takes no context. Add an optional third argument meta?: { ops?: unknown[] } to ModelObserver.notify and to every updateFieldsIfUnchanged / updateDocumentIfUnchanged / mutateDocumentData static (storyboard.ts:181, script.ts:224, js-script.ts:139, image-document.ts:245, timeline-sequence.ts:293, application.ts:352, workflow.ts:132). Do not use AsyncLocalStorage: the ops belong to one write, not to a request. onModelChange (packages/websocket/src/unified-websocket-runner.ts:9217) copies meta.ops into resource.ops. ResourceChangeMessage in packages/protocol/src/messages.ts:902 gets ops?: DocumentOp[]. DocumentOp lives in packages/protocol/src/document-ops.ts: { tool: string; input: unknown }, the same shape the headless bridges replay. The web narrows per surface. S0.2 Merge engine web/src/stores/documentMerge.ts, pure, no store access: interface MergeResult<TDoc> { doc: TDoc; conflicts: Conflict[]; } interface Conflict { unit: { kind: string; id: string; label: string }; external: unknown; // the value the draft refused reason: "edited" | "deleted" | "dangling" | "replaced"; } mergeByUnits<TDoc>(base, draft, server, adapter): MergeResult<TDoc> adapter lists the unit collections of a document (shots, clips, layers, …), the id field per collection, and the scalar fields that are last-write-wins. The engine is a three-way merge by unit id: unit changed on the server only: take the server value unit changed in the draft only: keep the draft both changed: keep the draft, emit edited conflict with the server value deleted on the server, changed in the draft: keep the draft, emit deleted server-only write with no ops (resource.ops missing): whole document is one unit; a dirty draft keeps everything and emits one replaced conflict Ops are used for one thing: to know which units the external write touched, so a unit the draft and the server both differ in but the write did not touch is not a conflict. Without ops, “touched” is “differs from base”. S0.3 documentSync changes documentSync.ts subscriber gains merge(server, ops) next to reload. handleDocumentResourceChange calls merge when dirty, reload when clean. warnChangedElsewhere goes. Conflict state: web/src/stores/ConflictStore.ts, keyed ${type}:${id}, holds Conflict[]. Actions accept(unitId) and discard(unitId). Accept applies the external value through the surface store’s own mutation so it lands on the undo stack. S0.4 Banner web/src/components/ui_primitives/ConflictBanner.tsx, one component, props { conflicts, onAccept, onDiscard }. Mounted by each editor shell. Text: “N changes made outside the editor conflict with your edits.” S0.5 Undo Merged external values must bypass undo. Per surface the mechanism differs and is listed in the slice. Rule: apply the merged document through a store path that does not push history. S0.6 Agent CAS retry The capabilities already retry CAS five times (storyboards.ts:88, scripts.ts:94, timelines.ts:465). The ADR says one reapply. Drop the constant to 1 in each. Not blocking. Tests: documentMerge.test.ts covers every branch above with a generic adapter. documentSync.test.ts covers dirty → merge, clean → reload, echo → ignore, no-ops → replaced. S1 Storyboard Store: web/src/stores/storyboard/StoryboardStore.ts, history via documentHistory.ts. Sync: web/src/hooks/storyboard/useStoryboardServerSync.ts:311. Write: packages/agents/src/capabilities/storyboards.ts:1339, ops add_shot / update_shot / remove_shot / reorder_shot / set_board. Units: shots[] by shot.id. Board scalars (brief, style, aspectRatio, model refs) last-write-wins. index renumbered after merge with renumberShots. Steps: Pass { ops } to Storyboard.updateFieldsIfUnchanged in editStoryboard and in the render tools (render_storyboard_stills writes one update_shot per shot). Adapter storyboardMergeAdapter in web/src/stores/storyboard/merge.ts. useStoryboardServerSync registers merge: run the engine, write the result with a store setter that does not call pushHistory, roll serverRevisions[id]. Mount ConflictBanner in StoryboardBoard. Accept for update_shot calls the store’s updateShot (checkpoints history). Test: agent still lands on shot 2 while shot 3 action text is dirty → no conflict, still present. Agent rewrites shot 3 action → conflict, draft text kept. S2 Timeline Store: web/src/stores/timeline/TimelineStore.ts, per-sequence instance, temporal() undo. Sync: web/src/hooks/timeline/useTimelineExternalSync.ts:27. Write: packages/agents/src/capabilities/timelines.ts:661, ops are the ui_timeline_* names. Units: tracks[], clips[], markers[], transcript[] by id. Scalars fps, width, height, scriptEnabled last-write-wins. Steps: Pass { ops } to TimelineSequence.updateDocumentIfUnchanged in editTimeline, assemble_script_timeline, assemble_storyboard_timeline. Adapter in web/src/stores/timeline/merge.ts. A clip whose trackId names a track deleted in the draft is dangling, dropped and listed. useTimelineExternalSync.merge: apply with temporal.getState().pause() / resume() around the set so no history entry is pushed. Then reflowGenerated. Banner in the timeline editor shell. Test: useTimelineExternalSync.test.tsx, dirty clip trim plus external add_text_clip → both present, no conflict. S3 Sketch Store: SketchSessionStore.ts (meta, CAS) and components/sketch/state/useSketchStore.ts (layers, custom delta history historySlice.ts). Sync: registered at SketchSessionStore.ts:1018, reload via clearHydrated + query invalidate. Write: packages/agents/src/capabilities/sketches.ts:807, ops add_layer, remove_layer, rename_layer, set_layer_props, reorder_layer, duplicate_layer, select_layer, resize_canvas. Units: layers[] by layer.id, layerBindings by layer id. Pixel data is opaque: a layer whose bitmap changed on both sides is one edited conflict, no pixel merge. activeLayerId is draft-only, never merged. resize_canvas is a whole-document replaced conflict when dirty. Steps: Pass { ops } in editSketch and in the generate path (one set_layer_props per generated layer). Adapter in web/src/stores/sketch/merge.ts. Order is array position, so a merged layer list is: draft order, with server-only new layers inserted at their server index. Merge path in SketchSessionStore: apply through useSketchStore with skipHistory: true on the layer set action (add the flag to historySlice). Banner in the sketch editor shell. Test in SketchSessionStore.autosave.test.ts: dirty brush stroke on layer A plus external add_layer B → both present. S4 Script Store: web/src/stores/script/ScriptStore.ts, documentHistory. Sync: web/src/hooks/script/useScriptServerSync.ts:233. Write: packages/agents/src/capabilities/scripts.ts:1170, ops add_speaker, set_speaker, set_speaker_voice, remove_speaker, add_section, add_line, set_line_text, set_line_speaker, remove_line; voice_script_lines writes one take per line. Units: cast[] by speaker.id, sections[] by section.id, sections[].lines[] by line.id, line.takes[] by take.id. Nested: the adapter lists lines as a unit collection under each section. A take added on the server to a line whose text is dirty is not a conflict: takes and text are separate fields of the unit. The engine must merge at field level inside a unit for this case, so the adapter marks takes as a mergeable sub-collection. Steps: Pass { ops } in editScript and voice_script_lines. Adapter in web/src/stores/script/merge.ts. useScriptServerSync.merge sets through withScript with checkpoint off. Banner in the script editor shell. Test: user edits line 4 text while voice_all runs → every line gets its take, line 4 keeps the draft text and is marked stale. S5 JS script Store: web/src/stores/jsScript/JsScriptStore.ts, documentHistory. Sync: web/src/hooks/jsScript/useJsScriptServerSync.ts:277. Write: packages/agents/src/capabilities/js-scripts.ts:443, no op list, save_js_script writes a whole document. Units: code is one unit. inputs[], outputs[] by port.name, tests[] by test.name. Scalars name, description, timeoutMs, secrets last-write-wins. Steps: save_js_script derives ops from the diff it made (set_code, set_ports, set_meta, set_tests) and passes them. This is the one capability where the ops come from a diff, because the tool takes a whole document. Adapter in web/src/stores/jsScript/merge.ts. A dirty code plus external set_code is one edited conflict; the banner shows the external body in a diff view. That is the only surface where accept needs a diff, use the existing -diff pack’s viewer or a plain two-pane. Merge sets without checkpoint. Banner in the JS script editor. Test: dirty code plus external set_tests → tests merged, code kept, no conflict. S6 Application No zustand draft. Puck holds the document, ApplicationAppBuilder.tsx keeps the CAS token in revisionRef. isDirty is hard-coded true and reload is a no-op, so today every external write raises the conflict banner even when the user has not typed. Write: packages/agents/src/capabilities/apps.ts:337, ops are the ui_app_* steps. Units: ui.content[] by props.id (nested by parentId + slot), operations[], variables[], resources[] by id. Steps: Pass { ops } in editApp. Real isDirty: compare Puck’s current data with the last loaded document by reference or hash. Clean → reload sets Puck data from the server. Adapter in web/src/components/appbuilder/merge.ts. Apply the merged document with Puck’s dispatch({ type: "setData" }), which does not create a history entry in Puck; verify against the Puck version in use. Replace setConflict("external") with the shared banner. Test in ApplicationAppBuilder.test.tsx: clean editor reloads, dirty widget label plus external add_component → both present. S7 Workflow graph Store: web/src/stores/NodeStore.ts per workflow, temporal() undo, explicit workflowIsDirty. Reload path is separate: workflowResourceReloader → WorkflowManagerStore.refreshWorkflow:620, bails when dirty. Write: packages/agents/src/capabilities/ui.ts:122 and workflows.ts (:275, :435, :490), ops ui_add_node, ui_connect_nodes, ui_update_node_data, ui_delete_node, ui_delete_edge, ui_move_node, ui_set_node_title. Units: nodes[] by id with fields data, ui_properties.position, title; edges[] by id. An edge whose source or target is not in the merged nodes is dangling, dropped and listed. ui_move_node touches only position, so a moved node with dirty data is not a conflict: the adapter marks position, data, title as separate fields of the unit. Steps: Pass { ops } in ui.ts and the three workflows.ts writes. Adapter in web/src/stores/workflowMerge.ts. refreshWorkflow: when dirty and resource.ops present, merge instead of bail. Apply with temporal.pause(), set workflowIsDirty unchanged, roll etag and updated_at. Banner in the editor shell. Accept for a node runs updateNodeData (history on). Test in WorkflowManagerStore.save.test.ts: dirty node data plus external ui_add_node + ui_connect_nodes to a node the draft deleted → new node present, edge dropped and listed. Order and size Slice LOC estimate Depends on S0 600 — S1 storyboard 250 S0 S2 timeline 300 S0 S3 sketch 350 S0 S4 script 300 S0, field-level merge in S0.2 S5 JS script 250 S0, diff view S6 application 400 S0, Puck dirty detection S7 workflow 350 S0 Each slice is one PR. S0 ships with S1 in the same PR if a standalone S0 has no consumer. Not in scope Text merge inside one field. A field is one value. Pixel merge in sketch layers. Presence or cursors. Locks.--- # Document synchronization Source: https://docs.nodetool.ai/document-sync.html Document synchronization web/src/stores/documentSync.ts is the canonical seam for editor synchronization. It owns save debounce, serialization, flush behavior, compare-and-swap recovery, and routing external changes. Editor integrations supply document conversion, persistence, dirty-state detection, and merge behavior. They must not recreate that lifecycle in a hook or store. ADR 0001 defines how external changes merge with an open draft. The implementation plan remains in document-sync-plan.md. Checked integration inventory npm run check:document-sync scans non-test TypeScript under web/src and rejects synchronization entry points outside this inventory. The check follows renamed imports and generic calls. An intentional new integration must use the shared seam and update the inventory in the check and this table together. Entry-point category Current integrations Shared lifecycle controller construction JS Script useJsScriptServerSync.ts; Script useScriptServerSync.ts; Storyboard useStoryboardServerSync.ts; Timeline useTimelineAutosave.ts; Sketch SketchSessionStore.ts External-change subscription Application ApplicationAppBuilder.tsx; JS Script useJsScriptServerSync.ts; Script useScriptServerSync.ts; Storyboard useStoryboardServerSync.ts; Timeline useTimelineExternalSync.ts; Sketch SketchSessionStore.ts Save-flush registry declaration jsScriptSaveRegistry.ts; storyboardSaveRegistry.ts The two save-flush registries expose a pending shared-controller flush to agent bridges. They do not implement debounce, retries, compare-and-swap recovery, or external-change handling. All four migrated editors, Script, Storyboard, Timeline, and Sketch, construct the shared controller instead of retaining a separate synchronization lifecycle.--- # DSL Native Flow — Technical Design Source: https://docs.nodetool.ai/dsl-native-flow-design.html DSL Native Flow — Technical Design Status: In progress — the public surface runs in the QuickJS guest (§0) Last updated: 2026-08-17 — recorded the guest pivot (§0) and rewrote §1 0. The QuickJS pivot What changed: the public surface is guest code, not a host import. §4–§6 describe the host API, which is now internal. A user writes flow code in a Code node, where it runs in the sandbox like every other body. The typed callables live in a shipped pack, @nodetool-ai/sandbox-flow (packages/sandbox-packs/sandbox-flow/), built from packages/dsl/src/flow/generated/ — one guest module per namespace, emitted by the same npm run codegen pass as the graph tree. Each call crosses to the host through the capability module @nodetool-ai/sandbox-nodetool/flow (invoke_node, open_node_stream, take_node_stream, close_node_stream), which lands on the host flow of §4–§6. Point by point against what came before: No @nodetool-ai/dsl/flow export. startFlow, invoke, invokeStream and the streaming adapters stay in packages/dsl/src/flow/, consumed by the capability module. They are the backend, not the API; a caller who wants them is inside NodeTool. Streaming input is arrays only, v1. §4.4’s AsyncIterable on an input handle does not cross the guest boundary, so a run-contract node’s inputs widen to T | T[] and nothing more. The host adapter still accepts an iterable — nothing in the guest can hand it one. Correlation and emitGroup are unchanged (§5): a group flattens to its member emissions, lineage is not modelled, and a .stream on a run-contract node yields {slot, value}. Isolation comes back. §2 counted the sandbox boundary as a cost the host surface avoids. It is now paid per call — one crossing for a one-shot call, one per item for a stream — and what it buys is that model-authored flow code is no longer trusted code. Open item. Pack discovery refuses a non-relative import in authored pack code (inspectJavaScript, packages/node-sdk/src/sandbox-pack-discovery.ts), so the pack’s guest core cannot yet name the capability module. Either that rule gains an allowance for @nodetool-ai/sandbox-nodetool/*, or the bridge ships as a host module (SANDBOX_HOST_MODULES). A second, smaller one: the host mounts a capability facade by scanning the body’s static imports (packages/agents/src/codeact/capability-modules.ts), so a body using the pack must also import "@nodetool-ai/sandbox-nodetool/flow" itself. Mounting what a resolved pack module imports would remove that line from every program. 1. Summary A third way to run nodes, cheaper than the graph: call a node as a typed async function and write the control flow in plain JavaScript. No graph, no edges, no WorkflowRunner. await is the edge, a variable is the wire, Promise.all is the fan-out. // A Code node body. import "@nodetool-ai/sandbox-nodetool/flow"; import { template } from "@nodetool-ai/sandbox-flow/nodetool.text"; import { textToSpeech } from "@nodetool-ai/sandbox-flow/gemini.audio"; const draft = await template({ string: "Write a haiku about {{topic}}", values: { topic: "rain" } }); const speech = await textToSpeech({ text: draft.output }); return speech.output; Each call resolves the node class from the registry, assigns the inputs, injects secrets, runs process()/genProcess() through the node’s own toExecutor() bridge, and returns the outputs record — the same execution the kernel’s NodeActor performs, minus the actor. The existing graph DSL (workflow() + run()) is untouched, and the sandbox gains a pack rather than a mode. Where a graph is the deliverable — something to open in the editor, validate, supervise, or hand to the server — build one with @nodetool-ai/sandbox-dsl instead (§5). 2. Why Three execution paths exist today, and each pays for something a plain script does not need: WorkflowRunner (kernel). Actor per node, message-passing mailboxes, correlation lineage, EOS propagation, ProcessingMessage fan-out. Right for the editor and for supervised runs; overhead and indirection when the caller is code that just wants values. QuickJS sandbox (Code node, CodeAct). Every value crosses a host↔guest marshalling boundary, the guest has its own module loader and interrupt budget, and host packs bridge one call at a time. The isolation is the point — and the cost. For code we already trust, it buys nothing. Graph DSL run(). Typed authoring, but the built graph still executes on WorkflowRunner, so branching, loops, and retries must be expressed as nodes (If, ForEach, …) instead of if, for, try. The native flow keeps the DSL’s typed node surface — the generated factories’ Inputs/Outputs types, checked at compile time — and swaps the execution substrate for direct async calls. 3. Design goals A node call is one function call. No graph build, no scheduler, no serialization. The only machinery per call: registry resolve, secret injection, process(). JavaScript is the control plane. Branches, loops, retries, timeouts, concurrency — all plain JS. The API ships no control-flow combinators. Same types, same nodes, one codegen. The flow surface is generated from the same registry metadata as the graph factories, in the same npm run codegen pass, gated by the same codegen:check. Same observability. Each call opens a node.process OTel span; LLM calls inside it still emit llm.chat/llm.stream spans and cost-ledger rows, because providers go through the same ProcessingContext. Nothing new in the kernel or node-sdk. The flow layer is a consumer of NodeRegistry, toExecutor(), and ProcessingContext as they exist. 4. The API 4.1 startFlow and the Flow handle import { startFlow, type Flow, type FlowOptions } from "@nodetool-ai/dsl/flow"; const flow: Flow = await startFlow({ userId: "1", // default "1", matching the CLI secretResolver, // (key, userId) => value — same SecretResolver as RunOptions registry, // optional NodeRegistry override bridgeOptions, // Python worker bridge config (lazy, see 4.6) signal, // AbortSignal — cancels in-flight calls onCall // (event: FlowCallEvent) => void, see 4.7 }); A Flow owns what RunOptions configures today: one ProcessingContext (job id, user, secrets, storage), the resolved registry chain (caller override → NodeRegistry.global → the lazily registered builtin packs, exactly as run() in core.ts builds it), and at most one Python bridge. Lifecycle: interface Flow { run<T>(body: () => Promise<T>): Promise<T>; // sets the ambient flow (ALS) for body readonly context: ProcessingContext; // escape hatch for advanced callers close(): Promise<void>; // closes the Python bridge, ends spans [Symbol.asyncDispose](): Promise<void>; // `await using flow = ...` } 4.2 Node calls For every registered node, codegen emits a callable in the flow namespace, sharing the graph DSL’s generated Inputs/Outputs types with Connectable<T> collapsed to T — a value is a value, there are no handles: // generated — flow/gemini.audio.ts export function textToSpeech( inputs: TextToSpeechInputs, // { text?: string; model?: ...; voice_name?: ... } opts?: CallOptions // { flow?: Flow; signal?: AbortSignal } ): Promise<TextToSpeechOutputs>; // { output: AudioRef } Resolution of the executing flow, in order: opts.flow → the ambient flow set by an enclosing flow.run() (AsyncLocalStorage) → a process-default flow created lazily on first use (env-var secrets only, no Python bridge). The default flow makes a five-line script work with zero ceremony; flow.run() is for anything that needs configured secrets, cancellation, or isolation between concurrent flows. Under the hood a call is: registry.resolve({id, type, properties: inputs}) // fresh instance per call → executor.initialize() → executor.process(inputs, flow.context) // secrets injected by toExecutor() → executor.finalize() One-shot nodes return the record process() resolves to. A node whose genProcess yields multiple records is drained and the call returns the last value per slot — the same fold run() applies to terminal outputs today. Callers who want the stream use .stream(). 4.3 Streaming output Every generated callable for a streaming-output node also carries a .stream member: for await (const chunk of agents.agent.stream({ objective, model })) { // chunk: Partial<AgentOutputs> — one genProcess yield process.stdout.write(chunk.chunk ?? ""); } .stream(inputs, opts?) returns AsyncIterable<Partial<Outputs>>, one item per genProcess yield. Early break/return on the iterator calls the generator’s return() so the node can clean up. Non-streaming nodes get no .stream member — the type surface tells you which is which. 4.4 Streaming input Nodes with the run(inputs, outputs, context) contract consume live streams. In the flow API their generated inputs accept AsyncIterable<T> (or an array, wrapped for convenience) on stream-typed handles, and the call returns an async iterable of emissions: const events = code.code.stream( { code: body, items: readLines(file) }, // items: AsyncIterable<string> ); for await (const { slot, value } of events) { ... } The adapter bridges the caller’s iterables into a StreamingInputs implementation (per-handle queues, EOS on iterable end) and surfaces outputs.emit(slot, value) as { slot, value } items. emitGroup flattens to its member emissions — correlation tokens are a runner concept and do not cross this boundary (see §5). The awaited form (await code.code({...})) is also defined for these nodes: drain everything, return last value per slot. 4.5 Errors, cancellation, control flow A node that throws rejects the call with the original error. There is no verdict machinery, no --supervise, no per-node error message stream — the caller’s try/catch is the supervisor. Retry, fallback, timeout, and concurrency limits are the caller’s JS (Promise.all, AbortSignal.timeout, p-limit-style helpers); the API deliberately ships none, because shipping them would be rebuilding the runner one combinator at a time. FlowOptions.signal aborts the flow: in-flight calls reject with AbortError, and the signal is forwarded into ProcessingContext so providers that honor abort stop mid-request. Per-call CallOptions.signal scopes cancellation to one call. 4.6 Python nodes Same policy as run() in core.ts: a node type no TS registry resolves is a Python candidate. The flow connects the bridge lazily on the first such call (via connectPythonBridgeForGraph’s single-node equivalent), reuses it for the rest of the flow, and closes it in close(). bridgeOptions and the NODETOOL_WORKER_URL env behave exactly as documented for RunOptions. 4.7 Observability Each call runs inside a node.process span carrying node.type, duration, and status — the same span name the kernel emits, so a trace of a native flow and a trace of a runner execution read with the same analyzer. FlowOptions.onCall receives { type, phase: "start" | "end" | "error", durationMs, error? } for callers who want progress without OTel. Cost rows land in the prediction ledger unchanged: providers bill through ProcessingContext, which the flow supplies. 5. What the runner provides that this does not Stated, not hidden. Choosing the flow API means choosing away: Runner feature Native flow answer node_update / job messages, editor progress UI onCall hook + OTel spans; no job row, no editor surface Correlation lineage, emitGroup atomic frames, zip semantics Gone. Grouping is the caller’s data structure Supervision (--supervise, escalations, verdicts) try/catch Per-item fan-out via streaming edges Promise.all(items.map(...)) Trigger nodes (webhook/tick/file-watch entry points) Out of scope — they need a resident run loop Graph-level static validation (nodetool validate) The type checker. Inputs are compile-time checked; there is no graph to validate Persistable/sharable artifact (workflow JSON, editor round-trip) None. A flow is code. When shareability matters, use the graph DSL That last row is the real boundary: if the artifact must open in the editor, be validated, supervised, or run by the server, build a graph. The flow API is for programs. 6. Codegen and package layout packages/dsl/src/flow/core.ts — startFlow, Flow, ALS ambient context, the invoke path (resolve → secrets → process), the streaming-input adapter, the default-flow singleton. Reuses the builtin-pack registration list currently inlined in core.ts run() by extracting it to a shared buildBuiltinRegistry() (one list, two consumers). packages/dsl/src/flow/generated/ — one module per namespace, emitted by the same npm run codegen script from the same registry metadata. Each node emits: the plain-value Inputs type (Connectable stripped), the callable, and .stream when is_streaming_output or the run contract is present. codegen:check covers both trees, so a renamed node cannot leave a flow callable behind. Package export: @nodetool-ai/dsl/flow. The root export stays the graph DSL; the two surfaces share types.ts (refs, model types) and the generated literal types. The Code node keeps its usesStreamInputContract probe on the body, applied at call time instead of graph-build time. 7. Implementation plan Eight tasks, ordered by dependency. T1–T3 are the core and land together or in sequence; T4 unlocks the typed surface; T5–T8 are independent once T4 is in. Every task ends with npm run test --workspace=packages/dsl green plus the repo gate (npm run typecheck && npm run lint), and any task touching packages/dsl re-runs npm run codegen:check. T1 — Extract the shared registry/executor chain Refactor only; no behavior change. Files: packages/dsl/src/core.ts → new packages/dsl/src/registry.ts. Steps: Move the builtin-pack registration block from run() (the registerBaseNodes + five try/catch optional packs, core.ts:340–377) into buildBuiltinRegistry(): Promise<NodeRegistry>, memoized per process — run() currently rebuilds it per call, which the flow API must not do per node call. Move the resolution chain (opts.registry → NodeRegistry.global → builtin → Python candidate) into createExecutorResolver(opts, bridge): (node) => NodeExecutor. Re-wire run() through both helpers. Acceptance: tests/core.test.ts and tests/integration.test.ts pass unchanged; registry.ts exports are consumed by core.ts only (until T2). T2 — Flow core: startFlow, ambient context, invoke path Files: new packages/dsl/src/flow/core.ts, packages/dsl/src/flow/invoke.ts; new test packages/dsl/tests/flow-core.test.ts. Steps: Flow / FlowOptions / CallOptions types as in §4.1–4.2. Flow owns one ProcessingContext (job id flow-<uuid>, userId default "1", secretResolver, FileStorageAdapter on the default assets path — same wiring as run-node.ts). Ambient context: module-level AsyncLocalStorage<Flow>; flow.run(body) = als.run(flow, body). Flow resolution helper resolveFlow(opts?: CallOptions): Flow — opts.flow → als.getStore() → lazy process-default flow (env secrets only, created once, never auto-closed). invoke(nodeType, inputs, opts) in invoke.ts: resolve executor via T1’s chain, fresh instance per call, initialize() → drain genProcess(inputs, ctx) folding last-value-per-slot → finalize() in a finally. Rejects with the node’s error unwrapped. close() + [Symbol.asyncDispose]: idempotent; closes the Python bridge when one was connected (T7); subsequent calls on a closed flow reject with FlowClosedError. onCall events (start/end/error with type, durationMs) and a node.process span per call via the existing tracing entry point the kernel uses. Tests (hermetic, no keys): one-shot call returns outputs (nodetool.text.Concat); multi-yield genProcess folds to last value; ambient vs explicit {flow} precedence (two concurrent flow.run bodies don’t cross); default-flow fallback works with no startFlow; error from process() rejects and finalize() still ran; closed flow rejects; onCall sequence recorded. Acceptance: all of the above green; no import of WorkflowRunner anywhere under src/flow/. T3 — Streaming both directions Files: packages/dsl/src/flow/streaming.ts; test packages/dsl/tests/flow-streaming.test.ts. Steps: invokeStream(nodeType, inputs, opts): AsyncIterable<Record<string, unknown>> — yields each genProcess record; iterator return()/throw() forwarded to the generator so early break runs node cleanup; finalize() on completion either way. StreamingInputs adapter for run-contract nodes: accepts AsyncIterable<T> | T[] | T per stream-typed handle; per-handle queue with EOS on source end; implements stream, any, first, and the envelope variants with synthesized plain envelopes. StreamingOutputs adapter: emit(slot, value) pushes {slot, value} into the returned async iterable; emitGroup flattens to member emissions (correlation tokens dropped — §5). Awaited form for run-contract nodes = drain the emission iterable, fold last-value-per-slot. Backpressure: the emission queue is unbounded is not acceptable — cap it (default 1024) and make emit await the consumer past the cap. Tests: stream yields in order; early break triggers generator cleanup (probe with a finally in a test node); array input wraps to a stream; interleaved two-handle any() order; EOS terminates run(); emission backpressure (producer awaits when the consumer is slow); emitGroup flattening. T4 — Codegen: the typed flow surface Files: packages/dsl/scripts/codegen.ts, packages/dsl/src/flow/generated/ (emitted), packages/dsl/src/flow/index.ts; extend packages/dsl/tests/codegen.test.ts. Steps: Emit one flow module per namespace next to the graph tree, from the same metadata pass. Per node: a plain-value Inputs type (today’s literal types minus the Connectable<...> wrapper — emit the inner type directly rather than unwrapping with a conditional type, so hover types stay readable), the callable (inputs, opts?) => Promise<Outputs> delegating to T2’s invoke. .stream member only when is_streaming_output (the flag read at codegen.ts:376) or the class declares the run contract; typed AsyncIterable<Partial<Outputs>>. Stream-typed input handles on run-contract nodes widen to T | T[] | AsyncIterable<T>. Code node special case: the callable applies usesStreamInputContract to inputs.code at call time to pick the invoke path. --check covers the flow tree (it already diffs the whole emitted set; assert with a deliberate stale file in the test). Namespace re-exports from flow/index.ts; keep the graph DSL’s namespace names. Acceptance: npm run codegen is idempotent; codegen:check fails on a hand-edited flow file (prove it once, revert); tsc accepts text.template({string: "x"}) and rejects {string: 1} in a type-level test. T5 — Package export and docs Files: packages/dsl/package.json, packages/dsl/README.md, AGENTS.md (this repo file’s DSL row), this doc’s status line. Steps: add the ./flow export map entry (mirroring the root entry’s four conditions, including nodetool-dev); README section with the §1 example; note in the exports table. Acceptance: node -e 'import("@nodetool-ai/dsl/flow")' resolves from a built workspace; npm run build:packages clean. T6 — Abort and error semantics Files: flow/core.ts, flow/invoke.ts; test packages/dsl/tests/flow-abort.test.ts. Steps: FlowOptions.signal and CallOptions.signal combined per call (AbortSignal.any); already-aborted signal rejects before resolve. Forward the combined signal into ProcessingContext so providers that honor abort stop mid-request; nodes that ignore it still reject at the next yield boundary. Streaming iterables settle their pending next() with the abort rejection and run generator cleanup. Tests: pre-aborted call rejects with AbortError and never invokes process(); mid-stream abort cleans up; flow-level abort rejects two concurrent in-flight calls. T7 — Python bridge lazy-connect Files: flow/core.ts (+ a small helper in packages/runtime if connectPythonBridgeForGraph’s graph-shaped signature doesn’t fit a single node type — prefer reusing its internals via a connectPythonBridgeForTypes(types, hasTsExecutor, opts) overload). Steps: on the first call whose type no TS registry resolves, connect once (single in-flight promise so concurrent first calls share it); reuse for the flow’s lifetime; close in close(). bridgeOptions / NODETOOL_WORKER_URL as in §4.6. Tests: unknown type with no bridge configured rejects with the existing “Unknown node type” error; bridge connect is invoked at most once under concurrency (fake bridge); integration test behind the existing worker-availability guard used by the graph DSL’s integration suite. T8 — Harness registry entry Files: packages/cli/src/harness/registry.ts. Steps: add a dsl-native-flow harness entry covering the packages/dsl surface, naming the T2/T3/T6 suites as the keyless selfcheck so nodetool harness gate runs them on diffs touching packages/dsl/src/flow/. Acceptance: nodetool harness audit passes with no new undocumented gap; harness gate --dry-run on a flow-touching diff plans the suites. 8. Out of scope Running untrusted code. The QuickJS sandbox remains the only surface for user- or model-authored code the host has not reviewed. Trigger nodes and resident flows. A migration of existing graph workflows. workflows export-dsl keeps emitting the graph DSL; a flow-emitting exporter is possible later but is not part of this design.--- # Editor Panels Source: https://docs.nodetool.ai/editor-panels.html The NodeTool Workflow Editor is surrounded by a left panel, an Inspector, a bottom panel, and a composer bar over the canvas. This page covers each panel in depth. Left Panel Opens from the icons down the left edge. It’s a tabbed drawer: click an icon to expand, click the same icon to collapse. The rail has five views: Documents, Chats, Library, Nodes, and More. The logo above them opens the app menu (see App Menu). Documents A searchable tree of every document in the active project, grouped as Workflows, Apps, Creative documents (sketches, scripts, storyboards, timelines, and entities), and Agents & code (JS scripts). Click a row to open the document as a workspace tab. Workflows, sketches, and the other kinds also have their own list views with richer actions, such as creating a sketch or timeline. For Apps, the list view has two header buttons: one scaffolds an app from a workflow and one starts an empty app. See Mini Apps. For sketches, see Sketch Editor. Chats Saved agent conversations for the active project. Click one to open it as a tab, or start a new one. See Chat. Library Your assets as a grid. Drag a file onto the canvas to create the matching input node. Open in full page opens the Assets explorer. Nodes The node browser. Search and browse all available nodes, organized into sub-tabs: All, I/O, Image, Image AI, Video, Video AI, Audio, Audio AI, 3D, Agents, and Control. Drag a node onto the canvas to add it. More A searchable list of additional panels, followed by the app pages. Group Entries Workflow tools Favorite Nodes (your starred nodes), Recent Nodes (nodes you added lately), Workflow Settings Agent tools Skills Workspace Workspace file browser App pages Tutorials, Examples, Costs, Model Manager, Package Manager (development builds only), Assets, Collections, Workspaces, Memory Workflow Settings edits the open workflow: name, description, tags, Run Mode (Workflow, Chat, App, or Tool), the associated workspace folder, and the Tool Name used when the workflow is exposed as a tool. It only appears while a workflow tab is active. Workspace shows the file tree of the selected workspace. Pick the workspace from the dropdown. Double-click a file to open it as a workspace tab. Open in Folder and Open Externally act on the selected file, and Open Externally is available in the desktop app only. Right Panel (Inspector) The right panel hosts the Inspector with a Cost estimate section under it. The panel opens by itself when you select a node, and i toggles it. Closing it yourself stops selection from reopening it until you open it again. (Logs, Queue, Trace, and Version History are not here. They live in the Bottom Panel.) Inspector — Node Properties The Inspector has three tabs: Params renders every property with the right input (number, slider, model picker, asset selector, dropdown, color picker, and so on), marks required ones, and shows validation errors for that node. I/O lists the node’s inputs and outputs. Help shows the node’s documentation, type, and namespace. With several nodes selected, the Inspector edits the properties they share and says how many nodes it changes. Mixed values are flagged. When no node is selected the Inspector has nothing to show, but the Cost estimate section below it stays. Edit the workflow’s own name, description, and tags in More → Workflow Settings. Cost estimate Under the Inspector, a collapsed Cost estimate section prices the open workflow for one run. Its header shows the total, or incomplete when some nodes have no known price. Expand it for a table with a row per node that uses an AI model: the node, its provider and model, the units, and the cost. A ~ means the figure rests on an assumed default, ≥ means it leaves out a known cost, and a question mark marks a node with no known price, which is left out of the total. The section is present whenever a workflow tab is open, and it takes at most 45% of the panel’s height. For how prices are found, what the markers mean, and the other places NodeTool shows costs, see Costs and credits. Bottom Panel The bottom panel docks runtime diagnostics and secondary workflow tools. Drag its top edge to resize. Its views are grouped: Run: Logs, Queue, Workers Workflow: Versions Debug: Trace Its header also carries the node and edge counts of the open workflow and a live readout of the server’s CPU and memory use. The figures come from the system_stats frame the server pushes every 5s (see WebSocket API), so on a local install they describe your own machine. A server that enforces auth (a hosted deployment) sends no such frame by default, because the numbers would describe a shared container, so the header shows the counts alone. Set NODETOOL_SYSTEM_STATS=1 to force the broadcast on, or 0 to turn it off. Logs Log lines from your runs, newest first, for the open workflow. Toggle Info, Warn, and Error to filter by severity. A node’s View logs button on its error panel opens this view filtered to that node and run, shown as a chip you can clear. A fullscreen button expands the table. l toggles the panel. Queue Jobs for your workflows in four columns: Running, Queued, Completed, and Cancelled. Workers The GPU workers panel. It lists provisioned worker instances with status, uptime, and estimated cost, provisions a new worker from a profile, and stops one or all of them. GPU pods bill continuously, so stop workers you no longer need. Versions A saved workflow keeps a history of versions, marked manual, autosave, or checkpoint. Review past versions, compare two, restore one, or delete one. Save Before Running in Settings → General → Autosave adds a checkpoint before each run. Trace The execution trace of a run, with per-node timing. A run selector in the header switches between recent runs. The toolbar copies the trace to the clipboard, exports it as JSON, or clears it. Ctrl/⌘ + Shift + T toggles the panel. Composer Bar A dock floating over the canvas. Its top part is the chat and media prompt (generated media is added to the canvas as nodes). Under it sits the row of workflow controls. Drag the handle at the left of the row to move the dock, and double-click it to reset the position. On narrow screens the dock is fixed to the bottom. Control When shown Action Add node Graph view Open the node menu Conversation Always Show or hide the conversation above the dock, with a count badge Auto Layout Graph view, not on mobile Auto-arrange the graph Save Not on mobile Save the workflow Trigger toggle Workflow has trigger nodes Arm or disarm the workflow’s triggers, with per-trigger schedule and last-fired details. See Triggers Stop While a run is starting, queued, or running Cancel the run Run Always Run the entire workflow. The label reads Run entire workflow, Starting, Queued #n, Running, Stopping, or Error · Retry, and a badge counts additional queued runs. Elapsed time shows while running. ⋮ Workflow actions Always Overflow menu (see below) The ⋮ overflow menu contains: Chain View / Graph View (toggle), Instant Update (on/off, re-runs the downstream part of the graph when you change an input value), Stop (while running), Auto Layout and Save (on mobile), Mini Map (show/hide), Download JSON, and Panels… (on mobile). Download JSON saves the workflow as <name>.json with an empty id. There is no Pause, Resume, or Fit button in the dock. A run cannot be paused or resumed. Tab Bar Controls The workspace tab bar holds a Project selector, a + New menu for creating or opening documents, and your open tabs. At the right edge, an activity indicator and the Notifications button show pending warnings and agent messages. App Menu (logo dropdown) The logo at the top of the left rail opens the app menu: Settings, Help, and Downloads. Settings opens as a workspace tab. The other app pages are in More. Customizing the Layout Each panel stays on its own edge. Click a rail icon to open or collapse it, and drag its inner edge to resize. Open or collapsed state and size are remembered between sessions. Three combinations you’ll land on most often: Left panel only, canvas taking the rest: Left panel plus the Inspector, for editing node properties while browsing: Everything open, with the logs docked at the bottom for a run: Next Steps Workflow Editor — building on the canvas Chat — how the in-editor chat works Configuration — settings that affect the editor--- # Desktop App Views Source: https://docs.nodetool.ai/electron-views.html The NodeTool desktop app is built on Electron. It reuses the web app for all workflow editing but adds a handful of native surfaces: a boot splash, package manager, log viewer, tray menu, pinned windows, and a native menu bar. This page catalogs every desktop-only view with screenshots. Everything inside the main window is covered by User Interface and Workflow Editor. Boot Message (Splash) A splash appears in the main window while the backend and web app come online. It shows the NodeTool logo and the latest boot message from the backend, for example “Setting up Python runtime (one-time, ~1-2 min)…”, “Installing Nodetool Python packages…”, or “Updating 2 package(s)…”. During setup it also shows a progress bar and a carousel of supported providers with a Get API Key link for each. A Show Log toggle expands the live startup log, and a folder button opens the raw log file. If the backend fails to start, the splash shows “Backend failed to start” with the error and three buttons: Retry start, Open logs, and Reinstall environment. There is no separate install wizard. Python is an optional runtime. You install it from the Runtimes panel of the Package Manager, and it is also installed automatically the first time you install a Python package there. Node packs and runtimes are managed from the Package Manager. Package Manager The Package Manager is a page inside the main window, not a native window. Open it from Tools → Package Manager in the menu bar or from the logo menu (Package Manager, which shows only when the app is served from localhost). It opens as a workspace tab on the /packages route. The left rail switches between two tabs: Software: system runtimes, grouped as All runtimes, Languages, Media & docs, and AI runtimes. Node Packs: grouped as Included, Registry, and Third-party. Each list is searchable (press /) and has status filters such as Installed, Not installed, Updates, Enabled, and Disabled. The page remembers the last tab and category. When a registry pack has a newer version, a notice reading “New Nodetool packs available” appears at the bottom right with the installed and latest versions. The notice points to the Package Manager for updating. See Node Packs for where packs come from and how to publish your own. Log Viewer A separate window (1200×800) that shows the backend log. Open it from Tools → Log Viewer or the tray. Each row is color-coded as info, warning, or error. The toolbar has a Search logs… box, a level filter (All, Info, Warning, Error), an auto-scroll toggle, Copy for the filtered rows, and Clear. Click a row to select it, Shift+click for a range, Ctrl/⌘+click to toggle. Selected rows are copied to the clipboard. Update Notification Auto-updates are opt-in and run only in packaged builds. Turn them on in Settings. When a newer release exists, the app downloads it in the background and shows a card at the top right of the main window: “Version X is available. Downloading in the background…” with a View Release Notes link. Once the download finishes the card reads “Version X has been downloaded and will be installed on restart.” with a Restart to Update button. A system notification also appears for both events. The update channel (stable or nightly) comes from the updateChannel setting. Other Windows Workflow Execution Window A frameless, transparent, always-on-top window that loads the workflow as an app (apps/index.html?workflow_id=...). It opens when you run a workflow from a global workflow shortcut and the workflow’s run mode is not headless. Headless workflows run without a window and write the last output to the clipboard. Mini-App Window A standard window for one workflow’s mini app, opened from Mini Apps in the tray. It is 80% of the primary display up to 1200×900, with a minimum of 800×600, and the title is “ - NodeTool". Closing it doesn't quit NodeTool. Chat Window Standalone chat opened from the tray or Tools → Chat. It uses the /standalone-chat route with the same size rules as the mini-app window. Only one chat window exists at a time. Opening it again focuses the existing one. Same content as the Chats panel. Performance Monitor A small window (700×420) opened from Tools → Performance Monitor. It shows a table of process metrics. System Tray The tray icon stays running while NodeTool is active. Click or right-click it to open the menu. From the top, the menu has: Item What it does Status line “NodeTool: Running on " or "NodeTool: Stopped" (not clickable) Start Service / Stop Service Start or stop the backend. Each is enabled only when it applies Show NodeTool Bring up the main window, or recreate it if it was closed Chat Open the standalone chat window (needs a running backend) Mini Apps Submenu listing your workflows by name. Disabled when the backend is stopped or there are no workflows Log Viewer Open the Log Viewer window Settings Open Settings in the main window Open Log File Open the raw log file in the OS file handler On Close Behavior Radio list: Ask Every Time, Quit Application, Keep Running in Background Quit NodeTool Stop the backend and exit On Windows, double-clicking the icon also shows the main window. On Close Behavior stores the windowCloseAction setting (ask, quit, or background). With ask, closing the last window shows a dialog with Quit and Keep Running in Background and a Remember my choice checkbox. Native Menu Bar The main window has a native menu bar built in menu.ts. On macOS the app menu (NodeTool) holds the standard About, Services, Hide, Hide Others, Unhide, and Quit items. Quit uses the platform default and has no explicit accelerator. File Item Shortcut Action Save Ctrl/⌘ + S Save the active workflow Settings   Open Settings in the main window New Workflow Ctrl/⌘ + T Open a blank workflow in a new tab Close Tab Ctrl/⌘ + W Close the active editor tab Edit Item Shortcut Undo, Redo, Cut, Copy, Paste, Select All Platform defaults Duplicate Ctrl/⌘ + D Duplicate Vertical Ctrl/⌘ + Shift + D Group Ctrl/⌘ + G Align none Align with Spacing Ctrl/⌘ + Shift + A View Item Shortcut Action Reload Platform default Reload the window Toggle Developer Tools Platform default Open or close DevTools Fit View Ctrl/⌘ + 0 Fit the graph to the viewport Reset Browser Zoom, Browser Zoom In, Browser Zoom Out Platform defaults Zoom the whole page, not the canvas Snap to Grid   Checkbox that mirrors the editor’s snap-to-grid setting Toggle Full Screen Platform default Enter or exit fullscreen DevTools only work in unpackaged (dev) builds, where Ctrl/⌘ + Shift + I also toggles them. Tools Item Action Chat Open the standalone chat window Log Viewer Open the Log Viewer window Performance Monitor Open the performance monitor window Model Manager Open the Model Manager tab Package Manager Open the Package Manager tab Help Open the Help dialog Downloads Open the model downloads dialog Vaults A radio list of your vaults with the active one checked. Picking another vault restarts the backend against that vault’s database and reloads the main window. Manage Vaults… opens Settings. Window / Help Window offers the platform-standard Minimize. Help has Learn More (opens https://nodetool.ai), Keyboard Shortcuts (Ctrl/⌘ + /), and System Information. The System Information dialog lists app, Electron, Chrome, and Node.js versions, OS details, install, conda, data, and log paths, and Python and CUDA status, with a Copy to Clipboard button. Opening a workflow execution window removes the application menu until the menu is rebuilt. Where the Electron Code Lives For contributors: electron/src/main.ts — entry point, window lifecycle. electron/src/window.ts — main window. electron/src/workflowWindow.ts — workflow execution, mini-app, and chat windows. electron/src/tray.ts — tray icon and menu. electron/src/menu.ts — native menu bar. electron/src/components/ — React components for the splash and update notices. electron/pages/ — the Log Viewer page. electron/src/updater.ts — auto-update flow. electron/src/perfMonitor.ts — performance monitor window. See the Electron developer guide for build and debugging tips. Related Docs Installation — download and first-run Configuration — settings persisted per-user Troubleshooting — boot and install issues Deployment — self-hosted / cloud alternatives--- # Entities Source: https://docs.nodetool.ai/entities An entity is a named, reusable thing that should look the same every time you generate it: a character, a location, a visual style, or a prop such as a product. Each entity has a reference image and a descriptor, a short description that NodeTool pastes verbatim into every prompt that uses the entity. The verbatim descriptor and the reference image are what keep a character’s look steady across shots. Entities support the Creative Agent and storyboards. See Creative Agent for how they fit into a production. Entity kinds Kind Use it for Character A person or creature that stays recognizable Location A place with a consistent setting and atmosphere Style A visual language to apply across generations Prop A product or object that should keep its appearance Only a character has the optional Voice id field, which holds a provider-specific voice id for text-to-speech. Open the library There are two places to manage entities. Sidebar panel. Open Entities in the left sidebar. It lists the entities for the active project as a grid of cards. The Open in full page button in the panel header opens the full library as a workspace tab. Full-page library. The page titled Entities shows every entity you have, with the caption “Reusable characters, locations, styles, and props”. It has an Add entity button. With no entities it shows “No entities yet”. Each card shows the reference image, the name, a kind chip, and the first two lines of the descriptor. Two buttons sit on the card: Edit entity and Remove entity. Create an entity Every entity is built on an image. There are two ways to start. From the full-page library Add entity opens a three-step flow. Back to entities returns to the grid. Details. Pick the Entity type, enter a Name, a Descriptor, and optional Tags (comma-separated). Name and descriptor are required. Describe only stable visual traits in the descriptor, for example “A young astronaut with cropped black hair and a worn orange flight suit”. The step also offers Start with a blank reference, which makes a plain white canvas the reference image and jumps to the review step. It fills the name with “Untitled entity” and the descriptor with “A reusable visual entity.” only where you left them empty. Reference. Choose the reference image. Choose from assets picks an existing image from your assets. Generate with AI opens Generate a reference image, where you pick a view, edit the prompt, and choose an image model. The views depend on the kind. A character offers Full body, Portrait, and Character sheet. A location offers Establishing view, Detail view, and Location sheet. A style offers Style sample, Style sheet, and Texture detail. A prop offers Product view, Detail view, and Turnaround sheet. An image that is already an entity cannot be picked again. Review. Check the image, name, kind, descriptor, and tags, then choose Create entity. NodeTool keeps an unfinished flow as a draft, so leaving the page does not lose your input. From an existing image In the sidebar panel, the New entity button (+) opens Pick a reference image. Choose an image and the New entity dialog opens with the image, Kind, Name, Descriptor, Voice id (optional) for characters, and Tags (comma-separated). Create stays disabled until name and descriptor are filled. If you have no images, the picker says “No images”. Generate or upload one first. Edit or remove an entity Edit entity opens the Edit entity dialog with the same fields. Change image swaps the reference image. The entity keeps its id when you swap, so storyboards and scripts that already use it stay linked. Choose Save to apply the change. Remove entity removes the entity but leaves the image in your assets. Script speakers linked to a removed entity keep the stale id until you clear it. Reference images The first reference image is the primary one. When an entity is used in a generation, its reference image goes along with the prompt as an image reference for models that accept one. An entity is stored on its image asset as metadata, so it needs no separate storage and appears wherever assets do. How entities are used in generation When an entity applies to a prompt, NodeTool appends a block to it: Consistency references: - Nova: A young astronaut with cropped black hair and a worn orange flight suit The rule for which entities apply is the same on every surface. With explicit entities selected, exactly those apply. With none selected, an entity applies when its name appears in the prompt text, ignoring case. With an empty prompt, every entity applies. An entity with an empty descriptor is skipped. Where this happens: Surface What it does Storyboard The Entities field under Board settings pins a cast to the board. Styles and locations season every shot. Characters and props apply to shots that name them. Each shot’s Edit view shows the cast as chips you can include or exclude for that shot. See Creative Agent. @ mentions The mention picker in the chat composer and the Prompt node lists entities first. A picked entity carries its descriptor and reference image into the generation. Workflow nodes nodetool.creative.ApplyEntities injects descriptors into a prompt and outputs prompt and reference_images. LoadEntity, ListEntities, and CreateEntity read and write the library. See the node table in Creative Agent. Entity picker Entity properties on workflow nodes open a picker titled Pick an entity. It has a search box (“Search entities by name, kind, or tag”) and filters for All, Characters, Locations, Styles, and Props. Agent tools The agent and the in-browser assistant use these tools. Tool What it does list_entities Lists entities with id, name, kind, and descriptor. Filters by kind, text in the name or descriptor, and project. get_entity Reads one entity in full, including tags, reference images, voice, palette, and LoRA. apply_entities Returns a prompt seasoned with descriptors, plus the asset ids of the reference images. create_entity Tags an image asset as an entity. The image must exist already, so generate it first. update_entity Changes any field, or swaps the picture. Only the fields you pass change. delete_entity Removes the entity marker and keeps the image. ui_entity_list, ui_entity_apply The same list and apply behavior from the browser. Tools take the entity id, which is the id of its image asset.--- # Error Tracing Source: https://docs.nodetool.ai/error-tracing.html Error Tracing NodeTool records server failures, failed workflow runs and client crashes as redacted error traces in its own database. Agents can query them, a bug report can attach them, and on the hosted service they are queryable in Supabase. Nothing is sent to a third-party error tracker. What is recorded Source When trpc A tRPC procedure fails with a 5xx status http A REST handler throws with a 5xx status job A workflow run fails, or fails before it starts web / electron The route error boundary or a panel error boundary catches a crash Client mistakes (4xx) are not traced. Each trace stores the error type, the message, the stack, a fingerprint that groups repeats, the app version, the platform, and a small context. The context keeps only identifiers listed in ERROR_TRACE_CONTEXT_KEYS (packages/models/src/error-trace-redaction.ts): job, workflow, node and thread ids, the route, the HTTP status and similar. Prompts, node inputs, outputs and request bodies are never stored. Redaction redactErrorTrace runs before every insert, on both the local capture path and the sync ingest path. It removes provider credentials by shape, JWTs, bearer tokens, PEM keys, secret-named key=value pairs, signed URL parameters, URL credentials, the values of credential variables in the server environment, email addresses, non-loopback IPv4 addresses, home-directory names, data: URLs and long opaque strings. Messages are capped at 2,000 characters and stacks at 40 frames. The table is in the personal-data registry, so account export includes the traces and account erasure deletes them. A process stores at most 10 traces of one fingerprint and 300 traces in total per minute. Traces older than 30 days are pruned at startup and once a day. Reading traces Surface How Agents (chat and MCP) list_error_traces, get_error_trace, export_error_report CLI nodetool errors list, show <id>, export [ids…] [-o file], sync, prune tRPC errorTraces.list, summary, get, report Bug report dialog The “Recent server errors” section attaches the last hour as server-errors.md Every surface reads only the caller’s own traces. A trace id can be given in full or as its 12-character prefix. Supabase On a deployment whose database is Supabase PostgreSQL, the traces live in nodetool_error_traces in that database. The create_error_traces migration enables row-level security and adds the error_traces_owner_read policy, so the Supabase Data API returns a signed-in user only their own rows and returns nothing to anonymous callers. The server connects as the table owner and is not affected. Query across users in the Supabase SQL editor, for example: SELECT fingerprint, count(*), max(created_at) AS last_seen, min(message) AS message FROM nodetool_error_traces WHERE created_at > (now() - interval '1 day')::text GROUP BY fingerprint ORDER BY count(*) DESC; Syncing a local install Sync is off by default. To push a local install’s traces to a NodeTool server, set both variables: Variable Value NODETOOL_ERROR_TRACE_SYNC_URL The server’s base URL. Must be https, except for localhost NODETOOL_ERROR_TRACE_SYNC_TOKEN An access token for your account on that server The server then pushes unsynced local traces every 5 minutes to errorTraces.ingest, and nodetool errors sync pushes them on demand. The receiving server stores them under the token’s user and redacts them again. Received traces are never forwarded. A local install never holds database credentials for the cloud. Other settings Variable Effect NODETOOL_ERROR_TRACES 0, false or off stops server-side capture NODETOOL_ERROR_TRACE_RETENTION_DAYS Retention window in days. Default 30--- # creative-pipeline — a live run’s output Source: https://docs.nodetool.ai/evals/creative-pipeline/ creative-pipeline — a live run’s output What one run of the creative-pipeline eval produced with --live, kept here so the suite’s output can be looked at without paying for a run. The suite is headless by default: every generate and render is a status flip against in-memory state, which is what makes it cost the agent loop and nothing else. --live additionally routes the same tool calls to fal, so these files are what the model directed, not illustrations of it. The commission The brief (LANTERN_BRIEF) asked for a vertical social spot for a cold brew: 9:16, under 12 seconds, must feature hands and sunrise, no logo. claude_agent_sdk/sonnet read the brief, proposed three concepts, committed to one, built a style frame, wrote and rendered four shots, assembled the cut, measured it, trimmed it, and signed it off — 162 tool calls, delivered at 11.80s. Files File Phase style-frame.webp Sketch — the look the shots follow keyframe-shot_{1..4}.webp Storyboard — one still per shot clip-shot_{1..4}.mp4 Storyboard — each still animated Stills come from openai/gpt-image-2, clips from fal-ai/ltx-2-19b/distilled/image-to-video. Compressed for the repo: stills to WebP at 720px, clips to h264 at 540px. Raw output became 1.8 MB, which is the difference between a fixture and a liability. The originals are not kept. Reproducing FAL_API_KEY=$FAL_KEY IS_SANDBOX=1 npx tsx \ packages/agents/scripts/dump-creative-run.ts full-pipeline claude_agent_sdk sonnet 220 --live Output lands in nodetool-debug/ (gitignored) with a full transcript and per-phase state snapshots alongside the media. Roughly $2.60 for the agent loop and $0.17 for the media — flux/schnell at $0.003/megapixel and ltx-2-19b/distilled/image-to-video at $0.0008. The model driving the pipeline costs an order of magnitude more than the pipeline’s output. Why not a cheaper image model The first pass used flux/schnell at $0.003 per megapixel, chosen on cost. It was the wrong trade: flux mangles hands, and this brief requires them in three of four shots. A model that cannot draw the thing being commissioned is not cheap. gpt-image-2 costs materially more per image and renders anatomy that survives a look. Swapping it in introduced a different brief violation, which is the more interesting half. flux could not render legible text, so it never breached the brief’s no logo rule by accident. gpt-image-2 can, and branded the bottle in shot 3 with “COLD BREW COFFEE” in raised lettering. Better capability, new failure mode. The prompt is why. Sonnet never carried no logo into any image prompt — it lives in the brief, and the model read the brief, but the constraint did not survive into the generation call. Shot 3 was regenerated with the constraint appended explicitly. Two things these files show that the eval cannot The planted defect is conservative. The suite simulates rendered clips coming back 1.35× longer than requested. LTX returned 4.84s takes for 3s requests — 1.61×. Real video models overshoot harder than the eval assumes. Two brief violations the predicates cannot see. clip-shot_3 contains neither hands nor sunrise; the model caught that in its own review and filed it, while the suite’s mustFeature check passed because it only asks whether each element appears somewhere in the shot list. And the same shot came back branded, which forbiddenAvoided also passed — that check reads shot action text and layer names, so it can see the word “logo” in a prompt and never what the picture actually contains. Both are the same limit. The predicates grade the plan; only a human or a vision model grades the artifact. Worth knowing before trusting a green run to mean the deliverable is on brief.--- # Workflow execution lifecycle Source: https://docs.nodetool.ai/execution-lifecycle.html Workflow execution lifecycle Construction inventory before convergence This inventory comes from repository-wide searches for new WorkflowRunner, WorkflowRunner(, connectPythonBridgeForGraph, hydrateGraphNodeFlags, and withExplicitNodeFlags. Paths identify call sites without depending on line numbers. Tests under tests/ and __tests__/ construct the kernel directly. Construction site Classification Responsibilities before migration packages/execution/src/session.ts Top-level server execution Normalize, hydrate, preflight, resolve executors, construct context and runner, connect and close Python, timeout, cancel, live input and properties, message capture, cost ledger, worker boundaries, optional output naming and persistence, scratch cleanup packages/execution/src/service/workflow-run.ts Top-level server execution Normalize, metadata hydration, preflight, registry or host resolver, start shared Python worker, workspace context, runner, interactive cancel, progress and terminal persistence, output naming, result presentation. No run timeout or scratch cleanup packages/dsl/src/core.ts Top-level server execution Build graph with explicit flags, construct context, caller/global/builtin/Python resolution, connect and close Python, runner, throw on failed result or node error, select last output. No preflight, timeout, persistence, live inputs or property updates scripts/run-workflow.mjs Top-level server execution Load file, prepare registry and context, select providers, start local Python worker, resolve executors, runner, print result, close Python on success. No cancellation, timeout or workspace cleanup packages/workflow-runner/src/run.ts Browser/portable execution Registry hydration and validation, context and runner, AbortSignal cancellation, queue/wake message streaming, completion and result propagation, listener cleanup. No Python, persistence or output rewriting packages/workflow-runner/e2e/src/browser-entry.ts Specialized browser harness Prepare context, registry and sandbox catalog, hydrate graph, runner, record messages and traces, browser test controls. No server lifecycle packages/core-nodes/src/nodes/run-inner-graph.ts Nested execution Normalize child graph, inherited metadata and executor resolution, inherited context, runner, failure interpretation and parent output mapping. No independent Python, persistence, timeout or workspace cleanup examples/workflow_runner/js/main.js Browser execution, different class Constructs the example’s HTTP/WebSocket client named WorkflowRunner, not the kernel runner. Owns transport and UI Electron’s createWorkflowRunner() constructs a client store, not the kernel runner. README examples are documentation rather than execution entry points. Boundary Top-level Node/server execution uses ExecutionSession. Public APIs keep their own graph preparation and result presentation. Nested execution can construct a child kernel runner with its inherited environment. Browser execution uses the portable runWorkflow API. Kernel tests and browser harnesses can construct the runner directly. npm run check:execution-boundary guards production sources. Session responsibility audit Behavior Ownership Normalize, hydrate, construct runner, cancel, timeout, terminal result Canonical lifecycle Python connection, worker job boundaries and owned bridge close Canonical lifecycle. A shared host bridge remains host-owned Scratch cleanup, including setup failure Canonical lifecycle buildWorkspaceExecutionContext Server default. An injected context overrides secrets, storage, workspace and generation durability Headless permission gate Server default, explicitly disabled by hosts preserving another permission policy Cost ledger Optional integration, enabled by default for existing server callers Durable generation tracking Context preparation, enabled by the default server context and saved-workflow service. Injected contexts choose their own hooks Output name rewriting Optional presentation policy, enabled with requireTerminalResult Persistence Optional host hooks. Job rows, progress, interactive sessions and receipt formatting remain in adapters Buffered message iteration Portable coordination shared with the browser runner The DSL keeps its explicit graph flags and output keys, secret resolver and registry precedence. Its preflight and cost recording remain disabled. Its result adapter continues throwing on failed runs and node error messages. Shared-server bridge bootstrap and metadata preparation remain host work. WebSocket graph preparation still precedes session normalization and hydration. Nested execution retains its parent context and output mapping. Adapter changes DSL run() and runGraph() retain their signatures and return records of the last value for each nonempty output. Registry resolution stays caller registry, global registry, builtin packs, then Python. The DSL no longer connects or closes Python, constructs a runner, or invokes the kernel run method. Graph conversion, context inputs, registry policy and result-to-exception conversion stay in the adapter. The saved-workflow service and scripts/run-workflow.mjs also use the session. The file runner keeps its existing preflight, permission and cost-recording policies through explicit options rather than inheriting server defaults. The service retains ownership checks, preflight before lazy bootstrap, metadata preparation, job progress, background receipts and interactive verdicts. Its shared Python bridge now receives session job boundaries without being closed. Actor failures now produce a canonical failed run. The legacy DSL check for node error messages remains separate from the session and is tested against a message-only result. A cancellation can also retain node errors while reporting a cancelled terminal status. The session adds these options for known host requirements: Option Requirement executorResolverFactory DSL registry precedence needs the bridge the session owns hasTsExecutor Python detection must consult every DSL TS registry bridgeOptions Preserve DSL remote worker configuration without wrapping bridge lifecycle preflight Preserve DSL behavior and avoid repeating the service’s pre-bootstrap preflight installHeadlessPermissionGate Preserve an injected DSL context’s permission policy context and recordCosts already existed. The DSL uses them to preserve its secret resolver and avoid adding persistence. Behavior verification The DSL differential tests retain the previous kernel path as a test oracle. They compare outputs, status, errors and retained messages for independent outputs, actor failure and an empty terminal. Job IDs and durations are excluded from comparison. Registry tests cover caller precedence, global precedence over builtin packs, builtin fallback and Python fallback with required secrets. Python tests stub the transport rather than starting a worker. Lifecycle tests cover bridge closure on completion, failure and partial setup, workspace cleanup on setup failure and execution errors, explicit cancellation and timeout. The cleanup test fails against the previous session when hydration throws after connecting the bridge. Existing runtime tests cover closure after local or remote bridge connection failures. Portable tests cover ordered delivery, terminal delivery after abort, queue draining, listener cleanup and abort before start. The cancellation terminal test fails against the previous portable runner. Message-buffer tests drain a large backlog and exercise overflow. Architecture tests exercise renamed and namespace imports while ignoring comments and types. No output, status, retained-message or exception differences were found in the representative DSL comparisons. Observable lifecycle changes are scratch cleanup after setup failure, shared-worker job boundaries in service runs, and portable terminal delivery after cancellation. Early portable iterator closure now cancels and waits for the run instead of leaving it executing without a consumer. Remaining duplication The saved-workflow service and WebSocket host still prepare metadata before the session hydrates the graph. Moving that preparation requires preserving unknown nodes, list property types and each surface’s output keys. Server preflight is also performed before lazy service bootstrap, using the shared preflight helper. Shared bridge bootstrap remains a host responsibility because its lifetime spans runs. Nested graph normalization and output mapping remain local to the child execution contract.--- # Execution Strategies Source: https://docs.nodetool.ai/execution-strategies.html This page covers two distinct mechanisms that are easy to conflate: Workflow execution — how the kernel runs a graph of nodes. This is an in-process actor model with message-passing; there are no per-job threads, subprocesses, or containers. Code execution — how the Code node runs arbitrary user JavaScript in a QuickJS WebAssembly guest. That sandbox covers the code inside one node, not the workflow. See also Architecture for the system overview and Automatic Message Correlation for the lineage model the scheduler relies on. Workflow Execution: the actor model A workflow is a DAG. The one allowed cycle is a loop that closes on a Loop node’s feedback input. The runtime lives in packages/kernel/: WorkflowRunner (packages/kernel/src/runner.ts) orchestrates one job. NodeActor (packages/kernel/src/actor.ts) runs a single node. NodeInbox (packages/kernel/src/inbox.ts) buffers each node’s inputs per handle and tracks how many upstream sources remain open. There is no JobExecutionManager and no pluggable “threaded / subprocess / docker” strategy for running a workflow. The whole graph runs concurrently in one event loop via async actors that pass messages to each other’s inboxes. What WorkflowRunner does WorkflowRunner.run(request, graphData) first emits job_update with status running, so a run that fails validation still tells clients it started. Then it performs, in order: Bypass rewrite — rewriteBypassedNodes re-routes around nodes flagged ui_properties.bypassed. Invalid-edge filtering — drops edges whose source or target node is missing. Correlation analysis — analyzeCorrelation (mandatory) computes the static lineage scope of every node input/output. Issues abort the run with a GraphValidationError before any actor starts. Graph + node validation — structural validation, an optional per-node validateNode callback, and a check that every workflow input node (nodetool.input.*) has a value, either in the request params or in its own value property. Inbox initialization — one NodeInbox per node, seeded with the count of incoming data edges per handle (and one __control__ upstream per unique controller). Node initialization — resolves and caches one executor instance per node. Input dispatch — _dispatchInputs runs each external input node’s process() once and delivers values to downstream inboxes. Non-streaming inputs then signal end-of-stream (EOS); streaming-output inputs (e.g. RealtimeAudioInput) stay open for later pushInputValue() calls. Actor spawning — _processGraph creates a NodeActor per node and runs them all with Promise.all. Actors block on their inboxes until upstream data arrives, so nodes whose inputs are ready run concurrently while the rest wait. The runner also routes outputs downstream (_sendMessages), tracks per-edge message counters for the UI (throttled edge_update events), propagates EOS (_sendEOS), and emits job_update / output_update messages through the ProcessingContext. Job outcomes run() returns a RunResult with status completed, failed, or cancelled. Precedence at finalization is cancel > failed > completed: a node error fails the whole job, and cancel() aborts the run-level AbortController and closes every inbox to unblock waiting actors. The four actor modes NodeActor picks a mode from the node’s hydrated behavior flags. (The runner requires a hydrated graph: the flags below must be set, or streaming nodes would silently run as one-shot process() calls.) Mode Trigger Behavior Buffered default Gathers inputs via the correlated scheduler and calls process() once per ready input set. Streaming input is_streaming_input Node drains its inbox itself via NodeInputs (run(inputs, outputs, ctx)), emitting through NodeOutputs. Streaming output is_streaming_output Calls genProcess(), which yields partial frames; each yield is routed downstream as it is produced. Controlled is_controlled Waits for control events on the __control__ handle, caches data inputs, and re-runs process() per run event until controllers signal EOS. Buffered and streaming-output nodes share the same correlation-aware gather path (_runCorrelated); the difference is whether the executor exposes process() or genProcess(). Two cases are checked before those modes: Trigger entry. When a delivered trigger event targets a node with is_trigger, the node does not listen. It emits the event payload on its declared outputs and completes. A run started from the editor’s Run button falls through to the normal modes. Loop. nodetool.control.Loop runs in a kernel-owned loop mode. The kernel handles iteration, feedback matching, and termination, and the node’s process() is never called. max_iterations defaults to 10 and is clamped to See Workflow Loops. Correlation-aware scheduling (_runCorrelated) Buffered/streaming-output nodes do not simply wait for “all inputs.” They schedule per correlation key so that fan-out/fan-in over iterations stays correctly paired. _runCorrelated (in actor.ts) classifies each connected data handle by its static scope: max-scope — scope length equals the node’s invocation scope; bucketed per projected lineage key. One handle may be the repeating driver (repeats_per_key), firing once per arriving envelope. strict-prefix sticky — a shorter non-empty scope; the latest value is kept per projected parent key (side inputs that change less often). empty — empty scope; a single sticky value (or the node’s declared property default). As envelopes arrive on iterAnyWithEnvelope(), the actor records each into its bucket and re-evaluates which keys are now ready. A key is ready when every handle has a value for it (lists wait for the handle to close so the full set is captured). Each ready key fires process()/genProcess() with its matched inputs, and the produced outputs inherit (or mint) the correct lineage so downstream joins line up. Source nodes (no connected data handles) fire once with empty inputs. Inbox safety limits (max_pending_keys, max_pending_messages_per_key) abort the run if pending keys grow without bound — typically a missing upstream close. See correlation-design.md for the full model: lineage as root-id → item-token maps, static scope analysis, and done / dropped-key / scope-close propagation. Sync modes (legacy framing) The older zip_all / on_any / sticky sync-mode vocabulary maps onto the correlation scheduler: zip_all-style “wait for all handles” is the max-scope readiness rule, sticky is the strict-prefix / empty-scope handle classes, and on_any is the per-arrival driver behavior. New nodes declare correlation via input scope and output_correlation rather than a single sync_mode flag. Control events Controller nodes (e.g. agent nodes) drive controlled nodes over control edges. The runner builds a _control_context describing each controlled node’s properties and “run” action schema, injects it as an input, and exposes sendControlEvent(targetNodeId, properties) (via ProcessingContext.setSendControlEvent) so a controller can dispatch a run and await that node’s next output. Responses are tracked FIFO per node so a burst of concurrent control calls doesn’t drop a waiter. Code Execution: the sandboxed Code node Separately from workflow scheduling, the Code node (nodetool.code.Code, packages/code-nodes/src/nodes/code-node.ts) executes user JavaScript in a QuickJS WebAssembly guest — see @nodetool-ai/agents/js-sandbox. It isolates the code in one node, not the workflow, and runs the same way in the browser and on the server: no Docker, no subprocess, no host interpreter. The guest gets standard JavaScript plus a fixed set of bridges: fetch(), workspace, getSecret(), sleep(), progress(), crypto, format, the image, audio, video, media, and canvas helpers, and emit() and output() for results. Dynamic inputs arrive on the inputs object. Libraries come from sandbox packs that the body imports. See JavaScript Sandbox for the full surface, limits, and security model. Cancellation and shutdown Workflow: WorkflowRunner.cancel() aborts the run-level AbortController (observed by node code via inputs.signal) and closes every inbox so waiting actors unblock; the job finalizes as cancelled. Code node: the QuickJS guest is torn down with its host call; the run’s abort signal ends any in-flight bridge call. Related Architecture — system components, message types, job lifecycle. Automatic Message Correlation — lineage, scopes, and the scheduler design. packages/kernel/ — runner.ts, actor.ts, inbox.ts, correlation-analysis.ts. packages/code-nodes/ — the sandboxed Code node.--- # Failure-mode roadmap Source: https://docs.nodetool.ai/failure-mode-roadmap.html Failure-mode roadmap Validated against main on 2026-08-22. Open items name the remaining code, tests, and acceptance conditions. Completed work is recorded at the end. Do not implement a check until it fails on a reproduction or positive-control fixture. An audit must enumerate its targets and assert that it found them. 1. Add an authenticated production /mcp endpoint Status: OPEN. Issue #5126. packages/websocket/src/server.ts mounts /mcp only outside production. The development mount always uses user "1", so it cannot be enabled unchanged. Python is not part of this item; its production opt-in already exists. Implementation: Add NODETOOL_ENABLE_MCP=1. Keep the production route absent by default. When enforceAuth is true, use the existing global bearer-token hook and build agentToolsScope from req.userId with source "http-session". When enforceAuth is false, require a new NODETOOL_MCP_TOKEN of at least 32 characters. Validate it inside the global onRequest hook, before the generic remote-client denial. Compare it in constant time. Do not add a general /mcp public-route exemption. Add "http-token" to the MCP scope source union and use user "1" only after the token passes. Return 401 for missing or invalid credentials. Keep the global rate limit, CORS policy, and JSON-RPC input validation on the route. Add both variables to the config setting catalog and its Zod validation. Add production route tests under packages/websocket/tests/ for both auth modes, a non-loopback client, invalid credentials, the default 404 result, and authenticated GET/DELETE requests after session initialization. Add an opt-in probe to scripts/docker-smoke.mjs. Send a complete MCP initialize request, including jsonrpc, id, protocol version, capabilities, and client information. Document the two MCP flags in docs/configuration.md and AGENTS.md. Acceptance: Authenticated production derives the MCP user from the validated session. Local production requires the dedicated token. Production without the enable flag returns 404. npm run test --workspace=packages/websocket passes. 2. Include missing secrets in validate_workflow Status: OPEN. validate_workflow calls validateGraph in packages/agents/src/capabilities/workflows.ts. Changing modelSelectionError does not change this capability. Implementation: Add optional availableSecrets(keys) to CreateCapabilityRunOptions and propagate it through createCapabilityRun to CapabilityRun. Pass it to validateGraph from validate_workflow. Enumerate every createCapabilityRun call site in an audit test. Server, CLI, and MCP hosts with a real secret resolver must inject a callback that uses their ProcessingContext. Hermetic eval hosts must omit it explicitly. A missing_secret issue must name the key and direct the agent to Settings > Credentials. Mention request_secret only when that capability is present in the run’s registered capability set. Tests must show that a missing key produces an issue, an available key clears it, and an absent callback preserves current output. Run npm run test --workspace=packages/agents -- workflows. 3. Make media resolution the browser rendering boundary Status: OPEN. Follow-up to #4873, #4929, #5028, #5078, #5122, and #5123. The reported leaks are fixed. Canonical resolution exists in web/src/utils/resolveMediaUri.ts and web/src/hooks/useResolvedMediaUri.ts. asset:// remains a valid stored locator, so a zero-hit grep is invalid. Implementation: Add a ResolvedMediaUrl branded string to resolveMediaUri.ts. Brand only a non-empty resolved URL; keep missing results null or undefined. Extend ResponsiveImage and VideoPlayer instead of creating parallel image/video primitives. Add one audio primitive. They accept a MediaLocator and resolve it before setting src. Migrate chat media, storyboard cards, sketch layers, script shot chips, node previews, and App Builder media widgets. Check in a consumer inventory for the named surfaces and assert that each uses a locator-aware primitive. Add a design-lint rule that rejects an asset:// string literal in a JSX src. Include a positive-control fixture. Acceptance: upload -> graph input -> render works for image, video, and audio; the primitives cover stored locators and HTTPS URLs; the lint fixture proves the rule can fail. 4. Preserve provider behavior through decorators Status: OPEN. Issue #5109. CassetteProvider inherits generateLoop and can drop an inner override. A method-existence test cannot detect this. For each decorator method, either forward it or reject construction when the behavior cannot be preserved. Cover the actual BaseProvider contract: generateMessage, generateMessages, traced wrappers, generateLoop, model discovery, media generation, tool support, error classification, lifecycle, and cost. Replace the diagnostic expectation in packages/runtime/tests/providers/provider-decorator-inertness.test.ts with behavioral equivalence checks using unique sentinel results. Start with CassetteProvider and add each later wrapper to an explicit inventory. In record mode, compare forwarded behavior with the inner provider. In replay mode, compare the recorded normalized contract and cost; do not require calls to the inner provider. The current implementation must fail the new test before the fix. Then run npm run test --workspace=packages/runtime -- provider-decorator-inertness. 5. Inventory and consolidate SSRF screening Status: DONE, shipped 2026-08-22. See SSRF screening inventory and consolidation under Completed work. The numbering stays so older references still resolve. 6. Add property tests to seven pure helpers Status: OPEN. Use one PR per row. Before adding fast-check, record why the current dependencies and table-driven tests are insufficient, check its latest maintenance date, and inspect its transitive tree with npm ls fast-check. Add it only to the owning workspace. Issue Helper Workspace Property #4909 normalizeGraph and graph validation execution, node-sdk Malformed arrays never execute; normalization is idempotent. #4910 deriveImageSizePreset web Rotated dimensions preserve the preset; positive dimensions return a valid preset. #5035 clampTimeoutSeconds agents Positive finite inputs stay in bounds; sub-second values never become zero. #5091 matchesFileWatchPattern automation-nodes Line terminators and glob metacharacters do not alter escaped filename semantics. #4939 hasYieldStatement node-sdk Comment markers in strings do not change executable yield detection. #4987 findMissingModelNodes web A missing provider is reported; a valid provider/model pair is not. #5116 mergeIntoSequence web Reassembly preserves track indexes and order, including duplicate indexes. Each PR pins the original counterexample, defines generator bounds and a replay seed, runs in the normal workspace suite, and proves failure by restoring the original defect once. Before generating values, the test must define its independent oracle: the complete valid preset set for #4910, parser-derived executable yield locations for #4939, documented glob semantics for #5091, and stable source order as the tie-breaker for duplicate indexes in #5116. Split #4909 into one execution normalization property and one node-sdk validation property. 7. Complete editor input-path coverage Status: OPEN. Use three separate PRs. 7A. Shortcut action mapping web/src/config/shortcuts.ts already owns shortcuts and rejects duplicate slugs. Add a typed action ID used by menus and handlers. Test missing actions, duplicate normalized combinations within the same active editor context and OS, Electron-only actions in web menus, and the command-menu shortcut on Windows and macOS. Duplicate combinations in disjoint contexts remain valid. 7B. Canvas drop journeys Add web Playwright journeys for file drop, node creation from a dropped connection at the cursor, and run-selected with multiple nodes. Use npm --prefix web run test:e2e. For the Windows Explorer defect, first add an Electron Playwright dependency, config, script, packaged-app fixture, and Windows CI job. Update AGENTS.md and Electron testing docs with the command. Run the case on Windows; a Windows-style string on macOS is not sufficient. 7C. Dialog containment audit No dialog primitive clamps to the viewport today — PositionedDialog, which did, is deleted. Enumerate the dialog primitives and direct users in a checked-in audit with a non-zero count. A dialog fails the audit when its rendered bounds extend outside a 600 x 600 px viewport or its content cannot scroll into view. Migrate failing callers and add one 600 x 600 px test per migrated primitive. Completed work An eval case or a suite for each agent capability: shipped 2026-08-22. packages/cli/src/harness/capability-table.ts names every exported capability — covered, or carrying a written gap note — with the implementation file, the suites the capability-suites selfcheck runs, and the eval cases whose expect.requiredTools demand them. The live tally is nodetool harness capabilities. scripts/sync-capability-coverage.mjs (npm run capabilities:sync / :check) derives everything but the gap notes from the live registry, the agent suites, and the eval case files, so a new capability with no check fails the check rather than review. Each entry carries a fingerprint of what the capability declares, and nodetool harness gate --base <ref> refuses a contract change whose coverage mapping stood still while saying nothing about a refactor. Fixtures in packages/cli/tests/harness-registry.test.ts, the table’s own audit in capability-coverage.test.ts, the rule in packages/agents/AGENTS.md, and .github/pull_request_template.md. Live provider contract probes: shipped 2026-08-22. The manifest packages/runtime/src/providers/contract/probe-manifest.ts names, per entry, the provider, the endpoint, the production decoder, a checked-in raw HTTP response fixture, the required fields whose removal must break the check, and — for one entry per provider — the single live request, capped at one request and USD 0.05 per provider. The decoders were extracted out of the provider methods (decodeChatCompletion, decodeOpenAIModelList, decodeGeminiGenerateContent, decodeGeminiModelsPage, decodeFalLanguageCatalog, the fal extract*Url family, kieEnvelopeError, decodeKieTaskSubmission, decodeKieRecordInfo, decodeKieResultUrls), so fixtures and live responses parse through the code a run uses. runProbes reports network failures apart from schema failures, and retains only the response shape (summarizeShape) plus redacted messages, so no credential, prompt, request id, or signed URL reaches an artifact. Nightly: .github/workflows/provider-contract-probe.yml; offline half: packages/runtime/tests/providers/provider-contract-probes.test.ts, wired into harness gate through the new provider-clients surface. Docs: docs/provider-contract-probes.md. Generated provider metadata drift: shipped 2026-08-22. Both generators have a fixture mode — packages/{fal,kie}-codegen/src/fixture-generate.ts — that reads only the schema fixtures under fixtures/, named by fixtures/generator-manifest.json, with no network, no pricing, and no timestamps. scripts/provider-codegen-check.mjs generates into a temporary directory and diffs every declared node-source and static-metadata output against fixtures/expected/, failing on a difference, an absent fixture, or a run that compared nothing. Root commands generate:fal:check and generate:kie:check and the workflow .github/workflows/provider-codegen.yml run it; fixture-generate.test.ts in each package covers the same comparison, byte stability, and the absent-fixture failures. Missing secrets in validate_workflow: shipped 2026-08-22. CapabilityRun.availableSecrets carries the host’s answer, contextSecretAvailability builds it from a context that can reach a store, and getAllMcpTools takes the factory as secretAvailability. The missing_secret issue names the key and Settings → Credentials, and adds request_secret only where the run can raise the dialog. Covered by packages/agents/tests/mcp-tools.test.ts and the call-site audit capability-run-secrets-audit.test.ts. Workflow credential preflight: shipped 2026-08-22 in node-sdk, execution, and CLI. The two execution credential suites cover it. ExecutionSession preflight: shipped 2026-08-22. The model and credential checks moved to packages/execution/src/preflight.ts (no models import), ExecutionSession.create selects its context and refuses through ExecutionPreflightError before the Python bridge and persistence.onAccepted, and runWorkflow refuses through the same contract. session-preflight.test.ts covers both directions; execution-session-hydration-audit.test.ts audits every host. Production Python opt-in: NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 is implemented in python-stdio-bridge.ts and documented in docs/configuration.md. tRPC POST batches: issue #3979 is closed. Both tRPC clients set methodOverride: "POST". Vite preload recovery: issue #4203 is closed. Dedicated tests cover stale deploys, non-stale failures, and the guard window. Stale Prompt value: issue #3786 is closed. PromptComposerBody commits edits synchronously; its commit and run-from-here suites cover the defect. Media resolution as the browser rendering boundary: shipped 2026-08-22. ResolvedMediaUrl in web/src/utils/resolveMediaUri.ts is the brand; ResponsiveImage, VideoPlayer, and the new AudioPlayback take a locator and resolve it. The rendering surfaces are inventoried in web/src/__tests__/mediaResolutionBoundary.test.ts, and design-tokens/no-unresolved-media-src (fixture web/scripts/test-media-src-rule.mjs) rejects a locator literal in a JSX url attribute. SSRF screening inventory and consolidation: shipped 2026-08-22. isBlockedIpLiteral in packages/runtime/src/providers/safe-url.ts is the one address table; the sandbox’s network-guard.ts imports it and the websocket runner’s private copy (isSafeExternalUrl) is gone. fetchExternalMedia (packages/runtime/src/external-media-fetch.ts) is the media-ref egress policy — safeFetch, with NODETOOL_ALLOW_PRIVATE_MEDIA_FETCH=1 as the self-hosted LAN opt-out — and the fifteen media and result-download sites that used a bare fetch now call it or safeFetch. The inventory is packages/runtime/tests/url-egress-inventory.ts (86 entries: guarded, fixed-host, private-integration, browser, sandbox-bridge, screening); the reader-facing doc, with the DNS-rebinding decision and the contribution checklist, is docs/url-egress-inventory.md; and url-egress-audit.test.ts discovers URL surfaces from source in both directions, so a new unclassified fetch(url) fails. Cross-origin media CSP: issue #5125 is fixed and covered by web/src/__tests__/contentSecurityPolicy.test.ts.--- # FAQ Source: https://docs.nodetool.ai/faq Each answer links to the page that covers the topic in full. Getting started Is NodeTool free? The app is free and open source under AGPL-3.0. With your own API keys you pay each cloud provider directly, and local models cost nothing after the download. The hosted Cloud also offers a curated model catalog paid with NodeTool credits. See Costs and Credits. What does a run cost? The cost depends on the models you pick. NodeTool estimates the cost of a workflow before you run it, in the Cost estimate section of the right panel, and records what each provider call cost afterward. See Costs and Credits. Do I need an API key? You need one model source before an agent or a generation node can run. That can be a cloud provider with a key, a provider you sign in to (a Claude subscription in the desktop app, a ChatGPT account, or Hugging Face), or a local runner such as Ollama. See Connect an AI provider. Which operating systems are supported? The desktop app ships for macOS (Apple Silicon and Intel), Windows (x64), and Linux (AppImage and Flatpak). See Desktop App. Where do I start? Install the app, connect a provider, then pick a type on the Home tab and describe what you want in one sentence. The Getting Started guide walks through a first project, and the Templates Gallery has examples you can copy. Models and hardware Which models can I use? NodeTool reaches cloud models from more than 30 providers, including OpenAI, Anthropic, Google Gemini, xAI, FAL, and Replicate, and it runs local models too. See Models and Providers for the catalog and Providers for setup. Can I use NodeTool offline? Yes, with local models. The desktop app runs its server on your machine, and local models run without a connection once they are downloaded. Cloud providers need internet. See Models and Providers. Do I need a GPU? No. Cloud providers need no graphics card. For local models, NVIDIA cards run the HuggingFace pack, llama.cpp, and Ollama, Apple Silicon Macs also run MLX, and a CPU works but slowly. See What different tasks need and GPU requirements. How big are local models? Expect 4 to 20 GB per model. Models, Python, and model runners download only when you install them. See What downloads later and Models Manager. Can I run Ollama or llama.cpp models? Yes. Install Ollama from ollama.com, pull a model with ollama pull <model>, and it appears in NodeTool while Ollama runs. In-process llama.cpp installs from Package Manager → Software. A llama-server you run yourself connects through LLAMA_CPP_URL. See Connect an AI provider. Data and privacy Where are my files stored? On the desktop, the database, assets, and vector store live in ~/.local/share/nodetool/ on macOS and Linux and in %APPDATA%\nodetool\ on Windows. See Where your data lives. Does my data leave my machine? Only when a node calls a cloud provider you configured, when you download something, or when you turn on an optional feature such as update checks. See Privacy and Data Handling. Are my API keys safe? Keys are stored encrypted with AES-256-GCM in the local database. The encryption key lives in your operating system keychain, or in SECRETS_MASTER_KEY if you set it. See Privacy and Data Handling. Studio, Cloud, and hosting What is the difference between Studio and Cloud? Studio is the desktop app, which can run local models and work offline. Cloud is the hosted browser version with the same canvas and nodes, and no local models. Both come from the same AGPL-3.0 source. See Studio or Cloud. Can I host NodeTool myself? Yes. The repository ships a docker-compose.yml that runs one server. It has no login by default, so keep it on a private network or enable Supabase authentication first. See Self-Hosted Deployment. Can other tools drive NodeTool? Yes. NodeTool has a REST and WebSocket API, a CLI, and an MCP server that gives agents such as Claude Code access to workflows and generation. See API Reference, CLI Reference, and MCP Server. Using NodeTool Can I import my ComfyUI workflows? You can run them, not convert them. Export the workflow from ComfyUI in API (prompt) format and load it into a Run ComfyUI Workflow node, which turns its load and save nodes into typed inputs and outputs. See ComfyUI. Can I turn a workflow into an app for other people? Yes. A mini app puts a form or interface in front of a workflow, with the workflow’s inputs and outputs as widgets. See Mini Apps and App Builder. Can I add my own nodes? Yes. Node packs add nodes to the node menu, and you can write your own in TypeScript. See Node Packs and the Custom Nodes Guide. What are the keyboard shortcuts? Press Ctrl + / or ⌘ + / in the app. The full list for every editor is in Keyboard Shortcuts. Licensing and updates What license is NodeTool under? AGPL-3.0. The full text is in the repository’s LICENSE.txt. The AGPL has conditions for modified versions offered over a network, so read it before you redistribute or host a changed copy. How do I update the desktop app? Turn on Automatic Updates in Settings and the app downloads releases in the background, then shows a Restart to Update button. You can also reinstall from the latest download. See Upgrading. Will an update delete my workflows? An update does not remove your data directory. The server applies database migrations at start. Back up the data directory before a major update. See Upgrading. Where can I see what changed in a release? Release notes are on GitHub, and the Changelog page summarizes recent months. Something broke. Where do I look? Check the logs in Tools → Log Viewer, then the Troubleshooting guide.--- # Your First Workflow Source: https://docs.nodetool.ai/first-workflow Quick Start runs a finished template. This page builds a workflow yourself, one node at a time. You start with an empty canvas and end with a picture generated from one short idea. The workflow has four nodes: Node Does String Input Holds your idea as text Agent Rewrites the idea as a detailed image prompt Text To Image Generates the picture Preview Shows the picture on the canvas Data flows left to right: String Input → Agent → Text To Image → Preview. Before you begin You need a language model for the Agent and an image model for Text To Image. Both come from providers you connect in Settings → Models & Providers, as described in Quick Start. Text To Image defaults to FLUX.1 Schnell on FAL, so a FAL key is the fastest way to get an image model. The capability matrix lists other providers. Note: The image costs money at the provider’s rate. The Agent call is text and costs cents. 1. Create an empty workflow Use + New → New workflow. A workflow tab opens with an empty canvas. You can also press Ctrl/⌘ + K and pick New Workflow from the Command Menu. 2. Add the String Input node Press Space on the canvas, or double-click empty space. The node menu opens. Type string input in Search for nodes…. Click String Input. The node appears on the canvas. Type your idea into the node’s Value field, for example: a lighthouse on a cliff at dusk Then give the node a name. Select it and set Name to idea in the Inspector on the right. An input node needs a name before the workflow can run, and the node shows Name required until it has one. The name becomes the node’s title and labels this value when the workflow runs as an app or over the API. 3. Add the Agent node Press Space, search for agent, and click Agent. Drag it to the right of String Input. Select the node. The Inspector opens on the right with three tabs: Params, I/O, and Help. On the Params tab: Press Select Model and choose a language model. Replace System (it starts as “You are a friendly assistant”) with instructions for this job, for example: Rewrite the user's idea as one detailed image prompt. Describe subject, light, and style. Reply with the prompt only. Tip: Leave Prompt alone. You fill it by connecting a node in the next step. 4. Connect String Input to Agent Drag from the output circle on the right of String Input. Release on the Prompt input circle on the left of Agent. A line appears. Connections only accept matching types, so text goes to text. Each input takes one connection. The I/O tab in the Inspector lists the node’s inputs and outputs. 5. Add the Text To Image node Press Space, search for text to image, and click Text To Image. Place it to the right of Agent. In the Inspector’s Params tab you can set: Property Default Use Model FLUX.1 Schnell (FAL) The image model Prompt A cat holding a sign that says hello world Replaced by the connection below Negative Prompt empty What to avoid in the image Aspect Ratio 1:1 Shape of the image Resolution 1K Short edge of the image Connect Agent’s text output to Text To Image’s Prompt input. The Agent’s written prompt now replaces the default prompt. 6. Add a Preview node You can add Preview from the node menu, but a shortcut is faster: Drag from Text To Image’s output circle and release on empty space. In the connection menu that appears, pick Preview. NodeTool adds a Preview node already connected to the image. Preview has one input, Value, and shows whatever reaches it. Text To Image also shows its own result inside its node, and Preview gives you a separate, larger place to look at it. You can send the Agent’s text output to a second Preview node the same way. Then you see the written prompt next to the picture. 7. Save and run Press Ctrl/⌘ + S to save. Then click Run entire workflow in the bottom toolbar, or press Ctrl/⌘ + Enter. If a model is not selected, NodeTool stops the run and marks the unset model field in the Inspector. Pick a model and run again. While it runs: A moving colored ring marks the node that is working. The Agent writes its text as it arrives. The run control reads Starting, Queued, or Running. Completed in badges show how long each node took. When the run finishes, the picture appears in Text To Image and in Preview. Generated images are saved as assets automatically. If a node fails, it gets a red Failed in badge and shows the error inside the node. View logs opens that run’s logs in the bottom panel, where Queue shows the jobs. See Workflow Debugging and Troubleshooting. 8. Change something and run again Edit the Value in String Input and press Run entire workflow again. Only the idea changed, so the Agent writes a new prompt and the image updates. Other changes to try: Pick a different Aspect Ratio on Text To Image. Fill Negative Prompt to exclude things from the image. Change the Agent’s System text to ask for a specific art style. Select a node and press B to disable it. A disabled node and its connections are left out of the run. Under the Inspector, Cost estimate prices the workflow for one run. What you just did Nodes. You placed four single-purpose boxes and set their properties in the Inspector. See Key Concepts. Connections. You wired outputs to inputs, and NodeTool accepted only matching types. See Data Types and Connections. Agent node. You used an Agent to rewrite a short idea into a detailed prompt. See Key Concepts. Run and preview. You ran the graph, watched each node report progress, and inspected the result. See Workflow Editor. Assets. The generated image was saved to your library automatically. See Asset Management. Troubleshooting A node failed or the run will not start? See Troubleshooting or ask on Discord. Next steps Workflow EditorSelecting, grouping, shortcuts, and the rest of the canvas. Templates GalleryStart from a shipped workflow instead of an empty canvas. Mini AppsTurn this workflow into a screen others can run. Models & ProvidersChoose models or run them locally.--- # Production deploy (Fly.io) Source: https://docs.nodetool.ai/fly-production-deploy.html Production deploy (Fly.io) The active release destination is the Docker host. Follow Production Deploy (Docker) for GitHub releases. This guide covers manual operation of the retained Fly deployment. Self-hosting uses Self-Hosted Deployment. The app is nodetool on Fly.io, configured by fly.toml at the repo root. Its machines run in fra, next to the Supabase database, and the deploy replaces them one at a time so the others keep serving. That needs at least two machines (fly scale count 2); with one, the drain is an outage window rather than a handover. What triggers a deploy GitHub no longer releases to Fly. The historical fly-deploy.yml filename runs Deploy to Docker and Pages follows that workflow. A manual Fly release uses scripts/fly-rolling-deploy.sh with a previously built commit image or immutable digest. Review database compatibility and trigger dispatch before using Fly as the rollback destination. Do not enable a second automatic release path. What the rollout does scripts/fly-rolling-deploy.sh takes the image ref and does the whole release. In order: Migrate. flyctl machine run --rm --restart no node /app/backend/db-migrate.mjs on the new image. fly.toml’s release_command covers this for a plain fly deploy, but machine update has no release phase, so the script runs it. A failed migration aborts before any machine serves the new code. List the app machines. fly_process_group == "app", matched exactly. A machine without that tag is a one-off like the migration machine. For each machine, in turn: drain it, update it to the new image, wait for /health to answer status: "ok", then move to the next. Draining is the point of the script. Fly’s own restart sends SIGTERM and waits at most 300 s, and a chat turn can run for half an hour. So each machine is first sent SIGUSR2, which starts the drain described in websocket-api.md § Draining: /health answers 503 so the proxy stops routing new clients here, idle sockets close with 1012, new chat_message and run_job are refused, and turns already in flight finish. Only when /health reports turns and jobs both 0 is the machine replaced. A healthy rollout looks like this, with no warnings: ==> [<id>] draining [<id>] draining: turns=0 jobs=0 ==> [<id>] updating to ghcr.io/nodetool-ai/nodetool:main-<sha> ==> [<id>] waiting for health ==> [<id>] healthy on ghcr.io/nodetool-ai/nodetool:main-<sha> Sending the signal The image ships no procps and the server is not PID 1, so there is no pkill. The script scans /proc in pure shell over flyctl ssh console -C, skipping its own pid — the scan’s own command line contains server.mjs, so without that it signals itself. Bootstrapping a machine that cannot drain SIGUSR2’s default disposition is terminate. An image built before the drain handler landed has no handler, so the signal kills the server instead of draining it, and the machine exits 140 (128 + 12). The script therefore sends SIGUSR2 once and gives the machine DRAIN_START_TIMEOUT_SECONDS (30) to acknowledge it in /health. If it does not, that is an image without the handler: the machine is updated with no drain, which is the only way to reach an image that can drain at all. Re-signalling instead kills every restart until Fly stops restarting the machine. This path is a one-time transition and it cuts in-flight turns. It is expected only when the running image predates the handler. Machine-state facts the script depends on These are not obvious and each one caused a failed deploy: flyctl machine update preserves a stopped machine’s state. It rewrites the config and returns without booting anything — no start event at all. A machine updated out of stopped must be started explicitly with flyctl machine start. flyctl machine run returns when the machine starts, not when its command exits, and its exit code says nothing about the command’s. Read the command’s exit code from the Machines API after waiting for the machine to be destroyed. flyctl machine status has no --json flag. Machine state and exit codes come from https://api.machines.dev/v1/apps/<app>/machines/<id> with Authorization: Bearer $FLY_API_TOKEN. The events array is newest-first. fly.toml sets auto_stop_machines = "off". Nothing stops an app machine on purpose, so a stopped one is always a machine to repair, whatever its exit code — including no exit event at all, which reads back as unknown. machine update can collide with an update Fly is already applying. An overlapping run, a retried deploy, or Fly replacing a machine on its own makes the API answer aborted: machine is replacing: concurrent update in progress and flyctl exit 1. That conflict clears on its own, so the update is retried UPDATE_ATTEMPTS times with a doubling delay; any other update failure still aborts the rollout on the first attempt. A machine mid-boot is not a machine that failed. machine update returns before the new process is up, so the post-update state check waits READY_STATE_GRACE_SECONDS (60) before a non-running state counts as a failure. Tuning Every timeout is an environment variable with a default, so a deploy can be re-run with a different budget without editing the script: Variable Default What it bounds MIGRATE_TIMEOUT_SECONDS 600 the migration machine reaching destroyed DRAIN_START_TIMEOUT_SECONDS 30 acknowledging SIGUSR2 before the bootstrap path DRAIN_TIMEOUT_SECONDS 2700 turns and jobs reaching zero READY_STATE_GRACE_SECONDS 60 booting before a non-running state is a failure READY_TIMEOUT_SECONDS 300 /health answering 200 after the update UPDATE_ATTEMPTS 5 attempts at an update that conflicts with another UPDATE_RETRY_SECONDS 10 the first delay between those attempts, doubling FLY_APP (nodetool) and FLY_REGION (fra) are overridable the same way. When a deploy fails Read the machines first. The rollout stops on the machine it was working on, so the app is usually still serving from the others. flyctl machines list -a nodetool --json | jq -r '.[] | "\(.id) \(.state) \(.config.image)"' # state and the exit codes behind it, newest first. # `flyctl auth token` still works as a fallback but is deprecated; mint one # with `fly tokens create deploy -a nodetool`. curl -sf -H "Authorization: Bearer $FLY_API_TOKEN" \ https://api.machines.dev/v1/apps/nodetool/machines/<id> | jq -r '.state, (.events[] | "\(.type) \(.status) exit=\(.request.exit_event.exit_code // "-")")' A machine is stopped. auto_start_machines = true means the proxy may boot it on the next request. To restore capacity now: flyctl machine start <id> -a nodetool. Re-running the deploy also repairs it. Exit code 140. SIGUSR2 with no handler. The running image predates the drain handler; the bootstrap path above is what gets past it. The rollout aborted but the machines are fine. Re-run the workflow. The script is idempotent: a machine already on the target image drains, updates to the same image, and reports healthy. For a manual Fly rollback, run bash scripts/fly-rolling-deploy.sh <image> with a reviewed, previously built image. GitHub’s production workflow now targets Docker and must not be used to operate Fly. Testing the script scripts/__tests__/fly-rolling-deploy.test.sh drives the whole script against stubbed flyctl and curl on PATH. It runs no real Fly command and touches no network, and covers the paths that are painful to reach for real: a legacy SIGUSR2 exit, an image that never acknowledges the drain, a machine left stopped by an earlier attempt, a machine stopped with no exit event, and a terminal machine that must abort the rollout. npm run test:scripts # via the Vitest wrapper bash scripts/__tests__/fly-rolling-deploy.test.sh # directly, prints each assertion The assertion that matters most is that a bootstrapped machine ends up started, not merely updated. A test that checks only “was it updated” passes against a machine that never came back. When you change the script, prove the new test can fail: point DEPLOY_SCRIPT at the previous version and watch it go red. git show HEAD:scripts/fly-rolling-deploy.sh > /tmp/old-deploy.sh DEPLOY_SCRIPT=/tmp/old-deploy.sh bash scripts/__tests__/fly-rolling-deploy.test.sh Related websocket-api.md § Draining — the drain protocol the script drives Self-Hosted Deployment — running your own server, outside Fly Deployment Guide — server vs. worker, and the self-host paths--- # Game Draft Storage Compatibility Source: https://docs.nodetool.ai/game-draft-storage.html Game draft storage compatibility Before rolling back the server, check the persisted draft format as well as the database schema. The game document’s schemaVersion and engineVersion do not describe draft storage compatibility. Version identifiers The game model stores immutable draft files at <source_root>/drafts/<version_id>.json. The games.draft_version_id column points to the current draft. Each game_draft_changes.before_digest value identifies the before-file used to undo that change. Legacy identifiers are the 64-character SHA-256 digest of the exact file bytes. The publication-safety implementation introduced by PR #6180 writes identifiers with three parts: <sha256>.<baseUpdatedAt encoded as base64url>.<unique version ID> The reader accepts both formats and verifies the file against the digest part. The captured base token and unique suffix let cleanup distinguish concurrent write attempts, even when their content is identical. Server rollback The reader before PR #6180 compares the file hash against the entire stored identifier. It rejects a dotted identifier with Game draft source is corrupt. A database schema check alone will not detect this incompatibility. Keep the compatible draft reader and cleanup behavior when rolling back other server changes. If the target cannot retain them, require a reviewed offline conversion before starting that server: A1: Stop every game writer and cleanup process. Back up the game database rows, change rows and workspace draft files together. A2: Verify the exact bytes of every referenced current draft and before-file against its digest. Stop if a file is missing or corrupt. Replacing it with a published revision would discard unpublished work. A3: Copy verified files to the legacy digest filenames and update both current draft pointers and history before-file references consistently. Keep the original files and backup available for reversing the conversion. A4: Read the converted current drafts and undo sources with the target reader before restarting writers. Compare their content with the backup. Do not strip suffixes from database values without creating and verifying the corresponding files. Do not run old and new writers together during conversion. Legacy content-addressed paths lack the write-attempt identity used by the newer cleanup rules. Legacy file retention Normal orphan cleanup preserves legacy digest-only files because they carry no captured base token. A separate maintenance sweep must exclude old writers and preserve every file referenced by a current draft or retained change. File age alone does not prove that a legacy file is safe to delete. See production rollback for the deployment procedure.--- # Game Editor Source: https://docs.nodetool.ai/game-editor.html Build a game in a workspace tab, play it without leaving the editor, and ask the agent to change it. Quick access: Click + New in the workspace tab bar and choose Game under Guided flows. The new game opens as a tab and is playable at once. Overview The Game Editor edits one game document. The built-in engine simulates it, renders it, and exports it as a standalone web player. A game is either 2D or 3D, and it keeps that dimension for its whole life. A game document holds: Scenes. Each game has an entry scene, and scenes can switch to each other during play. Entities. Each scene lists entities. An entity has a transform and optional components such as a sprite, body, collider, camera, or light. Behaviors. Built-in behaviors such as movement, patrol, and collectible, plus JavaScript scripts. Assets. Named slots bound to images, audio, fonts, and in 3D, models. Input actions. The names scripts and behaviors read, such as left and jump. You edit the game as a draft. The draft saves automatically about half a second after each change. Publish turns the draft into an immutable revision.   2D 3D Space Top-down or side-view, in world units Meters, Y up, forward is -Z Starter A top-down room with a player, a gem, and four walls An exploration level with a capsule player, ramp, moving platform, crate, door, checkpoint, and goal Viewport Canvas with selection, move, scale, and rotate handles Orbit camera with a Move, Rotate, and Scale gizmo Rendering Canvas 2D, or WebGPU when the game uses GPU effects WebGL2 Physics 2D bodies and colliders, tilemaps, one-way platforms 3D bodies, colliders, a character controller, a follow camera Start a game From a guided flow + New in the tab bar lists the guided flows. Game creates a 2D game named “Untitled game” in the selected project and opens it. The 3D choice is not in this menu. From the new-project screen The new-project screen has a Start with a guided flow row. The Game card appears after you choose More formats. Beside it, Game card starts as picks 2D game or 3D exploration. The text in the prompt box names the game. It is not sent to the agent. If the box is empty the game is called “New game”. From an example Open the Examples page from the logo menu and choose the Games tab. Each card shows a poster, a description, and the controls. Play and edit copies the game and its media into your current project and opens it. Example Dimension Summary Kindle 2D A platformer through a drowned temple Lumen 2D Fly a firefly through a night forest Neon Drift 2D An arena shooter with waves AETHER // Skybound 3D Floating platforms and lasers BLACKSITE 3D A first-person mission with drones The server reads example games from *.game.json files. Set NODETOOL_EXAMPLE_GAMES_DIR to point it at another folder. A folder that does not exist yields no examples. See Configuration. From the agent Ask the agent in chat to create a game. It calls create_native_game with a project, a name, and a dimension. For 3D it can pass template: "exploration". See Ask the agent. There is no blank-game entry. Every new game starts from a playable starter, so you delete or replace what you do not need. Editor layout The editor is a toolbar over three docks and a viewport. Drag the edge of a dock to resize it. Area What it does Toolbar Play controls, Publish, panel toggles, save status, and a live readout of tick, score, win state, and renderer Scene tree (left) The scenes and entities of the game Viewport (center) The game canvas, where you select and move entities Inspector (right) Properties of the selected entity, scene, or game Assistant (right) The agent chat for this game Script pane (bottom) A code editor, open while you edit a script The toolbar buttons are Play, Play in new tab, Stop, Step, Save play state, Load play state, Publish, a scene tree toggle, an inspector toggle, and Show assistant. The status text reads Draft saved, Saving draft, Unsaved changes, or Draft not saved. Under the toolbar, the editor can also show: A Changes list of edits the agent made, with View in chat and Undo for each group. A Retained construction section when a game was generated from a construction program. See Rebuild a generated game. A conflict banner when the draft changed somewhere else, for example from the agent or another tab. Accept or discard each conflicting unit. On a narrow screen the 2D editor stacks the scene tree and inspector under the viewport and opens the assistant as a bottom sheet. The 3D editor keeps its docks. Edit scenes and objects 2D Scene tree. Each scene lists its entities in a group named Entities. A second group named Prefabs appears when the scene has entities marked prefab-only, which scripts and the spawn behavior instantiate at run time. The search box filters by entity name or ID. Select an entity with a click. Shift-click or Ctrl-click or Cmd-click adds to the selection. Drag an entity onto another to nest it, or between rows to reorder it. Alt+arrow keys do the same from the keyboard. The + beside a scene adds an entity: Sprite, Static wall, Collectible, Trigger, Camera, or Empty. Sprite is disabled until the game has an image asset. Viewport. Click a sprite to select it, or drag on empty space to box-select. Drag a selected entity to move it. Handles on a selected sprite scale and rotate it, and Shift constrains the change. Alt-click cycles through overlapping sprites. Action Control Pan Right or middle drag, hold Space and drag, or turn on Pan Zoom Mouse wheel. On a Mac, pinch or Ctrl+scroll Reset camera Home Frame the selection F Nudge Arrow keys move 0.25 units, Shift+arrow moves 2.5 Copy, paste, duplicate Ctrl or Cmd with C, V, D Delete Delete or Backspace Undo, redo Ctrl or Cmd with Z, and with Shift for redo Keyboard shortcuts work while the viewport has focus. The overlay buttons Selection, Grid, Snap, Colliders, Lights, Backgrounds, and Camera turn the matching overlay on or off. G toggles snap. Inspector. It has two tabs. Entity (or Scene when nothing is selected) shows the selection. For an entity it lists name, parent, Prefab only, Transform, and each component present. Add component offers Sprite, Tilemap, Camera 2D, Body 2D, Collider 2D, Animator, Visual animation, Audio source, and Light 2D. Add behavior offers Movement, Patrol, Collectible, Health, Trigger, Spawn, Scene transition, Win when collected, Lifetime, and Script. A scene tab edits its name, music, lighting, and backgrounds. Game edits pixels per unit, the entry scene, input actions, collision layer names, render effects (bloom, LUT, brightness and contrast), and asset bindings. Fields that fail validation show their error beside the field. The scene tree marks an entity with a validation error or a script error in red. Revisions. The scene tree dock has a collapsed Revisions section. It lists the 10 most recent revisions. Restore to draft loads one into the draft without changing the published history. If the saved draft source is unavailable, the editor offers published revisions for recovery. When this tab still has a local draft, Export local draft saves its document, pending edits, and undo history as JSON before restoring is enabled. Keep the tab open until the download finishes. Confirming Restore to draft replaces the local draft and clears its undo history. 3D Scene tree. A Scene selector switches scenes. The list below it is flat. A child shows an arrow, and a character or camera shows a tag. Add box, Add sphere, and Add light create a primitive with a static body and matching collider, or a point light. Model assets. The collapsed Model assets section binds a 3D model. Enter an owned model asset ID and a slot name, then choose Prepare and install model. Each model slot has Edit model, which opens the model in a model editor tab. After you save changes to the model, prepare and install it again. The engine does not pick up edits on its own. Viewport. The toolbar above the canvas has a Transform mode selector (Move, Rotate, Scale), Local space, Snap, Gizmo settings, Align to ground, Overlays, Fly camera, and Frame selection. The orientation gizmo in the top-right corner shows the world axes. Click an axis end to look along that axis, and use the button below it to switch between perspective and orthographic views. Click an entity to select it. Shift-click, Ctrl-click or Cmd-click adds it to the selection or removes it, and a click on empty space clears the selection. The scene tree uses the same modifiers. A multi-selection shares one gizmo and outlines every selected entity. A child whose parent is also selected moves with the parent. Each gizmo drag and each Align to ground is one undo step for the whole selection. Gizmo settings holds the pivot, the snap increments and two switches. The pivot is Selection centre (the mean of the selected origins), Active entity (the last one selected) or Individual origins, where each entity rotates and scales around its own origin. Snap increments default to 0.25 meters for move, 15 degrees for rotate and 0.25 for scale. Surface snap makes move drags rest the selection on the surface under the pointer. Orbit around selection moves the orbit centre to the selection whenever it changes. Align to ground drops each selected entity onto the surface below it, or onto the ground plane at height zero. These settings are editor preferences kept in this browser, not part of the game. Action Control Orbit Drag Pan Shift-drag Zoom Scroll Fly camera Turn on Fly camera, then W, A, S, D to move, R and F to rise and descend, drag to look Frame the selection F Switch local and world space X Next pivot mode . (period) Select all, clear the selection Ctrl or Cmd with A, Esc Align to ground End Front, right, top view Numpad 1, 3, 7. Add Ctrl or Cmd for back, left and bottom Orthographic view Numpad 5 The 3D editor has Undo and Redo buttons, and Ctrl or Cmd with Z works inside the dock area. Inspector. With nothing selected it shows scene settings and any validation errors. With an entity selected it shows Position (meters), Rotation (degrees), Scale, and the entity’s components, such as primitive, body, collider, camera, light, and character. A Behaviors section lists behaviors, with Add script, Edit script, and Remove. Delete entity removes the entity. What differs The 3D editor has no copy and paste, no keyboard nudge, no scene tree search or nesting by drag, no prefab group, no Revisions list, and no Run 10 s check. Edit those from the agent or the document operations. Script entities A script is a behavior on an entity. In the inspector choose Add behavior, then Script in 2D, or Add script in 3D. Choose Edit script to open the script pane. The pane is a code editor with type hints for the engine’s script API, and each edit goes into the draft. A script is a function expression that receives the current tick, input, events, and its own entity, and returns { state, commands }. /** @type {GameScript} */ ({ state }) => ({ state, commands: [] }) Rules that apply in both dimensions: Keep values you need later in returned state. The engine reuses a sandbox only for expressions proven to read their input without retaining hidden state. Other sources run in a fresh context on every call. Input mutations never reach another call. Commands change the world. Shared commands include emit, hud, playAnimation, despawn, spawn, and sceneTransition. Physics ray and shape query results arrive on the next tick. The global world API reads the current tick’s starting state immediately. Events a script receives come from the previous tick. The engine runs at a fixed 60 ticks per second and seeds random, so a replay gives the same result.   2D 3D Position input entity.x, entity.y, touching sides XYZ entity.position, velocity, grounded, and axes and look input Movement commands setVelocity, setPosition, setVisual characterIntent, setKinematicPose, impulse, setVelocity, teleport, setVisual Spatial queries world.get and world.query world.get and world.query, plus ray and shape commands with a queryId Camera The camera entity Perspective or orthographic, fixed or follow Read the world during a script call world.get(id) returns an entity record, or undefined if the ID is absent. world.query({ source?, tag?, near?, radius?, limit? }) returns matching IDs in runtime entity order. Both read one immutable tick-start snapshot of active entities, before movement behaviors or script commands run. Returned values are private copies. They do not expose changes from earlier scripts in the same tick. In 2D a record contains id, source, x, y, velocityX, velocityY, and grounded. In 3D it contains id, source, position, velocity, and grounded. Positions and velocities use { x, y, z } in 3D. Pass near: { x, y, z? } together with a nonnegative radius. Distance uses entity centers and includes the radius boundary. Omitted z is zero. source matches the exact source field, the original entity ID rather than a 3D prefab asset ID. Documents currently have no authored tags, so every tag filter returns an empty array. /** @type {GameScript} */ (input) => ({ state: world.query({ near: { x: input.entity.x, y: input.entity.y }, radius: 120, limit: 8 }), commands: [] }) limit defaults to 1024 and accepts integers from 0 to 1024. Each call allows 64 combined get and query attempts. Exceeding that count fails the call even if the script catches the exception. Query arguments are limited to 4096 JSON characters, get IDs to 1024 characters, and each response to 64 KiB. The existing script time, memory, input, and output budgets still apply. Queries are available during the function call, not while its source initializes. The legacy input.world array keeps its original shape and population. In 2D it includes active entities with a collider or camera, whereas global world queries include every active entity. Its mutable copy remains private to the call. This API is additive. The 3D input retains contractVersion: 3, and documents need no schema version change or migration. The pane footer shows the source length against the 16,384 character limit, the maxCommands limit, and the maxTickMs limit. In 2D you can change both limits from the script behavior in the inspector. The defaults are 16 commands and 8 ms. The limits are 1 to 64 commands and 1 to 50 ms. On Linux and in the browser, maxTickMs counts wall time. Linux thread CPU accounting can lag behind a short call, so wall time keeps the supported 1 ms limit enforceable. Waiting for a processor or a blocking host callback counts against this limit. Other server and CLI hosts use thread CPU time when available, with wall time as the fallback. When a script throws during play, the 2D pane shows the tick and the message. Replay to tick N reruns the game to the tick before the failure so you can inspect it. Ask the assistant drafts a message with the scene, entity, behavior index, tick, and error. 2D also has Run 10 s, which runs the game headlessly for ten seconds of ticks and reports script calls, total script time, and a per-entity breakdown. The 3D editor shows Replay before error in the error line. Agents and JS scripts can build a whole document in code with the @nodetool-ai/sandbox-game pack. Ask the agent Show assistant opens a chat docked beside the viewport. The agent knows which game is open, which entities you selected, and which script you are editing. Welcome examples are “Add a boss that appears at wave 5”, “Make the forest darker and add fog”, and “Why does the player pass through this wall?”. The agent follows the native-game skill and edits the draft through tools: Tool Use get_native_game Read an outline, one entity, or the whole document edit_native_game Apply ordered operations. The whole edit is validated before it is written capture_native_game_frame Render frames so it can look at the result playtest_native_game Run recorded inputs and check assertions autoplay_native_game Steer a 2D player toward a target or win. 3D is not supported, so give a route instead generate_game_asset Generate or import an image, speech, music, sound effect, font, or 3D model, and bind it to a slot install_native_game_asset Install a staged candidate asset publish_native_game Publish, only when you ask build_native_game Write a standalone web build or a Tauri desktop app project list_example_games, get_example_game, install_example_game Study or copy a shipped game For a new game or an art overhaul the agent also loads game-direction. It fixes a player promise, a style string, a layer plan, feel parameters, and a pacing sheet first, then keeps asset prompts consistent with that spec. Agent edits land in your draft. The Changes list groups them by chat message. View in chat jumps to the message, and Undo reverts that group. Your later edits are merged, and overlapping ones show as conflicts. Play Button What it does Play Starts a play session from the current draft. While playing the button reads Pause Pause Stops the clock. The viewport shows the paused frame, and the Runtime state panel shows the selected entity Step Advances one tick. Available only while paused Stop Ends the session and returns to editing Save play state Stores a snapshot of the running session in this browser Load play state Restores that snapshot Play in new tab Opens the player page for this game Play uses the keyboard. Click the canvas to give it focus. In 2D, arrow keys and W, A, S, D map to left, right, up, and down, Space maps to space, and other keys map to their lowercase key name. A key counts only if the game lists it under input actions. In 3D, W, A, S, D or the arrow keys move, Space jumps, and the right mouse button looks around. A session plays the document as it was at the moment you pressed Play. If you edit during play, a note says The draft changed. Stop and play again to apply it. In 2D, changing only asset bindings does not require a restart. Saved play state is browser storage keyed by game ID. It does not travel with the game, and another browser does not see it. If the game uses a required GPU effect and the browser has no WebGPU, play fails with an error. An optional effect is dropped and the toolbar reports Canvas 2D. Publish Choose Publish. The button is disabled while the draft is saving. The 2D dialog lists the changes since the last revision. On Publish it validates the game first and shows any errors instead of publishing. The 3D dialog says it creates an immutable revision from the current draft. Revision message is optional and holds up to 500 characters. Publishing writes a new revision and makes it the game’s current revision. The draft stays editable and starts from the published content. If another session published first, you get Game was modified concurrently. Reload, merge your edits, and publish again. Publishing does not make the game public. The game belongs to your account and its project. The player URL Play in new tab opens /game/<game ID>. That route is protected and loads the game through your own account, so only the owner can open it. It loads the saved draft, not the last published revision, and shows the game name, Play, Stop, a score, and a status. The button is disabled until the draft is saved. A 3D game loads the 3D player in the same route. To give someone a game that runs without NodeTool, publish and then build a standalone web player or a desktop app. Build a standalone web build A standalone build is a folder of static files for one revision. It plays without a NodeTool server. Use one of these: The editor. Choose Export build in the toolbar, then Web player. The download is a zip of the last published revision, not the draft. The agent. Ask it to build the game. It calls build_native_game for an owned revision and writes the folder under the project workspace, then returns the path. The CLI. nodetool game build game.json --out game-build builds from a game file. See nodetool game for flags, and for validate, simulate, and capture, which check a game without the editor. The folder holds index.html, game.json, a player script, and an assets/ folder of media named by digest. A 2D build adds a stylesheet and a script runtime. A 3D build adds manifest.json, the models, colliders, and pinned runtime files. Serve a 3D build over static HTTP. Installed media must still exist, match its recorded digest, and use a supported format, or the build fails. On a touch screen the 2D player adds a floating stick for left, right, up, and down, and one button per other input action. The first 3D release targets desktop web and Electron. Build a desktop app A desktop build wraps the standalone web player in a Tauri project. It compiles to a native app for Windows, macOS, and Linux that plays the published revision offline. The editor. Choose Export build, then Desktop app. The zip holds a desktop/ folder. To compile it right away, see Build the executable in the editor. The agent. Ask it for a desktop build. It calls build_native_game with target: "tauri". The CLI. nodetool game build game.json --out game-desktop --target tauri --name "Cave Run". Add --compile to build the executable right away. Build the executable in the editor The desktop app and a local server can compile the executable without leaving the editor. Choose Export build, then Desktop executable. Cargo runs on this computer and its output streams into the Console panel, warnings and errors included. When it finishes, the status row shows the executable’s path, and the desktop app offers Show in folder. The executable lands in bin/ inside the build’s project folder under the project workspace. Cancel stops cargo. A hosted server refuses the build, because it would run a compiler on the server. Cargo comes from the Rust package in the Package Manager, or from a rustup install. If neither is found, the editor offers to install Rust. The first build compiles Tauri itself and takes several minutes. Later builds share that work and finish much faster. The compiler also needs the platform toolchain: the Visual Studio C++ Build Tools on Windows, the Xcode Command Line Tools on macOS, and the WebKitGTK development packages on Linux. The project holds dist/ (the web player), src-tauri/ (the native shell, icons, and tauri.conf.json), package.json, and a README. The app takes the game’s name, and its identifier defaults to ai.nodetool.game.<name>. To make installers, install Rust and the Tauri prerequisites for your platform, then run npm install && npm run build in the project. Installers land in src-tauri/target/release/bundle/. Each platform builds its own installers, so build the Windows app on Windows and the macOS app on a Mac. cargo build --release in src-tauri/ makes the executable alone. WebGPU effects depend on the system web view. Where it has no WebGPU, as in WebKitGTK on Linux, the 2D player skips them and says so in its status line. Generate assets The game documents bind art and sound by slot name, so you can replace placeholders without touching entities. Ask the agent to generate an asset for a slot. It generates, prepares, installs, and binds it. Image preparation can trim alpha, resize, set a pivot, slice a sprite sheet into aligned frames, build a 16-variant terrain tileset, or make a color lookup table for a LUT effect. In a workflow, three nodes support the same job: Load Game Template, Slot Prompt, and Stage Game Assets. Stage Game Assets validates generated media and stages candidate bindings. Staged media is a candidate until installed. 2D fonts, music, lighting, backgrounds, effects, and visual animation need a game at schema version 2. The editor disables the matching Add buttons for a version 1 game. Rebuild a generated game A game made from a construction program keeps that program in the document. The Retained construction section shows its source and inputs. Use Preview rebuild to see changed entities and dependencies, Highlight changes to select them in the viewport, and Apply rebuild to write them into the draft. A rebuild is blocked while it has conflicts or the draft is stale. If your local edits conflict with the new construction, a dialog lets you keep an entity detached or discard those edits. Limits A game is 2D or 3D. It cannot change dimension, and mixed 2D and 3D scenes are not supported. Multiplayer, terrain streaming, navigation meshes, vehicles, ragdolls, retargeting, and root motion are outside this release. A script is at most 16,384 characters and runs under its command and time limits. A failed tick stops the session. Reset it or load a state from before the failure. Per game, the schema allows up to 8 render effects, 32 collision layer names, and 32 lights and backgrounds per scene. A sprite has up to 5 visual animation tracks. The runtime limits events and spawned instances. Exceeding a limit stops the session. A 3D game needs WebGL2. Playtest captures in 3D need Chromium. 3D triangle-mesh colliders require static bodies. Physics hierarchies reject scale and competing pose owners. Saved play state lives in one browser. The player page is not a public link. Share a standalone build instead. Related Sketch Editor for the images you bind as sprites and backgrounds 3D Editor for the glTF models a 3D game binds Asset Management for the media in your project Workspaces for where game files and builds are stored CLI Reference for nodetool game validate, simulate, capture, and build Configuration for NODETOOL_EXAMPLE_GAMES_DIR--- # Retained game construction Source: https://docs.nodetool.ai/game-retained-authoring.html Retained game construction Native 2D and 3D drafts can retain the construction that built them. Preparation produces accepted resources. A bounded construction program uses literal inputs and pinned asset bindings to produce a native document. Play executes that document with separate session state. Construct and save Read the game builder pack. constructGame({inputs, seed}, (inputs, builder) => { return document; }) returns a bundle for saveGame. The callback has explicit inputs and cannot close over outer variables. The builder provides typed parameters, prefab definitions and instances with explicit keys. Names and construction order do not supply IDs. Retained metadata stores the program, inputs, authoring seed, generated baseline, parameter schemas and prefab relationships. Native documents keep their existing 2D or 3D schema version. Documents without retained metadata remain supported. Review and apply a rebuild nodetool.games.previewAuthoring(id, {program}) returns a candidate with changed entities, dependencies, conflicts and a restart requirement. Omit program to rebuild the saved construction. Preview writes no draft or publication. nodetool.games.applyAuthoring(id, candidate) rebakes and compares the result to the preview. It saves only if the draft timestamp and digest still match and all conflicts are resolved. Publishing remains a separate operation. The Game editor exposes construction source, inputs and typed parameter controls. Preview highlights affected entities. Apply is disabled for stale candidates or unresolved conflicts. Rebuilds reject external capabilities, including network, workspace, media storage, secrets and host modules. Use builder.random() for seeded authoring randomness. Continue editing Moving a generated object records property overrides. A later scale change can propagate while its overridden position remains intact. Deleting a generated object records suppression, which survives later rebuilds. The inspector exposes inherited, generated and overridden fields, prefab identity, reset and detach. Suppressed objects can be restored from the construction panel. Removing a referenced or overridden generated object, colliding with a manual object’s ID, or making competing changes to scene settings produces a conflict. Reset the manual override, detach the object, or adjust construction to resolve it. Behavior arrays are atomic override values. Rebuilding does not migrate live behavior state. Atomic edits can preserve captured manual ownership alongside field operations. set_override_membership targets one scene, entity and property path, with an override payload or null to remove membership. set_authoring_membership targets suppression or detachment, with present and optional occurrence positions. These operations preserve retained construction definitions and reject ownership that disagrees with the final fields. Positions describe the array after that operation and can shift when later edits insert or remove entries. Active retained-game sessions keep their accepted definition and asset bindings. Restart play to adopt a rebuilt draft. Runtime spawn, collection and death state never becomes authored content. Verification The delivery fixture moves a collectible, suppresses another, changes player and prefab parameters, then rebuilds twice and verifies a fixed-input win with snapshot restoration and captured frames. Use the existing game replay harness for additional routes.--- # Quick Start Source: https://docs.nodetool.ai/getting-started.html One sentence in, a finished video out. About 30 minutes, mostly render time. No account. You bring your own API keys. Tip: The plan is text and costs cents. The clips cost dollars. You check the plan before you pay for clips. 1. Install and connect Download NodeTool from nodetool.ai, install it, and open it. You do not need a graphics card, because the models run in the cloud. OS details: Installation. Connect your models. NodeTool uses AI from other companies. An API key is the password that lets NodeTool use your account with them. They bill you for what you use. Open Settings → Models & Providers, then sign in or paste a key. Connect a provider for each row you need: You need For Get it from Language model Plans the video OpenAI, Anthropic, Gemini, Groq, Mistral, or a Claude subscription Video model Renders the clips FAL, Replicate, Gemini, OpenAI, kie.ai, and others Voice model (optional) Reads the voiceover OpenAI, ElevenLabs, Gemini, and others Tip: The fastest setup is FAL plus one language provider. The capability matrix lists which provider covers which model type. 2. Say what you want On the Home tab, pick Video, or use + New → New timeline. Type one sentence: A 15-second ad for a desk lamp: a quiet desk at night, the lamp switches on. Press Continue. If you close the tab, NodeTool keeps your progress. Other ways to start: Drop your media — use your own clips, audio, or images. Start from a script — write the words first. Start with a blank timeline — skip the plan. 3. Plan the beats A beat is one clip in your video. Pick a template: Template Length Shape 15s ad 15 s 16:9 30s spot 30 s 16:9 60s explainer 60 s 16:9 9:16 social clip 20 s 9:16 Trailer 45 s 16:9 Music video 60 s 16:9 Slideshow 40 s 16:9 Then: Pick a Model at the bottom, then press Plan the beats. This step costs one language model call. Edit the plan. Each beat has a description, a length, a transition, and a voiceover line. The total length turns to a warning color when the plan is too long. Re-plan writes the beats again and follows your edits. Press Continue to look. 4. Pick the look and generate Set the look: Aspect ratio — comes from the template. You can change it. Video model — each tile plays a sample clip. Voiceover — on or off. Pick a voice when it is on. Music — adds one music track under the whole video. Check the cost next to the button. Press Generate your video. The timeline opens and the clips fill in as they finish. 5. Edit and export Press Retry N failed to run failed clips again. You can close the tab during generation. The clips are there when you come back. Edit the cut. Drag to move, drag an edge to trim, S to split, Delete to remove. These are the default NodeTool keys. Press ? to see them and to switch the keyboard preset to Premiere Pro or Final Cut Pro. Or ask the Editor Assistant: “fade out the last clip”. Press Export video to save an MP4 file. Save as Asset keeps it in your library. Every step is also an agent tool. You can click, chat, or do both. Coding agents get the same tools over MCP: NodeTool as an MCP Server. What you just did Providers and API keys. You connected a language model and a video model from your own accounts. See Models & Providers. Beats. You turned one sentence into a plan of clips before paying for any of them. See Creative Agent. Timeline. You generated the clips onto tracks you can trim, split, and reorder. See Video Editor. Assets. You exported the result and saved it to your library, where any workflow can reuse it. See Key Concepts. Agent. You used the Editor Assistant on the same timeline you edit by hand. See Chat. Troubleshooting Something stuck? See Troubleshooting or ask on Discord. Next steps Your First WorkflowBuild a four-node image workflow from an empty canvas. Creative AgentControl each shot, stills first, and keep characters consistent. Video EditorTrim, split, and layer clips on the timeline. Key ConceptsWorkflows, assets, sketches, timelines, and agents in one page. Questions: GitHub--- # Chat & Agents Source: https://docs.nodetool.ai/global-chat-agents.html Most tasks in NodeTool start with the agent. Describe what you want. The agent plans the steps, connects the nodes, picks models, runs the task, and edits your open document. Every message runs the agent loop, so there is no mode to switch on. What it can build What you ask for What it works on Learn more Create, change, or fix a workflow Workflows Workflow Editor Add a layer, mask, or fill Sketches Sketch Editor Add a track, cut, or animation Timelines Video Editor Create shots, stills, clips, or a cut Storyboards Creative Agent Add voiced lines or subtitles Scripts Creative Agent Build forms, fields, buttons, or outputs Mini Apps App Builder Add a level, an enemy, or a rule Games Game Editor The agent uses the same actions the interface offers, so you can watch changes land as they happen. The result is a normal document you can keep editing yourself. Core guides Chat — How to use the composer, threads, permissions, and the agent. Agent Memory — How the agent shares results between the steps of one run. Agent CLI — Run the agent from your terminal. Chat CLI & Chat Server — Automate chats or build your own interface. NodeTool as an MCP Server — Drive NodeTool from Claude Code, Codex, OpenCode, or any MCP client. Chat API — Start chats, get responses, and run tools from your own code. Typical workflows Build a workflow from a prompt Open Chat, pick an AI model, and describe what the workflow should do. The agent will plan the steps, make sure they work with the available tools, save the workflow, and run it. Edit what you’re working on With a workflow, sketch, or timeline open, just ask for the change you want: “add an upscale step” or “put the narration on a new audio track.” The agent will update your document right away. Run a saved workflow from chat Save your workflow in the editor, then select it in the chat or ask for it by name. The results will appear directly in your thread. Control how much the agent does automatically You can set permissions for each thread: Plan gives you a proposal and runs nothing, Default asks before actions, and Auto runs routine work unattended but asks once before an action it labels high risk, such as deleting, publishing, or spending. Learn more in the Chat guide. Run agents behind the scenes Use the Agent CLI for automated scripts, or the Chat API to connect the agent to a backend service. For more examples, check out the Cookbook, or see the Deployment Guide to learn how to put agents into production.--- # Chat Source: https://docs.nodetool.ai/global-chat.html Chat is where you tell NodeTool what to make. The agent behind it plans, calls tools, builds and edits the document you have open, generates images, video, and speech, and runs your saved workflows, all from one composer. It talks to any configured provider. Overview Builds and edits workflows, sketches, timelines, storyboards, scripts, and mini apps Every turn runs the agent loop. The assistant plans and calls tools as a task needs them 20+ providers (OpenAI, Anthropic, Gemini, Ollama, …) One composer for chat, image, video, speech, and music generation Tools: web search, files, code execution, HTTP, and more Permission modes (Plan / Default / Auto) set how much the agent may do without asking Run saved workflows from chat Multiple threads, persistent history Standalone tray window Persistent WebSocket connection — reconnects after reloads. Getting Started Opening Chat From the App: Click the Chats icon in the left rail, then pick a thread or start a new one Standalone Window: Click the NodeTool system tray icon and select Chat for a dedicated, focused window Chrome Side Panel: Install the Chrome extension and click Open chat in its popup. The panel is chat only, but it reads and writes the same threads, and the agent can act on the tab you have open Choosing a Model Click the model chip in the composer to open the model picker. Available models depend on your configured providers: Cloud models – OpenAI GPT, Anthropic Claude, Google Gemini (requires API keys) Local models – Ollama, LM Studio models (requires local installation) Configure providers in Settings > Models & Providers. See Models & Providers. Composer Modes The same composer generates more than text. Click the mode chip to switch between: Mode What it does Chat Talk to the model, run tools, and drive agents Generate Images Text-to-image with your chosen image model Edit Images Image-to-image edits on a dropped image Generate Videos Text-to-video Animate Image Turn a still image into a clip Reference to Video One clip from several attached images and videos — a character plus a garment, a product plus a location, a clip whose motion the new shot follows Generate Speech Text-to-speech with a voice picker Generate Music Text-to-music with your chosen music model and a duration Each mode swaps in its own controls — resolution, aspect ratio, duration, voice — and attaches them to the message so the server routes to the right provider call. Reference to Video takes every image and video on the message, plus the reference image of any @-mentioned entity, so the prompt describes the action and the camera rather than the subjects. Its audio chip chooses between what the model scores and the first reference video’s own track; only some models accept the second, and one that does not rejects the request. Mentions: @ for assets and entities Type @ in the composer to reference things by name instead of hunting for them: Entities — characters, locations, styles, and props from your entity library appear first. Picking one inlines its name into the message and attaches its reference image, so generations stay consistent with the entity’s canonical look. Assets — recent and saved library files. Picking one attaches it to the message like a drag from the asset library. The same picker works inside the workflow editor’s Prompt node, where a picked entity becomes a chip that expands to its descriptor and reference image at generation time. Slash commands: / for skills Type / at the start of a word in the composer to open a list of your skills and the skills NodeTool ships. Picking one inserts /<skill name> into the message so the agent loads that skill for the request. The list filters by skill name and description as you type. See Skills for how to write one and for the full catalog. Conversation Threads Chat organizes conversations into threads: Create threads – Click the New Chat button to start a fresh conversation Switch threads – Pick a thread from the Chats panel in the left rail; it opens as a workspace tab Delete threads – Remove conversations you no longer need Message history – Scroll through past messages with cursor-based pagination Message caching – Recent messages are cached locally for fast loading Each thread keeps its own model, permission mode, and history. Agents The Agent Loop Every turn in Chat runs the same agent loop — there’s no separate mode to turn on. The assistant decides for itself, per request, whether to break a task into steps, select tools, and execute a multi-step plan, or just answer directly. Planning – The agent analyzes your request and, when the task warrants it, creates a plan with ordered steps Tool selection – For each step, the agent chooses from the available tools Execution – Steps run in sequence, with results feeding the next step Adaptation – The agent adjusts its plan based on intermediate results Reporting – Progress, tool calls, and reasoning stream in real time Permission Modes The permission chip sets how far the agent may act on its own. It’s per-thread, so a scratch thread can run wide open while a production one stays cautious: Mode Behavior Plan Read and propose only. No actions taken. Default Reads run automatically; actions ask first. Auto Routine work runs unattended; the agent asks once before an action it declares risky. In Plan mode a request that needs several steps is decomposed by the task planner rather than described in prose: the agent calls create_plan, and the plan arrives as a DAG of tasks and steps with the independent ones marked, so you can see what would run concurrently. Nothing in it executes. A question that needs no plan still gets a plain answer. Switch to Default or Auto and say to run it, and the agent calls execute_plan with that same plan — not a fresh one derived from the conversation. You are asked once, for the plan as a whole, and then tasks stream in as they finish, independent ones together. Ask for a change first (“run it without the last task”) and the amended plan is what runs. In Auto the agent labels every code action it writes low or high risk. A low action — reading, computing, work you asked for and can undo — runs with no prompt. A high one — deleting or overwriting something, publishing or sending anything outside the account, spending real money — asks you first, once, showing the program it wants to run. Unlabeled counts as high. “Allow for this chat” stops the asking for the rest of the thread. Agent Capabilities When a request calls for it, the assistant can: Capability Examples Building workflows Plan a graph, validate it against the node library, save it, run it, debug what failed Editing open documents Add a node, rewire an edge, add a timeline track, paint a sketch layer, place an app widget Web research Search the web, browse pages, extract content File operations Read, write, and organize files in your workspace Code execution Run JavaScript in a sandboxed environment Data analysis Perform calculations, query vector databases Document processing Extract text from PDFs, process emails Asset management Create, organize, and index assets HTTP requests Call external APIs and process responses Workflow execution Run saved NodeTool workflows with custom inputs Watching an Agent Work As the agent runs, the thread shows the task plan, each step’s status as it starts, completes, or fails, the tools it calls and their results, and — on models that support it — its reasoning. Building and Editing From Chat Building a workflow Describe what the workflow should do and the agent runs its build loop: plan the graph, validate it against the node library, save it as a workflow, run it, and debug whatever failed. The result is a normal workflow in your library, opened in the editor like any other. Editing what you have open The agent reads the document in front of you and edits it with the same actions the interface offers. Ask it to add a node, rewire a connection, or explain what a branch does, and watch the change land on the canvas. The same holds on every other surface: Open document Ask for Workflow “Add an upscale step after the image node and preview both.” Sketch “Add a shadow layer at 40% opacity, multiply blend.” Timeline “Put the narration on a new audio track and fade the last clip out.” Storyboard “Render stills for every shot, then clips for the ones I picked.” Script “Voice every draft line with the cast voices and send it to a timeline.” Mini app “Add a tone dropdown and show the result underneath the button.” Each editor tool names the document it acts on, so a request that targets something you don’t have open comes back saying which documents are. Running a saved workflow Choose a workflow in the composer, or ask for it by name. The agent calls it with your inputs and streams the results into the thread. Save the workflow in the workflow editor first. Available Tools Chat agents have access to these tools: Tool Category What It Does Browser Navigate web pages, extract content, take screenshots Search Web search via multiple search providers Filesystem Read/write files, list directories, manage workspace Code Execute JavaScript in a sandboxed environment, including calculations HTTP Make HTTP requests to external APIs PDF Extract text and data from PDF documents Email Read and process email messages Assets Upload, organize, and manage NodeTool assets Vectors Query and manage vector database collections Google Interact with Google APIs (search, drive, etc.) MCP Tools (Model Context Protocol) Add external MCP servers in Settings → MCP Servers → External servers (local app only). Click Add server and give the server an id (lowercase letters, digits, and _, up to 32 characters), a name, and one of two transports: stdio: a command, arguments (one per line), environment variables, and an optional working directory. Stdio servers are unavailable on some hosted deployments. http: an http or https URL and optional request headers. Environment and header values can reference stored secrets as ${SECRET_NAME}. A server’s tools join the agent’s toolbelt on the next message, named mcp_<id>_<tool>. Test connects to a server and checks it. Disable turns a server off without deleting it. See the MCP documentation for available MCP servers. To use NodeTool from another agent instead, see NodeTool as an MCP Server. Standalone Chat Window Access chat in a focused, dedicated window outside the main app: Click the NodeTool icon in your system tray Select Chat from the menu A new window opens with just the chat interface The standalone window is useful for: Quick questions without switching to the full app Running agents in the background while doing other work Using chat as a general-purpose AI assistant Next Steps Chat & Agents – Agent CLI and API integration Chat API – Programmatic access for running chats Chat CLI – Command-line chat interface Agent CLI – Run the agent loop from the terminal Models & Providers – Configure AI providers Cookbook – Agent workflow patterns--- # Glossary Source: https://docs.nodetool.ai/glossary.html NodeTool terminology in plain language. Simple explanations followed by technical details where relevant. Core Concepts NodeTool The platform you’re using! NodeTool is a visual environment for building AI workflows. You connect nodes (building blocks) to create automations without writing code. Workflow A workflow is your project – a collection of connected nodes that accomplish a task together. Think of it like a recipe: each step (node) does something specific, and together they create the final result. Technical: A directed acyclic graph (DAG) of nodes describing an end-to-end task. Node A node is a single building block in your workflow. Each node does one specific job, like “generate text” or “resize image.” You connect nodes together to build workflows. Technical: A processing unit with typed inputs, outputs, and configurable properties. Edge / Connection The lines that connect nodes together. They show which node’s output feeds into another node’s input – like pipes carrying water between stations. Input Where data enters – either into a workflow (like uploading a file) or into a specific node (like connecting another node’s output). Output Where results come out. Output nodes show your final results; other nodes have outputs that connect to the next node in the chain. AI & Models Model A pre-trained AI program that has learned to do a specific task. For example, a language model generates text, an image model creates pictures. You don’t train models – you just use them. Example: GPT is a language model; Stable Diffusion is an image model. Provider A service that runs AI models for you. Providers can be: Cloud providers like OpenAI, Anthropic, or Google Local engines like Ollama or llama.cpp that run models on your computer Technical: Adapter that talks to an external AI service (OpenAI, Anthropic, Gemini, Ollama, ComfyUI, etc.). Agent A special type of AI that can plan and execute multi-step tasks. Unlike simple chat, an agent can break down complex requests, use tools, and work through problems step by step. Technical: Multi-step planner/executor that can call tools or workflows. LLM (Large Language Model) A type of AI model that understands and generates text. LLMs power chat assistants, writing assistants, and text analysis tools. Examples: GPT-4, Claude, Llama. Permission Mode A per-thread setting that controls how far the agent acts without asking: Plan (read and propose only), Default (ask before actions), or Auto (routine work runs unattended, high-risk actions still ask). Inference The process of using a trained AI model to generate results. When you “run” a model, you’re performing inference. User Interface Canvas The main work area where you build workflows by placing and connecting nodes. Pan around by dragging, zoom with scroll. Preview Node A special node that shows you intermediate results while your workflow runs. Add previews anywhere to debug or monitor progress. Mini-App A simplified interface for running a workflow. Mini-Apps hide the complexity and show only the inputs and outputs, making them easy to share with non-technical users. Chat NodeTool’s AI assistant interface. Chat with AI models, run workflows conversationally, or let the agent plan and execute multi-step tasks on its own. Inspector / Properties Panel The panel (usually on the right) that shows settings for the selected node. This is where you configure how each node behaves. Workspace The tabbed surface the app opens on. Every document and app page is a tab in it; with no tabs open it shows the new-project surface instead. Running & Execution Run Execute your workflow and see the results. Click the Run button or press Ctrl/⌘+Enter. Job A single workflow execution. When you click Run, NodeTool creates a job that tracks progress, handles errors, and delivers results. Technical: A single workflow execution orchestrated by the kernel’s WorkflowRunner, which runs each node as a NodeActor. Streaming Receiving results progressively as they’re generated, rather than waiting for everything to complete. Many AI nodes stream their output so you see progress in real-time. Execution Strategy Not a setting. The kernel runs every workflow in the server process, with one actor per node and no per-job threads, subprocesses, or containers. The execution_strategy field on a job is stored and never read. See Execution Strategies. Documents & Editors 3D Model A glTF asset (.glb or .gltf) that you block out with shapes and lights in the 3D editor and render as a reference image for a shot. See 3D Editor. App Builder The Design view of an app tab, where you place widgets on a canvas and wire them to workflow inputs and outputs to make a Mini-App. See App Builder. Clip One item on a timeline track: imported media, a generated result bound to a workflow, or an authored text, shape, or 3D model item. See Video Editor. Game A 2D or 3D game document with scenes, entities, behaviors, and assets, which the built-in engine plays in the editor and exports as a standalone web player. See Game Editor. JS Script A saved JavaScript document that runs in the QuickJS sandbox and can call nodes. The Code node uses the same sandbox. See JS Scripts and JavaScript Sandbox. Sketch A layered image document with blend modes, painting tools, and AI generation onto a layer. See Sketch Editor. Storyboard The shot-by-shot surface of a video project. It renders cheap stills to choose from before video generation spend. See AI Video Production. Timeline A multi-track sequence of video, audio, image, overlay, and subtitle clips that you edit, preview, and export. See Video Editor. Agents, Automation & Billing Creative Agent The script-to-screen video pipeline: a Director agent writes the screenplay, the storyboard gates spend shot by shot, and one click assembles the cut into the timeline. See Creative Agent. Credit The unit of NodeTool’s hosted balance. Credits are spent only by calls through the nodetool provider on the hosted cloud. Calls on your own provider keys never use them. See Costs and Credits. Director The agent in the Creative Agent pipeline that turns a brief into a typed screenplay of scenes and shots. See Creative Agent. Entity A reusable character, location, style, or prop that you define once and cast into shots so it stays consistent. See Entities. MCP (Model Context Protocol) An open protocol for giving an agent tools. NodeTool runs as an MCP server, so Claude Code, Codex, Cursor, and other clients can use workflows, assets, and editors. See MCP Server. Memory Short notes that an agent saves in one conversation and recalls in later ones, such as project facts, decisions, and references to assets it made. See Memory. Skill A saved set of instructions for one kind of work. The agent reads its name and description each turn and loads the body when needed or when you pick it with /. See Skills. Trigger A node that starts a workflow run on a schedule, a file change, a webhook, or by hand, so you do not press Run. See Triggers. Infrastructure Server The background program that actually runs your workflows. The desktop app starts one on your own machine and talks to it; you never have to think about it unless you’re hosting NodeTool for other people. Technical: Process that runs workflows and exposes HTTP/WebSocket endpoints (via nodetool serve, optionally with --host, default 127.0.0.1, and --port, default 7777). API Server The part of the server that answers requests from the app: saving and loading workflows, starting runs, returning results. Technical: Node.js HTTP server (via @nodetool-ai/websocket) handling the OpenAI-compatible /v1/chat/completions endpoint and REST routes such as /api/workflows. Proxy An optional service that sits in front of NodeTool to handle security, routing, and SSL certificates for production deployments. Technical: Reverse proxy that terminates TLS and forwards to the API server. Thread ID An identifier that tracks a conversation in Chat. Each chat thread has its own history and context. Technical: Conversation identifier for chat/agent sessions; used by WebSocket and SSE streams. Data & Storage Asset Any file you use in NodeTool – images, audio, documents, etc. Assets are stored and managed so you can reuse them across workflows. Collection A searchable database of documents used for RAG (Retrieval-Augmented Generation). Collections let AI answer questions using your own documents. Vector Database A special database that stores text as mathematical vectors, enabling semantic search (finding content by meaning, not just keywords). AI Techniques RAG (Retrieval-Augmented Generation) A technique where an AI model answers questions using your own documents as context. Instead of relying only on its training data, the model retrieves relevant snippets from your documents first, then generates an answer based on that context. This dramatically reduces hallucinations and keeps answers grounded in your data. In NodeTool: Use the Hybrid Search node to retrieve documents, then pass them to an Agent node through a Template node. Embedding A mathematical representation of text (or images) as a list of numbers (a “vector”). Embeddings capture meaning, so similar concepts have similar numbers. This is what makes semantic search possible — finding content by meaning rather than exact keyword matches. Prompt The text instruction you give to an AI model. Good prompts are specific and clear. In NodeTool, prompts are usually set as text inputs or constructed using Template nodes that fill variables into a text template. Temperature A setting that controls how creative or deterministic an AI model’s output is. Low temperature (0.0–0.3) produces consistent, factual responses. High temperature (0.8–1.2) produces more varied, creative results. Quantization A technique for making AI models smaller and faster by reducing the precision of their numbers. A Q4 model uses 4-bit precision (smaller, slightly less accurate), while Q8 uses 8-bit (larger, more accurate). Quantized models let you run larger models on less powerful hardware. Fine-tuning The process of further training an existing AI model on your own data to specialize it for a particular task. NodeTool can use fine-tuned models from providers like HuggingFace or OpenAI. Development Terms DAG (Directed Acyclic Graph) The technical name for how workflows are structured. “Directed” means data flows one way; “Acyclic” means no loops. NodeTool handles this automatically — you just connect nodes and it figures out the execution order. DSL (Domain Specific Language) NodeTool’s TypeScript API (@nodetool-ai/dsl) for building workflows in code rather than the visual editor. Useful for automation, testing, and custom integrations. Node Pack A collection of related nodes bundled together. Install node packs to add new capabilities to NodeTool (e.g., additional model providers or data processing tools). API Key A secret string that authenticates you with a cloud AI provider. You get API keys from provider dashboards (OpenAI, Anthropic, Google, etc.) and enter them in NodeTool’s Settings → Models & Providers panel. WebSocket A communication protocol that allows real-time, bidirectional data flow between NodeTool’s frontend and backend. This is what enables live streaming of workflow results and chat responses. SSE (Server-Sent Events) A one-way streaming protocol used by NodeTool’s Server API to push workflow progress and results to clients in real-time. Similar to WebSocket but simpler and HTTP-based. See Also Key Concepts – Deeper explanation of core ideas Getting Started – Hands-on tutorial Models & Providers – Setting up AI models--- # Agent-Owned Godot Projects — Design Question Source: https://docs.nodetool.ai/godot-agent-owned-project-design-question.html Agent-Owned Godot Projects — Design Question Status: Open design question Decision requested Design the NodeTool game architecture for this product direction: The agent owns and maintains a game. It manages the game source and generated assets, may write any Godot code the game needs, and works in a dedicated project folder. A shipped template may seed the game, but it must not remain the authority over the source after creation. The review must decide what a game is in NodeTool, where its files live, how the agent and workflows update it, and how NodeTool runs untrusted agent-authored Godot code safely. The answer should replace or narrow the current model in which a workflow graph fills a fixed template and an export node produces the Godot project. This document describes the current implementation, the requested product model, constraints, conflicts, and review questions. The reviewer should not need prior discussion to understand the decision. Requested review output Return a design with these sections: Recommended ownership model and why it is better than the alternatives. The smallest agent-facing interface that covers project creation, source editing, generated-asset installation, verification, playtesting, and build export. Persistent data shapes and ownership relationships among a NodeTool project, workspace, game, workflow, asset, and chat thread. Folder layout and path rules for local and virtual workspaces. End-to-end sequences for creating a game, editing code, regenerating one asset, verifying, and exporting a build. Security model for running agent-authored GDScript and importing a project. Local and cloud execution behavior. Migration from the current template/export-node design. Failure modes, recovery behavior, versioning, and concurrent-edit rules. Test seams, harness coverage, and staged implementation plan. Name rejected alternatives and the facts that rule them out. Mark assumptions that require product confirmation. Product intent The game is a long-lived software project, not a one-time media export. The agent should be able to: create or open a Godot project inspect and edit any source file in that project add scripts, scenes, resources, tests, and configuration generate and revise images, sprite sheets, tilesets, sound effects, and music install generated assets at stable project-relative paths keep code changes when an asset is regenerated run checks and a playtest loop export a distributable build separately from the source tree resume the same game in later turns without rediscovering its location A template is allowed as a starter. It may provide a working loop, placeholder assets, scenes, scripts, input mappings, and smoke tests. After the initial copy, the mutable project folder is authoritative. A template update or asset workflow rerun must not reset agent-authored source. The phrase “separate folder” has four intended separations that the design must make explicit: shipped templates are read-only package files each mutable game has its own workspace-relative source directory generated source assets are installed inside that game without sharing paths with another game distributable builds and archives do not overwrite the source directory Whether each game needs its own Workspace row, or only its own directory inside a project workspace, is open. Current architecture NodeTool projects A NodeTool project is a database row that groups related work. The row contains an id, user id, name, free-text kind, lifecycle timestamps, and an optional project-agent thread id. It does not contain a root directory or game-specific state. The project schema is in: packages/models/src/schema/projects.ts packages/protocol/src/api-schemas/projects.ts packages/models/src/project.ts The project document union currently includes storyboards, scripts, timelines, sketches, applications, and JavaScript scripts. There is no game document type. Workspace files can appear as restorable project tabs when their workspace belongs to the project, but a folder is not itself a project document. Deleting a project deletes rows with the same project_id, including assets, workflows, threads, jobs, predictions, and workspaces. This ownership rule is implemented in Project.deleteOwned in packages/models/src/project.ts. Workspaces A runtime Workspace is the file interface used by agents, nodes, and workflow runs. Paths are relative to its root. The interface rejects traversal outside that root and supports local folders and virtual object storage. Relevant files: packages/runtime/src/workspace.ts packages/runtime/src/storage-workspace.ts packages/models/src/workspace.ts packages/models/src/schema/workspaces.ts packages/websocket/src/lib/workflow-workspace.ts A workspace database row has id, user_id, name, path, project_id, and is_default. A workflow may hold a workspace_id. The runtime interface has these important properties: interface Workspace { readonly localDir: string | null; key(path: string): string; read(path: string): Promise<Uint8Array | null>; readText(path: string): Promise<string | null>; write( path: string, data: Uint8Array | string, contentType?: string ): Promise<void>; exists(path: string): Promise<boolean>; list( path?: string, opts?: { recursive?: boolean } ): Promise<WorkspaceEntry[]>; delete(path: string): Promise<boolean>; deleteAll(path: string): Promise<number>; copy(from: string, to: string): Promise<void>; move(from: string, to: string): Promise<void>; materialize(path: string): Promise<string>; absorb(localPath: string, path: string): Promise<void>; scratchDir(): Promise<string>; } localDir is null for cloud-backed workspaces. Byte movement must use the workspace interface. A host program either needs a local workspace or must materialize input into scratch storage and absorb outputs explicitly. A project-agent chat does not currently resolve its workspace from the project id. Chat workspace resolution uses the conversation’s bound workflow when one exists and otherwise uses the user’s default workspace. The durable lookup is threadWorkspaceWorkflowId in packages/websocket/src/session/chat-turn.ts. This means a project can own workspace rows while its agent thread still writes to a different workspace unless a workflow binding supplies the intended one. Assets Generated media is stored as a NodeTool asset. Assets have project_id, workflow, node, and job provenance. Media tools may also write a workspace copy when given an output path. Relevant files: packages/models/src/schema/assets.ts packages/runtime/src/context.ts packages/agents/src/tools/asset-persist.ts ProcessingContext.createAsset stamps the current projectId, workflowId, jobId, and node id. A Godot project cannot use an asset:// identifier as a runtime file path. The bytes must be copied into the project directory and its Godot resources must reference the copied path. The current Godot writer does this during project export. This creates two related records: the NodeTool asset, which carries provenance and can be reused the installed file inside the Godot project, which Godot reads The new design must define which record owns the stable mapping and how a regeneration updates it. Shipped Godot templates Three hand-written Godot 4.3 projects are checked into: packages/godot-templates/templates/platformer/ packages/godot-templates/templates/topdown/ packages/godot-templates/templates/shmup/ Each template contains: manifest.json project.godot assets/ scenes/ scripts/ test/smoke.gd The package loader is packages/godot-templates/src/index.ts. It resolves the source or packaged template directory, lists templates, locates Godot, and runs headless checks. Packaged backends stage the template directory as a runtime asset. Each template runs with checked-in placeholder assets before generation. Its manifest.json declares: template id and Godot minor version image, sprite-sheet, tileset, sound-effect, and music slots exact cell sizes, frame counts, durations, loop flags, and generation prompts a short list of hook scripts or scenes intended for edits The manifest schema is gameAssetManifest in packages/protocol/src/game-assets.ts: { version: 1, template: string, godot: string, slots: GameSlotSpec[], hooks: string[] } Slot ids map to deterministic files. Dots become underscores: Slot kind Installed path sprite sheet assets/sprites/<slot>.png and .tres tileset assets/tiles/<slot>.png and .tres image assets/images/<slot>.png sound effect or music assets/audio/<slot>.<extension> and import sidecar Asset checks and slot metadata The generation pipeline validates one asset at a time through game checker nodes: nodetool.game.SpriteSheet nodetool.game.Tileset nodetool.game.SeamlessImage nodetool.game.SoundEffect nodetool.game.MusicLoop A checker emits a stored image or audio reference. Its metadata contains a SlotFill under metadata.nodetool_slot. The fill records the slot id and layout facts such as cell size, animation frame ranges, duration, or loop flag. The checker output carries both the accepted bytes and the metadata needed to write Godot resources. A bare SlotFill is insufficient because it does not locate the media bytes. resolveFills in packages/game-nodes/src/fills.ts rejects a bare fill and requires the checker’s media output or a full filled-slot record. The pure writer in packages/godot/src/writer.ts accepts a template manifest and filled manifest. It produces text resources and a list of asset copies. It does not read pixels. checkGodotProject validates generated resource ids and references. Project layout join joinGodotProject in packages/game-nodes/src/project.ts currently combines four responsibilities: validate fills against the selected template manifest generate Godot resources and copy media bytes copy or preserve template source files check references and optionally invoke headless Godot It has two layout modes: create copies the template and writes generated resources refresh preserves every existing template file, including scripts and scenes, while replacing generated resources and copied asset files refresh is selected when project.godot already exists and overwrite is false. It also rewrites references when generated audio uses a different file extension from the placeholder. This preservation rule protects edits, but it also leaves the template export operation as the mechanism that updates an agent-owned project. Export Godot Project node nodetool.game.ExportGodotProject is implemented in packages/game-nodes/src/nodes/game.ts. Inputs: { template?: string; name?: string; fills?: (ImageRef | AudioRef)[]; directory?: string; verify?: boolean; } Outputs: { output: { directory: string; verified: boolean; archive: string }; directory: string; files: string[]; verified: boolean; verification: Record<string, unknown>; errors: string[]; archive: string; } The node: requires a workspace and project name loads a shipped template accepts zero, some, or all template slots keeps template placeholders for omitted slots defaults the destination to games/<slug> never overwrites existing scripts and scenes during refresh writes <directory>.zip defaults verification to true reports verified: true only when Godot ran and returned no errors and no dangling references The node is the final node in the guided Game flow’s generated asset workflow. The graph is expected to remain rerunnable as an asset-generation recipe. Agent Godot capabilities The agent capability module is: packages/agents/src/capabilities/godot.specs.ts packages/agents/src/capabilities/godot.ts It exposes: Capability Current behavior list_game_templates Lists shipped templates, slots, and hooks export_godot_project Requires every manifest slot, joins a template with checked assets, and writes a project verify_godot_project Checks references, imports the project, checks scripts, and runs test/smoke.gd The export capability defaults to godot/<name>, unlike the node’s games/<slug>. It accepts overwrite. The default is false, so later exports preserve existing scripts and scenes. The generic file capabilities already let an agent read, create, and edit text files anywhere inside its resolved workspace: read_file write_file edit_file list_directory glob grep The current godot-game skill in .agents/skills/godot-game/SKILL.md tells the agent to export first, then edit the manifest’s hook files. It permits a new scene when a hook needs one, but the template and hook list still define the expected customization surface. This conflicts with the requested ability to write any Godot code and restructure the project. Guided Game flow The existing product design is docs/creation-flows/game-prd.md. Its main decisions are: the game document is a workflow, not a new document type settings.game stores setup state on that workflow a model writes the game design, then a pure graph builder creates one generation and checker chain per template slot every checker feeds one ExportGodotProject node the graph run writes games/<slug> and a zip the project agent takes over after the first build later graph runs regenerate assets while refresh mode preserves edited code The graph is the stored generation recipe. The exported directory is the playable deliverable. No persistent game record links that directory, its installed files, its source assets, and its workflow except by conventions in workflow settings and paths. Verification and Godot execution verifyWithGodot runs three checks: godot --headless --path <dir> --import --quit godot --headless --path <dir> --check-only -s <script> for every .gd godot --headless --path <dir> -s res://test/smoke.gd Verification runs only when Workspace.localDir is available and a Godot binary is found. A virtual workspace reports that verification was skipped. runGodotHeadless in packages/godot-templates/src/index.ts calls child_process.spawn directly with a timeout. It captures stdout and stderr, but it does not isolate filesystem access, network access, subprocesses, or other host capabilities available to GDScript. Headless Godot is not a code sandbox. The agent-facing verify_godot_project capability is currently categorized as read, even though it executes project scripts. The export capability is categorized as write, and the workflow node can verify as part of a graph run. This classification was acceptable only under an assumption that shipped scripts and narrow hook edits were trusted enough to run. It must be revisited when the explicit product requirement is arbitrary agent-authored Godot code. Current lifecycle The guided path works as follows: A workflow stores the brief, template, style, models, and project name. The graph generates each manifest asset and validates it with a checker. ExportGodotProject copies the template to games/<slug> and installs the checked assets. The export optionally runs headless Godot and creates a zip. The project agent edits hook scripts and scenes in the exported directory. A later graph run regenerates assets and calls the export node again. Refresh mode preserves source files and replaces generated resources and media. This creates two possible authorities: the workflow and template define the initial structure and asset slots the exported folder contains the current game code and may diverge from both The current design avoids overwriting the folder but does not model that folder as the primary game resource. Requested lifecycle The requested model starts from an agent-owned project folder: Create a game identity and assign its source directory. Optionally seed that directory from a template exactly once. Let the agent inspect, add, move, or edit any Godot source file. Generate assets through direct media calls or reusable workflows. Install selected asset versions at stable paths in the game directory. Update Godot resources or scenes without resetting unrelated source. Check references, parse scripts, and run approved verification or playtests. Keep the same game identity, directory, and agent thread across later turns. Export distributable builds and archives outside the source directory. Templates and workflows become optional inputs to the game. Neither is the game itself. An illustrative layout is below. It is not a decision: games/<game-id>/ project.godot scenes/ scripts/ assets/ generated/ imported/ test/ .nodetool/ game.json asset-map.json builds/<game-id>/ <platform exports and source archives> The reviewer should decide whether NodeTool metadata belongs in hidden files, database rows, workflow settings, asset metadata, or a combination. Godot must not depend on NodeTool-only metadata to run after download. Facts that expose the design conflict F1 — The current document is a workflow. The Game flow decision says the workflow is the game document, while the requested model says the mutable Godot source directory is the game. F2 — Workspace selection is not project selection. A project-agent thread resolves a workflow workspace or the user default. It does not automatically resolve a workspace owned by the project. F3 — Export paths disagree. The graph node defaults to games/<slug> and the agent capability defaults to godot/<name>. F4 — Slot completeness disagrees. The graph node accepts partial fills and placeholders. The agent export capability requires every manifest slot. F5 — Code authority is implicit. Refresh preserves source files, but no game record declares that the folder is authoritative or records which template files have diverged. F6 — Asset identity is split. NodeTool owns the generated asset record, while Godot owns a copied file. The mapping exists only while export computes it from slot metadata and deterministic paths. F7 — The hook model is too narrow. The current skill directs edits toward manifest hooks. The requested agent can replace the scene tree, add systems, and write any code. F8 — Verification executes code. The capability is classified as a read and the export node verifies by default, but arbitrary GDScript executes with the Godot process’s host permissions. F9 — Cloud storage is not an executable folder. File editing works through a virtual workspace, but Godot verification currently requires localDir. F10 — Source archives are coupled to asset refresh. The export node writes a zip every time it updates assets, although source archiving and generated asset installation are different operations. Constraints and proposed invariants The reviewer may challenge these, but must address each one. The mutable Godot source must never live under packages/godot-templates/templates/. Every agent file operation must remain confined to the resolved workspace. A game must have a stable id. Renaming a game must not silently move or fork its source. The source directory must remain stable across chat turns and workflow runs. Generated asset installation must be idempotent and must not rewrite unrelated source files. Regenerating one asset must preserve scripts, scenes, and all other installed assets. Godot paths must be normal res:// project paths. Downloaded projects must run without NodeTool. NodeTool asset provenance must survive installation so the agent can identify and replace the source asset later. A skipped verification must never be reported as successful verification. Running agent-authored code must be a distinct, correctly classified action with an explicit trust and isolation model. Local and cloud workspaces must share file semantics. Execution may differ, but the difference must be represented in results rather than hidden. Deleting a NodeTool project must have defined behavior for the game source, installed asset copies, generated asset records, builds, and active jobs. Concurrent workflow and agent writes must not silently discard source edits. Existing template-generated games need a migration or compatibility path. Candidate ownership models The review should compare at least these options. O1 — Folder convention only A game is a directory such as games/<id>. The project agent stores the path in memory or workflow settings and uses generic file tools. No new game record or document type is added. This has the smallest implementation but weak identity, discovery, migration, concurrency, asset mapping, and deletion semantics. A later conversation may not know which folder is authoritative. O2 — Game document that points to a directory Add a persistent game resource with an id, project_id, workspace_id, source path, Godot version, optional template origin, optional generation workflow ids, and asset mappings. The source remains normal workspace files. The document provides identity, UI discovery, lifecycle, and agent operations. The design must prevent the database document and source metadata from becoming two competing authorities. O3 — NodeTool project kind with one assigned workspace root Treat a NodeTool project whose kind is game as the game identity. Assign one workspace or workspace subdirectory as its source root. The project agent owns all source and generated assets there. No separate game row exists. This keeps one product-level identity, but the current project model permits many documents and workspaces and does not have a canonical root. The free-text kind field is not currently an invariant. O4 — Workflow remains the game record Keep settings.game and add a durable source-directory pointer, asset map, and agent tools around it. The workflow remains the record and asset-generation recipe while the folder is its mutable output. This preserves the guided flow but continues coupling game identity to one workflow. It must explain games created without a workflow and games with multiple asset workflows. O5 — Game resource owns source, workflows are attached recipes Create a game resource that owns the directory and metadata. Zero or more workflows may generate or transform its assets. Templates are creation inputs. The project agent operates on the game resource and generic file tools are scoped to its source root. This most directly matches the requested lifecycle, but adds a resource type, server capabilities, UI integration, persistence, deletion, copying, and migration work. Design questions Q1 — Canonical identity: Is the game a NodeTool project, a new game document, a workflow, or only a workspace folder? Which object owns the stable id? Q2 — Source authority: Which state is authoritative for project structure, installed asset mappings, Godot version, and template origin? What is stored in the database versus the source tree? Q3 — Workspace assignment: Does every game receive a workspace row, or a reserved directory inside a project workspace? How does the project-agent thread resolve it without relying on a workflow binding? Q4 — Agent scope: Should generic file tools be scoped to the game source root, or should game-specific tools accept project-relative paths? How is accidental editing of another game prevented? Q5 — Template role: Is template use a one-time copy, a recorded upstream, or an ongoing merge source? Is creating a blank Godot project supported? Q6 — Asset installation: What interface copies a NodeTool asset into the game, chooses a stable destination, writes .tres or scene references, and records provenance? Is the current slot contract retained as an optional helper rather than the project model? Q7 — Asset updates: How does a later generation replace one installed asset without rerunning a whole export? How are file-extension changes, sprite-frame layout changes, and references handled? Q8 — Workflow relationship: Can a game have zero, one, or many asset workflows? How does a workflow target a game without owning it? Should the ExportGodotProject node be replaced by an asset-install node, narrowed to a one-time bootstrap node, or retired? Q9 — Agent interface depth: What small interface hides workspace resolution, asset materialization, provenance updates, reference checks, verification results, and build output handling? Avoid one tool for each file operation when generic file tools already cover source editing. Q10 — Arbitrary Godot code: Which Godot files may the agent write? The product direction says any file, so any restriction must be a security rule, not a template convention. Q11 — Execution security: Is agent-authored GDScript trusted, approved per run, containerized, delegated to an isolated worker, or never run on the host? How are imports, editor plugins, network access, subprocesses, and filesystem access handled? Q12 — Permission categories: Which operations are reads, writes, executes, or external actions? verify_godot_project cannot remain a read if it runs arbitrary project scripts. Q13 — Cloud behavior: Should cloud verification materialize the whole source tree into scratch storage, run in an isolated worker, and absorb only declared outputs? If execution is unavailable, what checks still run? Q14 — Builds: Where do desktop, web, mobile, and source-zip exports live? Are builds workspace files, assets, jobs, or all three with one owner? Q15 — Versioning and recovery: Are game source snapshots stored by the workspace, as archives, in Git, or as game-document versions? What happens after a bad agent edit? Q16 — Concurrency: How are agent edits, workflow asset installs, Godot imports, and builds serialized or checked for stale writes? Q17 — UI integration: How does a creator open the game, inspect files, see installed assets and verification status, download source, and start the project agent? Q18 — Migration: How are existing workflow-backed games with settings.game and games/<slug> adopted without moving or overwriting their source? Q19 — Copy and delete: What does copying a game copy? What does deletion remove? How are shared NodeTool assets handled? Q20 — Extensibility: Can the same ownership model later support imported Godot repositories, Git-backed projects, multiple levels, plug-ins, and games that do not use a shipped asset manifest? Minimum product scenarios The selected design must describe these scenarios without hidden manual steps. A1 — Start from a template Create a game named “Autumn Fox.” Seed the platformer template into its source directory. Give the project agent the game identity and source root. Let the agent replace the movement script and scene structure. Verify or report why execution is unavailable. Reopen the same game in a later conversation. A2 — Start without a shipped template Create an empty Godot 4 project. Let the agent write project.godot, scenes, scripts, and tests. Add generated art and audio. Verify and export it without inventing a template manifest. A3 — Regenerate one asset Select the current player sprite’s NodeTool source asset. Generate and check a replacement. Install it into the existing game. Update dependent Godot resources only when the layout changed. Preserve all source edits and other assets. Record the new provenance and keep the old version recoverable. A4 — Multiple asset workflows Attach one workflow for character sheets and another for music. Run either workflow independently. Install selected outputs into one game. Do not make either workflow the game’s owner. A5 — Agent-authored code fails The agent edits several scripts. A syntax check or smoke test fails. Return exact script errors and retain the changed source for inspection. Let the agent repair or restore the previous version. Never claim that a skipped or failed check passed. A6 — Cloud workspace Create and edit a game in object storage. Install generated assets without requiring a local filesystem. Run storage-only validation. Either execute Godot in an isolated worker or return a specific unavailable result. Preserve the same source and asset mappings when later opened locally. A7 — Existing guided-flow game Open a workflow with settings.game and an existing games/<slug> folder. Adopt that folder as the authoritative source without recopying its template. Keep the existing generation workflow as an attached recipe. Preserve all edited scripts, scenes, generated assets, and verification history. Decision criteria A preferred design should: make the mutable game source the clear authority let the agent write arbitrary valid Godot source without template-specific restrictions keep the agent’s file scope narrow enough to prevent cross-game writes separate project creation, asset installation, source verification, and build export keep workflows useful without making a workflow mandatory hide asset-copy and Godot-resource bookkeeping behind a small interface preserve NodeTool asset provenance and project ownership work with local and virtual workspaces classify and isolate code execution correctly provide recovery from bad edits and stale concurrent writes give the UI and agent one stable game id migrate current games without data loss keep a downloaded Godot project independent of NodeTool Risks the review must address R1 — Host code execution: GDScript run by Godot can act with the Godot process’s OS permissions. A timeout is not isolation. R2 — Split authority: Database metadata, workflow settings, and source files can disagree about the current game. R3 — Lost edits: An asset refresh, template update, or stale agent write can overwrite source created in another turn. R4 — Broken references: Moving or changing generated files can leave .tscn, .tres, scripts, and import sidecars pointing at missing paths. R5 — Untraceable assets: A copied PNG or WAV can lose the id and generation record of its NodeTool source. R6 — Cloud divergence: A design that depends on localDir may work in Electron and fail in cloud storage. R7 — Unbounded project copies: Source archives and build outputs can become large duplicates with unclear retention. R8 — Path identity: Deriving a directory from a mutable display name can orphan the project after rename or collide with another game. R9 — Project deletion: Workspace files and shared assets have different ownership semantics and may be deleted too much or too little. R10 — Tool sprawl: Adding shallow tools for every Godot file action can make the agent interface larger without hiding difficult behavior. Existing implementation that can be reused The new design should preserve useful modules unless the ownership model makes them obsolete: slot schemas and fill acceptance in packages/protocol/src/game-assets.ts deterministic Godot resource ids and resource writers in packages/godot/ game checker nodes in the image and audio node packages media-ref byte loading and project-scoped asset creation the workspace interface and local or virtual adapters template loading and placeholder projects in packages/godot-templates/ reference checks in packages/godot/src/check.ts and packages/game-nodes/src/project.ts Godot import, script-check, and smoke-run result shapes project ownership, project-agent threads, and resource-change notifications the CLI harness registry and existing game pipeline fixtures Reuse does not require preserving the current ExportGodotProject interface or keeping a workflow as the game record. Existing tests and harness coverage Current behavior is covered in these areas: packages/godot/tests/ tests deterministic project writing and reference checks packages/godot-templates/tests/ tests template manifests and optional real Godot execution packages/game-nodes/tests/game-nodes.test.ts tests the export node, partial fills, refresh behavior, outputs, and errors packages/agents/tests/capabilities-godot.test.ts tests agent export and verification packages/protocol tests the game manifest, slot prompts, and graph builder packages/cli/fixtures/graph-resources/platformer-asset-pack.fake.json runs a graph fixture through project export packages/cli/src/harness/registry.ts registers the Godot game pipeline checks web/src/components/setup/game/__tests__/ tests guided-flow summaries and verification reporting The review should identify which tests remain contract tests, which should be replaced, and what new end-to-end fixture proves agent-owned source survives asset updates. Likely affected areas This list is an impact map, not a proposed patch: Area Current role packages/protocol/src/game-assets.ts Template slot and filled-asset contracts packages/godot/ Pure Godot text-resource writer and checker packages/godot-templates/ Shipped starter projects and headless runner packages/game-nodes/ Template nodes, fill resolution, project join, export node packages/agents/src/capabilities/godot* Agent template export and verification tools .agents/skills/godot-game/SKILL.md Current template-first agent procedure packages/runtime/src/workspace.ts Local and virtual file interface packages/models/src/{project,workspace}.ts Persistent project and workspace ownership packages/protocol/src/api-schemas/projects.ts Project resources shown in the UI packages/websocket/src/session/chat-turn.ts Project-agent workspace resolution packages/websocket/src/trpc/routers/projects.ts Project summaries, tabs, lifecycle web/src/components/setup/game/ Guided game creation and landing state packages/protocol/src/game-graph.ts Current template-slot workflow builder docs/creation-flows/game-prd.md Current workflow-as-game product decision packages/cli/src/harness/registry.ts Required headless verification surface Out of scope for this design question choosing a specific game premise, visual style, or shipped template improving model-generated sprite quality implementing a Godot source editor in the browser adding a general Git hosting product deciding which platforms NodeTool should export first implementing the selected architecture in this document The design may require Git, a browser editor, or an isolated worker later. If so, state the dependency and keep the initial interface compatible with it.--- # Google Workspace and Email Source: https://docs.nodetool.ai/google-workspace NodeTool reaches Gmail in two separate ways. The Google Workspace integration signs in with your Google account and gives agents 20 tools for Drive, Gmail, Docs, Sheets and Calendar. The email tools read a Gmail mailbox over IMAP with an app password and work on any install, including a local one.   Google Workspace Email tools Tools 20 (google_* and gmail_*) 3 (search_email, archive_email, add_label_to_email) Credential Google sign-in with OAuth scopes Gmail address and app password Available on Installs with a login, or when forced on Every install Set up in Settings, Models & providers Settings, Models & providers, Services & Advanced Which installs have Google Workspace The integration authenticates with the Google access token that a Supabase Google login returns. There is no API key to paste. A local install has no login, so the tools and the connect card are hidden there. Install Google Workspace Supabase auth mode (hosted, or a server with SUPABASE_URL and SUPABASE_KEY set) On Local mode, no login Off Local server pointed at a hosted Supabase project Off until you set NODETOOL_GOOGLE_WORKSPACE=1 NODETOOL_GOOGLE_WORKSPACE accepts 1 or true to force the integration on and 0 or false to force it off. When it is unset, the server follows its auth mode. See Configuration and Authentication. A server that runs the integration also needs GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET, the same OAuth client you configured for the Google provider in Supabase. Supabase does not refresh Google tokens, so without these the connection stops working about an hour after you connect. Connect a Google account Open Settings and go to Models & providers. Find the Google Workspace card at the top. It shows a Not connected chip. Click Connect. NodeTool sends you to Google’s consent screen. Approve the requested access. Google returns you to NodeTool, and the card shows Connected with the account name. Connecting is a separate step from signing in. Gmail and Drive are restricted Google scopes, so the login screen asks for identity only. You grant Workspace access when you click Connect. The consent screen requests these scopes: Scope Used for drive Drive search, read, and file creation gmail.modify Reading Gmail, changing labels gmail.send Sending email documents Google Docs spreadsheets Google Sheets calendar Google Calendar The server stores the token as an OAuth credential for your account, one per Google account. Token status routes return metadata only and never the token itself. If a tool reports No Google account connected, the credential is missing or has expired. Click Connect again. What agents can do The tools are available to the chat agent and to the MCP agent tools on servers where the integration is on. Code nodes and code actions call them as functions of the google module, for example gmail_search from @nodetool-ai/sandbox-nodetool/google. See Categorize Mails for a workflow that does this. Every tool belongs to the external permission category because each one acts on a third-party service. A failure comes back as an error result, not a thrown exception, so an agent can reconnect or try another file and continue. Tool list Tool What it does google_drive_search Search Drive with a plain phrase or Drive query syntax. Up to 100 files google_drive_get_file Get name, MIME type, size, owners and link for a file google_drive_read_file Read a file as text. Docs, Sheets and Slides export to plain text or CSV google_drive_create_file Create a text file, optionally in a folder (default is the My Drive root) gmail_search Search with Gmail query syntax such as from:alice@example.com is:unread newer_than:7d. Up to 50 messages with subject, sender, date and body gmail_get_message Fetch one message by id gmail_send_message Send a plain-text email with to, subject, body, optional cc and bcc gmail_modify_labels Add or remove labels on a message. Remove INBOX to archive and UNREAD to mark read gmail_list_labels List label ids and names google_docs_read Read a document’s title and text google_docs_create Create a document, optionally with starting text. Returns the id and URL google_docs_append Append text to the end of a document google_sheets_read Read an A1 range such as Sheet1!A1:D50 google_sheets_append Append rows below the last populated row of a range google_sheets_update Overwrite a range with new values google_sheets_create Create a spreadsheet, optionally seeded with rows from A1. Returns the id and URL google_calendar_list_calendars List calendar ids and names google_calendar_list_events List events in a time window, soonest first. Times are RFC3339 and the start defaults to now. Up to 250 events google_calendar_create_event Create an event with a summary, start, end, and optional description, location and attendees google_calendar_delete_event Delete an event by id Calendar tools use your primary calendar unless you pass a calendar_id. Email tools over IMAP The email tools do not use the Google sign-in. They log in to imap.gmail.com on port 993 with a Gmail address and an app password, so they work on a local install with no login. To set them up: Turn on 2-step verification for the Google account, then create an app password. Open Settings, go to Models & providers, and open the Google Mail entry in the Services & Advanced section. Enter the address in Email address and the password in App password. You can also set GOOGLE_MAIL_USER and GOOGLE_APP_PASSWORD as environment variables. The stored value is read first. Both are required. Tool Parameters What it does search_email subject, text, since_hours_ago (default 6), max_results (default 50) Search the inbox, newest first. Returns message_id, subject, sender and body archive_email message_ids Move messages to [Gmail]/All Mail, which removes them from the inbox add_label_to_email message_id, label Add a label to a message The message_id values these tools return are IMAP UIDs. They are not the message ids that gmail_* tools use, so do not pass one to the other. Disconnect Click Disconnect on the Google Workspace card. NodeTool deletes your stored Google credentials from the server and shows “Disconnected Google Workspace. Use Connect to restore access.” The Google tools then return No Google account connected until you connect again. Disconnecting removes the credential on the NodeTool server only. To revoke the grant on Google’s side, remove NodeTool from your Google account’s third-party access page. For the HTTP routes behind the card (/api/oauth/google/*) see API Reference.--- # Graph Resources — Technical Design Source: https://docs.nodetool.ai/graph-resources/design.html Graph Resources — Technical Design Status: shipped, phases 1–5. Companion: tasks.md. Precedent: script-storyboard-link/design.md for the “pure functions in a shared package, consumed by editor, agent tools and nodes alike” pattern this design continues. The game export section is historical. The built-in game engine design replaces that pipeline. 0. The problem in one paragraph Storyboards, scripts, timelines and entities are the documents a director approves. Workflow graphs are the thing that re-runs. Today the two do not meet: a graph can read a script or a timeline (ScriptRef, TimelineRef, nodetool.script.*, nodetool.timeline.*) but there is no StoryboardRef, no entity value type, no nodetool.storyboard.* and no nodetool.entity.*. The creative nodes (Director, ScreenplayShots, ApplyEntities) pass screenplays and entities as untyped dicts and never touch a persisted board or the entity library. So every variant of an approved piece is re-directed by chat, which is exactly the work an ecommerce, marketing or game customer wants to stop paying for. The design makes the four documents typed graph values, adds the nodes that copy and fill them, and keeps the template the human approved read-only. One sentence: approve once in the editor, re-run many times in a graph. 1. Principles P1. A document enters a graph by reference, leaves as a new row. Refs are {type, id, data?} like ScriptRef. Nodes that derive (recast, fill, retarget) create a new row and stamp lineage; they never write the source. A batch cannot overwrite the board a person approved. P2. Pure first, IO second. Every mapping (recast a board, fill a script, retarget a sequence, prompt for a slot) is a pure function in a shared package with a fixture test. The node, the agent capability and the editor call the same function. P3. Re-runs are cheap by construction. Identity is by name, not by row: CreateEntity upserts on (project, kind, name), RecastStoryboard reuses its previous copy for the same substitution, and render nodes skip a shot whose render_inputs.prompt_hash is unchanged. A monthly re-run pays only for what changed. P4. The spend gate survives. Stills and clips are separate nodes, clips refuse a shot without a selected keyframe by default, and both take max_shots. A graph can run to the stills and stop, as the storyboard loop does. P5. No new agent tools. The capabilities already cover headless editing. What is new is the node surface and the shared functions under it. 2. Types 2.1 Protocol (packages/protocol/src/api-types.ts, creative.ts) /** Reference to a persisted storyboard, editable in the board editor and * passable between workflow nodes. Mirrors ScriptRef / TimelineRef. */ export interface StoryboardRef { type: "storyboard"; id?: string | null; /** Optional inline StoryboardDocument, for tests and the debug harness. */ data?: unknown; /** * Set only by a node that created or derived the row in this run. A ref * from the picker or from LoadStoryboard never carries it (§4.2, write * contract). */ writable?: boolean; } Entity already exists (creative.ts, type: "entity", id = asset id). It becomes a node value type named entity with no new interface. The one addition is a lineage marker on entities a graph creates: EntityMarker └─ source?: { workflow_id?: string; key?: string } // `key` is the upsert identity a batch used (e.g. a SKU); lets a re-run // find its own entity even after a rename. Storyboard document lineage (packages/models/src/storyboard.ts, StoryboardDocument, mirrored in api-schemas/storyboards.ts): StoryboardDocument ├─ templateId?: string | null // board this one was recast from └─ recastKey?: string | null // the canonical substitution mapping // (§3.1), so a re-run finds its copy Script document lineage (packages/models/src/script.ts): ScriptDocument └─ templateId?: string | null // script this one was filled from Timeline sequence lineage (packages/timeline/src/types.ts): TimelineSequence └─ templateId?: string | null // sequence this one was retargeted from All optional, all passthrough: old rows load unchanged. 2.2 Node value types Type name Wire shape Default (ref-defaults.ts) Picker storyboard StoryboardRef {type:"storyboard", id:null, data:null} StoryboardProperty over DocumentPickerProperty (useStoryboards) entity Entity {type:"entity", id:"", kind:"prop", name:"", descriptor:""} EntityProperty reusing EntityAssetPickerDialog list[entity] Entity[] [] same picker, multi game_slot GameSlotSpec (protocol game-assets.ts) — (output only) none slot_fill SlotFill — (output only) none The entities property on nodetool.image.TextToImage, nodetool.video.ImageToVideo, nodetool.video.TextToVideo and nodetool.creative.ApplyEntities changes from list[dict] to list[entity]. The runtime value is the same object, so saved graphs keep working; the editor gains a picker and a typed handle. Entity resolution rule. An entity value’s id is its identity; every other field is a cache. A consumer calls resolveEntities(values, context) (packages/runtime/src/entities.ts): when descriptor is empty and the context has getEntity, the library copy wins. A picked entity therefore follows later edits in the library, and an inline entity (tests, DSL) still works with no database. Registration points, one line each: web/src/config/data_types.ts (colour, icon), web/src/components/node/PropertyInput.resolver.tsx, packages/core-nodes/src/nodes/constant.ts (nodetool.constant.Storyboard, nodetool.constant.Entity), packages/dsl/src/types.ts (StoryboardRef, Entity), scripts/verify-backend-bundle.mjs type list. 2.3 ProcessingContext model interfaces (packages/runtime/src/context.ts) Same shape as the script trio, installed in packages/websocket/src/session/model-interfaces.ts and the CLI’s context: getStoryboard: ({userId, id}) => StoryboardResponse | null createStoryboard: ({userId, name, projectId?, document}) => StoryboardResponse updateStoryboard: ({userId, id, document, timelineId?, baseUpdatedAt?}) => StoryboardResponse | null findRecastStoryboard: ({userId, projectId?, templateId, recastKey}) => StoryboardResponse | null // the copy a previous run derived, asked for by identity: // `listStoryboards` answers with a window of the most recently // updated rows, which a catalog batch outgrows listEntities: ({userId, projectId?, kind?, tags?, nameContains?, limit?}) => Entity[] getEntity: ({userId, id}) => Entity | null upsertEntity: ({userId, projectId?, kind, name, descriptor, imageAssetId, description?, tags?, voiceId?, source?}) => Entity // finds (project, kind, name) or `source.key`; updates the // marker in place, else tags the image asset listGameTemplates: () => { id, manifest }[] entityFromAsset moves from packages/agents/src/capabilities/entities.ts to packages/models/src/entity.ts so the websocket host and the agent capability share one reader. The capability keeps its behaviour and imports it. 3. Shared functions (pure) 3.1 packages/storyboard (new, depends on protocol + timeline) Why a package: packages/timeline already holds buildStoryboardTimeline, and protocol holds shot-prompt.ts and script-link.ts. Recast and the render plan are storyboard-only and need both, and both video-nodes and agents must import them. The package mirrors packages/timeline in role. // recast.ts export interface RecastInput { document: StoryboardDocument; // the template, as it is now /** Entities on the source board, resolved. */ boardEntities: Entity[]; /** Incoming cast. Each replaces the board entity it targets. */ cast: Array<{ entity: Entity; replaces?: string /* board entity id or name */ }>; /** The copy a previous run made for this mapping, when reusing (below). */ existing?: StoryboardDocument; } export interface RecastResult { document: StoryboardDocument; // new or re-derived copy, lineage stamped substitutions: Array<{ from: Entity; to: Entity }>; /** Shots whose rendered prompt changed and lost their takes. */ invalidatedShotIds: string[]; /** Shots whose prompt is unchanged and kept keyframe/clip versions. */ keptShotIds: string[]; /** Shots the template no longer has; their takes on `existing` are gone. */ droppedShotIds: string[]; recastKey: string; } export function recastStoryboard(input: RecastInput): RecastResult; Rules, each pinned by a test: A cast entry with replaces targets that board entity. Without it, the entry replaces the single board entity of the same kind; if there are several or none, the entry is appended and nothing is renamed. Substitution rewrites the old name to the new name, whole word, case insensitive, in action, motion, dialogue, narration, slug, and in each shot’s explicit entity_ids, and on entityIds/screenplay.entity_ids. Invalidation is by prompt, not by touch: for each shot the function recomputes keyframePrompt and clipPrompt with the new cast through injectEntities, hashes them the way RenderInputs.prompt_hash is written today, and clears keyframe*/clip*/status only when the hash moved. A product-only frame on a board whose model changed keeps its takes. templateId = source board id. recastKey is the mapping, not the result: one token per resolved substitution, <sourceEntityId>><destEntityId>, one per append, +<destEntityId>, sorted and joined with ,. Input order does not matter; role assignment does. On a board with characters A and B, A→X, B→Y and A→Y, B→X are different keys and different copies. Copies keep the template’s shot ids, so a later re-derive merges by id. timeline_id is not copied; the approved cut is carried by AssembleTimeline (§4.2). Reuse re-derives, it does not return. With existing set, the function still derives from the current template and the current cast, every time. There is no shortcut keyed on a fingerprint: such a key would have to cover every derivation input (template text, board entity descriptors, the incoming entities’ names and descriptors, the mapping), and the derivation is one pass over one document. It then merges: for every shot id in both documents the existing copy’s keyframe*, clip*, status and render_inputs are carried over, and the hash rule above decides whether they survive. Shots new on the template appear; shots the template dropped disappear and are named in droppedShotIds. So a template edit to one shot’s action invalidates that shot on every reused copy and no other; a renamed destination entity (same id, a CreateEntity upsert under the same key) has its new name rewritten into the shots and invalidates exactly the shots that name it; an edited descriptor invalidates the shots it seasons, because the descriptor is in the injected prompt. The next only_stale render sees each new prompt against the old render_inputs. // render-plan.ts export interface ShotRenderPlan { shotId: string; kind: "keyframe" | "clip"; prompt: string; referenceAssetIds: string[]; model: { provider: string; model: string }; aspectRatio: string; /** True when render_inputs match this plan: nothing to do. */ fresh: boolean; } export function planShotRenders( doc: StoryboardDocument, entities: Entity[], kind: "keyframe" | "clip", targets?: string[] ): ShotRenderPlan[]; This is the logic render_storyboard_stills and filterStale run inline today, lifted so the node and the capability plan identically. The IO half, renderShots(context, ref, plans), lives in the same package under io/: it calls context.runGeneration with persist, appends the version with render_inputs, and writes the board through updateStoryboard with a reload-and-retry on StoryboardConflictError (the same posture patchShot has in the capability). The capability’s render tools switch to it in the same PR so there is one render path. 3.2 packages/timeline additions // fill-text.ts export function fillTimelineText( seq: TimelineSequence, values: Record<string, string> ): { sequence: TimelineSequence; filled: string[]; unresolved: string[] }; // `{{key}}` in ClipTextStyle.text and caption text. Unresolved keys are // reported, never blanked. // clone.ts export function cloneTimelineForBoard( seq: TimelineSequence, from: { boardId: string }, to: { boardId: string } ): TimelineSequence; // A new sequence (fresh id, `templateId` = the source sequence id). Every // clip owned by `from.boardId` (`storyboardBoardId` matches) is re-stamped // to `to.boardId` with its media cleared; every other clip and every track // is copied verbatim. The next re-assemble fills the owned clips from the // copy's renders and, through `foreignTimelineParts` (`reassemble.ts`), // leaves the text overlay, the music bed and any other edit alone. Hand // edits on the shot clips themselves (a trim, a moved cut, a transform) // survive too: the re-assemble refills each shot clip by the shot it plays // (`refillShotClips`, `reassemble.ts`) and swaps only the media, appending a // shot added since and dropping the clip of one deleted. A script-linked // board is still rebuilt, because there the words decide the timing. // retarget.ts export function retargetSequence( seq: TimelineSequence, aspectRatio: string, fit: "cover" | "contain" ): { sequence: TimelineSequence; croppedClipIds: string[] }; // frameSizeForAspect for the new canvas; clip transforms rescaled about the // clip centre; `cover` scales to fill and reports every clip that now crops, // `contain` letterboxes. Text clips rescale font size with the short edge. 3.3 packages/protocol additions // script-fill.ts export function fillScript( doc: ScriptDocument, values: Record<string, string> ): { document: ScriptDocument; filled: string[]; unresolved: string[] }; // `{{key}}` in ScriptLine.text. Takes are kept: `needsVoicing` already marks // a line stale when its text moved, so a later VoiceScript pays only for // changed lines. // game-slot-prompt.ts export function slotPrompt( slot: GameSlotSpec, style: Entity | null, cast: Entity[] ): { prompt: string; width: number; height: number; checker: Record<string, unknown> }; // spritesheet: width = cell.w × max frames, height = cell.h × animations; // tileset: a near-square grid of `count` cells; image: `size`; sfx/music: // no size. `checker` is the exact prop bag the matching nodetool.game.* node // wants, so a graph can Switch on slot.kind and wire one node per kind. 4. Nodes All nodes are tagAsServer (they need the model interfaces). Handle names are the ones an agent will see through search_nodes; keep them. 4.1 nodetool.entity.* (packages/core-nodes/src/nodes/entity.ts) Node Inputs Outputs LoadEntity entity: entity (picker) or name: str + kind? entity, descriptor: str, name: str, kind: str, reference_image: image, voice_id: str ListEntities kind?, tags?: list[str], name_contains?: str, project?: str entities: list[entity]; streams entity per item CreateEntity image: image, kind, name: str, descriptor: str, description?, tags?, voice_id?, key?: str (upsert identity, e.g. a SKU) entity, created: bool CreateEntity upserts (§2.3). A batch that runs twice yields the same entity ids twice. 4.2 nodetool.storyboard.* (packages/video-nodes/src/nodes/storyboard.ts) Node Inputs Outputs LoadStoryboard storyboard storyboard (same ref), shots: list[dict], entities: list[entity], style: str, aspect_ratio: str, name: str, image_model, video_model, shot_count: int StoryboardShots storyboard streams shot: dict, index: int, slug: str, keyframe: image, clip: video; output: list[dict] at the end (the ScreenplayShots contract, over a persisted board) CreateStoryboard screenplay: dict (from Director), name?, brief?, cast: list[entity], image_model?, video_model?, project?, reuse_existing: bool = true storyboard (writable), shots: list[dict], shot_count: int, created: bool RecastStoryboard storyboard, cast: list[entity], replaces?: list[str], name?: str, reuse_existing: bool = true storyboard (the copy), invalidated: list[str], kept: list[str] RenderStills storyboard, targets?: list[str], max_shots: int = 24, concurrency: int = 2, only_stale: bool = true storyboard, keyframes: list[image], rendered: list[str], skipped: list[str], failed: list[str] RenderClips storyboard, targets?, max_shots: int = 8, require_keyframe: bool = true, concurrency: int = 1, only_stale: bool = true storyboard, clips: list[video], rendered, skipped, failed AssembleTimeline storyboard, name?: str timeline: timeline, skipped_shots: list[str], retimed: list[dict] CreateStoryboard is the brief-first entry: Director writes the screenplay and this node saves it as a board a person can open in the board editor. With reuse_existing it returns the board it stamped with recastKey: "created:<name>" on an earlier run, so a re-run renders only stale shots, and a hand-made board with the same name is never handed out as writable. RecastStoryboard with reuse_existing looks up a board in the same project with the same templateId and recastKey (findRecastStoryboard, §2.3 — scoped, because a listing is windowed and a catalog batch outgrows it) and passes it to recastStoryboard as existing: the copy is re-derived from the current template (§3.1), so a template edit reaches every reused copy on the next run and nothing else is re-rendered. The output ref carries writable: true. Write contract. A StoryboardRef is read-only unless it carries writable: true, and only a node that created or derived the row in this run sets it: CreateStoryboard, RecastStoryboard, and the render nodes on the ref they pass on. The picker (nodetool.constant.Storyboard), LoadStoryboard and StoryboardShots never set it. RenderStills, RenderClips and AssembleTimeline refuse a ref without the flag with an error naming allow_writes, the per-node override for a graph whose whole purpose is to render the board a person picked. Permission does not depend on whether any derived row exists yet, so a first-run miswire (picker → RenderStills) fails before any spend, and a derived board picked from the library is as protected as a template, whatever its lineage. AssembleTimeline writes timeline_id on the board. When the board has no cut yet and its template has one (templateId → that board’s timeline_id), it first clones the template’s sequence for the copy (cloneTimelineForBoard, §3.2), then re-assembles in place: the shot clips are refilled from the copy’s renders where they sit, and the text overlay, the music bed and every other track survive. A copy therefore inherits the approved cut, not a bare assembly. When the board is script-linked it calls buildLinkedTimeline, as the capability does. Models come from the board (imageModel/videoModel), with an optional image_model/video_model input to override. Unset is an error naming find_model, never a default. 4.3 nodetool.script.* additions (packages/video-nodes/src/nodes/script.ts) Node Inputs Outputs WriteScript model: language_model, brief: str, format: str, cast: list[entity], language?: str, pace?, name?: str script (new row), line_count: int FillScript script, values: dict, name?: str script (new row), filled: list[str], unresolved: list[str] WriteScript runs the ScriptWriterInput prompt from script-authoring.ts and maps cast to speakers with entityId and voice.voice from voice_id. This is the script-first path the link design’s §4 deviation asked for. 4.4 nodetool.timeline.* additions (packages/video-nodes/src/nodes/timeline.ts) Node Inputs Outputs FillTimelineText timeline, values: dict, name? timeline (new row), filled, unresolved RetargetTimeline timeline, aspect_ratio: str, fit: "cover" \| "contain" = "cover", name? timeline (new row), cropped: list[str] Both create a new sequence with templateId; RenderTimeline is unchanged. 4.5 nodetool.game.* additions packages/image-nodes keeps the checkers. The template and export nodes go in a new packages/game-nodes (depends on @nodetool-ai/godot, godot-templates, runtime): Node Inputs Outputs LoadGameTemplate template: str (select from listGameTemplates) manifest: dict, slots: list[game_slot]; streams slot: game_slot SlotPrompt slot: game_slot, style?: entity, cast: list[entity] prompt: str, width: int, height: int, kind: str, checker: dict, seconds: float ExportGodotProject template: str, name: str, fills: list[slot_fill], directory: str, verify: bool = true directory: str, files: list[str], verified: bool, errors: list[str] The existing checkers gain a slot: game_slot input that fills cell_width/cell_height/animations/count/slot_id from the slot, so a SlotPrompt → generator → checker chain wires without hand-copied numbers. ExportGodotProject calls writeGodotProject and, when GODOT_BIN is set and the workspace is local, the same headless verify the capability runs. 5. Examples, end to end Each example ships as a workflow JSON under packages/base-nodes/nodetool/examples/nodetool-base/ and as a nodetool.fake.*-backed fixture the harness runs without spend (§7). E1. Per-SKU product ads (ecommerce) Setup, once, on the surfaces: a director builds the board “Hero 9:16” with a prop entity named Product, a character entity, a style entity, both models set, stills approved, clips rendered, cut assembled, a text clip reading {{name}} — {{price}}. The board and the cut are the templates. Graph (Per-SKU Ad Factory): LoadCSVAssets(folder: products/) → dataframe ForEachRow(dataframe) → row {sku, name, image, descriptor, price} LoadImageFile(row.image) → image CreateEntity(image, kind: prop, name: row.name, descriptor: row.descriptor, key: row.sku) → entity Constant.Storyboard("Hero 9:16") → template RecastStoryboard(template, cast: [entity], replaces: ["Product"]) → board' (invalidated: shots naming Product) RenderStills(board') → board' (only stale shots render) RenderClips(board', require_keyframe: true) → board' AssembleTimeline(board') → timeline' (clones the template cut, re-assembles the shot clips, keeps the overlay and music) FillTimelineText(timeline', values: {name: row.name, price: row.price}) → timeline'' RenderTimeline(timeline'') → video Collect(video) → Output "ads" What it costs: per SKU, one still and one clip for each shot whose prompt names Product; the style frame and any product-free shot are kept from the template. Second run with two new rows: CreateEntity returns the same entity for the eight old SKUs, RecastStoryboard returns the eight old copies, only_stale renders nothing on them, and only the two new SKUs are paid for. A WebhookTrigger on a product-created event in place of the CSV turns it into a listener. What it does not do: publish. The output is a list of videos. A Shopify or ad-platform node is a separate integration (§9). E2. Localized explainer (marketing) Setup: a script written on the script surface with a character entity Mara as the narrator (voice_id set) and lines carrying {{product}} and {{offer}}; a board of six b-roll shots, stills approved, no clips. Graph (Localized Explainer): Constant.Script("Explainer template") → script Constant.Entity(Mara) → narrator Constant.Storyboard("B-roll") → board Cross(rows: offers.csv, languages: ["en","de","fr"]) → (row, lang) WriteScript(model, brief: row.brief, cast: [narrator], language: lang, format: "narrator") → script' // or FillScript when only values change VoiceScript(script') → script' // pays per changed line ScriptToTimeline(script') → timeline' StoryboardShots(board) → keyframes AddClips(timeline', clips: keyframes, image_duration_ms: 4000) → timeline' ScriptToSubtitles(script', format: srt) → subtitles RenderTimeline(timeline') → video The board is read only; nothing renders on it. For a value-only variant (FillScript instead of WriteScript), unchanged lines keep their takes and VoiceScript voices only the lines whose text moved. E3. Game re-skin (gamedev) Setup: a style entity Cave Pixel and a character entity Pip created from an approved sprite, both in the library. Graph (Platformer Asset Pack): LoadGameTemplate("platformer") → streams slot SlotPrompt(slot, style: Cave Pixel, cast: [Pip]) → prompt, width, height, kind, checker Switch(kind) spritesheet → TextToImage(prompt, width, height) → game.SpriteSheet(image, slot) → fill tileset → TextToImage(...) → game.Tileset(image, slot) → fill image → TextToImage(...) → game.SeamlessImage(image, slot)→ fill sfx → TextToAudio(prompt, seconds) → game.SoundEffect(audio, slot) → fill music → TextToMusic(prompt, seconds) → game.MusicLoop(audio, slot) → fill TryCatch(around each branch) → on error, one retry with the checker's message appended to the prompt Collect(fill) → fills ExportGodotProject("platformer", name, fills, directory: "game/", verify: true) → directory, verified, errors A re-skin is swapping the style entity on the SlotPrompt node and running again. Cave Pixel seasons every slot the same way the storyboard render path seasons a shot, so the pack is consistent by construction. E4. One cut, three ratios Constant.Timeline("Launch film 16:9 cut") → timeline (the approved cut, read-only) RetargetTimeline(timeline, "9:16", fit: cover) → t1 (new row; cropped: [...]) RetargetTimeline(timeline, "1:1", fit: cover) → t2 (new row) RenderTimeline(t1), RenderTimeline(t2), RenderTimeline(timeline) The graph starts from the cut, not the board. A picker ref is read-only under the write contract, and RetargetTimeline derives a new row from it without writing the source, so the approved trims and placements carry over unchanged. Re-assembling from the board would rebuild the shot clips (§3.2). Honest limit: a 16:9 clip cropped to 9:16 loses the sides. cropped names the clips so the director can decide which shots need a real 9:16 board (the launch-kit rule of a board per ratio still holds for hero shots). 6. Web StoryboardProperty and EntityProperty (web/src/components/properties/), registered in PropertyInput.resolver.tsx; data_types.ts entries. Constant nodes nodetool.constant.Storyboard / Entity in the node menu under the existing Timeline / Script entries. Output handles of type storyboard / timeline / script on a finished run open the document (the TimelineProperty open-in-editor affordance, reused). A board created by RecastStoryboard shows a “recast from " chip in the board header, linking back. Nothing else in the editors changes. 7. Testing and harnesses Vitest, pure: recastStoryboard (targeting, whole-word rename, hash-based invalidation, kept takes, lineage), planShotRenders (fresh detection matches filterStale on the capability’s fixtures), fillScript, fillTimelineText, retargetSequence (crop report, text scaling), slotPrompt (sizes per kind, checker bag matches each checker’s props). Every suite has a case that must fail: a rename that would hit a substring inside another word, a placeholder with no value, a board with two entities of the target kind and no replaces. Recast fixtures also pin: the swapped assignment (A→X, B→Y versus A→Y, B→X) yields two keys; a re-run after editing one shot’s action on the template, cast ids unchanged, invalidates that shot on the reused copy and keeps the rest; a re-run after renaming a destination entity under the same id, template unchanged, rewrites the name and invalidates only the shots naming it; a cloned cut keeps its foreign clips and, after fillTimelineText, carries the filled overlay text. Vitest, nodes: each node against a ProcessingContext with in-memory model interfaces (testing.ts already builds one), including the write contract (picker → RenderStills refused on a board with no children; a derived board picked from the library refused the same way; allow_writes admits both), the upsert returning the same id twice, and the E1 fake-mode run asserting the filled {{name}} — {{price}} overlay is present on the exported sequence. Harness registry (packages/cli/src/harness/registry.ts): a graph-resources entry whose selfcheck runs the suites above plus nodetool debug on the four example workflows in fake mode (nodetool.fake.GenerateImage in place of the generators; render nodes take a provider override the fixture sets to fake). harness gate picks it up on diffs under packages/storyboard/, the new node files, and the examples. validate_workflow on each example must pass with models stamped. npm run check:agents-docs for the new package’s AGENTS.md. 8. What deliberately does not change The storyboard, script and timeline editors’ data model beyond the three optional lineage fields. buildStoryboardTimeline, buildLinkedTimeline, buildScriptTimeline. The agent capabilities’ names and arguments. render_storyboard_stills and render_storyboard_clips switch to renderShots internally; their outputs are unchanged and their eval cases prove it. Director, ScreenplayShots, ShotBatch, ShotChain: still the path for a graph that directs from a brief with no board. 9. Out of scope, named Publish nodes (Shopify media, YouTube/TikTok upload, Meta ad creative, SMTP). They are the step that makes E1 automation rather than file generation, and they are integration work with no dependency on this design. Separate design. Cost estimation for render nodes. Shot.cost_estimate is unpopulated today; max_shots is the cap this design offers. A LinkedAssemble node for a board plus script pair: AssembleTimeline handles the linked case by reading script_id, which covers it. 10. Risks R1. A graph draws over the approved board. Mitigated by P1 (derive nodes create rows), the write contract in §4.2 (a ref is read-only unless the run derived it), and the lineage chip. R2. Two branches write one board. A ForEachRow that fans out and each branch renders on the same recast copy conflicts on revision. renderShots reloads and retries on conflict, and RecastStoryboard gives each substitution its own row, so parallel SKUs never share a board. R3. Rename collides. “Nova” inside “Novak”. Whole-word, case-insensitive matching with a fixture; RecastStoryboard reports substitutions so the run log shows what was renamed. R4. Hash drift between capability and node. If the prompt hash is computed in two places they diverge and only_stale re-renders everything. One function, planShotRenders, used by both; a test asserts the capability’s filterStale output equals the plan’s fresh set on the same fixture. R5. retargetSequence on transforms with keyframes. Keyframed transforms must rescale every keyframe, not just the base. Covered by a fixture with an animated clip and the validator run on the output.--- # Graph Resources — Tasks Source: https://docs.nodetool.ai/graph-resources/tasks.html Graph Resources — Tasks Companion: design.md. Each phase ships alone and leaves the repo green. After any code change run the four mandatory checks (npm run test:affected, npm run typecheck, npm run lint, npm run dev:nodetool -- harness gate --base origin/main). Phase 1 — Types and entity nodes Protocol: StoryboardRef. Add to packages/protocol/src/api-types.ts next to ScriptRef, with the writable flag (design §2.1); storyboardRefDefault in packages/core-nodes/src/nodes/ref-defaults.ts; nodetool.constant.Storyboard and nodetool.constant.Entity in constant.ts; StoryboardRef + Entity in packages/dsl/src/types.ts; the storyboard and entity names in scripts/verify-backend-bundle.mjs. Test: DSL generation emits storyboard/entity typed inputs for the constant nodes. Lineage fields. templateId/recastKey on StoryboardDocument (packages/models/src/storyboard.ts, api-schemas/storyboards.ts), templateId on ScriptDocument and TimelineSequence, source on EntityMarker (creative.ts zod + reader). Tests: old fixtures parse unchanged; the fields round-trip. entityFromAsset to models. Move to packages/models/src/entity.ts; the agent capability imports it. No behaviour change; existing capabilities-entities tests stay green. Model interfaces. getStoryboard/createStoryboard/updateStoryboard, listEntities/getEntity/upsertEntity, listGameTemplates on ProcessingContextModelInterfaces (packages/runtime/src/context.ts), forwarders on the context, implementations in packages/websocket/src/session/model-interfaces.ts and the CLI context builder. upsertEntity matches source.key, then (project, kind, name). Tests: ownership refusal, upsert returns the same id twice. resolveEntities. packages/runtime/src/entities.ts: fill an entity value from the library when descriptor is empty and getEntity is wired; pass through otherwise. Unit test both branches. nodetool.entity.*. LoadEntity, ListEntities, CreateEntity in packages/core-nodes/src/nodes/entity.ts per design §4.1. Node tests against an in-memory context. list[entity] on generators. Retype entities on nodetool.image.TextToImage, nodetool.video.ImageToVideo, nodetool.video.TextToVideo, nodetool.creative.ApplyEntities; call resolveEntities before injectEntities. Test: a saved graph carrying the old list[dict] value still runs. Web pickers. StoryboardProperty (over DocumentPickerProperty, new useStoryboards hook if none) and EntityProperty (reusing EntityAssetPickerDialog) in web/src/components/properties/; PropertyInput.resolver.tsx cases; data_types.ts entries. Jest: picker renders, selection writes the ref shape. Phase 2 — packages/storyboard and the storyboard nodes Package scaffold. packages/storyboard with AGENTS.md + CLAUDE.md, an entry in packages/AGENTS.md, workspace + turbo wiring, dependency order protocol → timeline → storyboard. npm run check:agents-docs green. recastStoryboard. Per design §3.1 with the fixture suite: explicit replaces, single-of-kind, ambiguous kind appends, whole-word rename (Nova/Novak case), explicit entity_ids rewrite, hash-based invalidation keeping product-free shots, lineage stamping, recastKey stable under cast order and different for the swapped assignment, re-derive over existing (edited template action invalidates one shot, renamed destination entity under the same id invalidates only the shots naming it, added shot appears, dropped shot reported). planShotRenders + renderShots. Lift the plan and the IO from packages/agents/src/capabilities/storyboards.ts; the capability’s render_storyboard_stills/render_storyboard_clips/filterStale call the package. Test asserting filterStale and plan.fresh agree on the capability’s fixtures; storyboard tool-loop eval unchanged. nodetool.storyboard.*. LoadStoryboard, StoryboardShots, RecastStoryboard, RenderStills, RenderClips, AssembleTimeline in packages/video-nodes/src/nodes/storyboard.ts per design §4.2, tagged server. Node tests: reuse_existing returns the prior copy, only_stale skips fresh shots, require_keyframe skips, the write contract (picker ref refused on first run and for a derived board, allow_writes admits, derived refs carry writable), AssembleTimeline clones the template cut for a copy and keeps its foreign clips, takes the linked path when script_id is set, and writes timeline_id. Web lineage chip. “Recast from " in the storyboard header when `templateId` is set, linking to the source board. Phase 3 — Script and timeline derivations fillScript. packages/protocol/src/script-fill.ts, tests for filled/unresolved reporting and take retention. nodetool.script.WriteScript / FillScript. In packages/video-nodes/src/nodes/script.ts per design §4.3; WriteScript maps cast entities to speakers (entityId, voice.voice from voice_id). Node tests with a stubbed provider. cloneTimelineForBoard, fillTimelineText, retargetSequence. In packages/timeline per design §3.2; fixtures with a text clip, a music bed on a foreign track, a keyframed transform, and the validator run on every output. Clone fixture: owned clips re-stamped and cleared, foreign clips and tracks byte-identical. nodetool.timeline.FillTimelineText / RetargetTimeline. Per design §4.4; both create a new sequence with templateId. Phase 4 — Game nodes slotPrompt. packages/protocol/src/game-slot-prompt.ts; tests per slot kind asserting sizes and that checker matches each nodetool.game.* checker’s prop names. packages/game-nodes. LoadGameTemplate, SlotPrompt, ExportGodotProject per design §4.5, with AGENTS.md/CLAUDE.md and the packages/AGENTS.md entry; base-nodes re-exports. Export test against the platformer golden fixture in packages/godot. Checker slot input. SpriteSheet, Tileset, SeamlessImage, SoundEffect, MusicLoop accept slot: game_slot and derive their numeric props from it when connected. Tests: connected slot wins over stale hand-typed values. Phase 5 — Examples, harness, docs Example workflows. Per-SKU Ad Factory, Localized Explainer, Platformer Asset Pack, Three Ratios under packages/base-nodes/nodetool/examples/nodetool-base/, models unset, each passing validate_workflow once models are stamped. Fake-mode fixtures. The same graphs with nodetool.fake.* generators and provider: fake on the render nodes, runnable by nodetool debug with no keys, E4 included. The E1 run asserts the filled overlay text is on the exported sequence; the E4 run asserts the source cut’s clip placements are unchanged on both retargets. Harness registry. graph-resources entry in packages/cli/src/harness/registry.ts whose selfcheck runs the pure suites and the fake-mode debug runs; nodetool harness audit clean; harness gate maps diffs under packages/storyboard/, packages/game-nodes/, the new node files and the examples to it. Docs and skills. Node table in docs/creative-agent.md § Graph templates; the new node types in .claude/skills/video-workflow/SKILL.md and .claude/skills/godot-game/SKILL.md (the graph path next to the invoke_node loop); docs/harnesses.md entry for the harness; the AGENTS.md harness table row. Move this design’s status line to shipped per phase. Phase 5 deviations, and why Models are stamped on the examples, not left unset. validateGraph treats a *_model property as required, so RenderStills/RenderClips with the board’s models and no override fail npm run validate:examples (--warnings-as-errors) — the shipped gate every example passes through. The four examples therefore carry the same model selections the rest of the library does. The node still prefers the board’s model only when the override is the empty selection, which a saved example can no longer be. Per-SKU Ad Factory runs once per SKU rather than looping a CSV. A nodetool.code.Code node is the only way to pull a typed string out of ForEachRow’s row dict, and its outputs are dynamic_outputs, which the correlation analyzer never gives a scope — every downstream node then reads “multiple values at empty scope; only the last is kept” and the fan-out collapses to one item. The example takes the SKU’s fields as inputs instead; the batch driver is a webhook or a per-row call, as design §5 already allows. No Localized Explainer fake-mode fixture. getScript / createScript / updateScript are wired only in packages/websocket/src/session/model-interfaces.ts, not in documentModelInterfaces(), which is what the CLI installs — so every nodetool.script.* node fails under nodetool debug with “ProcessingContext model interface ‘getScript’ is not configured”. The other three fixtures run. ExportGodotProject is terminal in the E3 fixture. Wiring any node downstream of it makes the run never complete: the export emits, the downstream node runs, and then the runner’s promise never settles and the process exits 0 with no bundle.--- # Guided Creation Flows Source: https://docs.nodetool.ai/guided-flows A guided flow is a short sequence of steps that ends in a document you keep editing. You describe what you want, review what NodeTool proposes, and only then does it spend model credits. The flow saves its progress on the document it creates, so a reload or a second window resumes at the same step. There are seven flows. Flow What it makes Steps Image A sketch with variations to pick from Idea, Brief, Look Workflow A graph, checked and run once Idea, Plan, Build Entity A reusable character, location, style, or prop Details, Reference, Review Storyboard A rendered board of shots See AI Video Production Video A cut on the timeline, no board See AI Video Production Script Voiced lines ready to place See AI Video Production Game A playable game in the built-in engine See Game Editor Start a flow From the new-project screen. The screen has a Start with a guided flow section. It shows the Storyboard, Image, and Workflow cards. Choose More formats to also show Entity, Video, Script, and Game. Fewer formats hides them again. Text typed in the prompt box is not sent to a flow unless you click its card, and the Image card uses it as the first brief. From the tab bar. The + New menu lists the same flows in its first section. The menu names the Video card Timeline and the Image card Sketch. Each choice creates an untitled document in the selected project and opens it as a tab. A failed start shows the notification “Could not start the guided flow”. From Studio. The Studio home offers only Storyboard, Video, and Script. From the Entities page. Add entity runs the Entity flow in place. Every step has a primary button named for its outcome, such as Generate your image. When the button is off, a line beside it says why, for example “Pick an image use case”. Where a step spends credits, the cost estimate sits beside the button. Step 1 of the Image, Workflow, and Entity flows has Other ways to start, which skips the flow. Flows opened from the new-project screen also show Change flow on step 1, which asks for confirmation before it discards the draft. Image The Image flow creates a sketch. It turns a sentence into a brief, then renders one, two, or four variations for you to pick from. The sketch editor is described in Sketch Editor. Step 1: Idea The heading is What image do you want? Describe the picture in the Your image box. One sentence is enough. The box is required. Three example briefs below it fill the box when you click one. If the flow was started from the new-project prompt with reference images or entities attached, they appear under Came with your prompt. The brief refinement reads the reference images. Other ways to start skips the brief: Upload an image to edit opens the sketch editor with your PNG, JPEG, WebP, or GIF as the first layer. Start with a blank canvas opens the editor on an empty canvas. The footer has a model picker for the language model that writes the brief. Choose Continue. Step 2: Brief First choose what the picture is for. The cards set the canvas size and the default number of variations. Use case Description Canvas Default variations Product shot One object, clean background, ready to place 1024 x 1024 4 Portrait A person, framed head and shoulders 819 x 1024 4 Key art A poster frame with room for a title 683 x 1024 2 Social post A square that survives a feed 1024 x 1024 4 Logo A flat mark on a plain field 1024 x 1024 4 Concept art A world, painted wide 1024 x 683 2 Texture A flat surface that tiles 1024 x 1024 2 Choose Refine the brief. The language model expands your sentence into five fields and nothing is rendered yet. If the brief already matches your sentence and use case, the button reads Continue to the brief and makes no model call. The review step shows Your brief with five editable fields: Subject, Composition, Lighting, Style words, and Leave out. Edit anything before it is rendered. Re-refine runs the expansion again over the text in the boxes. Under How many variations? choose 1, 2, or 4. Choose Continue to look. Step 3: Look The heading is Choose the look. Size offers five tiles: Square (1:1), Portrait (4:5), Landscape (3:2), Story (9:16), and Banner (21:9). Only the sizes the selected model accepts are shown. Style offers style presets and a No style tile, which renders exactly what the brief says. Model is picked in the footer. If no image provider is connected, the step says so and points you to Settings. Choose Generate your image. The estimated cost appears beside the button when it can be measured. The contact sheet The variations render as a contact sheet. Each one has these buttons: Button Result Sketch editor Opens the editor with that variation Save to entities Saves the variation as an entity New node canvas Starts a node canvas from the image Image to video Sends the image toward video Regenerate Renders that variation again. A failed one reads Try … again Download Saves the image file Below the sheet, Make more variations renders another batch, Back to generation settings returns to the Look step, and Open editor goes to the sketch editor with every variation present. Workflow The Workflow flow plans a graph from a task, shows you the plan, then builds the graph, validates it, and runs it once with sample inputs. Planning places nothing and runs nothing. For the editor itself see Workflow Editor. Step 1: Idea The heading is What should this workflow do? Describe only the task in the The task box, for example “Summarize a PDF and email it”. Add files, services, and destinations later. Three example tasks sit below the box. Other ways to start: Start from an example opens a searchable browser of shipped workflows. Picking one copies it and opens it in the editor. Import a workflow reads a JSON file or a DSL .ts file. Start with a blank canvas skips the plan. Step 2: Plan Choose what kind of workflow it is. The category changes which nodes the planner considers and which run mode step 3 starts on. It places nothing, so a wrong choice costs a re-plan, not a graph. Category Description Starts as Content pipeline Text in, drafted and formatted text out Run by hand Media batch The same render over a list of inputs Run by hand Data extraction Pull structured fields out of documents or pages App with a form Research agent An agent that searches, reads and reports back App with a form Automation on a trigger Runs on a schedule or a webhook, with no one watching On a trigger Chat app A conversation with your data behind it App with a form Choose the planner model in the footer, then Plan the steps. A shipped example brief can use a stored plan with no model call. If the plan already matches your brief and category, the button reads Continue to your plan. The review shows the plan as a numbered list of inputs, steps, and outputs. You can edit titles and descriptions, reorder steps, remove a step, and use Add a step. Two markers can block Continue to setup: A red marker, “No node for this step”, means the step names a node type that does not exist. Pick a real node type in the field beside it. An amber marker, “No … provider connected”, means the step needs a model role that no connected provider covers. Connect opens provider setup in place. Every step needs a title and every output needs a name. Re-plan asks the model again. Step 3: Build The heading is Models and how it runs. Pick a model for each role the plan needs from the providers you have configured. Choose how the workflow starts: Run mode After the test run Run by hand You open it and press run App with a form You put a shareable form over the inputs On a trigger A schedule or a webhook starts it Under Sample inputs you can change the values used for the test run. Choose Build your workflow. The button states that it builds, checks, then runs the workflow once. Cost is unknown until the run returns. The landing checklist When the build finishes, the canvas opens with a checklist of three lines: Graph built with the node count, Validated, and Test run. If something failed, the checklist says what needs a fix. The next-step button depends on the run mode: Continue on canvas for Run by hand Create Mini App for App with a form. See Mini Apps Add a trigger for On a trigger. It opens the trigger-node picker. See Triggers Entity The Entity flow creates a reusable character, location, style, or prop with a reference image. Entities are described in Entities. Step 1: Details The heading is What should stay consistent? Choose the entity type: Character, Location, Style, or Prop. Enter a Name. The placeholder is “Nova”. Enter a Descriptor. This text is added to every prompt that uses the entity, so describe only stable visual traits. Optionally enter Tags (optional) as a comma-separated list. The name and descriptor are required. Start with a blank reference under Other ways to start makes a plain white 1024 x 1024 image, fills an empty name with “Untitled entity” and an empty descriptor with “A reusable visual entity.”, and jumps to the review. Choose Choose a reference. Step 2: Reference The heading is Choose a reference image. Pick the clearest image of the traits to preserve. Choose from assets opens the asset picker. After a pick the button reads Change reference. Images that are already entities are not offered. Generate with AI opens Generate a reference image. Choose a view for the entity type, edit the Prompt, pick an Image model, and choose Generate reference. The image is made at 1K resolution and selected for you. Choose Review entity. Step 3: Review The heading is Review your entity. It shows the reference, the name, the type, the descriptor, and the tags. Choose Create entity. The entity is then available anywhere entities can be added. If you leave before this step, the draft is kept and restored when you return. Storyboard, Video, and Script These flows build video material. AI Video Production documents all three, with their steps, creative context fields, and review screens. Storyboard takes a sentence to a rendered board in four steps. You review the screenplay, assign entities, choose the look, then render stills and clips. Video takes a sentence to a cut on the timeline with no board. Each beat stays a separate clip. Script takes a topic to voiced lines. You write or import the script, add production context, assign speakers and voices, and can create a linked storyboard from it. Game The Game flow creates a playable game in the built-in engine. + New creates a 2D top-down room game named “Untitled game”. On the new-project screen, Game card starts as picks 2D game or 3D exploration. See Game Editor.--- # Agent harness reference Source: https://docs.nodetool.ai/harnesses.html Agent harness reference The full reference for every headless harness, CLI command, and agent tool surface: what it checks, what it does not simulate, and why it is shaped the way it is. The index — which harness answers which need — is AGENTS.md § Agent Harnesses & Tooling. Flag-level reference for the CLI: cli.md. Packed CLI smoke After npm run build:packages, run node scripts/smoke-packed-cli.mjs to install the public package tarballs into an isolated project and exercise the CLI. Add --core-only to repack the test CLI without on-demand node packs. The published CLI manifest retains its packs until the final switch. The core smoke runs JSON and TypeScript workflows, executes a workflow through local MCP, starts the HTTP server, checks catalog validation for an absent FAL pack, and decodes audio and video through the installed media codecs. It audits nested installed packages for accidental pack or optional-library dependencies. The compiler exceptions and their range checks are defined in plan D23. The smoke prints du -sk node_modules for the installed tree. This is the Linux measurement used to establish the later size gate. The dedicated packed-core-cli.yml workflow runs the core smoke on pull requests, including stacked branches. Release CI runs it beside the full packed smoke. Release catalog pins After building packages and running node scripts/prepare-publish.mjs, run node scripts/publish-packages.mjs --pack-only --tarballs-dir <empty-directory> on Linux x64. It packs the catalog registrar packages and their companion libraries, measures each pack in a fresh production install, writes integrity and size pins, rebuilds the catalog artifact, and packs the remaining packages. It preserves every previously packed tarball and performs no publication. The catalog also records original companion hashes, exact npm runtime alias identities and shared-library pins. Staging verifies these before promotion. The install size includes the pack’s production dependency tree. Measurement uses local core directory overrides before final core artifacts exist. npm packs those directories according to their files fields, with scripts disabled. Each install has a fresh project and home, with a shared download cache inside the release artifact directory. Run the deterministic pin, measurement and packing-order regressions with: node scripts/run-vitest.mjs run --config vitest.config.ts scripts/__tests__/pin-catalog.test.mjs scripts/__tests__/publish-packages.test.mjs scripts/__tests__/install-pins.test.mjs scripts/__tests__/staging-release.test.mjs scripts/__tests__/publish-tarballs.test.mjs The release-pins job in packed-core-cli.yml verifies actual Linux measurement and packing without publishing. CLI Two entry points: nodetool (management commands) and nodetool-chat (interactive chat). # From source (no build needed — uses tsx): npm run dev:nodetool -- <command> # nodetool commands npm run dev:chat -- [flags] # interactive chat # From built dist (requires npm run build:packages): npm run nodetool -- <command> npm run chat -- [flags] nodetool chat Every chat session runs the unified agent loop. There is no mode to select: -a, --agent and --no-agent are accepted for backwards compatibility and do nothing (packages/cli/src/index.ts marks both [deprecated] No-op). # Interactive chat npm run dev:chat -- --provider openai --model gpt-5.4-mini npm run dev:chat -- --provider anthropic --model claude-sonnet-5 # Piped input (non-interactive) echo "research 5 AI topics" | npm run dev:chat -- --provider openai --model gpt-5.4-mini # Connect to running WebSocket server npm run dev:chat -- --url ws://localhost:7777/ws Chat flags: -p, --provider <name> anthropic, openai, gemini, xai, groq, mistral, deepseek, moonshot, minimax, cerebras, meta, alibaba, together, openrouter, requesty, opper, api_route, huggingface, replicate, kie, aki, ollama, lmstudio, claude_agent_sdk, codex, gmi, mlx, node_llama_cpp (any registry provider id also works, e.g. vllm, llama_cpp) -m, --model <id> Model ID (e.g. claude-sonnet-5, gpt-5.4-mini) -w, --workspace <path> Workspace directory for file tools --tools <list> Comma-separated tool names -u, --url <ws-url> Connect to WebSocket server instead of local provider --no-read-only-search Disable the read-only run_search fan-out primitive (on by default) --cost-cap <usd> Ceiling on provider spend for one turn, shared by every loop it starts; 0 lifts it. Default: NODETOOL_AGENT_TURN_COST_CAP_USD --timeout <s> Wall-clock bound on one turn, in seconds; 0 leaves it no time at all. Default: NODETOOL_AGENT_TURN_DEADLINE_MS -a, --agent [mode] [deprecated] No-op --no-agent [deprecated] No-op --cost-cap and --timeout override two of the five NODETOOL_AGENT_* settings a chat turn on the server reads; the other three (concurrency, total turns, unpriced-token ceiling) come from the settings alone. The budget is one object per turn, shared by every loop the turn starts — a sub-agent, an execute_plan DAG, an AgentNode reached through run_node — so a ceiling bounds the run rather than each loop. A turn a ceiling refuses says which one: Stopped: turn budget of $5 reached. nodetool agent run takes the same two flags, bounding the whole command instead of one turn, and exits non-zero on a budget stop. Interactive chat fills the terminal and restores the previous screen on exit. Page Up/Page Down scrolls history, Ctrl+G follows the latest output, and Ctrl+O shows tool details. Alt+Enter or Ctrl+J inserts a newline. Escape cancels a turn without releasing the next prompt until cleanup finishes. Interactive commands and session storage are documented in the chat CLI reference. Use --resume [id] to reopen a saved conversation, /sessions to choose one, and /export <path.md> to export the transcript. Local and remote approval requests appear in the terminal. The chat-terminal registry selfcheck drives the Ink UI with terminal streams and tests keyboard input, viewport bounds, sessions, approval cancellation, and WebSocket event handling without a model key. nodetool serve npm run dev:nodetool -- serve # Start on localhost:7777 npm run dev:nodetool -- serve --host 0.0.0.0 # Bind all interfaces npm run dev:nodetool -- serve --port 8080 # Custom port MCP bundle (.mcpb) for Claude Desktop npm run build:mcpb # → dist/nodetool.mcpb (runs an end-to-end smoke test) Builds a one-file MCP bundle that Claude Desktop (and other MCPB-aware agents) installs by drag-and-drop. The bundle is a stdio↔streamable-HTTP bridge (scripts/mcpb/bridge.mjs, packed by scripts/build-mcpb.mjs) that talks to a running NodeTool server’s /mcp endpoint — no native modules, so one artifact covers macOS/Windows/Linux. When the server isn’t running the bridge starts anyway in offline mode: it serves a nodetool_status tool with startup instructions, retries in the background, and hot-attaches (with list_changed notifications) when the server appears — including after a mid-session app restart. User config in the bundle: server URL (default http://127.0.0.1:7777/mcp) and an optional bearer token. For CLI agents (Claude Code, Codex) use nodetool mcp install instead. To reach a deployed server rather than a local one, the client points at /mcp with a token minted in Settings → MCP → Connect an agent remotely — see docs/mcp-production.md. Every release builds and attaches nodetool-<version>.mcpb to the GitHub Release (release.yaml, built once on Linux since the bundle is cross-platform). The desktop app ships the same bundle: the electron build runs prepare-mcpb and bundles nodetool.mcpb as an extra resource (electron-builder.json). Settings → MCP → Claude Desktop → Install Extension hands it to the OS (window.api.mcp.installBundle → MCP_INSTALL_BUNDLE IPC → electron/src/mcpBundle.ts), which opens Claude Desktop’s install dialog (falling back to reveal-in-folder when no handler is registered). The button is desktop-only — it’s hidden in the browser/remote UI. nodetool run (DSL Workflows) npm run dev:nodetool -- run workflow.ts # Run a TypeScript DSL file npm run dev:nodetool -- run workflow.ts --json # Output results as JSON Supervised runs (--supervise) --supervise puts an agent on the failure path: a node invocation that throws after its own error handling raises an escalation, and the agent answers with one verdict — retry, repair the output, skip the item, or fail. Without the flag no escalation is ever constructed and the run is unchanged. Available on nodetool run, nodetool workflows run, and nodetool debug (server surface). The flags configure ExecutionSessionOptions.supervisor — the one integration point every surface shares; no CLI code touches WorkflowRunner. npm run dev:nodetool -- workflows run <id> --supervise npm run dev:nodetool -- run workflow.ts --supervise --max-decisions 5 npm run dev:nodetool -- debug <id> --supervise --supervisor-cost-cap 0.25 npm run dev:nodetool -- workflows run <id> --supervise \ --supervisor-model openrouter/openai/gpt-5.4-mini --max-retries 1 --supervise Supervise this run (off unless passed) --max-decisions <n> Decisions allowed in the run (default 10) --max-retries <n> Retries per node invocation (default 2) --supervisor-cost-cap <usd> Ceiling on supervisor spend (default 0.50) --supervisor-model <provider/model> Default anthropic/claude-sonnet-4-6, or NODETOOL_SUPERVISOR_MODEL Each decision prints a ⛨ line as it happens and the run ends with a supervised summary (⛨ supervised: 2 skipped, 1 retried, 3 decisions, +$0.0200). With --json the decisions appear as interventions (run commands; nodetool run wraps them as {results, interventions}) or server.summary.interventions plus a server.supervised rollup (debug). It is the Intervention record from @nodetool-ai/protocol, which the editor surface consumes unchanged. Supervisor spend goes into the prediction ledger nodetool costs reads, one row per billable decision, attributed to the run and tagged supervisor in node_type. Every supervisor failure (timeout, unparseable verdict, exhausted budget, cancelled run) resolves as fail. Details: docs/workflow-supervisor-design.md. nodetool debug (Workflow Debug Harness) Runs a workflow end-to-end on the server (headless kernel WorkflowRunner) and optionally in a real browser (Playwright driving the e2e_runner harness), then writes a self-contained debug bundle and prints an agent-friendly verdict. Built for iterative troubleshooting: run → read the report → edit → re-run. The cheap server run (workflow JSON + all messages/logs/outputs/errors) is on by default. The expensive parts are opt-in flags: --browser (Playwright + Chromium), --trace (OpenTelemetry SDK + span overhead), --stages (a screenshot per run stage). # Server surface only (default) — accepts a workflow id, JSON file, or DSL .ts file npm run dev:nodetool -- debug <workflow_id> npm run dev:nodetool -- debug workflow.json --params '{"prompt":"hi"}' # Opt into the expensive parts: npm run dev:nodetool -- debug <id> --trace # OTel trace (timing/tokens/cost) npm run dev:nodetool -- debug <id> --browser # real-browser surface (Playwright) npm run dev:nodetool -- debug <id> --stages # per-stage screenshots (implies --browser) # Print the full machine-readable report to stdout for an agent to parse npm run dev:nodetool -- debug <workflow_id> --json npm run dev:nodetool -- debug <id> --no-server --browser # browser only npm run dev:nodetool -- debug <id> --out ./mydebug # custom bundle dir npm run dev:nodetool -- debug <id> --timeout 60000 # per-surface timeout (ms) npm run dev:nodetool -- debug workflow.json --watch # re-run on file change, print a verdict diff npm run dev:nodetool -- debug <id> --supervise # supervise the server surface (see above) The --watch flag (file targets only) re-runs after every save and prints just what changed since the last run — verdict ok/fail transitions, newly-appeared and resolved issues, and token/cost movement — so the edit→verify loop is a live diff instead of a fresh full report each time. The bundle (nodetool-debug/<id>-<ts>/ by default) contains: report.json # the full DebugReport (workflow JSON, both surfaces, verdict) report.md # human-readable summary workflow.json # the resolved graph (runner shape) server/messages.jsonl # every processing message (logs, node IO, outputs, errors) server/trace.jsonl # OpenTelemetry spans (timing, tokens, cost) — only with --trace browser/record.json # the browser RunRecord (events, logs, node IO, artifacts) — only with --browser browser/screenshot.png # canvas screenshot of the finished graph browser/stages/ # canvas screenshots at each stage — only with --stages browser/console-errors.log Agents can also debug a workflow on a running server via the debug_workflow tool. It posts to POST /api/workflows/:id/debug, which runs the workflow and returns the same execution summary and verdict the CLI harness computes — per-node status and errors, logs, LLM calls, outputs — plus the job record and the graph overview. The summary reducer and triage live in @nodetool-ai/execution/debug, so CLI and agent surfaces cannot drift. With interactive: true, run_workflow and debug_workflow put the calling agent on the failure path the way --supervise puts an LLM supervisor there: a failing node invocation parks the run and the tool returns the escalation (status: "escalated" with the supervisor’s Escalation record — redacted inputs, error detail, allowedActions). The agent answers via resolve_workflow_escalation — retry, substitute, skip, end_stream, or fail, kernel-enforced against the allowed set — and gets back either the next escalation or the run’s final report. HTTP surface: POST /api/workflows/:id/run|debug {interactive: true} plus GET/POST /api/debug/sessions/:id[/verdict|/cancel] (packages/execution/src/service/debug-sessions.ts). Escalations the agent leaves unanswered fail closed on the decision timeout (default 10 min). The browser surface is exposed in web/ as npm run test:debug-harness (env: NODETOOL_DEBUG_GRAPH, NODETOOL_DEBUG_OUT, NODETOOL_DEBUG_PARAMS). nodetool app debug (App-Builder Debug Harness) Runs a mini app headlessly for agent debugging: validates every widget binding against the workflow’s inputs/outputs/variables, simulates the app the way the web runtime does (seed input defaults, apply params, click the Run button or a scripted interaction sequence), executes the workflow on the kernel runner, folds the streamed messages into the app’s reactive values, and reports each widget’s final state plus a verdict. Three target kinds, all producing the same report: an application id (read straight from the applications table, no server), an ApplicationBundle JSON file (the app plus the full graphs of the workflows it binds — operations reference bundle keys, so it runs without touching the database), and — legacy — a workflow id or workflow JSON file carrying graph + app_doc, whose document is lifted onto the host workflow. npm run dev:nodetool -- app debug <application_id> npm run dev:nodetool -- app debug my.app.json # ApplicationBundle file npm run dev:nodetool -- app debug workflow.json --params '{"prompt":"hi"}' npm run dev:nodetool -- app debug <id> --no-run # static wiring check only npm run dev:nodetool -- app debug <id> --json # full AppDebugReport for agents # Scripted interactions: set values, change inputs, click widgets (by # component id, unique type, or unique label), and run or cancel an # operation by id npm run dev:nodetool -- app debug <id> --interact \ '[{"set":{"key":"prompt","value":"hi"}},{"click":"Button-1"}]' npm run dev:nodetool -- app debug <id> --interact \ '[{"set":{"key":"tone","value":"terse","operationId":"draft"}},{"run":"draft"},{"cancel":"draft"}]' The harness runs every declared operation, not just the first: each resolves its own graph, and state is keyed per operation. The verdict catches app-level failures a workflow-only run can’t: bindings that reference missing inputs/outputs/variables, apps with no run trigger, and display widgets that never receive a value from a completed run. It also catches what the operation/variable layer makes mis-configurable — an output mapped to an undeclared variable, a mapping keyed on a node the workflow lacks, an event naming an operation the document never declares, a widget showing execution state of an operation nothing can run, and an elapsed timeoutMs. A persist: true variable that is instance-scoped warns rather than being silently downgraded. Three warnings cover the app an agent builds that wires up correctly and still fails a user. A run button with no disabledWhen on op:<id>/exec#running is a race the user drives — no policy refuses the second click, so it cancels the job and restarts it (replace), stacks another run (queue), or starts one alongside (parallel). An operation nothing binds exec#error to fails invisibly: the app looks idle and says nothing. And a media input widget — Image, Sketch Pad, Camera Capture, Audio, Audio Recorder, Video, Document — filling an input with no default, behind a run trigger guarded on nothing, lets the run start with the input unset, which is a paid call the user did not get to fill in. A Workflow Form is not one of those: it renders every input of its operation rather than binding one, so there is no single binding to guard. The bundle (nodetool-debug/app-<id>-<ts>/) contains report.json/report.md, app.json (the app document), workflow.json, and server/run-N.messages.jsonl per triggered run. The report carries final variable values, the activity label stream, and each invocation’s policy decision, so an agent can see why a run was replaced, queued, or timed out. Simulator code: packages/execution/src/app-debug/ (@nodetool-ai/execution/app-debug), so every host — the CLI, the agent build loop, the server — simulates an app the same way. The CLI keeps target resolution and bundle writing in packages/cli/src/app-debug/. Conditions and formatting are simulated: after every fold the harness evaluates each widget’s visibleWhen/disabledWhen, a click or change on a widget that is hidden or disabled fails the step and names the condition, a run trigger whose condition never held is an error, and a widget with a format template reports what the template renders. Resource collections come from an in-memory provider the script seeds — {"seedResource":{"id":"<binding>","items":[…]}} as an interaction step, or a resource:<binding> key in --params. A from: "resource" input then resolves through it, resource widgets report their collection in the report, and a resourceCommand mutates it; running an operation whose input reads an unseeded binding fails and says how to seed it. Not simulated headlessly (the report lists this too, under notSimulated): layout, styling, focus, and scroll; and the stored collections themselves — a run never reads the database, and openResource has no editor to open. The shipped example apps are curated ApplicationBundle files in packages/base-nodes/nodetool/examples/apps/, built from the spec in scripts/example-apps/apps.mjs by node scripts/build-example-apps.mjs. The build resolves every workflow, input, and output by name against the shipped template graphs, validates each bundle with nodetool app debug --no-run, and writes the preview bundles in web/public/app-preview/. Example workflows carry no app_doc. --regen -p <provider> -m <model> answers a different question — would nodetool app build produce these apps today? It derives a BuildSpec from each shipped bundle, builds it, and prints the drift (operations, variables, and widgets compared by what they show, not by their ids, so two builds of one app differ only where they really differ). It writes nothing: the curated bundles stay hand-approved, and drift between two model runs is a signal to read, not a patch to apply. Add --app <slug> for one app. The server lists them at GET /api/applications/examples and installs one with POST /api/applications/examples/:slug/install, which goes through the normal bundle import. Marketing screenshots come from web/scripts/screenshot-app-previews.mjs (renders web/app-preview.html headlessly → marketing/public/apps/<slug>.png). Marketing chat screenshots The chat panel on the marketing site is shot the same way — from the real UI, not mocked up. npm run chat-shots (in web/) builds the stills the casts embed, then replays each cast in web/src/demo/chat/marketing/ through web/demo.html?chat=<id>&t=<ms>&bare=1 and writes marketing/public/chat/<id>.webp plus the size manifest the gallery’s next/image needs (marketing/src/data/chatShots.generated.ts). The casts are authored rather than recorded, and the run blocks tRPC, so a re-run reproduces the same frames on any machine with no backend, no model call, and no credits spent. Changing the prose in a cast changes how the answer wraps, so re-run the script rather than editing a .webp: the height of each shot is measured from the rendered thread, not declared. The storyboard surface loop replays the same SCRAPHEART board through the real StoryboardBoard (web/src/demo/doc/storyboardAssistantCast.ts), with its keyframes inlined by web/scripts/build-storyboard-cast-stills.mjs. The board is a shot grid, so six cards are two rows and the surface is shorter than a 1080-line frame: the loop lays it out in a frame 1.5× smaller and scales it back up (zoom in demo/src/hero/SurfaceLoop.tsx) rather than leaving a third of the frame empty. panPx, which scrolled the taller list the board used to be, is still there for a cast whose surface overflows. Re-render with npm run render:surfaces in demo/, then encode marketing/public/surface-storyboard.{mp4,webm} and its -poster.webp. The storyboard’s still frames come from web/scripts/screenshot-trailer-surfaces.mjs, which replays the same cast through Playwright: trailer-storyboard.webp for the movie-trailer use-case page, and one hero written to two trees — marketing/public/screen_storyboard.png (landing page, /creatives, the Product Hunt slide) and docs/assets/creative-agent/storyboard-surface.png. Both are backend-free, so a re-run reproduces them with no server and no credits spent. Marketing project screenshots The landing page’s project section (marketing/src/components/ProjectSection.tsx) tells one session end to end, and three of its four frames are the project views. Those come from the documentation screenshot suite — npm run screenshots in web/ drives the real app against the seeded projects in packages/websocket/src/screenshot-projects.ts and writes docs/assets/screenshots/project-*.png. npm run project-shots (in web/) then re-encodes those PNGs as WebP into marketing/public/projects/ and writes marketing/src/data/projectShots.generated.ts with the size next/image needs. The two halves are split because only the first needs a browser, a backend and a seeded database; re-encoding is deterministic and needs none of them. That is what npm run project-shots:check reads: it re-encodes into memory and fails when a committed WebP or a recorded size has drifted from its source PNG, so a stale marketing copy cannot ship unnoticed. A shot may declare a height to crop to its top band (the new-project surface pins its blank-document strip to the bottom of the viewport, leaving 270px of empty column in the middle at 900px) — a crop and nothing else: no compositing, no re-rendering. The fourth frame is the chat one, and it is the same session on purpose: the six keyframes rendered in it are the six stills on the board the project frames show. nodetool app build (Mini-App Build Harness) Turns a prompt — or a hand-written spec.json — into a verified ApplicationBundle, without touching the database. Six stages run in order: spec pins what the app must do, plan builds one workflow per operation with authorGraph (or binds one you pin), author drives the real ui_app_* tool contract to place and wire the widgets, check validates the app’s wiring against those graphs, run replays every interaction on the kernel and asserts what each widget ends up showing, and judge asks a model whether each interaction achieved what was asked — the one question a structural check cannot answer. The judge sees only a Check+Run-green app, one call per interaction, given the spec’s intent, the steps, and the widget states they left behind. A verdict of not-achieved becomes the round’s complaint and routes to the Author with the judge’s reasons. It fails closed: a judge that times out, errors, or answers with something unparseable scores that interaction as not achieved. Its model is configured apart from the builder’s (--judge-model, NODETOOL_APP_JUDGE_MODEL), defaulting to a configured model the builder did not use, because a model grading its own work is the weakest reviewer available; report.judge.model records which one ran. --no-judge skips the stage, and the verdict’s notSimulated then says nothing scored the app. Everything wrong at the end of a pass becomes one complaint, and the next round edits the document rather than rebuilding it. The loop fails closed: a budget that runs out, an issue that reappears after being fixed, or a cancelled signal ends the build as failed with the reason named — there is no bundle behind a failed verdict. npm run dev:nodetool -- app build "an app that drafts a note from a prompt" -p anthropic -m claude-sonnet-5 npm run dev:nodetool -- app build spec.json -p openai -m gpt-5.4-mini --json npm run dev:nodetool -- app build "..." -p anthropic -m claude-sonnet-5 --workflow <id> # bind, never plan npm run dev:nodetool -- app build "..." -p anthropic -m claude-sonnet-5 --max-repairs 1 --cost-cap 1.00 npm run dev:nodetool -- app build "..." -p anthropic -m claude-sonnet-5 --judge-model openai/gpt-5.4-mini npm run dev:nodetool -- app build spec.json -p anthropic -m claude-sonnet-5 --no-judge # structural only npm run dev:nodetool -- app build "..." -p anthropic -m claude-sonnet-5 --supervise npm run dev:nodetool -- app build spec.json -p anthropic -m claude-sonnet-5 --watch -p, --provider <name> -m, --model <id> builder provider/model (required) --judge-model <provider/model> judge model (env NODETOOL_APP_JUDGE_MODEL; default: a configured model ≠ the builder's) --workflow <id> pin an existing workflow (repeatable, operation order) --max-repairs <n> --cost-cap <usd> --timeout <ms> --out <dir> --json --no-judge --watch --supervise --max-decisions <n> --max-retries <n> --supervisor-cost-cap <usd> --supervisor-model <provider/model> --supervise and its four bounds are the same flags nodetool run, workflows run, and debug carry, with the same defaults (env: NODETOOL_SUPERVISOR_MODEL); see Supervised runs above. They apply to the Run stage, whose interactions execute on the kernel — buildApp itself is never supervised. Each decision lands in that interaction’s run report and rolls up into report.supervision (the Intervention records plus the run summary), and the CLI prints the usual ⛨ lines. A supervised run’s shape is a decision rather than a defect: once the supervisor has skipped or repaired something, what the run produced less of is recorded as a warning instead of an issue the Author is asked to repair. The interaction’s expectations stay errors — supervision does not excuse the contract the spec pinned. --watch (spec-file targets only) re-builds after every save and prints just what changed since the last build — verdict ok/fail transitions, the stage it ended on, issues that appeared and resolved, and cost movement. It reuses debug --watch’s differ, so both harnesses read the same. The bundle directory stays at nodetool-debug/app-build-<slug>-watch so each re-build overwrites the last. A build is a model run: every save spends money. The bundle (nodetool-debug/app-build-<slug>-<ts>/) holds report.json (the BuildReport), report.md, spec.json, app.bundle.json (the deliverable, written only for a green build), and interactions/<name>/run-N.messages.jsonl per replayed run. Exit code 0 only when verdict.ok. Build spend lands in the prediction ledger nodetool costs reads, one row per stage, tagged app-build. Harness code: packages/agents/src/app-build/ (buildApp, the spec/author/judge stages, the ui_app_* bridge the app-tools eval also scores); the CLI keeps the flags and the bundle. Design: docs/mini-app-build-harness-design.md. On the server: POST /api/applications/build The same buildApp runs on the server: POST /api/applications/build {prompt | spec, provider, model, workflow_ids, max_repairs, cost_cap_usd, timeout_ms} returns the BuildReport. Provider and model come from the body, and the server falls back to NODETOOL_APP_BUILD_PROVIDER / NODETOOL_APP_BUILD_MODEL. The cost cap defaults to the harness’s own $2. There is no build_app agent tool. An agent builds an app the way a person does — declare the operations, place the widgets, and grade every change with debug_app / ui_app_debug — instead of handing the job to a second agent it cannot see into. The route stays for the CLI, the eval suite, and a caller that wants the batch build. Off the browser that path runs through create_app and edit_app. edit_app takes [{tool, input}, …] naming the same ui_app_* tools the Puck editor exposes, replays them against the saved document through app-build/bridge.ts — the headless twin the Author stage and the app-tools eval already drive — and saves once, CAS on updated_at. Call it with no steps to get the tool catalog and the app’s current state. The tools themselves stay in one implementation, so the browser and the headless path cannot drift. A build runs for minutes, so poll: true returns a session id immediately and the caller reads GET /api/debug/sessions/:id until it settles, or cancels with POST /api/debug/sessions/:id/cancel — the same session machinery an interactive debug_workflow run uses (packages/execution/src/service/debug-sessions.ts). A cancelled build settles as failed with reason: "cancelled". The bundle behind a green verdict is offered, never installed: it becomes an application through the normal POST /api/applications/import-bundle. Server code: packages/agents/src/app-build/build-service.ts. nodetool validate (Static Workflow Check) Checks a workflow against the node registry without running it — unknown node types, missing required properties, unselected models, model properties naming an unregistered provider or a model id that provider does not offer, dangling and mis-typed edges, dynamic slots typed with a JSON-Schema/TypeScript name instead of NodeTool’s (integer → int), DSL wiring handles left in a property bag (a connection that was never made), and Code node bodies. On a DB-id target, where the store is reachable, it also warns about declared credentials (required_settings, a Code node’s secrets) this install cannot resolve. Returns in well under a second, so it’s the cheap pre-flight before an expensive debug run. Accepts a workflow id, JSON file, or DSL .ts file. File/DSL targets need no database. Model references are found wherever they sit — a top-level property, an entry in a list[…_model], one nested in a settings object, or a dynamic slot value. Both catalogs fail toward silence: an empty provider list means the registry could not be reached, and a catalog only enumerable over the network (Anthropic, Ollama, ASR ids anywhere) reports nothing rather than calling a real id a typo. The check runs at graph creation time too — validate_workflow sits on the authoring agent’s belt, create_workflow refuses to save a graph whose provider or model the model hallucinated, and one whose model properties are left unselected (nothing stamps models in at run time, so every Agent node would die on “Select a model”) — and POST /api/workflows/:id/run|debug refuses the run with a 400 before the job row exists, instead of failing on the model node after the upstream half of the graph has been paid for. The same refusal covers credentials: a run whose selected providers have no resolvable key (secret store, then env) is refused with 400 naming each missing secret. A nodetool.code.Code node’s code is parsed, not just stored: a body that is not valid JavaScript, uses export at the top level, imports a specifier no installed pack serves (a node declares no packages — its imports are the declaration, resolved against the catalog), reads a bare name that is not a sandbox API — including one of the node’s own inputs, which arrive on the inputs object, so a bare read is a ReferenceError too — never returns, or leaves a declared output unset on some return path is reported against the node. A named inputs.<name> read or stream("name") / emit("name") call is not an error: the validator, the editor and the graph tools all count it as a declared handle. The analysis lives in @nodetool-ai/node-sdk (code-analysis.ts, code-node-validation.ts), so the graph validator, the submit_code planner and the editor read one AST. npm run dev:nodetool -- validate <workflow_id> npm run dev:nodetool -- validate workflow.json npm run dev:nodetool -- validate workflow.json --json # machine-readable report npm run dev:nodetool -- validate <id> --warnings-as-errors # exit non-zero on warnings too The same check is exposed to agents through the validate_workflow tool: pass an inline graph ({nodes, edges}) to check a graph being built, or a workflow_id to fetch and validate a saved one. The validator core is validateGraph in @nodetool-ai/node-sdk. The credential warning reaches that tool too, on a graph as well as a saved workflow. The run answers which of the declared names this install holds (CapabilityRun.availableSecrets, built from the context by contextSecretAvailability), and the issue tells the agent where a person sets one — plus request_secret where the run can raise that dialog, and never on a headless run, where the call fails closed. A run with no reachable store carries no callback and the check is skipped: nothing could answer, and reporting every declared key as absent would warn on every graph. The hosts that inject are audited by packages/agents/tests/capability-run-secrets-audit.test.ts, which records the runs that deliberately omit it and how many calls each is allowed. nodetool timeline validate / debug (Timeline Harness) Checks a timeline sequence without rendering it, and replays a scripted edit session against it. The target is a timeline JSON file — a bare TimelineDocument or anything carrying one under document, so a timeline.get tRPC response works as-is — or a timeline_sequences row id. A path that exists on disk wins over an id. npm run dev:nodetool -- timeline validate <timeline_id> npm run dev:nodetool -- timeline validate sequence.json --json npm run dev:nodetool -- timeline validate <id> --warnings-as-errors npm run dev:nodetool -- timeline debug sequence.json \ --interact '[{"tool":"add_track","input":{"type":"audio","name":"Music"}}, {"tool":"animate_clip","input":{"target":"shot","animations":[{"role":"in","preset":"fade"}]}}]' npm run dev:nodetool -- timeline debug <id> --out ./mydebug --json validate reads what a headless check can decide: a clip on a track the document does not have, a field the schema round trip would strip, an animation preset that does not exist, baked curves a custom animation could not render from, timings that cannot render. debug runs the same check, then executes each --interact step against the headless ui_timeline_* bridge — the one the timeline-tools eval drives — and validates the document the session left behind. A step names a tool with or without the ui_timeline_ prefix; a failing step is recorded and the script continues, so one bad target does not hide everything after it. Rendering, playback, decode, and generation are not simulated; the report lists that under notSimulated. Database-backed edit_timeline also runs media adapters. get_clip_frames uses the same contract as ui_timeline_get_clip_frames: it decodes source video frames or renders one text, shape or 3D clip. Frame inspection leaves the saved document unchanged. Check each returned frame’s complete flag before treating it as visual evidence. generatively_edit_clip requires a video-to-video provider/model pair headlessly. It submits through the generation lifecycle, stores its captured source and unselected timeline destination, and appends the persisted result as an inactive take. A document retry reuses the submission. Durable recovery attaches completed output with the same take id. apply_take accepts the candidate explicitly and returns undo_version; restore that version with restore_timeline_version to undo the acceptance. The same static check is exposed to agents through the validate_timeline tool: pass an inline document to check a timeline being built, or a timeline_id to validate a saved sequence (scoped to the requesting user). The timeline assistant is told to call it after edits, before the user renders. The bundle (nodetool-debug/timeline-<id>-<ts>/) holds report.json, report.md, and timeline.json (the input document). Exit code 0 only when the verdict is ok. Validation and report rules live in @nodetool-ai/execution/timeline-debug; the CLI keeps target resolution, the interaction script, and the bundle. nodetool timeline render (GPU Frame Render) Renders a timeline through the GPU compositor behind the Render Timeline node, from the same targets as validate. --stills writes selected frames as PNGs named by timeline index, which is how an export is checked frame by frame: preview_timeline_frame draws through Canvas 2D and can differ from the GPU render. --sheet [columns] combines selected frames into one labelled PNG in place of loose stills. --only <names-or-ids> isolates clips while keeping their ancestor groups and matte sources. --frames coverage:<file> selects integer frame indices from the Frame column of a builder’s leading comment and reports skipped rows. --scale makes a fast draft with the full-size layout. npm run dev:nodetool -- timeline render sequence.json --stills --frames 175,290,376-384 --scale 0.5 npm run dev:nodetool -- timeline render sequence.json --only title --frames coverage:build-sequence.ts --sheet --out review.png npm run dev:nodetool -- timeline render <timeline_id> --out ad.mp4 JSON results include complete and clip/time diagnostics for missing required content. Incomplete final exports retain the artifact and exit with status 1. Stills and contact sheets retain diagnostics for review and exit with status 0. Video exports mix audio-track clips, MIDI clips, and embedded video audio at their timeline offsets. Stills, contact sheets, PNG sequences, and video frame selections carry no audio. Flags are in CLI Reference. nodetool timeline score (Timeline Craft Scorecard) Deterministic, no-LLM craft scorecard for a timeline document, from the same targets as validate. It exists to make “does an agent-authored timeline approach the density and polish of the shipped showcase examples” a repeatable measurement instead of a by-hand agent run + timeline render --sheet + eyeballing session, so an agent-harness change can be scored against a fixed baseline. The core (packages/execution/src/timeline-debug/score.ts, scoreTimelineCraft) computes nine metrics — animations and keyframes per visible second, distinct animated properties, distinct effect types, showcase-only feature flags used (repeater, motionBlur, layout, blendMode, mask, animationLinks, temporalEcho), style tracks, text animators, authored transitions, and scene count (distinct clip start-time boundaries) — against the reference band: the same metrics computed at runtime for the shipped examples (cadence, kite, prism, serein, tidewater, voltra; t-minus-30 is footage-based and excluded) via @nodetool-ai/timeline/examples/node, the loader the product installs and previews them from, so the band never drifts from a second copy of the set. Each metric earns 0.4 credit outright, plus up to 0.6 more for reaching the reference median (capped at 1.0 — exceeding the median earns nothing further). The score is the mean credit across metrics, as a percentage, minus 5 points per distinct showcase-tier warning code the document carries (validateTimelineSequence(doc, { tier: "showcase" })’s showcase_* codes — new codes land here automatically, since the check runs, not a copy of it). Distinct, not per instance: four showcase_text_collision findings on four clip pairs are one craft gap, not four — penalizing by instance count would let one noisy, high-cardinality check dominate the score. showcaseWarnings still lists every instance; showcaseWarningCodeCounts breaks that list down by code (e.g. showcase_text_collision ×4), so volume stays visible even though the penalty is per code. The floor keeps one missing technique from cratering an otherwise dense, polished document; the median cap keeps a bloated document from buying an unbounded score by piling on one metric. npm run dev:nodetool -- timeline score <timeline_id> npm run dev:nodetool -- timeline score sequence.json --json npm run dev:nodetool -- timeline score <timeline_id> --min-score 70 npm run dev:nodetool -- timeline score <timeline_id> --sheet review-sheet.png --min-score <n> is the only way this command fails: without it, the score is information and the exit code is always 0. --sheet <out.png> renders a GPU contact sheet of 12 evenly spaced frames through runTimelineRender — the same code path timeline render --sheet uses, never a second renderer. Flags are in CLI Reference. scripts/motion-craft-eval.mjs is the eval loop built on top: it runs nodetool agent run against a motion-graphics brief (default: a 20-second launch spot for a fictional app, dense kinetic type/shapes/data/transitions), captures the NDJSON event stream to nodetool-debug/motion-craft/<timestamp>/events.jsonl, extracts the timeline the run settled on (the last set_timeline_document that resolved successfully — direct tool call or nested inside an execute_code sandbox action — falling back to the last one at all, then the last create_timeline), scores it, and writes report.md (brief, tool-call counts, errors, the scorecard, the contact-sheet path, the agent’s final answer). --runs N repeats the loop and reports mean/min/max score. Running it end to end costs a real model call and up to 30 minutes, so it is not part of any automated check; only its event-parsing unit (parseTimelineRunEvents) is covered, against a trimmed extract of a real recorded run (scripts/__tests__/fixtures/motion-craft-eval-events.sample.jsonl). node scripts/motion-craft-eval.mjs node scripts/motion-craft-eval.mjs --runs 3 -p claude_agent_sdk -m opus node scripts/motion-craft-eval.mjs --brief-file brief.txt --max-iterations 40 --timeout 900 3D clips in preview_timeline_frame preview_timeline_frame composites a sequence at one or more timecodes on @napi-rs/canvas and returns the frames with a per-layer report. A model3d layer is the one kind that path cannot draw itself — there is no WebGL behind that canvas. So the pass collects every 3D layer of every requested instant before compositing anything, groups them by the glTF plus the options that fix a renderer (lighting, intensity, background, animation selection), and hands each group to one headless Chromium page that loads the model once and returns one PNG per frame. Grouping is what makes it affordable: a launch costs about a second, so six timecodes of a turntable are one launch and six frames, not six launches. No Chrome on the host — or a launch that fails — is not an error the call throws. The group’s layers are left out and reported as a model3d_unavailable degradation naming the clip. The result, frame and contact sheet report complete: false when content is skipped or approximated. Frames expose degradations alongside the existing degraded field. Inspect these fields before judging the image. An incomplete preview cannot verify the composition. visual_output_verified stays false because rendering or comparing pixels does not judge whether the visual result meets the brief. compare_timeline_frames returns null scores for incomplete pairs and null aggregate metrics if either preview is incomplete, so two frames missing the same layer cannot establish visual equivalence. Two fields on a 3D layer’s report say what the pixels do not: camera — the pose the layer was drawn with (mode, azimuth, elevation, fov, zoom, and the glTF camera’s name in scene mode), the clip’s authored camera already folded with the four animated camera channels. A turntable mid-sweep and a static pose are the same picture at one instant. animation_time_sec — the glTF animation’s own clock: the clip’s source time times animation.speed. It is not the timeline time, so a trimmed or retimed clip’s animation is readable from the report. A clip whose bake is still current draws as a video layer instead, which the decoder seeks by clip-local time; a hash that no longer matches puts the live 3D layer back. The pre-pass is packages/agents/src/timeline-preview/model3d.ts, its suite packages/agents/tests/timeline-model3d-frames.test.ts, and the renderer it calls is renderGlbFramesHeadless (packages/video-nodes/src/nodes/model3d/render3d-headless.ts), which the single-frame RenderToImage path also goes through. A diff under either package runs both suites through harness gate (the timeline-model3d harness). The Blender side has a coverage gap. The sampled producer behind a bake is exercised by packages/blender-nodes/tests/render-animation-sampled.test.ts, which runs only where Blender is installed — BLENDER_PATH, or blender on PATH — and skips everywhere else, CI included: CI installs no Blender because EEVEE timed out on its GPU-less runners. It therefore runs only where a developer has Blender. Two defects in that producer surfaced only when CI still ran it: a named animation that selected nothing because the glTF importer’s active action played over the muted NLA tracks, and RGB output where render_image and render_passes both write RGBA, which made a bake frame incomparable to a still. Audio-driven timeline motion bake_audio_animation measures a clip’s audio (an envelope follower, or one pulse per detected onset in mode: "beats") and writes the result as a source-anchored custom animation on another clip’s property — commonly scale, so a title or a shape pulses with the beat. The write is additive: a clip can carry a hand-authored preset (a pop entrance, a slide) alongside the bake, and the two compose the way any two animations driving the same channel do — scale: "multiply" in ANIMATED_PROPERTY_FOLD (packages/timeline/src/animation/types.ts), so the resolved scale at any instant is the product of both, not one replacing the other. A re-bake with new settings replaces only the curve it produced before (custom.bakedFrom.kind), never a hand-keyframed animation. packages/agents/tests/timeline-audio-drive-demo.test.ts is the product demonstration: a 4-second music bed with four kick bursts drives a “DROP” text clip’s scale in beats mode, on a clip that also carries a 600ms pop entrance. It asserts on the saved document (exactly one baked animation next to the untouched pop, one peak per kick), on motion (resolveAnimatedLayerProps sampled at each kick and between them, checking the resolved scale equals the product of the pop’s own sample and the baked curve’s own sample), and on pixels (preview_timeline_frame’s rendering path draws the title measurably larger on a kick than between kicks). capabilities-bake-audio-animation.test.ts pins the arithmetic itself — the two clock hops from audio-source time through timeline time to the target clip’s own source time — with a synthesized WAV and no network. Generated mattes isolate_subject cuts the subject out of a video clip and carries the mask on that clip as generatedMatte (D2), so a trim, a split or a speed change keeps the cutout aligned with no second clip to sync. It sends the clip’s whole source asset to fal-ai/birefnet/v2/video with output_mask: true, downloads the grayscale mask through fetchExternalMedia — the media-egress door, https to a public host with every redirect hop re-checked — and stores it as an asset; the mask is read back at the clip’s own source time, which is why a submission trimmed to the clip’s window would need an offset the document does not carry. A clip that already has a current cutout is handed it back with reused: true and nothing is spent; a matte whose source asset was replaced under it is stale and is regenerated instead. The knobs on the result — invert, strength, feather, and putting an earlier version back — are the free set_generated_matte op, which both the headless bridge and applyTimelineOp run through the same helpers in packages/timeline/src/generatedMatte.ts. The provider is never called in a check. isolateSubjectOnClip (packages/agents/src/capabilities/timeline-isolate-subject.ts) takes its runner, its mask store and its persist step as dependencies, so capabilities-isolate-subject.test.ts drives the whole state machine with a fake runner and asserts the document: the in-flight write keeps the previous asset visible, a finished run pushes the displaced result onto versions and keeps the user’s own knobs, and a failed regenerate leaves the previous ready matte selected rather than costing the user a working cutout. The HTTP door the inspector posts to is POST /api/timelines/:id/isolate-subject (timeline-isolate-subject-route.test.ts), which is the capability’s inputs minus timeline_id. nodetool timeline versions (Timeline Version History) timeline versions reads and writes a sequence’s snapshot history against the local database — manual saves, the autosaves timeline.update writes at most every five minutes, and the pre-restore snapshot that makes a restore undoable. All five subcommands take --json. npm run dev:nodetool -- timeline versions list <timeline_id> --save-type manual --limit 10 npm run dev:nodetool -- timeline versions show <timeline_id> 3 --json npm run dev:nodetool -- timeline versions create <timeline_id> --name "before the recut" npm run dev:nodetool -- timeline versions restore <timeline_id> 3 npm run dev:nodetool -- timeline versions delete <timeline_id> 3 --yes restore mirrors the tRPC router: it snapshots the current state as a restore version, CAS-writes the old document and its render settings back onto the sequence, then runs the same static check timeline validate runs. An old document is restored against today’s schema, so what it used to pass is not what it passes now — a restore whose document no longer validates exits non-zero and prints the issues. Agents get the same history headlessly: list_timelines, list_timeline_versions, get_timeline_version (read one snapshot’s document without restoring), create_timeline_version (manual snapshot), delete_timeline_version, and restore_timeline_version, which snapshots the pre-restore state first and returns the post-restore validation. None of them needs an open editor or a running server. nodetool sketch validate / debug (Sketch Harness) Checks a sketch (image document) without opening an editor, and replays a scripted edit session against it. The target is an image document JSON file — a bare {sketch, layerBindings} object or anything carrying one, so a sketch.get response or an image_documents row works as-is — or an image_documents row id. A path that exists on disk wins over an id. npm run dev:nodetool -- sketch validate <image_document_id> npm run dev:nodetool -- sketch validate sketch.json --json npm run dev:nodetool -- sketch validate <id> --warnings-as-errors npm run dev:nodetool -- sketch debug sketch.json \ --interact '[{"tool":"add_layer","input":{"name":"Shadow"}}, {"tool":"set_layer_props","input":{"target":"Shadow","opacity":0.4,"blendMode":"multiply"}}]' npm run dev:nodetool -- sketch debug <id> --out ./mydebug --json validate reads what a headless check can decide: a duplicate layer id, an activeLayerId or binding pointing at a layer the document lacks, opacity or a blend mode no compositor ships, a binding with no workflow or prompt behind it, and fields a schema round trip would strip. debug runs the same check, then executes each --interact step against the headless ui_sketch_* bridge — the one the sketch-tools eval drives — and validates the document the session left behind. A failing step is recorded and the script continues. The replay bridge paints new strokes on Skia canvases, but does not load the saved layer bitmaps, run generation providers or resolve placed image bytes. The report validates structure and always returns visual_output_verified: false. Its notSimulated list records these limits. Media tool results and eval final state also return this flag. A successful replay does not verify media output. The same static check is exposed to agents through the validate_sketch tool: pass an inline document to check a sketch being built, or an image_document_id to validate a saved one (scoped to the requesting user). edit_sketch places an image on a layer as well as editing the layer stack: set_layer_image (or image on add_layer) points a layer at an asset id, an asset:// locator, a data: URL or an http(s) URL, and the editor resolves and draws it on load — the same reference a sketch seeded from an asset carries, so nothing inlines a bitmap into the document. An asset id that resolves to nothing is refused rather than stored, because a stored one shows up as an empty layer. With an editor open, ui_sketch_place_image does the same against the live canvas. Agents also get the version history headlessly: list_sketches, create_sketch (a blank canvas, then edit_sketch), list_sketch_versions, get_sketch_version (read one snapshot’s document without restoring), create_sketch_version (manual snapshot), delete_sketch_version, and restore_sketch_version, which snapshots the pre-restore state first and returns the post-restore validation. The bundle (nodetool-debug/sketch-<id>-<ts>/) holds report.json, report.md, and sketch.json (the input document). Exit code 0 only when the verdict is ok. Validation and report rules live in @nodetool-ai/execution/sketch-debug; the CLI keeps target resolution, the interaction script, and the bundle. nodetool sketch versions (Sketch Version History) sketch versions reads and writes an image document’s snapshot history against the local database — manual saves, the autosaves sketch.update writes at most every five minutes, and the pre-restore snapshot that makes a restore undoable. The per-layer generation takes (sketch.versions.* in the tRPC router) are a different thing: those record one generated image on one layer, these snapshot the whole document. All five subcommands take --json. npm run dev:nodetool -- sketch versions list <image_document_id> --save-type manual --limit 10 npm run dev:nodetool -- sketch versions show <image_document_id> 3 --json npm run dev:nodetool -- sketch versions create <image_document_id> --name "before the repaint" npm run dev:nodetool -- sketch versions restore <image_document_id> 3 npm run dev:nodetool -- sketch versions delete <image_document_id> 3 --yes restore mirrors the tRPC router (sketch.documentVersions.restore): it snapshots the current state as a restore version, CAS-writes the old document and its canvas settings back onto the image document, then runs the same static check sketch validate runs. An old document is restored against today’s schema, so what it used to pass is not what it passes now — a restore whose document no longer validates exits non-zero and prints the issues. Layer bitmaps stay opaque to that check. nodetool jsscript (JS Script Harness) A JS script is a named, versioned script document — a body plus declared ports, secrets, a timeout, and saved test cases (docs/js-script-document-design.md). The target of every command is a script JSON file (a bare JsScriptDocument or anything carrying one under document) or a js_scripts row id. A path that exists on disk wins over an id; file targets need no database. npm run dev:nodetool -- jsscript validate <id|file.json> [--json] [--warnings-as-errors] npm run dev:nodetool -- jsscript run <id|file.json> --inputs '{"numbers":[1,2,3]}' npm run dev:nodetool -- jsscript run <id|file.json> --input-streams '{"numbers":[1,2,3]}' npm run dev:nodetool -- jsscript test <id|file.json> --json npm run dev:nodetool -- jsscript debug <id|file.json> \ --interact '[{"tool":"set_code","input":{"code":"await output(\"n\", 1);"}}]' npm run dev:nodetool -- jsscript versions list|show|create|restore|delete <id> validate reads what a headless check can decide: the body’s syntax, imports against the installed catalog (a script has no packages setting), undefined names, undeclared inputs.* reads, outputs no emit/output call reaches, duplicate or non-identifier port names, and tests naming ports the script does not declare. A body that declares outputs and returns them instead of emitting them is an error — a script has no legacy return contract. Zero saved tests and a declared secret this install lacks are warnings. run executes the body once in the QuickJS sandbox. A body that reads its inputs with stream is fed with --input-streams '{handle: [item, …]}' instead of --inputs; a staged handle the script does not declare is refused. test runs the document’s own saved cases (which stage their own items in inputStreams), grades them the way test_code grades a case list, and exits non-zero on any failure — the keyless selfcheck the harness gate runs, against packages/cli/tests/fixtures/js-script-sum.json, js-script-running-total.json and js-script-timeline-pack.json, which imports a shipped sandbox pack. run and test open the database and install the sandbox pack catalog, so a body can import packs and save through nodetool.*. debug replays each --interact step against the headless ui_jsscript_* bridge (tool names with or without the prefix; a failing step is recorded and the script continues), validates the document the session left behind, and writes nodetool-debug/jsscript-<id>-<ts>/ with report.json, report.md and jsscript.json. versions restore snapshots the pre-restore state first and re-validates against today’s schema, so a restore that no longer validates exits non-zero. Not simulated: the editor, persistence of a debug session, and secret values. Agents reach the same surface through the js-scripts capability module — list_js_scripts (id, name, description, ports: the discovery surface), get_js_script, save_js_script (validated first, CAS on update), validate_js_script, run_js_script and test_js_script. A script runs inside its own envelope: every installed sandbox pack and every @nodetool-ai/sandbox-nodetool/<namespace> module by import, its declared secrets intersected with whatever allowance the invoking context carries, its own timeout, and the same imported / nodetool.* belt a Code node has. Composition is bounded like sub-agents: depth cap 4 and a script id chain, so a cycle fails the call naming it. Validation and report rules live in @nodetool-ai/execution/js-script-debug; the CLI keeps target resolution, the interaction script, and the bundle. Eval suite: nodetool eval jsscript-tools. Script voicing tools (no workflow, no browser) An agent voices a script and cuts it without authoring a workflow: voice_script_lines synthesizes each line with its cast voice and saves the take onto the line, and assemble_script_timeline lays the voiced takes end to end into a saved timeline sequence — which validate_timeline then checks. list_scripts and get_script find the script and report each line’s status (draft, stale, voiced, no_voice). Voicing defaults to every line that is draft or stale, so one call covers a script; a line uses its own voice unless the call overrides provider+model+voice for all of them. Word timings come from a best-effort transcription pass and ride into the assembled clips as captions. The voice, staleness, and script → timeline rules live in @nodetool-ai/timeline (effectiveVoice/needsVoicing/buildScriptTimeline), shared with the editor and the nodetool.script.* nodes. Code: packages/agents/src/capabilities/scripts.ts. The ui_script_* tools remain the path when the script is open in a browser. Storyboard render tools (no workflow, no browser) An agent takes a storyboard from directed to delivered without authoring a workflow: create_storyboard makes a blank board (then edit_storyboard adds shots), render_storyboard_stills calls the image model per shot and saves each still as the shot’s keyframe, render_storyboard_clips animates those keyframes into clips, revise_storyboard_clip revises one take, and assemble_storyboard_timeline lays the rendered clips into a saved timeline sequence — which validate_timeline then checks. list_storyboards and get_storyboard find the board and its shot ids. get_storyboard_one_take reads a board’s one-take prompt with the block compiled from the board and its [Image N] references, and update_storyboard_one_take writes the prompt, the render settings (duration_seconds, aspect_ratio, resolution, model), and each shot’s end_state and sound. edit_storyboard’s set_board operation accepts screenplay-level narration and music_prompt strings. It preserves the screenplay’s other fields and creates a screenplay when needed. An empty string clears the supplied direction. On an unlinked board, assembly turns these fields into draft narration and music clips. Linked boards use their script’s voiced takes for audio. Both render tools default to every eligible shot missing that output. With stale_only, they select existing versions whose render inputs changed. Provider and model come from the call, then the shot, then the board. An unset model returns an error naming find_model. Stills request 1K and clips request 1080p. Clip calls can override resolution. Clip renders compile creative-context and shot production bindings, local direction, duration and take count before provider dispatch. The still is optional, per shot. A shot’s render_mode decides where its clip comes from: "keyframe" (the default) animates the selected still with image_to_video, "direct" skips the still and generates from the prompt with text_to_video. Set it with edit_storyboard, or pass mode to render_storyboard_clips to override every selected shot for one call. render_storyboard_stills skips direct shots unless they are named in targets. Keyframe stays the default because it is what makes a board affordable and coherent: a still is the cheap unit to iterate on and the anchor that holds a character, a palette and a lighting setup steady across shots. Direct earns its place on two shapes. First-frame conditioning biases the sampler toward the reference appearance, so a heavy-motion shot comes out stiffer than the same model’s text_to_video. And the native-audio models (synced dialogue, diegetic sound) are weakest on their image path. A direct shot’s prompt therefore carries the framing and the board style too, since no still carries them in. The prompts, entity seasoning, and shot → timeline mapping are the editor’s own (entitiesForShot in @nodetool-ai/protocol, buildStoryboardTimeline in @nodetool-ai/timeline), so a headless render matches one done in the UI. Code: packages/agents/src/capabilities/storyboards.ts. The ui_storyboard_* tools remain the path when the board is open in a browser and the user should watch it fill in. Shipped example storyboards Boards ship the way workflows and apps do — a file on disk, read without a user, installed into a library with one insert. The bundles are packages/base-nodes/nodetool/examples/storyboards/<slug>.storyboard.json (the storyboards sibling of the example workflows, which is where exampleStoryboardsDir looks by default in the monorepo, the packaged backend, and the server image). storyboards.examples lists them and storyboards.installExample installs one; the web offers them under Examples → Storyboards and New → New storyboard…. Each shot includes its direction and a rendered still. The boards ship without video clips. The media are package:// assets under assets/nodetool-base/storyboards/<slug>/, so one copy on disk serves every user and an install writes no bytes. node scripts/build-example-storyboards.mjs builds them from scripts/example-storyboards/boards.mjs and preserves the checked-in stills. The stills were rendered through the NodeTool CLI, then resized to 1280×720 JPEGs for shipping. To render a new shot, run npm run dev:nodetool -- generate atlascloud bytedance/seedream-v4.5 "<shot prompt>" --aspect-ratio 16:9 -o <shot>.jpg. --draw-stills draws a vector fallback from the spec when no image provider is available. Add --check for the CI shape (bundles unchanged, every named media file present), or --board <slug> for one. npm run validate:examples checks each shipped board’s shot text and that every still it names is on disk, and scripts/verify-backend-bundle.mjs checks the same files were staged into the packaged bundle. Shipped compositions A composition is a group clip with parameters, stored as a JSON asset carrying metadata.nodetool_composition the way entities carry their marker. Six ship with the product in packages/base-nodes/nodetool/examples/compositions/ (title card, lower third, caption bar, callout, end card, logo sting), built from the spec in scripts/example-compositions/compositions.mjs by node scripts/build-example-compositions.mjs (--check fails when a bundle drifts from its spec). The directory is registered in PACKAGE_RUNTIME_ASSETS, so the backend bundle stages it and the verifier checks it. Agents reach them through list_compositions, get_composition, save_composition (extract a group from a saved timeline), delete_composition and the edit_timeline op insert_composition; the pure instantiateComposition/extractComposition live in packages/timeline/src/composition.ts. Shipped recipes A recipe is a named outcome, the shipped example app that reaches it, and the ordered example workflows that app is made of — “one packshot per SKU becomes the whole channel set”, four to six shipped workflows deep. The manifests are packages/base-nodes/nodetool/examples/recipes/<slug>.recipe.json (the recipes sibling of the example workflows, where exampleRecipesDir looks by default in the monorepo, the packaged backend, and the server image). They hold no graphs and no app documents: each step names a shipped example and each entry in apps a shipped bundle in packages/base-nodes/nodetool/examples/apps/, and listExampleRecipes (packages/websocket/src/lib/example-recipes.ts) resolves both against what the install actually ships, reading each step’s node count, thumbnail and models out of the graph. A recipe with a step or an app that no longer resolves is dropped from the listing rather than half-offered. The first entry in apps is the one built for that chain and binds every workflow the chain names — tests/example-recipes.test.ts fails when it stops covering the steps. Any entries after it cover part of the chain and are worth having anyway. workflows.recipes serves them; the web app shows them above the gallery on Examples, where the card’s primary action installs the recipe’s app (POST /api/applications/examples/:slug/install, which also creates the workflows it binds), a step opens that example as a workflow, and “Add all” copies the whole chain into the library as graphs. The site builds its /recipes pages and the downloadable .nodetool bundles from the same manifests (marketing/scripts/generate-recipes.mjs, with the sample renders and page order in marketing/scripts/recipes.mjs), so the page and the product name one list of workflows. The recipes harness is the gate on that: its selfcheck runs the resolver’s suite (packages/websocket/tests/example-recipes.test.ts) and the site generator in --check mode, and it fires on any diff touching a manifest, the resolver, the Examples listing, or the recipe pages. npm run validate:examples also checks every step, alternative and hero resolves, and scripts/verify-backend-bundle.mjs checks the manifests and the examples they name were staged into the packaged bundle. graph-resource fixtures The nodes that put an approved storyboard, script, timeline or entity into a graph (nodetool.storyboard.*, nodetool.entity.*, nodetool.script.*, nodetool.timeline.*, nodetool.game.*) are only interesting when they are wired together, and every graph that wires them reads documents a person approved and spends money on renders. This harness runs them anyway. npm run fixtures:graph-resources # all of them node scripts/graph-resources-fixtures.mjs e1 --keep # one, keeping the scratch install scripts/graph-resources-fixtures.mjs builds a scratch install — its own DB_PATH and ASSET_FOLDER under a temp directory, NODETOOL_ENABLE_FAKE_PROVIDER=1 — and seeds it with the world the graphs read: a template board with a Product prop entity and a style entity, the approved cut behind it carrying a {{name}} — {{price}} overlay on its own track, a b-roll board whose stills are stored assets, and a 16:9 launch cut. Then it runs each graph in packages/cli/fixtures/graph-resources/ through nodetool debug and reads the result back out of the database. The fixtures are the shipped examples with the generators swapped for nodetool.fake.* and the board’s models set to the fake provider, so a run costs nothing and needs no key. Three claims a green run alone would not make: Fixture What it asserts per-sku-ad-factory.fake.json The exported sequence carries the overlay with both placeholders replaced — so the template’s cut survived recast, render and assembly, and FillTimelineText reached the clip that inherited it. topdown-native-asset-pack.fake.json Every top-down slot receives a staged asset binding and content-addressed path. three-ratios.fake.json Both retargets are new rows naming the source as their template, and every clip keeps the name, start and duration the approved cut gave it. The verdict comes from the debug bundle, not the exit code: a graph carrying a sandboxed nodetool.code.Code node has been seen to exit 0 and write no bundle after a node failed, so a missing bundle is a failure here. Two shapes the fixtures deliberately do not have, both because of defects outside this harness: No Localized Explainer fixture. nodetool.script.* needs the getScript / createScript / updateScript model interfaces, and those are wired only in packages/websocket/src/session/model-interfaces.ts — not in documentModelInterfaces(), which is what the CLI installs. Every script node fails under nodetool debug with “ProcessingContext model interface ‘getScript’ is not configured”. The pure half of the harness is the derivation suites — packages/storyboard, packages/timeline’s derive, protocol’s script-fill and game-slot-prompt, packages/game-nodes, and the node suites in core-nodes and video-nodes. nodetool harness gate runs both halves on any diff touching packages/storyboard/, packages/game-nodes/, the derivation sources, the node files, the four shipped examples or the fixtures themselves. 3D scene tools (no editor, no browser) An agent builds and fixes a 3D model without an editor open: list_model3ds finds the .glb/.gltf assets, create_model3d makes one holding an empty glTF scene (optionally applying operations in the same call), get_model3d lists every object with its transform, visibility and material color plus the scene’s world-space bounds, edit_model3d runs the ui_3d_* verbs — add and delete primitives and lights, set transforms, rename, show and hide, recolor, select — against the stored document and saves it back over the same asset, validate_model3d checks a document statically, and render_model3d renders one through headless Blender and stores the PNG. The operations, the units (Euler degrees, CSS hex) and the “uuid or name” addressing live in @nodetool-ai/model3d, shared with the browser editor, so a model built headlessly opens there unchanged and an edit touches only the nodes it names — an imported model keeps its meshes, textures, skins and animations. Object ids are stamped into node.extras.nodetool_id, because glTF addresses nodes by array index and a delete renumbers them. The camera has no headless equivalent: ui_3d_frame_scene and ui_3d_capture_view need a WebGL context, and get_model3d’s bounds are what answers “how big is this and where is it” without one. Implementations: packages/agents/src/capabilities/model3d.ts. The ui_3d_* tools remain the path when the model is open in a browser. Native game pipeline (templates, staging, playtest, web build) A built-in game has one mutable draft and immutable published revisions in a project workspace. The @nodetool-ai/game-runtime package validates and simulates it at a fixed tick rate. Legacy 2D uses WebGPU quads or Canvas 2D. Explicit schema version 3 games use Rapier WASM physics and Three.js WebGL2. Both starter templates run with placeholder geometry before media generation. Captures use the selected renderer and standalone builds include the runtime and verified media. create_native_game {project_id, name, dimension: "3d"} starts an exploration blockout. The editor provides orbit and fly cameras, picking, transform gizmos, snapping, diagnostic overlays, script editing and draft undo. Play pins a cloned draft. Existing script source edits compile during play, preserving tick and script state. Invalid edits retain the previous source and report a Console error. Other edits require Restart from tick 0. playtest_native_game accepts recorded analog axes and look, spatial and grounded assertions, and snapshot replay checks. autoplay_native_game returns an unsupported diagnostic for 3D. Use a recorded route to verify its completion. generate_game_asset imports owned GLB or glTF models and stages normalized GLB candidates. Dependencies must come from the owned workspace. Models and prepared colliders require explicit installation with the draft timestamp. Captures return committed simulation hashes and projected bounds. The renderer package’s tests exercise skinning, shadows, context recovery, disposal and a closed-network standalone completion route. Desktop capture uses Playwright, which the Package Manager installs. A build without it reports unavailable capture capability. See the 3D design and runtime contract. The nodetool.game.* nodes retain asset checks. LoadGameTemplate lists native asset slots. SlotPrompt prepares a request for one slot. SpriteSheet, Tileset, SeamlessImage, SoundEffect, and MusicLoop check generated media. StageGameAssets validates the fills and writes content-addressed candidates to the project workspace. Staging does not alter a published game revision. install_native_game_asset installs a selected candidate binding into the draft with a draft timestamp check. Existing scene and behavior edits remain in place. browse_native_game_assets lists the draft’s assets with where each is used, its prefabs and scenes, the staged candidates, and each slot’s generation prompt. With digest and slot it returns an installable binding for one candidate. The editor’s Assets panel calls the same catalog and filter. Tests: packages/agents/tests/capabilities-game-asset-browser.test.ts and packages/websocket/tests/trpc-game-asset-browser.test.ts. Agents can build a complete document with @nodetool-ai/sandbox-game and save it through edit_native_game’s atomic set_document op. The builder supplies entities, scripts, asset registration, and tile and arc helpers. Retained construction adds hermetic preview/apply, typed parameters, prefab relationships, property overrides and deleted-instance suppression. See Retained game construction. list_example_games, get_example_game, and install_example_game expose the shipped benchmarks and copy their verified assets into an owned project. generate_game_asset prepares aligned sprite sheets, terrain edge variants, and packed grade LUTs. It also generates sound through sound-effect nodes and imports TTF/OTF fonts. autoplay_native_game steers standard controls to target contacts or an observed win and returns a replayable route and level stats. Its search is bounded. A stalled search is inconclusive. playtest_native_game accepts up to 18,000 ticks and can assert engine wins or scripted victory events. capture_native_game_frame supplies images for review against the locked game-direction spec. Schema validation and a successful route do not measure visual quality. Publishing the draft with publish_native_game creates a revision when the user asks. The web Game tab supplies direct editing, play, pause, step, reset, a scene tree, and draft review. The nodetool game CLI validates, simulates, captures, and builds a standalone web player. Run nodetool game smoke <directory> after export to render 300 scripted frames in Chromium and check errors, missing assets, and tick progress. The game-flow selfcheck includes the smoke suite. A legacy export node reports a migration diagnostic; external-engine scene behavior requires manual reconstruction. Game flow (guided design and asset graph) The Game entry opens a guided flow: idea, design, review, look, then build. Its document is a workflow row carrying settings.game, the same record the headless capabilities use. design_game writes a reviewed game design to workflow settings, and build_game creates or selects a game in the workflow’s project and stages a generation graph. The web flow (web/src/components/setup/game/) designs with the same prompt and parser, then builds without a graph: it creates the template’s game, records its id on the workflow, and generates each image slot through games.generateAsset with the chosen style and image model. Sound slots keep the template’s built-in effects. The built-in templates and their manifests live in packages/protocol/src/native-game-templates.ts, and the inspiration chips for them in NATIVE_GAME_INSPIRATION_CHIPS. The pure design, prompt, and graph placement contracts live in packages/protocol/src/game-design.ts, game-flow-prompt.ts, and game-graph.ts. Re-running the graph produces asset candidates; it does not overwrite a published game. The game-flow harness covers those protocol contracts, node registration, native staging, agent capabilities, persistence, simulation, and renderer behavior. npm run fixtures:graph-resources runs the platformer asset graph with fake generators and checks that every declared slot gets a staged binding and path with a content digest. It does not call a media provider. Entity library tools (no browser) The reusable production entities — characters, locations, styles, props — are image assets carrying a marker under metadata.nodetool_entity. list_entities lists them (filtered by kind or text), get_entity reads one in full, and apply_entities pastes their descriptors into a prompt and returns the reference-image asset ids to pass to an image model. create_entity tags one of the caller’s image assets as an entity (the same marker write the browser’s Save Entity does; generate or save the image first), update_entity changes an existing entity’s fields — or the picture it shows, via asset_id — and delete_entity untags one (marker cleared, asset kept). Swapping the picture writes reference_asset_id onto the marker instead of moving the marker to the other asset, so the entity keeps the id boards and scripts cast it by; asset_id: null puts it back on its own bytes. The browser’s entity editor writes the same field. The injection rule is injectEntities in @nodetool-ai/protocol, shared with the browser’s ui_entity_apply and the Director node: with explicit entity_ids exactly those apply, otherwise the entities whose name appears in the text (all of them when the text is empty). An id that resolves to nothing comes back in missing_entity_ids — otherwise the prompt returns unseasoned and looks fine. Implementations: packages/agents/src/capabilities/entities.ts. Media analysis tools (no model, no ffmpeg) understand_video asks a model what a clip is about. These five measure what it is, and an agent could not get at any of it before without shelling out to ffprobe and parsing text. Tool Answers analyze_audio Duration, sample rate, channels, codec; EBU R128 integrated loudness and loudness range, peak/RMS dBFS, crest factor, clipped samples, DC offset; an RMS/peak envelope over time with the loudest and quietest moments analyze_audio_spectrum Ten named octave bands (sub_bass → air) with each one’s share of the energy, the dominant frequency, and spectral centroid/rolloff/flatness/bandwidth averaged and as a series detect_audio_events Silence and sounding segments, onset times, tempo in BPM with a confidence analyze_video Duration, resolution, frame rate, rotation, both codecs; brightness, contrast, saturation and motion per sampled frame; dominant palette; darkest, brightest and busiest moments detect_video_scenes Cut times and per-shot start/end/duration/brightness/motion/palette, plus black-frame and frozen-frame runs They take the same reference forms read_media_bytes takes — an asset id, an asset:// URI, a /api/storage/ key, a URL, a data: URI — and a video’s soundtrack is a valid analyze_audio target, so a clip needs no demux first. Decoding is Mediabunny’s, the library the sandbox’s audio.*/video.* already use, so none of this depends on ffmpeg being installed. nodetool.audio.GetAudioInfo is the contrast: it sniffs magic bytes, and reports a duration only for WAV. Loudness follows ITU-R BS.1770-4 with EBU Tech 3342 for the range, K-weighted at the file’s own sample rate rather than through 48 kHz coefficients — so packages/agents/tests/audio-dsp.test.ts can pin the scale’s own anchor, a 1 kHz sine at -20 dBFS on two channels reading -20.0 LUFS. Cuts are decided from the luma histogram, not from a pixel difference, so a whip pan inside one shot does not read as an edit. Two things the answers say about themselves rather than leaving to be assumed. Every series is decimated to a point budget and every decode stops at a duration cap, both reported (decimated, truncated). And tempo.reliable is false unless the novelty curve carried at least four onsets and the autocorrelation actually found a period — speech and room tone otherwise produce a confident-looking BPM from nothing. The math is pure and lives apart from the capability: packages/agents/src/analysis/audio-dsp.ts (FFT, spectral features, K-weighted gated loudness, silence, onsets, tempo) and video-frames.ts (per-frame statistics, histograms, motion, palettes, cuts), both tested on signals whose answers are known analytically. media-decode.ts is the Mediabunny seam. Live browser tools (your own signed-in Chrome) The browser_* capabilities drive one real Chrome page action by action — browser_view (URL, title, indexed interactive elements, screenshot), browser_navigate, browser_click, browser_input_text, browser_press_key, browser_select_option, browser_move_mouse, browser_scroll, browser_console_exec, browser_console_view, browser_capture_media, browser_upload_asset, browser_restart, browser_status. Element indexes are rebuilt on every view, so a caller views before it acts on an index. The page is either a headless Chrome the process launched or, through the Chrome extension relay (native messaging host and a local socket), the tab the user is already signed in to — cookies, sessions and 2FA in place, which is what makes Midjourney, Sora and the rest reachable at all. The action loop is the same either way, so only two capabilities mention transports: browser_status reports the one in force (and whether an extension is actually attached, so an agent learns that before spending a 30-second attach timeout), and browser_restart changes it. The action loop is its own package, @nodetool-ai/browser — CdpPage, the session, media capture, file upload, and the extension transport — and it knows nothing about agents, nodes, assets or workflows: inputs are plain values, a screenshot comes back as base64. The capability module (packages/agents/src/capabilities/browser.ts) imports it directly and owns the half that needs a ProcessingContext, turning those bytes into an asset reference and an asset id into bytes. That split is what lets the capabilities and the lib.browser.Screenshot node share one implementation. One session exists per process and every caller shares it — which is why the cloud profile drops them all (packages/agents/src/browser-gate.ts): one shared page across tenants is a single-tenant shape, and the node catalog already agreed by leaving lib.browser out of CLOUD_NODE_NAMESPACES. Extension setup, the wire protocol and its limits: docs/chrome-extension.md. Code authoring tools (no workflow, no browser) An agent writes, checks, and debugs a nodetool.code.Code body without authoring a workflow: validate_code runs the same static check the workflow validator runs (syntax, imports against the installed catalog, undefined names, undeclared inputs.* reads, outputs unset on a return path), run_code executes a body in the QuickJS sandbox with given inputs and returns outputs, logs, and error (yield bodies return the collected streamed items), and test_code grades a case list — inputs plus expected outputs per case — as the regression check after an edit. Execution matches the Code node: the body-shaping rules (implicit return, yield collection, output normalization) live in @nodetool-ai/node-sdk (code-body.ts), shared with packages/code-nodes, so a body that passes the harness runs the same way inside a workflow. Harness runs are hermetic: no node toolbelt, and only the secrets a call names in secrets are readable. These are not a second CodeAct surface — execute_code remains how an agent acts; this harness authors node code. Implementations: packages/agents/src/capabilities/code.ts. In the editor, the Code node’s assistant dialog (code editor + chat side panel, ui_code_* tools) drives the same loop while the user watches. nodetool node run (Single-Node Harness) Runs one node in isolation — instantiate it, feed it a property bag, print what it emits — without authoring a whole workflow. --no-secrets skips the DB for a hermetic run. npm run dev:nodetool -- node run nodetool.text.Concat --props '{"a":"hi ","b":"there"}' npm run dev:nodetool -- node run <type> --props '{...}' --no-secrets # hermetic, no DB npm run dev:nodetool -- node run <type> --props '{...}' --json nodetool generate (Media Generation) Generate an image from any registered provider straight to a file — no workflow. Positional <provider> <model> <prompt>, with lenient name matching (fal-ai → fal_ai, flux-schnell → fal-ai/flux/schnell via the provider’s model manifest). Currently covers text-to-image (and image-to-image with --image). Resolves the provider key from the secret store or env (e.g. FAL_API_KEY). npm run dev:nodetool -- generate fal-ai flux-schnell "a red fox in snow" -o fox.png npm run dev:nodetool -- generate fal-ai flux-schnell "a logo" --aspect-ratio 1:1 -n 4 npm run dev:nodetool -- generate fal-ai flux-dev "restyle this" --image in.png --strength 0.6 npm run dev:nodetool -- generate fal-ai --list-models # discover model ids npm run dev:nodetool -- generate fal-ai flux-schnell "..." --json # machine-readable nodetool eval (Agent Evaluation Suites) Runs the graph authoring eval suite (authorGraph over the typed DSL pack) against any registered provider and reports metrics: success rate, expectation score, one-shot rate (graphs delivered in the first authoring round), authoring rounds (execute_code actions), tool calls, duration, and cost. Cases and expectations live in packages/agents/src/evals/. npm run dev:nodetool -- eval graph-planner --list # show cases npm run dev:nodetool -- eval graph-planner -p anthropic -m claude-sonnet-5 npm run dev:nodetool -- eval graph-planner -p ollama -m qwen-3.5:4b --cases summarize,branch-both-paths npm run dev:nodetool -- eval graph-planner -p openai -m gpt-5.4-mini --json --out report.json npm run dev:nodetool -- eval graph-planner -p anthropic -m ... --min-success 0.8 # non-zero exit below threshold Running a suite from CI. The suites that drive a model run on request: .github/workflows/agent-eval.yml is workflow_dispatch only, with inputs for the suites, the provider, the model, the cases, and a floor that overrides every suite’s default for that run. Each suite carries a default floor recorded in the workflow next to the reason for the number; the run uploads each suite’s JSON and renders one table across all of them. It reports — it opens no PR, is not a required check, and starts on nobody’s schedule, because a model run costs money and varies enough that an automatic red would as often mean an average night as a regression. Two flags exist for the free half. nodetool eval <suite> --list marks the cases that need no API key and no network, and --keyless runs exactly those, asking the suite rather than naming ids (a suite with none refuses the flag). --min-cases <n> fails a run that examined fewer cases than that — the failure a success rate cannot express, since a suite reports 0 over an empty set and 100% over the survivors of a set that shrank. No API key? Use the Claude Agent provider. In keyless environments — Claude Code on the web, CI sandboxes — the claude_agent_sdk provider runs every eval suite on the session’s own Claude credentials, no secret store needed. In the web sandbox (uid=0) set IS_SANDBOX=1 so the nested CLI accepts the permission bypass: IS_SANDBOX=1 npm run dev:nodetool -- eval graph-planner -p claude_agent_sdk -m claude-sonnet-5 Details on env stripping and the uid=0 blocker: docs/AGENTS.md § Claude Agent provider. A graph-e2e suite takes the same planner all the way through: it plans a workflow, executes it on the kernel with the case’s inputs, and has an LLM judge decide whether the outputs achieve the case’s goal. A case succeeds only if all three hold, and that end-to-end rate is what --min-success gates on. Two cases are deterministic (exact string and arithmetic results, judge skipped) and run without model providers; the rest need one, and cost inference twice — once for the run, once for the judge. npm run dev:nodetool -- eval graph-e2e --list npm run dev:nodetool -- eval graph-e2e -p anthropic -m claude-sonnet-5 npm run dev:nodetool -- eval graph-e2e -p openai -m gpt-5.4-mini --timeout 600000 A code-gen suite drives CodePlanner over the Code-node authoring shapes (reshape, merge, compute, parse, split, format, validate, seed) and reports first-pass and post-repair acceptance separately; --min-success gates on post-repair. npm run dev:nodetool -- eval code-gen -p anthropic -m claude-sonnet-5 The task planner has a suite of its own, scoring the plan without running it: task-planner (multi-task DAG quality — parallel width, decomposition size, tool routing, no synthesis task). npm run dev:nodetool -- eval task-planner -p anthropic -m claude-sonnet-5 A codeact suite scores the CodeAct execution mode (steps act by writing sandboxed JavaScript over the toolbelt instead of JSON tool calls — docs/codeact-design.md) on offline instrumented cases: required tools invoked, action rounds within bounds, result correct. npm run dev:nodetool -- eval codeact -p anthropic -m claude-sonnet-5 A subtask suite scores delegation: each of its seven cases hands the parent an objective it should hand to a run_subtask child, and the check is that the child — not the parent — ran the inherited tools. The instrumented tools record the subtask depth of every call, so “the parent did it itself” scores differently from “the parent delegated”. It also covers subtask count, recursion depth, error propagation, and whether the delegated result reached the parent’s answer. npm run dev:nodetool -- eval subtask --list npm run dev:nodetool -- eval subtask -p anthropic -m claude-sonnet-5 Alongside graph-planner (graph authoring) there are eleven tool-loop suites that drive a real provider through the frontend ui_* tool contract against a headless bridge — no browser — and score the multi-turn tool-calling flow structurally: tool-loop (graph editor), workflow-escalation, script-tools, jsscript-tools, sketch-tools, timeline-tools, storyboard-tools, model3d-tools, app-tools, memory-tools, and creative-pipeline. Same flags, metrics, and --min-success CI gate as graph-planner. Details: packages/agents/AGENTS.md. workflow-escalation runs the graph tools over objectives that are missing something only the user can decide — a name, permission to delete, a choice between two node types — plus an ask_user tool wired to a scripted user. Each case scores both the question the model asked and whether the graph it went on to build matches the answer, and one case pins every value so that asking at all is the failure. npm run dev:nodetool -- eval timeline-tools --list npm run dev:nodetool -- eval script-tools -p anthropic -m claude-sonnet-5 npm run dev:nodetool -- eval sketch-tools -p ollama -m qwen-3.5:4b --min-success 0.8 npm run dev:nodetool -- eval workflow-escalation -p anthropic -m claude-sonnet-5 An app-build suite scores nodetool app build end to end: medium-complexity prompts (two operations, a persisted setting, a streaming output, a gated second step, a condition that hides something) go through spec → plan → author → check → run → judge, and a case counts as green only when the build’s verdict is ok and the delivered bundle has the shape asked for. It reports the one-shot rate (green with zero repair rounds — the PRD’s north star), the green-within-budget rate that --min-success gates on, repair rounds, cost, and wall clock. Two deterministic cases author from a script over template graphs, call no provider, and run on every PR in the Quality Gate; the full suite runs nightly (.github/workflows/app-build-eval.yml). npm run dev:nodetool -- eval app-build --list npm run dev:nodetool -- eval app-build -p anthropic -m claude-sonnet-5 # The deterministic cases — no API key needed; the provider is never called. npm run dev:nodetool -- eval app-build --cases greeting-card,draft-then-publish \ -p ollama -m none --no-find-model --min-success 1 nodetool jtbd (Jobs To Be Done — the optimization loop) An eval suite scores a model. A job asks whether the product let the agent get something done, and keeps enough of the run that an outer agent can say what to change. A job is one objective taken end to end across whatever surfaces it needs, stated the way a user would state it (“when I have a scene to shoot, I want it broken into shots, so I can see the coverage before I spend on renders”), handed to the agent in the user’s own words, and graded on the world it left behind. No job names a tool: which tool to reach for, in what order, is what is under test. The worlds are the same headless bridges the tool-loop suites drive, so a job cannot drift from the tool contract those suites pin. npm run dev:nodetool -- jtbd list [--json] npm run dev:nodetool -- jtbd run -p anthropic -m claude-sonnet-5 npm run dev:nodetool -- jtbd run --jobs workflow-from-prompt,timeline-assemble-cut -p openai -m gpt-5.4-mini npm run dev:nodetool -- jtbd run -p anthropic -m claude-sonnet-5 --min-achieved 0.8 npm run dev:nodetool -- jtbd optimize -p openai -m gpt-5.4-mini # review the recorded runs run writes one bundle per job under nodetool-debug/jtbd/<job>/: report.json (transcript, tool calls, outcomes, friction) and review.md, the same thing rendered for a person to read. The transcript is the point. runToolLoop used to drain the provider stream and discard every ProviderMessageEvent, so a run recorded which tools fired but never what the model was told or what it said between calls — enough to score a run, not enough to diagnose one. It now keeps the whole conversation. Before any model reviews a run, the pure pass in packages/agents/src/jtbd/friction.ts derives what a transcript decides on its own, and each signal names an owner. A tool that errored repeatedly, or answered the same call identically three times, is a harness finding — the fix is a schema or an error string in our code. A run that called no tool at all is a prompt finding. A run that took twice the turns it should have is unattributed, because whether that is a prompt failing to describe the short path or a tool surface forcing the long one is exactly the judgement a pure pass cannot make. A rule that guesses confidently sends the fix to the wrong file. optimize is the outer half: it hands one run — the system prompt verbatim, every assistant turn, every call with its arguments and result — to a different model and asks what one change would make the next run go better. It must name a target and a change; “improve the prompt” is rejected by the parser. Proposals land in proposals.json next to the run. It proposes; it never applies. Nothing in the loop edits a prompt or a tool. That is the anti-slop ratchet’s posture — it opens a PR, it merges nothing — and for the same reason: a loop that rewrote its own prompts on a model’s say-so would have no reviewable step between a bad diagnosis and a shipped regression. Run and review are separate commands because the bundle is the handoff. A run costs model time; reviewing it is a different model, a different prompt, often a different day. Splitting them means a bundle can be re-reviewed with a better optimizer without paying for the runs again, and that a person can read the transcript before any model proposes anything. The catalogue’s own invariants are tested rather than trusted (packages/agents/tests/jtbd-friction.test.ts): every job states a purpose, grades at least one outcome, names no tool in its objective, and fails its own checks on an untouched world — a job whose outcomes pass before the agent does anything is measuring nothing. nodetool packs compile (Sandbox npm Modules) A sandbox pack can declare a guest module by npm dependency name instead of authoring code ({"name": ".", "kind": "js", "npm": "js-yaml"}). This builds it: esbuild bundles the dependency with pinned resolver conditions and no externals, a scope-aware scan rejects free references to globals the guest lacks, and a capability-free QuickJS probe imports the bundle to prove it initializes. Results are cached by content digest — never by version — under <user cache>/nodetool/sandbox-modules. npm run dev:nodetool -- packs compile # every installed pack npm run dev:nodetool -- packs compile --json # machine-readable report npm run dev:nodetool -- packs compile --force # recompile and re-probe npm run dev:nodetool -- packs compile --pack-search-path <node_modules dir> Everything that stops a module short of admission is a named skip, not an error: npm-module-builtin-import (the dependency needs node:*), npm-module-unresolved, npm-module-too-large (1 MB cap), npm-module-forbidden-global, and npm-module-probe-failed. The skips reach the Package Manager through packs.sandboxModules diagnostics. The server compiles during its own catalog refresh and Electron compiles after an install, so the command is for a warm cache and for diagnosing one pack. The CLI’s synchronous registry build never compiles: it reads the cache, re-hashing every recorded input first, and a miss surfaces as pending-compile naming this command. Compiler: packages/sandbox-compiler. Design: docs/sandbox-package-design.md § Config-only modules from npm packages. Every library the sandbox offers is an importable pack. There is no library global — the data.* namespace is gone. The packs live in packages/sandbox-packs/, each a package.json manifest plus a SKILL.md, and every one of them is available out of the box. The current list — which library each wraps and whether it runs guest-side or host-side — is the table in packages/sandbox-packs/README.md; read it there rather than from a copy that drifts. guest means the compiler bundles the library into QuickJS. host means it runs where the sandbox runs — needed when the library wants Node builtins or a DOM, when it carries a limit the guest could not enforce on itself (zip’s 50 MB inflation cap), or when the code is NodeTool’s own and a config-only pack therefore cannot ship it. Two packs are authored guest code rather than a library: -dsl builds a workflow graph, -flow calls nodes as typed async functions (docs/dsl-native-flow-design.md). Both are generated from packages/dsl and rebuilt by npm run build:sandbox-dsl / build:sandbox-flow. Native flow: call nodes from sandboxed code A third way to run nodes, next to WorkflowRunner and the graph DSL: guest code in the QuickJS sandbox calls a node as a typed async function and writes the control flow in plain JavaScript. await is the edge, a variable is the wire, Promise.all is the fan-out — no graph, no edges, no runner. import "@nodetool-ai/sandbox-nodetool/flow"; // mounts the bridge (body-side, required) import { concat } from "@nodetool-ai/sandbox-flow/nodetool.text"; const r = await concat({ a: inputs.left, b: inputs.right }); await output("joined", r.output); One module per node namespace, generated by the same npm run codegen pass as the graph DSL and shipped as the sandbox-flow pack. Streaming-output nodes carry .stream(inputs) — an async iterable over cursor calls; early break closes the stream and runs node cleanup. Errors reject the call; try/catch is the supervisor. Each call bridges to the host’s registry/invoke path through the @nodetool-ai/sandbox-nodetool/flow capability module (invoke_node, open_node_stream/take_node_stream/close_node_stream), so every invocation passes the per-call permission gate, bills through the invoking run’s ProcessingContext, and is bounded by a recursion depth cap of 4 and 16 concurrently open streams per run. v1 limits: streaming inputs accept arrays only (no live guest-produced streams), and the body must import the capability module itself for the facade to mount — the pack’s SKILL.md states both imports. The host backend is packages/dsl/src/flow/ (internal; @nodetool-ai/dsl/flow exists for the hidden import, not as a public surface — programs that must open in the editor, be validated, or run on the server still build a graph). The capability implementation is packages/agents/src/capabilities/flow.ts. Diffs touching either run the dsl-native-flow harness selfcheck via nodetool harness gate. Design and pivot record: docs/dsl-native-flow-design.md. The -aws, -notion, -supabase and -twilio packs are the host case: they replace the S3, Notion, Supabase and Twilio nodes. Each builds an authenticated request — -aws signs one with SigV4 — and none of them sends it. The guest passes what comes back to its own fetch, so the run’s fetch cap and SSRF guard still apply. Credentials come from nodetool.secrets.get(name), which a Code node can narrow to the names it declares in its secrets property. A host pack’s manifest entry is {"kind": "host", "host": "<id>"}, and the id resolves only through NodeTool’s own SANDBOX_HOST_MODULES table, which pins the one package allowed to declare it — a third-party pack can never bring host code. The implementations live in packages/agents/src/host-modules/, with every safety limit inside them. Apify is not one of them any more. A -apify pack of exactly this shape existed and was removed: the request-builder pattern requires the guest to hold the credential (nodetool.secrets.get("APIFY_API_TOKEN")) and to do its own fetching and polling, which is the wrong trade for a service that runs third-party code, on third-party machines, against a URL a model chose, and bills for it. Apify is now a capability module (@nodetool-ai/sandbox-nodetool/apify): the token never leaves the host, every actor passes an allowlist and a session budget, actor inputs are SSRF-screened, cancellation aborts the remote run, and files it produces become NodeTool assets. See docs/apify-integration.md. SerpAPI is a capability module (@nodetool-ai/sandbox-nodetool/serpapi), and the engine list is discovered rather than declared. SerpAPI is one endpoint whose engine parameter selects which of ~120 contracts applies — Google and its verticals, Bing, Baidu, DuckDuckGo, Yandex, Naver, YouTube, Amazon, eBay, Walmart, Yelp, TripAdvisor, the app stores — so list_serpapi_engines and get_serpapi_engine_schema read SerpAPI’s own engine table and an engine it ships tomorrow is callable with no diff here. serpapi_search runs any of them; the key stays on the host, api_key and output are refused from a caller, and the parameter bag is checked against the engine’s contract before the call — SerpAPI ignores an unknown parameter, so a typo is otherwise a billed search that answers a different question. web_search stays what it is: one query against whichever SERP_PROVIDER this install configured. See docs/serpapi-integration.md. Google Workspace is a capability module too (@nodetool-ai/sandbox-nodetool/google), and the only Drive/Gmail/Docs/Sheets/ Calendar surface — the fourteen lib.google.* nodes are gone. It authenticates with the token the user’s Google sign-in returns rather than an API key, which the host resolves and refreshes; a guest never sees it. Its twenty calls are the fourteen the nodes made plus six they never offered — get one Drive file, get one Gmail message, list labels, create a spreadsheet, list calendars, delete an event — and a missing or revoked credential comes back as {error} telling the user to sign in again. A server with no Google login offers none of it — see NODETOOL_GOOGLE_WORKSPACE in docs/configuration.md. NodeTool’s own settings are a capability module (@nodetool-ai/sandbox-nodetool/settings, also nodetool.settings.*), and the shape of it is the point: list_settings, get_setting and set_setting cover ordinary configuration, list_secrets reports which credentials this install holds without their values, and there is no set_secret. The definitions come from settingCatalog() in @nodetool-ai/config — the same table the tRPC settings router answers settings.list from — so the capability knows which names hold credentials instead of guessing from the name, and refuses to read or write one. Setting a secret goes through a bespoke dialog. request_secret takes a name, a reason and a help URL — never a value. The host sends a secret_request frame, the user types the key into a card in their own client, that client saves it with its own settings.secrets.upsert call, and the answer coming back (secret_request_response) says saved or declined and nothing else. The credential therefore never enters the guest, the websocket payload, the chat transcript, or the model’s context; the run learns only that a secret now exists, and reads it — if at all — through nodetool.secrets.get under its own declared secretScope. The dialog is a host capability, not a fallback: a headless run (a workflow on the kernel, the CLI, an eval) carries no CapabilityRun.secretPrompt and the call is refused by name rather than quietly writing something nobody approved. The last three replaced nodes rather than bridges. lib.browser.WebFetch, DownloadFile, Browser and SpiderCrawl are the fetch capability plus -html; lib.excel.* is -xlsx; lib.ocr.* is -ocr; and lib.tensorflow.* is -tfjs. Each was a chain of near-identical nodes that one script now expresses; only lib.browser.Screenshot (a real page over CDP) and lib.sqlite.GetDatabasePath stayed nodes. These packs are still not workspaces — no host code may import one — so npm links nothing into node_modules and discovery reads them from disk instead: packages/sandbox-packs/ in a checkout, and _sandbox/ next to server.mjs in the packaged desktop app and the Docker image, where bundle-backend.mjs stages every pack in that directory and verify-backend-bundle.mjs fails a build that misses one. shippedPackSearchPaths() (packages/node-sdk/src/pack-loader.ts) resolves both, and puts the shipped root last: a pack of the same name installed through the Package Manager shadows the copy in the app. Declaring a specifier from a pack this host does not carry still fails validation with “Install <pack>”. See packages/sandbox-packs/README.md. nodetool affected (Changed-File → Workspace Mapping) Maps changed files (or the git working tree) to the minimal set of workspaces to rebuild/test: the owning package plus its downstream dependents, and a build:packages only when a decorator package (loads from dist/) is affected. Avoids reflexively running the full 1–2 min build. Workspaces come from the root package.json, not from a scan of packages/ — reliability/harness is a workspace too, and a scan of one directory reported every change under it as belonging to nothing. reliability/journeys/ maps to the harness that runs it (EXTRA_WORKSPACE_PATHS in packages/cli/src/affected/affected.ts). npm run dev:nodetool -- affected # uses git working-tree changes npm run dev:nodetool -- affected --base main # diff against a ref npm run dev:nodetool -- affected packages/cli/src/x.ts # explicit files npm run dev:nodetool -- affected --json npm run mutate (Mutation Testing for TypeScript) Mutates one TypeScript site at a time (< to <=, && to ||, true to false, delete !, ?. to ., …), runs the owning package’s tests, and reports each mutant the tests fail to kill. A port of unclebob/mutator. Use it to check that a new test actually pins the behavior it claims to. npm run mutate -- packages/kernel/src/graph.ts # one file npm run mutate -- --base origin/main packages/kernel/src # functions changed on this branch npm run mutate -- --scan packages/kernel/src # list sites, run nothing npm run test:mutation --workspace=packages/kernel # the package's configured run node scripts/mutation-score.mjs --survivors kernel # survivors from the snapshots Tests: a Vitest package runs vitest related --run for the mutated file. A mutant is killed when that fails or exceeds ten times the baseline duration. Coverage: sites on lines the package’s V8 coverage run never hits are reported as uncovered and skipped. --no-coverage runs them too. Snapshots: .metrics/mutate/<namespace>.json records each function’s hash and each outcome, so a rerun only retries survivors and functions whose text changed. Isolation: mutants run in overlays under target/mutation-workers, one per core, each holding a copy of the package. The checkout is never edited. Exit codes: 0 all killed, 2 baseline failed, 3 a mutant survived. Details: packages/mutator/README.md. npm run probe:providers (Provider Contract Probes) Asks OpenAI, Gemini, fal, and KIE for one real response each and decodes it with the same production decoder a run uses. A cassette proves NodeTool still handles a response a provider gave us once; it cannot notice that the provider changed the response today. npm run probe:providers # one request per provider, keys from env npm run probe:providers -- --json --out report.json npm run probe:providers -- --only openai.chat-completion npm run probe:providers -- --strict-network # also fail on an unreachable provider The offline half needs no key and runs on every diff touching packages/runtime/src/providers/: each manifest entry decodes a checked-in raw HTTP response fixture, and every declared required field is deleted once to prove the check can fail (npm run test --workspace=packages/runtime -- provider-contract-probes). Network failures are reported apart from schema failures. No body reaching the decoder (DNS, timeout, 5xx, an HTML gateway page) is a network failure and does not fail the nightly job; a response that no longer decodes is a schema failure and does. Budget: one request and USD 0.05 per provider per run, enforced by the runner. Retained artifacts hold response shapes and redacted messages, never a body — no credential, prompt, request id, or signed URL survives. Manifest: packages/runtime/src/providers/contract/probe-manifest.ts. Details: docs/provider-contract-probes.md. npm run crap (CRAP Scores for TypeScript) Scores each TypeScript function with CRAP = CC² × (1 − coverage)³ + CC, a port of crapper’s TypeScript rules. 1–5 is low risk, 5–30 is worth a look, and above 30 the function is complex and under-tested. Use it to find which function to split or test first. npm run crap -- --no-coverage --top 20 packages/kernel # complexity only npm run crap -- --changed --run-coverage # working-tree changes, with tests npm run crap -- --base origin/main --run-coverage --threshold 30 # CI: exit 2 above 30 npm run crap -- --lcov packages/kernel/coverage/lcov.info --json packages/kernel Complexity is 1 plus each if, loop, catch, ?:, case/default, &&, ||, ??, logical assignment, and ?. token. Entries are functions not nested in another entry: top-level functions, arrow functions, values a call wraps (const Card = memo(() => …) is Card), top-level object-literal members, and every class member. A route callback (app.get("/users", handler)) is its own entry, GET /users. Other nested callbacks count toward their enclosing function, so a store built inside create((set) => …) is one entry, and its score is the sum of every action. A component’s score likewise includes its inline handlers. Move an action or handler to a module-level function to score it alone. Coverage is LCOV. A function with BRDA branch records is scored by branches, otherwise by line hits. A function the report leaves out scores 0%. By default the tool reads coverage/lcov.info in each workspace that owns an analyzed file. --run-coverage first runs that workspace’s related tests (vitest related or jest --findRelatedTests) with LCOV output, and Vitest writes the report even when a test fails. When several reports cover one file, a line or branch counts as covered if any report hit it. Coverage lands on src/ only when tests load the source. A test that imports its own package by name resolves to dist/ and records nothing for src/. A package whose tests import it by name aliases that name to src/index.ts in its vitest.config.ts. Test files, declaration files, and tests/, __tests__/, dist/, and node_modules/ are skipped. Rules are pinned by scripts/__tests__/crap-score.test.mjs. nodetool harness (Registry, Coverage Audit, and the Gate) The machine-readable inventory behind harness-first engineering (docs/HARNESS_FIRST.md): every headless harness in the repo, every product surface with the code paths it owns, and which harnesses cover which surface. An uncovered surface must carry a written gap note; one without it fails audit and the registry test. Shipping a new surface means adding it to packages/cli/src/harness/registry.ts — with its harness or its debt written down. gate makes the registry executable: it maps a diff onto surfaces by path and runs the selfcheck of every harness covering a touched surface — keyless, deterministic invocations like validate:examples, the Ring 0 reliability journeys, a shipped-bundle wiring check, the app-build deterministic cases. The diff selects the checks, not the author. Harnesses that need a target or key are printed as manual work, never silently skipped. A selfcheck with cost: "expensive" (a Blender render, the packaged-backend bundle) is skipped by default even when its surface is touched; pass --expensive to include it. npm run dev:nodetool -- harness list # every harness + capabilities npm run dev:nodetool -- harness audit # surface coverage + documented gaps npm run dev:nodetool -- harness audit --strict # exit 1 while any gap remains npm run dev:nodetool -- harness gate --base main # run the selfchecks this diff demands npm run dev:nodetool -- harness gate --dry-run # plan only npm run dev:nodetool -- harness gate --all # every selfcheck (--expensive to widen) npm run dev:nodetool -- harness gate --strict # also fail on a touched gap surface npm run dev:nodetool -- harness gate --timeout 900 # per-selfcheck timeout, in seconds npm run dev:nodetool -- harness gate --deps # also route through workspace dependents npm run dev:nodetool -- harness gate --include-suites # also run suite-only selfchecks npm run dev:nodetool -- harness gate --jobs 4 # cheap selfchecks four at a time npm run dev:nodetool -- harness capabilities # capability coverage + documented gaps A changed file the registry’s paths cannot place on any surface fails the gate outright, as long as it looks like source rather than prose, a test, or config: a new file the registry has never heard of is exactly the case a coarse-grained paths list is meant to catch. --strict widens that to a touched surface only a gap note covers, the ratchet to pull once a gap is closed. --timeout <seconds> bounds each selfcheck; one that runs past it is killed and counted as a failure, so a hang fails the gate instead of the CI job’s own timeout cutting it off with no verdict. The Quality Gate’s harness-gate leg (.github/workflows/quality-checks.yml) runs harness gate --base origin/main --timeout 900 on every PR and harness gate --all on push to main, so this is exercised on every diff, not only when an agent remembers to run it locally. capabilities is the same invariant one rung down, over packages/cli/src/harness/capability-table.ts: every exported agent capability names the suites a selfcheck runs over it, the eval cases that drive a model through it, or a written gap note. The table is derived — npm run capabilities:sync rewrites it from the live registry, the agent suites and the eval case files, and npm run capabilities:check fails when it is stale or when a new capability arrives with no check and no gap note. It also carries a fingerprint of what each capability declares, so harness gate --base <ref> can refuse a contract change that left its coverage mapping untouched while saying nothing about an ordinary refactor. See packages/agents/AGENTS.md § Capability coverage. nodetool reliability (Cross-Surface Journey Diffs) Runs a journey from reliability/journeys/ on every execution surface it declares and diffs each non-oracle surface against the kernel oracle. A journey is a small workflow plus the invariants its run must hold — lifecycle pairing, terminal uniqueness, cleanup leaks — and what it proves is that the kernel runner and the ws-server produce the same stream for it. Reach for it after a change to execution: harness gate already runs the Ring 0 journeys on such a diff, and this is how you run one by hand. Run it from dist, not from source: the journey fixtures use decorators, which the dev:nodetool transform rejects (Decorators are not valid here). Build the packages first. npm run nodetool -- reliability list # journeys + their surfaces npm run nodetool -- reliability run linear-text-pipeline npm run nodetool -- reliability run <journey> --surface kernel # repeatable npm run nodetool -- reliability run <journey> --faults provider-429 --diff npm run nodetool -- reliability update-goldens <journey> --faults replaces the journey’s own matrix for that run. The provider-seam faults are implemented (provider-429, provider-500, provider-timeout, truncated-stream, malformed-sse, slow-drip, cost-omission); the ws/bridge/host/client names are recognized but report as unimplemented. update-goldens rewrites expected/ from a fresh unfaulted kernel run — it cannot tell a fixed bug from a new one, so read the diff before committing it. nodetool package (Node-Pack Authoring) Manages TypeScript node packages — the packs contributing node types to the registry — not the sandbox packs nodetool packs handles. init scaffolds a package (prompting for name, description, author), list reports what this install has, and docs / node-docs / workflow-docs generate a pack’s Markdown. npm run dev:nodetool -- package list [--available] [--json] npm run dev:nodetool -- package init npm run dev:nodetool -- package docs [-o docs] [--compact] npm run dev:nodetool -- package node-docs [-o docs/nodes] [-p <namespace>] npm run dev:nodetool -- package workflow-docs [-o docs/workflows] [-e <dir>] nodetool workflows The workflow document tools (ui_get_graph, ui_add_node, ui_connect_nodes, ui_update_node_data, ui_delete_node, ui_delete_edge, ui_move_node, and ui_set_node_title) report source: "editor" for a live editor graph and source: "server" for saved state. Reads and successful mutations return document_revision, a content hash of the exposed persistent graph fields. Pass it as based_on_revision on the next mutation. A changed graph returns a revision conflict before applying the edit. Unsaved editor changes produce a different revision from the saved graph. The revision does not cover workflow metadata or transient editor selection and execution state. Reads and writes the local database directly — no running server needed. Pass --api-url <url> (or set NODETOOL_API_URL) to target a remote server instead. The same applies to jobs, assets, and models list/ollama/huggingface. npm run dev:nodetool -- workflows list # List all workflows npm run dev:nodetool -- workflows list --json # JSON output npm run dev:nodetool -- workflows get <workflow_id> # Get workflow details npm run dev:nodetool -- workflows get <id> --json # JSON output # Run workflow by ID (uses local DB), JSON file, or DSL file npm run dev:nodetool -- workflows run <workflow_id> npm run dev:nodetool -- workflows run <workflow_id> --params '{"key": "value"}' npm run dev:nodetool -- workflows run workflow.json npm run dev:nodetool -- workflows run workflow.ts npm run dev:nodetool -- workflows run <id> --json # JSON output # Export workflow as TypeScript DSL npm run dev:nodetool -- workflows export-dsl <workflow_id> npm run dev:nodetool -- workflows export-dsl <id> -o output.ts # Write to file npm run dev:nodetool -- workflows export-dsl workflow.json # From JSON file # Export workflow as a shipped template: materialize its referenced assets into # the package's constant asset dir (rewriting refs to package://<pkg>/<file>) # and write the example JSON. The assets ship with the build and resolve on any # install via /api/assets/packages/<pkg>/<file>. npm run dev:nodetool -- workflows export-example <workflow_id> npm run dev:nodetool -- workflows export-example <id> --package nodetool-base npm run dev:nodetool -- workflows export-example workflow.json -o example.json # Export/import a portable .nodetool bundle (zip): one or more workflow graphs # plus the bytes of every asset they reference, sharable as a single file (refs # become bundle://<file> inside, rewritten back to asset:// on import). Also # exposed over the API (GET /api/workflows/:id/export-bundle, POST # /api/workflows/export-bundle {workflow_ids}, POST /api/workflows/import-bundle) # and in the editor command menu (Export/Import Workflow as Bundle). npm run dev:nodetool -- workflows export-bundle <id> [<id2> ...] -o my-pack.nodetool npm run dev:nodetool -- workflows import-bundle my-pack.nodetool # → local library # Rewrite saved Code node bodies for the `inputs` object. A declared input used # to arrive as a global of its own name, so an old body ReferenceErrors on its # first read. The rewrite is done on the AST — a name in a string, a comment, an # object key, or a local binding is left alone — and is safe to re-run. npm run dev:nodetool -- workflows migrate-code-inputs --dry-run npm run dev:nodetool -- workflows migrate-code-inputs [--user-id <id>] [--json] nodetool apps Mini apps as portable artifacts, straight against the local database. An ApplicationBundle is one JSON file carrying the app document plus the full graph of every workflow its operations bind; inside it an operation’s workflowId is a bundle-local key, and import creates the workflows and rewrites the keys to the new ids. The bundle logic is pure and lives in @nodetool-ai/app-runtime, so the CLI, POST /api/applications/import-bundle, and the example-app installer all produce the same rows. npm run dev:nodetool -- apps list # id, name, operations, updated_at npm run dev:nodetool -- apps export-bundle <application_id> -o my.app.json npm run dev:nodetool -- apps export-bundle <id> --released # the released snapshot, not the draft npm run dev:nodetool -- apps import-bundle my.app.json --project default A bundled workflow carrying a sourceId gets a row id derived from it, so two bundles that ship the same workflow reuse the row instead of duplicating it — which is what keeps installing several example apps from filling the library with copies of one template. nodetool jobs npm run dev:nodetool -- jobs list # List jobs npm run dev:nodetool -- jobs list --workflow-id <id> # Filter by workflow npm run dev:nodetool -- jobs get <job_id> # Job details npm run dev:nodetool -- jobs get <job_id> --json Agents and MCP clients cancel a job with cancel_job (nodetool.jobs.cancel(id) in an action), and the web uses the jobs.cancel procedure. Both mark the row cancelled and stop a run executing in the server process, so a cancelled job is never overwritten as completed. nodetool assets npm run dev:nodetool -- assets list # List assets npm run dev:nodetool -- assets list --query "photo" # Search npm run dev:nodetool -- assets list --content-type image/png # Filter by type npm run dev:nodetool -- assets get <asset_id> # Asset details nodetool collections (RAG Vector Store) Manages the vector-store collections that back RAG: CRUD, document indexing, and semantic search. Runs in-process against the default vector provider (sqlite-vec unless NODETOOL_VECTOR_PROVIDER points elsewhere) — no server needed. npm run dev:nodetool -- collections list # List collections + counts npm run dev:nodetool -- collections create my_docs --embedding-model <id> npm run dev:nodetool -- collections index my_docs notes.md report.txt # Chunk + index files npm run dev:nodetool -- collections query my_docs "how does X work" -n 5 # Semantic search npm run dev:nodetool -- collections get my_docs # Metadata + document count npm run dev:nodetool -- collections delete my_docs --yes # Delete (skip confirm) nodetool costs Aggregates the per-call cost records NodeTool tracks for every provider call, read straight from the local DB — no server needed. LLM calls carry token counts; image, video and audio generation carries the billing unit, the quantity and the unit price behind the charge (costs list prints them in a units column, e.g. 5 × seconds @ $0.2050). npm run dev:nodetool -- costs summary # Overall + per-provider/model npm run dev:nodetool -- costs list --limit 20 # Recent calls npm run dev:nodetool -- costs list --provider anthropic # Filter by provider/model npm run dev:nodetool -- costs by-provider # Grouped by provider npm run dev:nodetool -- costs by-model --provider openai # Grouped by model Generation spend reaches the ledger through attachRunCostLedger (@nodetool-ai/execution), which ExecutionSession attaches to every run — so a CLI run, a debug run, an app run and the websocket server all record the same way, once. A node that knows its own charge (FAL, kie) reports it with context.setProviderCost() and that number wins, reconciled to the provider’s actual billed amount afterwards where a billing API exists. Everything else is priced off @nodetool-ai/model-pricing from the prediction message the capability call emits. A nodetool generate run records its own row: it calls the provider directly and no runner would see it. A model in no price catalog still gets a row, with a null cost. It shows in list as unpriced and is counted as unpriced in every aggregate, so the totals read as a lower bound rather than as free — an empty report is worse than no report. nodetool generations The record of every media generation — image, video, audio, 3D — read from the local database. A generation is one provider call, tracked as a predictions row opened before the call (running) and closed with its outcome (completed, failed, cancelled, or interrupted when a restart orphaned it), its cost, the provider’s request id, and the assets it produced. Every surface that asks a provider for media goes through ProcessingContext.runGeneration (or runGenerationWith around a call the capability switch has no case for), and packages/execution/tests/generation-seam-audit.test.ts fails on a provider media call outside it. Design: docs/media-generation-tracking-design.md. npm run dev:nodetool -- generations list [--status running] [--provider fal] [--capability text_to_video] [--thread-id <id>] [--job-id <id>] [--since <iso>] [--json] npm run dev:nodetool -- generations get <generation_id> --json npm run dev:nodetool -- generations await <generation_id> [--timeout 300] # exit 1 while still running npm run dev:nodetool -- generations cancel <generation_id> npm run dev:nodetool -- generations reconcile <generation_id> # ask the provider what it billed npm run dev:nodetool -- generations sweep # close orphaned rows, drain the reconcile queue once Reconciliation is a queue on the table: a row with a provider request id and no billed amount yet is retried with backoff (1, 5, 30, 120, 720 minutes) by a worker the server starts, and a provider with no billing API leaves the queue as unavailable. FAL and kie reconcile; the sweep runs at every server start. Two more commands ask the provider instead of the local table — its own record of what this account ran, from any machine or from the provider’s own site, at the price it billed: npm run dev:nodetool -- generations provider-list --provider fal_ai [--model <endpoint>] [--since <iso>] [--limit 50] [--json] npm run dev:nodetool -- generations provider-get <request_id> --provider fal_ai [--model <endpoint>] [--json] That is BaseProvider.listGenerations / getGeneration (packages/runtime/src/providers/provider-generations.ts), which a provider overrides only where it publishes a history API. Exactly those providers declare list_generations / get_generation, and every other provider throws ProviderGenerationsUnsupportedError, a capability answer rather than a failure. FAL answers both through its Platform APIs: a listing with no model filter comes from the billing feed, which records the charge and not the outcome (those rows read status: "unknown", and the page says so), while a listing filtered by endpoint reads the request history for status, timings and output urls with the costs merged in. AtlasCloud publishes a prediction lookup and no listing, so it answers provider-get only. Both need the provider’s own key, and FAL’s Platform APIs need it admin-scoped. Agents reach the same record through the generations capability module (list_generations, get_generation, await_generation, cancel_generation, reconcile_generation, plus list_provider_generations and get_provider_generation for the provider’s own record), and every generation capability (generate_image, generate_video, generate_speech, the storyboard renders, …) returns the generation_id next to the asset. background: true returns the id at once and leaves the follower to finish the job; at most 16 may be open per run, and await_generation collects them. Durable generation recovery selfcheck The generation-recovery harness exercises durable acceptance, lease fencing, FAL queue submission and binding, receipt ordering, destination attachment recovery, and signed webhook inbox deduplication with local databases and fake HTTP responses. It never calls a paid provider endpoint. It also holds the recovery decoder level with the response shapes the live FAL extractors read, so a shape the live path saves cannot be recovered as a completion with no asset. npm run dev:nodetool -- harness gate packages/execution/src/generation-recovery-worker.ts The gate selects the focused model, execution, runtime, and websocket suites registered in packages/cli/src/harness/registry.ts. Live FAL callback and billing canaries remain separate operational checks. nodetool storage Asset objects live at <userId>/<assetId>.<ext> so the owner is the leading path segment — the boundary a Supabase RLS policy or S3 bucket policy can enforce on the object itself. migrate-keys moves objects written under the older flat layout. Required on Supabase/S3 when upgrading; the local file backend falls back to the flat key on a miss. npm run dev:nodetool -- storage migrate-keys --dry-run # Report, write nothing npm run dev:nodetool -- storage migrate-keys # Move them npm run dev:nodetool -- storage migrate-keys --user-id <id> --json nodetool auth Signs in to providers that use an account instead of an API key. auth claude runs the same OAuth flow the claude CLI does and writes the tokens to the Claude Agent SDK’s credential file ($CLAUDE_CONFIG_DIR/.credentials.json, default ~/.claude/.credentials.json), so a NodeTool login and a claude login are interchangeable — the Claude Agent provider picks it up with no extra configuration. npm run dev:nodetool -- auth claude login # browser + loopback callback npm run dev:nodetool -- auth claude login --manual # paste the code (headless/remote) npm run dev:nodetool -- auth claude login --console # Console (API-billed) account npm run dev:nodetool -- auth claude status npm run dev:nodetool -- auth claude refresh --force npm run dev:nodetool -- auth claude logout The same flow is exposed over HTTP at /api/oauth/claude/{start,complete,tokens,disconnect} and as a sign-in card on the Models & Providers settings page. Details: packages/runtime/src/providers/oauth/README.md. nodetool secrets npm run dev:nodetool -- secrets list # List secret keys npm run dev:nodetool -- secrets store OPENAI_API_KEY # Store (prompts for value) npm run dev:nodetool -- secrets store MY_KEY --description "..." npm run dev:nodetool -- secrets get OPENAI_API_KEY # Print value nodetool worker (Rented GPU Workers) Provisions a RunPod/Vast worker a NodeTool instance attaches to for Python nodes, and manages the HuggingFace cache on it over the WebSocket bridge — no server needed for worker models. A worker bills by the minute, so --idle-timeout and stop are part of the flow. npm run dev:nodetool -- worker profile add hf-a40 --target runpod \ --image ghcr.io/nodetool-ai/nodetool-worker:latest --gpu "NVIDIA A40" --idle-timeout 15 npm run dev:nodetool -- worker create --profile hf-a40 --attach npm run dev:nodetool -- worker models list # attached worker npm run dev:nodetool -- worker models download --repo-id stabilityai/sdxl-turbo npm run dev:nodetool -- worker list npm run dev:nodetool -- worker stop --all Full reference: docs/cli.md § nodetool worker, walkthrough: docs/worker-deployment.md. nodetool telegram (Telegram Bridge) Turns Telegram private-chat messages into turns on a running server’s agent loop. The bridge holds no credentials and no conversation state — threads, tools, permissions, and cost tracking stay on the server, which needs NODETOOL_INTEGRATION_TOKEN set or the linking routes do not exist. Long polling only; TELEGRAM_WEBHOOK_URL makes serve refuse to start. npm run dev:nodetool -- telegram register-commands # setMyCommands (deploy step) npm run dev:nodetool -- telegram serve --config ./telegram-bot.json Env: TELEGRAM_BOT_TOKEN, NODETOOL_INTEGRATION_TOKEN, NODETOOL_API_URL. File (optional): allowUsers, editThrottleMs, maxQueuedTurns. Full reference: docs/cli.md § nodetool telegram, design: docs/telegram-bot-design.md. nodetool settings & info npm run dev:nodetool -- settings show # Show env config npm run dev:nodetool -- settings show --json npm run dev:nodetool -- info # System info, API key status npm run dev:nodetool -- info --json Global Options The read commands (workflows, jobs, assets, models) hit the local database, providers, and caches by default — no server required. Pass --api-url <url> (env: NODETOOL_API_URL) to route through a remote server instead. Observing Agent Execution NodeTool emits a hierarchy of OpenTelemetry spans that an analyzer agent can ingest to study and optimize prompts/agents/workflows: workflow.run (kernel WorkflowRunner) node.process (kernel NodeActor — one per node) agent.execute (Agent.execute) agent.plan (TaskPlanner / authorGraph / CodePlanner) llm.chat / llm.stream (BaseProvider) agent.step (CodeActExecutor) llm.chat / llm.stream IO- and CPU-bound work gets its own spans, tagged nodetool.task.kind = io or cpu so a backend can split wall time by resource: Span Kind Where <METHOD> <route> (server) — Every API request except /health and /ready. Continues an incoming traceparent GET, POST, … (client) — Every outbound fetch: provider APIs, S3, Supabase Storage, safeFetch workspace.read / write / list / materialize / absorb io StorageWorkspace subprocess.run cpu runHostBinary, the video nodes’ ffmpeg/ffprobe, server-side thumbnails and audio extraction image.encode_png / rasterize_svg / extract_region / thumbnail cpu Image codec helpers and asset thumbnails audio.peaks, video.encode_proxy, ffmpeg.trim_video_window cpu Waveform peaks, preview proxies, video-edit source windows storage.store / retrieve / delete / list io A run’s asset storage (ProcessingContext.storage and assetStorage), tagged with the backend class Other activity spans: Span Where provider.<method> Every non-chat provider call: image, video, audio, 3D, embeddings and rerank. Carries gen_ai.system, gen_ai.operation.name and gen_ai.request.model, never the prompt python.execute / python.execute_stream A node run in the Python worker, with node.type ws.command Every /ws command except stream_input frames, with rpc.method. Like an API request span, it starts its own trace Wrap new IO or CPU work with withTaskSpan(kind, name, attributes, fn) from @nodetool-ai/runtime. It runs fn directly when telemetry is off. Declare a new span name in SAFE_SPAN_NAMES and each metadata attribute in SAFE_TRACE_ATTRIBUTES (packages/protocol/src/run-trace.ts). An undeclared name or attribute is stored as owner content, so external sinks and retained summaries see content.span without it. packages/protocol/tests/run-trace-span-names.test.ts fails when a literal span name in the source is undeclared. Every llm.chat / llm.stream span carries gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.total_tokens, and gen_ai.usage.cost_usd. Token counts also appear in the llm_call message events emitted by BaseProvider. Sinks Multiple sinks can run simultaneously (each gets its own span processor). The network sink sends OTLP/HTTP JSON. Configure a collector that accepts JSON on its HTTP traces endpoint. # JSONL log file (analyzer-friendly — one span per line) NODETOOL_TRACE_FILE=/tmp/nodetool-trace.jsonl npm run dev:chat -- --agent npm run dev:chat -- --agent --trace-file /tmp/nodetool-trace.jsonl # Stdout — pretty (human) or json (JSONL) NODETOOL_TRACE_STDOUT=pretty npm run dev:chat -- --agent npm run dev:chat -- --agent --trace-stdout pretty npm run dev:chat -- --agent --trace-stdout json # OpenTelemetry — Traceloop cloud TRACELOOP_API_KEY=your-key npm run dev:chat -- --agent # OpenTelemetry — custom OTLP backend (Jaeger, Grafana, etc.) OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npm run dev:chat -- --agent # Full traces URL, used as given; OTEL_EXPORTER_OTLP_HEADERS adds auth headers OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://otlp.example.com/v1/traces npm run dev:server # Debug logging (all LLM calls, planning details) NODETOOL_LOG_LEVEL=debug npm run dev:chat -- --agent The --trace-file and --trace-stdout flags also work on the nodetool CLI: npm run dev:nodetool -- --trace-file trace.jsonl run workflow.ts npm run dev:nodetool -- --trace-stdout pretty workflows run <id> JSONL trace schema Each line in the file is one span: { "trace_id": "...", "span_id": "...", "parent_span_id": "...", "name": "agent.plan", "kind": "INTERNAL", "start_time_ms": 1700000000000, "end_time_ms": 1700000001234, "duration_ms": 1234, "status": { "code": "OK" }, "attributes": { "agent.objective": "...", "agent.kind": "plan", "agent.provider": "anthropic", "agent.model": "claude-sonnet-5", "gen_ai.usage.input_tokens": 150, "gen_ai.usage.output_tokens": 80 }, "events": [], "resource": { "service.name": "nodetool" } } See packages/agents/AGENTS.md for agent architecture, parallel execution, skills, and tuning. App runs phase 1 Run the durable-operation check from the repository root: npm run test --workspace=packages/execution -- app-operation app-run-generation The fixtures open a server-owned app run before execution, propagate its identity to child work, fold its outputs into revisioned instance state, and read its generation attachment back from the database. They also check transport retries, failed preparation, cancellation, and stale state writes. The app-runs entry in the harness registry selects this check for instance, run, and app-runtime changes. Run traces phase 2 Run the stored-trace check from the repository root: npm run test --workspace=packages/execution --workspace=packages/runtime --workspace=packages/models --workspace=packages/websocket -- run-trace external-trace-privacy http-tracing The fixture executes a scripted app with a scripted provider and reads the stored app.run, script.run, capability.call and llm.* tree without an external sink. It checks captured request/response and activity events, run-resolved secret masking and content-free visitor runs. The bridge check verifies that live readers receive committed snapshots only. The same check exercises external content filtering, ownership, deletion and retention. The run-traces entry in the harness registry selects this check for the trace store and its execution hosts. Model and runtime suites cover content separation, duplicate events, limits, parent deletion, late writes, expiry, export/erasure, external sink filtering and recording failures. HTTP and WebSocket checks reject cross-owner ancestry. The model PostgreSQL test uses an isolated test database and actual non-bypass roles to verify owner reads and denied client writes. Set NODETOOL_TEST_POSTGRES_URL to a local PostgreSQL administrative connection to run that check. Run readers phase 3 nodetool runs list, show, trace, logs and tail read the same owner-scoped service as the tRPC runs router and nodetool.runs.* capabilities. Start with show <id> --json, then focus trace on the span named in summary.first_failed_span_id. Resource ids accept their exact 12-character prefixes. Trace and span ids remain full OpenTelemetry ids. Run the reader check from the repository root after building packages: npm run test --workspace=packages/execution --workspace=packages/models --workspace=packages/websocket --workspace=packages/agents --workspace=packages/cli -- runs-phase3 app-debug-versioned-targets app-debug-workflow-inputs-phase3 capabilities-runs runs-command runs-integration run-readers job-find runs-trpc sandbox-api-coverage The check invokes the real CLI on a persisted failing script and verifies its failed span and ancestor path. It compares reader adapters, exercises bounded queries and cancellation, and checks that executing debug_app operations return run ids with origin debug. Static debug validation creates no run. Debug workflow runs with widget property bindings record their inputs as parameters and node_properties, alongside the pinned workflow snapshot. Traced jobs read stored events even when content expired. Only jobs without a registered trace may fall back to their legacy logs. tail --json emits newline-delimited JSON with snapshot updates and a terminal result. The store retains the latest snapshot of each span, so readers receive an explicit resnapshot indicator when replay cannot reconstruct intermediate updates. The CLI deduplicates event identities across polling and resume. See run commands for filters and output flags. The run-readers entry in the harness registry selects the check for these surfaces. Run inspection phase 4 The bottom panel’s Trace and Logs tabs share a durable run picker. The picker lists app runs, workflow jobs and chat turns. Browser spans appear above server spans, and focused spans expose their bounded content. AgentActivity reloads stored events by instance and operation. “View trace” opens the producing run, and “Ask the agent” passes run and span ids into chat. Run the browser-ingestion check from the repository root after building packages: npm run test --workspace=packages/websocket -- browser-run-spans-phase4 browser-app-run-phase4 app-runs-workflow The check uses production HTTP routes and the native UI harness. It verifies owner and visitor restrictions, payload caps, retries, cycle rejection, server-span protection, browser-root recovery and first-run W3C ancestry. The browser-run-spans and run-inspection-ui entries in the harness registry run ingestion, recorder, replay and panel checks when these surfaces change. The real browser journey starts a keyless app operation, reads its stored ui.action → app.run ancestry, reloads the page, recovers the operation’s run, and checks Trace and Logs against the reader API. A second journey runs the real in-browser kernel and reads its app.run → workflow.run → node.process ancestry from the same store: npm run test:journeys --workspace=web -- run-observability.spec.ts The registry lists this browser journey as a manual check. It owns development servers through the existing journey fixtures. To isolate ports, set SCREENSHOT_BACKEND_PORT, SCREENSHOT_WEB_PORT and PROXY_API_TARGET as described in web testing. App instances and history phase 5 Instance management and historical results use the existing instance API and common run readers. The app-instance-history selfcheck covers bounded instance lists, operation-specific run recovery, revision-checked version advancement, and preservation of old execution snapshots: npm run test --workspace=packages/models --workspace=packages/execution --workspace=packages/websocket -- app-instance run-list-app-metadata runs-phase3 app-operation The browser completion journey uses real persistence and scripted execution. Run it after building packages: npm run test:journeys --workspace=web -- app-instances.spec.ts run-observability.spec.ts The registry lists browser journeys as manual checks. Instance tabs retain the application and instance IDs separately. Selecting a historical run is inspection state and does not replace working variables or invoke an operation.--- # HuggingFace Integration Source: https://docs.nodetool.ai/huggingface.html HuggingFace Integration NodeTool uses Hugging Face in five ways. Each has different requirements. Surface Runs Needs Where it appears HuggingFace provider Hosted, on Hugging Face Inference Providers HF_TOKEN Generic nodes (chat, image, video, TTS, ASR, embeddings) huggingface.* nodes Hosted, on the same router HF_TOKEN One node per Inference Providers task transformers.* nodes Locally, in-process, ONNX The Transformers.js runtime, no key Local small models nodetool-huggingface pack Locally, Python (Diffusers, Transformers) The pack, from the Package Manager Local image, audio, and speech models Model Manager Hub source Downloads Hub repos to disk A token for gated repos Models Manager HuggingFace provider The provider (id huggingface) routes generic nodes to Hugging Face Inference Providers through the @huggingface/inference client. It offers text-to-image, image-to-image, text-to-video, text-to-speech, speech recognition, and embeddings, plus chat. Tool calling is off. Model lists come from the Hub. For each task NodeTool asks for models with warm inference, sorted by likes, limited to 100, and caches the answer for 10 minutes. A token is required. Without one the provider does not register as configured. If the @huggingface/inference package is missing, the provider reports it instead of failing at the first call. See Providers for how it fits with other providers. huggingface.* task nodes Nineteen nodes call the Inference Providers router at https://router.huggingface.co. Chat completion uses the OpenAI-compatible /v1/chat/completions route. Every other task posts to /hf-inference/models/{model}. Each node has a model field that defaults to a recommended model for its task. Enter any repo id that serves that pipeline to use another. Group Nodes Text Chat Completion, Text Generation, Summarization, Translation, Fill Mask, Question Answering, Table Question Answering, Feature Extraction, Text Classification, Token Classification, Zero Shot Classification Image Text to Image, Image to Image, Image Classification, Image Segmentation, Object Detection Audio Automatic Speech Recognition, Audio Classification Video Text to Video Text to Image takes a prompt, negative prompt, width, height, guidance scale, inference steps, and seed. Node references are in the huggingface namespace. These nodes need a Hugging Face access token with the Inference Providers permission, stored as HF_TOKEN. HUGGINGFACE_API_KEY is also accepted. Without a token the node fails with HF_TOKEN is not configured. Transformers.js nodes (local ONNX) The transformers.* nodes run ONNX models inside the NodeTool backend. They need no key and no Python. Text: Text Classification, Token Classification, Question Answering, Summarization, Translation, Text Generation, Fill Mask, Feature Extraction, Zero Shot Classification Vision: Image Classification, Object Detection, Image To Text, Zero Shot Image Classification Audio: Automatic Speech Recognition, Audio Classification, Text To Speech Install the Transformers.js runtime from the Package Manager first. Without it, the first run fails and names the missing runtime. Models must be ONNX exports, usually from the Xenova/* or onnx-community/* Hub organizations. Each node defaults to a recommended repo. Models download on first use into <data-dir>/transformers-js-cache, or the directory in the TRANSFORMERS_JS_CACHE_DIR setting. This cache is separate from the Hugging Face Hub cache. The same models also serve the transformers_js provider for chat, TTS, ASR, and embeddings. Node references are in the transformers namespace. Local Python pack The optional nodetool-huggingface pack adds local Diffusers and Transformers nodes for image, audio, and speech models. Install it from Package Manager → Python packs, which sets up Python first if it is missing. It runs on PyTorch 2.14, and the desktop app installs the PyTorch build that matches your graphics card. See GPU requirements. Without the desktop app, see Python nodes without the desktop app. NodeTool does not install it by default. When the Python worker runs, its local Hugging Face provider appears as huggingface-local, so it does not clash with the hosted huggingface provider. The Model Manager recognizes the model types this pack uses, such as Flux (including Kontext, Canny, Depth, and Redux), Stable Diffusion 1.5, XL, and 3, Qwen Image and Qwen Image Edit, ControlNet, IP Adapter, and LoRA. MLX variants are also detected. The pack’s own node reference ships with the pack. Authentication and gated models Create a token at huggingface.co/settings/tokens. Which token a feature reads depends on the feature. Feature Token source HuggingFace provider and huggingface.* nodes The stored HF_TOKEN secret, then the HF_TOKEN environment variable. Paste it into the HuggingFace card in Settings → Models & Providers, or run nodetool secrets store HF_TOKEN Model Manager downloads, local or to an attached worker The stored HF_TOKEN secret, then the HF_TOKEN, HF_API_TOKEN, or HUGGING_FACE_HUB_TOKEN environment variable, then the token file from hf auth login ($HF_TOKEN_PATH, or token under $HF_HOME, default ~/.cache/huggingface/token). The server sends the token to a worker, which has no token of its own nodetool models download-hf The environment variables, then the token file. The stored secret is not read Hub search in the Model Manager The HF_TOKEN or HUGGING_FACE_HUB_TOKEN environment variable, sent when set Downloads a Python node starts while it runs The stored HF_TOKEN when the node declares it, otherwise the worker’s environment and token file A token pasted in Settings covers Model Manager downloads. For a gated repo, download the model in the Model Manager before you run the workflow. A Python node that downloads on its own may not receive the Settings token. In that case set HF_TOKEN in the environment before starting NodeTool, or run hf auth login. Gated models Some repos, such as FLUX.1 dev, require you to accept a license: Open the model page on huggingface.co while signed in. Accept the license or choose Request access, and wait if approval is manual. Make sure the token above has read access, then retry the download. A blocked download shows a message starting Hugging Face blocked this download with these steps. Model cache Hub downloads use the standard Hugging Face hub cache, ~/.cache/huggingface/hub by default. HF_HUB_CACHE, HUGGINGFACE_HUB_CACHE, HF_HOME, or XDG_CACHE_HOME relocate it, in that order of precedence, the same as the huggingface_hub library. The Model Manager’s Installed source lists what is in it and can delete cached models. See Models Manager. Troubleshooting HF_TOKEN is not configured. A huggingface.* node ran without a token. Add HF_TOKEN in Settings or the environment. The token needs the Inference Providers permission. @huggingface/inference is required for HuggingFaceProvider. The client package is missing. It is a dependency of @nodetool-ai/runtime, so a normal install has it. Reinstall NodeTool, or run npm install in a source checkout. 401 or 403 on a download. The repo is gated or private, or the token is missing. Follow Gated models. The CLI and Python nodes may not read the Settings token. See the table in Authentication and gated models. A transformers.* node fails on first run. Install the Transformers.js runtime from the Package Manager. A model is missing from the HuggingFace provider’s picker. The list contains only the 100 most-liked models with warm inference per task. Use a huggingface.* node and enter the repo id, or download the model through the Model Manager. Related documentation Models Manager: browse, download, and delete local models Supported Models: local engines and model types Providers: every provider and its key huggingface nodes and transformers nodes: node reference Hugging Face provider guide: how the provider is built Hugging Face Hub--- # image-editor.md Source: https://docs.nodetool.ai/image-editor.html This page has moved. Please go to Sketch Editor.--- # Indexing & Vector Stores Source: https://docs.nodetool.ai/indexing.html NodeTool ships a lightweight ingestion pipeline for semantic search and retrieval-augmented generation (RAG) tasks. The indexing logic is split across @nodetool-ai/vectorstore (store, chunking, and embedding) and @nodetool-ai/websocket (the upload route). Overview Collection metadata (CollectionResponse in @nodetool-ai/protocol packages/protocol/src/api-types.ts) carries the collection’s name, document count, free-form metadata, and workflow_name — the resolved name of the workflow id stored under metadata.workflow, shown in listings. Vector store – the default backend is SQLite-vec (@nodetool-ai/vectorstore packages/vectorstore/src/sqlite-vec-store.ts). Embeddings flow through the VectorProvider abstraction — see Vector Storage for swapping backends (Supabase/pgvector, or the Pinecone stub). Indexing route – handleCollectionRequest() (@nodetool-ai/websocket packages/websocket/src/collection-api.ts) handles the multipart upload and performs the ingestion itself. It is the one collection endpoint that stayed on REST; every other CRUD and query endpoint moved to the tRPC collections router. Default Flow handleCollectionRequest() matches POST /api/collections/:name/index, resolves the collection through getDefaultVectorProvider().getCollection(), and checks ownership with canAccessCollection() — someone else’s collection answers 404, not 403, so the endpoint cannot be used to probe for names. The request must be multipart/form-data with a file field. Otherwise the route answers 400, and 405 for a method other than POST. A file above getMaxUploadBytes() (NODETOOL_MAX_UPLOAD_BYTES, 1 GiB by default) answers 413. The file is read as UTF-8 text, split with splitDocument() from @nodetool-ai/vectorstore (2000 characters per chunk, 1000 overlap), and upserted into the collection as <file name>#<n> chunks carrying source and start_index metadata. An empty or whitespace-only file produces zero chunks. The route reads text only, so it does not extract text from PDF, DOCX, or other binary formats. It also passes no embedding function when it opens the collection, so it stores chunk text without calling the collection’s embedding model. To embed during ingestion, use the vector.Index* nodes below. The route returns { path, chunks, error }. A CollectionNotFoundError becomes 404; any other provider error is logged and returned as a generic 500, because provider errors carry SQL text, file paths, and upstream URLs. Indexing from a workflow Index nodes write to a collection and query nodes read from it. Each takes a collection input (see the vector.Collection node, which gets or creates a collection by name and embedding model). Node Purpose vector.IndexString Index a string under a document id. vector.IndexTextChunk Index a single text chunk. vector.IndexAggregatedText Index several text chunks at once with aggregated embeddings from Ollama. vector.IndexImage Index a list of image assets or files. vector.IndexEmbedding Index a single precomputed embedding vector, with optional metadata. vector.QueryText, vector.QueryImage Similarity search by text or image. vector.HybridSearch Fuses a semantic ranking with a keyword-filtered one by reciprocal rank fusion. vector.Count, vector.GetDocuments, vector.Peek Inspect a collection. vector.RemoveOverlap Removes overlapping words between consecutive strings in a list. Definitions are in packages/core-nodes/src/nodes/vector.ts. To index a PDF, extract its text first with a node such as lib.pdf.ExtractText. Configuring the vector store The default backend is local SQLite-vec. Switch backends with NODETOOL_VECTOR_PROVIDER. Variable Description Default NODETOOL_VECTOR_PROVIDER sqlite-vec, pinecone, or supabase sqlite-vec VECTORSTORE_DB_PATH Local SQLite-vec database file vectorstore.db in the NodeTool data dir ($XDG_DATA_HOME/nodetool, falling back to ~/.local/share/nodetool) PINECONE_API_KEY Required to select pinecone, which is a stub that fails on use — SUPABASE_URL and SUPABASE_KEY (or SUPABASE_SERVICE_ROLE_KEY) Required when provider is supabase. Install the SQL migration once. — See Vector Storage for backend-specific setup. CLI & API Integration POST /api/collections/:name/index (see @nodetool-ai/websocket packages/websocket/src/collection-api.ts) triggers ingestion via HTTP (multipart/form-data file upload). The MCP server (@nodetool-ai/websocket packages/websocket/src/mcp-server.ts) exposes the CodeAct action tool execute_code plus view_image and the direct set. An IDE plug-in reads collections from inside an action — nodetool.collections.list() and nodetool.collections.query(). It does not index assets. A deployed server exposes no separate admin ingestion endpoints. The /admin/* surface @nodetool-ai/deploy’s client used to carry was ported from the retired Python server and never mounted; use POST /api/collections/:name/index against the deployment instead. Troubleshooting Supabase errors – verify SUPABASE_URL, the key, and that packages/vectorstore/sql/supabase-migration.sql ran on the project. Fall back to local SQLite-vec by setting NODETOOL_VECTOR_PROVIDER=sqlite-vec. Pinecone errors – the provider is not implemented. Use sqlite-vec or supabase. Large files – ensure VECTORSTORE_DB_PATH has disk headroom, or move to a remote backend. Related Documentation Providers – selecting embedding models for ingestion nodes. Workflow API – details on RunJobRequest. Storage Guide – configuring persistent storage for uploaded documents.--- # Install NodeTool on Windows, macOS, or Linux Source: https://docs.nodetool.ai/installation.html Download, install, open, connect a provider. There is no setup wizard, but NodeTool can’t run an agent or generate an image until it has a model to call, so connecting a provider is part of installing. The larger pieces some workflows need, such as local AI models, download later and only when you actually use them. The short version Download NodeTool from nodetool.ai Run the installer Open NodeTool Connect an AI provider — see below The file you get is a .dmg on macOS, an .exe on Windows, and an AppImage on Linux. Exact steps for each are below: macOS, Windows, Linux. macOS Download the .dmg from nodetool.ai. There are two builds: one for Apple Silicon (arm64) and one for Intel (x64). To see which Mac you have, open the Apple menu → About This Mac. Open the downloaded file and drag Nodetool into your Applications folder. Open it from Applications. Released builds are signed and approved by Apple, so macOS should not warn you or block the first launch. If a workflow records audio or video, macOS asks once for microphone or camera permission. Approve it or those nodes won’t work. On Apple Silicon Macs with macOS 14 or newer, local models can run on Apple’s MLX framework. MLX is an optional Python pack: install MLX from Tools → Package Manager → Python packs, which sets up Python first if it is missing. Intel Macs and Macs on macOS 13 or older cannot run MLX or the HuggingFace pack. They run cloud providers, Ollama, and llama.cpp. Windows Download the installer (Nodetool-Setup-<version>.exe) from nodetool.ai. It is built for 64-bit x64 Windows. Run it. You choose where it installs. It then adds a desktop shortcut and opens NodeTool when it finishes. Approve the Windows firewall prompt. NodeTool runs a small server on your own machine (on port 7777) that the app window talks to. Nothing is exposed to the internet. Both the installer and the app are code-signed. To run local models on your graphics card, see GPU requirements. Linux Download the AppImage from nodetool.ai. It is the only Linux package NodeTool ships today. Mark it executable and run it: chmod +x Nodetool-*.AppImage ./Nodetool-*.AppImage There is no install step. An AppImage is a single self-contained file that runs wherever you put it. Prefer Flatpak? Unsigned builds are produced from every change to the project. See Flatpak CI Builds. They are not on Flathub yet. Connect an AI provider A fresh install has no model behind it. Agents, chat, image and video generation, speech — every one of them calls a model, and until NodeTool has somewhere to call, those nodes fail with a missing-provider error. Do this before your first workflow. Two ways to get one, and you can mix them: A cloud provider (fastest). Open Settings → Models & Providers and connect one. Some providers sign you in without a key: a Claude subscription (desktop app only), a ChatGPT account (Codex), and Hugging Face. For the rest, paste an API key: a password-like string you create on that company’s website, which lets NodeTool use your account there. OpenAI — chat, images, video, speech, transcription, embeddings. The broadest single key. Anthropic — Claude chat models, the usual choice for agents. Text only. Google Gemini — chat, images, Veo video, speech, transcription, embeddings. FAL or Replicate — image, video, and audio generation across many models. Note: NodeTool is bring-your-own-key. It never marks up a provider’s price, and the provider bills you directly. Keys are stored encrypted (AES-256-GCM) in a local database, not in a plaintext config file. The encryption key lives in your operating system’s keychain (macOS Keychain, Windows Credential Manager, or the Linux Secret Service). The SECRETS_MASTER_KEY environment variable overrides it. Local models (no key, no bill). Install Ollama, pull a model with ollama pull <model>, and it shows up in NodeTool automatically. This covers chat and embeddings, so agents work. Image and video generation mostly need either a cloud provider or a graphics card and the local model downloads described below. Wherever a missing provider blocks you — a model dropdown, a node warning, the getting-started checklist — NodeTool opens the connect dialog in place, so you don’t have to hunt through settings. Each connected provider carries a Test button that calls the provider to confirm the key still works. From the terminal: nodetool secrets store OPENAI_API_KEY # prompts for the value, stores it encrypted nodetool secrets list # list stored keys (values are never shown) nodetool secrets get OPENAI_API_KEY # print one stored value The full list of 30+ providers, what each one can generate, and which key it needs is in Providers. What downloads later The app itself is small. NodeTool starts without a Python environment. These pieces arrive only when you install them: Python and Conda. Some nodes are written in Python rather than JavaScript, and they need this to run. Install it from Tools → Package Manager → Software, or accept the Install required runtimes prompt that appears when a workflow contains nodes that need a runtime. The prompt installs runtimes only. A workflow with no Python nodes never needs Python. Allow several GB of disk space. Python node packs. HuggingFace, MLX, and Wan2GP nodes come in packs you install from Tools → Package Manager → Python packs. Installing a pack sets up Python first if it is missing. The HuggingFace pack downloads a PyTorch build for your graphics card, which takes several GB. See Node Packs. In-process model runners. llama.cpp (the llama.cpp local provider) and whisper.cpp run inside NodeTool. Install them from Package Manager → Software. Ollama is a separate program. Download it from ollama.com and keep it running. NodeTool does not ship or start it. A llama-server you run yourself connects through LLAMA_CPP_URL. See Providers. The models themselves. Several GB each, depending on the model. Download them from Tools → Model Manager. No graphics card, or no room for the downloads? Use a cloud provider with your own API key instead — see Connect an AI provider above, Providers, and Models & Providers. What different tasks need What matters is the kind of hardware you have, not the exact model of graphics card: Your hardware What runs the model Good for NVIDIA graphics card HuggingFace pack (PyTorch), llama.cpp/GGUF, Ollama Making images and video, running compressed language models AMD graphics card (Linux) HuggingFace pack (PyTorch ROCm build), llama.cpp, Ollama Same as NVIDIA Apple Silicon Mac MLX pack, HuggingFace pack, llama.cpp, Ollama Language models, image understanding, Flux No graphics card, CPU only llama.cpp, Transformers.js, Ollama Works, but slowly Anything, using an online service The provider’s servers Every kind of task, nothing to download Supported Models compares all of them. GPU requirements Python nodes run on PyTorch 2.14. Which PyTorch build you get, and so which cards and drivers work, depends on how you installed Python nodes. Rows marked inferred follow from PyTorch’s published platform support and have not been tested on every card. Desktop app. Before it installs the HuggingFace or MLX pack, the app detects your GPU and picks the matching PyTorch build: Hardware Build the app installs NVIDIA Turing (RTX 20xx, GTX 16xx) or newer CUDA 12.8, which needs driver 570 or newer NVIDIA Maxwell, Pascal (GTX 10xx) or Volta CUDA 12.6 (inferred) NVIDIA Kepler and older CPU. The app logs that the card is too old AMD on Linux, RDNA 2 or newer (RX 6000 and later) ROCm 7.2 (inferred) AMD on Linux, older cards (Polaris, Vega, RDNA 1) CPU, with a warning AMD on Windows CPU, with a warning. PyTorch reaches these cards only through DirectML, which has no 2.14 build. llama.cpp still uses the card through Vulkan Intel Arc and Intel integrated graphics (Windows, Linux) Intel XPU (inferred) Apple Silicon, macOS 14 or newer The PyPI build, which uses Metal (MPS) Intel Mac, or a Mac on macOS 13 or older None. The HuggingFace and MLX packs are hidden, because PyTorch 2.14 and MLX publish no build for them If PyTorch publishes no 2.14 build for the chosen GPU index, the app installs the CPU build instead and logs a warning. Python nodes then run on the CPU. After a driver update, install the pack again to retry the GPU build. Detection runs on every install. install.sh. On Linux, install.sh lets uv pick the CUDA, ROCm or Intel XPU build from the installed driver (--torch-backend auto). On macOS it installs the PyPI build, which needs Apple Silicon and macOS 14 or newer. Manual pip install torch. The PyPI build on Linux uses CUDA 13.0, which needs driver 580 or newer and a Turing or newer card. On a GTX 10xx or with an older driver, Python nodes fail with a CUDA error. Install a CUDA 12.6 or 12.8 build from pytorch.org instead, or set NODETOOL_TORCH_DEVICE to cpu. PyPI ships only a CPU build for Windows, so a GPU build on Windows always comes from pytorch.org. Docker worker image. The worker image installs PyPI’s CUDA 13.0 build, so the GPU host needs NVIDIA driver 580 or newer. See Worker deployment. llama.cpp and Ollama do not use PyTorch, so these limits do not apply to them. On a card PyTorch cannot use, they still run on the GPU. Rough VRAM (or unified memory on a Mac) per model family: Model family Memory 7–8B language model, 4-bit GGUF or MLX 6–8 GB SDXL 8–12 GB Flux, full precision 24 GB or more; less with CPU offload, but slower Wan video models 8 GB and up, depending on resolution and length The Model Manager’s Get Started tab estimates fit for your machine. To pick which GPU Python nodes use, set NODETOOL_TORCH_DEVICE in Settings to cuda, cuda:1 (a second card), mps, or cpu. See Configuration. Other ways to install If you don’t want the desktop app: Command line only — install the nodetool command without the desktop app. With npm (Node.js 22 or newer): npm install -g @nodetool-ai/cli nodetool serve # API on 127.0.0.1:7777; use --host and --port to change This runs every TypeScript node. Python nodes need a Python environment, which you set up yourself: see Python nodes without the desktop app. See the CLI Reference. On your own server (Docker) — run NodeTool’s backend on your machine or a remote host: cp .env.example .env docker compose up -d See Self-Hosted Deployment for logins, upgrades, and remote hosts, or the Deployment Guide for the full picture. From source code — for people who want to change NodeTool itself: nvm use npm install npm run build:packages npm run dev Needs the Node.js version in .nvmrc. For Python nodes, set up a Python environment as described next. Python nodes without the desktop app The desktop app manages Python for you. With the npm CLI or a source checkout, you create the environment yourself. The server starts Python nodes by running python -m nodetool.worker --stdio, so nodetool-core and every pack you want must be installed in the interpreter it picks. On Linux and macOS, install.sh does steps 1 to 3 below for you. It creates a Python 3.11 environment with ffmpeg in ~/.local/share/nodetool/conda_env and installs nodetool-core plus the packs you name with --pack (huggingface, mlx, or wan2gp): curl -fsSL https://raw.githubusercontent.com/nodetool-ai/nodetool/main/install.sh | bash -s -- -y --pack huggingface It prints the nodetool serve command to run at the end. On Linux the server finds that environment by itself. On macOS it does not search there, so always start the server with NODETOOL_PYTHON set as the script shows. To set up the environment by hand: Create an environment named nodetool with Python 3.11 or newer: conda create -n nodetool python=3.11 ffmpeg -c conda-forge conda activate nodetool A plain python -m venv works too. Install ffmpeg separately then. Install PyTorch for your hardware before any pack that needs it. On Linux and macOS, pip install torch gets the CUDA 13.0 build or the Apple Silicon build. On Windows, or for a ROCm or other CUDA build, use the command from pytorch.org/get-started for PyTorch 2.14. Check GPU requirements first. Install nodetool-core and the packs from PyPI in one command, so pip resolves their shared dependencies together: pip install nodetool-core nodetool-huggingface # add nodetool-mlx on Apple Silicon nodetool-wan2gp is the Wan2GP pack. See Wan2GP. Start the server so it finds that interpreter: NODETOOL_PYTHON="$CONDA_PREFIX/bin/python" nodetool serve On Windows, NODETOOL_PYTHON is %CONDA_PREFIX%\python.exe. In a venv, use NODETOOL_PYTHON="$VIRTUAL_ENV/bin/python", or %VIRTUAL_ENV%\Scripts\python.exe on Windows. With the conda environment named nodetool active, the server finds it without the variable. The server tries NODETOOL_PYTHON first. Without it, it uses the active conda environment when that environment is named nodetool or conda_env, then the desktop app’s environment and nodetool environments under miniconda3 or anaconda3. The full order is in Configuration. To check the environment, run python -c "import nodetool.worker" in it. With no match, the server log says Python not found — Python nodes will not be available. When the worker fails to start, the error names the interpreter it tried. If installing goes wrong Most install problems are one of these. For problems that show up once NodeTool is running, see Troubleshooting. A node says no provider is configured — nothing is connected yet, or the key you pasted is for a provider that can’t do what the node asks (Anthropic makes no images, FAL runs no chat). Connect one in Settings → Models & Providers and press Test; the capability matrix shows which provider covers which modality. The Python install fails — it needs an internet connection and free disk space. Retry from Tools → Package Manager. NodeTool doesn’t see my graphics card — on NVIDIA, open a terminal and run nvidia-smi. That is the same check NodeTool runs for Help → System Information, and it shows the driver version. Compare it with GPU requirements. On a card PyTorch does not support, set NODETOOL_TORCH_DEVICE to cpu so Python nodes run on the CPU, or use an online service instead. A model download stalls or fails — usually disk space or network. Model Download Troubleshooting covers disk space, resuming, and HuggingFace download limits. The app can’t reach its own server — approve the firewall prompt for NodeTool’s local server on port 7777. Running the Docker version instead? See Deployment Troubleshooting. Still stuck — ask on Discord or open a GitHub Issue. Include your operating system and your NodeTool version (Help → System Information). Uninstalling Windows — Settings → Apps → Nodetool → Uninstall. macOS — drag Nodetool from Applications to the Trash. Linux — delete the AppImage file. Your settings live in ~/.config/nodetool/settings.yaml (macOS and Linux) or %APPDATA%\nodetool\settings.yaml (Windows). Delete that folder too if you want to start completely fresh. Next steps Quick StartTurn one sentence into a finished video. Your First WorkflowBuild a four-node image workflow from an empty canvas. ProvidersWhat each provider can generate and which key it needs. Models & ProvidersChoose models or run them locally.--- # JavaScript Sandbox Source: https://docs.nodetool.ai/javascript-sandbox Navigation: Architecture Chat & Agents CodeAct design Sandbox packages For the saved script document editor that runs in this sandbox, see JS Scripts. Every piece of JavaScript NodeTool did not write itself runs in one place: a QuickJS WebAssembly guest built by runInSandbox (packages/agents/src/js-sandbox.ts). A workflow’s Code node, an agent’s code action and a planner’s graph program all enter through that function, and they all get the same engine, the same limits, and the same marshaling rules. The guest has its own heap inside the WASM instance, so a runaway or hostile program cannot corrupt the host V8 heap the way it could under node:vm. What it can reach is a curated set of host bridges, and each one is a capability the caller granted for that run. Where it runs Caller Code What runs nodetool.code.Code packages/code-nodes/src/nodes/code-node.ts A user’s node body, with dynamic inputs on inputs CodeAct step / chat turn packages/agents/src/codeact/ One model-written action per execute_code call validate_workflow with code packages/agents/src/graph-dsl.ts A legacy graph DSL program with no host access at all Browser runner packages/workflow-runner/ The same Code nodes, in the page, fetching modules over HTTP One engine (loadQuickJs, the quickjs-ng release variant) is loaded once per process and shared. Each invocation gets a fresh runtime and context. Anatomy of a run Resolve limits. resolveSandboxLimits applies defaults and clamps every caller override to a hard ceiling. A caller can tighten a limit, or raise it within bounds, but never switch a protection off. Build the bridges. buildSandbox constructs the host-side objects (fetch, workspace, crypto, …) bound to this run’s context, limits and abort signal. Each async bridge is wrapped in neverReject and guardAbort. Build the entry module. buildEntryModule parses the code with acorn, hoists static import declarations above the wrapper, and emits the rest as the body of a top-level-awaited async IIFE, so return value becomes the module’s default export. Code acorn cannot parse falls through to wrapCode unchanged, so a syntax error reaches the user as they wrote it. Install the module loader — always, because it is also what the wrapper’s own Node-compat preamble resolves through. That preamble compiles ~12KB of polyfills into every fresh runtime, and the prelude below deletes most of what they install, so BOOTSTRAP_MODULE_SOURCES (sandbox-bootstrap-modules.ts) serves four of them as empty modules and node:util as the two classes the guest keeps. node:url still comes from the wrapper: URL and URLSearchParams are guest capabilities. Measured: 5.96ms → 3.28ms of setup per run. Guest specifiers resolve only what the run declares; without modules, every guest import resolves nothing. Init prelude. eval, Function and the wrapper’s unconditional stubs (Buffer, process, env, Headers, Request, Response, performance) are deleted. Four of those no longer exist to delete — the bootstrap above never installs them — and the deletions stay as the guard for a wrapper release that puts them back some other way. The entry module additionally deletes the timer globals, which the wrapper library re-installs on every evaluation. Run, under an interrupt handler on a CPU deadline and a wall-clock race. Serialize. The guest encodes its result with the JSON transport (see Marshaling rules) and the host decodes it, then serializeResult walks the returned value (cycle-safe, depth-capped at 32) converting typed arrays at any depth, then truncates to maxOutputSize. Object-typed globals are deep-replaced on the host, which is how the Code node’s state survives between invocations. The guest surface Two kinds of thing, and the difference matters: capabilities are globals, libraries are imports. Capabilities (globals) Global What it does console.* Log lines land in the run’s logs array, capped by count and bytes (MAX_COLLECTED_CONSOLE_LOGS / MAX_COLLECTED_CONSOLE_CHARS in js-sandbox.ts) with one notice line when cut. A Code node also posts each line as a log_update fetch(url, options?) HTTP, returning {ok, status, headers, body, json, text(), arrayBuffer(), bytes()}. A Uint8Array body is sent as raw bytes workspace.* read, write, list, readBytes, writeBytes, stat, root, copy, move, mkdir, remove. Needs a ProcessingContext getSecret(name) The run’s secret store, limited to the run’s declared secret scope nodetool.secrets.* get (throws when unset), tryGet, list — over the same bridge sleep(ms) The only timer crypto.* randomUUID, getRandomValues, digest, hmac (WebCrypto-backed; SHA-1/256/384/512) format.* number, date, relativeTime, list — host Intl, which QuickJS does not ship. All four are async image.* info, stats, decode, blank, pad, grid, resize, crop, rotate, flip, adjust, composite, convert. Transforms return image handles audio.* info, normalize, trim, concat, mix, reverse, fadeIn, fadeOut, repeat. Transforms return audio handles video.* info, trim, resize, rotate, addAudio, extractAudio, extractFrame. Transforms return video, audio, or image handles as appropriate media.* bytes, text, info read a document/image/audio/video input; toDocument, toImage, toAudio, toVideo build one to emit or output. Needs a ProcessingContext canvas.measureText / createCanvas(w, h) Canvas 2D drawing, recorded in the guest and replayed on a real host context by await surface.toBytes() assetToSandbox(assetId, path) / sandboxToAsset(path) Move an asset in and out of the workspace progress(percent, message?) Fire-and-forget progress, rate-limited and capped emit(name, value) / output(name, value) The Code node’s output contract: stream a value now, or record a handle’s final value. Awaitable; emit awaits drain under backpressure toBase64 / fromBase64 / toHex / fromHex / parallelMap Pure guest helpers, no host call behind them Core JavaScript — JSON, Math, Date, Map, Set, RegExp, URL, URLSearchParams, TextEncoder/TextDecoder — is QuickJS’s own, not a host-bridged version. Callers add their own globals through RunSandboxOptions.globals: inputs and state for the Code node, tools/finish/__graphQueues for CodeAct, node/graph for the graph DSL. Names in RESERVED_SANDBOX_NAMES cannot be overwritten this way. media.* — media inputs and outputs A document, image, audio or video input arrives in the guest as a ref object, not as bytes. media is the bridge that resolves one, and the bridge that turns bytes back into a ref the next node can read: const bytes = await media.bytes(inputs.pdf); const page = await media.text(inputs.notes); // utf-8 unless `encoding` says otherwise const { mimeType, size } = await media.info(inputs.pdf); await output("report", await media.toDocument(bytes, { mimeType, filename: "report.pdf" })); Call Returns media.bytes(ref) Uint8Array media.text(ref, { encoding? }) string media.info(ref) { type, mimeType, uri, size } media.toDocument(bytes, { mimeType?, filename? }) DocumentRef media.toImage(bytes, { mimeType? }) ImageRef media.toAudio(bytes, { mimeType? }) AudioRef media.toVideo(bytes, { mimeType? }) VideoRef Every call is async — await all seven. A ref resolves from any of the forms the rest of NodeTool produces: asset://, a storage path (/api/storage/<key>), an https: URL, a data: URI, a package:// asset shipped inside a node package, and a plain file path. Pass the whole input object, not its uri — media.bytes(inputs.pdf) — so the ref’s own type travels with it. These are capabilities, not libraries: they need a ProcessingContext, so they resolve inside a workflow run and throw in a bare runInSandbox call with no context. For the libraries that read what the bytes contain — a PDF, a spreadsheet, a zip — import a sandbox package. Media editing handles image.*, audio.*, and video.* accept a media ref, encoded bytes, or a handle from an earlier operation in the same run. A transform returns a small sandbox://media/<id> handle with its media type. The encoded payload stays on the host while calls chain: const joined = await audio.concat([inputs.intro, inputs.voiceover]); const clip = await video.trim(inputs.video, { start: 2, end: 12 }); const finished = await video.addAudio(clip, joined, { keepOriginalAudio: true }); await output("video", await video.toAsset(finished)); Use <type>.bytes(handle) only when code must inspect the encoded payload. Use <type>.toAsset(handle) or media.toImage/toAudio/toVideo(handle) before the run ends when the result must persist. Handles do not work in a later run. Audio and video transforms use Mediabunny. Browsers use WebCodecs, and Node uses Mediabunny’s server codec adapter. The sandbox does not expose workflow nodes or their packages. Libraries (imports) There is no library global. Every library the sandbox offers is a sandbox package the run declares and imports: import yaml from "@nodetool-ai/sandbox-yaml"; const config = yaml.load(inputs.text); await output("config", config); NodeTool ships these packs (packages/sandbox-packs/, one directory each): -dates (date-fns), -yaml (js-yaml), -markdown (marked), -qr (uqr), -color (culori), -decimal (decimal.js), -jmespath (jmespath), -stats (simple-statistics), -rrule (rrule), -gif (gifenc), -dsl (NodeTool’s graph builder), -flow (NodeTool’s node callables), -game (NodeTool’s game document builder) and -timeline (motion-graphics timelines as code) run inside the guest; -csv (papaparse), -html (cheerio + turndown), -xml (fast-xml-parser), -xlsx (exceljs), -diff (diff), -zip (fflate), -ocr (tesseract.js), -tfjs (TensorFlow.js and its model zoo), -docx (docx), -mammoth (mammoth), -epub (epub2), -fabric (Fabric.js — SVG and vector scenes), -pdflib (pdf-lib), -pptxgen (PptxGenJS), -chrono (chrono-node), -exif (exifr), -expr (expr-eval), -ics (ics), -subtitle (subtitle), -tokens (js-tiktoken), -pptx (office-text-extractor) and -pdf (pdf-parse) run on the host behind a generated facade, because they need Node builtins or a DOM, or carry a limit the guest could not enforce on itself (zip’s 50 MB inflation cap), or hold state no run can keep alive (the tfjs weights). Four more carry NodeTool’s own code rather than a library — -aws signs a request with SigV4, and -notion, -supabase and -twilio build an authenticated one — and none of them sends it: the guest passes what comes back to its own fetch, under the run’s fetch cap and SSRF guard. Every shipped pack is available by default — a checkout, the desktop app and the server image each read them from where their own build put them. A third-party pack is installed through the Package Manager and is discovered the same way. See Sandbox packages and packages/sandbox-packs/README.md. Limits Every default below is overridable per invocation through RunSandboxOptions.limits and clamped to the ceiling in the last column. Limit Default Enforced by Ceiling Execution time 30 s (timeoutMs) interrupt handler on a CPU budget + wall-clock race — CodeAct action timeout 600 s DEFAULT_CODEACT_ACTION_TIMEOUT_MS — Suspended time 30 min suspendAllowanceMs, only with a clock — Guest heap 64 MB runtime.setMemoryLimit 512 MB Call stack 512 KB runtime.setMaxStackSize 8 MB Fetch calls 20 per run counter in the bridge 100 Fetch body 1 MB truncation in the bridge 50 MB Fetch timeout 15 s per-request AbortController 120 s Fetch redirects 5 hops the bridge — Output size 100 KB serializeResult 10 MB Random bytes 64 KB per call crypto.getRandomValues clamp — Progress reports 1000 per run, one per 100 ms counter + timestamp — Host module text input 5 MB host-modules/limits.ts — Host module byte input 10 MB host-modules/limits.ts — Image input 25 MB, 32 M pixels, 16384 px longest edge assertSurfaceSize — Run media handles 256 MB total encoded payload SandboxMediaStore (runMediaBytes) 2 GB Canvas ops 10 000 per render renderCanvas — Tool calls per action 50 DEFAULT_MAX_TOOL_CALLS_PER_ACTION — QuickJS’s memory limiter counts its own heap objects. String and typed-array payloads are not charged against it, so memoryLimitBytes bites on object allocation, not on new Uint8Array(n). Concurrency The sandbox is fully asynchronous, and a bridge call starts its host-side work when it is invoked, not when it is awaited. Promise.all over five fetches therefore takes one round trip, not five: const pages = await Promise.all(urls.map((u) => fetch(u))); parallelMap(items, fn, concurrency = 5) is the bounded form — order-preserving, maximum concurrency 32, rejecting on the first failure. Parallel calls count against the per-run fetch cap exactly like serial ones. Timer globals (setTimeout, setInterval, setImmediate and their clears) are deleted inside the user-code module. Their callbacks would fire through ctx.callFunction with errors discarded, outside the never-reject and abort-guard conventions every bridge follows. sleep is the only timer. Marshaling rules Anything crossing the WASM boundary follows four rules. Break one and the symptom is silent data corruption, not an error. Host async functions never reject. A failing bridge resolves a tagged {__nodetool_sandbox_error__: true, name, message} object, and a guest prelude re-throws it as a real Error. This works around a handle leak in @sebastianwessel/quickjs@3.0.1 that trips an assertion (list_empty(&rt->gc_obj_list)) when the runtime is freed. Binary crosses asymmetrically. Guest → host is native: typed-array serializers registered with addSerializer turn a guest Uint8Array into a host one. Host → guest is not: a returned Uint8Array would arrive as a numeric-keyed plain object, so byte-producing bridges return a base64 marker object that the guest prelude rebuilds. Follow this for any new binary bridge. Results are scanned at any depth. serializeResult walks for typed arrays through the whole value; the streaming path always nests bytes two levels down, and a Uint8Array that falls to JSON.stringify becomes {"0":137,"1":80} — lossy and indistinguishable from a user’s integer-keyed map. Object globals sync back. After the guest runs, object-typed globals are deep-replaced on the host. Primitives pass by value and do not sync. Structured data crosses as JSON, not as handles. The wrapper marshals an object node by node — four evalCode compilations plus a property-by-property descriptor copy each — so a run handed 5 000 rows spent ~2 s reading them and ~0.8 s handing them back. sandbox-json-transport.ts moves one string instead: the guest builds its result with JSON.stringify and the host parses it, the host encodes injected globals for the prelude to parse, and emit / output arguments take the same path. Typed arrays and strings over 8 KB ride a sidecar the marshaler moves whole; dates, non-finite numbers and undefined properties carry markers, because JSON alone would flatten or drop them. A value the encoder cannot represent — a function, a bigint, a Map, a class instance — falls back to the wrapper’s own marshaling. A cycle is refused outright: the wrapper follows it until the runtime aborts, so a cyclic global fails the run by name and a cyclic result comes back as String(value). The shared engine loader releases the primitive property handles that @sebastianwessel/quickjs@3.0.1 forgets while reading descriptors, then returns QuickJS’s equivalent static handles. It also releases forgotten symbol description handles with their context. This applies before bootstrap on both execution paths, so engines remain reusable. A worker reporting an interpreter failure is discarded. Guest exceptions leave the worker reusable. See the reproduction and upstream issue draft. The agents and sandbox compiler keep @sebastianwessel/quickjs in the 3.0 patch series with ~3.0.1. Its 3.1 release uses quickjs-emscripten-core 0.32, while the selected WASM variant uses the 0.31 ABI. Mixing those versions loses host function references and leaves asynchronous callbacks unresolved. Upgrade the wrapper, core, and WASM variant together and verify host callbacks on both execution paths. Security model The guest starts with less than plain QuickJS, and every capability past that is one the host granted. No dynamic code generation. eval and Function are deleted before any user code evaluates. No ambient modules. A body that imports nothing gets no loader at all. With one, only the packs the run resolved and their intra-pack siblings resolve; dynamic import() is always denied. Enforcement sits in the normalizer, not the loader, because QuickJS serves an already-cached module without consulting the loader — that is what keeps node:buffer and the rest of the wrapper’s compat preamble out of reach after bootstrap. Scoped secrets. A run may declare the secret names it needs (limits.secretScope, host-set). The getSecret bridge refuses every other name, so a node that talks to one service cannot read another’s credentials — and because the check is at the bridge, nodetool.secrets.get cannot route around it. An absent scope is unscoped, which is what a Code node authored before scopes existed still gets; an empty declared scope denies everything. Writing one is a separate question with a stricter answer: guest code cannot set a secret at all. The settings capability module (@nodetool-ai/sandbox-nodetool/settings) reads and writes ordinary configuration, refuses get_setting/set_setting on anything the catalog marks a credential, and offers request_secret — which takes a name and a reason, never a value. It opens a dialog in the user’s client, the user types the key there, and the client saves it with its own authenticated call, so the credential never passes through the guest, the websocket frame, or the model’s context. A run with no interactive client is refused by name rather than falling back to a write nobody saw. SSRF guard. fetch refuses loopback, link-local and private ranges, including IPv6 forms and IPv4-mapped addresses, and re-checks on every redirect. limits.allowPrivateNetwork lifts it; it is host-set only, so guest code cannot enable it for itself. Workspace containment. workspace.* resolves inside the workspace root and re-checks the symlink-resolved real path immediately before each operation. limits.filesystemAccess: "host" lifts that to the whole filesystem the process can reach. Both switches exist because lib.http and lib.os nodes always had that reach and a Code node replacing one must match it; both default to the restrictive value, and the graph migration sets the filesystem switch only on nodes rewritten from such a node. The cloud profile ignores both switches (isHostAccessEnabled in packages/agents), because on a shared server they would reach the operator’s disk and network. Cancellation. Once the abort signal fires, every subsequent bridge call fails fast and the guest unwinds. A purely CPU-bound loop still runs to its execution timeout — QuickJS’s wrapper exposes no interrupt input — but runInSandbox returns as soon as the signal fires. The platform API is scoped to the caller’s own rows. Every capability that reads or writes NodeTool’s own data resolves the row against the run’s user id, and missing and not-yours are one answer so a run cannot probe for ids. What is deliberately not reachable is written down per procedure in packages/websocket/src/trpc/sandbox-coverage.ts and checked against the live router: credentials, billing, other tenants, host control, the transcript of the run’s own behaviour, and anything that grants a third party access. Publishing is the one exception, and it asks — set_workflow_access is classified external, so the permission gate prompts before a workflow becomes readable outside the account. The gate is the host’s, not the sandbox’s. A Code node and a JS script read PERMISSION_GATE_CONTEXT_KEY off the run’s context, so a body reached from a chat in plan mode is refused a write capability through both doors — the @nodetool-ai/sandbox-nodetool/* import and the belt bridge. A kernel workflow run sets no gate and runs auto, since pressing Run on the graph was the consent. Bridges are the boundary, not the hiding. The host and WASM module dispatchers validate module identity, export name and argument list before any implementation loads, and their bindings are deleted before user code starts. A pack module that captures a binding during linking gains nothing beyond the run’s own declared surface. Known accepted risk: the realpath check and the filesystem call after it are separate awaits, so an in-workspace symlink swapped between them is a TOCTOU window. Closing it needs fd-based operations (O_NOFOLLOW/openat) that node:fs/promises does not expose. It requires a local attacker racing inside the workspace, on surfaces that run first-party or already-trusted code. The module system A pack declares its sandbox modules in the nodetool field of its package.json. Three kinds, one import surface: Kind Runs Declared as Guest JS inside QuickJS authored source, or {"npm": "<dependency>"} compiled by packages/sandbox-compiler Host JS where the sandbox runs {"kind": "host", "host": "<id>"}, resolved only through SANDBOX_HOST_MODULES in @nodetool-ai/protocol WASM host worker pool manifest exports with scalar-only signatures, behind a generated facade A host id resolves only if the registry pins that exact package as the one allowed to declare it, so a third-party pack can never bring host code. WASM calls are stateless by contract: each instantiates fresh from the cached module, runs, and is discarded. Compilation of an npm-declared module is cached by content digest, never by version: npm run dev:nodetool -- packs compile # every installed pack npm run dev:nodetool -- packs compile --force # recompile and re-probe Anything that stops a module short of admission is a named skip, not an error: npm-module-builtin-import, npm-module-unresolved, npm-module-too-large (1 MB), npm-module-forbidden-global, npm-module-probe-failed. The browser runner fetches module sources over GET /api/sandbox-modules/* by opaque module id, and the catalog authorizes and retrieves in one call — the route never touches the filesystem. Bodies are verified before they run, so the loading and denial contract is the same in the page as on the server. The Code node nodetool.code.Code is the sandbox as a workflow node. // inputs: { rows: [...], threshold: 10 } const kept = inputs.rows.filter((r) => r.score > inputs.threshold); progress(50, `kept ${kept.length}`); for (const row of kept) await emit("row", row); // streams as it goes await output("count", kept.length); // final, posted at the end // outputs: row, count Inputs arrive on the inputs object, never as globals of their own name. Sharing the global namespace let an input called env shadow a bridge and made every undeclared identifier ambiguous between a typo and a missing slot. Values are deep-copied through JSON before entering the guest. Outputs leave the body through two awaitable calls, and nothing else. await output(name, value) records a handle’s final value; all final values post together as one bag when the body completes, and a second output on the same handle throws. name must be a declared output handle. return is ordinary control flow — its value is ignored. Media inputs arrive as refs. Read one with media.bytes / media.text, and output one built by media.toDocument / toImage / toAudio / toVideo. Streaming. await emit(name, value) delivers {[name]: value} downstream immediately, while the body keeps running — call it any number of times per handle. Awaiting it applies backpressure once the queue is full. A body that emits nothing simply never streams; there is no separate streaming mode to opt into. Legacy bodies. A body that calls neither emit nor output runs the old return/yield contract — returned object keys as outputs, implicit return of the last expression, yield replayed after the run — for one more release, with a deprecation warning from validate_code and from the editor. See code-node-emit-design.md for the contract and the migration. state is a plain object that survives across streaming invocations and resets at the start of each workflow run. progress(percent, message) posts node_progress to the kernel — the same channel the Python worker uses — so a long snippet drives the node’s progress bar. Input streams stream is the input side of emit: four verbs that read the node’s connected handles as values arrive, instead of once per item. for await (const item of stream(name)) // one handle, in order, until EOS for await (const [handle, item] of stream.any()) // every handle, arrival order const item = await stream.first(name) // next value, undefined at EOS stream.open(name) // could more still arrive? stream(name) and stream.any() are async iterables that complete at end-of-stream; stream.first is for a value that arrives once; stream.open is synchronous and consumes nothing. Values are marshaled exactly like buffered inputs — JSON deep copy, media as refs readable through media.*. // Running total, live, with a summary at the end. let sum = 0; for await (const n of stream("numbers")) { sum += n; await emit("running", sum); } await output("total", sum); The body decides the mode. A body that mentions stream runs once for the whole stream and pulls its own items; a body that never mentions it keeps today’s contract — one invocation per incoming item, with inputs holding that item’s snapshot. Nothing is configured: hydration re-reads the body each time (usesStreamInputContract), so deleting the last stream call flips the node back. The split has three consequences worth knowing: inputs.<name> in a streaming body carries the node’s configured property values, never per-item edge data. A connected handle is reachable only through stream(name); reading one via inputs. is a validation error naming the call to use. Outputs leave through emit/output only. There is one invocation, so its return value is control flow. output finals post as one bag when the body ends; a body that throws mid-stream keeps what it already emitted and drops the finals. state still exists and is pointless here. One invocation means plain local variables already survive across items. Waiting is not executing: time parked on a take is clock-suspended, so timeout meters the body’s own work and a slow upstream cannot kill a correct consumer. What bounds the run is cancellation — a cancelled run unparks every take as end-of-stream and the body unwinds. Backpressure is free in the other direction too: the guest pulls, so an item the body has not asked for stays in the kernel’s inbox. run_code and test_code take the same bodies: stage items with input_streams: {handle: [...]} and the harness answers takes from them, interleaving stream.any() round-robin by index across the handles you declared. Design: code-node-input-streaming-design.md. Node props map onto sandbox policy: timeout (seconds, 0 for none), max_response_mb, allow_local_network → limits.allowPrivateNetwork, allow_host_filesystem → limits.filesystemAccess, secrets → limits.secretScope. Modules come from the body itself: the packs its static imports name are resolved against the installed catalog. A pack no install serves fails the node before the guest starts rather than surfacing as a resolve error inside it; version or digest drift only warns on the node’s log. On a server host the node’s code also gets the nodetool object model, backed by the agent toolbelt. A JS script run uses that same belt, and one function assembles it for both (assembleSandboxToolbelt). The belt is loaded lazily and only on Node, since the in-browser runner bundles this module: without one, nodetool.capabilities() reports {} and each method throws naming the tool it needs instead of a ReferenceError. run_code stays hermetic so an authoring probe does not get the belt. It is not the chat belt: ui_* needs a browser, and a session can add tools of its own. What it does carry is everything a chat action can reach on the server, Apify and SerpAPI included — a call a chat made and a node then repeats resolves to the same tool. Static checking. nodetool validate parses each Code node body and reports what a run would hit: invalid JavaScript, top-level export, an import no installed pack serves, a bare read of a name that is neither a sandbox API nor one of the node’s own inputs (they live on inputs, so a bare read is a ReferenceError), an inputs.<name> the node does not declare, an emit/output call naming a handle the node does not declare, and a declared handle no reachable call ever writes. The analysis lives in @nodetool-ai/node-sdk (code-analysis.ts, code-node-validation.ts), so the validator, the submit_code planner and the editor read one AST. Agents CodeAct: code as the action space Every agent step acts by writing a program, not by emitting one JSON tool call. The model sees a single provider tool, execute_code({code}); the program runs in the sandbox with the step’s toolbelt exposed as async functions, and one round trip can chain, loop over, branch on and reduce any number of tool calls. Design and the research behind it: CodeAct design. What an action gets on top of the standard surface: import { <name> } from "@nodetool-ai/sandbox-nodetool/<namespace>" — the belt, reached the way a library is. There is no tools.<name>() global: a name a capability module owns comes from that namespace, a client ui_* tool from .../ui, and anything the session added at its own call site from .../session. One import shape covers all three, and a body’s imports say what it uses before it runs. A tool returning an {error} payload throws in the guest, so try/catch is the idiom. Each invocation surfaces to the host as a tool_call_update (id codeact_<n>), so composition inside one action stays observable. tools still exists, as a thrower. Every property access reports the import that replaces it — read off the __toolModules map the host installs — so a body written against the old global says import { web_search } from "@nodetool-ai/sandbox-nodetool/web"; rather than ReferenceError. state is gone from CodeAct. It used to persist across the actions of a step (and, on the chat runner, across a thread via an LRU), which made cross-turn carry process-local and silently lossy. Durable carry is thread memory now: generation results are already assets (asset:// refs), and an action records what later actions or turns need with nodetool.memory.save. Handles from image.* / audio.* / video.* die between actions; convert with nodetool.media.toImage / toAudio / toVideo first. finish(result) — completes the step. For schema’d steps the host validates, and an invalid result throws in the guest with the violation list, so the same action can repair. nodetool.searchTools(query) — in-sandbox discovery for tools the prompt lists by name only, past the disclosure threshold. Each hit carries name, signature, description, and the module / specifier / import line that makes it callable, so discovery answers “where do I import this from” in the same round trip as “what are its arguments”. Deferred tools stay callable; the split spends prompt tokens, not capability. nodetool.packs — what this action may import: list() reports every installed pack and whether the session allows it, modules(pack) the specifiers it declares, exports(specifier) the function names one module exports, and docs(specifier) the pack’s SKILL.md. nodetool.* — the platform as objects (workflows, graph(), nodes, agents, models, media, assets, jobs, collections, web, memory, and the rest), each method wrapping a belt tool. A method whose backing tool is absent throws naming it. nodetool.media.understandVideo(video, prompt, model) is the video half of the judging methods: it hands a whole clip to a model that reads video (Gemini) and answers prompt as {text}. nodetool.media.ffmpeg / ffprobe / downloadVideo — the host binaries. ffprobe(path) reads a file’s format and streams; ffmpeg(args) takes argv, and that argv is bounded on the way out: every path is resolved against the workspace and refused when it lands outside (symlinks resolved), inputs may open local files only (-protocol_whitelist file,crypto,data before every -i, plus a token scan for ://, concat:, pipe:, /dev/…), and the run is bounded on wall clock, captured output, artifact size, and how many host binaries may run at once. Fetch what you need with downloadVideo or fetch, then pass the local path. downloadVideo carries its own boundary: the URL must be public (loopback, link-local and the private ranges are refused, by literal and by DNS answer), the download is capped at 2 GiB, and the run ignores config files — yt-dlp reads yt-dlp.conf from its working directory, which is the workspace guest code writes, and that file can ask for --exec. openWorkflow(id) — when the belt carries the ui_* document tools, a graph object model whose synchronous mutators queue operations against a local mirror, replayed through the same tool contract by await wf.commit(). The action inherits exactly the privileges tool mode already granted: every imported belt function is a tool the model could have called directly, and per-step allowlists stay a privilege boundary. What is genuinely new is composition — one action can chain calls without per-call visibility in the provider transcript — which the per-action tool-call cap, the tool_call_update events and the action timeout bound. Package consent A Code node runs a body a person saved; an action is code the model just wrote. So an action imports only what the session allowed (sandboxPackages), and the prompt advertises only those specifiers, one sanitized line each, never the installed catalog. A session that allowed nothing imports nothing, and an off-allowlist import stops the action before the guest starts — the model sees the refusal as its observation and can correct it. A session with packages also carries get_sandbox_package_docs, which serves one pack’s SKILL.md and wraps an untrusted pack’s body in <untrusted-package-docs>: reference, never instructions. The suspending clock A chat action that calls a gated tool parks on a person’s answer. Charged to the same budget, that wait kills the program that asked, and answering then resolves nothing. createSandboxClock gives the caller clock.suspend(): suspended time is added back to timeoutMs, suspensions nest, and the engine’s own abort moves out to timeoutMs + suspendAllowanceMs as the backstop for a prompt nobody answers. The interrupt handler still cuts a runaway loop at exactly timeoutMs of running time. The websocket chat runner owns one clock per turn and suspends it around every tool- and plan-approval round trip. Other agent surfaces validate_workflow evaluates a legacy graph DSL program (its code parameter) in the sandbox with no host access — only node() and graph() — so a malformed or hostile program cannot reach anything. The plain js tool is gone. execute_code is the one code path an agent acts through; it declares the session’s sandbox packages. run_code remains as a hermetic authoring harness — it executes a Code-node body with no toolbelt and only the secrets the call names — and library-backed work outside an agent goes through a Code node that imports the pack. Failure modes Symptom Cause url.searchParams.set() does not affect the parent URL QuickJS URL limitation. Build the query with URLSearchParams directly A Uint8Array arrives as {"0":137,…} A host → guest byte path that skipped the base64 marker convention An import resolves nothing The run declared no modules, or the specifier is off the allowlist setTimeout is not defined Deleted deliberately. Use sleep, Promise.all or parallelMap A bare identifier is a ReferenceError in a Code node Node inputs live on inputs, not in the global scope nodetool.* throws naming a tool The host has no toolbelt (browser runner, no context) or that tool is not on the belt tools.<name> is gone The flat toolbelt was retired for imports. The message names the import that replaces the call, or points at nodetool.searchTools(name) when no module owns the name Could not find export '<name>' in module '…/<namespace>' The namespace is mounted but this host’s belt has no such tool. nodetool.searchTools(name) lists what it does have; ui_* is browser-only and never on a Code node’s belt A Code node has an empty nodetool.capabilities() and cannot import @nodetool-ai/sandbox-nodetool/* The host never called setCodeNodeAgentsModule. The node resolves @nodetool-ai/agents by bare specifier, which resolves in a checkout and nowhere in the bundled backend (esbuild inlines the workspace packages into server.mjs); packages/websocket/src/server.ts hands over the inlined copy at bootstrap A CPU-bound loop outlives its cancellation The signal ends runInSandbox, but the guest loop still runs to the execution timeout Extending it A new bridge: add it in buildSandbox, wrap async work in neverReject + guardAbort, return bytes as a base64 marker object, add the name to EXPOSED_BRIDGE_NAMES, and describe it in packages/agents/src/code-gen/sandbox-manifest.ts. The manifest reads limits and names out of js-sandbox.ts rather than restating them, so a prompt derived from it cannot advertise an API the sandbox does not marshal — tests/sandbox-manifest-drift.test.ts holds that line. A new library: ship a sandbox pack. Guest-side if it compiles under the admission probe, host-side if it needs Node builtins, a DOM, or a limit the guest could not enforce. Host implementations live in packages/agents/src/host-modules/ with every safety limit inside them, where nothing can route around them. Tests: packages/agents/tests/js-sandbox.test.ts (surface, limits, async concurrency), js-sandbox-modules.test.ts, js-sandbox-wasm.test.ts, host-modules.test.ts, codeact-executor.test.ts, chat-codeact.test.ts, nodetool-api*.test.ts, and packages/sandbox-compiler/tests/packs.test.ts for every shipped pack through the real install path. Related CodeAct design — the action protocol and its research Sandbox packages — the pack system, trust model, milestones Chat & Agents — the agent surfaces whose actions run here Execution strategies — where sandboxed work sits among the run modes--- # JS Script Documents: a first-class, executable script type Source: https://docs.nodetool.ai/js-script-document-design.html JS Script Documents: a first-class, executable script type Status: implemented (phases 1–3). Owner: code-nodes / agents / web. 2026-08-12. Where the implementation differs from the design below, the difference is noted inline under Implementation note. Problem Reusable JavaScript exists in NodeTool in exactly two forms, and neither is a document a user can open, edit, test, and share: A Code node body — a code string inside one workflow’s graph. It cannot be reused by a second workflow without copy-paste, it has no version history of its own, and editing it means opening the workflow that happens to contain it. A sandbox pack — an npm-shaped artifact installed through the Package Manager. It is a distribution format, not an editing surface: changing one means republishing a package. The gap shows up in three places. A mini app that needs ten lines of glue (reshape an API response, format a date, merge two lists) must author a whole workflow around one Code node. An agent that has debugged a useful body with run_code has nowhere durable to put it. And two Code nodes that need the same helper each carry their own diverging copy. This design adds a JS script document: a named, versioned, user-owned script with declared inputs, outputs, packages, secrets, and saved test cases. It opens in its own workspace tab (Monaco editor plus an assistant chat panel, the Code node assistant’s layout as a full surface), runs and tests against the QuickJS sandbox, and is invocable from mini apps (as an operation), from Code nodes (as a linked body), from agents (belt tools), and from other scripts (through the same belt). Naming: the scripts table, the script tab type, and the nodetool.scripts guest namespace are all taken by the video/voiceover script feature. This feature uses js_scripts / jsscript / ui_jsscript_* / *_js_script throughout. Document shape One JSON document per script, stored whole and validated with Zod at the wire boundary (schema in packages/protocol/src/api-schemas/js-scripts.ts, types exported from @nodetool-ai/protocol): interface JsScriptDocument { schemaVersion: 1; description: string; // what it does — agents read this to pick a script code: string; // the body, emit/output contract only inputs: JsScriptPort[]; // { name, type } — same TypeMetadata names nodes use outputs: JsScriptPort[]; packages: SandboxModuleDeclaration[]; // sandbox packs the body may import secrets: string[]; // names readable via getSecret; [] = none timeoutSeconds: number; // default 30, max 120 (MAX_TIMEOUT_SECONDS) tests: JsScriptTestCase[]; // saved cases, same shape test_code grades } interface JsScriptTestCase { name: string; inputs: Record<string, unknown>; inputStreams?: Record<string, unknown[]>; // staged items, for a `stream` body expect?: Record<string, unknown>; // per-handle structural compare expectedStreamed?: { name: string; value: unknown }[]; } Decisions baked into the shape: Emit/output contract only. Scripts are a new surface that launches after docs/code-node-emit-design.md; there is no legacy return-contract path to support. validate_js_script rejects a body that calls neither emit nor output while declaring outputs — no deprecation warning, an error. Tests live in the document. test_code already grades a case list; a script carries its own regression suite, so the editor’s Test button, the assistant’s repair loop, and the CLI harness all run the same cases. A saved script with zero tests validates with a warning. Ports are declared, not inferred. validateCodeNodeBody already takes availableInputs/declaredOutputs as lists; for a Code node the graph derives them, for a script the document declares them. Everything downstream (app input mapping, Code node linking, run_js_script argument checking) reads the declared ports. The script pins its own execution envelope (packages, secrets, timeout). A caller can narrow it — never widen it. Invoking a script does not grant it the caller’s secrets; it runs with the intersection of its declared secrets and what the invoking context allows. Implementation note. A script declares no packages. Every installed sandbox pack and every @nodetool-ai/sandbox-nodetool/<namespace> module resolves by import; a packages field on an old save is dropped on parse. Storage and API Copy the sketch pattern (image_documents + image_document_versions), the full-featured reference for document types: packages/models/src/schema/js-scripts.ts — table js_scripts: id, user_id, project_id, name, document (TEXT JSON), created_at, updated_at, indexes on user/project/updated. Postgres mirror in schema-pg/. packages/models/src/schema/js-script-versions.ts — table js_script_versions: id, js_script_id (cascade), user_id, name, version, save_type ("manual" | "autosave" | "restore"), document, created_at, unique on (js_script_id, version). packages/models/src/js-script.ts — class JsScript extends DBModel, beforeSave bumps updated_at monotonically and asserts the document validates; CAS save via updateFieldsIfUnchanged. Migration entries in packages/models/src/migrations/versions.ts (create_js_scripts, create_js_script_versions). tRPC router packages/websocket/src/trpc/routers/js-scripts.ts: list / get / create / update / delete, update carrying the read updatedAt as the CAS token, plus a nested documentVersions sub-router (list / get / create / restore / delete) copying sketch.ts — restore snapshots the current document as a restore version before overwriting, so a restore is itself undoable. Mounted as jsScripts in trpc/router.ts. One run endpoint for non-tRPC callers (mini apps, harnesses): POST /api/js-scripts/:id/run {inputs, input_streams?} executes the script and streams the same message shapes a workflow run streams (node_progress for progress(), per-emit output messages, a final output bag), so the app-runtime fold consumes it unchanged. Editor tab New WorkspaceTabType member "jsscript" in web/src/stores/WorkspaceTabsStore.ts; the exhaustive switch in TabContent.tsx forces the new case, JsScriptSurface.tsx. The surface is the Code node assistant dialog promoted to a document editor: Left: Monaco (useMonacoEditor), editing the persistent document — not a draft. Autosave through useJsScriptServerSync (copy useScriptServerSync: load on mount, 750 ms debounce, CAS on updatedAt, retry after 5 s, registerDocumentSync for cross-client resource_change). Right: assistant chat panel, the same panel component the Code node assistant uses, wired to the ui_jsscript_* tools below. Where the node assistant edits a draft and Apply commits to the node, the script assistant edits the document directly — undo is the document’s own history plus the version snapshots. Header/sidebar: ports (inputs/outputs with types), secrets list, timeout — the same controls the Code node property panel renders for its dynamic slots. There is no packages picker: every installed pack resolves by import (see the implementation note above). Bottom: run console. A Run button (prompting for input values from the declared ports), a Test button running the saved cases, and a console showing logs, streamed emits, final outputs, and errors — the run_code/test_code result shapes rendered. Threading a tab type through the shell touches the known list: PanelStore.LeftPanelView ("jsscripts") + JsScriptListPanel / CreateJsScriptButton in PanelLeft.tsx, OpenMenu.tsx, openDocument.ts (OPENABLE_TYPES), uiContext.ts (TAB_TYPE_TO_SURFACE), documentSync.ts (SyncedDocumentType), resourceChangeHandler.ts, IconForType.tsx, and UiSurfaceType in packages/protocol/src/api-types.ts. ui_jsscript_* tools Frontend tools in web/src/lib/tools/builtin/jsscript.ts (registered via builtin/index.ts), executing against a per-document bridge (jsScriptAgentBridge.ts + useJsScriptAgentBridge, mounted from the surface — same seam as scriptAgentBridge.ts). Every tool takes the script id; ui_open_document bridges “an id exists” → “the tools work”. Tool Effect ui_jsscript_get_state document + validation issues + last run/test result ui_jsscript_set_code replace the body ui_jsscript_set_ports replace inputs/outputs ui_jsscript_set_meta name, description, secrets, timeout ui_jsscript_set_tests replace the saved case list ui_jsscript_run run with given inputs, return outputs/streamed/logs/error ui_jsscript_test run the saved cases, return the grade report Run and test execute server-side (the browser calls the run endpoint / capability, not an in-browser QuickJS), so the editor, the assistant, and the CLI harness exercise one execution path. Execution semantics Execution reuses runCodeBody (packages/agents/src/capabilities/code.ts) — the same core run_code/test_code already share, with withToolbelt on — parameterized by the script document instead of ad-hoc arguments: Globals: { inputs, state: {} }. Fresh state per run; inputs JSON deep-copied. Limits: secretScope = the script’s declared secrets intersected with the caller’s allowance; timeoutMs from the document, capped by the caller. Packages resolve through the process SandboxModuleCatalog; an undeclared import fails validation before it fails at run time. The Code-node toolbelt. A script run prepends the same imported / nodetool.* prelude a Code node does, wired to assembleSandboxToolbelt() (getBuiltinTools() plus getAllMcpTools({ registry })). Chat ui_* tools stay off this belt — they need a browser. run_code / test_code stay hermetic so an authoring probe cannot spend money. Tool-backed calls (media generation, workflow runs, run_js_script) use each tool’s own permission gating; a script that calls run_js_script on itself still hits the cycle gate. Invocation By agents (and by other scripts) One capability module js-scripts (packages/agents/src/capabilities/, specs + implementation, registered in both halves of capabilities/registry.ts): Tool Effect list_js_scripts id, name, description, ports — description is the discovery surface get_js_script full document save_js_script create or update (CAS) validate_js_script static check of a document (inline or by id) run_js_script execute by id/name with inputs; returns outputs/streamed/logs/error test_js_script run the saved cases; returns the grade report Because these are belt tools, every existing surface reaches them with zero new plumbing: agent JSON tool calls, CodeAct actions, MCP, and — the “invoked by other scripts” requirement — a Code node body or CodeAct action calling tools.run_js_script({name, inputs}) through the existing tool bridge. Guest→guest goes through the host, so budgets, depth, and secret scoping stay host-enforced. Recursion is bounded the way sub-agents are bounded: run_js_script carries a depth counter on the ProcessingContext (the SUBTASK_DEPTH_KEY pattern) plus the chain of script ids; a script id already on the chain fails the call with the cycle named. Default depth cap: 4. A direct guest import of another script (a nodetool-script:<id> specifier served by the catalog, next to how mountCapabilityModules mounts platform modules) is deferred: it would inline the callee into the caller’s capability envelope, so scoping and staleness need their own design. The tool path covers composition until then. From Code nodes A Code node can link a script instead of writing a body by hand. Linking materializes the pinned version: its code, packages, secrets and timeout are copied onto the node in one undoable edit, and the optional script prop ({ id, version }) records where they came from. The editor offers: Link script — pick a script; its body, ports, packages, secrets and timeout are copied onto the node, whose code renders read-only while linked, with an “update to latest” affordance that re-copies and re-pins. Extract to script — lift the node’s current body, ports, packages, and secrets into a new script document and link it. The inverse, detach, clears the script prop; the body is already inline. At run time CodeNode executes its own properties — there is no resolution step, no database read, and no resolver on the ProcessingContext. A graph therefore runs wherever it is opened, and hydration answers the one question it has to answer statically: resolveStreamingInput reads node.properties.code to decide whether the node streams its inputs, which it can only do because the body is right there. Resolve-at-run would have made a linked streaming script hydrate buffered and fail on its first stream call. Implementation note. The link is checked, not resolved. validateGraph is synchronous, so it takes an injected jsScriptLookup: a caller prefetches every link with collectJsScriptLinks and passes a lookup over what it found. The materialized body is validated exactly like an inline one — streaming rules included — and the lookup only answers the freshness question, comparing the script’s declared ports against the node’s slots (jsScriptPortMismatches in @nodetool-ai/node-sdk, js-script-link.ts). A link nothing can answer for is a warning (js_script_unverified without a lookup, js_script_missing with one): execution no longer needs the row, so a script that moved or was deleted costs freshness, not the run. The models-layer resolver (createJsScriptResolver) survives for the surface that still resolves a pinned version — a mini-app script operation — and for the editor’s link and update-to-latest, which read it over tRPC. From mini apps OperationBinding (packages/app-runtime/src/document.ts) today binds only a workflowId. It gains a discriminated target: type OperationTarget = | { kind: "workflow"; workflowId: string; workflowVersion?: number } | { kind: "script"; scriptId: string; scriptVersion: number }; (workflowId on the binding stays readable for one schema version; APP_SCHEMA_VERSION bumps to 4 with a lift in documents.ts.) Implementation note. The lift lives in document.ts, next to parseApplicationDocument — documents.ts is the sketch/timeline ref reader and has nothing to do with app documents. workflowId is not duplicated: a workflow target stores its id there as it always did, and only a script target stores a target. operationTarget(binding) is the normalizer that derives the union, and it is the only place the two kinds are told apart. A script operation maps widgets to the script’s declared inputs and its outputs to displays/variables exactly as a workflow operation maps input and output nodes — the mapping layer already works on names. Execution goes through the run endpoint above; because it streams the same message shapes, the app-runtime fold (fold.ts), the app debug harness simulator, and the web runtime consume script operations without a second code path. nodetool app debug validates a script target the way it validates a workflow target: a binding naming a port the script does not declare is an error. Implementation note. The run endpoint still answers with plain JSON, so the stream the fold consumes is synthesized from that result by one adapter, jsScriptRunMessages (@nodetool-ai/execution/app-debug, script-operation.ts): one output_update per emit in call order, one per final output, then the terminal job_update. A script’s ports stand in for node ids, since a script has no nodes. Executing a body lives above @nodetool-ai/execution, so the simulator takes an injected runScript; the hosts that own both wire createJsScriptAppRunner (@nodetool-ai/agents) — the server’s /api/applications/debug, the debug_app capability, and nodetool app debug. A host with no runner reports the operation as unexecutable rather than skipping it. The browser runs one too: useAppRuntime loads the script document, derives its IO from the declared ports, calls the run endpoint, and folds the synthesized messages — so a script operation and a workflow operation differ only in how the messages are produced. The pure half of the adapter (scriptRunMessages, scriptInvocationInput, scriptPortIO) therefore lives in @nodetool-ai/app-runtime (script-run.ts), which the browser can depend on; @nodetool-ai/execution re-exports it and keeps the two parts that need a parser — binding the pinned Zod document, and reading the body to decide whether the mapped values are staged as streams. The app builder authors one: an operation’s Runs field picks Workflow or JS script, and a script target picks from the user’s scripts. Switching what an operation runs clears its input and output mappings, in updateOperation rather than in the panel, because they key on the old target’s node ids or port names. A positive scriptVersion pins both the bindable ports and execution to the owned immutable snapshot. The run endpoint receives that version through script_version and rejects missing or malformed snapshots without falling back to the live script. Version-zero draft targets use the saved head. Ordinary script-editor runs omit script_version and retain saved-head behavior. ApplicationBundle.scripts carries each pinned document under a bundle-local key. Because a script row has its own version numbering, import snapshots every carried script and re-pins the operations to the version it created (pinScriptVersions); the number the export carried means nothing in the importing database. @nodetool-ai/app-runtime stays dependency-free, so the carried document is typed structurally (BundleJsScriptDocument) and parsed against the pinned Zod contract wherever it is actually run. ApplicationBundle gains scripts: BundledJsScript[] beside workflows, with the same bundle-local key indirection, so an app that uses scripts stays a single shareable file. Validation validate_js_script wraps validateCodeNodeBody (packages/node-sdk/src/code-node-validation.ts) with the document’s declared ports and the installed sandboxModuleCatalog — the analysis (syntax, missing installed packs, undefined names, undeclared inputs.* reads, unreachable output handles) is shared, not forked. A script does not declare packages. On top of the body check, document-level rules: duplicate or non-identifier port names; a port type the type system lacks a test case referencing an undeclared input, staging a stream for one, or expecting an undeclared output declared outputs with no reachable emit/output call (from the body check), and a body using the legacy return contract (error, not warning) zero saved tests (warning) a declared secret name that does not exist in the secret store (warning — the store is per-install) the streaming rules under Input streaming The check runs at save time (beforeSave), in the editor (issues panel), in the save_js_script capability, and in the CLI harness. CLI harness and registry Following the sketch/timeline pair, packages/cli/src/commands/js-script.ts (registered in nodetool.ts; heavy deps imported lazily inside actions): nodetool jsscript validate <id|file.json> [--json] [--warnings-as-errors] nodetool jsscript run <id|file.json> --inputs '{"a":1}' [--input-streams '{"nums":[1,2]}'] [--json] nodetool jsscript test <id|file.json> [--json] # saved cases nodetool jsscript debug <id|file.json> --interact '[{"tool":"set_code",...}]' nodetool jsscript versions list|show|create|restore|delete <id> [...] debug replays a scripted session against the headless ui_jsscript_* bridge and validates the document the session left behind; test is the selfcheck-friendly command (a fixture script with deterministic cases runs keylessly). Report types live in @nodetool-ai/execution/js-script-debug; the CLI keeps target resolution and the bundle, as sketch/timeline do. Registry (packages/cli/src/harness/registry.ts): harness entries jsscript-validate, jsscript-run, jsscript-test (selfcheck: cheap, no-db on file targets), jsscript-debug, jsscript-versions; a jsscript surface entry listing the paths above — the registry test fails without it. Evals: a headless surface packages/agents/src/evals/surfaces/js-script.ts mirroring the tool names/shapes, wired as nodetool eval jsscript-tools, with cases covering authoring from a prompt, adding tests, and a repair loop (a failing case the model must fix). Input streaming A body reads its inputs one of two ways, and the body says which. The rules are the Code node’s — code-node-input-streaming-design.md — reused whole, because a body that runs in a script must run the same way in a node. A script whose body calls stream(name) / stream.any() / stream.first() / stream.open() is a streaming-input script, detected with usesStreamInputContract. There is no document flag: adding the call is what makes it one, and deleting it is what makes it buffered again. let total = 0; for await (const n of stream("numbers")) { total += n; await emit("running", total); } await output("total", total); Callers stage the items. A script has no inbox of its own — nothing upstream produces into it — so whoever runs it supplies what the body will pull: Surface How run_js_script input_streams: {handle: [item, …]} POST /api/js-scripts/:id/run input_streams in the body nodetool jsscript run --input-streams '{"numbers":[1,2,3]}' The editor’s Run dialog one JSON array per declared input A saved test case inputStreams on the case Wire names are snake_case (input_streams) and the stored document field is camelCase (inputStreams), matching expectedStreamed beside it. A staged handle the script does not declare is refused by every surface rather than silently yielding nothing. Staged items reach the guest through the same pre-staged inbox run_code uses, so arrival order for stream.any() is round-robin by index across the handles in declaration order. Validation runs validateCodeNodeBody with every declared input passed as both available and connected: a script has no node-configured property values, so each declared input is fed by the caller, which is what the graph calls a connected handle. Saying so gives the script exactly the rules it should have — stream("x") on a declared input never warns that nothing feeds it, and reading a declared input through inputs.x in a streaming body is the error naming stream("x"). A streaming body on the return contract stays an error. Two rules are the document’s own, both warnings: a case staging inputStreams for a body that never calls stream, and a streaming body whose cases stage nothing — each case then proves only that the body survives an empty inbox. A mini-app operation stays buffered. A widget holds one value per input, not a stream of them. When the operation’s script streams, each mapped value is staged as a one-item stream and inputs is left empty (scriptOperationInvocation) — the split a graph run makes, where a connected handle is reachable through stream and never through inputs. Security A script executes as its owner for storage/versions, but its runtime capability envelope is only what the document declares: secretScope = declared secrets ∩ caller allowance, sandbox default fetch caps, no toolbelt, no filesystem beyond the workspace default. run_js_script across users is not a thing: scripts are scoped to the requesting user, same as sketches and timelines. Version restore re-validates against today’s schema (the timeline/sketch rule): an old document is restored, then checked, and a failing restore exits non-zero / returns the issues. The script body is untrusted input to every consumer: the editor renders results, never evals; the only executor is the QuickJS sandbox. Phases Document + editor: model, migration, router, tab surface, server sync, Monaco + assistant panel, ui_jsscript_* tools, run/test console. A user can author, run, and test a script end to end. Agent + harness surface: js-scripts capability module, nodetool jsscript commands, harness/surface registry entries, jsscript-tools eval. Agents and other scripts can invoke scripts. Attachment: Code node script link (link/extract/detach, pinned version, graph validation), mini-app OperationTarget + bundle scripts + app debug coverage, the web app runtime’s execution path, and the app builder’s target picker. Landed. Each phase lands green on its own; nothing in phase 1 depends on a later phase’s schema. Testing Model/router (packages/models, packages/websocket): CAS conflict on concurrent update; version restore snapshots first; document validation rejects bad ports/tests. New checks proven failable with inverted fixtures. Validation (packages/node-sdk + capability tests): each document-level rule red on a fixture built to violate it, green on the shipped fixtures. Execution (packages/agents): run_js_script exposes tools / nodetool and can call list_js_scripts; run_code stays hermetic; secret intersection, depth cap and cycle error (including a nested tools.run_js_script of the same id), emit ordering parity with run_code. Web (web/): surface renders, autosave CAS retry, bridge registration/teardown, tools against a mounted surface (Jest, RTL). Harness: jsscript test on a fixture with one deliberately failing case exits non-zero; harness audit passes with the new surface entry. E2E (phase 3): a mini app with a script operation runs under nodetool app debug; a workflow with a linked-script Code node runs under nodetool debug. In the browser (useAppRuntime): the ports become the bindable IO, a run reaches the endpoint with the mapped values keyed by port name and never touches a workflow runner, and a failed run lands as a failed invocation. Switching an operation’s target drops its stale mappings (doc-ops), proven failable by inverting the check. Out of scope Guest→guest import of scripts (module specifier) — deferred, see Invocation. TypeScript bodies, npm dependencies beyond declared sandbox packs, and script sharing/marketplace. Mobile editing — mobile opens documents read-only per its architecture; script edits arrive through the chat agent’s tools like other documents.--- # JS Scripts Source: https://docs.nodetool.ai/js-scripts A JS script is a saved JavaScript document. It has a body, declared input and output ports, the secret names it may read, a timeout, and saved test cases. Unlike the body of a Code node, which lives inside one workflow, a script is a standalone document you can reuse, run on its own, and test. Scripts run in the server’s QuickJS sandbox, never in the browser. For what the sandbox allows, its limits, and its imports, see JavaScript Sandbox. Create a script Open JS Scripts in the left sidebar. Select the New JS script button (+). NodeTool creates “Untitled JS script” and opens it as a workspace tab. The sidebar panel lists your scripts, newest first, and Search JS scripts… filters them by name or description. Select a script to open it in a tab. The editor The script tab has four regions. Editor. The script body in Monaco. Every keystroke autosaves. Undo and redo work as in other documents. Run console. Below the editor, with Run and Test buttons and the result of the last run. Script tab. On the right, the settings panel described below. Assistant tab. On the right, next to Script. See Assistant. On a phone these become four panes, Code, Run, Script, and Assistant, with one visible at a time. Writing the body The host wraps the body in an async function, so write top-level statements only. Do not write export or wrap the body in a function. inputs is already in scope. Read an input as inputs.<name>. Send a result out with await output(name, value) or await emit(name, value). Do not return outputs. A body that returns its outputs fails validation. Import any installed sandbox pack or @nodetool-ai/sandbox-nodetool/<namespace> directly. await output("total", inputs.numbers.reduce((a, n) => a + n, 0)); Media handles are valid only for the current run. Persist them before output, for example with await image.toAsset(frame). A body that reads its inputs with stream runs once and pulls items as they arrive. For the full model, see the nodetool-js-scripting skill and JavaScript Sandbox. Declare inputs and outputs In the Script tab, the Inputs and Outputs lists declare the script’s ports. Select Add to add a row, then enter a name and choose a type. Port names must be valid identifiers, because the body reads them as inputs.<name>. The available types are str, int, float, bool, list, dict, image, audio, video, document, and any. The ports are the only source for the run dialog, the test cases, and the validation checks. The same panel holds: Field What it does Description What the script does. Agents read it to decide whether to call the script. Show in node menu Adds the script to My Nodes so it drops onto a graph like any node. Category sets the group, and defaults to My Nodes. Timeout (seconds, max 120) How long a run may take. The default is 30. Secrets Under Secrets, list the names of the secrets the body may read, such as OPENAI_API_KEY. The body reads one with nodetool.secrets.get. The list is a ceiling, not a grant. A run gets only the names that appear both in this list and in what the calling context allows, so declaring a name never widens access. The validator warns when a declared secret is missing on the current install. Run a script Select Run in the console. The Run script dialog asks for one value per declared input, using the control that matches the port type. The list, dict, and any types take JSON text. A value that does not parse is passed as a string. A script with no inputs says “This script declares no inputs. Run it as it is.” If the body reads inputs with stream, each field instead takes a JSON array of items to stage, for example [1, 2, 3]. The console then shows Last run with its status and duration, plus Logs, Streamed values from emit, and Outputs. Failures show the error in a banner. Runs always use the saved script. If the document has unsaved changes, the console says “Runs use the saved script”, and the run and test tools flush the live document first. Tests A test case has a name, the inputs to run with, and the expected result. It can compare the final outputs per port (expect) and the ordered sequence of emit calls (expectedStreamed). A case for a body that uses stream stages items with inputStreams. The editor has no form for writing cases. Ask the assistant to add them, or edit them through the CLI or the agent tools described below. The Test button reads “No tests” until a case exists, then “Test (N)”. It runs the cases one at a time and reports “N passed, M failed”. Each failing case lists the output name with its expected and actual values. Assistant The Assistant tab (“JS Script Assistant”) edits the document directly. Its changes autosave, with no Apply step. Ask it to write or change the script, add an error output, or write a test for the empty case. Its tools: Tool What it does ui_jsscript_get_state Reads the name, document, validation issues, and the last run and test results ui_jsscript_set_code Replaces the body ui_jsscript_set_ports Replaces input and output ports ui_jsscript_set_meta Sets name, description, secrets, and timeout ui_jsscript_set_tests Replaces the saved test cases ui_jsscript_run Runs the saved script with given inputs ui_jsscript_test Runs and grades the saved cases. Fails when there are none. Outside the editor, agents manage scripts with list_js_scripts, get_js_script, save_js_script, validate_js_script, run_js_script, test_js_script, edit_js_script, and delete_js_script, plus version tools for listing, reading, creating, restoring, and deleting versions. Use a script in a workflow Two paths exist. Node menu. Turn on Show in node menu in the Script tab. The script appears under the category you set in the node menu. Code node. The Code node has a Script property that links it to a script version. Linking copies that version’s code, secrets, and timeout into the node and pins the version. Editing the script later does not change the workflow. Use Update to latest to re-copy and re-pin. Command line nodetool jsscript works on a script JSON file or a saved script id. Full options are in CLI reference. Command What it does nodetool jsscript validate <script_id_or_file> Checks syntax, imports, undefined names, undeclared inputs.* reads, unreachable outputs, port names, and tests. --warnings-as-errors fails on warnings. nodetool jsscript run <script_id_or_file> Runs the body once. Pass --inputs '{"a":1}' or, for a body that uses stream, --input-streams '{"nums":[1,2,3]}'. nodetool jsscript test <script_id_or_file> Runs and grades the saved cases. Exits non-zero if any case fails. nodetool jsscript debug <script_id_or_file> Replays scripted ui_jsscript_* steps against the headless bridge and validates the result. nodetool jsscript versions Lists, creates, and restores versions of a script. nodetool jsscript validate script.json nodetool jsscript run script.json --inputs '{"numbers":[1,2,3]}' nodetool jsscript test script.json--- # Key Concepts Source: https://docs.nodetool.ai/key-concepts.html The handful of ideas you need before building anything. No prior experience assumed. If you want a quick definition of a single word instead, the Glossary is faster. What NodeTool is NodeTool is a visual way to use AI. Instead of writing code, you place boxes on a page and draw lines between them. Each box does one job, and the lines carry results from one box to the next. Ask instead of build. Describe what you want, and an AI agent builds the boxes and lines for you. You can read and change what it builds. Run locally. Models that run on your computer use your own hardware, keeping your data private on your disk. Use your own accounts. When a workflow uses an online AI service like OpenAI or Anthropic, you use your own API keys. You pay the provider directly. Mix local and online. A single workflow can use a local model for one step and an online service for another. Open source. NodeTool is open source software (AGPL-3.0). You can run the exact same code we use. The building blocks The agent The agent is the AI assistant in the Chats panel that turns a goal in plain English into steps. It can build workflows, call tools, and edit your documents directly. How it differs from a plain chat model: It works directly on your open documents. Whether you have a workflow, sketch, or timeline open, the agent acts on it using the same tools you do. You can see its changes and undo them if needed. Everything remains editable. The agent doesn’t create unchangeable results. You can open, understand, and modify anything it builds. You control the agent’s permissions with the permission mode in the chat composer. Plan only reads and proposes, with actions blocked. Default runs reads and asks before each action. Auto runs routine work unattended and still asks once before an action the agent declares high risk, such as a delete, a publish, or spending money. See Chat for more details. Nodes A node is one box that does one thing. Node What it does Example Text To Image Turns a description into a picture “Sunset over mountains” → an image Agent Works out the steps for a task and carries them out “Summarize this document” → an organized summary Text To Speech Reads text aloud A blog post → an audio file Filter String Keeps only the text that matches a rule Keep only lines that contain a keyword Every node takes things in on its left side, sends results out its right side, and has settings that appear in the panel on the right when you click it. Workflows A workflow is a set of nodes joined by connections that you run as one unit. When you run it, your input enters on the left, each node starts as soon as everything it needs has arrived, and results appear on the right as they are produced. Examples of what a workflow can be: A description, into an image model, saved as a file. A PDF, split into pieces, filed away, then searched to answer a question. A story, turned into character descriptions, then portraits, then a video. Reach for a workflow when you want to do the same thing repeatedly: produce media, convert files, ask a model something, index documents, or prepare material for one of the other editors below. Connections A connection is a line that carries one node’s output into another node’s input. Drag from a node’s output on the right to another node’s input on the left. NodeTool checks that the two ends match, so an image output only connects to an input that accepts images and you can’t wire something nonsensical. Hover over a line to see what is passing through it. The full list of types and the conversion rules are in Data Types and Connections. Assets An asset is any file NodeTool stores for you: an image, video, audio clip, PDF, text file, 3D model, or anything else a node can read or write. Assets live in the Asset Explorer and can be used again in any workflow, sketch, timeline, or chat. Assets are the common currency between the different editors: Drag an image asset onto a workflow and it becomes an input. Save a workflow’s result as an asset to use later. Drag video, audio, or images onto a timeline to become clips. Open an image asset in the Sketch Editor to paint on it. Group document assets into a collection so an AI can search them. Sketches A sketch is an image made of stacked layers, the way Photoshop or GIMP work. Use it to paint, hide parts of an image, retouch, combine images, or have AI generate a layer in place. A sketch sits between editing by hand and automating with a workflow: Start from a blank page or an image you already have. Paint, or build the picture up in layers. Attach a layer to an image workflow so it regenerates when its inputs change. Flatten the sketch back into an ordinary image asset. Use that image in a workflow or a timeline. Timelines A timeline arranges media over time on parallel tracks, the way a video editor does. It holds video, audio, still images, overlays, and clips generated by AI. Timelines are where results become finished media: Drag existing assets onto tracks. Add a clip that is produced by a workflow. Trim, split, reorder, and stack clips. Regenerate a generated clip after you change its settings. Export the whole sequence as a video asset. Agent nodes An Agent node is an agent placed inside a workflow, as opposed to the agent above, which builds workflows. It takes a goal written in plain English, works out the steps itself, and uses tools such as web search, file access, or running code to get there. Use one when a step of your pipeline is describable but not scriptable. A normal node does one fixed thing; an Agent node decides what to do. Mini-Apps A Mini-App is a form or dashboard built on top of one or more workflows, with the nodes and lines hidden. Create and open them from the Apps panel in the left sidebar. Give one to someone who should never have to look at a canvas. See Mini Apps. How everything fits together Most work in NodeTool follows one loop: Get some material. Upload files, produce them with a workflow, paint a sketch, or export a timeline. All of it becomes assets. Build a workflow around them. It reads assets, calls models, converts media, and writes new assets. Polish the result in the right editor. Sketches for still images, timelines for anything with a duration. Feed the polished result back in. A flattened sketch or an exported video is just another asset a workflow can read. Share it. Keep the workflow to run again, wrap it as a Mini-App, or publish the finished file. The editors are separate because painting an image and cutting a video are different problems, but they share one pool of assets and one set of AI services. The agent works in every one of them, so any step of the loop is something you can do by hand or ask for. graph LR A[Assets] --> B[Workflow] B --> C[Generated assets] C --> D[Sketch] C --> E[Timeline] D --> A E --> A B --> F[Mini-App] Worked example: a short product video Upload your product photos as assets. Run a workflow that writes the copy, makes background images, and records a voiceover. Open the main photo in a sketch to touch it up and cut out the background. Drag the image, the voiceover, and the generated clips onto a timeline. Attach one clip to an image-to-video workflow and regenerate it until it looks right. Export the timeline as your finished video. Models A model is a trained AI you call from a node. You don’t train it, you use it. Kind of model Produces Used for Image Pictures Posters, concept art, mockups Video Clips Animation, motion Audio Sound Narration, music, effects Text Words Scripts, summaries, analysis On your machine, or online   On your machine Online service Cost Free once downloaded The provider’s price, billed to your account Where it runs Your computer Their servers Speed Depends on your hardware Depends on theirs Internet Works offline Required To set up Download 4-20 GB per model Paste an API key You can mix them and choose per node, so an expensive online model can handle the one step that needs it while the rest runs locally. Words you’ll see Term What it means Workflow Nodes joined by lines Node One box that does one job Edge / connection A line carrying data between nodes Input / output Where data enters and leaves a node Preview A node that displays whatever reaches it, for checking your work Run Execute the workflow Asset A stored file used by workflows, sketches, timelines, chats, or collections Sketch A layered image document for painting, masking, and combining Timeline Tracks for arranging video, audio, and images over time Clip One piece of media placed on a timeline track Generated clip / layer A clip or layer produced by a workflow instead of imported Stale A generated result whose settings changed since it was last produced Agent The assistant you ask for changes, and the node that plans its own steps Permission mode How far the agent may act without asking: Plan, Default, or Auto Model The trained AI a node calls Provider Whoever runs the model: your own machine, OpenAI, FAL, and so on What happens when you press Run On Ctrl/⌘ + Enter: NodeTool reads the lines to work out which node depends on which. Each node starts the moment its inputs have arrived. Nodes that don’t depend on each other run at the same time. Results appear in preview and output nodes while they are still being produced, rather than only at the end. graph LR A[Input: Prompt] --> B[Agent: Plan] B --> C[Text To Image] B --> D[Summarizer] C --> E[Preview: Image] D --> F[Preview: Text] Here the Agent goes first. Text To Image and Summarizer both wait only on the Agent, so once it finishes they run side by side. Data flows one way, so NodeTool works out the running order for you. The editor refuses a connection that would close a cycle. The one exception is a Loop node, whose feedback inputs can take a value back from later in the graph. See the Loop node reference. If you write code Everything above is also available as a TypeScript API, so you can build and run the same workflows from code. Piece What it is Graph Nodes plus connections. Build one with workflow(...), run it with run(...) or runGraph(...) (@nodetool-ai/dsl, packages/dsl/src/core.ts). DSL The TypeScript DSL (@nodetool-ai/dsl), typed factory functions for building graphs in code. WorkflowRunner Runs each node as an actor, passes messages between them, streams progress back (@nodetool-ai/kernel). ProcessingContext Everything a running node can reach: job and user IDs, auth token, workspace, assets, providers (@nodetool-ai/runtime). How a node type is found A saved workflow refers to nodes by a type string (package.Namespace.Class). The node registry resolves it in this order: An exact match in the registry The same string without a trailing Node If neither matches, the namespace (everything before the last dot) is loaded on demand, then the lookup runs again That is why loading a graph doesn’t require importing every node module first. A type that still has no match is reported as unknown. See the Developer Guide and Custom Nodes. Next steps Quick StartTurn one sentence into a finished video. Your First WorkflowBuild a four-node image workflow from an empty canvas. ChatThe agent, its tools, and permission modes. Models & ProvidersChoose models or run them locally. Looking up a word? The Glossary has single-word definitions.--- # Keyboard Shortcuts Source: https://docs.nodetool.ai/keyboard-shortcuts This page lists the shortcuts in each part of NodeTool. In the app, press Ctrl + / (Windows/Linux) or ⌘ + / (Mac), or ?, to open the shortcut sheet for the part you are in. Reading the tables Ctrl/⌘ means Ctrl on Windows and Linux and ⌘ on Mac. Alt/⌥ means Alt on Windows and Linux and Option on Mac. Single-letter shortcuts such as F or B work when focus is on the canvas and not in a text field. Shortcuts marked “desktop app” are registered only in the Electron app. The browser leaves Ctrl + T and Ctrl + W to the browser. The node editor list comes from web/src/config/shortcuts.ts. The timeline, sketch editor, 3D model editor, and game editor each keep their own keys, listed in their own sections. Node editor Workflow Windows/Linux Mac Action Ctrl + Enter ⌘ + Enter Run the workflow Esc Esc Stop the running workflow Ctrl + S ⌘ + S Save the workflow Ctrl + T ⌘ + T New workflow tab (desktop app) Ctrl + W ⌘ + W Close the workflow tab (desktop app) Ctrl + 1 to 9 ⌘ + 1 to 9 Switch to workflow tab 1 to 9 Ctrl + PageUp / PageDown ⌘ + PageUp / PageDown Previous or next tab Ctrl + Shift + [ / ] ⌘ + Shift + [ / ] Previous or next tab (alternate) Ctrl + Alt + ← / → ⌘ + ⌥ + ← / → Previous or next tab (alternate) Editing nodes Windows/Linux Mac Action Ctrl + C / X / V ⌘ + C / X / V Copy, cut, paste nodes Ctrl + Z ⌘ + Z Undo Ctrl + Shift + Z ⌘ + Shift + Z Redo Ctrl + A ⌘ + A Select all nodes Ctrl + D ⌘ + D Duplicate selected nodes Ctrl + Shift + D ⌘ + Shift + D Duplicate selected nodes vertically Ctrl + G ⌘ + G Group selected nodes Delete or Backspace Backspace Delete selected nodes B B Disable or enable selected nodes (a disabled node is excluded from runs) C C Collapse or expand selected nodes Ctrl + I ⌘ + I Show or hide the node info panel Ctrl + right-click ⌘ + right-click Reset a property to its default ← → ↑ ↓ ← → ↑ ↓ Nudge selected nodes Shift + C Shift + C Select all connected nodes Shift + I Shift + I Select nodes connected into the selection Shift + O Shift + O Select nodes that receive the selection’s output Adding nodes Key Node added at the cursor Space Opens the node menu (double-clicking the canvas does the same) Shift + P Prompt Shift + G Text to Image Shift + V Text to Video Shift + L Agent Layout Key Action A Align selected nodes Shift + A Align and distribute spacing Shift + ← / → / ↑ / ↓ Align to the left, right, top, or bottom edge Shift + H Align centers Shift + E Align middles (vertical center) Shift + D Distribute horizontally V Stack selected nodes in one column G Arrange selected nodes in a grid View and navigation Windows/Linux Mac Action F F Fit all or selected nodes into view Ctrl + F ⌘ + F Find in workflow Ctrl + 0 ⌘ + 0 Reset zoom to 50% Ctrl + = / Ctrl + - ⌘ + = / ⌘ + - Zoom in or out by 20% Ctrl + Alt + 5 / 1 / 2 ⌘ + ⌥ + 5 / 1 / 2 Zoom to 50%, 100%, or 200% Ctrl + Alt + N / P ⌘ + ⌥ + N / P Move keyboard focus to the next or previous node Alt + ← → ↑ ↓ ⌥ + ← → ↑ ↓ Move focus to the nearest node in that direction Enter Enter Select the focused node Ctrl + Alt + B ⌘ + ⌥ + B Go back to the previously focused node Esc Esc Exit keyboard navigation Panels and app Windows/Linux Mac Action 1 / 2 1 / 2 Toggle the Workflows or Assets panel I I Show or hide the Inspector W W Show or hide Workflow Settings O O Show or hide the Operator panel L L Toggle the Logs panel (not while editing a timeline) Ctrl + Shift + T ⌘ + Shift + T Toggle the Trace panel Ctrl + K ⌘ + K Open the command menu (works on every view) Ctrl + , ⌘ + , Open Settings in a workspace tab Ctrl + / or ? ⌘ + / or ? Open the keyboard shortcuts panel Chat Windows/Linux Mac Action Enter Enter Send the message Shift + Enter Shift + Enter Insert a new line Esc Esc Stop a reply in progress, when the message box has focus Ctrl + Shift + O ⌘ + Shift + O Start a new conversation in the focused chat Ctrl + Shift + L ⌘ + Shift + L Put the cursor in the focused chat’s message box Assets Windows/Linux Mac Action F2 F2 Rename the selected assets Shift + click Shift + click Extend the selection from the last selected asset Ctrl + click ⌘ + click Add or remove an asset from the selection ← / → ← / → Previous or next asset in the asset viewer Ctrl + ← / → Control + ← / → Jump five assets back or forward in the viewer Ctrl + Backspace or Ctrl + Delete ⌘ + Backspace or ⌘ + Delete Clear the asset search field Timeline editor The timeline has three key layouts: NodeTool, Premiere Pro, and Final Cut Pro. Open the shortcut sheet with ? and pick a layout. The choice is stored in the timelineKeyboardPreset setting. The tables below show the NodeTool layout first. On Mac, Ctrl in these tables means ⌘. Group Key Action Tools V Select tool Tools C Cut (blade) tool Tools N Toggle snapping Tools Esc Clear selection and return to Select Editing S Split selected clips at the playhead Editing Ctrl + Shift + K Cut all tracks at the playhead Editing Delete or Backspace Delete selected clips Editing Shift + Delete Ripple delete (closes the gap) Editing Ctrl + D Duplicate Editing Ctrl + Shift + D Duplicate with a 1 second gap Editing Ctrl + T Cross-fade into selected clips Editing Alt + T Fade selected clips in and out Editing Ctrl + A Select all clips Editing E Extend the selected edit point to the playhead Editing Ctrl + Shift + ← / → Trim the edit point one frame Editing Ctrl + Shift + Alt + ← / → Trim the edit point ten frames Clipboard Ctrl + C / X / V Copy, cut, paste at the playhead Move ← / → Nudge selected clips one frame Move Shift + ← / → Nudge selected clips one second Playback Space Play or pause Playback Alt + ← / → Step the playhead one frame Playback J / K / L Shuttle back, stop, shuttle forward (press again for faster) Playback I / O Mark in, mark out Playback Ctrl + Shift + X Clear in and out Playback ↑ / ↓ Previous or next cut Playback M Add a marker at the playhead Playback Shift + M / Ctrl + Shift + M Next or previous marker Keyframes Alt + K Keyframe the selected clip at the playhead Keyframes Alt + Shift + K / Ctrl + Alt + Shift + K Next or previous keyframe Source Shift + E Append the source range to the end Source W Insert the source range at the playhead Source D Overwrite with the source range at the playhead View + or = / - Zoom in or out View Shift + Z Zoom to fit content History Ctrl + Z Undo History Ctrl + Shift + Z or Ctrl + Y Redo Mouse Ctrl + drag edge Roll the cut with its neighbour Mouse Alt + drag Disable snapping while moving or trimming Mouse Ctrl + drag clip Insert on drop and push later clips right Premiere Pro and Final Cut Pro layouts change these keys: Action Premiere Pro Final Cut Pro Select tool V A Cut tool C B Split at playhead Ctrl + Alt + K Ctrl + B Cut all tracks Ctrl + Shift + K Ctrl + Shift + B Delete Delete Shift + Delete Ripple delete Shift + Delete Delete Duplicate Alt + D Ctrl + D Nudge one frame Alt + ← / → , / . Toggle snapping S N Zoom to fit \ Shift + Z Cross-fade into selection Ctrl + D Ctrl + T Insert / overwrite source range , / . W / D Append source range Shift + E E In the Premiere Pro and Final Cut Pro layouts, bare ← / → step the playhead one frame. The NodeTool layout uses Alt + ← / → for that because bare arrows nudge clips. Shuttle, marking, marker, cut navigation, keyframe, clipboard, and undo keys are the same in all three layouts. See the Video Editor guide for the editor itself. Sketch editor The sketch editor reads Ctrl as ⌘ on Mac. Holding Ctrl/⌘ switches temporarily to the Move tool, except while the Crop, Segment, or Transform tool is active. See the Sketch Editor guide for what each tool does. Group Key Action Tools V B P E G I Move, Brush, Pencil, Eraser, Fill, Color Picker Tools Q S J T F C Blur, Clone Stamp, Adjustments, Gradient, Transform, Crop Tools M W Rectangle Select, Magic Wand Select Tools U L R O A Shape, Line, Rectangle, Ellipse, Arrow Edit Ctrl + Z Undo Edit Ctrl + Shift + Z or Ctrl + Y Redo Edit Ctrl + C / X / V Copy, cut, paste Edit Ctrl + Shift + V Paste masked Edit Ctrl + T Free transform Edit Ctrl + Shift + T Repeat last transform Edit Ctrl + Shift + Alt + T Repeat last transform on a copy Edit Delete or Backspace Clear the layer Edit Ctrl + Backspace Fill with the background color Edit Alt + Backspace Fill with the foreground color Edit Ctrl + I Invert colors Edit ← → ↑ ↓ Nudge Edit Esc Cancel or deselect Selection Ctrl + A Select all Selection Ctrl + D Deselect Selection Ctrl + Shift + D Reselect Selection Ctrl + Shift + I Invert selection Canvas Ctrl + 0 Fit to screen Canvas Ctrl + 1 Zoom to 100% Canvas + or = / - Zoom in or out Canvas Tab Toggle panels Color X Swap foreground and background colors Color D Reset colors Paint [ / ] Decrease or increase brush size Paint { / } Decrease or increase hardness Paint 0 to 9 Set the tool opacity preset Layers Ctrl + J Layer via copy Layers Ctrl + Shift + J Layer via cut Layers Ctrl + ] / [ Move the layer up or down Layers panel ↑ / ↓ Previous or next blend mode (when focus is in the Layers panel) Transform tool Enter / Esc Commit or cancel the transform Transform tool . Reset the transform box Transform tool Ctrl + Z / Ctrl + Shift + Z Undo or redo a transform step Crop tool Enter / Esc Commit or cancel the crop Sketches saved from a standalone sketch page also respond to Ctrl + S or ⌘ + S. 3D model editor These keys work only while the 3D model editor is the visible tab. Windows/Linux Mac Action G G Translate gizmo R R Rotate gizmo S S Scale gizmo Delete or Backspace Delete or Backspace Delete the selected object and its children Ctrl + S ⌘ + S Save the scene to the asset See the 3D Editor guide. Game editor The 2D viewport and the scene hierarchy handle keys while they have focus. The editor does not handle these keys while a game is playing. Key Action Ctrl/⌘ + Z / Ctrl/⌘ + Shift + Z Undo or redo Ctrl/⌘ + C / V Copy or paste the selected entities Ctrl/⌘ + D Duplicate the selected entity Delete or Backspace Remove the selected entities and their children ← → ↑ ↓ Nudge the selection by 0.25 units Shift + ← → ↑ ↓ Nudge the selection by 2.5 units F Center the 2D camera on the selection Home Reset the camera G Toggle snap to grid (2D viewport) Space (held) Pan with the mouse in the 2D viewport Alt + ← → ↑ ↓ Move the selected entity within the scene hierarchy The 3D viewport has its own tool keys: Key Action W Move tool E Rotate tool R Scale tool F Frame the selection X Switch the gizmo between local and world space . Next pivot mode: selection centre, active entity, individual origins Ctrl/⌘ + A / Esc Select every entity in the scene, or clear the selection End Align the selection to the ground Numpad 1 / 3 / 7 Front, right or top view. Add Ctrl/⌘ for back, left or bottom Numpad 5 Switch between perspective and orthographic views W A S D, R / F Fly the camera, with R and F rising and descending (when Fly camera is on) While a game is playing, W, A, S, D, the arrow keys, and Space go to the game. Which keys do what depends on the game’s input actions. See the Game Editor guide. Desktop app menu The desktop app menu bar adds accelerators that send the same actions to the editor. Ctrl/⌘ + S, + T, + W, + D, + Shift + D, and + G match the editor keys. Two differ from the web editor: Fit View is Ctrl/⌘ + 0 and Align with Spacing is Ctrl/⌘ + Shift + A. See Electron Views for the full menu.--- # M1 implementation plan — guest JS end to end (feature-flagged) Source: https://docs.nodetool.ai/m1-implementation-plan.html M1 implementation plan — guest JS end to end (feature-flagged) Task breakdown for milestone M1 of sandbox-package-design.md. M0 shipped the catalog, discovery, and install trust flow; M1 makes a declared sandbox module importable in guest code, on every execution path, behind a parity flag. Each task lands green on its own and carries its tests. Landed. Tasks 1 and 2 in d7d0b47, task 3 in 8ac8ebe, tasks 4 and 5 in 1142c8f. M1 is complete behind NODETOOL_SANDBOX_MODULES_V1; see the M1 checkpoint in sandbox-package-design.md. Task 1 — Entry builder: AST transform for static imports (landed, d7d0b47) Replace wrapCode()’s string wrapping (packages/agents/src/js-sandbox.ts:2194) with an AST-based entry builder for code that contains imports: Parse user code (acorn, as code-analysis.ts already does), hoist static ImportDeclarations to module top level, and wrap the remainder in the async IIFE that keeps return and top-level await meaning what they mean today. Import-free code keeps the current fast path byte for byte, so existing behavior and tests do not move. The timer-deletion statements wrapCode emits stay in the entry module, but hardening for loaded modules moves to the loader (task 2): modules must evaluate under hardened globals, which the entry module runs too late to guarantee. Preserve the CodeNode genProcess streaming rewrite (packages/code-nodes/src/nodes/code-node.ts:402) on the transformed body. Tests: extend packages/agents/tests/js-sandbox.test.ts wrapCode describe — hoisting, implicit return, top-level await, return inside the body, source of syntax errors still pointing at user code. Task 2 — Runtime loader: the enforcement boundary (landed, d7d0b47) runInSandbox gains a modules option carrying a SandboxModuleResolution (@nodetool-ai/protocol), and installs a custom module loader/normalizer on the QuickJS context that resolves only the resolved module ids and their intra-pack siblings: Everything else fails at resolve with an error naming the node’s packages declaration: node:*, the wrapper’s compat modules, absolute and encoded paths, computed dynamic imports, sibling escapes, module-cache hits. Hardening precedes evaluation: the deleted globals and timer removal are in effect before any loaded module’s top-level code runs. Guest modules evaluate under the existing interrupt handler and memory limit. No new budget knobs. When modules is absent or empty, behavior is exactly today’s: no loader surface, imports are a syntax/validation error. Tests: adversarial fixtures in js-sandbox.test.ts — every denial case above, plus a two-module pack with an internal helper, and a check that a module cannot import another pack’s module undeclared. Task 3 — Code node packages + validation + CLI/server wiring (landed, 8ac8ebe) CodeNode (packages/code-nodes/src/nodes/code-node.ts) gains a packages property: a list of SandboxModuleDeclaration. process and genProcess resolve it through context.sandboxModuleCatalog and pass the resolution to runInSandbox. A missing catalog or failed resolution is a node error naming the pack. validateCodeNodeBody (packages/node-sdk/src/code-node-validation.ts) learns declarations: an import whose specifier is declared is legal; an import missing from packages stays an error (message updated — the “sandbox has no module loader” wording goes away); unknown specifier errors name the pack; declared-but-unused warns; version and digest mismatch warn (catalog statuses already carry these). validateGraph (graph-validation.ts:708) threads the catalog so offline validation resolves declarations; nodetool validate, node run, and debug reproduce a failing import headlessly through the same catalog the CLI already installs. Tests: code-node-validation.test.ts, code-node.test.ts end-to-end with a fixture pack, context.test.ts untouched (injection default already covered). Task 4 — CodeAct session allowlist + one-line prompt tier (landed, 1142c8f) CodeActExecutorOptions and ChatCodeActSessionOptions gain a session package allowlist (default: trusted packs only, per the design’s trust model). Each step’s generated code is parsed for imports; specifiers on the allowlist are resolved through the catalog and mounted, anything else is an action error the model sees as an observation. buildCodeActSystemPrompt advertises one line per session-allowed specifier — specifier plus the manifest-derived description under the strict length/character limits (SandboxModuleSummary.description is already capped at 160) — never the whole installed catalog. The sandbox manifest (code-gen/sandbox-manifest.ts) gains the packages section that pins this tier, and the drift tests (codeact-prompt-drift.test.ts, sandbox-manifest-drift.test.ts) extend to cover it. Every “no module loader” prompt string moves to the new truth: modules exist, only declared/allowed ones resolve. Tests: codeact-executor.test.ts (allowlisted import mounts, undeclared import fails as observation), drift tests, chat-codeact.test.ts (chat sessions default to no packages). Task 5 — Parity flag, browser refusal, docs (landed, 1142c8f) A NODETOOL_SANDBOX_MODULES_V1 opt-in flag, placed where server, CLI, and validation can all read it (not in websocket — the existing SDK flag module is the pattern, not the home). While the flag is off the loader never mounts and validation treats imports as today. While flagged on, parity is the gap: validation reports the rollout issue — “Sandbox package imports are unavailable in the browser runner until module delivery is enabled” — for any graph using packages, and the browser runner refuses such nodes with the same message. BrowserEligibility (web/src/lib/workflow/browserWorkflowRunner.ts) becomes property-aware for this one check. Nothing persistent is written: no node platform metadata changes. Docs: design doc status line moves to “M1 implemented”, CLAUDE.md sandbox notes updated in the same PR. Tests: flag module unit tests, browser-platform.test.ts, a graph-validation case for the rollout warning. Sequencing Tasks 1 and 2 are one work package (both live in js-sandbox.ts; the loader decides where hardening runs, which constrains the transform). Task 3 depends on 1+2. Tasks 4 and 5 depend on 3 (validation wording, resolution plumbing). Order: (1+2) → 3 → (4+5), one agent per work package, sequential on this branch. Exit criteria An authored fixture pack’s module imports and runs in a Code node via server execution, CLI node run, and CodeAct — and is refused with the parity message in the browser runner. Every adversarial fixture (deny node:*, compat modules, computed imports, path escapes, undeclared siblings) fails on the loader, not only in static validation. npm run check green; drift tests pin the new prompt tier.--- # M2 implementation plan — delivery parity (removes the flag) Source: https://docs.nodetool.ai/m2-implementation-plan.html M2 implementation plan — delivery parity (removes the flag) Task breakdown for milestone M2 of sandbox-package-design.md. M1 makes declared sandbox modules importable on the server and CLI behind the NODETOOL_SANDBOX_MODULES_V1 parity flag; the browser runner refuses them. M2 delivers module source to the browser, runs the same loading/denial contract there, and removes the flag. Parity restored is the exit criterion. Landed. Tasks 1, 2 and 5 in 126eceb; tasks 3, 4 and 6 in the commit that carries this banner. The NODETOOL_SANDBOX_MODULES_V1 flag no longer exists; see the M2 checkpoint in sandbox-package-design.md. Grounding: browser runs execute the kernel WorkflowRunner client-side through runBrowserWorkflow (packages/workflow-runner/src/browser.ts), routed by web/src/lib/workflow/browserWorkflowRunner.ts and browserRunnerCore.ts (worker first, main-thread fallback). ProcessingContext reads the process catalog default, which nothing sets in the browser — the browser catalog is the missing piece, not the runner. Task 1 — authorizeDelivery: the catalog’s delivery half The design deferred this method until a route exists; M2 is that route. Extend the SandboxModuleCatalog contract (packages/runtime/src/sandbox-module-catalog.ts) with the asynchronous, entitlement-aware operation: authorizeDelivery(moduleId, requester): Promise<AuthorizedSandboxModuleDelivery | SandboxDeliveryRefusal>. Authorization and retrieval are one operation, so the checks cannot drift apart. AuthorizedSandboxModuleDelivery (schema in packages/protocol/src/sandbox-package.ts) carries browser-safe content (JS source now; bytes later for WASM), media type, the module-graph digest, and dependency module ids — never a filesystem path. The node-sdk implementation (createSandboxModuleCatalog) resolves the opaque module id against the discovery graph. v1 entitlement policy: any authenticated client of this server may fetch modules of installed packs; the seam exists so a later private-pack policy is a policy change, not a plumbing change. Tests: sandbox-module-catalog.test.ts — unknown id, internal file id (delivered — the browser loader needs the whole graph), digest matches the resolution’s digest. Task 2 — Server route: GET /api/sandbox-modules/:moduleId New route module in packages/websocket/src/routes/ following the existing route pattern (routes/assets.ts et al). Delivery is by opaque module id, never path segments: scoped names contain / and encoded slashes are router-hostile. The route resolves through the catalog’s authorizeDelivery only — no route-to-filesystem translation. Responses carry the content digest (header) and long-lived caching headers keyed on it; unknown id → 404; refusal → 403. The registry summary field (sandboxModules: string[]) and per-file hashes stay out of scope (ecosystem milestone). Tests: websocket route tests — happy path, 404, digest header, and the response never containing an absolute path. Task 3 — Browser catalog: fetch, verify, cache A web-side SandboxModuleCatalog implementation (web/src/lib/workflow/sandboxModuleCatalog.ts): fetches by module id from the route, verifies the content digest client-side (crypto.subtle), and caches by digest (in-memory; digest-keyed, so a pack upgrade is a natural miss). resolveForExecution is synchronous by contract, and fetching is not: the run path prefetches. runBrowserGraphJob / updateBrowserJobNodeProperties (web/src/lib/workflow/browserWorkflowRunner.ts) collect the packages declarations from the graph’s Code nodes, prefetch every declared module and its dependency ids before the run starts, then hand the warmed catalog to the run’s ProcessingContext (both the worker and the main-thread fallback — the worker needs the declarations or the fetched sources passed across the boundary). A fetch or digest failure before the run is a job error naming the pack, same wording as the server’s resolution errors. Tests: browserWorkflowRunner.test.ts (prefetch collection, failure surfaces as job error), catalog unit tests with a mocked route (digest mismatch rejects and does not cache). Task 4 — One contract, two runtimes: shared fixtures Extract the M1 loading/denial fixtures (declared-import happy path, intra-pack helper, node:* denial, compat-module denial, computed dynamic import, path escapes, undeclared specifier, hardening order) into data-driven fixture definitions importable from both suites — home: packages/agents with a browser-safe export, mirroring the js-sandbox subpath pattern. Node side: the M1 vitest suite consumes the shared fixtures. Browser side: a Playwright leg in the existing e2e-runner harness (web/src/e2e_runner/, npm run test:e2e-runner) runs the same fixtures through a real browser QuickJS, via a fixture pack the e2e server has installed. Exit gate for the milestone: both suites green on identical fixture data. Task 5 — Electron staging for bundled builtins scripts/bundle-backend.mjs learns to stage sandbox module files of bundled builtin packs under _sandbox/<pack>/, and scripts/verify-backend-bundle.mjs verifies them. Installed packs are unaffected (they resolve through the optional-node root as today). No builtin ships sandbox modules yet, so this lands as mechanism plus a fixture-driven verification test; npm run backend:smoke stays the proof that staging did not break the artifact. Task 6 — Remove the flag Delete NODETOOL_SANDBOX_MODULES_V1, the validation rollout issue (“Sandbox package imports are unavailable in the browser runner…”), and the browser-runner refusal from M1’s Task 5. The property-aware eligibility check stays only if the flag removal leaves no other use for it. Because M1 wrote nothing persistent (no node platform metadata), removal is code-only — workflows saved during M1 need no migration. Docs: design doc status line moves to “M2 implemented”; CLAUDE.md browser-runner notes updated in the same PR. Tests: the M1 rollout-issue and refusal tests flip to asserting the paths are gone; browser-platform.test.ts gains the positive case. Sequencing Task 1 → Task 2 → Task 3 form the delivery chain. Task 4 needs 3 (browser loading exists) and is the parity proof. Task 5 is independent and can run any time. Task 6 lands last — the flag comes out only after Task 4’s both-runtimes gate is green. Exit criteria The M1 fixture pack’s module imports and runs in the browser runner, fetched by opaque id, digest-verified, cached by digest. The shared denial fixtures fail identically on Node and in the browser. The parity flag, the rollout validation issue, and the browser refusal no longer exist in the tree. npm run check and the e2e-runner leg green.--- # M3 implementation plan — npm compilation Source: https://docs.nodetool.ai/m3-implementation-plan.html M3 implementation plan — npm compilation Task breakdown for milestone M3 of sandbox-package-design.md: the config-only module form. A pack declares {"name": ".", "kind": "js", "npm": "js-yaml"} and NodeTool produces the guest module from the npm dependency — bundled, statically scanned, admission-probed, and cached by content. Grounding, verified against the tree: Discovery already parses the npm field and records the entry as a skip: sandbox-pack-discovery.ts emits npm-module-unsupported (“requires the M3 compiler”) and gives the module id npm:<name> with an empty dependency list. The catalog excludes those ids from resolution. M3 replaces that skip with a compiled artifact. The QuickJS engine dependencies live only in packages/agents, and agents depends on node-sdk — so node-sdk can never import the engine for the admission probe. esbuild is likewise not a node-sdk dependency. The compiler cannot live inside node-sdk without spreading esbuild concerns through it, which the design forbids anyway. Task 1 — Pay down the M-1 measurement debt The 1 MB npm-bundle cap is promised “validated in M-1”, and the design says to measure before the number lands anywhere user-visible. Bundle the real candidates with the exact Task 2 options — zod, date-fns, js-yaml, papaparse, fast-xml-parser, diff, fflate, cheerio — record the sizes in this document, and keep or move the cap on evidence. cheerio is the expected casualty (the design already hedges “if the bundle admits it”); a candidate that fails is recorded as out of scope for the bridge-pack milestone, not silently dropped. Measured candidates esbuild 0.28.1, conditions ["import","module","default"], mainFields ["module","main"], platform: "neutral", format: "esm", target: "es2022", no externals, no minification — the exact Task 2 options. Bundled from the versions installed in this tree. candidate version bundle inputs verdict zod 4.4.3 483.3 KB 80 scan error — const F = Function (×2); warns on navigator date-fns 4.4.0 175.6 KB 305 admitted — 250 exports through the probe js-yaml 4.3.0 101.4 KB 2 admitted — 15 exports through the probe papaparse 5.5.3 — — bundle failed — imports node:stream fast-xml-parser 5.7.3 142.1 KB 27 scan error — window && window.parseInt (×3) diff 4.0.4 35.1 KB 2 scan error — setTimeout (×2) fflate 0.8.3 60.8 KB 2 admitted — 49 exports; warns on queueMicrotask, setTimeout cheerio 1.2.0 — — bundle failed — imports 25 Node builtins Every rejection is a real one rather than a tooling artifact: zod reaches the Function constructor for its compiled validators, fast-xml-parser reads a bare window with no typeof guard, diff schedules with setTimeout. Each would throw in the guest at the line the scan names. cheerio is the expected casualty and papaparse joins it — both need the host bridge path, so both are out of scope for the bridge-pack milestone. Cap decision: keep 1 MB. The largest bundle measured is zod at 483 KB; the largest admitted one is date-fns at 176 KB. The cap is about 2× the worst candidate and 6× the worst realistic one, no candidate fails on size, and nothing measured argues for moving the number either way. It lives in NPM_BUNDLE_MAX_BYTES (packages/sandbox-compiler) and is re-checked against SANDBOX_PACKAGE_LIMITS.npmBundledJsBytes in node-sdk, so an artifact that grew past it after compilation is still refused at discovery. Reproduce with nodetool packs compile --json against a pack declaring the candidate, or drive bundleNpmModule / scanBundle / probeBundle from @nodetool-ai/sandbox-compiler directly. Task 2 — The compiler package New workspace package packages/sandbox-compiler, the dedicated compiler module. It depends on esbuild, acorn, and the QuickJS variant directly; node-sdk stays free of all three. compileNpmModule({ packDir, npmName, options }) runs esbuild: bundle: true, format: "esm", platform: "neutral", explicit conditions and mainFields pinned as constants (neutral alone does not pin which conditional export wins), no externals, no minification (digests and review beat bytes here). An import of a Node builtin fails the bundle; the failure becomes a named skip status at discovery (“imports node:fs — needs the host bridge path”), never a generic error. Output larger than the cap is a named skip carrying the measured size. Task 3 — Scope-aware forbidden-global scan The bundle then passes the authored-module static checks plus a scope-aware scan (acorn scope analysis, not a regex): free references to process, Buffer, require, eval, Function, timer globals, WebAssembly, and DOM names. A hard reference errors; a feature-detected reference (typeof process !== "undefined") warns — and the warning is a heads-up, not a compatibility promise. Local bindings that shadow those names are not hits; that is what “scope-aware” buys. Task 4 — QuickJS admission probe Admission ends by importing the bundle in the real engine, because bundling proves resolution, not compatibility: The probe context is capability-free: no fetch, no workspace or secret bridges, no tools, nothing from js-sandbox.ts’s bridge surface. The compiler package instantiates the engine directly with a short deadline, the normal memory limit, and output/log caps. The probe proves module initialization only — top-level code runs and the exports object materializes. It cannot prove every export works; runtime loading stays authoritative after admission. A module that fails the probe is skipped with the probe’s error. Task 5 — Content-addressed cache Cache keys are digests, never pack/name@version (linked packs, transitive updates, lockfile changes, and esbuild upgrades all change output while the version stays put): Key: hash over esbuild’s metafile input list (every input file’s content hash), the esbuild version, the compiler package’s own version, and the normalized build options. Value: the bundled source, the metafile summary, the scan report, and the probe verdict — so a warm cache skips the probe too. Writes are atomic (temp file + rename); every path component is sanitized; the cache lives under the existing per-user cache root from @nodetool-ai/config paths. Task 6 — Catalog and host integration discoverSandboxPack accepts compiled-artifact input (an injected lookup, keeping discovery synchronous and engine-free): an npm entry with a cached artifact joins the source graph like an authored file; one without becomes a pending-compile status instead of today’s npm-module-unsupported. The graph digest for npm entries incorporates what the design requires: the bundled source, compiler options, and compiler version. computeSandboxModuleGraphDigest already hashes authored files; the npm branch lands here. Compilation runs where the process is already async: the server’s bootstrapNodeRegistry path and soft reload, the Electron installer after a sandbox pack lands, and an explicit nodetool packs compile CLI command. The CLI’s synchronous buildFullRegistry never compiles — it reads the cache, and a miss surfaces as the pending-compile diagnostic naming the command. In-flight runs keep the resolution they started with, as with every catalog swap. packs.sandboxModules diagnostics carry the new statuses; the Package Manager shows a pending/failed compile per module with the skip reason. Task 7 — Fixtures and tests Fixture npm packages checked in under the compiler package’s test directory (tiny, dependency-free, no network): a clean ESM utility, one importing node:fs (named skip), one with a hard process reference (scan error), one with a feature-detected reference (scan warning, admitted), one whose top-level throw fails the probe, one over the size cap. Cache tests: identical input hits; changing one source file, the esbuild version, or an option misses; concurrent writes stay atomic. An end-to-end test through the catalog: the fixture pack’s npm module resolves for execution with the compiled source and correct digest, and runs in the M1 loader. Sequencing Task 2 → 3 → 4 are the compile pipeline, in execution order. Task 5 wraps them; Task 6 wires the result into discovery, hosts, and CLI; Task 1 runs first (its numbers are Task 2 constants); Task 7 grows with each piece. M3 depends on M1’s loader for the end-to-end proof but not on M2 — a compiled npm module is server/CLI-usable while browser delivery ships separately, and delivery of compiled modules rides M2’s route unchanged (they are catalog modules like any other). Exit criteria @nodetool-ai/sandbox-yaml-shaped fixture (config-only manifest, no authored code) installs, compiles, probes, and its module imports and runs through the M1 loader on server and CLI. A builtin-importing, a scan-failing, a probe-failing, and an oversized candidate each surface as named skips in packs.sandboxModules, not errors. Cache invalidates on content, never on version alone. The measured candidate table exists in this document with the cap decision recorded; npm run check green.--- # M4 implementation plan — WASM execution (scalar-only) Source: https://docs.nodetool.ai/m4-implementation-plan.html M4 implementation plan — WASM execution (scalar-only) Task breakdown for milestone M4 of sandbox-package-design.md: declared WASM modules become callable from guest code through generated facades, on workers, instance per call, under layered budgets. Grounding: the static half already shipped in M0. validateWasmModule/parseWasm in packages/node-sdk/src/sandbox-pack-discovery.ts enforce the whole binary contract — magic/version, zero imports, every memory unshared with a maximum at or below both the manifest’s memoryPagesMax and the 4096-page host ceiling, declared exports present and function-kind, arity ≤ 8, ≤ 1 result, i32/f32/f64 value types only — and the resolution schema carries WASM bytes through the catalog. What does not exist is everything at run time: the M1 loader rejects kind: "wasm" resolutions with an M4 error. Task 1 — Validation gaps and manifest mapping Close the deltas between the shipped validator and the design text: Export-name identifier rule: a manifest export must be a valid, non-reserved JS identifier, and a binary export that is not ("foo-bar") must use the explicit { "wasm": "foo-bar", "as": "fooBar" } mapping (the schema already allows the object form — validate the as name). Skip reasons name the offending export and rule (“uses i64”, “is a memory, not a function”) rather than the generic unsupported- signature message, per “named in the skip reason, not silently dropped”. Per-call timeout and budget fields a manifest may lower (never raise): schema additions with the defaults as ceilings. Task 2 — The WASM host: workers, pool, instance per call New module in packages/agents (beside js-sandbox.ts, which owns the guest side and already runs on Node and browser): Compile once per process from the resolution’s bytes; WebAssembly.Module is structured-cloneable, so the compiled module crosses to workers without recompiling. Calls run on workers — Node worker_threads, browser Web Worker — from a process-wide pool of 4. A hard per-call timeout (5 s default) terminates and replaces the worker. Instance per call, stateless: each call instantiates fresh from the cached module inside the worker, runs, discards. Mutable globals and linear memory do not survive between calls; the docs say so. This is what makes terminate-and-replace coherent — a killed worker destroys nothing an invocation owns. Argument conversion is validated host-side before dispatch, never left to WebAssembly coercion: i32 requires a finite integer in int32 range (out-of-range rejects, no wrapping); f32/f64 accept any JS number including NaN and infinities, f32 rounding by WebAssembly’s rules. Return values mirror the same rules; a void export resolves undefined. Task 3 — Budgets Layered, defaults fixed by the design, a manifest may lower: Bound Default Process-wide worker pool 4 Per-invocation call concurrency 2 Calls per invocation 256 Aggregate WASM wall clock per invocation 30 s Per-call timeout 5 s Exhaustion errors name the budget and the module. No byte caps — those belong to the future byte ABI. Task 4 — Generated facades and the per-run dispatcher For a WASM entry, the catalog resolution carries a generated ESM facade: named async exports matching the manifest’s exports, each calling a per-run dispatcher. Authored JS in the same pack importing a sibling WASM specifier resolves to the same facade. The facade source and generator version join the module-graph digest (the digest schema already anticipates this). The dispatcher is the boundary, not the hiding: it serves only WASM modules declared for the run, validates module identity, export allowlist, and argument count/types on every call. The static analyzer and the M1 loader both deny direct imports of the private bridge module, and the dispatcher binding is removed before the user IIFE starts — but a module that finds the binding anyway gains nothing beyond the run’s declared WASM surface. The M1 loader’s “WASM is M4” rejection is replaced by facade mounting. Task 5 — Reference module and toolchain doc One scalar reference module built from a documented toolchain (Rust no_std → wasm32-unknown-unknown, or WAT via wabt — whichever the doc pins), checked in as bytes with its source and build instructions. It is the fixture for every execution test and the template third parties copy. The byte-oriented reference module from M-1 informs the future byte ABI and ships as documentation only. Task 6 — Contract tests, Node and browser Happy path: facade import, scalar round trips for all three types, void export, as-mapped name. Conversion rejections: i64-shaped values, out-of-range i32, non-numbers. Statelessness: a mutable-global counter reads fresh every call. Budgets: per-call timeout kills and replaces the worker (an infinite loop export), call-count and wall-clock exhaustion, concurrency cap. Boundary: undeclared module id at the dispatcher, direct bridge import denied, facade of a pack not declared for the run denied. The same fixtures run on Node (vitest) and browser (the M2 shared fixture harness). Sequencing Task 1 is independent and lands first. Task 2 → 3 → 4 build on each other; Task 5 supplies the fixture Task 2 needs on day one (build it first). Task 6 grows with each. Depends on M1 (loader, facade mounting); browser-side tests depend on M2’s harness. Independent of M3. Exit criteria The reference module’s export is callable from a Code node via its facade on server, CLI, and browser; the call runs on a worker, instance per call. Every budget bound is observable in a test, including timeout-terminate-replace. An adversarial guest cannot reach the dispatcher beyond the run’s declared WASM surface. npm run check green on Node; the shared fixture suite green in the browser.--- # M5 implementation plan — agent and UI disclosure Source: https://docs.nodetool.ai/m5-implementation-plan.html M5 implementation plan — agent and UI disclosure Task breakdown for milestone M5 of sandbox-package-design.md. The one-line prompt tier and the CodeAct session allowlist shipped with M1; M5 is the documentation layer on top: full SKILL.md retrieval under the trust rules, package-picker rendering, and Package Manager consent language. It lands only after the untrusted-content policy below is wired exactly as designed — quoting is risk reduction, not isolation, so the policy is about when the agent sees pack docs, not how they are delimited. Landed. All five tasks shipped together; see the M5 checkpoint in sandbox-package-design.md for what is where. Task 1 — Full SKILL.md parsing at discovery M0 validates only size and minimal frontmatter shape. Hoist the existing AgentSkill parser (packages/agents/src/agent.ts) so node-sdk can call it — per the design, format and parser are the existing skill machinery, moved to where discovery lives (the agents package depends on node-sdk, so the shared parser moves down the dependency order, likely into node-sdk itself or protocol, mirroring how code-analysis.ts serves both). Discovery output gains the parsed skill (title, description, body) per pack; parse failures stay warning-grade — a frontmatter typo must never disable working modules. A pack exposing several specifiers carries one skill with a section per module. Task 2 — Trust-scoped disclosure paths Trusted pack (on the pack-loader allowlist): its skill registers through the normal skill system and can be injected like any other skill. Untrusted pack: the body is never injected. The agent sees it only as untrusted tool output, and only after the package has been chosen for the session (on the CodeAct session allowlist from M1). A new tool — get_sandbox_package_docs(specifier) — serves the body wrapped as untrusted content, refusing specifiers not on the session allowlist. The ambient one-line tier stays exactly what M1 shipped: manifest-derived, strict length and character limits, never arbitrary pack text. M5 adds no third tier. Task 3 — Code node package picker The Code node property panel gets the package picker: session-installed sandbox packs with their one-liners, and the rendered SKILL.md on selection (same file the agent retrieves; rendering is for the human). The picker writes SandboxModuleDeclarations onto the node’s packages property (M1’s shape), stamping resolvedPackVersion and contentDigest from the catalog summary. The consent wording is the design’s: “runs inside your workflows with the node’s capabilities” — never “no trust needed”. UI per repo rules: ui_primitives only, design tokens, catalog data over the existing packs.sandboxModules tRPC surface (extended with the rendered skill body by opaque request, not ambient in the summaries). Task 4 — Package Manager consent language Settings → Packages already shows mode chips and the trust-and-rebuild action (M0). M5 adds the sandbox-consent sentence to sandbox-only and hybrid packs, the per-module one-liners, and the SKILL.md view. Install flow wording states what importing means at the moment of install, per the trust model. Task 5 — Drift tests and prompt-budget guard The M1 drift tests pin the one-line tier. M5 adds: the disclosure tool’s output is wrapped as untrusted content (string-pinned, like the existing prompt drift tests); the one-liner derivation enforces its length/charset limits against adversarial manifest fixtures (control characters, prompt-injection phrases in description — the limits are the defense, the test proves them); and skill registration for a trusted pack does not leak into a session where the pack is not allowlisted. Sequencing Task 1 → 2 (parser before disclosure), then 3 and 4 in parallel (both consume Task 2’s surfaces), Task 5 throughout. Depends on M1 (session allowlist, declarations); benefits from M2 (a picker-declared pack should run in the browser) but does not require it. Exit criteria A trusted fixture pack’s skill registers; the same pack untrusted yields docs only via the tool, only when session-allowed, wrapped as untrusted output. The picker produces valid stamped declarations and shows the consent wording. Adversarial manifest fixtures cannot place uncapped or unfiltered text into any always-on prompt tier. npm run check green.--- # M6 implementation plan — library packs Source: https://docs.nodetool.ai/m6-implementation-plan.html M6 implementation plan — library packs Landed, then reworked. The first M6 shipped three packs and kept the data.* bridges beside them — “an added import path, not a migration”. The repo owner rejected that: “I don’t want to keep the data.* bridges. I want every module to be imported properly — even the nodetool modules.” The rework makes import the only way to reach a library. What changed: data.* is gone. The namespace, its eleven members, its entries in the sandbox manifest and SANDBOX_GLOBALS, its prelude block, and every prompt and snippet that named it. Host-facade modules. A new manifest kind, {"kind": "host", "host": "<id>"}, is the host-JS analog of M4’s WASM path: a generated ESM facade over a per-run dispatcher, with the implementation in packages/agents/src/host-modules/ — the moved bridge code, limits included. A pack declares an id; NodeTool owns the code. Eight packs, two guest and six host (table below). Limits moved, not dropped. MAX_UNZIP_TOTAL_BYTES (50 MB), DEFAULT_SELECT_HTML_LIMIT/MAX_SELECT_HTML_LIMIT, and the shared 5 MB/10 MB input caps live in the implementations now, each with its test. zip moved from guest to host. M3 admitted fflate, so the first M6 shipped it in the guest. The 50 MB inflation cap cannot be enforced there — a guest module is the guest, so there is no boundary below it to cap — and the 64 MB heap does not replicate the policy. Host it is. Disposition Pack Library Runs Why @nodetool-ai/sandbox-yaml js-yaml guest 101 KB bundle, 15 exports through the probe @nodetool-ai/sandbox-dates date-fns guest 176 KB, 250 exports @nodetool-ai/sandbox-csv papaparse host imports node:stream, so the bundle fails @nodetool-ai/sandbox-html cheerio + turndown host 25 Node builtins; turndown wants a DOM @nodetool-ai/sandbox-xml fast-xml-parser host reads a bare window with no typeof guard @nodetool-ai/sandbox-xlsx exceljs host Node streams; never a guest candidate @nodetool-ai/sandbox-diff diff host schedules with setTimeout @nodetool-ai/sandbox-zip fflate host compiler admits it; the inflation cap does not survive the move The M3 measurement still decides guest admission. What it no longer decides is whether a library gets a pack: a library the compiler rejects gets a host pack instead of a global. The earlier alternative — vendoring an authored kind: "js" fork of each rejected library — stays rejected, for the same reason: owning someone else’s code and its licence trail forever is a larger commitment than one import path is worth. A host module costs nothing like that, because the code was already in this repo, running host-side, behind the bridge that just went away. Trust kind: "host" is the one manifest kind that names NodeTool’s own code, so it is the one that had to be closed. Two checks, both by identity and both repeated: Discovery refuses an id that is not in SANDBOX_HOST_MODULES, and refuses a known id declared by a package that table does not pin it to. The dispatcher repeats both when it is built — a resolution can arrive from a catalog this process did not build, since the browser fetches one over HTTP — and then validates the module key, the export name, and the argument list on every call. packages/sandbox-compiler/tests/packs.test.ts drives both refusals with real crafted manifests; packages/agents/tests/host-modules.test.ts drives the dispatcher’s. Migration Everything that named a data.* member moved to the import form with the matching packages declaration: the sandbox manifest and its drift tests, the node-sdk SANDBOX_GLOBALS set, the run_code/js tool prompts, the Code node’s own description, the editor’s chat sandbox docs, five HTML code snippets (which now carry a packages list the palette seeds onto the node), the CodeAct batching example, and the live API script. js-sandbox-data-libs.test.ts became host-modules.test.ts. data.parseYaml/toYaml had no host pack of its own — js-yaml already ships as a guest pack, so YAML is @nodetool-ai/sandbox-yaml and nothing was duplicated. data.parseXlsx and data.htmlToMarkdown had no pack at all before; they became @nodetool-ai/sandbox-xlsx and the second export of @nodetool-ai/sandbox-html. No member survived as a global. Task breakdown for milestone M6 of sandbox-package-design.md: one config-only pack per migratable bridge library, living in the monorepo, consumed only via install. The original plan is kept below as written; the banner above records where the rework departed from it. Task 1 — The pack workspace New directory packages/sandbox-packs/ holding one npm package per library, each just a package.json (config-only manifest, M3 form) plus SKILL.md: Pack npm dependency Note @nodetool-ai/sandbox-csv papaparse   @nodetool-ai/sandbox-yaml js-yaml   @nodetool-ai/sandbox-xml fast-xml-parser   @nodetool-ai/sandbox-diff diff   @nodetool-ai/sandbox-zip fflate see Task 3 @nodetool-ai/sandbox-html cheerio only if the M3 size measurement admitted it; otherwise dropped here and recorded     Each manifest is {"name": ".", "kind": "js", "npm": "<lib>"} — the single-module root form, so usage reads import Papa from "@nodetool-ai/sandbox-csv". These packages are not part of the root workspace build graph the way backend packages are: they ship no compiled output and are consumed only via the package-manager install flow. Whatever workspace wiring they get must not let anything import them in-process. Task 2 — Skills A SKILL.md per pack: the specifier, the main functions with one example each written for the sandbox context, and the gotchas that matter there — input size caps, the 64 MB guest heap, no timers, and where the hardened data.* bridge remains the better route. The M5 disclosure machinery serves them; nothing new here beyond the files. Task 3 — The zip-bomb boundary, written down fflate’s bridge enforces a 50 MB decompression cap (MAX_UNZIP_TOTAL_BYTES) that the guest heap does not replicate — a 64 MB heap does not bound what a stream inflates through. The sandbox-zip SKILL.md and the pack description must steer decompression of untrusted archives to the data.unzip bridge, and the design doc’s bridge table stays authoritative: exceljs (Node streams), turndown (DOM), format.* (no Intl), image/canvas (native), and every capability bridge stay host-side permanently. Nothing in this milestone touches a bridge limit. Task 4 — Registry and validation presence The registry index entries for the six packs carry the sandboxModules summary and a recommended mark. Presence is never assumed: nodetool validate on a workflow declaring an uninstalled bridge pack fails with “install " (the M1 unknown-specifier error already names the pack; this adds the install hint for known registry packs). Prompts advertise only installed packs — already M1’s rule; the test here proves an uninstalled bridge pack never appears in the one-line tier. Task 5 — End-to-end proof and CI An integration test installs sandbox-yaml through the real scripts-disabled install flow, compiles it (M3), and runs import yaml from "@nodetool-ai/sandbox-yaml" in a Code node on the kernel — the full path a third-party pack will take. The harness gate surface registry gains the sandbox-packs surface with this test as its selfcheck, keeping the packs from shipping uncovered. Sequencing Task 1 → 2 → 4 → 5; Task 3 is wording inside 1–2. Depends on M3 (the compiler is what turns these config-only manifests into modules) and M5 (skill disclosure). cheerio’s fate is decided by M3 Task 1’s measurement, not here. Exit criteria Each shipped pack installs, compiles where it compiles, and imports in a Code node and CodeAct step; every pack’s path is CI-proven end to end. A workflow naming an uninstalled pack fails validation with the install hint. No data.* global survives, and every limit those bridges enforced has an equivalent inside a host module, with its test. A crafted manifest declaring kind: "host" with an unknown or forged id is refused at discovery and by the dispatcher. npm run check green.--- # MCP OAuth Authorization — Design Source: https://docs.nodetool.ai/mcp-oauth-design.html MCP OAuth Authorization — Design Status: proposed — 2026-08-23. Spec: MCP authorization (draft), built on OAuth 2.1, RFC 9728 (protected resource metadata), RFC 8414 (authorization server metadata), RFC 8707 (resource indicators), RFC 9207 (issuer identification), and OAuth Client ID Metadata Documents. Problem Connecting a remote MCP client to /mcp today requires a person to mint an ntk_ token in Settings → MCP → Connect an agent remotely and paste it into the client’s config (docs/mcp-production.md). That works, but: Every spec-conforming MCP client (Claude Desktop, Claude Code, VS Code, ChatGPT connectors) already speaks the OAuth discovery flow: hit the server, get a 401, discover the authorization server, open a browser, done. NodeTool answers that first unauthenticated request with a bare 401 — no WWW-Authenticate header (packages/websocket/src/lib/ws-upgrade.ts) — so the automatic flow dead-ends and the user is back to pasting tokens. An ntk_ token is a full-API bearer for its minter. A credential that exists only so a chat client can call MCP tools should not also authorize every REST route, tRPC procedure, and asset download. Pasted tokens have no client identity: revocation is “which of these five tokens named ‘claude’ is the one I still use?”. Decision summary NodeTool is both the resource server and the authorization server. The MCP spec lets the AS be a separate entity, but there is no external AS to point at in either auth mode: Local mode has no IdP at all, and Supabase GoTrue does not serve arbitrary third-party OAuth clients (no RFC 8414 metadata for them, no resource indicators, no CIMD). So the websocket server grows a minimal OAuth 2.1 AS, issuer = NODETOOL_PUBLIC_URL. User authentication inside the flow reuses whatever the install already has: the web session (Supabase mode) or the trusted-local rules (Local mode). Authorization code + PKCE (S256) only. Public clients, no client secrets, token_endpoint_auth_methods_supported: ["none"]. No client-credentials or implicit grants. Client registration: Client ID Metadata Documents first, RFC 7591 dynamic registration as the compatibility fallback. CIMD needs no registration table for the client itself — the client_id is an HTTPS URL the AS fetches and validates. DCR stays because shipped clients still use it. Opaque tokens, DB-verified, audience-bound to /mcp. Same hash-and-compare scheme as ntk_ (packages/models/src/access-token.ts), new prefixes so the auth hook can route verification: nta_ (access, short-lived) and ntr_ (refresh, rotating). An nta_ token authorizes only /mcp; every other route rejects it. ntk_ tokens keep working everywhere, unchanged. One scope: mcp. The MCP tool surface is already permission-gated per-call inside the agent-tools layer; inventing granular OAuth scopes now would duplicate that gate with a second, coarser one. The scope string exists so challenges and step-up have somewhere to grow. Roles and endpoints MCP client NodeTool server (one Fastify app, NODETOOL_PUBLIC_URL) ---------- ------------------------------------------------------- Resource server: POST/GET/DELETE /mcp (exists) GET /.well-known/oauth-protected-resource/mcp (new) GET /.well-known/oauth-protected-resource (new) Authorization server: GET /.well-known/oauth-authorization-server (new) GET /oauth/authorize (new) POST /oauth/token (new) POST /oauth/register (new, RFC 7591) POST /oauth/revoke (new, RFC 7009) All five new prefixes are auth-exempt (lib/public-routes.ts) — none reads per-caller state before authenticating on its own terms — and excluded from the SPA fallback (server.ts), or /.well-known/* gets served index.html. The flow sequenceDiagram participant B as Browser participant C as MCP client participant N as NodeTool (RS + AS) C->>N: POST /mcp (no token) N->>C: 401, WWW-Authenticate: Bearer resource_metadata="…/.well-known/oauth-protected-resource/mcp", scope="mcp" C->>N: GET /.well-known/oauth-protected-resource/mcp N->>C: { resource: "…/mcp", authorization_servers: ["…"], scopes_supported: ["mcp"] } C->>N: GET /.well-known/oauth-authorization-server N->>C: AS metadata (PKCE S256, CIMD supported, registration_endpoint, iss supported) alt CIMD Note over C: client_id = https URL of its metadata document else DCR C->>N: POST /oauth/register N->>C: { client_id } end C->>B: open /oauth/authorize?client_id&redirect_uri&code_challenge&resource&scope&state B->>N: GET /oauth/authorize Note over N: authenticate user (web session / trusted local), fetch+validate CIMD, render consent B->>N: approve (authenticated tRPC call) N->>B: 302 redirect_uri?code&state&iss B->>C: authorization code C->>N: POST /oauth/token (code, code_verifier, resource, redirect_uri, client_id) N->>C: { access_token: "nta_…", refresh_token: "ntr_…", expires_in: 3600, scope: "mcp" } C->>N: POST /mcp, Authorization: Bearer nta_… N->>C: MCP response 401 challenge denyUnauthorized (packages/websocket/src/lib/ws-upgrade.ts) grows an optional challenge parameter. The /mcp path — and only it, for now — answers: HTTP/1.1 401 Unauthorized WWW-Authenticate: Bearer resource_metadata="<PUBLIC_URL>/.well-known/oauth-protected-resource/mcp", scope="mcp" An expired or invalid token gets the same header plus error="invalid_token". Insufficient scope (future, once more scopes exist) gets 403 with error="insufficient_scope" and the scopes the operation needs. scopeRefusal() in mcp-server.ts keeps its JSON-RPC -32001 body but the HTTP layer adds the header. The challenge is emitted only when the flow can complete: mcpHttpEnabled and configuredMcpUrl() non-null (lib/mcp-mount.ts already owns both). A server with no NODETOOL_PUBLIC_URL cannot name its own resource URI and answers the plain 401 it does today. Protected resource metadata (RFC 9728) Served at both well-known forms the spec’s fallback order probes — /.well-known/oauth-protected-resource/mcp (path-aware, matching the /mcp endpoint path) and the root document: { "resource": "<PUBLIC_URL>/mcp", "authorization_servers": ["<PUBLIC_URL>"], "scopes_supported": ["mcp"], "bearer_methods_supported": ["header"], "resource_name": "NodeTool MCP" } resource is the canonical URI (lowercase scheme/host, no trailing slash), derived from configuredMcpUrl() — not a fourth copy of the public-URL logic. Authorization server metadata (RFC 8414) /.well-known/oauth-authorization-server: { "issuer": "<PUBLIC_URL>", "authorization_endpoint": "<PUBLIC_URL>/oauth/authorize", "token_endpoint": "<PUBLIC_URL>/oauth/token", "registration_endpoint": "<PUBLIC_URL>/oauth/register", "revocation_endpoint": "<PUBLIC_URL>/oauth/revoke", "response_types_supported": ["code"], "grant_types_supported": ["authorization_code", "refresh_token"], "code_challenge_methods_supported": ["S256"], "token_endpoint_auth_methods_supported": ["none"], "scopes_supported": ["mcp"], "client_id_metadata_document_supported": true, "authorization_response_iss_parameter_supported": true } code_challenge_methods_supported is load-bearing: spec-conforming clients refuse to proceed when it is absent. iss is emitted on every authorization response (success and error) per RFC 9207, so the flag is true. Client registration Client ID Metadata Documents (primary) A URL-shaped client_id (https, with a path) is fetched by the AS at authorize time and validated: the document’s client_id must equal the URL exactly, the structure must carry client_id, client_name, redirect_uris, and the request’s redirect_uri must exactly match one of them. The fetch goes through safeFetch from @nodetool-ai/runtime — the SSRF guard the spec’s security considerations call for already exists and is mandatory repo policy for any URL somebody else chose (docs/url-egress-inventory.md gets a new row). Responses are cached respecting HTTP cache headers, capped (size ≤ 64 KB, timeout 5 s), and never trusted across the client_id mismatch. Dynamic registration (fallback) POST /oauth/register accepts {client_name, redirect_uris, grant_types, token_endpoint_auth_method: "none", application_type} and answers a generated client_id (ntc_<random>), no secret. Rows live in a new mcp_oauth_clients table. Registration is open (that is the point of DCR) but bounded three ways: the existing fastifyRateLimit registration caps requests, a route-level bodyLimit of 8 KB caps each one (the server’s own limit is 100 MB, which at ten requests a minute is a gigabyte a minute of durable client metadata), and the document itself is capped — client_name ≤ 256 characters, at most 10 redirect_uris, each ≤ 2048 characters. Rows are garbage-collected when they age out with no grant ever issued (30 days), on the server’s periodic cleanup tick. Redirect URIs at registration and at authorize time must be https://… or loopback (http://127.0.0.1[:port]/…, http://localhost[:port]/…); anything else is rejected. Authorize and consent GET /oauth/authorize validates the request (client, redirect_uri, S256 code_challenge, resource equal to the canonical /mcp URI when present, scope ⊆ {mcp}), then hands off to the web UI: a redirect to /oauth/consent?request_id=… inside the SPA, with the pending request parked server-side (same in-memory TTL-swept store pattern as oauthStateStore in packages/websocket/src/oauth-api.ts, 10-minute TTL). The consent page runs inside the normal web app, so user authentication is whatever the install already enforces: Supabase session in cloud mode, the trusted-local rules in Local mode (where the auth hook resolves userId: "1" on loopback). Unauthenticated users go through the existing login and return. The page shows the client’s client_name, the redirect URI host (spec requirement — with an explicit warning when every redirect URI is loopback-only, since CIMD cannot prevent localhost impersonation), and the scope. Approve calls a new authenticated tRPC procedure (agentAccess.approveOauthRequest) that mints the code and returns the redirect URL; the SPA navigates there. Deny redirects with error=access_denied. Both redirects carry state (echoed verbatim) and iss. Errors that arrive before a redirect_uri is validated (unknown client, bad redirect_uri) render in-page and never redirect — open-redirect rule. Token endpoint POST /oauth/token, application/x-www-form-urlencoded: grant_type=authorization_code — verify: code exists, unexpired (≤ 10 min); client_id matches; redirect_uri exact-matches the authorize request; code_verifier S256-hashes to the stored code_challenge (constant-time compare — PKCEHelper in packages/runtime/src/providers/oauth/pkce-helper.ts already implements S256); resource, when sent, equals the one bound at authorize time. Every one of those is checked before the code is spent, so someone holding a leaked code but not the verifier cannot burn it — redeeming first made the legitimate exchange look like a replay and revoked the grant. Only once the bindings hold is the code consumed, and a reused code then revokes every token already issued from it (OAuth 2.1 code replay rule). grant_type=refresh_token — rotate: verify the ntr_ token, issue a new access + refresh pair, invalidate the old refresh token. A rotated-out refresh token presented again is reuse-detection: revoke the whole grant. The rotation runs in one transaction, and only the unique-index violation on rotated_from counts as reuse — a transient write failure rolls the whole attempt back and propagates, rather than revoking a working grant. Response: access_token (nta_…, expires_in: 3600), token_type: "Bearer", refresh_token (ntr_…, absolute lifetime 30 days), scope. Token model and storage Three new tables (Drizzle, SQLite + PG mirrors, next to schema/access-tokens.ts): mcp_oauth_clients — DCR rows: id (ntc_…), client_name, redirect_uris (json), created_at, last_used_at. CIMD clients are not rows; their identity is the URL. mcp_oauth_grants — one per (user, client) consent: id, user_id, client_id (URL or ntc_…), client_name (denormalized for the UI), scope, resource, created_at, revoked_at. The revocation unit the settings UI operates on. mcp_oauth_tokens — id, grant_id, kind: access | refresh, secret_hash (SHA-256, timingSafeEqual compare — the AccessToken scheme reused verbatim), expires_at, rotated_from (refresh lineage for reuse detection), last_used_at. Authorization codes are in-memory only (TTL store): a code lives ≤ 10 minutes and the loss on restart costs one browser round-trip. Nothing durable, nothing to migrate. Audience enforcement is structural, not claim-parsing: the auth hook in server.ts routes on prefix. nta_ verifies against mcp_oauth_tokens and sets req.userId only when req.url is /mcp; on any other path it is denyUnauthorized. The stored resource is additionally compared to configuredMcpUrl() so a token minted when the server lived at one public URL does not survive a rehost. This is the RFC 8707 audience-validation MUST, implemented without JWTs. Conversely /mcp keeps accepting ntk_, delegated, and Supabase tokens — the OAuth path is additive; token passthrough is still impossible because /mcp mints nothing and forwards nothing. What each auth mode gets   Local (default) Supabase / cloud (Fly) Flow enabled when NODETOOL_PUBLIC_URL is set (else unchanged: loopback trust means most local clients never see a 401) when NODETOOL_ENABLE_MCP=1 and NODETOOL_PUBLIC_URL set — same gate as the mount User auth at consent trusted-local (userId: "1") Supabase web session Issuer scheme http://localhost tolerated (spec permits loopback redirect URIs; a non-TLS non-loopback issuer disables the flow per request: no challenge, AS routes 404) HTTPS required No new enable flag: the ability to answer the challenge is derived from configuration that already exists. NODETOOL_DISABLE_MCP_OAUTH=1 is the escape hatch for an operator who wants token-paste only. Security checklist (spec MUSTs → where enforced) PKCE S256 mandatory, plain rejected — token endpoint. Exact redirect_uri match, registered values only — authorize + token. Single-use codes; reuse revokes issued tokens — token endpoint. Refresh rotation + reuse detection for public clients — token endpoint. iss on all authorization responses — authorize. Audience binding: nta_ valid only for /mcp + stored resource check — auth hook. No token passthrough: /mcp never forwards inbound bearers upstream — unchanged invariant. CIMD fetch through safeFetch (SSRF), size/time caps, client_id equality — authorize. Consent shows client name + redirect host, warns on loopback-only URIs — consent page. AS endpoints HTTPS in production; loopback exception local — enforced per request by the shared gate (oauth/gate.ts). Secrets stored hashed; raw token appears once in the token response — token model. Rate limits on /oauth/token and /oauth/register — existing fastifyRateLimit. Size limits on /oauth/register (8 KB body, bounded name and redirect URIs) — register route. Code bindings validated before the code is consumed — token endpoint. Out of scope, deliberately Granular scopes / step-up. One mcp scope until a real consumer needs a narrower one; the challenge plumbing (403 + insufficient_scope) is where it would land. NodeTool as AS for anything but /mcp. The REST/tRPC surface keeps its existing token types. JWT access tokens / introspection endpoint. One process is both AS and RS; a DB lookup is simpler and revocation is immediate. The .mcpb bundle and stdio transport. Stdio explicitly stays on environment credentials per spec; the bundle’s bearer-token user config keeps working. Implementation phases Resource-server half — metadata routes, WWW-Authenticate challenge, nta_ verification in the auth hook, tables. Independently shippable: clients discover, fail at the missing AS, fall back to token paste. Authorization-server half — authorize/consent/token/register/revoke, the tRPC procedure, the SPA consent page. Settings surface — agent-access.ts router + AgentAccessSection.tsx: list connected clients (grant rows), revoke, and report auth_mode: "oauth" in mcpConnection. docs/mcp-production.md rewritten around “connect via browser” with token paste demoted to fallback. Tests: route-level Vitest in packages/websocket (challenge shape, metadata documents, full code+PKCE happy path, every rejection in the checklist above — each proven able to fail per the repo’s check rules), model tests beside access-token.test.ts, and a probe that the SPA fallback does not swallow /.well-known/*.--- # MCP on a production server Source: https://docs.nodetool.ai/mcp-production.html MCP on a production server NodeTool speaks MCP over streamable HTTP at /mcp. An MCP client — Claude Code, Claude Desktop, Codex, anything else — reaches that endpoint and gets the agent toolbelt: workflows, assets, nodes, collections, jobs. On a laptop this needs no setup. On a server it needs two decisions: whether to serve the endpoint at all, and how an agent proves who it is. This page covers both. For the surrounding deployment, see Self-hosted deployment. 1. Turn the mount on The published image, fly.toml, and docker-compose.yml all set NODETOOL_ENV=production, and /mcp is off in that mode. The endpoint carries the full toolbelt for whichever user it binds, so a deployment opts in. The NodeTool production deploy does it in fly.toml’s [env]; a compose file does it the same way: environment: NODETOOL_ENABLE_MCP: "1" Without it the route is never registered — /mcp answers 404 for everyone, authenticated or not — and the boot log names the flag: MCP over HTTP (/mcp) disabled in production; set NODETOOL_ENABLE_MCP=1 to enable The mount is not a second door. It sits behind the same auth hook as /api, and binds the user that hook resolved. A request it cannot authenticate is refused at initialize with 401, never given an anonymous session. The session id in the mcp-session-id header belongs to its owner: a second user presenting the same id gets 404 Session not found, the same answer as an id that never existed. 2. Connect a client The browser path: OAuth With NODETOOL_PUBLIC_URL set, /mcp answers a spec-conforming MCP client’s discovery flow instead of a bare 401. Every spec-conforming client — Claude Desktop, Claude Code, VS Code, ChatGPT connectors — already speaks it: point the client at https://your-server/mcp, it discovers the authorization server, opens a browser, and you approve. No token to copy. 1. Client: POST /mcp (no token) 2. Server: 401, WWW-Authenticate: Bearer resource_metadata="…/.well-known/oauth-protected-resource/mcp" 3. Client discovers the AS metadata, opens a browser to /oauth/authorize 4. You sign in (if not already) and approve the client in the consent page 5. Server redirects back to the client with an authorization code 6. Client exchanges the code for an access token and connects What gets minted is an nta_… access token, scoped to /mcp only — it does not authenticate /api, /trpc, or anything else, unlike a pasted ntk_ token (below). It expires in an hour and refreshes itself; revoking the grant in Settings → MCP → Connected clients kills both the access and refresh token immediately. NODETOOL_PUBLIC_URL is required because the server has to name its own /mcp resource and authorization endpoints in the discovery documents it serves — without it, the flow cannot start and clients fall back to token paste. The URL must be HTTPS, or loopback (http://127.0.0.1, http://localhost) for local development; with a plain-HTTP public URL the flow stays off and /mcp answers the plain 401. NODETOOL_DISABLE_MCP_OAUTH=1 turns the whole discovery + AS surface off (no challenge header, the OAuth routes 404) for an operator who wants token-paste as the only path. See MCP OAuth design for the full protocol. The fallback path: paste a token Open Settings → MCP → Connect an agent remotely. Name a token, pick a lifetime, and the app hands back a claude mcp add … command and a JSON block with the URL and the token already in them. claude mcp add --transport http nodetool https://your-server/mcp \ --header "Authorization: Bearer ntk_…" Or, for a client that takes a config file: { "mcpServers": { "nodetool": { "type": "http", "url": "https://your-server/mcp", "headers": { "Authorization": "Bearer ntk_…" } } } } What that token is: A bearer credential for one user, the one who minted it. It authenticates the whole API, not only /mcp, and carries exactly that user’s access — no more and no less than their browser session. Shown once. The server stores a SHA-256 of the secret half and nothing else, so the plaintext exists only in the response that minted it. Lose it and you revoke and mint again; there is no recovery. Revocable, immediately. Revoking deletes the row, and the next request on that token is refused. This is the difference from the delegated tokens a messaging bridge uses, which are stateless HMACs that live until they expire. Optionally expiring. A lifetime is offered but not required; the list shows each token’s last use, so a forgotten one is visible before you revoke it. Tokens work in every auth mode, including local, and reach every route — not only /mcp. A self-hoster exposing /mcp beyond loopback should still prefer the OAuth path above when the client supports it: the token it mints is scoped to /mcp alone, so a leaked one cannot reach the rest of the API. The other paths A Supabase access token. With SUPABASE_URL and SUPABASE_KEY set, the server accepts a Supabase JWT as the bearer token, like any web client. Fine for a script that already signs in; a poor fit for a config file, since the JWT expires within the hour and nothing refreshes it there. Network trust. In local mode (no identity provider configured), NODETOOL_TRUST_LOCAL_NETWORKS=<cidr> trusts that range as user 1 with no token at all. Scope it to a VPN: anything reaching the server from a trusted range gets the whole toolbelt, and there is nothing to revoke afterwards. A delegated token. A bot or bridge holding NODETOOL_INTEGRATION_TOKEN mints a short-lived token per connection through the integration routes. This is for a bridge acting on behalf of a linked account, not for a person configuring a client. 3. Check it From the machine the agent runs on: curl -i -X POST https://your-server/mcp \ -H "Content-Type: application/json" \ -H "Accept: application/json, text/event-stream" \ -H "Authorization: Bearer ntk_…" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{ "protocolVersion":"2025-06-18","capabilities":{}, "clientInfo":{"name":"probe","version":"1"}}}' A working setup answers 200 with an mcp-session-id header. An nta_ token from the OAuth flow works the same way in place of ntk_…. What the other answers mean: Answer Cause 404 The mount is off. Set NODETOOL_ENABLE_MCP=1 and restart. 401 No token, or one that is wrong, revoked, or expired. WWW-Authenticate with error="invalid_token" means an nta_ token failed; without an error param, either no token was sent or a non-OAuth token (ntk_, Supabase) was refused. No header at all means the OAuth flow is not configured on this server. 404 on a request that carries a session id That session belongs to another user. Initialize a new one. What stays local The Install on this machine buttons in the same settings panel write MCP config files into ~/.claude.json and friends. They are disabled in production (mcpConfig answers 503) because on a shared host those files belong to nobody. The remote paths above are what works there. Python nodes A separate flag, and a separate gotcha: the published image ships no Python worker, so NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 alone changes nothing but the error. See Self-hosted deployment › Python nodes.--- # NodeTool as an MCP Server Source: https://docs.nodetool.ai/mcp-server NodeTool runs as a Model Context Protocol (MCP) server. An agent harness such as Claude Code, Codex, or OpenCode starts it and gets NodeTool’s toolbelt: workflows, image, video, and audio generation, assets, nodes, collections, and the editors. The desktop app is optional. Install You need Node.js 22 or later. Run this in a terminal: npx -y --package=@nodetool-ai/cli nodetool mcp install Or install the CLI once and use the short form: npm install -g @nodetool-ai/cli nodetool mcp install The command does these steps: It finds Claude Code, Codex, and OpenCode on the machine. It starts the server once and lists its tools. This proves that the server works and fills the npx cache, so the harness’s first start is fast. It writes a nodetool entry into each client’s config. Restart the harness, then ask it: “Use NodeTool to list my workflows.” To install for one client, add --claude, --codex, or --opencode. Add --no-verify to write the config without starting the server first. Check the result with nodetool mcp status --check. Remove the entries with nodetool mcp uninstall. What the install writes The entry starts nodetool mcp serve over stdio. The command is nodetool when the CLI is on PATH, and npx -y --package=@nodetool-ai/cli nodetool when it is not. Add --npx to always use npx. Client File Scope Claude Code ~/.claude.json, key mcpServers User scope, every project Codex ~/.codex/config.toml, block [mcp_servers.nodetool] Every session OpenCode ~/.config/opencode/opencode.json, key mcp Every session Codex gets a 120-second start timeout and a 900-second tool timeout, because a video generation can run for minutes. OpenCode gets the 120-second timeout. Earlier versions wrote the Claude Code entry under projects["<home>"], so it worked only in sessions started in the home directory. A new install moves that entry to the user scope. Other clients Print a config block for Cursor, Claude Desktop, Windsurf, or any client that reads mcpServers JSON: nodetool mcp config { "mcpServers": { "nodetool": { "command": "nodetool", "args": ["mcp", "serve"] } } } To register the server with Claude Code by hand, run: claude mcp add --scope user nodetool -- npx -y --package=@nodetool-ai/cli nodetool mcp serve Claude Desktop also installs the bundled extension from Studio: open Settings → MCP Servers → Install Extension. Provider keys Generation tools call provider APIs, so the server needs your keys. Store a key once in the NodeTool secret store: nodetool secrets store FAL_API_KEY nodetool secrets store OPENAI_API_KEY Keys you add in Studio settings go to the same store. When a key is not in the store, the server reads the environment variable with the same name. To pass one through Claude Code, add -e FAL_API_KEY=... to claude mcp add. Tools that need no provider work without keys: node search, workflow validation, assets, and the local file tools. With and without Studio nodetool mcp serve checks for a running NodeTool server on 127.0.0.1 at port 7777 (or PORT), over HTTPS when TLS_CERT and TLS_KEY are set. When one answers its MCP handshake, the stdio process forwards every request to it. Actions then run in that process, and an open editor shows the agent’s changes as they happen. When no server answers, mcp serve runs its own MCP server over the default NodeTool data directory. The agent keeps full access to workflows, assets, and generation. To connect a client to a running server over HTTP instead, install with --http. That entry works only while the server runs: nodetool mcp install --http # http://127.0.0.1:7777/mcp nodetool mcp install --url http://127.0.0.1:8000/mcp For a NodeTool server on another machine, create an access token in Settings → MCP Servers → Connect an agent remotely. See MCP on a production server for the server side. What the agent gets The server registers a short list of direct tools. Everything else NodeTool can do is reached through execute_code, which runs sandboxed JavaScript that calls nodetool.<namespace>.<method>(). Through it the agent reaches workflow building and debugging, media generation, assets, collections, documents, web search, and memory. The agent finds a tool with nodetool.searchTools("query"). The full list of sandbox tools is in the CLI reference. Direct tools Tool Purpose execute_code Run sandboxed JavaScript that calls the nodetool.* modules. The main tool. view_image Return an image as MCP image content so the agent can see the pixels. Takes an asset id, asset:// URI, public https:// URL, or data: URI. A local connection also accepts an absolute disk path. upload_asset Save bytes in your asset library. Takes content_base64, or an absolute file_path on a local connection. Returns the asset id and URI. download_asset Return an asset you own as base64, or write it to an absolute output_path on a local connection. list_directory List files and subdirectories in a workspace directory. browser Fetch a web page and return its readable text. http_request Make an HTTP request and return the response body as text. download_file Download a file from a URL, or from an asset:// URI, a /api/storage/ key, or a data: URI, into the workspace. run_subtask Hand a task to a child agent that has the NodeTool toolbelt. find_model Find a real provider and model id for a capability such as image, video, or speech. list_models List the models available from the providers you configured. list_provider_models List the language models of one provider. search_nodes Search node types by keyword. get_node_info Return a node type’s inputs, outputs, types, and defaults. list_nodes List node types, optionally filtered by namespace. search_nodes, get_node_info, and list_nodes are offered only when the session has the node registry loaded. The file and search tools read_file, write_file, edit_file, glob, grep, web_search, and todo_write are not direct tools, because Claude Code and similar clients already provide their own. They stay available inside execute_code. Resources and prompts Name Kind Purpose nodetool://capabilities Resource The modules mounted for the session, every tool an action can call, and each tool’s permission category, as JSON. nodetool://sandbox Resource How to write execute_code: the guest contract, blocked globals, unavailable bridges, and worked examples. sandbox-action Prompt A complete execute_code body that lists workflows, picks a model, and generates one image. sandbox-asset Prompt A complete execute_code body that lists one asset and fetches its ref. <skill-name> Prompt One prompt for each skill the user can load: the shipped system skills and the user’s own skills. The prompt returns the skill body. The optional request argument adds the task. Claude Code shows each prompt as a slash command. Troubleshooting Symptom Fix The harness shows the server as failed Run nodetool mcp status --check. It starts each registered server and prints the error. npm error could not determine executable to run The command lacks --package=@nodetool-ai/cli. The package has two binaries. Reinstall with nodetool mcp install. The first start times out Run nodetool mcp install again. It fills the npx cache before the harness starts the server. nodetool mcp serve fails, but npx works A legacy Python nodetool is first on PATH. Install with --npx, or put the npm global bin first on PATH. A generation fails with a missing key Store the key with nodetool secrets store <NAME>.--- # Media Generation Tracking — Technical Design Source: https://docs.nodetool.ai/media-generation-tracking-design.html Media Generation Tracking — Technical Design Status: Draft, for review Owner: Matti Georgi Related: packages/execution/src/cost-ledger.ts, packages/runtime/src/cost-reconciler.ts, tool-class-retirement-design.md, HARNESS_FIRST.md 1. Summary A media generation request is money out and minutes of wall clock, and today the record of it is written after it succeeds, by a listener that never sees a failure, a cancellation, or a process restart. The row it writes names no asset, the asset names no row, and the reconciliation that turns an estimate into the billed amount is a detached promise that dies with the process. One primitive fixes this: a generation is a row opened before the provider is called and closed with a terminal status, a cost, and the assets it produced. A follower owns that row from submit to close. It is not a new runtime: it is the ledger listener, rewritten to track a lifecycle instead of recording a success, plus a durable reconcile queue and a startup sweep. The follower runs in the process that made the call and outlives the caller (a cancelled chat turn, a finished tool call), so the outcome is recorded even when nobody is waiting for it. Agents get the record as a capability module, generations: list, get, await, cancel, reconcile. Generation capabilities gain background: true, which returns the generation id at once and leaves the follower to finish the job. The CLI gets the same surface as nodetool generations. Current implementation boundary The durable path now has two identities. A generation is keyed by its idempotency key and input fingerprint. Each provider attempt has its own attempt number, provider request id, endpoint and callback credentials. The generation row and the attempt row are lease-fenced, so a stale worker cannot overwrite a newer owner. The row records four independent dimensions: Dimension States Meaning Submission accepted, submitting, submitted, submission_unknown Whether NodeTool accepted and identified the provider submission Provider unknown, queued, running, succeeded, failed, cancelled The provider’s observed execution state Output pending, saving, ready, retrying, unavailable Whether the provider result has been recorded and saved Attachment pending, attached, superseded, target_deleted, retrying Whether a destination attachment is current The public status is derived from those dimensions. completed means durable output is ready. A provider success with output still saving is recovering. needs_attention covers unavailable or retrying output, retrying attachments, and unknown submission. Existing interrupted rows remain readable. FAL queue calls use an explicit submit, bind and wait sequence. The submission POST is made once, its provider request id is persisted before the status or result wait, and binding does not submit again. A signed callback is accepted into the durable inbox before the HTTP response is sent. The inbox verifies the Ed25519 signature and five-minute timestamp window, bounds the raw body to 10 MiB, deduplicates deliveries, and refreshes FAL’s JWK set through a cached, coalesced request. When no callback is configured, or when a callback is not available, the recovery worker polls the bound FAL request instead. It consumes pending inbox deliveries before polling attempts. The callback URL is generated only from a valid https NODETOOL_PUBLIC_URL. Local servers normally omit it and use queue polling. A hosted server must set it to an address FAL can reach. The callback route is POST /api/providers/fal/webhook/:token. Durable cancellation records cancel_requested_at; it does not immediately claim that FAL cancelled the request or close the generation row. The worker uses the bound request id to request provider cancellation and records the observed result. The in-process registry can still abort a local call, which then closes through the normal generation seam. This implementation does not checkpoint arbitrary agent or workflow stacks. Persisted generation state survives a socket or worker restart, but resuming a paused agent turn or workflow execution requires a separate checkpoint design. 2. What is wrong today Read with packages/execution/src/cost-ledger.ts and packages/runtime/src/context.ts open. F1. Only success is recorded. recordFromMessage writes a row on prediction + status: "completed". A failed prediction, a cancelled one, or one interrupted by a restart writes nothing. kie bills a task at submit and Meshy bills a preview pass before refine, so a failure is not free. F2. Cost and asset are not linked. A nodetool_predictions row has no asset id and an nodetool_assets row has no prediction id. “What did this clip cost” and “which asset came out of this charge” are both unanswerable. F3. The message id is not the row id. runProviderPrediction mints a randomUUID() for the prediction message; Prediction.create mints another for the row. A client that watched the message cannot find the row. F4. Reconciliation is fire-and-forget. recordNodeProviderCost calls void reconcileProviderCost(...). No retry, no record of whether it ran, and it dies with the process. kie never reconciles at all: reportKieProviderCost (packages/kie-nodes/src/kie-base.ts) omits provider_request_id though the task id is in hand. F5. Five caller surfaces, three bypass the seam. Workflow nodes and agent capabilities go through runProviderPrediction. The chat turn’s direct generation (packages/websocket/src/session/chat-turn.ts), the generate_media RPC (session/inference.ts) and nodetool generate (packages/cli/src/commands/generate.ts) call provider.textToImage(...) and friends directly, each with its own asset save and its own Prediction.create. BYOK direct generation writes no row by design. F6. 3D generation is off the books. TextTo3DNode and ImageTo3DNode (packages/video-nodes/src/nodes/model3d/generation.ts) call provider.textTo3D directly. text_to_3d and image_to_3d are absent from ProviderCapability and from UNIT_BILLED_CAPABILITIES. Meshy and Rodin spend reads as free. F7. A completed prediction message carries the raw output bytes as data. Every listener, the chat socket forwarder included, receives a full video buffer. Nothing in web/src reads data; the chat and workflow reducers read status, node_id and logs. F8. An agent cannot see a generation. get_cost_summary aggregates; no capability lists what is running, what a call cost, or where its asset is. 3. Design goals One row per generation, opened before spend. The row exists from the moment the provider is asked, whatever happens after. Every terminal state is a row state. completed, failed, cancelled, interrupted. Silence is never “free”. The asset is the outcome. A completed generation names its assets, and each asset carries its generation id. One seam. Every surface that asks a provider for media goes through ProcessingContext. An audit fails the build on a new bypass. Reconciliation is durable. A charge that can be reconciled is reconciled, retried with backoff, and the row says whether it was. The follower is the capability. Rule 4 of HARNESS_FIRST: the CLI and agent-tool forms are the feature, the dashboard renders them. 4. Architecture caller (capability · node · chat turn · RPC · CLI) │ context.runGeneration(req) ← the one seam (§5) ▼ ProcessingContext (packages/runtime/src/context.ts) id = randomUUID() emit prediction{status:"running", id, origin, params} dispatchCapability(provider, req) inside a receipt scope (AsyncLocalStorage) provider code: recordGenerationReceipt({provider_request_id, cost?}) persist output → createAsset(…, metadata.generation_id = id) emit prediction{status:"completed", id, asset_ids, receipt, duration} or {status:"failed", id, error, receipt} or {status:"cancelled", id, receipt} return {id, output, assets} │ addMessageListener ▼ GenerationTracker (packages/execution/src/generation-tracker.ts) ← the follower running → Prediction.create({id, status:"running", …origin}) completed → price (receipt cost wins, else catalog) · update row · enqueue reconcile failed → update row · enqueue reconcile when a request id exists cancelled → update row · enqueue reconcile when a request id exists ReconcileQueue: rows with provider_request_id, reconciled_at null, attempts < 5 StartupSweep: rows still "running" from a dead process → "interrupted" │ ▼ nodetool_predictions (packages/models) ← one table, lifecycle in place ▲ ▲ │ generations capabilities │ nodetool generations · nodetool costs · Costs dashboard The dependency arrows are unchanged: runtime knows nothing about models, execution imports both, hosts attach the tracker where they attach the ledger today (packages/execution/src/session.ts, service/workflow-run.ts, packages/websocket/src/session/chat-turn.ts). D1. Reuse nodetool_predictions; do not add a generations table The table already carries status (default pending), error, provider_request_id, parameters, metadata, started_at, completed_at, duration, and the unit-billing columns. nodetool costs, get_cost_summary and the Costs dashboard read it. A second table would split “what ran” from “what it cost” and every consumer would join them. The lifecycle goes into the row that already exists; the write path changes from insert-on-success to insert-on-start plus update-on-close. D2. The listener and durable recovery worker have different jobs The ordinary generation tracker watches the messages the seam emits and closes the in-process row. The durable recovery worker handles accepted attempts after the submitting call or process is gone. It consumes a signed FAL inbox delivery when one exists, otherwise binds the persisted provider request id and polls the queue. It never replays a paid submit. Other providers still need their own submit and poll adapters before they can use this recovery path. D3. Receipts flow through AsyncLocalStorage, not through return types Twenty-five provider methods return bytes. Threading a request id through each signature is a large diff for one field. packages/runtime/src/invocation-account.ts already scopes {costUsd, createdAssets} per invocation with AsyncLocalStorage; the receipt scope is the same shape. A provider calls recordGenerationReceipt({provider_request_id, cost?}) from inside the method and the seam reads it after dispatch. FAL (client.subscribe returns requestId) and kie (taskId, creditsConsumed) fill it in first. D4. The seam persists the asset Today persistOutput in packages/agents/src/tools/asset-persist.ts saves the asset after the seam returns, so the seam cannot name it. runGeneration takes a persist option and saves through createAsset before it emits completed, stamping metadata.generation_id, workflow_id, node_id and job_id on the asset. The caller gets assets back and stops saving on its own. A host with no createAsset interface (a hermetic eval, the CLI without a database) gets the bytes and writes a workspace file as before; the row then records asset_ids: [] and metadata.persisted: "workspace". Durable recovery records provider output separately from attachment state, so this path does not imply that every provider result has been copied to owned storage. 5. The seam In packages/runtime/src/context.ts. export type GenerationSurface = | "workflow" | "capability" | "chat" | "rpc" | "cli"; /** Who asked. Every field the row and the asset carry back to the caller. */ export interface GenerationOrigin { surface: GenerationSurface; thread_id?: string | null; tool_call_id?: string | null; request_id?: string | null; // the rpc surface's request (§5.3) job_id?: string | null; node_id?: string | null; } export interface GenerationRequest extends ProviderPredictionRequest { origin?: Partial<GenerationOrigin>; // defaults from the context (§5.1) persist?: { name?: string; // asset name; default `<capability>-<ts>.<ext>` mime?: string; // default sniffed from bytes, then by capability parentId?: string; // asset folder }; signal?: AbortSignal; // cancel_generation reaches the provider through it } export interface GenerationResult<T = ProviderPredictionResult> { id: string; // the row id and the message id (F3) output: T; assets: AssetRef[]; // empty when nothing was persisted receipt: GenerationReceipt | null; duration_ms: number; } export interface GenerationReceipt { provider_request_id?: string | null; /** The provider's own charge, when it states one. Wins over the catalog. */ cost?: Pick<ProviderCost, "amount" | "currency" | "billing_unit" | "quantity" | "unit_price"> | null; } runGeneration(req: GenerationRequest): Promise<GenerationResult>; runProviderPrediction(req) stays and becomes (await this.runGeneration(req)).output, so the node call sites keep compiling. textToSpeechEncoded, textToMusic and streamProviderPrediction get the same lifecycle (they already emit running/completed/failed; they gain the origin, the receipt and the cancelled state). 5.1 Origin defaults The context knows userId, jobId, workflowId and threadId. A request that names no origin gets surface: "workflow" with the context’s job and workflow ids inside a run, and surface: "chat" with its thread id inside a chat turn. A capability passes surface: "capability" and its tool_call_id explicitly, because that is the one id the context does not hold. 5.3 The rpc surface’s request id generate_media answers with an rpc_response carrying no job_id and no thread_id, so WebSocketClientSession.sendMessage writes it to the socket that asked and drops it if that socket has gone. A client that reloads mid-render therefore cannot receive its reply, and re-subscribing to the request id afterwards produces nothing: the frame was already delivered to a socket that no longer exists. The row outlives the socket. origin.request_id puts the client’s own request id on it, indexed as (user_id, request_id), and lookup_generations answers {status, asset_ids, error} for a batch of them — scoped to the caller’s rows and capped at MAX_REQUEST_ID_LOOKUP. The row is the recovery mechanism; the subscription is only an optimization. A subscription cannot recover a reconnect at all: GlobalWebSocketManager.subscribe is a client-side map with no replay buffer, so a reply written to a socket that no longer exists reaches nobody, and a handler installed afterwards has nothing to receive. Re-subscribing to a running request would leave the surface waiting forever. So every in-flight request is watched — watchGeneration polls lookup_generations for all outstanding ids together, backing off from 2s to 15s, until each row reaches a terminal state or its 30-minute window runs out (after which the surface fails and offers Retry rather than rendering forever). The subscription runs alongside and wins when it can, which is only when the socket is the same one the request went out on. Watching starts at the send, not just at reattachment, because a socket that drops and reconnects without a page reload leaves the reply addressed to a server session that is gone in exactly the same way — and nothing re-runs reattachment in that case. It also closes the window between a lookup and the subscription that follows it, where a reply arriving in between would reach no handler. An id with no row comes back absent, which the client reads as “still unknown” and keeps watching: the row may not be open yet, and reading unknown as failed would discard a render still in flight. 5.2 Message changes predictionSchema in packages/protocol/src/messages.ts: status exposes pending, running, recovering, completed, failed, cancelled, needs_attention and interrupted. submission_status, provider_status, output_status and attachment_status expose the independent lifecycle dimensions. add origin: GenerationOrigin, asset_ids: string[], receipt: GenerationReceipt | null. data stops carrying bytes (F7). It stays in the schema, nullable, for the segmentation result (ImageSegmentationMask[] metadata without pixels) and the ASR transcript, which are results and not media. params is redacted before it is emitted or stored: byte fields, data: URLs and base64 blobs are replaced with {bytes: n}. The redaction is the one the supervisor uses for Escalation.inputs. 5.3 Capabilities added to the seam text_to_3d and image_to_3d join ProviderCapability, dispatchCapability and UNIT_BILLED_CAPABILITIES (F6). The two 3D nodes call runGeneration instead of context.getProvider(...).textTo3D(...). 5.4 Cancellation runGeneration registers {id, abort} in a process-level GenerationRegistry for the life of the call. The registry is the shape of packages/agents/src/background-subtasks.ts (a record map, a version counter, a waiter list), lifted to the process because a generation outlives the turn that started it. cancel_generation aborts it; the seam emits cancelled with whatever receipt the provider recorded before the abort. Providers that already take signal (image and video params) stop polling; the rest finish their current HTTP call and the result is discarded. Cancelling does not refund: a provider that bills at submit still bills, which is why a cancelled row keeps its request id and is reconciled. 6. The follower packages/execution/src/generation-tracker.ts, replacing the prediction branch of recordFromMessage. The node_update + provider_cost branch stays for the node packages that call a provider SDK directly (§8, S7). 6.1 Row lifecycle Message Row write running Prediction.create({id, user_id, status: "running", provider, model, capability, surface, thread_id, tool_call_id, job_id, node_id, workflow_id, project_id, document_id, parameters, started_at}) completed update({status: "completed", cost, billing_unit, quantity, unit_price, currency, provider_request_id, asset_ids, duration, completed_at, metadata.price_source}), then enqueue reconcile when provider_request_id is set failed update({status: "failed", error, provider_request_id, completed_at}); enqueue reconcile when a request id exists, because a failed call may be billed cancelled as failed, with status: "cancelled" Pricing order on completed: the receipt’s cost when the provider stated one (metadata.price_source: "provider"), else the catalog through priceGeneration ("model-catalog"), else cost: null with unpriced_reason, as today. A null stays null: an unpriced row is countable, a zero is a lie. A create that fails (no database on this host) is logged at warn and the tracker remembers the id, so the later update for that id is skipped instead of logged again. Bookkeeping stays best-effort for a hermetic run, and loud. 6.2 Reconcile queue State lives in the row: reconciled_at, reconcile_attempts, metadata.reconcile_error. The queue is a query, not memory: provider_request_id IS NOT NULL AND reconciled_at IS NULL AND reconcile_attempts < 5 AND status IN ('completed','failed','cancelled') A worker drains it on a timer (every 5 minutes; the first attempt for a fresh row after 60 s, because billing events lag the request) with attempts spaced 1, 5, 30, 120, 720 minutes. A provider with no reconciler marks the row reconciled_at = now, metadata.reconcile: "unavailable" on the first pass so it leaves the queue. The FAL reconciler stays as it is (packages/fal-nodes/src/fal-billing.ts); kie gains one over /api/v1/jobs/recordInfo, whose status payload already carries creditsConsumed. The worker is attached once per process by the server (packages/websocket) and by the CLI when it opens a database; an eval attaches none. 6.3 Startup sweep On attach, rows with status = "running" and started_at older than the process start are set to interrupted with metadata.interrupted_reason: "process restart", and enqueued for reconcile when they carry a request id. A row a live process is still driving is never older than its own start, so the sweep cannot close a real run. 6.4 Run totals packages/websocket/src/session/job-execution.ts accumulates a run’s cost for the job row from the same messages (_handlePredictionCost, _handleNodeProviderCost, runMeasuredCost), re-pricing each prediction itself. It keeps doing that, from the new receipt field first and the catalog second, so the job total and the row agree by construction. It does not write rows; that stays the tracker’s. 6.5 Double counting One generation, one row. Two guards: A node that goes through the seam and calls setProviderCost (the FAL and kie factories, once they move to the seam in S7) would write a row from the prediction message and a second one from node_update.provider_cost. The tracker records the invocation’s generation ids; a node_update whose invocation already produced a row with a receipt cost writes nothing, and a node_update whose row exists without a receipt updates that row instead of inserting. The kernel already scopes an invocation (invocation-account.ts), so the mapping is by invocation, not by guess. runDirectMediaGeneration (inference.ts) writes its own Prediction.create after reserveSpend. That write goes; the seam’s row is the row, and the nodetool-provider budget path keeps admitSpend/ reserveSpend/releaseSpend around the runGeneration call. 7. Schema packages/models/src/schema/predictions.ts and schema-pg/predictions.ts, one migration in migrations/versions.ts: Column Type Why capability text the ProviderCapability; today only in metadata surface text GenerationSurface thread_id text, index (user_id, thread_id) the chat turn that asked tool_call_id text the capability call that asked job_id text, index the run that asked asset_ids json text the outcome (F2) reconciled_at text queue state (§6.2) reconcile_attempts integer, default 0 queue state plus index (user_id, status) for list_generations. Existing rows keep status: "completed" and nulls elsewhere; nothing is backfilled. Assets: no column. metadata.generation_id on the asset row, written by the seam, is enough for get_generation and for an asset panel to show “cost $0.41, 6 s of Veo” on the clip. nodetool_assets already carries workflow_id, node_id and job_id. Aggregates (Prediction.aggregateDashboard, aggregateByUser, aggregateByProvider, aggregateByModel, listSpendByProject) add status != 'running' so an open row is neither summed nor counted as unpriced, and gain running_count and interrupted_count in AggregateResult. The filter excludes rather than allows because the token writers (chat-turn.ts _logProviderCall, cli/src/supervisor.ts, app-build/build.ts) all write completed today and none of them is touched here. Spend sums include a failed row that carries a cost, because that is money out. 8. Surfaces that must go through the seam Enumerated, not sampled. Each one either moves in this design or is named as staying out with the reason.   Surface Today Change S1 Agent capabilities: generate_image, edit_image, segment_image, generate_video, animate_image, generate_speech, generate_music (packages/agents/src/capabilities/media.ts) runProviderPrediction then persistOutput runGeneration with persist and origin: {surface: "capability", tool_call_id}; drop the capability-side save; return generation_id next to asset_id S2 render_storyboard_stills, render_storyboard_clips, revise_storyboard_clip (capabilities/storyboards.ts), voice_script_lines (capabilities/scripts.ts, which calls generateSpeech.impl) same as S1 same as S1; the shot and the take record the generation id next to the asset id S3 Workflow nodes through runProviderPrediction (image-nodes, video-nodes, audio-nodes, video-nodes/script.ts, llm-nodes/shots.ts) seam, no persist, autoSaveAsset on the websocket host (packages/websocket/src/session/asset-autosave.ts, driven per generation_complete, deduped on metadata.generation_index) unchanged call; origin defaults to the run. The node returns an inline ref, so the seam cannot name the asset. Linking goes the other way: the tracker keeps, per (job_id, node_id), the generation ids it opened and has not linked; autosave asks for them when it saves that node’s generation_complete, stamps metadata.generation_ids on the asset, and the tracker writes asset_ids on those rows. A node that made several generations in one invocation links all of them to the asset rather than guessing which one S4 3D nodes (video-nodes/src/nodes/model3d/generation.ts) direct provider.textTo3D (F6) runGeneration with the two new capabilities S5 Chat turn direct generation (chat-turn.ts handleMediaGenerationMessage) and generate_media RPC (inference.ts runDirectMediaGeneration) direct provider methods, own asset save, own Prediction.create (RPC) ctx.runGeneration with persist and origin: {surface: "chat" \| "rpc", thread_id}; delete both local storeAsset helpers and the RPC’s row write; BYOK gets a row, unpriced when the catalog has no price S6 nodetool generate (packages/cli/src/commands/generate.ts) direct provider methods, node:fs write, hand-rolled recordGenerationSpend build a ProcessingContext (the CLI already does for node run), runGeneration with origin: {surface: "cli"}; -o still writes the file; the row and the asset come from the seam when a database is open S7 Provider node packages that drive an SDK themselves: fal-nodes, kie-nodes, atlascloud-nodes, together-nodes, topaz-nodes, minimax-nodes, replicate-nodes, reve-nodes, huggingface-nodes, elevenlabs-nodes, integration-nodes/kie-dynamic.ts setProviderCost on FAL and kie only, autoSaveAsset for the media, no row for the rest Stays on the node_update path in this design. Two fixes ship with it: kie passes taskId as provider_request_id (F4) and fal-dynamic.ts calls reportFalCost like fal-factory.ts does. Moving these packages behind the seam means giving dispatchCapability an endpoint-shaped request, which is its own design. The audit (§9) lists them as an allowed set so the debt is visible, not forgotten 9. Audit packages/execution/tests/generation-seam-audit.test.ts, modelled on packages/runtime/tests/url-egress-audit.test.ts. It scans packages/*/src for a call to any media method of BaseProvider (textToImage, imageToVideo, textTo3D, … the list is read from base-provider.ts so a new method is covered the day it lands) outside packages/runtime/src/ and outside the S7 allow-list, and fails on the first unclassified call. It also asserts it found the S7 calls, so it cannot pass by matching nothing. It is inverted once in the PR that adds it: a bare provider.textToImage( in a scratch file under packages/agents/src/ must turn it red. 10. Capabilities New module packages/agents/src/capabilities/generations.ts + generations.specs.ts. Wiring follows the checklist every module pays: the spec import, the loader in MODULES, the name in DECLARED_CAPABILITY_MODULES and the entry in MODULE_SPECS of capabilities/registry.ts; the category snapshot in packages/agents/tests/capabilities-registry.test.ts; the generations namespace in the CodeAct API map (codeact/nodetool-api.ts), so guest code gets nodetool.generations.wait(id) next to nodetool.jobs.wait(id); the tRPC classification in packages/websocket/src/trpc/sandbox-coverage.ts for any procedure the dashboard gains to list generations (classified capability, since the module covers it); the module table in docs/AGENTS.md; and a capability-table.ts entry per capability naming packages/agents/tests/capabilities-generations.test.ts and packages/execution/tests/generation-tracker.test.ts. Name Category Input Returns list_generations read status?, provider?, capability?, thread_id?, job_id?, since? (ISO), limit (default 50, max 500) {generations: GenerationSummary[], next}: id, status, provider, model, capability, cost, currency, asset_ids, started_at, duration, error get_generation read generation_id the full row: parameters (redacted), price breakdown, receipt, reconcile state, assets as asset:// refs, origin await_generation read generation_id, timeout_seconds (default 300, max 1800) the row once settled, or its current public state with waited_seconds on timeout. In-process it subscribes to the registry; across processes it polls the row every 5 s cancel_generation write generation_id Local work can abort and close as cancelled. Durable work records a cancellation request; the worker must observe the provider result before closing the row. reconcile_generation external generation_id {before: cost, after: cost, reconciled: boolean, reason?}; runs the provider’s reconciler now, outside the queue’s schedule Every read is scoped: Prediction.find is unscoped today and stays that way for the ledger; the module uses a new Prediction.findForUser(userId, id) and Prediction.listGenerations(userId, filter), both with user_id in the WHERE. 10.1 background: true on the generation capabilities generate_image, edit_image, generate_video, animate_image, generate_speech, generate_music, render_storyboard_stills and render_storyboard_clips accept background: true. The capability calls runGeneration without awaiting the result, reserves the estimated cost against the run’s RunBudget (the same reservation a synchronous call makes), and returns {generation_id, status: "running", estimated_cost} in the same turn. The follower closes the row; await_generation collects it. A background generation an agent never awaits is still finished, recorded, and its asset saved, which is the whole point. Bounds: at most 16 background generations open per run (the number the native flow allows for streams), and none once the budget is exhausted. A run that ends with background generations open leaves them running: they are the follower’s, not the turn’s. 10.2 What is returned to the model Every generation capability adds generation_id to its result next to asset_id, so a model that wants the cost of the image it just made calls get_generation with an id it already holds. persistOutput’s SavedOutput shape is unchanged apart from that field. 11. Harness CLI: nodetool generations list [--status] [--provider] [--since] [--json], get <id>, await <id> [--timeout], cancel <id>, reconcile <id>, plus nodetool generations sweep (run the startup sweep and one queue drain by hand). nodetool costs list gains status and assets columns. Each subcommand takes --json; exit code is the verdict. Selfcheck: packages/execution/tests/generation-tracker.test.ts with the fake provider: running→completed writes one row with the asset id and the message id; failed writes a failed row; an aborted call writes cancelled; a row older than process start is swept to interrupted; a receipt cost wins over the catalog; a reconciler that throws increments attempts and leaves the row queued; five failures leave the queue. packages/agents/tests/capabilities-generations.test.ts drives the five capabilities and background: true through the headless run, and asserts a foreign user’s row is invisible. Registry: surface generation-tracking in packages/cli/src/harness/registry.ts with paths packages/execution/src/generation-tracker.ts, packages/agents/src/capabilities/generations*.ts, packages/runtime/src/generation-receipt.ts, covered by capability-suites; the audit test is named as its selfcheck. Eval: one creative-pipeline case where the objective is “render these four shots and tell me what each cost”, scored on render_storyboard_stills followed by get_generation or list_generations. 12. Out of scope Resumable polling for providers other than FAL. The durable recovery worker can bind and poll FAL attempts after a restart. Other providers still need provider-specific submit and poll adapters. The row records their lifecycle state, but this document does not claim that their media can be recovered. Moving S7 behind the seam (§8). Budget refusal before the call. admitSpend exists for the nodetool provider only; extending the pre-run gate to BYOK is a pricing question, not a bookkeeping one. UI. The Costs dashboard and an asset panel badge render the row; neither is designed here. 13. Rollout Phase Ships Gate P1 Schema, runGeneration + receipts + persist, tracker with queue and sweep, S1/S2 capabilities on the seam, generations module, CLI, audit (with S4/S5/S6 in the allow-list, marked TODO(P2)) tracker and capability suites; audit inverted; capabilities:check; harness gate P2 S4, S5, S6 on the seam; S7 fixes (kie request id, fal-dynamic cost); allow-list shrinks to S7 audit shrinks and stays green; generate_media route tests; CLI generate writes a row through the seam P3 background: true, await_generation, cancel_generation wired to the registry; aggregate changes and dashboard columns capability suite for background; costs route tests P1 is the bookkeeping the title asks for. P2 closes the bypasses. P3 is the ergonomics. 14. Risks R1. A row for every call doubles the write rate on the predictions table. Two writes per generation instead of one, on a table indexed five ways. A generation is seconds to minutes of provider time; two SQLite writes are not the bottleneck. Watched, not mitigated. R2. The sweep marks a slow generation interrupted after a restart while the provider finishes it. Correct: the process that would have collected the bytes is gone. The row says so, keeps the request id, and reconciles the cost. The media is lost until §12’s resumable polling. R3. A provider reports a request id but no reconciler exists. The row leaves the queue on the first pass as unavailable; the estimate stands. No retry storm. R4. Receipt scope leaks across concurrent generations. AsyncLocalStorage isolates per async context; a provider that schedules work outside the awaited chain (a detached timer) would record into nothing. The receipt call logs at debug when it finds no scope, and the tracker test runs two generations concurrently and asserts each row gets its own id. R5. Dropping bytes from prediction.data breaks a consumer. Enumerated: web/src/stores/workflowUpdates.ts, web/src/core/chat/chatProtocol.ts, web/src/hooks/timeline/useGenerateClip.ts, web/src/hooks/sketch/useGenerateLayer.ts, packages/websocket/src/chat-prediction-forwarder.ts. None reads data. Mobile is checked in the PR the same way. R6. background: true spends money a turn never reports. The reservation against RunBudget happens at submit, so the cap holds; the turn’s cost summary counts reserved spend as spent and says n generations still running.--- # Memory Source: https://docs.nodetool.ai/memory Memory is a store of short notes that an agent writes for itself and reads back in later conversations. An agent saves a project fact, a decision you approved, or a reference to an asset it generated. The next time you chat, it can find that note again without you repeating yourself. Quick Access: Open the app menu in the sidebar and choose Memory to see everything the agent has saved. What a memory holds Each memory is one row with these fields. Field What it holds Kind note, fact, preference, decision, or resource. An unknown value is saved as note. Title An optional short label Content The note itself, required Resources Optional typed references to things the note is about Thread The conversation the memory was saved in. Empty when it was saved outside a chat. Created and updated times Set by the store A resource reference has a type and an id, plus an optional uri and label. Known types are asset, workflow, collection, node, job, timeline, script, storyboard, image_document, thread, url, and other. Any other type string is also accepted. Asset references are checked against your asset library. NodeTool fills in the asset:// URI and the asset name, and drops any reference to an asset it cannot find. The tool result reports the dropped references. Other types are stored as given. How agents save and recall Agents use five tools. They are always part of the built-in tool set, so you do not turn them on. Tool What it does memory_save Saves a new memory. Takes content (required), title, kind, and resources. Returns the new memory_id. memory_list Lists memories, newest first. Takes limit (default 100, maximum 200), thread, and kinds. memory_search Finds memories by keyword. A memory matches only when every word in query appears in its title or content, ignoring case. Takes limit (default 25, maximum 200), thread, and kinds. memory_update Changes a memory by memory_id. Only the fields passed change. Passing resources replaces the whole list. memory_delete Deletes a memory by memory_id. For memory_list and memory_search, thread is either all (the default) or current. all reads every memory in your account. current narrows the result to the conversation the agent is running in. Each result item includes from_current_thread, so the agent can tell where a note came from. Every tool needs a signed-in user. Without one the call returns “No user context; cannot access memory.” What the agent sees at the start of a turn At the start of each chat turn, NodeTool adds a <memory> block to the message. It contains the memories saved in the current conversation, up to 100, with their resource references. For memories saved in other conversations, it adds only a count and a hint to use memory_search or memory_list. The block is built from your 400 newest memories, so an older memory may not be counted. The agent reaches older or cross-conversation notes by calling the tools. The block is marked as user data. The agent is told to use it as reference and not to follow instructions written inside a note. The block is added for chat turns that belong to a thread. A memory saved by a run with no thread has an empty thread field, so it does not appear in any conversation’s block. The agent can still find it with memory_list or memory_search. The Memory page Open the app menu in the sidebar and choose Memory. The page opens as a tab titled Memory. List. Each memory is a card with its kind, title, creation time, content, and resources. Image assets show as thumbnails. Other resources show as chips such as workflow: <id>. The page loads up to 200 memories. Search. The Search memories box runs the same keyword match as memory_search. Every word you type must appear in the memory. Filter. When the results hold more than one kind, chips appear above the list. Select a kind to narrow the list, or All to reset. Open the source conversation. Hover over a card and click the conversation icon (Open the conversation this came from). Delete. Hover over a card, click the delete icon, and confirm in the Delete memory dialog. Deleting cannot be undone. The page does not edit memories. To change a note, ask the agent to update it, and it calls memory_update. An empty page reads “Nothing remembered yet.” Memories appear as agents save them while they work. Scope and storage Memory belongs to your user account, not to a conversation. A memory saved in one chat is readable from every other chat. Other accounts never see it. The thread field records where a note came from and works as a filter only. Memories are rows in the NodeTool database. Deleting a chat thread also deletes the memories that were saved in it. Memories with no thread are not affected. How it differs from run memory NodeTool has a second, unrelated store also called memory. Agent Memory System describes context.memory, a scratch space shared by the steps of one workflow run through the list_shared, read_shared, and share_result tools. It is cleared when the run ends and is never shown on the Memory page.   Durable memory (this page) Run memory Lifetime Until deleted One run Scope Your account, all conversations One run and its steps Tools memory_save, memory_list, memory_search, memory_update, memory_delete list_shared, read_shared, share_result Visible to you Memory page Not shown Related Agent Memory System Chat & Agents Skills--- # Design: Mini-App Build Harness Source: https://docs.nodetool.ai/mini-app-build-harness-design.html Design: Mini-App Build Harness Author: Matti Georgi Status: Draft — for review Last updated: 2026-08-02 PRD: mini-app-build-harness-prd.md Implementation plan: mini-app-build-harness-implementation-plan.md How nodetool app build turns a prompt into a verified ApplicationBundle. The PRD says what and why; this says where the code goes, what each stage’s contract is, and which existing modules do the work. 1. Placement The loop needs three things at once: authorGraph (@nodetool-ai/agents), the app document ops (@nodetool-ai/app-runtime), and the headless app simulator (today packages/cli/src/app-debug/). The dependency graph already almost allows it: app-runtime ──┐ execution ────┼──▶ agents ──▶ { cli, websocket } kernel ───────┘ agents depends on app-runtime and execution today. cli and websocket both depend on agents today. One move makes it work: the app-debug simulation core relocates from packages/cli/src/app-debug/ to packages/execution/src/app-debug/. This is the same move debug_workflow already forced for the workflow harness — its summary reducer and triage live in @nodetool-ai/execution/debug precisely so CLI and server surfaces cannot drift. execution has every dependency the simulator needs (kernel, node-sdk, protocol, runtime), and the harness is already decoupled from CLI concerns: AppDebugDeps injects loadFromDb, loadApplication, and runOnServer, so the move carries no DB or filesystem code with it. The CLI keeps target resolution (files, DB ids), bundle writing, and the command itself, re-exporting the moved types. New code: packages/agents/src/app-build/ types.ts # BuildSpec, BuildReport, BuildComplaint, stage records spec.ts # Spec stage (StepExecutor + schema) bridge.ts # ui_app_* headless bridge, promoted from evals/surfaces/app.ts author.ts # Author stage (tool loop over the bridge) interactions.ts # Interaction-script derivation from the spec judge.ts # Judge stage (goal-judge pattern) build.ts # Orchestrator: stages, repair loop, budgets packages/execution/src/app-debug/ # relocated simulator (app-spec, runtime, harness) packages/cli/src/commands/app.ts # + `app build` subcommand packages/websocket/src/... # HTTP surface The evals keep working against the same surfaces: evals/surfaces/app.ts shrinks to a re-export of app-build/bridge.ts plus its cases, so the eval and the production author stage cannot diverge — the drift-prevention rule that file already documents, now enforced by sharing the implementation, not just the contract. 2. Types All in packages/agents/src/app-build/types.ts; the report shape also lands in @nodetool-ai/protocol if the web surface consumes it (M4), mirroring how Intervention moved. /** What the Spec stage produces and every later stage consumes. */ interface BuildSpec { title: string; operations: Array<{ id: string; // stable slug, e.g. "draft" objective: string; // authorGraph objective, or — workflowId?: string; // — bind an existing workflow instead inputs: Array<{ name: string; type: string; example: unknown }>; outputs: Array<{ name: string; type: string }>; streaming: boolean; }>; variables: Array<{ id: string; scope: "app" | "instance"; persist: boolean; writtenBy: string; readBy: string[] }>; widgets: Array<{ role: string; type: string; binding: string; label: string; container?: string; visibleWhen?: string }>; // condition source, verbatim interactions: Array<{ // the behavioral contract name: string; // "draft-then-approve" steps: InteractionStep[]; // the app-debug script language expect: Array<{ widget: string; check: "nonEmpty" | "equals" | "matches"; value?: unknown }>; }>; } /** One repair round's input: everything wrong, not a delta. */ interface BuildComplaint { round: number; issues: BuildIssue[]; // stage, severity, message, widget/op ref fingerprints: string[]; // stable ids for oscillation detection } interface BuildReport { target: { prompt: string; specPath?: string }; spec: BuildSpec; interactions: CompletedInteraction[]; // what Run replayed: authored steps // plus seeded inputs and derived runs, // each flagged (§3.5) stages: StageRecord[]; // one per stage execution, repairs included repairs: BuildComplaint[]; appDebug: AppDebugReport | null; // the final Check+Run evidence, verbatim judge: JudgeRecord | null; verdict: { ok: boolean; reason: string; notSimulated: string[] }; cost: { usd: number; byStage: Record<string, number> }; bundle: ApplicationBundle | null; // null only when verdict.ok is false } InteractionStep is the existing app-debug script language (set, click, change, run, cancel) — the spec’s behavioral contract is written in the oracle’s own vocabulary, so Run needs no translation layer. 3. Stages 3.1 Spec One StepExecutor call with BuildSpec as the outputSchema. The prompt carries the widget catalog (WIDGET_CATALOG names, binding modes, fields), the operation policy vocabulary, and the medium-complexity shapes from the guide. Host-side validation beyond the schema, before any planning token is spent: every widget type exists in WIDGET_CATALOG; its binding parses (parseBinding) and names a declared operation input/output or variable; every variable’s writtenBy/readBy name declared operations or widgets; every interaction’s steps reference declared widgets/operations, and every expect names a display widget; at least one interaction per operation. Failures bounce back into the open StepExecutor conversation as tool errors (the supervisor’s finish_step repair loop, reused as-is, MAX_REPAIR_ROUNDS = 3). A prompt that cannot be pinned — no operations derivable, or the model asks for widgets the catalog lacks — resolves as failed at this stage with the validation record as the reason. That is the PRD’s “refuse rather than guess.” app build spec.json enters here: parse, validate with the same code, skip the LLM. 3.2 Plan Per spec operation, in declaration order: workflowId set → load the workflow, validateGraph, extract IO (extractAppIO), and check the spec’s declared inputs/outputs against it. A mismatch is a spec-stage complaint, not a planning job. otherwise → authorGraph(objective, {...}) with inputs seeded from the spec’s examples and outputSchema derived from the declared outputs. The authoring sub-agent already checks its own graph with validate_workflow; the harness adds validateGraph + a check that the planned graph’s Input/Output node names cover the spec’s declared surface — the app cannot bind to an input the planner renamed. Missing names → one replan with the delta named in the objective; still missing → failed. Planned graphs are held in memory as bundle entries — nothing writes to the database. That is what makes the loop runnable in CI and what makes the output an ApplicationBundle rather than a pile of rows. 3.3 Author The tool loop from tool-loop-eval (provider generateLoop over HeadlessTool[]), promoted to production: the model drives the real ui_app_* contract against the bridge, starting from an empty document plus the bundle’s workflows as BindableWorkflows. The system prompt is the spec, rendered: operations to declare (with exact input/output mappings), variables, the widget list with bindings, the container structure. Termination: the model calls ui_app_finish (new bridge-only tool) or the turn cap (default 40 tool calls) trips. The bridge’s finalState() yields the Puck document + AppDocMeta, assembled into the draft ApplicationBundle. On repair rounds the loop resumes with the current document loaded and the BuildComplaint rendered as the opening message — the model edits, it does not rebuild. Fingerprints that reappear after being absent end the loop (§5). 3.4 Check validateApp from the relocated simulator, against the in-memory bundle (app debug’s existing ApplicationBundle target path, no DB). Errors → complaint; warnings ride along in the report but do not block. Then validateGraph per workflow again — authoring cannot change graphs today, but the gate is cheap and the invariant (“no LLM judge ever sees a statically invalid app”) should not depend on that staying true. 3.5 Run For each spec interaction, one simulator pass (runAppDebug equivalent on the in-memory bundle) with that interaction’s steps, then the expect checks against the folded final state: nonEmpty — the widget’s state key received a value from a completed run; equals / matches — value comparison after formatTemplate is applied, so the check sees what the user would see. Failed expectations and simulator verdict issues both become complaints. The existing per-operation policy/timeout machinery applies unchanged; a spec interaction may exercise cancel and replacement policies like a hand-written script does today. 3.6 Judge The goal-judge pattern from graph-e2e: one structured-output LLM call per interaction, given the spec’s plain-language intent, the interaction’s steps, and the final widget states (previewed with the same previewValue truncation the debug collector uses). Output: { achieved, confidence, reasons }. achieved: false → complaint carrying the reasons. The judge runs only on a Check+Run-green app, is configured independently (--judge-model, default: a different model than the builder), and is skipped for deterministic eval cases. Judge failures (timeout, unparseable) count as not-achieved — fail closed, same rule as everywhere else. 3.7 Repair The orchestrator (build.ts) is a plain loop, not an agent: spec → plan → author ─▶ check ─▶ run ─▶ judge ─▶ ok? ─▶ bundle + report ▲ │ │ │ └──────────┴────────┴───────┘ complaints, rounds ≤ maxRepairs Complaints route to Author by default. A complaint that names a missing graph input/output (Plan’s responsibility) routes to Plan for that one operation, then falls through Author again. Spec is never revisited after it validates — a bad spec fails, it doesn’t mutate, so the target the judge scores against is fixed for the whole build. Oscillation guard: every issue gets a stable fingerprint (stage + code + widget/op ref). A fingerprint absent in round N that reappears in round N+1 ends the build as failed with both rounds cited. Budgets: maxRepairs (default 3), wall clock (default 10 min), and USD. Dollar enforcement reuses TurnBudget from the runtime provider layer (built for the supervisor): every stage’s provider call reserves worst-case before the turn. Exhaustion mid-stage resolves that stage as failed and the build as failed — reported, never silent. 4. Oracle closures 4.1 Conditions and format (R6) @nodetool-ai/app-runtime/conditions.ts already exports parseCondition, evaluateCondition, readRef, and formatTemplate — the web runtime’s implementations. The gap is that the headless runtime never calls them. Fix, in the (relocated) simulator: After every fold step, evaluate each widget’s visibleWhen/disabledWhen against current state and record it in the widget’s simulated state. A click/change/set-via-widget step on a currently hidden or disabled widget is a run failure (“clicked Approve while hidden by draft == null”), because no user could perform it. Display expectations evaluate against the format-rendered value. The report’s notSimulated list drops these entries and keeps only what remains genuinely browser-only (layout, focus, CSS). The verdict gains one check: a widget whose condition never becomes satisfiable across all executed interactions (always-hidden trigger) is an error — today’s “app with no run trigger” check, generalized. 4.2 Resources (R7) The headless runtime’s resourceCommand branch currently no-ops (“headless runs have no resource providers attached”). Add: interface HeadlessResourceProvider { kind: ResourceKind; list(): ResourceItem[]; get(id: string): ResourceItem | null; apply(op: ResourceOperation): void; // the doc-ops mutation verbs } with one in-memory implementation, seedable from the interaction script ({ seedResource: { id, items } } step) or from --params. from: "resource" params resolve through it; resource widgets (picker, gallery, scene list) report their collection state like bound widgets report values. The web runtime is untouched — this provider exists only under the simulator. 5. Surfaces 5.1 CLI packages/cli/src/commands/app.ts gains build: nodetool app build "<prompt>" | spec.json -p/--provider, -m/--model builder provider (registry ids, as everywhere) --judge-model <provider/model> default: NODETOOL_APP_JUDGE_MODEL or a different model than the builder --workflow <id> (repeatable) pin existing workflows; Plan binds, never plans --max-repairs <n> --cost-cap <usd> --timeout <ms> --out <dir> --json --no-judge Bundle layout extends the app-debug convention: nodetool-debug/app-build-<slug>-<ts>/ with report.json (BuildReport), report.md, app.bundle.json (the deliverable), spec.json, and one interactions/<name>/ per interaction holding that pass’s run-N.messages.jsonl. Exit code 0 iff verdict.ok. 5.2 Server HTTP (M4) POST /api/applications/build {prompt | spec, options} runs the same buildApp() from @nodetool-ai/agents. Long builds use the existing debug-session machinery (packages/execution/src/service/debug-sessions.ts) for polling and cancel, and the bundle behind a green verdict is imported through the normal bundle-import path. There is no matching agent tool. A build_app capability shipped and was removed: it put a second, invisible agent between the user and their app, and the editor agent can already do every stage itself with the ui_app_* tools and debug_app. The route stays for the CLI and the eval suite. 5.3 Eval suite One EVAL_SUITES entry (eval.ts — suites are data): id app-build. A case is a prompt (or spec, for the deterministic ones), a target-shape checklist (operations ≥ 2, a persisted variable, a working conditional, …), and for deterministic cases exact expected widget values with the judge skipped. Metrics: one-shot rate (green with zero repairs), green-within-budget rate, repair rounds, cost, duration; --min-success gates on green-within-budget. Two deterministic cases (template-only workflows, no model calls in the product under test) run in CI on every PR; the full suite runs nightly. 6. Telemetry and cost attribution app.build (orchestrator) app.build.spec (StepExecutor → llm.chat) agent.plan (authorGraph, existing span, per operation) app.build.author (tool loop → llm.chat per turn) app.build.check app.build.run (→ workflow.run per interaction, existing) app.build.judge (→ llm.chat) Every provider call carries the existing gen_ai.usage.* attributes; ledger rows are attributed to the build id and tagged app-build in node_type, so nodetool costs answers “what did generated apps cost this month” the same way it answers it for the supervisor. 7. Failure modes Every row resolves as a reported failed build — no partial success, no silent bundle. Failure Resolution Spec unpinnable / fails validation after 3 bounces failed, validation record as reason Planner returns null / planned IO can’t cover spec after 1 replan failed, per-operation planning record Author turn cap trips without ui_app_finish current document goes to Check anyway; its complaints count as the round’s result Oscillation (fixed issue reappears) failed, both rounds cited Any budget exhausted (rounds, wall clock, USD) failed, best attempt’s report attached, budget named Judge timeout / unparseable interaction counts as not-achieved Kernel run crash during Run the simulator’s existing error surface → complaint; a second identical crash in the next round is oscillation Cancel (signal) aborts the in-flight provider call and kernel run; failed, reason: "cancelled"--- # Implementation Plan: Mini-App Build Harness Source: https://docs.nodetool.ai/mini-app-build-harness-implementation-plan.html Implementation Plan: Mini-App Build Harness Author: Matti Georgi Status: Draft — for review Last updated: 2026-08-02 PRD: mini-app-build-harness-prd.md Design: mini-app-build-harness-design.md Written to be executed by an implementing agent (Opus) PR by PR. Each PR names its files, the invariants it must hold, and the tests that prove it. PRs marked ∥ can proceed in parallel once their listed dependency lands. 0. Ground rules Every PR passes npm run check before commit. packages/cli, packages/execution, packages/agents, and packages/websocket are plain TS packages — no build:packages needed for their own tests, but run it before any end-to-end nodetool app debug smoke since the CLI loads decorator packages from dist/. Read before writing. The modules this plan touches are documented in-file: packages/cli/src/app-debug/harness.ts (orchestration), packages/agents/src/evals/surfaces/app.ts (bridge + drift rules), packages/agents/src/evals/tool-loop-eval.ts (loop driver), packages/cli/src/commands/eval.ts (“suites are data”). Honor the invariants their header comments state. The relocation PR (PR 3) is behavior-preserving and must be provably so: nodetool app debug output on the shipped example bundles is byte-identical before and after (modulo timestamps/paths). BuildSpec/BuildReport shapes change only via the design doc, in the same PR. No web UI work in this plan. The web surface (if any) consumes BuildReport after M4; it does not define it. Prose in reports and docs follows the repo style rules in docs/WRITING_STYLE.md. Phase M1 — close the oracle (independently shippable) PR 1 — Simulate conditions and format (R6) All in packages/cli/src/app-debug/ (pre-relocation — keep the diff where the code lives today; PR 3 moves it wholesale). runtime.ts: after every fold into app state, evaluate each widget’s visibleWhen/disabledWhen via parseCondition/evaluateCondition from @nodetool-ai/app-runtime (already exported — wire, don’t reimplement). Track per-widget { visible, disabled } in the runtime’s widget state. harness.ts: an interaction step targeting a hidden or disabled widget fails that step with a message naming the condition source (clicked "Approve" while hidden by \draft == null`). set` steps that address state keys directly (not widgets) are exempt — they model the runtime, not a user. Display checks and report.md previews render through formatTemplate before comparison. types.ts: widget final-state record gains visible/disabled; the report’s notSimulated (add the field if absent) no longer lists conditions/format. Verdict: new issue for a run-trigger widget whose visibleWhen never evaluated true in any executed interaction. Tests (packages/cli/tests/ next to the existing app-debug tests): a conditional Approve button — hidden click fails, satisfied click runs; a format template changes an equals check’s outcome; the never-visible trigger verdict fires; an app with no conditions produces an unchanged report. PR 2 — Headless resource provider (R7) ∥ (with PR 1) runtime.ts: HeadlessResourceProvider interface + InMemoryResourceProvider (design §4.2); the resourceCommand branch dispatches to it instead of no-opping. harness.ts + types.ts: seedResource interaction step; from: "resource" params resolve through the provider; resource widgets report collection state in the final report. Update the “Not simulated headlessly” paragraph in CLAUDE.md § nodetool app debug and mini-apps.md “Checking an app” — both currently list resources as unreachable. Tests: a picker-fed param runs with seeded items and fails with an actionable message when unseeded; a resource operation mutates the collection and the report shows it. PR 3 — Relocate the simulator to @nodetool-ai/execution (depends: PR 1, PR 2) Move app-spec.ts, runtime.ts, types.ts, and the simulation half of harness.ts to packages/execution/src/app-debug/; export from the package index the way execution/debug already does. markdown.ts moves too (it renders the report, needs nothing from cli). Stays in packages/cli/src/app-debug/: app-target.ts (DB/file resolution), bundle writing, the command wiring. It re-exports the moved types so @nodetool-ai/cli importers don’t break. The harness’s runOnServer / loadFromDb / loadApplication injections already isolate it from cli’s debug internals — the move must not grow new parameters. Where harness.ts imports ../debug/collector.js (previewValue) and ../debug/types.js, move those types and the small preview helper into execution as well or inject them; do not drag the whole cli debug module along. Proof of behavior preservation: script a before/after app debug --json run over every bundle in packages/base-nodes/nodetool/examples/apps/ and diff (normalize paths/timestamps). Keep the script in scripts/ for PR review, delete before merge or park under the eval fixtures. Phase M2 — the loop PR 4 — Promote the bridge, extract the tool-loop driver (depends: PR 3) packages/agents/src/app-build/bridge.ts: move the bridge implementation out of evals/surfaces/app.ts verbatim — tool contract, catalog use, and the doc-meta passthrough to @nodetool-ai/app-runtime doc-ops are already correct there. Add ui_app_finish (bridge-only termination tool) and a loadDocument entry so repair rounds resume from the current document. evals/surfaces/app.ts becomes: re-export of the bridge + the eval cases. The file’s header comment about drift updates to describe the new sharing. packages/agents/src/app-build/tool-loop.ts: extract the generic provider-driving loop from tool-loop-eval.ts (generateLoop over HeadlessTool[], turn cap, transcript capture) into a function both the eval and the author stage call. The eval keeps its scoring; the loop moves. Tests: existing app-tools eval passes unchanged (that suite is the regression test for this PR); a unit test drives the bridge through loadDocument → edit → finalState round-trip. PR 5 — Types, spec stage, interaction derivation ∥ (with PR 4, depends: PR 3) packages/agents/src/app-build/types.ts: BuildSpec, BuildIssue, BuildComplaint, StageRecord, BuildReport per design §2. spec.ts: one StepExecutor call with BuildSpec as outputSchema; the host-side validation list from design §3.1, implemented against WIDGET_CATALOG and parseBinding — reuse the supervisor’s finish_step bounce loop (it is a general StepExecutor feature since supervisor PR 3). specFromFile parses + validates without the LLM. interactions.ts: spec interactions are already InteractionStep[]; this module only completes them — prepend set steps for declared inputs that no step touches (using spec examples), and derive a minimal default interaction (the existing defaultInteractions shape) for any operation the spec forgot, flagging it as derived in the report. Tests: schema-valid-but-catalog-invalid specs bounce with the right issue; an unpinnable spec fails after 3 bounces; derivation covers a forgotten operation and never overwrites an authored step. PR 6 — Orchestrator, repair loop, app build CLI (depends: PR 4, PR 5) The core PR. packages/agents/src/app-build/build.ts + the CLI command. build.ts — buildApp(opts): Promise<BuildReport>: the stage sequence from design §3.7. Plan wraps GraphPlanner per operation (existing-workflow binding first, IO-coverage check, one replan). Author runs the tool loop from PR 4 with the spec rendered as system prompt. Check/Run call the execution simulator on the in-memory ApplicationBundle. Judge is a stub returning achieved: true this PR (lands in PR 7); the verdict says so in notSimulated. Repair: complaints as full issue sets, fingerprints, oscillation guard, routing (Author default, Plan for graph-IO issues) — design §3.7 verbatim. Budgets: maxRepairs, wall-clock, and USD via TurnBudget reservation on every provider call (spec, plan, author turns). Exhaustion → failed with the budget named. AbortSignal threads into every stage. Ledger: stage provider calls attributed to the build id, node_type: "app-build". Spans: app.build + per-stage children (design §6), via the existing agent span helpers. packages/cli/src/commands/app.ts: the build subcommand, flags and bundle layout per design §5.1. Exit 0 iff verdict.ok. Tests (packages/agents/tests/): a scripted provider (the eval harness’s mock pattern) drives a full green build over a template workflow with zero real model calls; a complaint round edits rather than rebuilds (assert the document diff is scoped); oscillation ends the build; each budget exhaustion fails closed; cancel mid-author aborts cleanly. CLI test: app build spec.json --json on a deterministic spec emits a parseable BuildReport and writes the bundle layout. Phase M3 — the score PR 7 — Judge stage (R8) (depends: PR 6) packages/agents/src/app-build/judge.ts: per-interaction structured-output call following evals/goal-judge.ts; inputs and fail-closed rules per design §3.6. --judge-model flag + NODETOOL_APP_JUDGE_MODEL; default resolution picks a model ≠ builder when possible and records which ran. Wire into build.ts behind --no-judge; judge complaints route to Author. Tests: scripted judge verdicts flip the build outcome; judge timeout counts as not-achieved; deterministic path never invokes it. PR 8 — app-build eval suite (R5) (depends: PR 6; judge cases depend: PR 7) Cases in packages/agents/src/evals/app-build-cases.ts: ≥ 8 prompts meeting the PRD §4 bar (each case declares which medium-complexity traits it exercises, and the set covers all of them), plus 2 deterministic spec-file cases over template graphs with exact expected widget values, judge skipped. Runner in evals/app-build-eval.ts; one data entry in EVAL_SUITES (packages/cli/src/commands/eval.ts) — follow the file’s “suites are data” rule, no new command block. Metrics and --min-success semantics per design §5.3. CI: deterministic cases join the Quality Gate; full suite gets a nightly workflow following the GenSpend sync workflow’s shape (report artifact, no auto-merge decisions). Tests: the suite runs end-to-end on the deterministic cases in CI with no provider keys. Phase M4 — surfaces PR 9 — build_app server tool + HTTP (R9) (depends: PR 6; better after PR 7) Register build_app next to debug_workflow; POST /api/applications/build running buildApp(); long-build polling and cancel through packages/execution/src/service/debug-sessions.ts. Auth and cost-cap defaults match debug_workflow’s posture. The tool result is the BuildReport; the bundle imports through the existing bundle-import endpoint, never auto-installs. Tests: route test with a scripted provider; cancel over HTTP aborts the build; the tool is listed for editor chat. PR 10 — Supervisor composition (R10) ∥ (with PR 9, depends: PR 6) --supervise + supervisor flags on app build configure ExecutionSessionOptions.supervisor for the Run stage’s kernel executions — the same single integration point every surface uses. Interventions land in the interaction’s run report and roll up into BuildReport. Test: a scripted supervisor skip during Run surfaces in the build report and does not count as a build failure when expectations still pass. PR 11 — P2 conveniences (R11, R12) (depends: PR 8) --watch on spec-file targets: re-run on save, print a verdict diff — reuse the debug --watch differ. Example-apps regeneration: scripts/build-example-apps.mjs gains a mode that runs buildApp from each curated spec and diffs against the shipped bundle, reporting drift without overwriting — the curated bundles stay hand-approved. Sequencing summary M1: PR1 ∥ PR2 → PR3 M2: PR4 ∥ PR5 → PR6 M3: PR7 → PR8 (deterministic cases of PR8 can start after PR6) M4: PR9 ∥ PR10 → PR11 The PRD’s launch gate (§8) is measurable after PR 8; nothing in M4 blocks it.--- # PRD: Mini-App Build Harness Source: https://docs.nodetool.ai/mini-app-build-harness-prd.html PRD: Mini-App Build Harness Author: Matti Georgi Status: Draft — for review Last updated: 2026-08-02 Design doc: mini-app-build-harness-design.md Implementation plan: mini-app-build-harness-implementation-plan.md Related: mini-apps.md · app-builder.md · nodetool app debug (AGENTS.md) · packages/agents app-tools eval 1. Summary Today an agent can check a mini app (nodetool app debug) and can edit one (the ui_app_* tools), but nothing closes the loop. There is no command that takes “build me a product-photo captioner with an approve step” and returns a working, verified app — workflows included. The Build Harness is that command: nodetool app build "Turn a product photo into three caption options; \ let me pick one and post-process it" -o captioner.app.json Plan the workflows, author the app document, validate the wiring, run every operation headlessly, judge the result against the intent, repair what failed, repeat until green or out of budget. Output: an ApplicationBundle that imports and runs anywhere, plus the debug bundle proving it works. The bet is simple: the model is good enough to one-shot a medium-complex app if the harness gives it the same feedback loop a human gets from the editor. Every piece of that loop already exists in the repo. This PRD wires them into one machine and puts a number on it. North-star metric: one-shot rate on the app-build eval suite — the share of medium-complexity prompts that produce a green app with zero human edits. 2. Problem 2.1 The loop is open The build story for an agent today: Plan a workflow — GraphPlanner exists, evaled (graph-planner, graph-e2e). Author the app document — ui_app_* tools exist, evaled (app-tools), and run headlessly against @nodetool-ai/app-runtime doc-ops. Verify — nodetool app debug validates bindings, simulates the runtime, runs the kernel, and emits an AppDebugReport with a verdict. Each stage is solid in isolation. But nothing drives them in sequence, feeds the verdict back into the tools, or decides when to stop. An agent in editor chat improvises this loop turn by turn; an agent in CI or the CLI cannot run it at all. The result: app-building quality is whatever the conversation happens to produce, and we cannot measure or regress-test it. 2.2 Verification has blind spots app debug cannot see everything the web runtime does, so a “green” app can still be broken on screen: visibleWhen / disabledWhen / format are not simulated — a widget hidden by a wrong condition reports as fine. from: "resource" params have no headless provider — any app that reads a picker cannot be exercised. Interaction scripts are hand-written JSON. Nobody writes them, so the approve-then-continue and slider-rerun shapes — the ones the guide teaches — go untested. A build loop is only as good as its oracle. These gaps are the oracle’s bugs. 2.3 There is no score graph-planner reports a one-shot rate for graphs. No suite reports one for apps. Without a number, model upgrades, prompt changes, and tool-contract edits ship blind — we find out an app shape regressed when a user hits it. 2.4 Why now The example-apps pipeline (scripts/build-example-apps.mjs) hand-curates bundles because generation isn’t trusted. Mobile ships app screens. The marketing site generates /apps/* landing pages. Every one of these consumes finished apps; none can produce them. App supply is the bottleneck, and the supply chain should be a prompt. 3. Users Agents first. The harness is an agent-facing product; humans read its output, they don’t operate it. User Surface What they need Editor chat agent ui_app_* + a build_app tool One tool call that returns a working draft to refine with the user CLI / CI agent (Claude Code, cron) nodetool app build Prompt in, bundle + verdict out, exit code that means something Eval runner nodetool eval app-build The one-shot number, per case, gated with --min-success Human builder App Builder Imports the generated bundle and tweaks in Design view — never debugs wiring by hand 4. What “medium complex” means The target is pinned so “one-shot” can’t be gamed by easy prompts. A medium-complex app has, per the eval suite’s case definitions: 2+ operations over 2+ workflows (e.g. draft → approve → publish), 8–15 widgets, at least one layout container deep, variables, including one persist: true app-scoped setting, one streaming output folded into a display widget, one gated flow — a second operation reading a variable the first wrote, one conditional (visibleWhen/disabledWhen) that must actually gate. Roughly: the “review before publish” and “settings that stick” shapes from the guide, combined. Below this is a form; above it is a product. 5. Product 5.1 The loop prompt ──▶ SPEC ──▶ PLAN ──▶ AUTHOR ──▶ CHECK ──▶ RUN ──▶ JUDGE ──▶ green? ──▶ bundle ▲ │ └──────────── REPAIR (bounded) ◀─────────┘ Stage What runs Exists today Spec LLM turns the prompt into an app spec: operations, widget list, variables, interaction expectations New Plan GraphPlanner authors each workflow the spec names (or binds an existing one by id) packages/agents GraphPlanner Author LLM drives ui_app_* against the headless bridge to build the document evals/surfaces/app.ts bridge, promoted out of evals Check Static wiring pass — app debug --no-run + validate per workflow packages/cli/src/app-debug/ Run Every operation, headlessly, with an interaction script derived from the spec (fill inputs, click run triggers, write the gate variable, click the second trigger) app debug runtime + new script derivation Judge LLM judge scores final widget states against the spec’s expectations, same pattern as graph-e2e’s goal judge evals/goal-judge.ts pattern Repair Failures (validation issues, run errors, judge misses) go back to the author stage as a diff-shaped complaint; bounded rounds New Two hard rules, borrowed from the supervisor PRD’s spine: Deterministic before generative. Check runs before Run, Run before Judge; an LLM judge never sees an app the static validator rejected. The cheap oracle always fires first. Fail closed on budget. Repair rounds (default 3), wall clock, and USD cost are capped. Out of budget → the harness returns the best attempt marked failed, with the full report of why. It never returns a silent half-working bundle as success. 5.2 Surfaces nodetool app build "<prompt>" # bundle + report to nodetool-debug/ nodetool app build "<prompt>" --workflow <id> # bind existing workflow(s), plan none nodetool app build "<prompt>" --json # full BuildReport for agents nodetool app build "<prompt>" --max-repairs 5 --cost-cap 1.00 nodetool app build spec.json # skip the Spec stage, build from a spec file build_app tool on the server (like debug_workflow): editor chat and MCP agents call the same loop, get the same BuildReport. nodetool eval app-build: the suite from §4, standard eval flags, --min-success as a CI gate. Two deterministic cases run without providers. 5.3 The report BuildReport extends the existing AppDebugReport: per-stage outcome, every repair round with what was complained about and what changed, per-stage token and dollar cost, and the final verdict. report.md reads top-down as a story: what was built, what broke, what fixed it, what it cost. Spans follow the existing OTel hierarchy (app.build → agent.plan / app.author / app.check / app.run / app.judge). 6. Requirements # Requirement Priority R1 nodetool app build runs the full loop from a prompt and emits an ApplicationBundle + BuildReport P0 R2 Author stage drives the real ui_app_* contract headlessly — no forked tool surface P0 R3 Interaction scripts derived from the spec; every operation and every declared event path exercised P0 R4 Repair loop consumes verdict issues as structured complaints; bounded by rounds, wall clock, and cost; fails closed P0 R5 app-build eval suite with ≥8 medium-complex cases and --min-success gating P0 R6 Headless runtime simulates visibleWhen/disabledWhen/format; a click on a hidden/disabled widget is a run failure P0 R7 Headless resource provider: seedable in-memory collections so from: "resource" params run P1 R8 LLM judge stage scoring widget end-states against spec expectations P1 R9 build_app server tool + HTTP surface P1 R10 --supervise composes: the run stage can put the supervisor on workflow failures P1 R11 Example-apps pipeline can regenerate a curated bundle via app build spec.json and diff against the shipped one P2 R12 --watch on a spec file: edit spec, re-run loop, print verdict diff (mirrors debug --watch) P2 7. Non-goals Pixel truth. No browser, no screenshots, no layout aesthetics. The browser surface stays where it is (web e2e); this harness judges wiring and behavior. A beautiful-but-miswired app fails; an ugly-but-correct one passes. Replacing the App Builder. Humans still refine in Design view. The harness produces the first 90%, not the last 10%. Open-ended agents. An app with no fixed shape is Chat’s job (mini-apps.md says so); the harness refuses specs it cannot pin to operations and widgets rather than guessing. Publishing. The output is a bundle. Releases, spending limits, and sharing stay in the app lifecycle, untouched. 8. Success metrics Metric Target Where it’s read One-shot rate, app-build suite (zero repair rounds) ≥ 60% at launch, ≥ 80% two model generations later eval report, CI gate Green-within-budget rate (≤ 3 repairs) ≥ 90% eval report Repair convergence median ≤ 1 round BuildReport Cost per green app ≤ $0.50 median on the suite prediction ledger (nodetool costs, tagged app-build) Wall clock per app ≤ 3 min median, providers warm eval report Oracle honesty 0 known false-greens: every R6/R7 gap closed or listed in the report’s notSimulated field BuildReport The last row is the one that keeps the rest honest: a rising one-shot rate against a blind oracle is noise, not progress. 9. Milestones M1 — Close the oracle gaps. R6 + R7 in app debug. Ship independently; existing users of the debug harness benefit immediately. M2 — The loop. Spec → Plan → Author → Check → Run wired into nodetool app build with repair (R1–R4). Judge stubbed to verdict-only. M3 — The score. app-build suite + judge stage (R5, R8), CI gate on deterministic cases, nightly full run. M4 — Surfaces. build_app tool (R9), supervisor composition (R10), then the P2 conveniences. 10. Risks Risk Mitigation Repair loops oscillate (fix A breaks B, fix B breaks A) Complaints carry the full issue set, not the delta; a round that reintroduces a previously-fixed issue ends the loop early as failed Judge grades its own homework (same model plans and judges) Judge model is independently configurable, defaults to a different model than the builder; deterministic cases skip the judge entirely Spec stage invents unbuildable apps Spec is validated against the widget catalog and operation schema before Plan spends a token; unpinnable prompts are rejected at spec time (§7) Headless/browser drift makes green apps break in the web runtime Author and Run stages consume @nodetool-ai/app-runtime doc-ops and fold — the same code the browser runs; anything the harness cannot share it must list in notSimulated Cost blowups on hard prompts Same posture as the supervisor: caps are defaults, not options; exhausting one is a normal, reported outcome--- # Building Mini Apps Source: https://docs.nodetool.ai/mini-apps-guide.html Eleven kinds of app, each built step by step. They all start from the same short recipe; what differs is which widgets you place and what you wire them to. New here? Read Mini Apps first — it explains what a widget, a binding, and an operation are in a page. Every widget and setting is listed in the Reference. Fix the workflow first The workflow decides what the app can show and change. Sort this out first and the app itself is twenty minutes of layout. Anything the user should change needs an Input node. That’s what the widget wires to. A value with no Input node can’t be edited from the app — except through the node-setting trick in recipe 5. Anything the user should see needs an Output node. A workflow that just ends, with no Output node attached, produces nothing an app can display. Give every input a sensible default. An app whose first Run needs four empty fields filled in gets abandoned. With defaults, the first run is one click. Run the workflow once on its own first. Debug a graph as a graph. An app over a broken workflow only tells you the app is broken. The starting recipe Every app below begins here. Open the Apps panel in the left sidebar. The two buttons in its header are Create app from workflow and New app. Click New app for an empty screen, or Create app from workflow to get a starting layout with one widget per Input and Output node, a Run and Cancel button, a progress bar, and an error alert. Either way the app opens as its own workspace tab. Switch the tab to Design. That’s App Builder: palette on the left, the app screen in the middle, the selected widget’s settings on the right. Drag widgets from the palette onto the canvas. Select a widget and pick what it’s wired to in the right-hand panel. The menu lists what the workflow actually offers: Input nodes for widgets the user types into, Output nodes and variables for widgets that show results. Add a Button and give it an On click event with the Run workflow action. Click Save. Switch the tab to Run and try it. This is what the person using the app sees — no palette, no inspector, no graph. Then run nodetool app debug <application_id> to check the wiring the way the runtime sees it. To edit the graph while you build, open Linked workflows on the app tab and click the workflow. It opens as a normal workflow tab; the app tab stays put. The assistant on the right of App Builder can read the app’s workflows and edit the app for you: place widgets, wire them up, declare variables and operations, and add the Input, Output, or Set Variable nodes a layout needs. Operations, variables, and resources are also editable by hand in the App Data panel (the button in the Design header). Resource actions beyond the four in the event menu go through the assistant. 1. Fill a form, get a result Shape: fill a few fields, click a button, see one result. Good for: image generation, rewriting copy, summarizing, translating, classifying. This is the default app, and what most workflows should get. Place a Text Input for each Input node the user should fill, and wire each one to its Input node. Give it a label and a placeholder that shows what good input looks like. Place a Button named after the result, not the machinery: “Write the caption” beats “Run”. Place one display widget per Output node, wired to it: Markdown for prose, Image, Audio, Video, JSON for structured data, Table for rows, Output when the type varies. Give each display widget a placeholder (“Your caption appears here”) so the app looks finished before the first run. Two layout habits worth having: put the fields and the button in one Panel and the results in another, and use Columns so fields sit left and results right on a wide screen. 2. Runs that take a while Shape: a run lasting 30 seconds to several minutes. Good for: agents, batch jobs, video generation, research tasks. Build recipe 1, then add signs of life. Without them the app looks frozen and the user clicks Run again. Place a Progress widget wired to op:main/exec#progress. Place a Text widget wired to op:main/exec#activity — the line the run writes about itself: the tool an agent is calling, the planning stage, the step it’s on. For an agent, place an Agent Activity widget wired to op:main/exec#transcript instead. It shows the agent’s text and every tool call with its outcome while the run is live. Set the Run button’s disabledWhen to op:main/exec#running is not empty, so it can’t be clicked twice. Place a second Button with the Cancel run action, and set its visibleWhen to op:main/exec#running is not empty. Place a Text widget wired to op:main/exec#error, shown only when that is not empty. An agent app with no activity on screen is the single most common complaint about Mini Apps. Add it. 3. Text that types itself out Shape: the answer appears a few words at a time. Good for: chat-style answers, long drafts, live transcripts. Build recipe 1 with a Markdown widget wired to the Output node that streams. That’s it. The pieces are collected as they arrive, so one streamed answer renders as one block of Markdown instead of a flicker of fragments. If that same text feeds a later step, wire the output to a variable as well (to: "variable" on the operation). It builds up there the same way, and the next run starts a fresh value rather than continuing the last one. 4. A gallery of results Shape: one run makes several results, and the user keeps generating. Good for: image variations, ad creative, thumbnail sets, name candidates. Point the display widget at an Output node the workflow emits more than once. The items collect into a list instead of replacing each other. Use Table for structured rows, or an Image widget for a stream of images. Add a second Button labelled “Make another”, with the Run workflow action, placed below the results. Now nobody scrolls back up to the form. If a new set should replace the old one, give the operation the replace rule, so a second click cancels the run in flight first. The example apps NodeTool ships use this pattern. Add one from Start from an app on the start page and open it in Design to see how it’s wired. 5. A slider that re-runs Shape: drag a slider, the workflow re-runs, the result updates. Good for: image enhancement, color grading, thresholds, strength and guidance settings. This is the one recipe that doesn’t need an Input node — a widget can drive a node’s setting directly. Place a Slider and wire it to the node setting rather than to an input: op:main/prop:<nodeId>#<property>. The picker lists any node with settings, not just Input nodes. Set min, max, and step to the setting’s real range. Add an On change event with the Run workflow action. Set the event’s Pacing to On release, so the slider fires once when the user lets go instead of once per pixel. Use Debounced for a text field, and Live only for something cheap. Give the operation the replace rule: a new value should cancel the run for the old one, not line up behind it. Pacing and the run rule work together. Live pacing plus a queue rule on a five-second workflow builds a backlog the user can’t escape. 6. Approve, then continue Shape: run a step, look at the result, then run the next one. Good for: draft then publish, extract then post, analyze then act, generate then upscale. This needs two operations and two variables. Declare them in the App Data panel or ask the assistant. Declare a text variable called draft and a text variable called decision. Declare an operation draft running the drafting workflow, with its text output wired into draft (to: "variable"). Declare an operation publish running the publishing workflow, with its input reading draft (from: "variable"). Place a Markdown widget wired to var:draft, so the user reads what’s about to be published. Add an Approval widget bound to var:decision. It writes approved or rejected when the person picks one. Do not use the draft’s non-empty value itself as approval. Place a Run button for draft with two click events: Run workflow for draft, and Set variable on decision with an empty value, so every new draft starts unapproved. Place a Run button for publish. Set its visibleWhen to var:decision equals approved, and its disabledWhen to op:draft/exec#running is not empty. The draft output is cleared when its source run starts and after a failed or cancelled run, so a partial or stale result is never shown as the draft. A condition holds one test against one value, so the gate is the single decision variable. The publishing workflow should still validate that its input is the approved artifact before performing an irreversible action. 7. Settings that stick Shape: a tool used every day that should remember how it’s set up. Good for: tone of voice, target language, output format, default model. In App Data, declare a variable with scope Per user and Remember between visits on (scope: "user", persist: true). Only user-scoped variables can be remembered; marking an instance-scoped one as persistent raises a warning instead of quietly working. Give it a default, which seeds the first session. A remembered value is loaded first and beats the default. Wire a Select or Switch to var:<id>, with an On change event using the Set variable action. Have the operation input that uses it read from: "variable". The setting now survives a reload and drops out of the form the user fills in each run. 8. One app, several modes Shape: one app with modes, each showing only what its mode needs. Good for: a tool that takes either an image or text, a form with an advanced section, an app that switches between two workflows. Place a Select wired to a variable — call it mode — with options image and text, and an On change event using the Set variable action. Give each mode-specific widget a visibleWhen of var:mode equals image (or text). For an advanced section, put the widgets in an Accordion that starts closed, or in Tabs. Layout widgets take no condition, so a condition has to go on each widget it should hide. To switch workflows rather than fields, declare two operations and give each Run button its own operation and its own condition. A condition compares one value against one fixed value. Anything more involved — “show this when the score is high and the language is German” — belongs in the graph: emit one flag from a node and condition on that. 9. Apps that edit a document Shape: the app reads and writes a real document rather than loose values. Good for: storyboard editing, sketch iteration, curated asset collections. Declare a resource in App Data: its kind (asset, timeline, storyboard, or sketch) and what it may point at (all documents of that kind, or one fixed document). A resource added there allows read. Ask the assistant to grant create, update, or delete. Place a Resource Picker to choose one, or a Resource Gallery to show a grid of them. Have the operation input that uses it read from: "resource", naming that resource. Storyboard Scenes edits the document directly, so it fires no event of its own. nodetool app debug does not read the browser’s resource library. Seed a deterministic collection with a seedResource interaction or a resource:<binding-id> parameter to exercise resource inputs headlessly. Test picker rendering, focus, and real document writes in the app. 10. A chat app Shape: a conversation the user adds to, one workflow run per turn. Good for: assistants, Q&A over a collection, anything where the previous turns matter. Build the workflow with a Message List Input (the whole conversation) or a String Input (just the latest message), an LLM node, and an Output node. Declare a variable — call it chat — to hold the conversation. Place a Chat Thread: bind it to var:chat, and set Live reply to the Output node so the answer appears as it streams. Place a Chat Composer below it: bind it to the workflow input, set Conversation variable to chat, and pick what it sends — Whole conversation for a Message List Input, Message text for a String Input. Its On click event runs the operation. Add a Model Select bound to the LLM node’s model property to let the user choose the model. Turn on image attachments in the composer when the model is multimodal — the message then carries content parts instead of plain text, which is what a Message Input node and every provider expect. 11. An app that shows a sketch or a timeline Shape: a run that produces an editor document, previewed in the app. Good for: generated artwork you want to see composited, a cut assembled from a script, anything where the result is a document rather than a file. Build the workflow so it ends at an Output node fed by a node that emits a sketch (nodetool.sketch.CreateSketch, nodetool.constant.Sketch) or a timeline (nodetool.timeline.AddClips, nodetool.script.ScriptToTimeline). Place a Sketch or Timeline widget and bind it to that Output node. Set a max height so a tall canvas or a long sequence doesn’t push the rest of the app off screen. These nodes emit a reference — {type: "sketch", id} — not an image or a video file, so the media widgets show nothing when bound to them. If you also want the flat result, add a RenderSketch or RenderTimeline node and bind an Image or Video widget to its output; the two can sit side by side. Before you share it Every input widget is wired to something real. nodetool app debug --no-run names the ones that aren’t. Every display widget actually receives a value from a finished run. The app has at least one way to start a run. Runs longer than a few seconds show progress, activity, or both. The Run button is disabled while a run is in flight. Errors are visible on screen, not swallowed. Inputs have defaults, so the first run is one click. Display widgets have placeholders, so the empty app looks finished. Buttons are named after results, not mechanisms. When something’s wrong What you see What’s usually causing it A widget stays empty after a run It’s wired to a node the run never emits, or to an Output node the graph doesn’t have. A widget does nothing at all Its wiring points at nothing. Run app debug --no-run. Values from another run show up Can’t happen by design — messages are matched by job id and foreign ones are dropped. If the values look wrong, the widget is wired to the wrong node. The slider fires dozens of runs Pacing is set to live. Use on-release. Runs pile up behind each other The operation’s rule is queue. Use replace. A remembered setting resets The variable is instance-scoped. Only user-scoped ones are remembered. The app broke after a graph edit An old name-based wiring pointed at a renamed node. Re-select it; the editor writes id-based wiring, which survives renames. Related Mini Apps — the concepts and how the runtime works Mini App Reference — widgets, bindings, schema App Builder — the editor Templates Gallery — example apps and workflows Workflow Debugging — debugging the graph underneath--- # Mini Apps on Mobile Source: https://docs.nodetool.ai/mini-apps-mobile.html The mobile app runs the same Mini Apps you build on the desktop. It reads them from the server, renders the document with native controls, and runs the workflow behind them. It does not edit them — the App Builder stays on the desktop. New to Mini Apps? Read Mini Apps first: it explains what a widget, a binding, and an operation are. Finding your apps The Apps browser lists every app on the server you are connected to, newest first. It is the home screen after you sign in, and the deep link nodetool://apps opens it. Each row shows the app’s name, how many operations it declares, and when it last changed. Pull down to refresh. Tapping a row pushes the app onto its own screen; nodetool://app/<application-id> opens one directly, which is what a notification or a shared link points at. If the list is empty, the screen reads “No apps yet. Build one in the desktop app builder.” Any app on the server is listed, published or not, and a new one shows up on the next refresh. Running an app An app screen renders the released snapshot if the app has been published, and the draft document otherwise. The heading comes from the document, the widgets below it are the ones you placed in the builder. Inputs, the Run button, and the progress bar behave as they do on the web. While a run is going the button shows a spinner, and the Progress widget shows what the run says it is doing — the node’s activity label outranks the static label the author set, so an agent workflow reports its step instead of spinning silently. A failed run puts its error in a banner above the app; the next run replaces it. Outputs fill in as the run streams them. Widgets Every widget in the builder’s catalog has a native renderer. The document decides layout and behavior; only the controls differ. These render differently enough to be worth knowing: Widget On the phone Columns Stacks vertically. Two columns side by side are unusable at phone width. ImageInput / AudioInput / VideoInput / DocumentInput Opens the system photo or file picker. The pick is uploaded to the server so the workflow gets a URL it can fetch; if the upload fails the local file is used, which still previews on the device. Sketch Pad An image picker. The app bundles no drawing surface, so the pad offers the camera and photo library for the same binding and says drawing needs the desktop app. Audio Recorder / Camera Capture Records for real: the microphone through a record/stop control with elapsed time and playback, the camera through the system camera sheet. Both upload the take, so the workflow reads the same {type, uri, asset_id} an upload writes. A denied permission says so and leaves the file picker as the way through. Workflow Form One row per workflow input, in graph order. Gallery Tapping a tile opens the pager; when the app wired a selectionBinding, tapping picks and a long press previews. Image Compare Two labelled panes, before above after — a wipe handle needs more width than a phone has, and a thumb would cover it. ResourcePicker / ResourceGallery A list of rows rather than a grid of tiles. Tapping one opens that document in the screen its kind already has — a storyboard in the storyboard editor, a sketch in the sketch viewer, an asset in the asset viewer. ChatThread / ChatComposer / ModelSelect Native bubbles, a multiline composer, and a two-step provider-then-model picker. A widget the phone has no renderer for says so in place — “… is not available on mobile” for a catalog widget, “Unknown widget” for one from a newer builder than the installed app. Nothing silently disappears. What the phone does differently One workflow per screen. The screen holds the workflow the app’s first operation binds. Other operations over that same workflow run normally, each dispatched by id. An operation bound to a different workflow refuses to run and says so, rather than running the wrong graph under its name. Every run is a server run. There is no browser worker on mobile, so the reactive-subgraph path the web runtime uses does not exist. A run action always executes the whole workflow on the server. Apps that lean on partial re-execution feel heavier here. Runs are matched by job id. A streaming message for any other job is dropped, so a run you started in the chain editor never folds into what the app shows. Persistent variables survive the app closing. A variable declared scope: user with persist: true is stored on the device, keyed by app, and restored before the document’s defaults are applied — last session’s value beats the declared default. Instance-scoped variables die with the screen, which is what their scope means. Requirements and limits The mobile app talks to a NodeTool server; set its URL in Settings and see Mobile App for connecting from a physical device, emulators, and firewalls. Known limits: No editing. Authoring an app document needs the desktop builder. Apps are listed per server. Switching servers in Settings switches the list. A Timeline widget shows a summary card (duration, track and clip counts), not the timeline itself. A Sketch widget renders the composited image, and falls back to a size and layer summary when it cannot load the document. Related Mini Apps — what an app is, and the vocabulary Building Mini Apps — nine app shapes, built step by step Mini App Reference — every widget and setting Mobile App — the rest of the mobile app--- # Mini App Reference Source: https://docs.nodetool.ai/mini-apps-reference.html Every part of a Mini App, in tables. For what these things are, read Mini Apps; for how to put them together, read Building Mini Apps. All of it is defined in packages/app-runtime, shared by the browser, the nodetool app debug harness, and the test suites — so all three behave the same. Widgets A widget is one thing on the app screen. Every widget except the layout widgets can be wired to a value (binding), shown only under a condition (visibleWhen), and greyed out under a condition (disabledWhen), on top of its own settings. format is offered on the widgets that show text: Heading, Text, Markdown, JSON, Output, Alert, Code, List, and Stat. The widget catalog (WIDGET_CATALOG in packages/app-runtime/src/widgets.ts) is the single list the editor, the debug harness, and the assistant all read. The editor palette groups widgets as Inputs, Chat & AI, Actions, Display, and Layout. Widgets that show something Widget Shows Heading Fixed text at H1–H3. Text Fixed or formatted text. Markdown Rendered Markdown. The right choice for streamed prose. Image An image. Fit contain or cover, fixed height, placeholder, optional download button and filename. Audio An audio file, with a player, placeholder, optional download button and filename. Video A video, with a player, max height, placeholder, optional download button and filename. Sketch A sketch document, layers composited. Max height, optional canvas size. Timeline A timeline sequence, with its tracks and clips. Max height, optional metadata. JSON Structured data, formatted. Table A list, as rows. Max height, placeholder. List A list as bullets, or numbered with Numbered. Placeholder. Key/Value A record as label and value pairs. Placeholder. Output A value whose type varies; picks a display based on what arrives. Optional media download. Progress How far along the run is. Agent Activity What an agent did during the run: its text and each tool call with the outcome, live. Bind it to op:<opId>/exec#transcript. Label, height, placeholder. Stat One number with a label and a caption, for reading at a glance. Placeholder. Alert A message box. Severity info, success, warning, or error, optional title and text. Pair it with visibleWhen to show an error only when there is one. Code Text in a code block. Language label, max height. Download A button that saves the bound file. Label, file name, placeholder. 3D Model A 3D model viewer. Height, placeholder. Chart Plots a bound dataframe, list of records, or list of numbers. Chart type line, bar, scatter, or pie, height, placeholder. PDF A PDF viewer. Height, placeholder. Gallery A list of images as tiles. Tile size, placeholder. Set selectionBinding and a tap picks one. Image Compare Two images under one wipe handle. binding is before, compareBinding after. Max height, placeholder. Sketch and Timeline take a document reference — {type: "sketch", id} or {type: "timeline", id} — which is what the nodes that produce them emit. They also accept the document inline, so a node that returns the payload rather than a saved id renders too. Binding one to Image or Video instead shows nothing: a reference is not a media URL. On mobile a Sketch draws the composited image and falls back to a size and layer summary when it cannot load it, and a Timeline shows a summary of duration, tracks, and clips rather than the sequence. Widgets the user changes “Commits” means the control reports a final value when you let go or click away. That’s what makes on-release pacing possible — see Triggers and pacing. Widget Takes Commits Workflow Form Every input of one operation at once, each rendered to match. no Workflow Input Whatever the bound Input node declares, rendered to match. no Text Input Text. Single-line or multiline. yes Number Input A number. Min, max, step. yes Slider A number. Min, max, step. yes Switch On or off. no Select One option from a fixed list. no Radio Group One option from a fixed list, stacked or inline. no Checkbox Group Any number of options from a fixed list, written as a list. Stacked or inline. no Choice Cards One option shown as a card with title, description, and image. Options are authored, or read from a bound output through optionsBinding. Columns. no Guided Steps One step of an ordered list, as buttons. Writes the step’s value, starts on the first step, and allowBack can forbid going back. no Date Input A date, or a date and time with Include time. Writes text. yes Image Input An image. no Sketch Pad A drawing the user makes on the spot. Canvas size, white or transparent paper. yes Audio Input An audio file. no Audio Recorder Audio recorded from the microphone, then and there. no Video Input A video. no Camera Capture Video recorded from the camera, then and there. no Document Input A document. no Color Input A color. no Data Table Input A table of rows. Max height. no File Path Input A file path as text. no Folder Path Input A folder path as text. no 3D Model Input A 3D model. no Image Size Input A width and height. no Media List Input A list of images, videos, audio files, or text (List of). no Resource Picker Picks which document a resource points at. no Resource Gallery The same, from a grid of tiles. Tile size. no Storyboard Scenes Edits the bound storyboard directly. Fires no event. no Choice Cards and Guided Steps suit the pick-then-continue step of an approval flow. See recipe 6. A Workflow Form is one widget where placing each input by hand would be several. It renders the inputs of the operation it names — the app’s only operation unless you pick another — so an Input node added to the workflow shows up in the app with no edit. Place the inputs one by one instead when the layout matters: a form renders them in graph order, top to bottom. The Audio Recorder and Camera Capture write the same {type, uri, asset_id} ref an upload writes, so bind them to an Audio or Video Input and the workflow reads one value either way. The recording uploads as an asset when it finishes. On the builder canvas they show a placeholder instead of asking for the microphone or camera, so laying out an app never raises a permission prompt. A Gallery with a selectionBinding is the generate-then-pick loop: bind the gallery to a run’s list of images, the selection to the input of the next step, and wire a run event to the selection so picking a tile starts it. What lands in the binding is the item the run emitted, not the URL it rendered. The Sketch Pad is the sketch editor’s canvas with a four-tool chrome — brush, pencil, eraser, fill, plus colors, stroke size, undo and clear. Each finished stroke flattens the drawing to a PNG and writes it as {type: "image", uri}, so bind it to an Image Input and the workflow reads it the way it reads an upload. White paper is the default because an image model is fed an opaque picture; transparent keeps the alpha for a mask or an overlay. Nothing is written until the user draws, so a pad wired to a run does not fire it on load. The drawing travels inline as a data URI, so keep the canvas near its 512×384 default rather than at the 2048px ceiling. Chat and AI Widget Does Chat Thread Shows a conversation. binding is the conversation (an app variable, or an output that emits messages), streamBinding the reply arriving from the current run. Max height, placeholder. Chat Composer Writes the next message and runs the operation. binding is the workflow input it sends to, historyBinding the conversation variable it appends to. Sends the message text, a message object, or the whole conversation. Optional image attachments. Model Select Picks a model and writes its reference. Kind: language, image, video, speech, transcription, or embedding. A conversation is a list of {role, content} objects. Anything else a binding holds — a streamed string, a media ref, a list of results — reads as one assistant turn, so a thread bound straight to an LLM output still renders. The thread does the bookkeeping a second turn needs: when a run settles, it folds the streamed reply into the conversation variable. An output slot is cleared at the start of every run, so a reply left there alone would disappear the moment the user sends again. This only happens when binding is a variable and streamBinding is an output. Buttons and layout Widget Does Button Runs its click action. Style contained/outlined/text, color primary/secondary/warning. Approval Two buttons that write approved or rejected to its binding. Label, description, and button labels (defaults Approve and Needs changes). Fires a change event. Panel A titled box holding other widgets. Columns Two side-by-side slots, left and right. Tabs Up to three tabs, each a slot (tab1, tab2, tab3) with its own label. Accordion A titled section that opens and closes. defaultOpen sets its first state. Spacer Empty vertical space of a set height. Divider A horizontal line. Layout widgets carry no binding, condition, or format. To show or hide a group under a condition, set visibleWhen on each widget inside it. Tabs and Accordion let the person using the app hide a group themselves. Bindings A binding is the string that says what a widget is wired to. You usually pick these from a menu rather than typing them. They point at node ids, so renaming a node in the graph editor never breaks an app. Binding Points at op:<opId>/in:<nodeId> An input of one of the app’s workflows. op:<opId>/out:<nodeId> An output of one of the app’s workflows. op:<opId>/prop:<nodeId>#<prop> A setting on a node, driven by a widget. op:<opId>/exec#<field> Run status: running, progress, error, activity, transcript. transcript is the agent text and tool calls of the latest run, in order. var:<variableId> A value the app remembers. view:<componentId>#<prop> State belonging to one widget. Never saved. node:<nodeId>#<prop> Old form of a node setting, resolved against the default operation. <name> Old form: a bare node name, looked up in the live graph. A bare name is looked up according to how the widget uses it: a widget the user types into looks for an Input node, a widget that displays looks for an Output node and then a variable, and a condition or format token tries outputs, then inputs, then variables. A name that matches nothing is an error, not a silent no-op. Actions A widget event runs one action. Actions are settings the runtime carries out — there’s nowhere to write code. Action Settings Effect run operationId Runs that workflow, following its concurrency rule. cancel operationId, optional invocationId Cancels that operation’s runs, or one specific run. setVariable variableId, and a value or the widget’s own value Writes a variable. toggleVariable variableId Flips an on/off variable. resourceCommand resourceBindingId, command read, create, update, delete, or upload on a resource. openResource resourceBindingId Opens that document in its editor. The visual editor offers Run workflow, Cancel run, Set variable, and Toggle variable. The resource actions are set through the agent or by editing the document. Triggers and pacing Events fire on click (buttons) or change (everything the user edits). A change event also has a pace, which decides how often it fires while the user is still editing: Pace Fires live On every change. release Once, when the control settles — the slider is let go, the field loses focus. Only offered on controls that commit. debounce Once, after a short pause in editing. Conditions visibleWhen and disabledWhen each hold a binding, an operator, and a value to compare against. A condition whose binding points at nothing is treated as no condition, so broken wiring never silently hides a widget. Operator Editor label True when notEmpty is not empty The value is set, non-empty, and not false. empty is empty The value is unset, empty, or false. eq equals Equal, after converting your value to the same type. neq does not equal Not equal. gt / gte is greater than / is at least Numbers. A non-number never satisfies these. lt / lte is less than / is at most Numbers. contains contains Text contains the substring, or a list contains the item. You always type the comparison value as text, and it’s converted to match what it’s compared against — so count gt "3" compares numbers and dark eq "true" compares on/off. Format templates format replaces {binding} tokens with the value, optionally passed through one filter: {op:main/out:n1|truncate:80}. An unknown filter or a binding that points at nothing renders as nothing. Filter Argument Result number digits The value as a number, optionally to N decimal places. date short Local date and time, or just the date with short. upper — UPPERCASE. lower — lowercase. join separator A list joined into text; defaults to ", ". truncate length Cut to N characters, with an ellipsis. Document schema The rest of this page is the saved shape of an app — useful when editing a document by hand or reading one the agent wrote. An app is stored as one ApplicationDocument. Ordinary apps use schema version 4. Version 5 is for apps that carry a recipe manifest, and a document with a higher version is rejected. Embedded version 1 and 2 documents ({ version, data } on a workflow) are upgraded on load, gaining one implicit main operation bound to the workflow they came from. interface ApplicationDocument { schemaVersion: number; // 4, or 5 with a recipe ui: PuckData; // the layout: { root, content, zones } operations: OperationBinding[]; resources: ResourceBinding[]; variables: VariableDeclaration[]; theme?: { id: string }; // "default" | "centered" | "card" recipe?: RecipeManifest; } Operations One operation is one workflow the app can run, plus its wiring. interface OperationBinding { id: string; name: string; workflowId: string; // empty for a script operation workflowVersion?: number; // fixed in a published app, latest in a draft target?: // set when the operation runs a script | { kind: "workflow"; workflowId: string; workflowVersion?: number } | { kind: "script"; scriptId: string; scriptVersion: number }; inputs: Record<string, InputMapping>; // keyed by node id outputs: Record<string, OutputMapping>; // keyed by node id policy: "parallel" | "replace" | "queue"; timeoutMs?: number; } Input mapping Where the value comes from { from: "widget" } The widget wired to it. The default when nothing else is set. { from: "variable", variableId } A value the app remembers. { from: "constant", value } A fixed value. { from: "resource", resourceBindingId } Whichever document that resource currently points at. Output mapping Where the value goes { to: "display" } The slot display widgets read. { to: "variable", variableId } A variable, and the display slot. policy If you start a run while one is already going parallel Start anyway. replace Cancel the running one, then start. The default. queue Wait for it to finish, then start. Several operations may point at the same workflow with different wiring, and an app that runs three workflows declares three operations. A script operation runs a saved JS script at a pinned version instead of a workflow. The script’s declared input and output names stand in for node ids, so inputs and outputs key on those names. Publishing checks that the pinned version exists. A public link cannot serve a script operation. Variables interface VariableDeclaration { id: string; name: string; type?: { type: string; optional?: boolean } | null; default?: unknown; scope: "instance" | "user"; // this open app, or saved per user persist: boolean; // only user-scoped variables may be saved } The visual editor’s variable picker lists the Set Variable channels the graph publishes. The App Data panel in the Design header declares variables with a type, default, scope, and persistence. Operations and resources are edited there too, and the assistant edits the same lists. Resources A resource is a handle to a real document the app may read or edit. interface ResourceBinding { id: string; name: string; kind: "asset" | "timeline" | "storyboard" | "sketch"; scope: { projectId?: string; fixedId?: string }; operations: ("read" | "create" | "update" | "delete")[]; } Recipe manifest A recipe is optional metadata that marks an app as a recipe: named, typed inputs, a fixed set of operations, and declared outputs. It holds a slug, typed inputs (text, number, boolean, image, video, audio, color, asset, entity, storyboard, timeline), the operations it uses with their intent, outputs, and optional creative strategy, preservation rules, media policy, presentation, and marketing fields. The document is rejected unless every recipe input has an app variable with the same id and a compatible type, every required input is mapped into a recipe operation, and every output binding resolves. An ordinary update cannot strip the manifest from an app that has one. Bundles A bundle is an app packaged for sharing: the app plus the full graph of every workflow it runs, in one JSON file. Export, import, and the shipped example apps all use it. interface ApplicationBundle { schemaVersion: number; // 1 name: string; description: string; app: ApplicationDocument; // operations reference workflows[].key and scripts[].key workflows: { key: string; name: string; description?: string; graph: Graph; sourceId?: string; // stable identity across installs version?: number | null; graphHash?: string | null; }[]; scripts: { key: string; name: string; document: JsScriptDocument; sourceId?: string; version?: number | null }[]; } Inside a bundle, each operation’s workflowId (or script id) holds a local nickname (key) instead of a real id. Importing creates the workflows and swaps the nicknames for the new ids, the same trick .nodetool workflow bundles use for their asset references. A workflow with a sourceId gets a stable row id, so importing two bundles that ship the same workflow reuses one row. A bundle exported with --released carries the locked-in graphs, so it reproduces exactly what that release ran. What one open app holds Group Keyed by Holds inputs opId:nodeId, or opId:nodeId#prop { value, dirty, revision }. dirty turns true once a widget rather than a default wrote it. outputs opId:nodeId { value, invocationId, status, revision }. Status is empty, pending, streaming, or done. variables variable id The current value. view componentId:prop State belonging to one widget. invocations job id { id, operationId, status, progress, error, startedAt }. A run’s status is pending, running, completed, failed, or cancelled. Streamed values accumulate: text is joined, structured items collect into a list. A message from a job this app didn’t start is thrown away. Agent tools The builder agent edits the open app through these tools. Every one takes an application_id — the app’s own id, listed in the ui_context block. A workflow id is never accepted; the workflows an app runs are named by target_workflow_id on its operations. Area Tools Layout ui_app_get_snapshot, ui_app_list_component_types, ui_app_add_component, ui_app_update_component, ui_app_remove_component, ui_app_select_component, ui_app_set_title Operations ui_app_list_operations, ui_app_add_operation, ui_app_update_operation, ui_app_remove_operation Variables ui_app_list_variables, ui_app_declare_variable, ui_app_update_variable, ui_app_remove_variable Resources ui_app_list_resources, ui_app_add_resource, ui_app_remove_resource Bindings ui_app_get_binding_targets Checking ui_app_debug ui_app_debug sends the open draft, unsaved edits included, to the server and returns the same verdict as nodetool app debug. With run false (the default) it is a free wiring check. With run true it executes the workflows and spends money, so run it once at the end. It also takes params, interact, and timeout_ms. REST API The same service backs the web app’s tRPC calls and these routes, which mobile, the CLI, and other clients use. Route Does GET /api/applications List apps. POST /api/applications Create an app, empty or from a workflow. GET, PUT, PATCH, DELETE /api/applications/:id Read, replace, update, or delete one app. GET /api/applications/:id/released-document The released snapshot and the graphs it pinned, or null. GET /api/applications/:id/export-bundle Download the app as a bundle. POST /api/applications/import-bundle Create an app and its workflows from a bundle. POST /api/applications/debug Run the debug harness on a document. POST /api/applications/build Build an app from a prompt. GET /api/applications/examples List the shipped example apps. GET /api/applications/examples/:slug One example as a bundle. POST /api/applications/examples/:slug/install Add an example to your apps. Publishing, versions, rollback, the budget, and the public link are tRPC procedures on applications (publish, versions, release, setBudget, deploy, and others). A public link is served at /a/<token>. CLI npm run dev:nodetool -- app debug <application_id> npm run dev:nodetool -- app debug my-app.json --params '{"prompt":"hi"}' npm run dev:nodetool -- app debug <id> --no-run # check the wiring, don't run npm run dev:nodetool -- app debug <id> --json # full report # Script the clicks and typing npm run dev:nodetool -- app debug <id> --interact \ '[{"set":{"key":"prompt","value":"hi"}},{"click":"Button-1"}]' npm run dev:nodetool -- app debug <id> --interact \ '[{"set":{"key":"tone","value":"terse","operationId":"draft"}},{"run":"draft"}]' # Timeout, report folder, build, and bundles npm run dev:nodetool -- app debug <id> --timeout 60000 --out ./app-report npm run dev:nodetool -- app build "<prompt>" -p anthropic -m claude-sonnet-5 npm run dev:nodetool -- apps list npm run dev:nodetool -- apps export-bundle <id> -o my.app.json --released npm run dev:nodetool -- apps import-bundle my.app.json --project <project_id> An interaction step is one of {"set":{"key","value","operationId"}}, {"change":"<widget>","value":…}, {"click":"<widget>"}, {"run":"<operationId>"}, {"cancel":"<operationId>"}, or {"seedResource":{"id","items"}}. --params sets values before the steps run. The command exits 0 when the verdict is ok. app build and the apps commands are covered in the harness reference and CLI. The harness runs every operation the app declares, not just the first. Widgets are clicked by component id, by type if only one exists, or by label if it’s unique. Results land in nodetool-debug/app-<id>-<ts>/ (or the --out folder) as report.json, report.md, app.json, workflow.json, and one server/run-N.messages.jsonl per run. The harness evaluates visibleWhen, disabledWhen, and format for scripted interactions. It does not render layout, styling, focus, or scroll, and it does not read the user’s resource library. Resource tests must seed an in-memory collection with seedResource or a resource:<binding-id> parameter. Reactive browser subgraph runs are reported separately because the harness executes the whole workflow. Related Mini Apps — concepts and runtime Building Mini Apps — recipes per use case App Builder — the editor CLI — the full command reference--- # Mini Apps Source: https://docs.nodetool.ai/mini-apps.html A Mini App is a simple screen in front of a workflow. Instead of a canvas full of connected nodes, the person using it sees a text box, a button, and a result. Say you built a workflow that turns a product photo into a caption. It has a dozen nodes: the model, the prompt, retries, formatting. You want a coworker to use it, but not to edit it, and not to have to understand it. So you build a Mini App: one field for the photo, one button that says “Write the caption”, and one box where the caption appears. Same workflow underneath, nothing to break. Building Mini Apps — eleven app shapes, each built step by step. Start here if you want to make one. Mini App Reference — every widget, setting, and field, in tables. App Builder — the editor you build the screen in. Mini Apps on Mobile — opening and running the same app from a phone. The words These five terms cover most of what the rest of these pages say. Term What it means Workflow The node graph that does the actual work. It already exists before you build an app. Widget One thing on the app screen: a text box, a button, an image, a slider. Binding The wiring between a widget and the workflow. “This text box fills the Prompt input.” “This image shows the Result output.” Operation One workflow the app can run, plus how its inputs and outputs are wired. An app that runs two workflows has two operations. Variable A value the app remembers between runs — a draft to review, a setting like “always translate to German”. Two more you meet in the workflow editor, not the app: An Input node is a node in the graph that marks a value coming in from outside. Widgets that the user types into bind to Input nodes. An Output node marks a value going out. Widgets that show a result bind to Output nodes. If a value has no Input node, no widget can change it. If a result has no Output node, no widget can show it. Getting these into the graph is the first step of building any app. What Mini Apps are good for A node graph is the right screen for the person who built the pipeline and the wrong one for everyone else. It shows model choices, retry paths, and half-built intermediate steps that someone who just wants a caption doesn’t need and shouldn’t be able to break. A Mini App narrows it to the two or three fields that actually change. The situation What the app does about it A coworker needs to run your workflow but not edit it They get fields and a Run button. The graph isn’t reachable from the app. The workflow has 40 nodes and 3 things worth changing Only those 3 get widgets. The rest stays fixed in the graph. A run takes 90 seconds and looks frozen Add a progress bar and a status line that says what the run is doing right now. The result arrives a word at a time Display widgets collect what streams in. Text builds up into one block instead of flickering. One screen should drive several workflows Give the app one operation per workflow. Each gets its own fields, results, and button. Someone must approve a draft before the next step Step one writes a variable, a widget shows it, a second button runs step two using that variable. Dragging a slider should re-run the workflow Bind the slider to a setting in the graph and have it re-run when you let go. The same tool is used daily with the same settings Mark a variable as remembered, and it survives across sessions. Don’t build one for a workflow you’ll run once, for a pipeline another program calls (use the Workflow API), or for an open-ended task with no fixed shape (use Chat). Where apps live An app is its own thing, not a setting on a workflow. It has its own name, version history, releases, and spending limit — none of which belong on a graph. Opening a workflow never creates an app, and a workflow without an app is just a workflow. Where How to get there Apps panel Left sidebar → Apps. Every app in the project, plus the New app and Create app from workflow buttons. Right-click a row to rename, duplicate, or delete an app. App tab Click an app. It opens as a workspace tab with four views: Design, Run, Preview draft, Settings. Linked workflows A Linked workflows menu on the app tab, listing the workflows this app runs. Click one to open it as a normal workflow tab. Terminal nodetool app debug <application_id>. See Checking an app. Design is where you place and wire widgets: Run is what the person using the app sees: the fields, the button, and the result. Each working instance runs its pinned version. A new default instance uses the release when one exists, or the unpublished draft. Preview draft freezes a separate draft working copy when requested, so you can test edits without changing a released instance. Settings holds publishing, versions, the spend budget, the public link, and recent runs. See Publishing and sharing. Apps run wherever their workflows run: the desktop app, a server you host, or NodeTool Cloud. On mobile, a published app opens as its own screen. Making one New app starts empty. You add an operation, pick the workflow it runs, and place widgets yourself. Create app from workflow builds a starting point for you. You pick a workflow, and the app gets one operation named main bound to it, a Workflow Input widget for every nodetool.input.* node, an Output widget for every nodetool.output.* node and Preview node, a Run button, a Cancel button that shows while a run is going, a Progress widget, and an error alert. If the workflow already carries an older embedded app document, that document is imported instead. This is a copy, made once. From then on the app and the workflow are separate, and editing one doesn’t change the other. Publishing and sharing Publish new version (Settings) saves the draft first, then takes a snapshot. The snapshot is the screen, the variables, and the graph of every workflow the app runs, frozen at that moment. It becomes the released version at once, so a published app keeps working the way it did on release day while you keep editing the drafts. Every version is listed with what it may touch (workflow count and resource access). Roll back to version N makes an older version the released one again. Settings has three more sections: Spend budget. A period (per day, per month, or lifetime), a maximum spend in USD, and a maximum number of runs. An empty field means no limit. Runs of the released app are checked against it before they reach a provider, and the section shows what has been used so far. Public link. Create public link makes a hidden /a/<token> URL. Anyone with it can open and run the released version without a NodeTool account. Runs execute on your account and count against the budget. Withdraw link turns it off. Links are served only when the server runs with NODETOOL_ENV=production, as nodetool.ai does, so a local install shows a note instead. A link needs a released version and a budget with at least one finite limit. Apps with script operations or resource bindings cannot be linked, and publishing such a change while a link is live is refused. Recent invocations. Each run of the released app with its operation, status, version, and cost (marked “est.” until the actual cost is known). To hand an app to someone else without a link, export a bundle: one JSON file holding the app and the full graph of every workflow it runs. They import it and get a working app plus its workflows. Bundles move through the CLI (nodetool apps export-bundle <id> and nodetool apps import-bundle <file>, see CLI) and through the GET /api/applications/:id/export-bundle and POST /api/applications/import-bundle routes. Inside the file the app refers to its workflows by a local nickname. Importing swaps those for the real ids of the workflows it creates. NodeTool ships example apps as bundles. Add one from Start from an app on the start page, or install it with POST /api/applications/examples/:slug/install. How it works Four pieces. Only one of them computes anything. App document (what you build in the editor) ui the layout: which widgets, where, wired to what operations which workflows the app runs, and how they're wired variables values the app remembers resources handles to an asset, timeline, storyboard, or sketch │ │ the wiring is checked against the real graphs ▼ Instance state (what one open copy of the app is holding) inputs · outputs · variables · widget state · runs in flight │ │ messages from the run update the state as they arrive ▼ Workflow runs (where all the actual work happens) The document is configuration, not code. Nothing in it branches, loops, or calls a model — that’s the graph’s job. This is on purpose: logic in a graph can be tested, cached, and read by the agent. Logic hidden in an app screen can’t. The app document Everything you build in the editor is saved as one document with these parts: ui — the layout. Which widgets are on the screen, where, and what each one is wired to. operations — the workflows this app can run. Each operation has a name, what it runs, wiring for each input and output, a rule for what happens when you click Run while it’s already running, and an optional time limit. An operation runs a workflow, or a JS script at a pinned version (see Reference). One app can run several workflows, or the same workflow twice wired differently (translateTitle and translateBody). variables — the values the app remembers, each with a type, a starting value, and whether it survives a reload. resources — handles to a real document (an asset, timeline, storyboard, or sketch) that the app is allowed to read or edit. theme — optional. A named page style picked in the editor’s Theme field: Default, Centered (content capped at 720 px), or Card (content in a bordered card, capped at 880 px). In the editor, the App Data button in the Design header opens a panel where you add and edit operations, variables, and resources by hand. The assistant edits the same lists. Operations are what make an app more than a form over one graph. A transcription app can run a transcriber, a summarizer, and a translator from one screen. Bindings A binding is a short string that says which slot a widget is wired to. You pick these from a menu in the editor; you rarely type them. They look like this: op:<opId>/in:<nodeId> an input of a workflow op:<opId>/out:<nodeId> an output of a workflow op:<opId>/prop:<nodeId>#<prop> a setting on a node, driven by a widget op:<opId>/exec#<field> run status: running | progress | error | activity var:<variableId> a value the app remembers view:<componentId>#<prop> state that belongs to one widget, never saved Two details worth knowing: Every binding names its operation, so two operations running the same workflow never read each other’s values. Bindings point at node ids, not names. Renaming an Input node in the graph editor can’t break an app. Older apps that used names still work — the name is looked up in the live graph, and the editor rewrites it to the id form when it can. A binding that points at nothing is reported as an error, not quietly ignored. Both the editor and nodetool app debug will tell you. What one open app holds While an app is open it holds a bundle of current values, split into five groups so nothing collides: Group Keyed by Holds inputs operation + node What’s currently in each field, and whether the user typed it or it’s still the default. outputs operation + node Each result and its status: empty, pending, streaming, done. variables variable The values the app remembers. view widget + property State belonging to one widget. Never saved. invocations job id One entry per run: status, progress, error, and what it’s doing. Keying inputs and outputs by operation and node is what lets one app run several workflows without their values landing on top of each other. The same code updates this state in the browser, in the nodetool app debug harness, and in the test suites. What the harness reports is what the browser does. What happens when you click Run The widget fires an action: run, cancel, setVariable, toggleVariable, or one of the resource actions. Actions are settings the runtime carries out; there’s nowhere to write a statement. If the operation is already running, its rule decides: parallel starts another anyway, replace cancels the one in flight first, queue waits. The values are collected. Each input takes its value from wherever it’s wired: the widget (the usual case), a variable, a fixed value, or the currently selected resource. The run starts, and every result slot it will fill goes to pending. Messages come back from the run and update the screen. Each one is matched to the run that produced it by job id, and anything from a job this app didn’t start is thrown away. That’s what stops a second tab, an overlapping run, or a run launched from a workflow tab from polluting what the app shows. Values land in their result slot — and also in a variable, if the operation is wired that way. Streamed text builds up; streamed items collect into a list. Clicking Run again starts fresh instead of appending to the last result. Saved working state Opening an app loads its server-owned default instance. Widget inputs, variable state and terminal outputs survive a reload and follow the authenticated account between browsers. Existing local variable caches are imported when the default instance is first created. An existing server instance wins over a browser cache. Each operation reserves a durable app run before collecting its inputs. The run stores its inputs, outputs, outcome and child generation attachments. It executes the definition pinned by its instance. Revision checks prevent a stale browser or delayed run from overwriting newer working state. A conflict keeps the run’s outputs and asks you to reload the instance. The instance header offers New, Rename, Duplicate, Delete, and advancement to a newer release. New starts from app defaults. Duplicate copies working values and the frozen definition, reuses media references, and starts with empty history. Each instance opens in its own workspace tab. Editing opens the application editor, while Run returns to a working instance tab. Run history is scoped to the instance. Selecting a run shows read-only inputs, outputs, stored agent activity, and links to its trace and documents. Document links open the current document. Historical inspection changes no working values and starts no execution. Expired or limited content is labelled. Each run also has View trace and Ask the agent. Ask the agent opens a chat tab titled “Inspect run” with the run id attached, and the first failed span when there is one. Deleting an instance removes its history and attachments while retaining generated library media. Deployed-app visitors keep local working state. Their server runs retain no visitor inputs or outputs, and they cannot manage owner instances. See App runs for the persistence and API contracts. Showing that something is happening Every operation exposes four things a widget can display, so an app over a slow workflow shows more than a spinner: running — true while a run is in flight. progress — how far along it is. error — what went wrong, if anything. activity — a line of plain text the run writes about itself: the tool an agent is calling, the planning stage, the step it’s on. If your app wraps an agent, put activity on the screen. Without it the user stares at a spinner for two minutes with no idea whether anything is happening. The little logic an app can do Three things, and the list is short on purpose: Show a widget only when something is true (visibleWhen). Grey a widget out while something is true (disabledWhen). Reformat a value before showing it (format) — round a number, shorten a string, join a list. A condition compares one value against one fixed value, using one of nine operators. Formatting runs a value through one of six filters. Anything past that belongs in the graph: compute the value in a node and show that node’s output. Checking an app nodetool app debug runs the whole app without a browser. It checks every binding against the real workflow, fills in the default values, clicks the Run button (or follows a script you give it), runs the workflow, and reports what every widget ended up showing. It runs all of the app’s operations, not just the first. npm run dev:nodetool -- app debug <application_id> npm run dev:nodetool -- app debug my-app.json # a bundle file npm run dev:nodetool -- app debug <id> --no-run # check the wiring, don't run npm run dev:nodetool -- app debug <id> --json # full report npm run dev:nodetool -- app debug <id> --timeout 60000 --out ./app-report The command exits 0 when the verdict is ok and 1 otherwise. --timeout is a per-run ceiling in milliseconds. An operation’s own time limit also applies, and the shorter one wins. --out sets the report folder. The target can also be a workflow id or file that carries an older embedded app document. It catches the mistakes a workflow-only test can’t: a widget wired to an input or output that doesn’t exist, an app with no way to start a run, a display widget that never receives anything, a result wired to a variable that was never declared, a button pointing at an operation the app doesn’t have, and a run that hit its time limit. It also warns about gaps that still run: a Run button with no disabledWhen on the running state, an input with no default, no widget showing an operation’s progress or error, an Output node no widget displays, and an operation no event runs. It follows your conditions too: a widget hidden or disabled by visibleWhen or disabledWhen can’t be clicked, so a script that tries reports the condition that blocked it, and a Run button no state ever makes visible is an error. A widget with a format template is reported as the template renders it. Resources work too, from a collection you seed. Give the run the items a picker, gallery, or scene list would show, and an input that comes from that resource is sent, a create/update/delete button changes the collection, and the report lists what it holds at the end. A run whose input reads a collection nothing seeded fails and tells you how to seed it. npm run dev:nodetool -- app debug <id> \ --interact '[{"seedResource":{"id":"boards","items":[{"id":"b1","name":"Opening"}]}},{"click":"Run"}]' npm run dev:nodetool -- app debug <id> --params '{"resource:boards":[{"id":"b1"}]}' What it can’t check: how the app looks — layout, styling, focus — and your stored resources, since the run reads the seeded collection rather than the database, and has no editor for openResource to open. Building an app from a prompt nodetool app build writes the app for you and checks it with the same harness. It plans one workflow per operation, places and wires the widgets, checks the wiring, replays every interaction, and asks a judge model whether each interaction did what was asked. It needs a provider and model. npm run dev:nodetool -- app build "an app that drafts a note from a prompt" -p anthropic -m claude-sonnet-5 npm run dev:nodetool -- app build spec.json -p openai -m gpt-5.4-mini --no-judge It writes an ApplicationBundle and a build report, and exits 0 only when the verdict is ok. The flags (--workflow, --max-repairs, --cost-cap, --judge-model, --watch) are in the harness reference. To debug the workflow itself, nodetool validate is the quick check and nodetool debug is the full run. See Workflow Debugging. Related Building Mini Apps — recipes, step by step Mini App Reference — widgets, bindings, schema App Builder — the editor Workflow Editor — building the graphs underneath Key Concepts — how workflows, assets, and apps fit together--- # Mobile App Source: https://docs.nodetool.ai/mobile-app.html Run Mini-Apps, chat with models, and follow your jobs from your phone or tablet. The mobile app is a companion to desktop and web: you build and edit workflows there, and run and review them here. It connects to any NodeTool server, whether your desktop, a self-hosted instance, or NodeTool Cloud. New here? Start on desktop with Getting Started. You must sign in. The app opens on a Login screen with Continue with Google (Supabase auth). A build without EXPO_PUBLIC_SUPABASE_URL and EXPO_PUBLIC_SUPABASE_ANON_KEY shows a banner saying login is not configured. Overview Feature Notes Mini-Apps The home screen. Run apps published on the server Chat Streaming responses across providers, image and video generation modes, and inline previews of sketches and timelines Jobs Review job history. A finished run sends a notification while the app is in the background Assets Browse, upload from the camera or photo library, and view Documents Edit storyboards. View timelines and sketches Desktop and web only Workflow editing, scripts, JS scripts, managing API keys, collections, and triggers, and editing timelines or sketches Platforms iOS, Android, browser Server Connect to any NodeTool server Getting the App From App Stores (Coming Soon) The app will be available on: iOS: Apple App Store Android: Google Play Store For Developers Build and run from source: # Clone the NodeTool repository git clone https://github.com/nodetool-ai/nodetool.git cd nodetool/mobile # Install dependencies (mobile/ is not a root workspace) npm install # Start the development server npm start Then: Press i for iOS Simulator (macOS only) Press a for Android Emulator Press w for web browser Scan the QR code with a development build on your phone Expo Go does not work. The app uses native modules that Expo Go does not ship (Google Sign-In, Sentry, speech recognition), so sign-in and voice input fail there. Build a development build with npm run build:dev. See mobile/README.md for the TestFlight route. Connecting to Your Server The mobile app requires a running NodeTool server. Configure Server URL Open the app Go to Settings (gear icon on the Workflows screen) Enter the URL in API Host Tap Test & Save to check the server and save the URL in one step, or Save Only to skip the check The button reports the result: Server Ready & Saved, Sign In Required, Connection Timed Out, Incompatible Server, or Network Error The default host is http://localhost:7777. The URL must start with http:// or https://. Server URLs by Platform Platform Server URL iOS Simulator http://localhost:7777 Android Emulator http://10.0.2.2:7777 Physical Device http://<your-computer-ip>:7777 Physical devices must be on the same network as your NodeTool server. AI Chat Chat with AI models from your mobile device. Features Streaming responses – See text appear in real-time Model selection – Choose from available AI models Markdown rendering – Code blocks and formatting Stop generation – Tap the stop button to cancel a response Multiple threads – Keep separate conversation topics and reopen them from the threads list Attachments and voice – Attach files, take or pick photos, and dictate with voice input Modes – Switch the composer between Chat, Image, and Video. Image and video modes expose aspect ratio, resolution, and variation or duration pickers Options bar – Toggle Agent and Help modes, add tools, and choose collections to search Inline previews – When a reply names a sketch or a timeline, it draws in the message with a chip that opens it. Other documents show the chip alone How to Chat Tap the Chat button in the Apps header Select a model (tap model name) Type your message Tap the send button (up arrow) Watch the AI respond in real-time Tips Use the + button to start a new conversation Use the history button to open past conversations. Each thread can be deleted there Tap the stop button to halt a response Switch models mid-conversation if needed Running Mini Apps The home screen is the Apps list. Mini Apps are a separate resource, authored in the desktop App Builder and stored on the server. Tapping an app runs it on its own screen. See Mini Apps on Mobile. The Apps header opens Chat, Documents, Jobs, Assets, and Settings. Workflows are not edited on mobile. Build and edit them in the desktop or web app, and run them here through a Mini App. Running a Mini App Tap an app on the Apps screen Fill in the inputs Tap Run View results as they stream in Supported Input Types Mini App inputs map from the workflow’s Input nodes to native controls: Kind Input nodes Text String, Text, Message Numbers Integer, Float Boolean Boolean toggle Media Image, Video, Audio, Document, Model3D Lists Image, Video, Audio, and Text lists Paths File path, Folder path, Folder Other Color, DataFrame, Select, Image size Models Language, image, video, TTS, ASR, embedding, and Hugging Face model inputs Documents The Documents screen lists your storyboards, timelines, and sketches. Kind On mobile Storyboard Edit shots and the board by touch, or ask the assistant Timeline View the tracks and clips. Edit in the desktop or web app, or ask the assistant. The viewer reloads when you come back to it Sketch View the layers Other documents, such as scripts, open in the desktop or web app. Mobile Settings Configure the mobile app from the gear icon: Section Purpose Appearance Light, Dark, or System theme Server Connection API Host with Test & Save and Save Only Account Signed-in email and Sign Out About App version and the GitHub repository Mobile Language Model Selection Tapping the model name at the top of a chat opens a two-step picker: Select a provider — the providers your server reports as supporting message generation. Select a model — the models offered by that provider. A search box appears in either step when the list has more than five entries, and a back arrow returns from models to providers. There is no API-key gating, disabled styling, or docs links in this picker. Screens The stack registers these screens: Login, Apps, App, Settings, Chat, Threads, Language Model Selection, Documents, Storyboard Editor, Timeline Viewer, Sketch Viewer, Assets, Asset Viewer, Jobs, and Job Detail. Deep links The app registers the nodetool:// scheme. Links work only while signed in. Finished runs send a local notification whose link opens the job. Link Opens nodetool:// or nodetool://apps Apps nodetool://chat/<threadId> Chat nodetool://threads, documents, assets, jobs, settings The matching screen nodetool://app/<applicationId> A Mini App nodetool://job/<jobId> Job detail nodetool://asset/<assetId> Asset viewer nodetool://document/<kind>/<id> Storyboard editor, timeline viewer, or sketch viewer (storyboard, timeline, sketch) nodetool://settings/models Model picker Server Requirements Your NodeTool server must be: Running – Start with nodetool serve --host 0.0.0.0 --port 7777 Accessible – On same network as your device Configured – With the models you want to use Starting the Server nodetool serve --host 0.0.0.0 --port 7777 nodetool serve binds to 127.0.0.1 on port 7777 by default. A physical device cannot reach that, so pass --host 0.0.0.0 to listen on your network. The iOS Simulator and Android Emulator reach the default binding through the addresses in the table above. Firewall Settings If connecting from a physical device, ensure: Port 7777 is open on your computer’s firewall Your router allows local network connections Both devices are on the same WiFi network Troubleshooting Cannot Connect to Server Symptoms: “Connection failed” or timeout errors Solutions: Verify the server is running Check the server URL in Settings For physical devices: Use your computer’s IP address (not localhost) Ensure same WiFi network Check firewall settings Android Emulator Connection Issues Problem: Cannot reach localhost Solution: Use http://10.0.2.2:7777 instead of localhost:7777. This is Android’s special IP for the host machine. App Crashes on Startup Solutions: Clear app data and restart Check that all dependencies are installed: npm install inside mobile/ Reset Metro bundler: npx expo start --clear Chat Not Streaming Symptoms: Responses appear all at once Solutions: Check WebSocket connection Verify server is running current version Try a different AI model Platform Notes iOS Requires Xcode for development (macOS only) Use iOS Simulator for testing Production builds via EAS Build Android Requires Android Studio for development Use Android Emulator for testing Use 10.0.2.2 for localhost access Web Works in any modern browser Good for testing without mobile device Run with npm run web Building for Production Use Expo’s EAS Build for production apps: # Install EAS CLI npm install -g eas-cli # Log in to Expo eas login # Build for Android eas build --platform android --profile preview # Build for iOS eas build --platform ios --profile preview mobile/package.json wraps these as npm run build:dev, build:preview, and build:production, each for all platforms. The profiles in mobile/eas.json are development, development-simulator, development-testflight, preview, preview-simulator, and production. For store submissions: # Google Play Store (AAB format) eas build --platform android --profile production # Apple App Store eas build --platform ios --profile production See EAS Build Documentation for details. Related Topics Getting Started – Desktop setup and first workflow User Interface – Full UI guide Mini Apps – What apps are and how they run Mini Apps on Mobile – Opening and running an app on the phone Chat & Agents – Chat features in detail API Reference – Server API documentation--- # Models & Providers Source: https://docs.nodetool.ai/models-and-providers.html Overview / start here. For the full catalog of model families see Supported Models; for connecting and configuring each provider see Providers; for the desktop download panel see Models Manager. Run models locally, through cloud APIs with your own keys, or both in the same graph. Local vs. cloud Local Data stays on your disk Free after download Works offline 4–20 GB per model; needs a capable GPU or Apple Silicon Cloud (BYOK) No download Runs on the provider’s hardware Latest model releases Billed by the provider, at the provider’s price — no NodeTool markup Requires internet; data goes to the provider Mixed Pick the best provider per node: ASR (Whisper) — local for sensitive audio Image generation — Flux locally for control, FAL/KIE cloud for speed Document processing — local for confidential files Cloud models NodeTool reaches thousands of cloud models across 30+ providers, all through the same generic nodes. The catalog below is grouped by what you’re generating — text, images, video, speech and music, 3D, embeddings. Every model runs on your own API key (BYOK): you’re billed by the provider at the provider’s price, with no NodeTool markup. Add a key in Settings → Models & Providers and the models show up in the node’s model dropdown. Most chat providers fetch their model list live from the provider’s API, so the list reflects new releases. A few use fixed lists, such as kie.ai’s chat gateway and Meta AI’s offline fallback. Image, video, audio, and 3D models come from manifests or static lists that each provider ships (packages/runtime/src/providers/ and the packages/*-nodes/ packages), so they track what NodeTool exposes. The model names below are examples and drift as providers release. Text & chat models (LLMs) The generic nodetool.agents.Agent and chat nodes route to whichever provider owns the model you pick. Swap claude-opus-5 for gpt-5.6 or gemini-3.1-pro-preview without changing the rest of the graph. Provider Model families OpenAI GPT-5.6 (plus the Sol, Terra, and Luna variants), GPT-5.5, GPT-5.5 Pro, GPT-5.4 with its Pro/mini/nano tiers, GPT-5, GPT-5 mini, GPT-5 nano Anthropic Claude Fable 5, Claude Opus 5, Claude Sonnet 5, Claude Opus 4.8/4.7/4.6, Claude Sonnet 4.6, Claude Haiku 4.5 Google Gemini Gemini 3.5 Flash, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite, Gemini 2.5 Pro xAI Grok 4, Grok 3, Grok Code DeepSeek DeepSeek-V3, DeepSeek-R1 (reasoning) Groq Llama, Qwen, GPT-OSS, Kimi K2 on LPU inference Mistral Mistral Large, Mistral Medium, Mistral Small, Mixtral, Codestral, Magistral Cerebras Llama, Qwen, GPT-OSS on wafer-scale inference Alibaba Cloud Qwen3-Max, Qwen-Plus, Qwen-Flash, Qwen-Turbo, Qwen3-VL GMI Cloud Llama, DeepSeek, Qwen (open-weight) Moonshot Kimi K2, Kimi latest Meta AI Muse Spark MiniMax MiniMax M2.7, M2.5, M2.1, M2 (plus high-speed variants) OpenRouter 300+ models proxied through one key (Claude, GPT, Gemini, Llama, Qwen, DeepSeek, …) Requesty Claude, GPT, Gemini, DeepSeek, Grok and more through one OpenAI-compatible router API Route Hosted Claude, GPT, Gemini and DeepSeek chat models through an OpenAI-compatible API Opper Claude, GPT, Gemini, DeepSeek, Kimi and more through one EU-hosted OpenAI-compatible gateway Together AI Llama, Qwen, DeepSeek, Mixtral, GLM, Kimi, and more open models Evolink GPT, Claude, Gemini, DeepSeek through one gateway key kie.ai GPT, Claude, Grok, Kimi, DeepSeek, and Gemini models through its chat gateway Codex (OpenAI OAuth) GPT chat models via your ChatGPT/Codex login Claude Agent SDK Claude via your local claude CLI subscription Ollama · vLLM · LM Studio · llama.cpp · llama.cpp local Any local open-weight model served by your own engine (Llama, Qwen, Gemma, DeepSeek, Mistral, Phi, GPT-OSS) HuggingFace The most-liked warm inference models on the Hub, through Inference Providers FAL Chat through fal’s OpenAI-compatible route (OpenRouter’s catalog) Reasoning, tool calling, and vision input are exposed where the provider supports them. For local text generation without an API key, see Supported Models (Ollama, llama.cpp, MLX, Transformers.js). Image generation models Generate and edit images through nodetool.image.TextToImage and nodetool.image.ImageToImage. Flagship models Model Provider Capabilities Key features Black Forest Labs FLUX.2 BFL T2I with control Photoreal images, multi-reference consistency, accurate text, flexible control Google Nano Banana 2.0 / Pro Google High-res T2I/Edit 2K output, 4K upscaling, better text and character consistency GPT Image 2 OpenAI T2I/Edit Photorealistic generation and instruction-based editing Ideogram V3 Ideogram T2I/Edit Typography rendering and artistic style control Seedream 4.5 ByteDance T2I/Edit High-fidelity generation and instruction-based editing Z-Image Turbo Z-AI T2I Fast generation with strong prompt adherence Full catalog by provider Provider Image models OpenAI GPT Image 2, GPT Image 1.5, GPT Image 1, GPT Image 1 mini Google Gemini 3.1 Flash Image (Nano Banana 2), Gemini 3 Pro Image (Nano Banana Pro) Black Forest Labs FLUX.1 [schnell], FLUX.1 [dev], FLUX.1 Pro, FLUX 1.1 Pro, FLUX 1.1 Pro Ultra, FLUX.2 [dev/flex/pro/max], FLUX.2 Klein, FLUX Kontext [pro/max], FLUX Fill, FLUX Canny, FLUX Depth, FLUX Redux, FLUX PuLID Stability AI Stable Diffusion 1.5, SD 2.1, SDXL, SDXL Lightning, Stable Diffusion 3 Medium, SD 3.5 [medium/large/large turbo], Stable Cascade Qwen Qwen Image, Qwen Image Edit (2509/2511/Plus), Qwen Image Max, Qwen Image Layered ByteDance Seedream 3.0, Seedream 4.0, Seedream 4.5, Seedream 5 Lite Ideogram Ideogram V2, V2 Turbo, V2a, V3 [balanced/quality/turbo], Ideogram Character Recraft Recraft V3, Recraft 20B, Recraft V4, Recraft V4 Pro (raster + SVG) Others Kolors, HiDream, Hunyuan Image 3, OmniGen v1/v2, Sana, PixArt-Σ, Aura Flow, CogView4, Luma Photon, GLM Image, Reve, Bria, ERNIE Image, Emu 3.5, Lumina, F-Lite, Fibo, Playground v2.5, Juggernaut, DreamShaper, Proteus, Recraft, Chroma, Janus, MiniMax Image-01, xAI Grok Imagine Upscale & restore Topaz, Real-ESRGAN, ESRGAN, SUPIR, Clarity Upscaler, SeedVR, GFPGAN, CodeFormer, AuraSR, SwinIR, DDColor Access aggregators (FAL, Replicate, kie.ai, and AtlasCloud) carry most of these families, plus community fine-tunes, LoRAs, and control variants. Video generation models Generate video through nodetool.video.TextToVideo and nodetool.video.ImageToVideo. Flagship models Model Provider Capabilities Key features OpenAI Sora 2 Pro OpenAI T2V/I2V up to 15s Realistic motion, refined physics, synchronized native audio, 1080p Google Veo 3.1 Google T2V/I2V with references Multi-image references, extended length, native 1080p with synced audio ByteDance Seedance 2.0 ByteDance T2V/I2V Cinematic video with stable characters and smooth motion Runway Gen-4 / Aleph Runway T2V/I2V/Extend Precise motion control and high fidelity Luma Ray 2 Luma AI T2V/I2V/edit Fast generation and creative editing xAI Grok Imagine xAI T2V/I2V/T2I Coherent motion with synchronized audio Alibaba Wan 2.6 Alibaba Multi-shot T2V/I2V Affordable 1080p, stable characters, native audio MiniMax Hailuo 2.3 MiniMax High-fidelity T2V/I2V Expressive characters, complex motion and lighting Kling 3.0 Kling T2V/I2V with audio Synchronized speech, ambient sound, and effects Full catalog by provider Provider Video models OpenAI Sora 2, Sora 2 Pro Google Veo 3.1, Veo 3.1 Fast, Veo 3.1 Lite, Gemini Omni Flash Kling Kling 2.1 [standard/pro/master], Kling 2.5 Turbo Pro, Kling 2.6, Kling 3.0, Kling Avatar, Kling Lip Sync ByteDance Seedance 1.0 [lite/pro], Seedance 1.5 Pro, Seedance 2.0 Alibaba Wan 2.1, Wan 2.2, Wan 2.5, Wan 2.6, Wan VACE MiniMax Hailuo 02, Hailuo 2.3, Video-01 [director/live] Runway Gen-3 Alpha, Gen-4, Gen-4 Turbo, Gen-4.5, Aleph Luma AI Dream Machine, Ray 2 [540p/720p], Ray Flash 2 PixVerse PixVerse v4, v4.5, v5, v5.6, v6 Vidu Vidu 2.0, Vidu Q1 Others LTX Video, LTX-2, Hunyuan Video (+ v1.5, Avatar, Foley), Mochi v1, CogVideoX-5B, Marey, Lucy, Magi, Kandinsky 5, Sana Video, SkyReels, Stable Video Diffusion, Zeroscope, Pika, xAI Grok Imagine Lip sync & avatars LatentSync, Lipsync 2 [pro], MultiTalk, OmniHuman, SadTalker, Live Portrait, Stable Avatar, DreamActor, InfiniteTalk Upscale Topaz Video, SeedVR, Crystal Video Upscaler Speech, audio & music models Generate speech through nodetool.audio.TextToSpeech, transcribe with nodetool.text.AutomaticSpeechRecognition, and create music through provider music nodes. Text-to-speech (TTS) Provider TTS models OpenAI TTS 1, TTS 1 HD, gpt-4o-mini-tts ElevenLabs Eleven v3, Turbo v2.5, Flash v2.5, Multilingual v2 Google Gemini TTS, Gemini 3.1 Flash TTS MiniMax Speech 02 [hd/turbo], Speech 2.6, Speech 2.8 Open models Kokoro, Orpheus, Chatterbox [HD/Pro/Turbo/Multilingual], Dia, F5-TTS, XTTS v2, VibeVoice, Zonos, CSM-1B, Index-TTS 2, Qwen 3 TTS, Maya, StyleTTS2, Parler-TTS, Tortoise, VoiceCraft, OpenVoice, Cartesia Sonic Speech recognition (ASR) Provider ASR models OpenAI Whisper Google Gemini audio transcription ElevenLabs Scribe (speech-to-text) Together AI Whisper Large v3, Voxtral, Parakeet HuggingFace Whisper and other ASR models, plus MLX Whisper locally Music & sound Provider Music & sound models Suno (via kie.ai) Suno — song generation, extend, cover, remix ElevenLabs V3 Dialogue, Sound Effects MiniMax Music 01, Music 1.5, Music 2.6 Google Lyria 3.5, Lyria 3 Clip Open models MusicGen, Stable Audio 2.5, ACE-Step, DiffRhythm, YuE, Riffusion, Flux Music, MMAudio, ThinkSound 3D generation models Generate 3D assets through nodetool.model3d.TextTo3D and nodetool.model3d.ImageTo3D. Model Provider Capabilities Key features Meshy AI Meshy (MESHY_API_KEY) T2M/I2M Textured mesh generation Rodin AI Rodin (RODIN_API_KEY) T2M/I2M High-fidelity 3D creation AtlasCloud also lists 3D models. Open 3D families such as Hunyuan3D and Trellis run through provider-specific FAL and Replicate nodes, not the generic nodes. See Providers for details. Embedding models Power RAG and semantic search through embedding nodes. Provider Embedding models OpenAI text-embedding-3-small, text-embedding-3-large, ada-002 Google gemini-embedding-2, gemini-embedding-001 Mistral mistral-embed Cohere embed-v4.0, embed-english-v3.0, embed-multilingual-v3.0 Voyage AI voyage-3.5 and the Voyage line Jina AI jina-embeddings-v3 Together AI m2-bert, BGE, and other open embedding models Ollama · llama.cpp local · Transformers.js Embedding models that run on your machine with no key (Transformers.js downloads ONNX models on first use) vLLM · OpenRouter · Replicate · HuggingFace Any embedding model the server or provider lists Using these models Access these models through NodeTool’s generic nodes: For Video: Use nodetool.video.TextToVideo or nodetool.video.ImageToVideo For Images: Use nodetool.image.TextToImage For 3D: Use nodetool.model3d.TextTo3D or nodetool.model3d.ImageTo3D For Music: Use the kie.ai Suno nodes (Generate Music, Extend Music, Upload And Cover Audio, Add Vocals, and others) Select Provider: Click the model dropdown in the node properties Configure API: Add provider API keys in Settings → Models & Providers Access via kie.ai (recommended for broad model support): Many of these models are available through kie.ai, an AI provider aggregator that often offers competitive or lower pricing compared to upstream providers. Configure using KIE_API_KEY in Settings → Models & Providers Access via fal.ai: Configure using FAL_API_KEY in Settings → Models & Providers Cost Considerations: Cloud models typically charge per generation. Check each provider’s pricing before extensive use. Local models are free after download but require capable hardware. Getting Started Option 1: Start with Local Models (Recommended) Open the logo menu and choose Model Manager On an empty install it opens on Get Started, which detects your hardware and lists current models that fit it. Pick a chat model and an image model. Download them. Progress shows under Downloads in the logo menu. Run templates that use those models. They work offline. Option 2: Start with Cloud Providers Get an API key from a provider: OpenAI – GPT, Whisper, Sora Anthropic – Claude models Google – Gemini models In NodeTool, go to Settings → Models & Providers Paste your API key Select the provider when using AI nodes Detailed Guides General Models Manager – Download and manage AI models Getting Started – First workflow Local AI Supported Models – List of local models (llama.cpp, MLX, Whisper, Flux) Cloud AI Providers Guide – Set up OpenAI, Anthropic, Google HuggingFace Integration – Access 500,000+ models Advanced ComfyUI – Run an exported ComfyUI workflow as a node, locally, on a GPU worker, or on Comfy Cloud Self-Hosted Deployment – Secure deployments Deployment Guide – Cloud infrastructure--- # Models Manager Source: https://docs.nodetool.ai/models-manager.html Local model panel. The Model Manager downloads and manages models that run on your machine or on an attached worker. For the model catalog see Supported Models. For provider keys see Providers. The Model Manager lists the local models NodeTool can run, lets you search the Hugging Face Hub, and downloads what you pick. Cloud models from OpenAI, Anthropic, and other API providers do not appear here because they need a key, not a download. Add keys in Settings → Models & Providers. Opening the Manager Open the logo menu and choose Model Manager. It opens as a page tab at the /models route. The page has four sources, switched with the toggle at the top right. Source What it shows Get Started Detects your hardware, explains the local engines, and lists current models sized for it Installed Models on disk: Hugging Face repos, Transformers.js models, and local runtimes (Ollama, llama.cpp, MLX) Recommended A curated catalog gathered from the nodes you have installed Hub A live search of the Hugging Face Hub, limited to the top 50 results by downloads On an empty local install the Manager opens on Get Started once, so you do not land on a blank list. Get Started The hardware card shows what NodeTool detected and a memory budget, which you can leave on Auto-detect or set from 4 GB to 48 GB. The engine guide covers Ollama, llama.cpp, Transformers.js, Hugging Face / Diffusers, and MLX (Apple Silicon only), and shows which ones you still need to install. Ollama is a separate download from ollama.com that must be running. The others install from the Package Manager. The model list below it can be filtered by capability: chat, vision, image generation, speech to text, text to speech, and embeddings. Entries that fit your budget sort first. Sizes and memory figures are approximate. Local and worker scope When a worker is attached, a second toggle switches between Local and the worker’s name. The Worker view lists models cached on the worker. While a worker is attached, every download goes to the worker, whichever view is open, because workflows run there. The header shows Downloads → <target> as a reminder. If the worker image is too old for model management, the Worker option is disabled. Browsing Models Categories The left sidebar, Model Categories, filters by model type with a count for each. Hugging Face types follow Hub pipeline tags such as text-generation, text-to-image, image-to-image, text-to-video, automatic-speech-recognition, text-to-speech, and feature-extraction. Each Hugging Face category has a View on Hugging Face link to the Hub’s list for that tag. Only categories present in the current list appear. Ollama and llama.cpp models group under their own types. In the Hub source the sidebar shows the full list of pipeline tags. Filters The filter bar above the list narrows the results. Filter Options I want to Chat & agents, Create images, Understand images, Transcribe speech, Generate speech & audio, Create video, Search & RAG Format GGUF, ONNX, Safetensors, PyTorch, TensorRT, MLX Status (Installed source only) All, Ready, Download required, Unavailable, each with a count Max size A slider from 0 to 50 GB. 0 means all sizes Selecting an active goal or format chip again clears it. Changing the source or the Local/Worker scope resets the search and all filters. Search and sort Search matches model names and repository ids. Sort by Best fit, Name, Size, Downloads, or Likes. The arrow button reverses the direction. Models that need a download or are unavailable always sort after ready ones. Reading a model row Each row can show these badges. A status badge with the reason on hover, for example when a runtime the model needs is not installed. A fit badge: Fits your machine, Tight fit, or Needs ~N GB. NodeTool estimates it from the model’s size against the memory budget on the Get Started tab. Works with N nodes, which opens a dialog listing the nodes that can use the model. The pipeline tag, which links to trending models with that tag on Hugging Face. Downloading Models Find the model in Get Started, Recommended, or Hub. Click Download. Follow progress in the Downloads dialog, opened from Downloads in the logo menu. The menu entry shows the combined percentage while downloads run. Download details Downloads continue in the background while you move around the app. The Downloads dialog shows each file’s progress. The download connection reconnects automatically after a drop, up to 5 attempts with exponential backoff. Gated Hub repos need a Hugging Face token. See HuggingFace Integration. Storage location Hugging Face models use the standard Hugging Face hub cache, ~/.cache/huggingface/hub by default. NodeTool resolves it the way the huggingface_hub library does: HF_HUB_CACHE, then HUGGINGFACE_HUB_CACHE, then $HF_HOME/hub, then $XDG_CACHE_HOME/huggingface/hub. Downloads, the Installed list, the downloaded badges, llama.cpp, whisper.cpp, and the Python worker all read the same directory. On Linux with XDG_CACHE_HOME set, as in the Flatpak build, the desktop app keeps using ~/.cache/huggingface/hub when it already holds models. The full rule is under HF_HUB_CACHE. Ollama keeps its models in its own directory. Transformers.js models download to <data-dir>/transformers-js-cache, or to TRANSFORMERS_JS_CACHE_DIR if set. On the desktop app, the Downloads dialog has Open HuggingFace folder and Open Ollama folder buttons. Managing Models Per-model actions Download fetches a model that is not on disk yet. Downloaded marks an installed model. When the Manager is used as a picker, a ready model shows Use instead. Copy copies the repo id, or the model name for Ollama. Show in File Explorer opens the model folder. It appears in the desktop app only. Delete asks for confirmation (Confirm Deletion), then removes the model. Hugging Face models are removed from the cache of the Local or Worker scope you are viewing, and Ollama models are removed through Ollama. View on HuggingFace and View on Ollama open the model’s page on that site, where you can read its README. Recommended models Many workflow nodes name recommended models. The Recommended source gathers them from the nodes you have installed and lists the ones you can download. If you have no nodes that run models, the list is empty. Choosing a model in a node A model property opens a picker for its type. The title says which: Select Language Model, Select Image Model, Select Video Model, Select TTS Model, Select ASR Model, Select Embedding Model, Select Music Model, Select Audio To Audio Model, Select HuggingFace Model, or Select Transformers.js Model. The picker lists only models of that type, with a provider rail on the side. Provider icons show which providers have a key. A provider without one shows API key required and a Connect this provider action. A provider can be disabled to hide its models from the picker. The star marks a favorite. The pin sets the model as the default for new nodes of that type. Cloud provider models Models from cloud providers appear in these pickers based on your configured keys. They run remotely and need no download. Configure keys in Settings → Models & Providers. See Models & Providers for setup details. Next Steps Models & Providers – Configure providers and API keys Installation – Hardware requirements for local models HuggingFace Integration – Browse and use HuggingFace models--- # Supported Models Source: https://docs.nodetool.ai/models.html The model catalog. For how to connect a provider, see Providers. For the desktop app’s download panel, see Models Manager. New here? Start with Models & Providers. NodeTool runs models from many providers, proprietary and open. Generic nodes (TextToImage, Agent, and others) work across providers, so swapping a model doesn’t change the graph. Local inference engines The engines below run on your machine. The Model Manager’s Get Started tab lists them with the formats each one loads and matches model suggestions to your hardware. For engines that run as a separate server (Ollama, vLLM, LM Studio, llama.cpp llama-server), see the Providers documentation. Ollama Ollama pulls and runs GGUF chat and embedding models by name. It is a separate program: download it from ollama.com and keep it running. NodeTool does not ship or start it. NodeTool connects to http://127.0.0.1:11434, or to the OLLAMA_API_URL setting. Models you pull appear in NodeTool automatically, and the Model Manager lists them under the llama_model type. llama.cpp & GGUF Format llama.cpp is a C/C++ inference library for LLMs on CPU and GPU using the GGUF format. It supports 1.5-bit through 8-bit integer quantization for lower memory use. NodeTool reaches it two ways. The llama.cpp provider talks to a llama-server you run. The llama.cpp local provider (node-llama-cpp) runs GGUF files inside the NodeTool backend, with no server. Install it as a runtime from the Package Manager. Transformers.js Transformers.js runs small ONNX models (embeddings, speech, classification, text generation) in-process through ONNX Runtime. It works on any platform with no Python. Install it as a runtime from the Package Manager. See HuggingFace Integration. MLX Framework (Apple Silicon) MLX is Apple’s open-source machine learning framework, optimized for Apple Silicon’s unified memory. It runs through the Python worker and needs the optional MLX pack: in the desktop app, install MLX from Tools → Package Manager → Python packs, which sets up Python first if it is missing. The Package Manager offers the pack only on Apple Silicon Macs, and it needs macOS 14 or newer. Capabilities: LLMs: Native optimization for Llama, Qwen, Mistral, and others. Vision: Multimodal models and FastVLM support. Image Gen: FLUX models ported to MLX for faster generation. HuggingFace Transformers Transformers and Diffusers are the Python libraries behind the optional nodetool-huggingface node pack, which you install from the Package Manager. They run Hub models locally on GPU, Apple Silicon, or CPU on PyTorch 2.14. See HuggingFace Integration and GPU requirements. Wan2GP Wan2GP runs Wan video models in a server you start yourself. The optional Wan2GP pack calls it over MCP, so no model loads inside NodeTool. See Wan2GP. Comparison Matrix Framework Formats Best Hardware Use Case Ollama GGUF CPU, GPU Simplest local chat models llama.cpp GGUF CPU, GPU Quantized models, edge devices Transformers.js ONNX Any Small models, no Python MLX MLX Apple Silicon Mac on-device models Transformers / Diffusers Safetensors, PyTorch Any Research, flexibility Supported Model Types NodeTool supports a wide range of model types across different domains. Below is an overview of the supported types and their available execution variants. Variants Key Full Precision: Standard execution using HuggingFace Transformers/Diffusers (supports CUDA, MPS, CPU). MLX: Optimized execution for Apple Silicon (M-series chips). Image Generation Model Type Description Variants Flux Text-to-image generation ✅ Full Precision✅ MLX Flux Fill Inpainting/Outpainting for Flux ✅ Full Precision✅ MLX Flux Depth Depth-guided generation ✅ Full Precision✅ MLX Flux Redux Image variation and mixing ✅ Full Precision✅ MLX Flux Kontext Context-aware generation ✅ Full Precision✅ MLX Stable Diffusion XL SDXL base and refiner models ✅ Full Precision Stable Diffusion 3 Latest Stable Diffusion architecture ✅ Full Precision Stable Diffusion SD 1.5, 2.1, and variants ✅ Full Precision Qwen Image Qwen-based text-to-image ✅ Full Precision✅ MLX Qwen Image Edit Instruction-based image editing ✅ Full Precision✅ MLX ControlNet Structural guidance (Canny, Depth, etc.) ✅ Full Precision✅ MLX (Flux) Text to Image Generic text-to-image models ✅ Full Precision Image to Image Image transformation models ✅ Full Precision Inpainting Mask-based image editing ✅ Full Precision Vision & Video Model Type Description Variants Image Text to Text Vision-Language Models (VLM) ✅ Full Precision✅ MLX (Qwen2-VL) Visual QA Visual Question Answering ✅ Full Precision Document QA Document understanding and QA ✅ Full Precision OCR Optical Character Recognition (GOT-OCR, etc.) ✅ Full Precision Depth Estimation Monocular depth estimation ✅ Full Precision Image Classification Categorize images ✅ Full Precision Object Detection Detect objects in images ✅ Full Precision Image Segmentation Pixel-level segmentation ✅ Full Precision Zero-Shot Detection Open-vocabulary detection ✅ Full Precision Mask Generation Segment Anything (SAM) variants ✅ Full Precision Video Classification Categorize video content ✅ Full Precision Text to Video Generate video from text ✅ Full Precision Image to Video Animate images ✅ Full Precision Text to 3D Generate 3D assets from text ✅ Full Precision Image to 3D Generate 3D assets from images ✅ Full Precision Natural Language Processing Model Type Description Variants Text Generation LLMs (Llama, Qwen, Mistral, etc.) ✅ Full Precision✅ MLX Text to Text T5, BART, and seq2seq models ✅ Full Precision Summarization Text summarization ✅ Full Precision Translation Machine translation ✅ Full Precision Question Answering Extractive QA ✅ Full Precision Text Classification Sentiment analysis, etc. ✅ Full Precision Token Classification NER, POS tagging ✅ Full Precision Zero-Shot Class. Open-vocabulary classification ✅ Full Precision Sentence Similarity Semantic similarity / Embeddings ✅ Full Precision Reranker Search result reranking ✅ Full Precision Feature Extraction General embeddings ✅ Full Precision Fill Mask BERT-style masked modeling ✅ Full Precision Audio Model Type Description Variants Text to Speech Generate speech from text ✅ Full Precision✅ MLX Speech Recognition ASR (Whisper, etc.) ✅ Full Precision✅ MLX Audio Classification Categorize audio events ✅ Full Precision Voice Activity VAD (Silero, etc.) ✅ Full Precision Audio to Audio Voice conversion, enhancement ✅ Full Precision Components & Adapters Model Type Description Variants LoRA Low-Rank Adaptation weights ✅ Full Precision (SD, SDXL, Qwen) IP Adapter Image Prompt Adapters ✅ Full Precision VAE Variational Autoencoders ✅ Full Precision CLIP Text/Image Encoders ✅ Full Precision T5 Encoder Text Encoders for diffusion ✅ Full Precision RealESRGAN Image Upscaling ✅ Full Precision Cloud Models In addition to local models, NodeTool provides access to cloud-based models through provider integrations. These models offer the latest capabilities in video, image, and audio generation. Video Generation (Cloud) Model Provider Key Features Resolution Max Duration Sora 2 Pro OpenAI Realistic motion, refined physics, native audio 1080p 15s Veo 3.1 Google Realistic motion, multi-image refs, synced audio 1080p Extended Seedance 2.0 ByteDance High-quality cinematic video, stable characters 1080p Variable Runway Gen-3 Alpha Runway Precise motion control, professional fidelity 1080p Variable Runway Aleph Runway Next-gen Runway video generation 1080p Variable Luma Luma AI AI-powered video modification and editing 1080p Variable Grok Imagine xAI Multimodal T2V/I2V with coherent motion 1080p Short clips Wan 2.6 Alibaba Multi-shot, stable characters, affordable 1080p Variable Hailuo 2.3 MiniMax Expressive characters, complex lighting 1080p+ Variable Kling 3.0 Kling Synced speech & effects, audio-visual coherence 1080p Variable Access via: nodetool.video.TextToVideo, nodetool.video.ImageToVideo nodes Image Generation (Cloud) Model Provider Key Features Output Quality FLUX.2 Pro Black Forest Labs Photoreal, multi-reference consistency, accurate text High Nano Banana 2.0 Google 2K native, 4K scaling, enhanced text & characters Very High GPT Image 2 OpenAI Photorealistic generation and instruction-based editing High Ideogram V3 Ideogram Exceptional typography, artistic style control High Z-Image Turbo Z-AI Fast generation with strong prompt adherence High Seedream 4.5 ByteDance High-fidelity generation and instruction-based editing High Access via: nodetool.image.TextToImage node Music & Audio Generation (Cloud) Model Provider Key Features Suno Suno Full song creation from text, extend/cover/remix, instrumental support ElevenLabs V3 Dialogue ElevenLabs Multi-speaker dialogue with emotional control ElevenLabs TTS Turbo 2.5 ElevenLabs Ultra-fast, natural text-to-speech ElevenLabs Sound Effect ElevenLabs Generate sound effects and ambient audio from text Access via: nodetool.audio.TextToSpeech for speech. ElevenLabs has its own provider and elevenlabs.* nodes. Suno runs through the kie.ai nodes. Advantages of Cloud Models Latest Technology: Access to newest architectures and training data No Local Resources: Run on any hardware without GPU requirements Instant Availability: No download or installation needed Continuous Updates: Models improve without local updates Considerations API Costs: Per-generation pricing varies by provider Internet Required: Cannot run offline Data Privacy: Content is processed on provider servers Rate Limits: Subject to provider API quotas Where the cloud models come from Most models above are offered by more than one provider, and the set changes as providers add models. The generic nodes list what your configured providers serve, so open the node’s model dropdown to see your options. The aggregators (kie.ai, FAL, Replicate, AtlasCloud, Evolink, OpenRouter) carry many of these families under one key. Prices differ between an aggregator and the upstream provider, so compare them before committing to a large run. For provider configuration and the capability matrix, see the Providers Guide.--- # Real-Time Collaborative Workflow Editing (“Multiplayer”) Source: https://docs.nodetool.ai/multiplayer-design.html Real-Time Collaborative Workflow Editing (“Multiplayer”) Author: Matti Georgi Status: Draft — for review Last updated: 2026-07-10 Reviewers: TBD Implementation plan: multiplayer-implementation-plan.md Summary NodeTool’s workflow editor is single-writer. Two people who open the same workflow silently overwrite each other; the only concurrency mechanism we have is a coarse “this resource changed, refetch it” push that discards the loser’s work. This doc proposes making the workflow canvas multiplayer: shared live editing with presence (cursors, selections), per-user undo, and convergent conflict resolution, built on a CRDT (Yjs) synchronized over our existing WebSocket server and persisted in our existing database. The central architectural decision is that the CRDT is an implementation detail of the editor, not a new source of truth for the platform. The workflow’s materialized graph JSON remains the interface for the runner, the REST/tRPC API, the CLI, version history, export, and validation. Everything outside the editor keeps working unchanged. That constraint is what makes this project a bounded, reversible investment rather than a rewrite. Estimated cost: ~2 engineer-weeks for Phase 0 (presence + write-conflict safety), ~7 engineer-weeks for Phase 1 (CRDT co-editing). Phase 0 ships value on its own and is a prerequisite for nothing; if Phase 1 is cancelled, Phase 0 stands. 1. Context NodeTool is a visual AI workflow platform: a React/ReactFlow editor over an actor-model execution kernel, deployed both as a desktop app (Electron, local single-user server) and as a hosted service (Fly.io, api.nodetool.ai). Over the last several quarters the platform grew the full perimeter of a collaborative product: Hosted multi-user deployment with auth and public/private workflow access (packages/websocket/src/trpc/routers/workflows.ts). Version history with manual and autosave snapshots (packages/models/src/schema/workflow-versions.ts, web/src/components/version/). Sharing surfaces: public workflows, mini-apps, portable .nodetool bundles. A live push channel: the server observes model changes and pushes resource-change events over WebSocket; the client invalidates TanStack Query caches (web/src/stores/resourceChangeHandler.ts). What is missing is the center of that perimeter: two people cannot work on the same canvas. Saves are last-write-wins. There is no presence, no awareness, no merge. The resource-change channel makes this worse in one respect: user B’s screen refetches and discards their in-progress edits when user A saves. AI workflows are collaborative artifacts by nature — a prompt author, a domain expert, and an engineer iterating on the same graph is the normal working mode, not the exception. Every adjacent category has already internalized this (Figma for design canvases; n8n shipped collaborative editing for workflow graphs). For a product with a hosted offering, multiplayer is not a feature among features; it is the difference between a tool and a platform. 1.1 Current editing pipeline (what we’d be changing) Each open workflow gets its own Zustand store created by createNodeStore (web/src/stores/NodeStore.ts), holding ReactFlow nodes/edges. Undo/redo is client-local via the zundo temporal middleware. Saves go through tRPC mutations to the workflows router, which writes the whole graph JSON column. No optimistic-concurrency check: the write path accepts whatever graph the client sends. Version snapshots (save_type: manual | autosave) are rows in nodetool_workflow_versions keyed by workflow. All realtime traffic (runs, chat, resource changes) flows through one WebSocket (/ws, Fastify + @fastify/websocket, MsgPack frames) behind the client’s GlobalWebSocketManager singleton. These are good bones. The transport, auth, persistence, and versioning layers all exist; we are adding a synchronization layer, not infrastructure. 2. Goals G1 — Convergent co-editing. N users edit the same workflow concurrently; all replicas converge to the same graph without lost updates. Target p95 remote-edit visibility < 500 ms on the hosted deployment. G2 — Presence. Live cursors, selections, and a “who’s here” roster, with join/leave latency < 2 s. G3 — Per-user undo. Undo reverts my operations only, never a collaborator’s. G4 — Zero blast radius outside the editor. No changes to the execution kernel, REST/tRPC API shapes, CLI, DSL export, bundles, validation, or mini-apps. The materialized graph JSON remains authoritative for all of them. G5 — Runs on our actual infrastructure. SQLite and Postgres schema parity, single-node Fly deployment, self-hostable with no new external services. G6 — Reversible rollout. Feature-flagged; disabling the flag loses no data and returns to today’s behavior. Non-goals Collaborative text editing inside prompt fields (character-level merge via Y.Text). Phase 2 candidate; v1 treats each property value as an atomic register. Comments, annotations, follow-mode, emoji reactions. Presence primitives make these cheap later; they are product work, not sync work. Offline-first editing with deferred merge. Desktop users editing purely local workflows are out of scope; collab requires a live connection to the owning server. (The CRDT gives us most of the machinery for this later; we are deliberately not paying the UX cost now.) Horizontal scale-out of the sync server. We run one Fly machine. The design documents the scale-out path (§7.4) but does not build it. Shared execution sessions. Runs stay per-user (per-user generations, per-user results). Broadcasting run status to collaborators is cheap and included; sharing run outputs across users touches asset ownership and is deferred. A new permissions system. We reuse access: public | private plus one additive collaborators table (§5.6). Org/team modeling is its own project. 3. Design overview One page, top to bottom: Document model. Each collaborative workflow has a Yjs Y.Doc: nodes and edges as Y.Maps keyed by id, node properties as nested Y.Maps with last-writer-wins leaf values. This mirrors the existing graph JSON one-to-one. Transport. A new MsgPack message family (collab_*) on the existing /ws socket: join/leave, Yjs sync protocol (state vectors + updates), and awareness (presence) frames. No new port, no new server process, no new auth path. Server. A CollabRoomManager in packages/websocket: one in-memory room per open workflow, holding the loaded Y.Doc, broadcasting updates to members, appending updates to the database, and periodically (a) compacting the update log into snapshots and (b) flushing the materialized graph JSON through the existing save path — which keeps versions, resource-change events, and every downstream consumer working untouched. Client. A YjsGraphBinding that two-way binds the Y.Doc to the existing per-workflow NodeStore. ReactFlow and every component above it are unaware anything changed. Undo/redo switches from zundo to Y.UndoManager (scoped to local origin) when a workflow is in collab mode. Persistence. Two additive tables: an append-only update log and periodic snapshots. Recovery = load latest snapshot + replay tail. The materialized JSON is always a valid fallback: worst case, we discard the CRDT state and re-seed it from workflow.graph, losing nothing but sub-second granularity. Phasing. Phase 0 — Safety + presence (no CRDT). Optimistic-concurrency check on workflow saves (reject stale writes instead of clobbering), presence roster + live cursors over an awareness channel, and a “someone else is editing” banner. Small, independently shippable, kills the silent-data-loss bug on day one. Phase 1 — CRDT co-editing. Everything above. Phase 2 (sketch only). Y.Text for prompt properties, comments, follow-mode, offline merge. ┌────────────┐ Yjs updates + awareness ┌─────────────────────────┐ │ Client A │◄────────────────────────────►│ /ws (existing socket) │ │ NodeStore │ │ CollabRoomManager │ │ ▲ binding│ │ ┌───────────────────┐ │ │ Y.Doc │ │ │ Room: workflow_id │ │ └────────────┘ │ │ Y.Doc (memory) │ │ ┌────────────┐ │ └───┬─────────┬─────┘ │ │ Client B │◄────────────────────────────►│ │ │ │ └────────────┘ └──────┼─────────┼────────┘ append│ │debounced flush ▼ ▼ crdt_updates/ workflow.graph JSON crdt_snapshots (unchanged interface: (new tables) runner, API, versions, export, validate, CLI) 4. Why a CRDT, and why Yjs The decision that shapes everything else. Stating the reasoning explicitly because it’s the part most likely to be re-litigated: A workflow graph under concurrent editing has exactly the conflict profile CRDTs are good at: a keyed collection of objects (nodes, edges) where concurrent operations mostly touch different keys (you move your node, I edit my prompt), and where the rare same-key conflict has an acceptable arbitrary-but-deterministic resolution (last writer wins on a property value). We do not need server-ordered intent preservation (OT’s strength, at the cost of a server that must serialize and transform every operation); we need convergence, offline-tolerant buffering during reconnects, and per-key merge — CRDT territory. Yjs specifically, over Automerge: Yjs is the ecosystem default (mature, MIT, ~10 years of production use, the engine behind most collaborative ReactFlow implementations), has an order-of-magnitude edge in update size and apply throughput in public benchmarks, ships the awareness protocol we need for presence as a sibling package (y-protocols), and — decisive for us — its sync protocol is transport-agnostic bytes we can frame in MsgPack on our existing socket rather than adopting a new server. Automerge’s nicer git-like history model buys us nothing here because version history already exists at the platform level. The considered-and-rejected alternatives are in §8. 5. Detailed design 5.1 Document model One Y.Doc per workflow. Top-level shared types: doc.getMap('nodes') // Y.Map<node_id → Y.Map> doc.getMap('edges') // Y.Map<edge_id → plain object (atomic)> doc.getMap('meta') // workflow-level fields editable in the canvas (name, description) Per node (Y.Map): Key Type Conflict semantics type, parent_id plain value LWW (rarely concurrent) position plain {x,y} value LWW, whole-point (concurrent drags of the same node: one wins; visually fine) width/height, ui state plain values LWW properties nested Y.Map<prop → value> LWW per property — the case that matters. A edits prompt while B edits temperature on the same node: both survive. Edges are atomic values, not nested maps: an edge’s identity is its endpoints, so “merging” two concurrent edits of one edge is meaningless — replace wholesale. Structural conflicts follow Yjs map semantics: concurrent delete-node vs. edit-node resolves to delete-wins (the map key is gone). The binding then garbage-collects edges referencing missing nodes inside the same transaction, preserving the graph invariant the kernel’s validateGraph expects. This is the one invariant the CRDT cannot express by itself and must be enforced in the binding on every remote transaction — it gets a dedicated property-based test suite (§7.3). Property values are the same JSON-serializable values stored in graph today, so Y.Doc ⇄ graph JSON conversion is mechanical and lossless in both directions (docFromGraph, graphFromDoc — pure functions, heavily unit-tested, shared between client and server via a new packages/collab workspace package). Not in the doc: run results, generations, node execution state, selection. Execution state stays in ResultsStore/LiveRunStore and flows through existing run channels; selection is presence (ephemeral), not document state. 5.2 Wire protocol New MsgPack frames on the existing /ws connection, multiplexed by workflow id. We reuse Yjs’s binary sync protocol as an opaque payload — we frame it, we don’t reimplement it. Message Direction Payload collab_join C→S {workflow_id} — authz check, room attach collab_joined S→C {state_vector, snapshot, members[]} collab_sync both {workflow_id, data: bytes} — y-protocols sync1/sync2 (state-vector exchange → diff) collab_update both {workflow_id, update: bytes} — incremental Yjs update; server rebroadcasts to other members and appends to log collab_awareness both {workflow_id, data: bytes} — y-protocols awareness delta (cursor, selection, viewport, user meta). Ephemeral: never persisted, never ordered. collab_leave C→S room detach; server broadcasts awareness removal Reconnect is the sync protocol’s normal case: the client keeps its Y.Doc, reconnects, exchanges state vectors, and receives/sends exactly the missed diff. Edits made while disconnected are buffered in the local doc and merge on reconnect — short-outage tolerance falls out of the design for free (distinct from offline-first, which we are not doing; the buffer lives only as long as the tab). Rate shaping at the client: position updates during drags are throttled to ~30 Hz and batched per Yjs transaction; awareness to ~15 Hz. Yjs batches transactions into single updates natively. 5.3 Server: CollabRoomManager Lives in packages/websocket, wired into the unified socket runner next to the existing message handlers. Room lifecycle. First collab_join for a workflow: load latest snapshot from workflow_crdt_snapshots, apply tail from workflow_crdt_updates; if neither exists (first-ever collab session), seed the doc from workflow.graph via docFromGraph. Last member leaves: flush, then evict after a 60 s grace period (fast rejoin skips the reload). On each update: apply to the room doc, rebroadcast to other members, append the raw update to workflow_crdt_updates (append-only insert — cheap, safe on SQLite and Postgres). Compaction. When a room’s log exceeds 500 updates (or on eviction), encode the doc as a snapshot row and delete the covered updates, in one transaction. Back-of-envelope: a heavy session is ~10 updates/s sustained ⇒ compaction roughly every minute under load; snapshot for a 500-node workflow is O(100 KB); the log never grows unbounded. Materialized flush. Debounced (2 s idle / 15 s max), the room writes graphFromDoc(doc) through the existing workflow save path. Consequences, all intentional: The ModelObserver → resource-change event fires as today, so non-collab clients (CLI, another user’s read-only view, mini-apps) see updates at the granularity they already expect. Autosave version snapshots keep accruing in nodetool_workflow_versions with no changes to that system. A run started mid-session sees a graph at most ~2 s stale — identical to today’s autosave behavior, and the run submit path can force a flush first (§5.7). Authority. While a room is live, the room doc is authoritative and direct graph saves from collab-mode clients are suppressed (the binding replaces the save path). A non-collab writer (CLI workflows run-adjacent tooling, older client) that writes workflow.graph while a room is live is detected via the Phase 0 optimistic-concurrency token; the room responds by rebasing: diffing the incoming JSON against graphFromDoc(doc) and applying the delta as a Yjs transaction. This is the ugliest corner of the design; it is bounded, and the alternative (locking out the CLI) breaks G4. 5.4 Client: YjsGraphBinding A per-workflow object owned by WorkflowManagerStore, created when a workflow opens in collab mode. Responsibilities: Doc → store: on remote transactions, translate changed keys into the minimal NodeStore mutation (add/remove/patch node or edge). ReactFlow, properties panels, and everything above the store are untouched — they already re-render from store changes. Store → doc: the store’s mutation actions (add node, move node, change property, connect edge) additionally write into the Y.Doc inside a transaction tagged with a local origin. Origin tagging is what prevents echo loops (remote transactions don’t re-enter the doc) and is what scopes undo. Undo/redo: Y.UndoManager tracking only local-origin transactions replaces the zundo temporal store while in collab mode. This is a real behavior change (undo history no longer survives a reload; a collaborator deleting the node you just edited makes your undo a no-op) — both are the industry-standard semantics and match user expectations from Figma/Google Docs. Presence rendering: awareness states drive remote cursors (canvas overlay), selection highlights (colored node outlines), and the avatar roster in the editor header. Each user gets a stable color derived from user id. Single-user and local-desktop workflows do not create a binding and keep today’s path bit-for-bit, including zundo. Yes, this is a dual code path (risk §9); the seam is one object with a narrow interface, and Phase 1 exit criteria include e2e coverage of both paths. 5.5 Presence & awareness y-protocols/awareness, payload per user: {user_id, name, color, cursor: {x,y} | null, selection: node_ids[], viewport?}. Ephemeral by construction — 30 s timeout evicts crashed clients, nothing persisted, nothing in the doc. Presence ships in Phase 0, before the CRDT: the awareness channel has no dependency on shared document state, and “I can see you’re here, and where” removes most accidental-conflict pain even before merges are automatic. 5.6 Permissions & sharing Reuse the existing model, one additive table: workflow_collaborators (workflow_id, user_id, role: editor | viewer, invited_by, created_at) — SQLite and Postgres schemas, additive Drizzle migration. Edit (join room, send updates): owner + editor collaborators. Observe (join room read-only, receive updates + awareness, send only awareness): viewer collaborators; optionally public workflows (flag-gated — live-observing a public workflow is a great demo and a real load consideration; default off at launch). Authorization enforced at collab_join and per collab_update server-side (a viewer socket sending an update is dropped and logged). All room traffic rides the already-authenticated socket. Invite UX (email/link) is product work layered on this table; out of scope for this doc beyond the schema. 5.7 Interaction with execution & caching The rule: runs consume the materialized JSON; the CRDT never touches the kernel. Run submit from a collab client forces a room flush, then follows today’s path. The kernel WorkflowRunner, actors, and the debug/validate CLI see an ordinary graph. Per-user run state is unchanged. Run status (node running/completed) is additionally mirrored into the initiating user’s awareness state so collaborators see “Anna is running this branch” — cheap, ephemeral, no new channels. The editor’s cached partial-run machinery (runSubgraph.ts, computeRunSignatures.ts, generations) already keys reuse off input signatures of current graph state; remote edits dirty signatures exactly like local edits. No changes, but this interaction gets explicit e2e coverage because it’s subtle: a collaborator editing an upstream node must invalidate my “run from here” cache, and signature hashing makes that automatic. 5.8 Version history Unchanged and load-bearing. Autosaves continue via the materialized flush; manual “save version” works from any collaborator. Version restore while a room is live is implemented as a room-level operation: server rebases the restored graph onto the doc as one transaction (same mechanism as §5.3 external-writer rebase), so a restore doesn’t fork the room. Versions remain our coarse-grained, human-meaningful history; the CRDT update log is fine-grained machinery, not a user-facing timeline (per-user attribution of changes is recorded in update metadata, so a future “who changed this node” feature has the data it needs). 5.9 Failure modes Failure Behavior Server restart / deploy Rooms are memory; clients reconnect, rooms lazily reload from snapshot+log. In-flight updates not yet appended are re-sent by clients via sync (client docs hold full state). Worst case loss: none, by protocol. Client crash mid-edit Their unflushed local ops are lost (same as today); doc consistent. Awareness timeout clears their cursor. Divergence bug (binding or protocol defect) Detection: clients periodically send a state-vector hash; mismatch → client discards local doc and re-syncs from server (server wins), incident logged with both docs’ encoded state attached. Corrupt/unloadable CRDT state Drop snapshot+log for that workflow, re-seed from workflow.graph. Loses at most the last flush interval (≤15 s) of granularity, never the graph. This is the design’s ultimate safety property and the reason G4 exists. Log growth (compaction failing) Alert at 10× threshold; room degrades to read-only presence rather than unbounded writes. 6. Rollout Flag: collab (server setting + per-workspace override). Off = today’s behavior exactly; Phase 0’s stale-write rejection ships behind its own sub-flag first since it changes save semantics for racing writers (from “silently clobber” to “second writer gets a conflict prompt” — strictly better, but a visible change). Phase 0 (≈2 eng-weeks): optimistic-concurrency token on workflow saves; awareness channel; presence UI (roster, cursors, “N others editing” banner). Dogfood on the team’s own hosted workspace. Phase 1 (≈7 eng-weeks): packages/collab (doc model + conversions, property-based tests), server room manager + persistence + compaction, client binding + undo swap, permissions table, e2e suite. Dogfood ≥2 weeks; exit criteria: zero divergence incidents, p95 edit latency <500 ms, both editor code paths green in CI. GA: default-on for hosted; self-hosters get it via normal upgrade; desktop-vs-cloud unchanged (local workflows never enter collab mode). Kill switch: disabling the flag stops room creation; materialized JSON is current to within the flush interval, so rollback is a no-op for data. Observability: OTel spans for room lifecycle and sync (collab.room.load, collab.update.apply); metrics: concurrent rooms, members/room, update rate, broadcast fan-out latency, compaction duration, divergence count (this one pages), flush lag. All through the existing tracing stack (NODETOOL_TRACE_FILE et al.). Testing: property-based convergence tests on the doc model (random concurrent op interleavings ⇒ identical graphFromDoc output, graph invariants hold); fuzzed delete-vs-edit races for the edge-GC invariant; multi-context Playwright e2e (two browsers, one workflow: edit propagation, cursor rendering, conflict cases, undo isolation, reconnect); chaos tests reordering/dropping/duplicating update frames at the transport seam. 7. Scale & capacity (back of envelope) Hosted deployment today: one Fly machine, hundreds of users, tens concurrent. Numbers for 50 concurrent rooms × 4 members, heavy editing (10 updates/s/room, ~150 B/update): broadcast egress ≈ 50 × 10 × 3 × 150 B ≈ 225 KB/s — noise. Memory: 50 loaded docs × O(1 MB) worst case ≈ 50 MB — fine. DB: 500 inserts/s worst-burst append-only — fine on Postgres; on SQLite, batched in the room’s write loop (single writer per process anyway). CPU: Yjs applies are microseconds. The binding constraint is rooms are process-local, so multiplayer requires all members of a room on one machine. Single machine today ⇒ non-issue. The documented scale-out path, when needed: consistent-hash workflow_id → machine (Fly-Replay header routing), which preserves the room-per-process model without a Redis/pubsub layer. Explicitly not built now. 8. Alternatives considered A. Per-node locking (“check out” a node) + today’s refetch. Simple, and some enterprise tools live with it. Rejected: locks rot (crashed clients, forgotten sessions), pessimistic UX at exactly the granularity people collide least, doesn’t solve the whole-workflow save race, and builds no foundation for anything in Phase 2. We’d pay UX cost forever for a dead end. B. Rebroadcast Zustand actions (naive event sync). Tempting because the store already exists. Rejected: no convergence guarantee under reorder/loss/reconnect — this is the architecture that produces ghost edges and split-brain canvases, and then you rebuild a worse CRDT case by case. Listed because someone always proposes it. C. Operational Transformation. Proven (Docs), but requires the server to serialize and transform every op with per-type transform functions — high-complexity server code for a graph type nobody has a battle-tested OT library for. CRDTs give convergence with an off-the-shelf, transport-agnostic library. Rejected on build-cost and risk. D. Automerge instead of Yjs. Credible; better history ergonomics. Rejected on update-size/throughput (order-of-magnitude in public benchmarks), smaller production footprint for canvas apps, and because its git-like history duplicates what our version system already provides. E. Hosted sync service (Liveblocks, PartyKit, etc.). Fastest path on the hosted product. Rejected on a product invariant: NodeTool is self-hostable (desktop, docker-compose, packages/deploy), and a collaboration feature that only works on our cloud — or that ships customer workflow content to a third party — breaks that promise. Yjs costs us the room manager (§5.3), which at our scale is small. F. CRDT as the sole source of truth (drop the JSON graph). Architecturally purer; rejected emphatically. It couples the kernel, API, CLI, exports, and versioning to the CRDT library, makes rollback a migration instead of a flag flip, and turns a 9-week project into a rewrite. The materialized-view pattern (§5.3) is the load-bearing decision of this doc. 9. Risks & open questions Dual editor code path (collab vs. solo binding, Y.UndoManager vs. zundo) is the top maintenance risk. Mitigation: narrow seam, both paths in CI. Open question for post-GA: converge solo mode onto a local-only Y.Doc to delete the fork. External-writer rebase (§5.3) is the fiddliest logic. Mitigation: it degrades safely (worst case equals today’s last-write-wins for that one write) and is fuzz-tested. ReactFlow render cost under remote-update storms (many nodes patched at 30 Hz). Mitigation: transaction batching + rAF coalescing in the binding; perf budget test with a 500-node graph and a simulated 4-editor storm. SQLite write contention on busy self-hosted single-file DBs. Mitigation: batched appends, compaction thresholds; needs measurement in dogfood, not speculation. Open: do public workflows allow anonymous observers at GA, or authenticated-only? (Load and abuse surface vs. shareability. Default: authenticated-only, revisit with data.) Open: awareness on mobile (mobile/ app is read-mostly today) — roster yes, cursors probably meaningless. Decide during Phase 0 UI work. Appendix A — Schema sketch (Drizzle, additive) workflow_crdt_updates (id, workflow_id, seq, update BLOB, user_id, created_at) index (workflow_id, seq) workflow_crdt_snapshots (id, workflow_id, seq_through, snapshot BLOB, created_at) index (workflow_id, seq_through desc) workflow_collaborators (workflow_id, user_id, role, invited_by, created_at) pk (workflow_id, user_id) Mirrored in schema/ and schema-pg/; migration is additive only — no changes to workflows or nodetool_workflow_versions. Appendix B — Glossary CRDT — Conflict-free Replicated Data Type: data structures whose concurrent edits merge deterministically without coordination. Awareness — Yjs’s ephemeral presence side-channel: per-client state (cursor, selection) that is broadcast but never part of the document. State vector — compact summary of “which updates a replica has,” used to compute the minimal diff on (re)connect. LWW — last-writer-wins register semantics for a single value. Materialized flush — writing graphFromDoc(doc) into the existing workflow.graph column so non-collab consumers stay oblivious.--- # Implementation Plan: Real-Time Collaborative Workflow Editing (“Multiplayer”) Source: https://docs.nodetool.ai/multiplayer-implementation-plan.html Implementation Plan: Real-Time Collaborative Workflow Editing (“Multiplayer”) Author: Matti Georgi Status: Draft — for review Last updated: 2026-07-10 Design doc: multiplayer-design.md This plan turns the design into ordered, reviewable PRs with file-level targets, exit criteria, and a rollout checklist. Estimates assume one engineer full-time; workstreams marked ∥ can run in parallel with a second engineer. 0. Ground rules Every PR passes npm run check (typecheck + lint + test) and follows repo conventions: strict TS, no any, ui_primitives only (no raw MUI), design tokens, Zustand selectors, tests in __tests__/. New backend code lands in a new workspace package packages/collab plus surgical changes to packages/websocket, packages/models, and web/. No changes to packages/kernel, packages/runtime, or any node package — if a PR touches those, it violates design goal G4 and needs a design amendment first. All schema changes are additive, mirrored in packages/models/src/schema/ and schema-pg/, registered in packages/models/src/migrations/versions.ts, and covered in migrations.test.ts (the suite asserts tableExists). Feature flags via the existing settings registry (packages/websocket/src/settings-registry.ts → registerSetting): collab_occ (Phase 0 save safety), collab_presence (Phase 0 UI), collab (Phase 1 CRDT). Everything defaults off until the rollout checklist says otherwise. Phase 0 — Save safety + presence (no CRDT) · ~2 eng-weeks Ships standalone value: kills silent last-write-wins data loss and adds presence. Nothing in Phase 0 depends on Yjs. PR-1: Optimistic concurrency on workflow saves (~3 days) Server — packages/websocket/src/trpc/routers/workflows.ts: Extend updateInput (the update mutation, ~line 489) and autosaveInput (~line 706) with optional base_updated_at: string. When present and ≠ the row’s current updated_at, throw a tRPC CONFLICT (the error formatter already maps ALREADY_EXISTS/CONFLICT; add a distinct STALE_WRITE code in packages/websocket/src/error-codes.ts so clients can discriminate). Return the server’s current updated_at and graph in the error payload. Omitted token ⇒ today’s behavior (older clients, CLI keep working). Client — web/src/stores/WorkflowManagerStore.ts (saveWorkflow, ~line 314) and the autosave path: Track updated_at from the last successful load/save per workflow; send as base_updated_at; update it from every save response. On STALE_WRITE: conflict dialog (ui_primitives) with three actions — Reload theirs (refetch, discard local), Overwrite (resend without token), Keep both (save local graph as a manual version via the existing versions.create mutation, then reload). Autosave never overwrites: on conflict it silently snapshots local state as an autosave version and backs off until the user decides. Interplay fix (required, not optional): web/src/stores/resourceChangeHandler.ts currently invalidates ["workflows"] on every workflow change, which refetches and clobbers a dirty open editor. Gate that invalidation: if the changed workflow is open and dirty, surface the conflict banner instead of refetching. Tests: router unit tests (token match / mismatch / omitted); WorkflowManagerStore tests for token threading and each conflict action; regression test for the dirty-editor refetch clobber. Exit: two clients racing on one workflow can no longer silently lose work, with collab_occ on. PR-2: Awareness channel, server side (~2 days) New module packages/websocket/src/collab/presence-registry.ts: in-memory Map<workflow_id, Map<socket_id, AwarenessState>>, join/leave/update, 30 s stale eviction, broadcast to co-members. Wire three message types into the dispatch switch in packages/websocket/src/unified-websocket-runner.ts (the case block starting ~line 5849): collab_join, collab_leave, collab_awareness. MsgPack frames like every other message; no new socket, no new port. Authz on collab_join: reuse the access check pattern from the workflows router (access !== "public" && user_id !== ctx.userId ⇒ deny, plus collaborators table once PR-9 lands). Message schemas in packages/protocol/src/ (new collab.ts), since protocol is the shared-types package. Note: protocol changes require cd packages/protocol && npm run build before mobile typecheck — CI already handles this. OTel: collab.presence.join/leave events; gauge for members per workflow. Tests: registry unit tests (join/leave/evict/broadcast); an integration test in packages/websocket/tests/ driving two fake sockets through join → awareness → leave. PR-3: Presence UI (~4 days) ∥ after PR-2’s protocol types merge web/src/lib/collab/PresenceChannel.ts: thin client over globalWebSocketManager (web/src/lib/websocket/GlobalWebSocketManager.ts — singleton, per AGENTS.md never a new socket): join on workflow open (hook into WorkflowManagerStore open/close lifecycle), throttle outgoing cursor/selection to ~15 Hz, resend state on reconnect (subscribe to the manager’s reconnect event). web/src/stores/CollabPresenceStore.ts: Zustand store keyed by workflow id → member list with cursor/selection; selectors per member to keep re-renders scoped. UI (all ui_primitives + design tokens): Roster: avatar row in the editor header; stable per-user color derived from user id hash. Remote cursors: absolutely-positioned overlay inside the ReactFlow viewport (transform flow-coords → screen via ReactFlow’s viewport state; rAF-batched). Selection highlights: colored outline on nodes present in a remote member’s selection. Banner: “N others viewing/editing” using the existing notification/banner primitives. Own selection/cursor published from existing ReactFlow callbacks (onSelectionChange, pane onMouseMove throttled). Tests: store tests; RTL component tests for roster/banner; one Playwright two-context smoke (second browser’s cursor appears) — this bootstraps the multi-client e2e harness Phase 1 needs. Phase 0 exit / dogfood gate: flags on for the team workspace ≥1 week; zero clobbered-save reports; presence latency < 2 s observed. Phase 1 — CRDT co-editing · ~7 eng-weeks PR-4: packages/collab — document model (~1 week) ∥ with PR-2/3 New workspace package (depends only on @nodetool-ai/protocol; register in root workspaces and build:packages order — it has no decorators, so no dist/-loading caveats): src/doc-model.ts: shared-type layout (nodes/edges/meta maps per design §5.1); typed accessors. src/convert.ts: docFromGraph(graph): Y.Doc, graphFromDoc(doc): Graph — pure, lossless, the contract everything else leans on. src/invariants.ts: enforceGraphInvariants(doc, transaction) — edge GC for dangling endpoints after delete-vs-edit races, duplicate-edge collapse; runs inside the same Yjs transaction. src/protocol.ts: sync/awareness frame encode/decode wrapping y-protocols (opaque bytes in MsgPack fields). Tests (the heart of this PR): round-trip property tests (fast-check) over generated graphs: graphFromDoc(docFromGraph(g)) ≡ g; convergence property tests — N docs, random interleaved concurrent ops, sync pairwise ⇒ identical graphFromDoc output and invariants hold; targeted delete-vs-edit fuzz for edge GC. Budget half the PR’s time for these tests; they are the cheapest place this project can fail. Deps: yjs, y-protocols, fast-check (dev). Pin exact versions; both are pure-JS ESM (no native modules, no sandbox install pain). PR-5: Persistence — schema + migrations (~2 days) packages/models/src/schema/workflow-crdt.ts (+ schema-pg/ mirror): workflow_crdt_updates (id, workflow_id, seq, update blob, user_id, created_at; index workflow_id+seq) and workflow_crdt_snapshots (id, workflow_id, seq_through, snapshot blob, created_at; index workflow_id+seq_through desc). Export from both schema/index.ts files. workflow_collaborators (workflow_id, user_id, role, invited_by, created_at; pk workflow_id+user_id). Migrations create_workflow_crdt_tables, create_workflow_collaborators appended in packages/models/src/migrations/versions.ts with down-migrations, covered in migrations.test.ts. DAO packages/models/src/workflow-crdt.ts: appendUpdate, loadSince(seq), latestSnapshot, compact(seqThrough, snapshot) (transactional), collaboratorsFor(workflowId). PR-6: Server — CollabRoomManager (~2 weeks) packages/websocket/src/collab/room-manager.ts + room.ts, replacing PR-2’s presence registry internals (presence becomes room-scoped; wire protocol unchanged so PR-3’s client needs no changes): Lifecycle: first join loads snapshot + update tail via the DAO, else seeds from workflow.graph with docFromGraph; last-leave flush + 60 s grace eviction. Update path: authz (editor role) → apply to room doc → rebroadcast to other members → batched append to workflow_crdt_updates (single writer loop per room; batch inserts for SQLite friendliness). Sync: collab_join reply carries state vector + snapshot; handle sync1/sync2 frames via packages/collab protocol helpers. Compaction: threshold 500 updates or on eviction; snapshot + delete-covered-updates in one transaction. Materialized flush: debounced 2 s idle / 15 s max; write graphFromDoc(doc) through the same internal save routine the update mutation uses (so updated_at advances, the ModelObserver resource-change event fires, and autosave versions accrue). Tag the write with a room origin so the OCC check (PR-1) knows it’s the authority. External-writer rebase: when a non-collab write lands on a workflow with a live room (detected via the OCC token path), diff incoming JSON vs graphFromDoc(doc) and apply the delta as a Yjs transaction with a rebase origin. Divergence guard: on a client-reported state-vector-hash mismatch, send authoritative re-sync; count it; log both encoded states at debug level. Run-submit hook: the run_job path forces a flush for workflows with a live room before job creation. OTel spans collab.room.load, collab.update.apply, collab.flush, collab.compact; metrics per design §6. Tests: room unit tests (lifecycle, authz, batching, compaction transactionality); integration tests with two fake sockets (join → concurrent updates → convergence → flush lands in workflows table → resource event observed); rebase fuzz tests; restart simulation (kill room, rejoin, sync from persistence). PR-7: Client — YjsGraphBinding (~2 weeks) ∥ with PR-6 against a stub room web/src/lib/collab/YjsGraphBinding.ts, owned per-workflow by WorkflowManagerStore: Store → doc: wrap the NodeStore mutation surface — onNodesChange, onEdgesChange, onConnect, addNode, updateNode, updateNodeData, deleteNode(s), setNodes, setEdges (all in web/src/stores/NodeStore.ts) — mirroring each mutation into a Y transaction tagged localOrigin. Position drags coalesced to ~30 Hz, one transaction per frame. Doc → store: deep-observe handler translating remote transactions into minimal store mutations, entering the store through a remote: true path so wrapped actions don’t echo back into the doc, and marked to not dirty the workflow (the store already has a shouldn’t-dirty pattern, NodeStore.ts ~line 808) — collab edits must not trigger the legacy autosave. Save suppression: in collab mode, saveWorkflow/autosave become no-ops for the graph (server room flush is the writer); manual “save version” calls the versions mutation directly. Undo/redo: Y.UndoManager over nodes/edges/meta, trackedOrigins: {localOrigin}; the editor’s undo/redo entry points dispatch to it in collab mode, zundo temporal path untouched otherwise. Keyboard shortcuts unchanged. Sync client: join/sync/update/reconnect state machine over globalWebSocketManager; buffered local edits merge on reconnect; periodic state-vector hash for the divergence guard; on server-forced re-sync, rebuild store from doc. Mode selection: binding created only when collab flag is on and the workflow is server-backed and shared/multi-member-capable; pure-local desktop workflows never enter collab mode. Tests: binding unit tests against an in-memory Y.Doc pair (each store action round-trips; remote ops produce exact store mutations; echo prevention; undo isolation); store integration tests for save suppression and no-dirty remote edits; jsdom perf smoke for the 500-node update-storm budget. PR-8: Collaborative signals in the editor (~3 days) Run-status mirroring: initiating client publishes {running_nodes} into its awareness state; collaborators render the existing node-running affordance in the runner’s color. Cache interplay e2e: collaborator edits an upstream node ⇒ my “run from here” signature dirties (should be automatic via computeRunSignatures; this PR proves it with a test, changes nothing). Version restore while a room is live: route the versions.restore mutation through the room rebase path (PR-6 mechanism) instead of a raw graph write. PR-9: Permissions + sharing surface (~3 days) Enforce workflow_collaborators roles at collab_join (editor/viewer/deny) and per collab_update (viewer updates dropped + logged). tRPC mutations: collaborators.list/add/remove on the workflows router; minimal share dialog in the editor header (ui_primitives) — add by email/user id, role toggle. Public-workflow live observation stays behind its own flag, default off. PR-10: Multi-client e2e + chaos suite (~1 week, overlaps 6–9) Extend the Playwright harness from PR-3: two browser contexts, one workflow — edit propagation, both-edit-same-node property LWW, delete-vs-edit edge GC, undo isolation, reconnect-and-merge, presence rendering, conflict dialog (Phase 0 path with flag off). Chaos at the transport seam: inject a frame-mangling proxy (drop/reorder/duplicate collab_update) and assert convergence via state-vector hash. Perf budget test: 500-node graph, 4 simulated editors at full rate, assert frame time and store-update latency budgets. Sequencing Phase 0: PR-1 ──► dogfood ─────────────────────────────┐ PR-2 ──► PR-3 ─────────────► dogfood │ ▼ Phase 1: PR-4 ─┬─► PR-6 ─┬─► PR-8 ─► PR-9 ─► PR-10 ─► dogfood ─► GA │ │ PR-5 ─┘ PR-7 ─┘ (PR-6 ∥ PR-7 via stubbed counterpart) Total: ~9 eng-weeks serial; ~6–7 calendar weeks with two engineers (PR-3∥PR-4, PR-6∥PR-7). Rollout checklist Phase 0 dogfood: collab_occ + collab_presence on for the team workspace ≥1 week. Gate: zero lost-work reports, zero false-positive conflict dialogs. Phase 0 GA: collab_occ default-on everywhere (it is pure safety); presence default-on for hosted. Phase 1 dogfood: collab on for team workspace ≥2 weeks. Gates: zero divergence-counter increments; p95 remote-edit latency < 500 ms; flush lag p99 < 20 s; compaction keeping logs < 2× threshold; both editor code paths (collab/solo) green in CI. Phase 1 GA (hosted): default-on; self-hosters receive via normal upgrade (migrations are additive; flag still available as kill switch). Kill switch drill (before GA): disable collab mid-session in staging; verify clients fall back to solo mode, materialized JSON is current, no data loss. Monitoring to have in place before step 3: divergence counter (pages), room count / members gauge, update rate, broadcast latency histogram, flush lag, compaction duration, workflow_crdt_updates row-count alert at 10× threshold. Risk register (implementation-level) Risk Mitigation Where NodeStore has mutation paths not on the wrapped action list (bulk ops, timeline/sketch side stores writing nodes) Audit every set() touching nodes/edges in NodeStore.ts during PR-7; add a dev-mode assertion that the doc and store hash-match after each transaction PR-7 Legacy autosave fires in collab mode and double-writes Save suppression tested explicitly; server rejects graph writes from collab-mode clients (origin check) PR-6/7 resourceChangeHandler refetch fights the binding Collab-mode workflows opt out of workflow query invalidation (binding is the sync channel) PR-7 Yjs update floods from position drags degrade ReactFlow 30 Hz coalescing + rAF apply batching; perf budget test PR-7/10 SQLite write stalls under append load on self-hosted boxes Batched appends in room writer loop; measured in dogfood before GA PR-6 Protocol drift between web and mobile Frames defined once in packages/protocol; mobile ignores unknown message types today (verify in PR-2) PR-2 Explicit non-goals of this plan (deferred with the design doc) Y.Text prompt merging, comments/annotations, follow-mode, offline-first merge, horizontal room scale-out, anonymous public observers, org/team permissions.--- # Native game effect baseline Source: https://docs.nodetool.ai/native-game-effect-benchmark.html Native game effect baseline npm run benchmark:effects --workspace=packages/game-renderer renders one sprite at 512 × 288 through Dawn, waits for the GPU queue, and reports the median of ten frames after three warmups. The measured device was an Apple M4 Pro with Metal 4. This is a correctness and local cost baseline, not a frame rate promise for other devices or viewports. Chain Median frame work Effect target memory Disabled 0.31 ms 0 bytes One bloom 0.77 ms 3,481,600 bytes Eight blooms 3.96 ms 3,481,600 bytes The default bloom radius is 8 output pixels, threshold 0.7, softness 0.1, and intensity 1. Radius 8 costs less than 1 ms for one effect in this fixture while showing a visible halo. The recipe uses 8-bit textures and reusable scratch targets. It makes no HDR claim. The benchmark includes the viewport overscan needed for light just outside the visible area. Target memory is the renderer’s reported effect allocation, including recipe scratch; texture assets and the instance buffer are counted separately in renderer statistics. Point light baseline The same command benchmarks a scene with 32 point lights on the Apple M4 Pro at 512 × 288. Lights use a reusable RGBA16F texture in WebGPU so overlapping lights retain irradiance above 1 until the sprite color is multiplied. The GPU row waits for submitted work and reports the median of ten frames after three warmups. The CPU row measures the light calculation over the same viewport and light layout; its scratch allocation is released after each frame. Path Median frame work Light target or scratch memory WebGPU 0.57 ms 1,179,648 bytes target Canvas2D/headless CPU calculation 18.91 ms 1,769,472 bytes scratch The WebGPU timing includes the light texture pass and sprite draw. The CPU timing excludes canvas readback, sprite drawing, and PNG encoding. These measurements describe this fixture and device, not a frame rate guarantee.--- # PRD: Native AI Media Editing Source: https://docs.nodetool.ai/native-media-editing/prd.html PRD: Native AI Media Editing Status: Draft Baseline: main at 1d16ec8 Tasks: tasks.md Related: AI Timeline Editing, Video Editor, Media Generation Tracking, docs/timeline-editor-prd.md 1. Summary NodeTool will edit existing media through fixed actions on the media itself. The primary interaction is: Select a clip, choose an action, generate a candidate, audition it in the current cut, then apply it. The first release ships one complete action: Edit video. It works for an imported or generated video clip, sends only the source window used by that clip, keeps the accepted take playing while generation runs, stores the result as an inactive candidate, and applies it through one undoable Use take operation. This is not a workflow feature and does not add a Maestro mode. Workflows may remain part of a clip’s creation history, but the editing interaction neither creates nor exposes a graph. 2. Problem statement NodeTool can generate timeline media and can revise a storyboard clip through video_edit, but it does not provide one reliable editing interaction for any existing timeline clip. The current pieces disagree at important boundaries: useTimelineDirectGenJob activates a successful result immediately and clears its trim points. composeGenerativeTakePatch stores a generated result as an inactive take unless the caller explicitly activates it. ClipVersionHistory restores a take as soon as its tile is clicked. It does not distinguish auditioning from accepting. generateRevisedClip gives storyboards a direct video_edit path, but that path is separate from timeline generation and does not establish shared candidate semantics. packages/timeline/src/generative.ts plans several operations, but no product surface drives that planner today. A generative request does not yet describe the selected clip’s source window and time mapping. Editing a four-second excerpt can therefore target the wrong frames or apply an invalid old trim to a new four-second asset. The result is useful groundwork without a safe end-to-end edit. 3. Product principles D1. Editing belongs to the selected media. Imported footage, direct generations, and workflow-generated clips receive the same eligible actions. D2. Generation and editing stay separate. Generation describes how the original shot was made. AI Edit transforms the selected take. Takes stores the alternatives. D3. Every result starts as a candidate. Completion never changes the accepted cut. Applying a candidate requires Use take. D4. Preserve the cut, not old source-time numbers. Apply rules preserve what plays, where it plays, and the unrelated editorial state. They remap or replace source-time fields when the generated asset has a different source timeline. D5. The entry surface owns the destination. A timeline edit creates a take on one clip instance. A storyboard edit creates a candidate for one shot. A library edit creates a derived asset. D6. Actions filter models. The selected action determines the required task or capabilities. The model that created the source is only a default when it is eligible. D7. No substitute operation is silent. Image-to-video from the last frame is not video extension. Prompt restyling is not masked replacement. D8. Inspector and agent use the same operation. Both call the same validation, submission, landing, audition, and apply logic. D9. Changes stay local by default. Editing one timeline clip does not replace its source asset elsewhere or update its storyboard shot, entity, or voice definition. 4. Current state Existing component What exists Boundary for this PRD web/src/components/timeline/Inspector/DirectGenClipPanel.tsx Prompt, model, output settings, cost estimate, and direct generation controls. Reuse its controls and model catalog patterns. AI Edit sits outside the generation panel dispatcher. web/src/hooks/timeline/useTimelineDirectGenJob.ts Direct generate_media submission, persisted pending requests, generation lookup, recovery, and asset landing. Extract or extend the reusable lifecycle. Do not reuse its automatic activation and trim reset for edits. web/src/hooks/storyboard/useGenerateShot.ts generateRevisedClip submits mode: "video_edit" with the shot clip as source. Route storyboard revisions through the same editing request and candidate rules. Server generate_media video-edit branch Calls provider.videoToVideo() and stores the resulting asset. Keep this direct execution path for P0. Complete its input and capability validation contract. web/src/components/timeline/Inspector/ClipVersionHistory.tsx and packages/timeline/src/takes.ts List, select, rename, and delete ClipVersion takes. Extend this system with candidate audition and explicit acceptance. Do not create another history model. packages/timeline/src/generative.ts Plans extend, range replacement, object edits, restyle, and regenerate. composeGenerativeTakePatch() defaults to an inactive take. Reuse its provider-independent planning and candidate composition after completing source context and provider mapping. web/src/components/timeline/Inspector/ClipTracking.tsx Numeric initial region controls. Track subject is disabled because no provider is connected. Object and character actions remain unavailable until preview selection and tracking execution exist. The guide at docs/ai-timeline-editing.md describes several intended operations. It is not evidence that every operation is executable. 5. Users and surfaces 5.1 Timeline The selected clip inspector gains an AI Edit section. Eligible actions also appear in the clip context menu. This section is independent of bindingKind, sourceType, and workflow ownership. The existing concepts remain distinct: Inspector concept Meaning Generation Inputs and binding that created the original shot. AI Edit A transformation of the selected take. Takes Original, accepted, and candidate alternatives for this clip. The top-level Generate control continues to create media. Editing requires an explicit target and starts from a selected clip or cut. 5.2 Preview P0 adds candidate auditioning to the existing preview: The accepted take remains active while a request is queued, running, failed, or completed. Selecting a candidate for audition changes preview state only. It does not mutate the persisted clip. Original and candidate playback use the same relative position in the clip. Original, Candidate, and Use take controls are explicit. Closing the audition or selecting another clip discards preview-only state. Use take applies the candidate in one undo entry. Mask drawing, subject selection, and tracking overlays are not part of P0. 5.3 Storyboard Revise take uses the same validated video_edit submission and provenance contract. The result lands as a shot clip candidate. It does not replace the selected shot clip until the creator accepts it. 5.4 Asset library and other destinations Controls may be reused later, but each entry point has a fixed default result: Entry point Default result Timeline clip Candidate take for that clip instance. Storyboard shot Candidate for that shot. Asset library New derived asset. Existing references stay unchanged. Selected cut Transition or bridge candidate with cut-specific timing. Audio or dialogue clip Audio take or aligned replacement layer. P0 covers timeline clips and storyboard shots only. 6. Action model Action Interaction Result Release gate Edit video Select a clip, enter an instruction, choose an eligible model and optional model-supported strength. Candidate video take. P0, existing video_edit execution. New take Rerun known creation inputs, optionally with changed direction. Alternative generation from the recipe, not a pixel transformation. P1, only when original inputs are complete. Extend Choose start or end, added duration, and continuation intent. Longer source candidate plus an explicit timing choice. P1, genuine extension provider task required. Change line delivery Choose a script-linked voiced clip and direction. Speech take aligned to the line. P1, reuse Script cast and line data. Remove subject Select a preview region or subject. Masked and tracked edit candidate. P2, preview selection, mask preparation, tracking, and inpaint task required. Replace subject or character Select a subject, then an Entity, reference image, or instruction. Reference-conditioned masked edit candidate. P2, references must reach the provider call. Expand frame Choose target framing and generated sides. Spatially expanded candidate. P2, outpaint task required. Enhance or upscale Choose an eligible enhancement and target resolution. Higher-resolution candidate. Sequence dimensions stay unchanged. P2, enhancement task required. Generate sound Use clip timing and available scene context. Aligned audio layer or audio take. P2, video-conditioned audio task required. Change voice Choose another delivery or replace recorded speech. Speech take, replacement layer, or lip-synced candidate depending on the chosen operation. P2, isolation and lip-sync are separate capabilities. Generate transition Select a cut between adjacent clips and set overlap or duration. Cut-level transition candidate. P2, dedicated cut apply operation required. New take and Edit video never share a label. Imported footage offers Edit video, but does not offer New take unless NodeTool has an explicit reproducible generation recipe. The initial reference-conditioned production route and the production-flow definition of New take are specified in the AI Video Production PRD (docs/creation-flows/ai-video-production-prd.md). This document owns the independently shippable P0 Edit video path, including its source-window rules and candidate application semantics. 7. P0 requirements: Edit video 7.1 Eligibility The action is enabled when all of the following are true: Exactly one video clip is selected. The clip has an active stored asset. The clip has a positive playable duration. The source window can be represented as an ordinary constant-speed range. At least one available model supports the video-to-video task. P0 rejects reverse playback, speed ramps, discontinuous mappings, and other complex retiming with a message that explains the unsupported edit. It does not process a different range as a fallback. 7.2 Source context Submission captures an immutable source snapshot before the request is sent: interface MediaEditSourceContext { sequenceId: string; clipId: string; sourceAssetId: string; sourceTakeId?: string; sourceStartMs: number; sourceEndMs: number; timelineStartMs: number; timelineDurationMs: number; speedMultiplier: number; } For a clip that plays source seconds 40 through 44, the edit input contains that four-second window. If the server must materialize the range before calling a provider, the materialized asset becomes an intermediate generation input, not a replacement for the library source. The output contract states whether the result represents the selected window from zero or preserves source time. P0 expects a window-relative output whose playable range starts at zero. 7.3 Request and model selection The UI chooses video_edit before choosing a model. The model list contains only catalog entries that support the mapped video-to-video task. Cloud shows the managed default and a small eligible alternative list without provider selection. Local builds retain provider and model controls. An ineligible remembered model is replaced by the eligible default or produces a clear choice request when no default exists. Submission uses the existing generate_media lifecycle, project context, generation record, asset storage, spend handling, cancellation, and recovery. The request stores its action and source snapshot before dispatch. 7.4 Candidate landing and provenance The result is appended to the existing take list through composeGenerativeTakePatch() or a shared equivalent. Landing does not change currentAssetId, activeTakeId, trim fields, clip status that controls the accepted media, or timeline structure. Before the first edit of a clip whose active asset is not represented in versions, NodeTool records that active media as the baseline take. Each editing take records: Parent take and source asset. Source window and time mapping captured at submission. Action and submitted parameters. Instruction, references, provider, and model. Request or generation identifier. Result asset, duration, status, cost, and creation time. These values come from the submitted request snapshot. Landing never rebuilds provenance from mutable clip fields. 7.5 Audition and apply The candidate tile has a non-persisted Preview action and a persisted Use take action. Previewing at clip-relative time t reads candidate time t, clamped to the candidate duration. Applying a normal full-window edit: Sets the candidate as currentAssetId and activeTakeId. Preserves clip startMs, durationMs, track, effects, transform, animation, captions, links, mute state, and unrelated audio. Maps the generated window to source time zero, so inPointMs becomes 0 and outPointMs becomes the applied playable duration, or uses the repository’s absent-field convention when zero-to-duration is implicit. Refuses a result shorter than the current timeline duration unless the creator explicitly trims the cut or chooses another take. Keeps extra result duration as unused source handles. It does not lengthen the cut automatically. Runs as one undoable timeline store operation. Applying a take affects only the selected timeline clip. A separate command may later update the originating storyboard shot. 7.6 Failure, cancellation, and concurrency A failed, cancelled, expired, or recovered request leaves the accepted take and cut unchanged. The failed request remains inspectable through generation history and can be retried with its captured inputs. One clip can have at most one active AI Edit submission in P0. Switching sequences or selected clips while a request runs cannot redirect its result. Destination identifiers are captured at submission. Deleting the destination before completion settles the request without recreating the clip. The generated asset remains available through the generation record. 8. Shared execution boundary The implementation has five shared operations: Capture and validate the editing target and source context. Resolve eligible models and prepare provider inputs. Submit and recover a direct generation. Attach the result as a candidate with immutable provenance. Audition and apply the candidate with destination-specific rules. The timeline inspector, storyboard revision control, and agent bridge call these operations. They do not write asset or take fields independently. The existing direct-generation hook remains valid for creating new media. Its completion policy must not be used for editing because it activates the output and clears trims. 9. Agent behavior The timeline agent receives an explicit operation with sequence and clip targets. A request such as “make this station deserted at night” creates a candidate and returns its take and generation identifiers. It does not apply the candidate unless the request explicitly asks to use it. Agent execution uses the same eligibility checks, source-window capture, model filtering, request snapshot, landing, and apply operation as the inspector. Headless and browser-host execution must produce the same stored take. 10. Testing decisions Requirement Test boundary A trimmed imported clip sends only its playable source window. Pure source-context tests plus a fake-provider integration test around generate_media. Complex retiming is rejected before spend. Planner tests in packages/timeline/tests/. Completion adds an inactive candidate and preserves the accepted cut. Timeline store and direct-generation landing tests. The first edit preserves imported media as a baseline take. Timeline take composition test. Provenance reflects submission-time values after the clip is edited while a job runs. Delayed fake-provider test that mutates the clip before landing. Recovery lands the same candidate once. Pending-request reload and reconnect tests based on useTimelineDirectGenJob recovery coverage. Preview changes no document state. React Testing Library test around preview selection and ClipVersionHistory. Use take is one undo step and preserves unrelated clip fields. Timeline store reducer test with a fully populated clip fixture. Model selection contains only video-edit-capable models. Model-task matching hook test. Storyboard and timeline produce the same request and provenance fields. Shared request builder contract test plus each host adapter test. The agent creates a candidate through the same operation. Timeline tool bridge test and harness selfcheck. P0 adds a fake-provider vertical integration test that selects a trimmed imported clip, submits an edit, reloads before completion, lands the candidate, auditions it, applies it, undoes once, and verifies the original cut. 11. Success criteria P0 ships when: Any eligible imported or generated video clip exposes Edit video in the inspector and clip context menu. A creator can submit an instruction with a compatible model and see an estimate before generation where pricing is available. The provider receives only the clip’s selected source window. The accepted take keeps playing throughout submission and completion. The result appears as an inactive candidate with complete submission-time provenance. The creator can compare original and candidate at the same relative time. Use take applies the candidate without moving or lengthening the cut and one undo restores the prior take. Closing and reopening the sequence recovers a pending or completed request without duplicating the candidate. Storyboard Revise take follows the same request and candidate rules. The agent can request the same operation and returns a candidate without applying it by default. 12. Out of scope for P0 User-authored workflows, action marketplaces, and a Maestro mode. Library editing controls. New take, extension, partial-range replacement, and ripple decisions. Region drawing, masks, tracking execution, object removal, and character replacement. Entity-conditioned video editing until references reach the provider call. Frame expansion, enhancement, upscaling, transition generation, sound effects, speech isolation, voice conversion, and lip-sync. Automatic updates to storyboard media, other timeline instances, entities, voices, sequence format, or export dimensions. 13. Risks R1. Providers interpret duration and source ranges differently. Normalize a window-relative input before the provider boundary and test each supported adapter. R2. Preview-only state can leak into persistence. Keep audition state in the timeline UI or playback store, outside the document store. R3. Existing take clicks mutate the cut. Introduce explicit Preview and Use take commands before exposing candidates. R4. Mutable completion handlers corrupt provenance. Persist the request snapshot before dispatch and land from that snapshot only. R5. Documentation overstates capability readiness. Action visibility comes from executable provider tasks and prerequisites, not the planner’s vocabulary or documentation alone.--- # Native AI Media Editing Tasks Source: https://docs.nodetool.ai/native-media-editing/tasks.html Native AI Media Editing Tasks Companion: prd.md. Each phase leaves main working. Tasks are in dependency order inside a phase. After code changes run: npm run test:affected && npm run typecheck && npm run lint && npm run dev:nodetool -- harness gate --base origin/main. Delivery graph P0 ships Edit video for timeline clips, shared storyboard revision, and agent parity. P1 adds recipe-based New take, true extension, and script-aware voice revision. P1 starts after P0. P2 connects spatial selection, tracking, references, and the remaining fixed actions. Each action stays hidden until its provider task and apply rule are executable. P0: Edit video T01: Edit one trimmed clip into an inactive candidate. Add a shared media-edit request that captures sequence, clip, active take, source asset, playable source window, timeline placement, and constant speed before dispatch. Map video_edit to model task support, submit through generate_media, and land through composeGenerativeTakePatch without changing the active take or trims. Reject complex retiming before spend. Reuse pending-request persistence, generation lookup, cancellation, and reconnect recovery from web/src/hooks/timeline/useTimelineDirectGenJob.ts. Store the immutable request snapshot with the pending entry so landing does not read mutable clip fields. Likely touchpoints: packages/timeline/src/generative.ts, packages/protocol/src/api-schemas/timeline.ts, web/src/hooks/timeline/, packages/websocket/src/session/commands.ts, and packages/websocket/src/session/inference.ts. Tests: a fake-provider integration edits source seconds 40 through 44, mutates the clip before completion, reloads, and receives exactly one inactive take with the submitted provenance. (PRD § 7.1 to § 7.4) T02: Expose Edit video in the selected clip UI. Add an AI Edit section outside the generation-panel bindingKind dispatch in web/src/components/timeline/Inspector/TimelineInspector.tsx. Add the same action to web/src/components/timeline/Tracks/ClipContextMenu.tsx. Render instruction, eligible model, supported strength when present, estimate, submit, progress, cancel, retry, and actionable errors with existing inspector primitives. Imported and generated video clips follow identical eligibility rules. Cloud hides provider selection and local builds keep existing provider controls. Tests: imported, direct-generated, and workflow-bound clips expose the action; audio, missing media, complex retiming, and no-compatible-model fixtures explain why it is unavailable. Blocked by T01. (PRD § 5.1, § 7.1, § 7.3) T03: Preserve a baseline and audition candidates without changing the cut. Before the first edit, add the active imported or generated asset as a baseline take when no version represents it. Extend ClipVersionHistory with explicit Preview, Original, Candidate, and Use take controls. Store audition state outside the timeline document. Sync original and candidate preview at clip-relative time and clear audition state when selection changes. Tests prove preview does not call restoreVersion, change currentAssetId, or create an undo entry. Blocked by T01. (PRD § 5.2, § 7.4, § 7.5) T04: Apply a candidate as one editorial operation. Add a shared take apply helper and one TimelineStore action that sets currentAssetId and activeTakeId, maps the window-relative result to source time zero, and preserves placement, duration, track, effects, transform, animation, captions, links, mute state, and unrelated audio. Reject a candidate that cannot cover the current duration. Keep extra duration as unused handles. One undo restores the prior take and mapping. Route Use take through this operation. Tests use a fully populated clip and verify every preserved field plus undo and redo. Blocked by T03. (PRD § 7.5) T05: Make request lifecycle destination-safe. Persist destination sequence and clip identifiers with the submitted snapshot. A result cannot land on the sequence open at completion, recreate a deleted clip, or append twice after socket response and generation lookup race. Keep failed and cancelled requests inspectable without changing accepted media. Add reconnect, switched-sequence, deleted-clip, cancellation, and duplicate-settlement tests beside the current direct-generation recovery coverage. Blocked by T01. (PRD § 7.6) T06: Route storyboard Revise take through the shared edit path. Replace the private request construction in web/src/hooks/storyboard/useGenerateShot.ts with the P0 request builder and generation lifecycle adapter. Land the result in the shot’s existing clip-version system as an inactive candidate. Accepting it changes only that shot. Keep shot and timeline destination adapters separate while sharing validation, submission, and provenance. Tests compare the submitted request and take metadata from both hosts. Blocked by T01 and T04. (PRD § 5.3, § 8) T07: Give the timeline agent the same edit and apply operations. Add explicit target and instruction contracts to the existing timeline frontend tool and handler bridge. Default to returning a candidate take and generation id without applying it. An explicit apply call uses the same T04 operation. Update the protocol contracts, browser handler, headless timeline capability or gap table, and surface eval. Add a harness selfcheck that edits a trimmed clip, inspects the inactive take, applies it, then undoes once. Blocked by T04 and T05. (PRD § 9) T08: Add the P0 vertical acceptance journey. Run a fake-provider journey from an imported trimmed clip through inspector submission, reload recovery, candidate comparison, Use take, render selection, and undo. Assert no workflow is created and no other clip or storyboard shot changes. Register the journey with the timeline harness gate. Blocked by T02, T04, T05, T06, and T07. (PRD § 10, § 11) P1: Known-input alternatives T09: Generate New take from a known recipe. Derive a reproducible recipe from the selected take’s captured generation inputs. Show New take only when prompt, references, task, and required model settings are known. Submit a new generation rather than video_edit, land it as an inactive candidate, and reuse P0 audition and apply. Imported recordings without a recipe do not show the action. Blocked by T08. (PRD § 6) T10: Extend a clip with explicit timing choices. Connect the planner’s extend operation only to models with a genuine extension task. Choose start or end, duration, and direction, then create a longer candidate. Applying offers three explicit outcomes: keep the cut and retain source handles, extend into available space, or ripple later clips. Each outcome is one undoable timeline operation and refuses collisions it cannot resolve. Do not substitute last-frame image-to-video. Blocked by T08. (PRD § 6) T11: Revise a script-linked line delivery. Resolve line text, cast member, voice, language, pace, and timing from Script data. Generate an alternative speech take and align it without asking for known context again. Keep recorded-speech replacement, isolation, voice conversion, and lip-sync as separate unavailable actions. Blocked by T08. (PRD § 6) P2: Spatial and capability-dependent actions T12: Select and track a subject on the preview. Replace numeric-only region entry with normalized preview selection and overlay rendering. Connect a real tracking provider to the seam in packages/agents/src/capabilities/timeline-track-object.ts. Store samples in the existing MediaTrack model and invalidate them when the active source changes. Keep the action disabled when no execution provider is available. Blocked by T08. (PRD § 6) T13: Forward Entity and reference inputs through video edits. Extend the video-edit provider request and model-task validation so selected Entity reference images and explicit asset references reach compatible providers. Record reference ids on the take. Selecting an Entity never mutates its definition or recasts a storyboard. Blocked by T08. (PRD § 6) T14: Remove or replace a selected subject. Use T12 tracks and masks with providers that declare inpaint, mask input, and, for replacement, object-replace support. A replacement accepts an instruction or T13 references. Land an inactive video candidate and reuse P0 comparison and apply. Refuse stale tracks and missing mask support before spend. Blocked by T12 and T13. (PRD § 6) T15: Replace a character with an Entity. Build the fixed character replacement interaction on T14 with an Entity picker and identity reference preview. Require reference-conditioned provider support. Do not present prompt restyling as identity replacement. Blocked by T14. (PRD § 6) T16: Expand frame and upscale as distinct actions. Expand frame chooses target aspect and generated sides and requires an outpaint task. Upscale chooses an enhancement operation and result resolution. Both create candidates. Neither changes sequence dimensions or reframes existing pixels automatically. Blocked by T08. (PRD § 6) T17: Generate video-conditioned sound as an aligned audio result. Use the selected clip window and scene context with a provider task that accepts video or timed scene input. Create an aligned audio clip or take on an audio track. Never replace the entire video soundtrack silently. Blocked by T08. (PRD § 6) T18: Add recorded voice replacement and lip-sync as separate actions. Voice replacement requires speech isolation or an explicit clean speech source. Lip-sync consumes the accepted replacement line and produces a video candidate. Model selection and copy state which operation will run. Blocked by T11 and provider support for each task. (PRD § 6) T19: Generate a transition at a selected cut. Target two adjacent clips and an explicit overlap or duration. Store the result as a cut-level candidate, preview it across the edit, and apply it through a dedicated undoable cut operation. Do not store it as a normal replacement take for either source clip. Blocked by T08. (PRD § 5.4, § 6) Completion checks Every visible action has an executable provider task and an apply rule. Every action stores submission-time provenance on its result. Browser and agent paths call the same validation and application logic. No action creates or exposes a workflow. Existing assets, other timeline instances, storyboard shots, Entities, voices, and sequence dimensions remain unchanged unless the creator invokes a separate scoped operation.--- # On-Canvas Node Editors Source: https://docs.nodetool.ai/node-editors Many image nodes do not show a plain list of fields. They show a preview and controls made for the job: a crop rectangle you drag, a layer stack, a histogram, a paint surface. These editors render inside the node on the canvas. A few also open a larger window. The Prompt node has a composer with inline asset and variable chips, and the Code node has an AI-assisted code dialog. Every edit writes to the node’s ordinary properties, so the Inspector, saved workflows, and the API see the same values. Nothing here is hidden state, with the exceptions noted under Painter. Quick Access: Add the node from the node menu, or select one already on the canvas. The editor is the node’s body. How these editors behave Previews show the node’s output. Most editors preview the result the server last produced, so the preview updates when you run the node or the workflow. Crop previews the incoming image instead, because you are choosing a region of it. Slider drags are live, and release commits. Dragging a slider writes the value as you go. Releasing the slider records one undo step for the whole drag. Input handles stay on the left edge. You connect images and masks to the same handles as on any node. Reset buttons return the editor’s controls to their defaults in one undoable step. For the canvas itself, see Workflow Editor. Compositor Node: nodetool.image.Compositor Stacks several images into one. The node body shows the composited result on top and one row per layer below it. Control What it does Stored as Layer thumbnail Shows the image connected to that layer Not stored. It follows the connection Opacity slider Sets the layer’s opacity from 0 to 1 layers[i].opacity Blend mode dropdown Sets how the layer combines with the ones below layers[i].blend_mode Show or hide button Turns the layer on or off without disconnecting it layers[i].visible Delete button Removes the layer, its input handle, and any edge into it Removes image_N and layers[i] layer add button Adds a layer and a new input handle named image_N Adds image_N and a default layers entry Each layer is a dynamic input named image_1, image_2, and so on. The layers property is a list of objects that lines up by position with the sorted image_N inputs. Layout editor The edit button at the bottom of the node (tooltip “Open layout editor”) opens a larger window with a live preview. It needs WebGPU. Without it the window shows “WebGPU unavailable” and the node still composites on the server. Canvas size: Width and Height set the output size. They write canvas_width and canvas_height. If unset, the size follows the first layer’s image, or 512 by 512. Stage: Select a layer, then drag its body to move it, drag a corner or edge handle to scale it, and drag the top handle to rotate it. Selected layer: Fields for X, Y, Scale X, Scale Y, and Rotation°. Scale ranges from 0.02 to 64 and rotation from -360 to 360 degrees. The reset button (“Reset transform”) puts the layer back at its default placement on the canvas. Layers: The same opacity, blend mode, visibility, delete, and add controls as on the node. A layer you have never moved has no transform. Once you move, scale, or rotate it, its layers[i].transform holds x, y, scaleX, scaleY, and rotation. Crop Node: nodetool.image.Crop Choose a region of an image by dragging a rectangle over it. Drag the interior to move the rectangle. Drag any of the four corners or four edges to resize it. A rule-of-thirds grid shows inside the rectangle. Aspect locks the ratio while you resize. Options are Free, 1:1 Square, 4:3, 3:2, 16:9, 3:4, 2:3, and 9:16. Choosing a ratio resizes the rectangle height from its current width. Size shows the width and height in pixels. You can type into either. With an aspect locked, changing one changes the other. Reset crop selects the whole image. The preview shows the image arriving on the image input. If the input is not connected, it shows the node’s constant image value, and after a run it falls back to the node’s own result. Stored as: left, top, right, bottom, in source pixels. Width and height are derived (right - left, bottom - top). The aspect choice is not stored. Levels Node: nodetool.image.Levels Adjusts tonal range per color channel, with a histogram of the node’s output above the controls. Histogram: The R, G, B, and L (luminance) toggle picks which histogram to draw. Channel: R, G, or B picks the channel the three sliders edit. Black: the input black point, 0 to 255. Gamma: the midtone curve, 0.1 to 10. 1.00 is neutral. White: the input white point, 0 to 255. Reset levels restores black 0, gamma 1, white 255 on all three channels. Stored as: r_black, r_gamma, r_white, g_black, g_gamma, g_white, b_black, b_gamma, b_white. Curves Node: lib.image.color_grading.Curves A slider-based tone control, not a drawn curve. Two groups sit under the preview. Group Slider Stored as Range Tonal Black Point black_point 0 to 0.5 Tonal White Point white_point 0.5 to 1 Tonal Shadows shadows -0.5 to 0.5 Tonal Midtones midtones -0.5 to 0.5 Tonal Highlights highlights -0.5 to 0.5 RGB Midtones Red red_midtones -0.5 to 0.5 RGB Midtones Green green_midtones -0.5 to 0.5 RGB Midtones Blue blue_midtones -0.5 to 0.5 Signed sliders fill outward from the center, and a thumb lights up when its value differs from the default. Reset all curves returns every slider to its default (Black Point 0, White Point 1, the rest 0). Mask Node: lib.image.Mask Composites two images through a mask. The node has three image inputs, image1, image2, and mask, and no controls. The preview has four tabs: Img 1, Img 2, Mask, and Result. The first three show what is connected to each input, and Result shows the node’s output. Use the tabs to check that each input is what you expect before running. Painter Node: nodetool.image.Painter Paint a mask directly on an image. The node outputs the painted layer as a mask. Source: Connect an image to image, or drop an image file onto the paint area. A dropped file uploads as an asset, becomes the node’s image, and sets the canvas size to the picture’s size. With no image the canvas is a blank 512 by 512. Tools: Brush and Eraser, plus a color swatch (Brush color, default white). Size (1 to 256 pixels, default 24) and Opacity (0 to 1, default 1) control the brush. Overlapping strokes within one stroke do not stack opacity. Background opacity (0 to 1) fades the source image behind your strokes so you can see the paint alone. Canvas width and Canvas height (1 to 4096) set the paint size when no image is connected. Undo, Redo, and Clear painted mask. The editor keeps the last 30 states. Stored as: The painted pixels are saved as a base64 PNG in mask_data. Dropping an image sets image, canvas_width, and canvas_height. The brush tool, size, opacity, color, and background opacity are editor settings. This body does not write them back to the node. Chroma key Node: lib.image.keyer.ChromaKey Removes a background color. The body shows the preview, a Key Color picker, and three sliders. Control Stored as Range Default Key Color key_color Any color #00ff00 Tolerance tolerance 0 to 1 0.1 Softness softness 0 to 1 0.05 Spill spill 0 to 1 0.5 key_color is stored as a color value ({ "type": "color", "value": "#00ff00" }). Prompt Composer Node: nodetool.text.Prompt A prompt editor with inline references. The text box reads “Write a prompt… @ to mention an asset”. Mention an asset or entity: Type @. A picker opens with Recent and Saved tabs of assets, plus an Entities group when you have library entities. Choosing one inserts a chip. Drop an asset: Drag an asset from the asset browser into the text. It becomes a chip at the drop point. Insert a variable: The bar under the text has one {{ name }} chip per available variable. Click one to insert it at the cursor. Chips come from the node’s own inputs first, then from variables set by Set Variable nodes anywhere in the workflow. Add a variable input: The variable add button creates a dynamic input named var_1, var_2, and so on. Each input also gets an output handle that passes its raw value through. Stored as: prompt holds plain text. Chips are only how the editor draws it: Chip Stored text Resolved Asset asset://<id>.<ext> Replaced with the image or audio content before the prompt reaches a provider Entity entity://<id> Expanded at generation time into the entity’s description and reference image Variable {{ name }} Replaced from the node’s input or the workflow variable at run time Every keystroke writes prompt right away, so a run always sends the current text. Because the stored value is plain text, you can also type {{ name }} or asset://... by hand. Code assistant Node: nodetool.code.Code The Code node’s Ask AI button (the sparkle icon in the node toolbar) opens the Code assistant, a large dialog for writing the node’s code with the agent. The palette and the handle context menus can also start it, which creates a Code node for the dialog to edit. The dialog has two sides: Left: the node title, Inputs and Outputs chips (name: type), and a JavaScript editor holding a draft of the code. Right: a chat panel, resizable by its edge. Ask in plain language, for example “merge the two lists on id” or “add an error output”. The agent edits the draft, not the node. It reads the draft and the declared ports, replaces the code, and declares input and output handles. It can also validate the code, run it with sample inputs, and test it. You see its edits appear in the editor. You can type in the editor too. Button Result Apply Writes the draft code to the node’s code property and updates the node’s dynamic inputs, their values, and its dynamic outputs, all in one undoable step. Ports the node already had keep their value Cancel Leaves the node unchanged. If the dialog started from the palette or a handle menu and nothing was applied, the node it created is removed and any edge it displaced is restored The code runs in a sandbox. Declared inputs arrive on an inputs object, and the keys of the returned object become output handles. See JavaScript Sandbox for what the sandbox allows. Other bodies These nodes also have their own bodies. Controls are sliders and number fields unless noted, and each writes the property named for it. Node type What the body adds nodetool.image.Blur Type dropdown (Gaussian, Box, Motion) and a size slider nodetool.image.Channels R, G, B, A, and Luminance selector over the preview nodetool.image.CanvasResize Mode toggle for fixed, scale, or padding, with a dashed outline of the source on the new canvas nodetool.image.Fit Width and height fields, size presets of 256, 512, 768, and 1024 nodetool.image.Resize Width and height fields with an aspect lock nodetool.image.ResizeImage One body for scale, resize, and fit behind a mode toggle nodetool.image.Scale A 0 to 10 scale slider and a resulting-size badge nodetool.image.Paste Left and Top fields, with an outline of the paste rectangle over the base image nodetool.image.RotateAndFlip Free-rotate slider with snap marks at 0, 90, 180, and 270, plus flip toggles and reset lib.image.color_grading.HSLAdjust Color-range dropdown with hue, saturation, and luminance sliders lib.image.effects.ColorOverlay, DropShadow, Outline Color picker with amount, offset, radius, intensity, width, or threshold sliders lib.image.warp.Offset, Pad Offset sliders with a Clamp, Repeat, Mirror toggle. Pad has four edge sliders and a fill color lib.image.filter.Invert, ConvertToGrayscale A Before and After toggle on the preview Gradient, Checkerboard, and Gaussian noise generators in lib.image.draw Color pickers and sliders for each generator property Background-removal nodes from Replicate and fal Image and Mask preview tabs and a Recalculate Mask button Many other lib.image.color, color_grading, filter, enhance, keyer, and warp nodes A shared slider body that builds one slider for each numeric property nodetool.control.Collection A tray where you drop media to curate a list that streams downstream nodetool.video.ExtractFrame Video preview with frame stepping and a frame field that writes time nodetool.generators.ListGenerator A live numbered list of generated items nodetool.variable.GetVariable A picker of variables published by Set Variable nodes nodetool.constant.Sketch, nodetool.constant.Timeline A preview of the sketch or timeline with the node’s fields beneath it Related Workflow Editor for the canvas Sketch Editor for full layered painting outside a node JavaScript Sandbox for what Code node code can do--- # Node Packs Source: https://docs.nodetool.ai/node-packs.html Node Packs extend NodeTool with additional nodes and sandbox libraries. Some packs ship with NodeTool and are switched on or off in the app. Others are npm packages you install. What’s in a Pack A pack is an npm package. The nodetool field in its package.json says what it contributes: Component Field Description Host nodes register An exported function that registers node classes. They appear in the Node Menu and run in the server process, so they need trust before they load. Sandbox modules sandboxModules Libraries that Code nodes and agent actions import. They run inside the JavaScript sandbox or through a host facade. npm dependencies dependencies Normal npm dependencies of the package. See the Custom Nodes Guide for the manifest and Sandbox packages for sandbox modules. Managing Packs in the App Open the Package Manager Open Tools > Package Manager in the desktop app, or go to /packages in the web UI. The left rail has four lists. Included, Python packs, Third-party, and Software Included lists the packs that ship with NodeTool. Switch a pack on or off, then restart the server. The base pack is always on. Provider packs that need an API key are not listed. Their nodes appear after you add the key in Settings > Models & Providers. Python packs lists the Python node packs that NodeTool offers. They install from PyPI. Use Install, Update, or Uninstall, or Update all when several have updates. Installing a pack sets up Python first if it is missing. A pack installs at its newest stable release, not a version tied to the app version. Prereleases are skipped. Each install or update resolves the new pack together with every Python pack already installed, so shared dependencies such as PyTorch stay at versions all of them accept. The HuggingFace pack gets the PyTorch build that matches your graphics card. See GPU requirements. A pack appears only where it can install. MLX needs an Apple Silicon Mac. HuggingFace runs on Windows, Linux, and Apple Silicon Macs. On a Mac, both need macOS 14 or newer. Third-party installs npm packs. Paste a package name such as @acme/cool-nodes or cool-nodes@1.2.3 and click Install. Restart the server to load it. Software manages runtimes such as Python and FFmpeg, in groups for languages, media, and AI. Press / to focus the search box. It filters by name and description. The status filter next to it shows all packages, the installed ones, or the ones not installed. On the Included list, the options are On and Off. Installed includes packages that have an update. The search and the filter stay set when you open another list. Installing and removing run only in the desktop app. In the web UI, the lists are status-only. Trust Installing a third-party pack runs none of its code and no lifecycle scripts. What happens next depends on what the pack contains: Sandbox modules only. The pack is active right away. Its modules run inside the sandbox with the capabilities of the node that imports them. Host nodes. The pack stays inactive until you switch on Trusted for it in the Third-party list. Trusting runs the pack’s lifecycle scripts against the artifact that was installed. The Third-party list shows each pack’s mode, either sandbox modules only, host nodes, or both, and whether it is active. Installing Packs via CLI # List the Python packs the desktop app offers nodetool package list --available # Install a pack into the project's node_modules npm install <pack-name> # Update a pack npm update <pack-name> # Remove a pack npm uninstall <pack-name> The server finds packs by scanning node_modules directories from the working directory upward. Add more directories with NODETOOL_PACK_SEARCH_PATHS (a list of directories) or NODETOOL_OPTIONAL_NODE_MODULES (one directory). Restart the server after any change. In production (NODETOOL_ENV=production, or NODETOOL_PACKS_REQUIRE_ALLOWLIST=1), only packs on the allowlist load. Set NODETOOL_PACKS_ALLOWLIST to a comma-separated list of package names, or edit the allow list in ~/.config/nodetool/packs.json. See the trust model. Built-in Node Libraries NodeTool ships these first-party packs. The ids are in BUILTIN_NODE_PACKS (packages/protocol/src/builtin-packs.ts). Pack Default Namespaces and contents Base Nodes Always on nodetool (agents, audio, code, constants, control flow, data, documents, generators, images, input/output, text, video, and more), openai, gemini, mistral, xai, lib, messaging (Discord, Telegram), vector FAL On fal image, video, and audio models Kie.ai On kie image and video models Replicate On replicate community models Hugging Face On huggingface Inference Providers nodes ElevenLabs Off elevenlabs text-to-speech and voice MiniMax Off minimax image, video, and audio Transformers.js Off transformers local ONNX models Topaz Labs Off topaz upscaling and enhancement Reve Off reve image generation and editing AtlasCloud Off atlascloud image and video Higgsfield Off higgsfield image and video Together AI Off together image, video, speech, and transcription Browse the full node library in the Node Reference. Anthropic (Claude) and Ollama are reached through the provider system and generic nodes (for example nodetool.agents.Agent), not as standalone node namespaces. See Providers. Publishing Your Own Pack Build the pack as described in the Custom Nodes Guide. Publish it to npm or provide a Git URL. To offer a Python pack in the Package Manager, publish it to PyPI and add it to PYTHON_NODE_PACKS in packages/protocol/src/python-packs.ts. nodetool package list --available prints the same list. Next Steps Developer Guide – Build custom nodes and packs Custom Nodes Guide – Step-by-step node development Node Patterns – Common node implementation patterns CLI Reference – Package management commands--- # Package Registry Guide Source: https://docs.nodetool.ai/packages.html NodeTool packages bundle reusable nodes, assets, and example workflows. The package registry discovers and registers node classes so workflows can reference them at runtime. Manage packs in the app The Package Manager (Tools > Package Manager in the desktop app, or /packages in the web UI) has four lists: Included shows the packs that ship with NodeTool. Each has an Enabled/Disabled switch, except the core pack, which is always on. Provider packs that need an API key are not listed here. Their nodes appear once you set the matching key. Python packs lists the Python node packs that NodeTool offers, with Install, Update, and Uninstall buttons. They install from PyPI at their newest stable release, resolved together with the packs already installed. See Node Packs. Third-party installs an npm package by name and lists the packs the app has installed. See Node Packs. Software manages runtimes such as Python and FFmpeg. Installing and removing run only in the desktop app. The web UI shows status. Toggling a pack takes effect after the NodeTool server restarts. The choices are saved in ~/.config/nodetool/packs.json (enabledBuiltins and disabledBuiltins). Package Anatomy A first-party package is an npm workspace package under packages/ that exports node classes and a registration function: package.json – declares the package name, dependencies, and build scripts. src/nodes/ – node implementations, one file per domain (e.g. list.ts, audio.ts). src/index.ts – exports all node classes and a register*Nodes() function. tsconfig.json – extends the workspace base config. examples/ – optional workflow examples. assets/ – optional static assets used by nodes. A third-party pack has the same layout plus a nodetool field in package.json. See the Custom Nodes Guide. Example package.json This is the shape of packages/text-nodes/package.json, trimmed: { "name": "@nodetool-ai/text-nodes", "type": "module", "version": "0.8.1", "main": "dist/index.js", "types": "dist/index.d.ts", "scripts": { "build": "node ../../scripts/build-typescript-workspace.mjs", "test": "node ../../scripts/run-vitest.mjs run", "lint": "node ../../scripts/run-tsc.mjs --noEmit" }, "dependencies": { "@nodetool-ai/node-sdk": "*", "@nodetool-ai/protocol": "*", "@nodetool-ai/runtime": "*" } } Every node package depends on @nodetool-ai/node-sdk, which provides BaseNode, the @prop decorator, and the NodeRegistry type. The lint script is a type check, not ESLint. Example tsconfig.json { "extends": "../../tsconfig.base.json", "compilerOptions": { "outDir": "dist", "rootDir": "src" }, "include": ["src"], "references": [{ "path": "../node-sdk" }] } Node Registration The packages the runtime registers directly export a constant array of node classes and a registration function. @nodetool-ai/base-nodes is one of them; it aggregates node groups from sibling packages: import type { NodeClass, NodeRegistry } from "@nodetool-ai/node-sdk"; import { CONTROL_NODES } from "@nodetool-ai/core-nodes/nodes/control"; import { TEXT_EXTRA_NODES } from "@nodetool-ai/text-nodes/nodes/text-extra"; export const ALL_BASE_NODES: readonly NodeClass[] = [ ...CONTROL_NODES, ...TEXT_EXTRA_NODES, // ... additional node groups ]; export function registerBaseNodes(registry: NodeRegistry): void { for (const nodeClass of ALL_BASE_NODES) { registry.register(nodeClass); } } At startup the server (packages/websocket/src/node-registry-setup.ts) creates one NodeRegistry and fills it in three steps: registerBuiltinPacks calls the registration function of each first-party pack that is enabled. The catalog is BUILTIN_NODE_PACKS in packages/protocol/src/builtin-packs.ts: base (always on), elevenlabs, minimax, transformers-js, fal, kie, topaz, reve, atlascloud, higgsfield, together, replicate, and huggingface. Only packs marked defaultEnabled load on a fresh install. loadInstalledPacks scans node_modules directories for packages with a nodetool field and calls the export it names (register by default). Trust rules apply. See Custom Nodes Guide. Metadata from a running Python worker fills any node type the registry does not know yet. Workflows referencing nodetool.text.Concat or mypack.math.AddOffset resolve through the registry without manual imports. Managing Packages via CLI List Packages nodetool package list nodetool package list --available # Python packs the app offers Without --available, it lists packages whose metadata it finds under a nodetool/package_metadata/ directory in the current workspace. With --available, it prints the name, repo_id, and description of each Python pack in PYTHON_NODE_PACKS (packages/protocol/src/python-packs.ts). Add --json for machine-readable output. Initialize a Package nodetool package init Prompts for a name, description, and author, then writes package.json, tsconfig.json, src/index.ts, and empty nodetool/package_metadata/, examples/, and assets/ directories. It asks before overwriting an existing package.json. The scaffold’s package.json carries the nodetool field ("apiVersion": 1, "register": "register") and src/index.ts exports an empty register function for you to fill in. Generate Documentation nodetool package docs # single index.md in ./docs nodetool package docs --output-dir docs # custom directory (default: docs) nodetool package docs --compact # shorter summaries for LLM prompts package docs reads nodetool/package_metadata/ in the current directory and writes a single index.md overview. It fails when that directory is missing. For per-node Markdown pages, use node-docs: nodetool package node-docs # one page per node (default: docs/nodes) nodetool package node-docs --package-name mypack # only nodes whose namespace starts with mypack nodetool package workflow-docs --examples-dir examples # docs for workflow JSON files (default output: docs/workflows) workflow-docs requires --examples-dir and accepts --package-name to filter on the package_name field of each workflow. The full set of package subcommands is: list, init, docs, node-docs, and workflow-docs. (Note: nodetool mcp install / nodetool mcp uninstall configure the MCP server, not node packages.) Building Packages Run these from the repository root against one workspace, or from inside the package directory: npm run build --workspace=packages/<name> # compile npm run lint --workspace=packages/<name> # type check, no emit npm run test --workspace=packages/<name> # Vitest npm run build:packages builds every backend package in dependency order. Publishing Packages Implement nodes under src/nodes/ extending BaseNode with @prop decorators. Export all node classes and a registration function from src/index.ts. Run npm run build to compile. Add example workflows in examples/ and assets in assets/ if relevant. Publish to npm or provide a Git URL. To offer a Python pack in the Package Manager, publish it to PyPI and add it to PYTHON_NODE_PACKS in packages/protocol/src/python-packs.ts. Workflow Integration Enabled and trusted packs register nodes with the runtime at startup: Node metadata is merged during startup so workflows referencing package.namespace.Node resolve without manual imports. Run npm run codegen --workspace=packages/dsl to regenerate typed factory functions from node metadata. Related Documentation CLI Reference – package subcommands. Configuration Guide – where package metadata is cached. Custom Nodes Guide – step-by-step node implementation. TypeScript DSL Guide – type-safe workflow definitions with @nodetool-ai/dsl.--- # packs.md Source: https://docs.nodetool.ai/packs This page has moved. Please go to Node Packs.--- # Parameter-based cost estimation from GenSpend data — implementation plan Source: https://docs.nodetool.ai/parameter-cost-estimation-plan.html Parameter-based cost estimation from GenSpend data — implementation plan Plan, 2026-08-12. Targets the GenSpend v1 API (https://genspend.io/api/v1/…), which evolves additively only. 1. The problem The cost estimate NodeTool shows (and gates budgets on) is unit_price × quantity, where quantity is only fan-out (num_images etc.). The unit is carried but never applied: A model billed per video second is estimated at one second, whatever duration the node sets. 103 of the 283 GenSpend catalog entries are per-video-second — for those, a 10s clip is under-estimated 10×. Resolution ladders are collapsed at sync time to a single number (scripts/sync-genspend-pricing.mjs:120-132): same model twice → keep the cheaper, different models on one id → keep the dearer. A seedance-2 node at 1080p and one at 480p estimate the same. Audio-state pricing (silent vs with-audio, up to 2× apart), reference-image surcharges, and reference-video re-rates are not represented at all. GenSpend now publishes all of this — per-spec variants[], surcharges[], capabilities (clip-length envelopes), and dataFlags[] — through GET /api/v1/export (~92 KB gzipped) and prices it with POST /api/v1/quote. The plan: pull the parameterized data in the nightly sync, compute locally from the shipped snapshot (NodeTool is local-first; the browser bundle and the packaged Electron backend must both price without a network call), and verify our arithmetic against /quote in the sync workflow, where the network already exists. 2. Current architecture (what changes, what stays) scripts/sync-genspend-pricing.mjs nightly pull → generated JSON ← CHANGES scripts/genspend/{match,normalize,inventory} GenSpend↔NodeTool id matching ← stays packages/model-pricing/src/genspend-catalog.ts typed catalog wrapper ← CHANGES (schema v3) packages/model-pricing/src/index.ts getModelUnitPrice lookup ← CHANGES (param-aware) packages/node-sdk/src/cost-estimate.ts estimateWorkflowCost ← CHANGES (units × params) packages/protocol/src/creative.ts NodeCostEstimate types ← CHANGES (additive) web/src/hooks/useWorkflowCostEstimate.ts editor hook ← CHANGES (+ bug fix) web/src/components/costs/WorkflowCostEstimatePanel.tsx the panel ← CHANGES packages/websocket/src/unified-websocket-runner.ts estimateRunCost budget gate ← inherits for free packages/websocket/src/sdk/sdk-static-preflight-service.ts preflight ← inherits for free The matching pipeline (alias → variant → receipt → provider-id → catalog, in scripts/genspend/match.mjs) is unchanged: it answers “which NodeTool provider:model_id keys does this GenSpend offering price”, and that question is the same whether the value stored is a scalar or a grid. FAL and kie keep their own catalogs and keep winning the lookup order — this plan parameterizes the GenSpend tier only (a later PR can apply the same unit arithmetic to the FAL per-video-second entries, which have the same defect). 3. Phase 1 — pull job: ship the grid, not one rung Files: scripts/sync-genspend-pricing.mjs, packages/model-pricing/src/genspend-catalog.ts, packages/model-pricing/tests/genspend-sync.test.ts, .github/workflows/genspend-pricing.yml. 3.1 Switch the fetch to /api/v1/export /export is the same projection as /models plus surcharges, open dataFlags, and the embedded usage block, assembled server-side so the two cannot drift. Keep the ETag / If-None-Match handling, the strong-ETag workaround, the retry loop, and the refuse-to-write-empty guard exactly as they are; only the URL and the envelope shape change (export has {schemaVersion, generatedAt, counts, usage, models}). 3.2 Schema v3: GenspendPrice grows a grid Bump SCHEMA_VERSION to 3 (which also correctly invalidates the stored ETag on the first run). Each entry keeps today’s scalar fields — unit_price / billing_unit stay the base-spec price, so every existing consumer keeps working unmodified through the migration — and adds: interface GenspendVariant { price_usd: number; unit_class: string; // may differ per row (e.g. per-generation duration rows) resolution?: string; // "480p" | "720p" | "1080p" | "2K" | "4K" | image sizes duration_seconds?: number; with_audio?: boolean; video_input?: boolean; tier?: string; is_base: boolean; } interface GenspendSurcharge { kind: "input_image" | "input_video_second" | "per_request"; spec?: string; // resolution scope for input_video_second unit_price_usd: number; free_allowance: number; label?: string; // per_request extras ("prompt expansion") } interface GenspendPrice { /* existing v2 fields unchanged: unit_price, billing_unit, unit_class, model_slug, match, live, source_url, tier?, resolution? */ variants?: GenspendVariant[]; // the provider's published grid, typed facets only surcharges?: GenspendSurcharge[]; clip_seconds?: { set?: number[]; min?: number; max?: number } | null; data_flags?: Array<{ kind: string; severity: "quote_wrong" | "spec_gap" | "cosmetic" }>; } Rules, following GenSpend’s own usage guidance verbatim: Store facets, drop raw spec strings. The raw string is GenSpend’s truth surface, but we only compute on the typed facets; a variant whose facets are all null beyond is_base contributes nothing and is dropped. This keeps the JSON small. The base-scalar pick changes for the better. Today the collapse picks a rung arbitrarily (cheapest/dearest by trust rule). With the grid shipped, unit_price becomes the base-spec row (is_base: true) when one exists, falling back to the current rule. The dearer-model budget-gate rule for two different models on one id stays. quote_wrong flags gate at sync time and at display time. The sync stores the flag; the calculator refuses to price a flagged entry (returns null → confidence “unknown”). Today that fires zero times; it keeps us honest when it doesn’t. clipSeconds: null means decline every duration — store it as-is, never as permissive. Cosmetic flags are dropped at sync time (display-text only, and we render our own text). 3.3 Size and diff hygiene The v2 JSON is ~2 600 lines. Variants exist mostly on video models (~100 keys), typically 4–8 rows each; surcharges on a handful. Expected growth: roughly 2–3×, still far below the 840 KB full export because we ship only matched keys and typed facets. Measure after the first build and record the number in the PR. Keep the updatedAt-carry-over so an unchanged nightly run still produces no diff. 3.4 Workflow: add a parity gate Extend .github/workflows/genspend-pricing.yml with a step after the sync, before the PR is opened: node scripts/genspend/parity-check.mjs re-prices ~15 fixed cases (cheapest-provider image, a resolution ladder, per-second × duration, with/without audio, additive reference images with an allowance, a resolution-scoped video re-rate, a rung we must decline, per-request extras excluded from the total) through our calculator against the freshly-built catalog and asserts each equals POST /api/v1/quote for the same step. This mirrors GenSpend’s own export-parity.ts, which currently passes 17/17 — a drift in our port fails the nightly job instead of shipping a wrong number. The script needs the network, so it runs in this workflow only, never in the PR quality gate. Prove the gate can fail once before merging: perturb one local price and watch the step go red. 4. Phase 2 — the shared calculator: params in, breakdown out Files: packages/model-pricing/src/index.ts (+ new genspend-calc.ts), packages/node-sdk/src/cost-estimate.ts, packages/protocol/src/creative.ts, packages/node-sdk/src/ (new pricing-params.ts), tests in both packages. 4.1 New lookup signature (additive) export interface ModelPriceParams { resolution?: string; // normalized: "480p".."4K", "512×512", … seconds?: number; // output duration withAudio?: boolean; referenceImages?: number; megapixels?: number; } export interface ModelParamPrice extends ModelUnitPricingLike { breakdown?: string; // "5 s × $0.205/s at 720p" assumptions?: string[]; // "resolution not set on node — priced at base spec 720p" warnings?: string[]; // "reference-image surcharge not captured — treat as at least" declined?: string; // set instead of a price when we refuse to extrapolate } export function getModelUnitPrice( model: SelectedModel, params?: ModelPriceParams ): ModelUnitPricingLike | null; // unchanged behavior when params omitted getModelUnitPrice keeps its exact current behavior with no params — every existing call site (runner budget gate, chat spend, preflight) compiles and behaves identically until Phase 3 threads params through. 4.2 Selection rules (a deliberate subset of GenSpend’s calculator) Implemented in packages/model-pricing/src/genspend-calc.ts, pure and I/O-free, mirroring the rules GenSpend documents (and per their handoff, never “scan variants and take the cheapest”): Narrow variants by every param we hold: resolution (normalized, 768p → 720p tier per their parser; we implement the same 5-tier video / 4-tier image mapping), with_audio (unset param → prefer the base spec, the honest default deliverable), video_input === false for plain t2v/i2v. No row matches a stated resolution → decline (declined: "no published price at 1080p"), never reuse another rung. Resolution unset on the node → base-spec row plus an assumptions entry. per-video-second / per-audio-second: multiply by seconds; clamp check against clip_seconds — outside the envelope → decline. seconds unknown → price one second and record the assumption (confidence stays “estimate”, the panel shows “per second”). per-image / per-generation: unit price as today; fan-out multiplication stays in estimateWorkflowCost. Surcharges: input_image adds max(0, refs − free_allowance) × rate. input_video_second replaces generation cost with rate × (input + output) seconds, scoped by resolution — no scoped row for the requested resolution → decline the re-rate and warn (understating is the direction that hurts). per_request is surfaced in warnings/assumptions, never added silently. (Reference params are unlikely to be readable from nodes in v1 — the shapes ship in the calculator anyway so the parity gate exercises them and the UI can grow into them.) quote_wrong data flag → decline. spec_gap → exact at base spec, adds a warning off it. Deliberately out of scope for v1 (declined or base-priced with an assumption, matching what the catalog can’t express): token-billed image inputs, per-candidate agentic billing, quality axes. 4.3 Parameter extraction from node data New packages/node-sdk/src/pricing-params.ts, the sibling of FAN_OUT_PROPERTY_NAMES (web/src/utils/aiModelNodes.ts moves its list here or re-exports — one source of truth): Duration: first present of duration, duration_seconds, seconds, num_seconds, video_length, video_duration with a finite positive value; also num_frames ÷ fps when both exist. Values may be strings on some nodes ("5", "5s") — parse leniently, extract strictly. Resolution: first present of resolution, image_size, size, quality; width × height → nearest tier. Normalize to GenSpend’s accepted sets; an unrecognized value maps to unset (assumption), never to a guessed tier. Audio: generate_audio, with_audio, audio boolean properties. Before writing the list, enumerate the actual property names: grep the generator manifests (packages/fal-nodes, packages/replicate-nodes, kie) and packages/base-nodes video/image node definitions, and record the census in the PR — the list must come from what nodes ship, not from memory. estimateWorkflowCost gains an optional getParams?: (node) => ModelPriceParams input the callers supply (keeps node-sdk hermetic), threads the result into getModelPrice, and: multiplies seconds-billed prices by extracted duration (the calculator returns the already-multiplied unit_price for per-second classes, so estimated_cost stays unit_price × quantity — fan-out still applies on top: 2 clips × 5 s each), copies breakdown / assumptions / warnings / declined onto the item, applies the existing isVagueBillingUnit guard to the getModelPrice path too (today it only guards FAL metadata — a "units"-billed GenSpend fallback is currently summed; that’s a bug). 4.4 Protocol (additive) NodeCostEstimate in packages/protocol/src/creative.ts gains optional breakdown?: string, assumptions?: string[], warnings?: string[], and estimated_cost gains a documented “lower bound when warnings is non-empty” semantic. CostConfidence stays as-is — “estimate” covers parameterized figures; declined entries are “unknown” with the reason in assumptions. 5. Phase 3 — wire the callers (and fix the data-shape bug) Files: web/src/hooks/useWorkflowCostEstimate.ts, web/src/utils/aiModelNodes.ts, packages/websocket/src/unified-websocket-runner.ts, packages/websocket/src/sdk/sdk-static-preflight-service.ts, tests. 5.1 The bug: node.data vs node.data.properties Editor nodes store property values under data.properties (web/src/stores/NodeData.ts:25-27), but useWorkflowCostEstimate.ts:58,65 passes node.data straight through — so selectedModel and nodeExpectedQuantity read the top level and find nothing. In the live editor, generic model-picker nodes almost certainly price as “unknown” and every quantity is 1; the hook’s unit test passes because its fixture is flat. Fix first, with a test whose fixture uses the real nested shape (reproduce the failure before enforcing the fix — watch the current code go red against the nested fixture). The server preflight already handles both shapes (sdk-static-preflight-service.ts:373-378) and is the pattern to follow. 5.2 Thread params Web hook: build getParams from extractPricingParams(node.data.properties) and pass it to estimateWorkflowCost. The hook already subscribes to the NodeStore via useSyncExternalStore, so a duration edit re-prices on the next store tick — verify a property edit actually bumps the subscribed version, and re-render cost is bounded by the existing memoized estimate. Runner: estimateRunCost / estimateNotetoolSpend / preflight pass the same extractor over the graph’s node data. The budget gate stays a floor by policy — declines contribute 0 to the reserved amount exactly as unknowns do today, and per-second pricing only raises the gate, never lowers it (base-spec fallback is by construction ≥ the old collapsed-cheaper number for same-model grids). 6. Phase 4 — the panel File: web/src/components/costs/WorkflowCostEstimatePanel.tsx (+ its test). Qty → Units. The quantity column becomes a units cell rendered from the item: 2 × 5 s @ 720p, 4 images, 1 generation. Fan-out and duration read as one phrase. Breakdown is the trust surface. A row with breakdown shows it in the row’s tooltip/expand (the panel already has per-row tooltips for unknowns); assumptions render as a muted sub-line (“resolution not set — priced at 720p”). Lower bounds are labeled. Any item with warnings (unpriced surcharge) renders its cost as ≥ $X and the panel total as ≥ $Y when any contributing item is a lower bound — “at least”, never a false exact. Declines are actionable unknowns. A declined item keeps the warning icon but the tooltip says why (“no published price at 1080p for this provider — try 720p”), which is the reason GenSpend returns null instead of extrapolating. Staleness stays visible. The attribution line already renders updatedAt; keep it, and add catalogGeneratedAt to the tooltip. All rendering through ui_primitives and design tokens per repo rules; no raw MUI. Out of scope here but noted for a follow-up: per-node price chips (FalPricingFooter, KieCreditsFooter) could show the parameterized figure for GenSpend-priced nodes; today they are FAL/kie-only. 7. Test plan Per repo rules: every new check is proven able to fail once; the parity gate must also assert it found and priced cases (an audit that matches nothing passes vacuously). Layer Test Sync genspend-sync.test.ts: v3 entries carry variants/surcharges/flags from a fixture export; base-spec pick; quote_wrong gating; cosmetic flags dropped; empty-variants entries identical to v2 shape; unchanged-run produces no diff Calculator packages/model-pricing/tests/: ladder narrowing, decline on stated-but-missing rung, base-spec on unset resolution, seconds multiplication, clip envelope decline, audio axis, additive refs with allowance, re-rate replace + scoped decline, per_request excluded, 768p → 720p tier parse, unrecognized resolution → assumption not guess Parity scripts/genspend/parity-check.mjs vs /api/v1/quote, nightly workflow only; seeded with GenSpend’s documented figures (seedance-2 720p/5s = $0.56, minimax-h3 2K/6s + 8 refs = $1.02, kie re-rate 5s+5s = $1.25) Estimator packages/node-sdk/tests/cost-estimate.test.ts: params threading, vague-unit guard on the model path, breakdown/assumption propagation, fan-out × duration composition Web hook nested data.properties fixture (red on current code first), duration edit re-prices Panel units cell, ≥ rendering, decline tooltip, assumptions sub-line Budget gate application-budget-gate.test.ts: per-second model with duration raises the reserved amount; decline reserves 0 (floor semantics unchanged) Post-change: npm run typecheck && npm run lint && npm run test, plus every workspace nodetool affected names. 8. Sequencing and risk Four PRs, each shippable alone: Sync v3 (schema + /export + parity gate). No behavior change downstream — new fields are dead weight until Phase 2. Riskiest diff is the generated JSON; the nightly PR review checklist covers it. Calculator + estimator (params accepted, unused by callers). Pure packages, dense tests. Callers + bug fix. The data.properties fix alone changes live numbers (nodes that priced as “unknown” start pricing) — call that out in the PR since the budget gate tightens. Panel. Risks worth naming: The catalog is a snapshot; GenSpend corrects continuously. Mitigated exactly as they advise: nightly re-fetch with If-None-Match, updatedAt visible in the UI, parity gate on every sync. Our port can drift from their calculator. The parity gate is the control; if it becomes a maintenance burden, the fallback GenSpend offers is vendoring their calc.ts + apply-surcharges.ts — keep our calculator’s interface narrow enough that swapping the implementation is contained to genspend-calc.ts. Property-name census is heuristic. A node whose duration property we don’t recognize prices at one second with an assumption line — visible, not silent. The census list is data, easy to extend. Tightened budget gate can refuse runs it used to admit. That is the point, but it lands in PR 3 with its own callout; the gate’s fail-open error handling is untouched. 9. Explicitly not in this plan Calling POST /api/v1/quote from the product at edit time (a keystroke stream describing what a user builds leaves the machine; GenSpend themselves recommend local compute for local-first products). A “why is this the price?” affordance that makes one deliberate /quote call for the breakdown string is a reasonable follow-up. Parameterizing the FAL/kie catalog tiers (same defect, different generators — follow-up). LLM token pricing (per-1m-tokens entries; prompt length is unknowable pre-run). Reading reference image/video counts from node inputs (shapes ship in the calculator; the extractor can learn them later).--- # 3D Clips on the Timeline — Implementation Plan Source: https://docs.nodetool.ai/plans/3d-timeline-clips-tasks.html 3D Clips on the Timeline — Implementation Plan Companion to 3d-timeline-clips.md (the technical design). Every task below is written to be executed by an agent with no prior conversation context. Each task’s first step is: read docs/plans/3d-timeline-clips.md in full. It is the contract; this file adds sequencing, file-level pointers and acceptance criteria. Ground rules for every task: After changes run the four checks in AGENTS.md § Mandatory Post-Change Verification: npm run test:affected, npm run typecheck, npm run lint, npm run dev:nodetool -- harness gate --base origin/main. packages/timeline is pure TypeScript: no DOM, no GPU, no three.js, no store imports. The render session lives in packages/video-nodes. Web UI: primitives from web/src/components/ui_primitives/ only, tokens per docs/DESIGN.md. Never import raw MUI. A new check must be shown failing once (AGENTS.md § Claims, Checks, and Measurements). Every task names the test that proves it. Commit per task with a conventional message. Do not reformat unrelated code. Dependency graph: M1: T1 (schema) ──► T2 (scene model + channels) T3 (render session) is independent T2 + T3 ──► T4 (browser source, preview, export) T2 + T3 ──► T5 (agent frames on the server) T1 + T2 ──► T6 (ops, tools, capabilities) M2: T4 ──► T7 (add clip, drag-and-drop) T4 + T6 ──► T8 (inspector) T4 ──► T9 (lane thumbnails, clip frames) T8 ──► T10 (preview orbit) M3: T6 + T8 ──► T11 (opaque bake) T11 ──► T13 (alpha bake) every task ──► T12 (docs, harness registry) Milestone 1 — A dropped GLB previews and exports T1 — Schema, types, defaults, validation Goal: a model3d clip round-trips through the wire schema and the document types, with defaults and validator codes. Read first: design §D1, §D2. packages/protocol/src/api-schemas/timeline.ts (timelineClip, clipShapeStyle as the pattern), packages/timeline/src/types.ts (ClipShapeStyle, TimelineClip), packages/timeline/src/defaults.ts, packages/execution/src/timeline-debug/validate.ts. Steps: Add "model3d" to mediaType in the Zod schema and the type. Add clipModel3DStyle (camera, animation, lighting, lightIntensity, background, bake) and model3dStyle: clipModel3DStyle.optional() on timelineClip, with the doc comment that explains why an unlisted field is stripped on PATCH. Mirror the types in packages/timeline/src/types.ts as ClipModel3DCamera, ClipModel3DAnimation, ClipModel3DStyle, and model3dStyle? on TimelineClip. defaults.ts: DEFAULT_MODEL3D_STYLE (orbit, azimuth 45, elevation 25, fov 35, zoom 1, all animations, loop on, speed 1, studio lighting, intensity 1, transparent). Match the RenderToImage prop defaults. validate.ts: model3d_style_missing (error) for a model3d clip with no model3dStyle or no asset; bake_stale (warning) when model3dStyle.bake.dependencyHash differs from computeModel3DBakeHash(clip) (T11 owns the real hash; land the function here as the style hash plus currentAssetId). field_stripped already detects fields the schema drops; add a case that proves a model3dStyle survives timelineClip.parse. Acceptance: packages/execution/tests/timeline-debug.test.ts (the validator suite): a model3d clip without a style yields model3d_style_missing; with a stale bake yields bake_stale; a valid clip yields nothing new. packages/protocol test: parsing a clip with model3dStyle keeps every field. T2 — Scene model kind, camera channels, orbit preset Goal: the pure scene model emits kind: "model3d" layers, samples four camera channels, and offers an orbit preset. Read first: design §D3, §D4. packages/timeline/src/render/sceneModel.ts (ActiveLayer, the shape branch of computeActiveLayersWithHorizon, MAX_VIDEO_LAYERS, DroppedLayerReason, clipSourceTimeSec, resolveAnimatedLayerProps), packages/timeline/src/animation/types.ts (ANIMATED_PROPERTIES, ANIMATED_PROPERTY_FOLD), animation/presets.ts (spin as the pattern), keyframes.ts (KEYFRAME_PROPERTIES). Steps: ActiveLayer.kind gains "model3d"; the layer carries model3dStyle and sourceTimeSec (from clipSourceTimeSec times animation.speed). In computeActiveLayersWithHorizon, a model3d clip with a fresh bake (hash matches, T1’s function) emits a video layer with assetId = bake.assetId and bakeSourceTimeSec = (timeMs - clip.startMs) / 1000 (design §D6, time origin); otherwise a model3d layer with assetId = effectiveAssetId(clip). Count live model3d layers against MAX_MODEL3D_LAYERS = 2; over it, push a dropped layer with the new reason model3d_layer_cap. Channels cameraAzimuth (add, 0), cameraElevation (add, 0), cameraZoom (multiply, 1), cameraFov (add, 0) in ANIMATED_PROPERTIES, the fold table and KEYFRAME_PROPERTIES. resolveAnimatedLayerProps returns them in anim. Export resolveModel3DCamera(style, anim): ClipModel3DCamera that folds the sampled channels into the style’s camera. Both hosts call it so the fold lives once. Preset orbit: role loop, params degrees (default 360) and direction, compiles to a linear cameraAzimuth curve over the clip. Acceptance: packages/timeline/tests/sceneModel*.test.ts: a model3d clip yields a model3d layer with the right sourceTimeSec after a trim and a 2x speed; a fresh bake yields a video layer whose bakeSourceTimeSec is 0 at the clip’s first frame for a clip with a 5 s in point and a 2x speed; a stale bake yields model3d; a third active 3D clip is dropped with model3d_layer_cap. Animation tests: orbit over a 4 s clip samples cameraAzimuth 180 at 2 s; a keyframed cameraZoom folds multiplicatively with the style’s zoom in resolveModel3DCamera. The horizon (nextChangeMs) is unchanged by 3D layers. T3 — Render session in video-nodes, multi-frame headless entry Goal: createModel3DRenderSession in the browser core, a multi-frame CDP entry in the page bundle, and renderGlbFramesHeadless on Node. RenderToImage keeps its behavior. Read first: design §D5. packages/video-nodes/src/nodes/model3d/render3d-core.ts, render3d-page.ts, render3d-headless.ts, render.ts, scripts/bundle-render3d-page.mjs, packages/video-nodes/tests/model3d-render.test.ts (covers computeFraming and orbitOffset). Steps: Split renderGlbToPng into createModel3DRenderSession(glb, options) (load, meshopt, bounding sphere, lights, background, renderer, mixer with the actions options.animation selects: the named clip, or every clip) and session.render(frame) (camera from ClipModel3DCamera through the existing computeFraming and orbitOffset, scene mode picks the named glTF camera, mixer.setTime(t) with t wrapped at the selected actions’ longest duration when loop is on and clamped to it when off, render, return the canvas). session.animations and session.cameras list the glTF’s names; an unknown clipName throws naming the available ones. renderGlbToPng becomes create, render once, convertToBlob, dispose. render3d-page.ts: __nodetoolRenderGlbFrames(glbBase64, options, frames[]) returns one PNG per frame from one session. render3d-headless.ts: renderGlbFramesHeadless(glb, options, frames), one Chrome launch, calls the new entry, returns Uint8Array[]. renderGlbHeadless stays and delegates for one frame. Rebuild the page bundle; PACKAGE_RUNTIME_ASSETS already registers it. Acceptance: Existing RenderToImage tests pass unchanged. New unit test on the pure part: scene mode with an unknown camera name throws a message naming the available cameras; the loop and clamp time mapping is a pure function (mixerTimeFor(t, duration, loop)) tested at t past the end for both modes. Headless tests on a fixture with two named animations (the existing harness, skipped without Chrome the way the current one is): selecting each name yields a different PNG at the same time; loop off at 1.5x the duration equals the last frame; rendering t = 0.2 after t = 5 equals rendering t = 0.2 first. T4 — Browser layer source, preview, export Goal: the live preview and the browser export draw model3d layers. Read first: design §D5 host 1. web/src/components/timeline/preview/compositeLayers.ts (CompositeSourceResolver), preview/gpu/types.ts (CompositeSource), preview/gpu/compositor.ts and canvas2dCompositor.ts (where a source becomes a texture or a drawImage), preview/PreviewCompositor.tsx (the resolver it builds, OffscreenVideoPool usage), render/TimelineRenderer.ts (the export resolver), Tracks/useAssetUrl.ts, utils/assetHelpers.ts. Steps: CompositeSource gains OffscreenCanvas. Both compositors accept it (copyExternalImageToTexture and drawImage both take one). Add the test in preview/gpu/__tests__/ that a canvas source composites. preview/Model3DLayerSource.ts: a pool keyed by clip id, cap 2, of { session, assetId, options: Model3DSessionOptions }. acquire(layer) fetches the GLB by asset URL, caches parsed bytes per asset id (least recently used, small cap), creates the session from the style’s lighting, intensity, background and animation, and returns null while loading. A layer whose asset id or session options differ from the pooled entry’s disposes it and creates a new one. frame(layer, anim) calls resolveModel3DCamera and session.render and returns the canvas. release(clipId) disposes. No WebGL is a Model3DUnavailable state the preview draws as an outlined placeholder with the clip name. PreviewCompositor.tsx and TimelineRenderer.ts: the resolver’s model3d branch calls the source. The export creates its own pool and disposes it when done. rasterClipFrames.ts keeps throwing for model3d; T9 handles it. Acceptance: A Jest test for Model3DLayerSource with a mocked session: the pool evicts at cap, returns null while loading, reuses a session across frames, re-creates the session when the clip’s background or lighting changes and not when only the camera changes, and reports unavailable without WebGL. Manual: drop a GLB on an overlay track over a video clip, scrub, play, export, and the exported MP4 shows the model at the same pose as the preview at 1 s and 3 s (compare with compare_timeline_frames against T5’s frames once it lands). T5 — Agent frames on the server Goal: preview_timeline_frame renders model3d layers through headless Chromium and reports them. Read first: design §D5 host 2. packages/agents/src/timeline-preview/frames.ts (the shape resolver branch, PreviewLayerReport, PreviewDegradation, Canvas2DDegradationReason), rasterize.ts (PreviewRasterizer cache), packages/video-nodes/src/nodes/model3d/render3d-headless.ts (T3’s renderGlbFramesHeadless). Steps: Before compositing, walk the requested timecodes, collect every model3d layer, group by asset id plus Model3DSessionOptions (lighting, intensity, background, animation), and call renderGlbFramesHeadless once per group with the folded camera and sourceTimeSec per frame. Cache the decoded images by (group key, camera, timeSec) for the call. The resolver’s model3d branch returns the decoded image; layerText style reporting gains camera and animation_time_sec on the layer report. Chrome missing or the launch failing is a PreviewDegradation with reason model3d_unavailable, not a thrown error; the layer is skipped. Acceptance: packages/agents/tests/timeline-model3d-frames.test.ts with the headless call mocked: two timecodes and one asset make one call with two frames; two clips on the same asset with different backgrounds make two calls with their own options; the report lists the layer with its camera; a rejected launch yields the degradation and a frame without the layer. The real headless path runs under the same Chrome-gated skip as the RenderToImage test. T6 — Ops, tools, capabilities Goal: an agent can add a 3D clip, set its style, and animate its camera from ui_timeline_edit and edit_timeline. Read first: design §D8. packages/timeline/src/ops/op.ts, ops/apply.ts (add_shape_clip, set_clip_params), packages/protocol/src/api-schemas/timeline-tool-params.ts (shapeStyleParams, withFieldNotes), web/src/lib/tools/builtin/timeline.ts (ui_timeline_edit), packages/agents/src/capabilities/timelines.ts and timelines.specs.ts, packages/agents/tests/timeline-op-parity.test.ts. Steps: Ops add_model3d_clip, set_model3d_style in op.ts and apply.ts. add_model3d_clip needs an asset id, lands on the named track or the first overlay track (creating one when none), default duration 4 s, default style. set_model3d_style is a deep partial patch. model3dStyleParams with field notes in timeline-tool-params.ts; the animate_clip preset list picks up orbit from the catalog. Wire both ops through ui_timeline_edit and the edit_timeline capability; the parity test enforces that both hosts expose the same op set, so it fails until both do. A timelines.specs.ts case: add a 3D clip, set orbit, preview a frame, expect a model3d layer in the report. Acceptance: timeline-op-parity passes with the two new ops. capabilities-timelines covers add, patch and the validator error for a missing asset. Milestone 2 — Editing surface T7 — Add clip menu and drag-and-drop Read first: design §D7. web/src/components/timeline/AddClipMenu.tsx, dnd/assetToClipAdapter.ts and its tests, web/src/utils/nodeGenerations.ts (how a model asset is recognized), README.md drop table. Steps: “3D model” entry opening the asset picker filtered to models; assetToClipAdapter accepts model/* or a .glb / .gltf name on video and overlay lanes and builds a model3d clip with the default style; the warning banner for other lanes stays. Acceptance: adapter tests for accept on video, accept on overlay, reject on audio; a model3d clip from the menu has the default style. T8 — Inspector section and keyframes Read first: Inspector/ClipShapeSection.tsx (pattern), Inspector/TimelineInspector.tsx, Inspector/ClipKeyframes.tsx, Inspector/InspectorPrimitives.tsx, usePersistedFold.ts. Steps: ClipModel3DSection with camera mode, the four orbit fields, scene-camera and animation pickers fed by the layer source’s session (a loading state until it resolves), loop, speed, lighting, intensity, background, and a Bake row (disabled with the reason until T11). Keyframes section lists the camera channels for a model3d clip. Acceptance: RTL tests: the section renders for a model3d clip only, patches model3dStyle.camera.azimuthDeg through patchClip, and the animation picker shows the names the session reports. T9 — Lane thumbnails and clip frames Read first: Tracks/clipThumbnails.ts, Tracks/useClipThumbnails.ts, Tracks/filmstripCells.ts, preview/rasterClipFrames.ts. Steps: a model3d clip requests cells from the layer source at the cell source times; renderRasterClipFrames handles model3d through the same source so ui_timeline_get_clip_frames returns real frames. Acceptance: thumbnails test with a mocked source; rasterClipFrames test that a model3d clip returns one data URL per requested time. T10 — Orbit gesture in the preview Read first: preview/TransformGizmoOverlay.tsx, preview/ClipTransformContextMenu.tsx, Model3DEditor.tsx orbit math. Steps: with a model3d clip selected, Alt-drag maps pointer delta to azimuth and elevation, Alt-wheel to zoom, written through patchClip on pointer up (one undo step per gesture). A readout shows the pose while dragging. Acceptance: RTL test that a drag of 100 px changes azimuth by the documented degrees-per-pixel and produces one store patch. Milestone 3 — Bake and documentation T11 — Opaque bake with Blender Read first: design §D6 in full. packages/blender-nodes/src/nodes/render-animation.ts, job.ts (RenderAnimationParams), packages/blender-nodes/blender_ops/ops/render_animation.py (run, _animate_orbit, the not animated and mode == "orbit" branch), render_image.py (write_still), packages/model3d/src/gltf.ts (GltfAnimation, accessor reads), packages/timeline/src/dependencyHash.ts, web/src/components/timeline/render/TimelineRenderer.ts and preview/PreviewCompositor.tsx (the video seek), packages/agents/src/timeline-preview/frames.ts (decodeVideoFrameAt and its video branch), web/src/stores/timeline/TimelineGenerationStore.ts (how a generated clip registers a job and stores a version). Steps: Blender prerequisite, the sampled producer: RenderAnimationParams gains frame_times?: number[] (model time in seconds per output frame), animation_name?: string, and cameras?: BakeCameraParams[] (one per frame). BakeCameraParams extends CameraParams with scene_camera_name? and target_offset?, the two fields ClipModel3DCamera carries that the existing DTO does not; the mapping from ClipModel3DCamera is one function in packages/timeline, tested both ways. With frame_times set, render_animation.py selects the named action (mutes every other NLA track) or, with animation_name absent, leaves every action playing, the D2 default; a model with no animations renders its rest pose at every entry. For each entry it calls scene.frame_set(frame, subframe) from the model time and the scene fps, places the orbit camera from that entry (scene mode keeps the glTF camera), and renders one still to frame_%06d.png; the op returns the sequence and the TS side muxes it with the ffmpeg helper. Without frame_times the op behaves as today, so the node is unchanged. packages/model3d: animationDurations(gltf) from each animation’s sampler input accessor max. packages/timeline: computeModel3DBakeSamples(clip, sequence, durations) returning { frameTimes, cameras } from clipSourceTimeSec at each output frame, animation.speed, loop wrap or clamp, and resolveModel3DCamera at each frame. computeModel3DBakeHash(clip, sequence) in packages/timeline, covering exactly the inputs design §D6 lists and nothing from bake itself. Replace T1’s placeholder. Every video decoder seeks by layer.bakeSourceTimeSec when present and clipSourceTimeSec otherwise: the preview resolver, the export resolver, and decodeVideoFrameAt in frames.ts, which takes the resolved seek time as an argument instead of recomputing it from the clip. bake_model3d_clip op: refuses a transparent style (until T13) with a message naming the reason; otherwise builds the sampled request from step 2 and the style, runs it as a generation job, muxes to MP4, and on completion stores model3dStyle.bake with the hash and a clip version. Inspector Bake row: run, progress, stale badge from bake_stale. Acceptance: Samples test: a clip with a 5 s in point and 2x speed yields frameTimes[i] = 5 + 2 * i / fps; a reversed remap yields a decreasing list; loop on past the animation’s end wraps, loop off clamps; the camera list under the orbit preset sweeps the degrees across the clip. Scene model: after a bake, changing the in point, the speed, the time remap, the duration or the sequence fps makes the layer live again; changing the clip’s start or opacity does not. Decoder tests: the browser resolvers (mocked video pool) and decodeVideoFrameAt (mocked forEachVideoFrame) each seek a fresh trimmed bake to 0 at the clip’s first frame and to 1 s one second later; the agent test asserts the first requested source timestamp is zero. Blender op test (skipped without BLENDER_BIN, the way the existing ones are): an animated fixture baked with frame_times: [0, 1.0, 0.5] renders a second frame equal to render_image at model time 1.0 and a third equal to 0.5, within the existing image-compare tolerance; the same fixture with a 180° camera sweep has different camera locations in its stats for the first, middle and last frame; a two-animation fixture with no animation_name differs from the same fixture with either name selected; a fixture with no animations renders identical frames at every entry; a fixture with a named glTF camera baked in scene mode reports that camera in its stats. The capability suite bakes an opaque fixture with the runner mocked. T13 — Transparent bake Read first: design §D6 output format. render_animation.py, packages/video-nodes/src/nodes/timeline/outputFormats.ts (the webm VP9 yuva420p entry), packages/video-nodes/src/nodes/ffmpeg-helpers.ts, packages/blender-nodes/blender_ops/ops/render_passes.py (RGBA PNG output). Steps: The sampled producer from T11 already writes PNGs; under a transparent style they carry alpha through film_transparent and color_mode RGBA. The bake op, for a transparent style, muxes the sequence to WebM VP9 yuva420p through the existing helper instead of MP4, and stores that asset. The browser video resolvers treat a decode error or a bake whose first frame reports no alpha as bake_undecodable: the live layer draws and the degradation is reported. Acceptance: A test that composites a baked transparent fixture over a solid color through the export path and asserts the background color is visible in a pixel the model does not cover. The op no longer refuses a transparent style. A mocked decode failure yields bake_undecodable and the live layer. T12 — Documentation and harness registry Steps: web/src/components/timeline/README.md drop table and a “3D clips” section; docs/harnesses.md entry for the 3D layer in preview_timeline_frame; packages/cli/src/harness/registry.ts maps a diff under packages/video-nodes/src/nodes/model3d/ to the T3 and T5 suites; AGENTS.md harness table row if a new command appears. Set this design’s status to implemented. Acceptance: npm run check:agents-docs and npm run capabilities:check pass; harness gate --dry-run on a diff touching render3d-core.ts lists the T3 suite.--- # 3D Clips on the Timeline — Technical Design Source: https://docs.nodetool.ai/plans/3d-timeline-clips.html 3D Clips on the Timeline — Technical Design Status: implemented. The editing surface is documented in web/src/components/timeline/README.md § 3D clips, the agent surface in docs/harnesses.md § 3D clips in preview_timeline_frame. Companion doc: 3d-timeline-clips-tasks.md (implementation plan, agent-consumable tasks). Goal Put a glTF scene on the timeline the way an image or a title goes on it: drop a model on a video or overlay track, scrub it, orbit the camera, key the camera, let its glTF animation play against the playhead, composite it over footage with the effects, masks, mattes and transitions every other clip has, and see the same picture in the live preview, the browser export, and the agent’s preview_timeline_frame. Final quality comes from a bake through Blender into an ordinary video asset the clip then plays. Non-goals A new track type. 3D is picture, so it lives on video and overlay tracks. Editing geometry, materials or lights from the timeline. That is the model editor and edit_model3d, on the asset. Per-clip overrides of individual scene nodes. A variant is a second asset. More than one model per clip. Two models are two clips, composed with a group. Rendering 3D in the server ffmpeg rough cut. It stays a rough cut, and a baked clip is a video clip it already handles. What exists Concern Where it lives Clip kinds mediaType in packages/protocol/src/api-schemas/timeline.ts and packages/timeline/src/types.ts; text and shape clips carry textStyle / shapeStyle and no asset. Scene resolution packages/timeline/src/render/sceneModel.ts: computeActiveLayersWithHorizon emits ActiveLayer { kind: "video" \| "image" \| "text" \| "shape" \| "caption" }, resolves groups, mattes, masks, transitions, and drops layers over MAX_VIDEO_LAYERS with a DroppedLayerReason. Animation packages/timeline/src/animation/: ANIMATED_PROPERTIES with a fold per channel, presets, and resolveAnimatedLayerProps sampling every channel for a layer at a time. Keyframes are curves on the custom preset (keyframes.ts). Browser compositing web/src/components/timeline/preview/compositeLayers.ts maps ActiveLayer to CompositeLayer for both the live PreviewCompositor and the TimelineRenderer export. A host supplies pixels through CompositeSourceResolver; CompositeSource is a video element, an image element or an ImageBitmap. Agent frames packages/agents/src/timeline-preview/frames.ts composites the same layers on @napi-rs/canvas; PreviewRasterizer draws text, shape and caption layers; the report lists degradations and dropped layers. 3D rendering, browser packages/video-nodes/src/nodes/model3d/render3d-core.ts: three.js renderGlbToPng on an OffscreenCanvas, orbit camera in azimuthDeg / elevationDeg / fovDeg / zoom, lighting presets, transparent background. 3D rendering, server render3d-headless.ts launches headless Chromium with SwiftShader and calls the bundled render3d-page.js over CDP. @nodetool-ai/agents already depends on @nodetool-ai/video-nodes. 3D rendering, final packages/blender-nodes: nodetool.blender.RenderAnimation renders a glTF’s animation or an orbit sweep to H.264 with the same camera vocabulary. 3D editor web/src/components/model_editor/Model3DEditor.tsx on react-three-fiber; Model3DCameraPose is orbit terms around a target. Generated clips bindingKind, paramOverrides, currentAssetId, versions, dependencyHash.ts and staleSet.ts decide when a generated clip is stale. Decisions D1. A model3d clip, not a 3D track mediaType: "model3d" joins image, video, text, shape. Tracks keep their five types. A track type changes what the mixer, ripple, drop rules and header controls do, and none of that differs for a 3D picture. Drop rules extend the existing table: a model/* or .glb asset lands on video and overlay lanes. D2. The model is the clip’s asset, the look is model3dStyle The glTF lives in currentAssetId, exactly where an image clip keeps its image. Replace, drag-and-drop, effectiveAssetId, thumbnails, versions and inputAssetHashes in the dependency hash all work unchanged. Everything the timeline adds on top is one optional block, persisted in the Zod schema (a field absent from the schema is stripped on PATCH): // packages/timeline/src/types.ts export type Model3DCameraMode = "orbit" | "scene"; export interface ClipModel3DCamera { /** `orbit` frames the model's bounding sphere; `scene` uses a glTF camera. */ mode: Model3DCameraMode; /** Orbit terms, the vocabulary of RenderToImage, the Blender job and the editor pose. */ azimuthDeg: number; elevationDeg: number; fovDeg: number; /** Distance multiplier on the auto-framed fit: >1 closer. */ zoom: number; /** Look-at offset from the bounding-sphere center, world units. */ targetOffset?: [number, number, number]; /** `scene` mode: the glTF camera's name; the first camera when absent. */ sceneCameraName?: string; } export interface ClipModel3DAnimation { /** glTF animation name; every animation plays when absent. */ clipName?: string; loop: boolean; /** Playback multiplier on top of the clip's own speed / time remap. */ speed: number; } export interface ClipModel3DStyle { camera: ClipModel3DCamera; animation: ClipModel3DAnimation; lighting: "studio" | "soft" | "flat"; lightIntensity: number; background: { transparent: true } | { transparent: false; color: string }; /** * A Blender render of this clip at a given style. The scene model plays it * as video while `dependencyHash` matches the live style; a style edit * makes it stale and the live layer draws again. */ bake?: { assetId: string; dependencyHash: string }; } TimelineClip.model3dStyle?: ClipModel3DStyle. Defaults live in packages/timeline/src/defaults.ts next to the text and shape defaults. D3. Time: the glTF animation follows the clip’s source time clipSourceTimeSec(clip, timeMs) already resolves in point, speed and time remap for video. A 3D layer uses the same number, times animation.speed, as the AnimationMixer time. Trimming hides animation instead of deleting it, a split copies the style and moves inPointMs, and a reversed remap plays the animation backwards, the same as video. With loop on, the mixer wraps at the animation’s duration; off, it holds the last frame. D4. Camera moves are animation channels Four channels join ANIMATED_PROPERTIES, each additive on top of model3dStyle.camera: Channel Fold Identity cameraAzimuth add 0 cameraElevation add 0 cameraZoom multiply 1 cameraFov add 0 Keyframes come free: KEYFRAME_PROPERTIES gains the four, the inspector’s Keyframes section keys them at the playhead, and keyframeValueAt samples them. One preset joins the catalog: orbit (role loop, degrees 360, direction) drives cameraAzimuth, so “make it a turntable” is one animate_clip call. The channels are ignored on any layer that is not model3d, the way wipeProgress is ignored without a wipe. D5. One render session, four hosts render3d-core.ts becomes a session: load once, render many. // packages/video-nodes/src/nodes/model3d/render3d-core.ts export interface Model3DRenderFrame { timeSec: number; camera: ClipModel3DCamera; // already folded with the sampled channels width: number; height: number; } export interface Model3DRenderSession { /** Names of the glTF's animations and cameras, for the inspector and the validator. */ readonly animations: readonly string[]; readonly cameras: readonly string[]; /** Draws one frame into the session's canvas and returns it. */ render(frame: Model3DRenderFrame): OffscreenCanvas; dispose(): void; } /** Everything a session fixes at creation. Two layers share a session only * when these are equal, so the key of every pool and every headless group is * `assetId` plus this object, never `assetId` alone. */ export interface Model3DSessionOptions { lighting: ClipModel3DStyle["lighting"]; lightIntensity: number; background: ClipModel3DStyle["background"]; animation: ClipModel3DAnimation; } export function createModel3DRenderSession( glb: Uint8Array, options: Model3DSessionOptions ): Promise<Model3DRenderSession>; animation selects the mixer’s actions: clipName plays that one glTF animation, absent plays all of them. render sets absolute time, so a scrub backwards after the end is a plain setTime. With loop on, the time wraps at the longest selected animation’s duration; off, it clamps to the last frame. A style edit to any session option disposes the session and creates a new one, in the browser pool and in the headless group alike, because a mixer whose actions were rebuilt mid-session is the same picture as a fresh one and harder to prove. renderGlbToPng becomes a one-frame call on a session, so RenderToImage keeps its behavior and its tests. The hosts: Live preview and browser export. web/src/components/timeline/preview/Model3DLayerSource.ts holds a session per active model3d layer, keyed by clip id and re-created when the clip’s asset or session options change, in a pool capped like the video pool (a WebGL context per session, browsers allow about sixteen). It loads the GLB through the asset URL, caches parsed scenes per asset id, and its resolver returns the session’s canvas as the layer’s CompositeSource. CompositeSource grows an OffscreenCanvas member; the WebGPU compositor’s copyExternalImageToTexture and the Canvas2D fallback both take one directly, so there is no per-frame bitmap transfer. Export steps the same resolver at exact frame times, so it is deterministic. Agent frames. frames.ts gathers the model3d layers across every requested timecode, groups them by asset id plus Model3DSessionOptions, and renders each group in one headless Chromium page through a new renderGlbFramesHeadless (one launch, N frames, PNG each). The PNGs decode with @napi-rs/canvas loadImage and join the layer list like a rasterized shape. No Chrome on the host is a PreviewDegradation with reason model3d_unavailable; the layer is left out and the report says so, so an agent never mistakes a missing renderer for an empty clip. Clip frames for the agent in the browser (ui_timeline_get_clip_frames, rasterClipFrames.ts) and lane thumbnails use host 1’s source. Bake (D6) is Blender, not the session, and is the only host that draws with a different renderer. Browser and server share the core, so the tone mapping, color space and lighting match to the pixel, the way text does today. D6. Bake with Blender for final quality A “Bake” action in the inspector and a bake_model3d_clip op run nodetool.blender.RenderAnimation and store the result in model3dStyle.bake. The scene model emits a video layer from bake.assetId while bake.dependencyHash matches the live document and the live model3d layer otherwise, so a stale bake is never played silently. The clip’s existing version history holds the bakes. Time origin. A bake is the clip’s evaluated picture, clip-local: frame i of the video is what the live layer draws at timeline time clip.startMs + 1000 * i / fps, with in point, speed, time remap, animation.speed and the camera curves already applied. The scene model therefore emits the baked video layer with bakeSourceTimeSec = (timeMs - clip.startMs) / 1000, and every video decoder seeks by that field when it is present instead of clipSourceTimeSec(clip, timeMs): the live preview, the browser export, and the agent frame path’s decodeVideoFrameAt. A trimmed clip whose source starts at 5 s bakes into a video that starts at 0 s and plays from 0 s in all three. Producer. The existing job renders consecutive scene frames at the output fps, which cannot express a trimmed, sped-up, reversed or looped clip. The bake sends Blender the evaluated sample list instead: render_animation gains frame_times: number[], one model time in seconds per output frame, plus animation_name and the camera per frame. The timeline side computes the list with the same functions the live layer uses: clipSourceTimeSec at each output frame, times animation.speed, wrapped or clamped by animation.loop against the selected animation’s duration, which packages/model3d reads from the glTF’s sampler input accessors. Blender then renders one still per entry, setting the scene frame from the model time (frame_set with the subframe), the orbit camera from that entry’s camera, and the selected action as the only one playing, and the frames are muxed with ffmpeg. This is the same still-per-frame loop the transparent bake needs, so both bake formats share one op path and the H.264 branch becomes the mux of an opaque sequence. A reversed remap is a decreasing list; a loop past the end wraps in the list; neither needs anything from Blender beyond frame_set. Dependency hash. computeModel3DBakeHash(clip, sequence) covers every input the picture depends on: model3dStyle without bake, the asset id, the clip’s animations (the camera curves and the orbit preset among them), inPointMs, outPointMs, durationMs, speed, the time remap, and the sequence’s fps, width and height. A change to any of them makes the bake stale; a change to the clip’s start, track, transform, opacity, effects, mask or matte does not, because those apply to the baked layer the way they apply to any video. Output format. The sampled producer writes a PNG sequence, RGBA under film_transparent. An opaque bake muxes it to MP4/H.264 yuv420p; a transparent bake encodes it to WebM VP9 yuva420p, the alpha format packages/video-nodes/src/nodes/timeline/outputFormats.ts already declares, both through the bounded ffmpeg runner the timeline node uses. Until the transparent encode ships (T13), bake_model3d_clip refuses a transparent style and says so. A browser that cannot decode the alpha bake (the video element errors or reports no alpha) falls back to the live layer with a bake_undecodable degradation rather than drawing an opaque box over footage. Camera. Blender’s orbit is keyframed only when the model has no animation of its own (render_animation.py, the not animated and mode == "orbit" branch), so an animated model with the orbit preset would bake without its camera move. The sampled producer above removes that branch: each frame_times entry carries its own resolved ClipModel3DCamera (the style folded with the sampled channels, the same resolveModel3DCamera the live layer calls), and the op places the orbit camera per still. A static camera, the orbit preset and custom camera keyframes are all just different lists; scene mode uses the glTF camera and ignores the orbit terms. Nothing is refused for its camera. D7. Editing surface Add. AddClipMenu gets “3D model” (pick a glTF asset). assetToClipAdapter accepts model/* content types and .glb names for video and overlay lanes. Inspector. ClipModel3DSection: camera mode, the four orbit fields, a scene-camera picker and an animation picker filled from the session’s cameras and animations, loop and speed, lighting and intensity, background, and Bake with its stale state. The Keyframes section offers the camera channels. Preview. With a model3d clip selected, Alt-drag on the preview orbits (azimuthDeg, elevationDeg), Alt-wheel changes zoom, and either writes model3dStyle.camera through patchClip like the 2D transform gizmo writes transform. Lane. A filmstrip like video, cells from the layer source at a few source times. D8. Agent surface Ops in packages/timeline/src/ops/op.ts, applied in apply.ts, parsed by timeline-tool-params.ts: add_model3d_clip { assetId, trackId?, startMs?, durationMs?, style? }, set_model3d_style { target, patch }, bake_model3d_clip { target }. Both ui_timeline_edit and the edit_timeline capability reach them through the shared op union. preview_timeline_frame reports a model3d layer with the camera it drew and the animation time, next to the text a text layer reports. validate_timeline adds model3d_style_missing (a model3d clip without a style or asset) and bake_stale (a bake whose hash no longer matches, as a warning). The model3d capability suite and the timeline capability suite each gain a case; harness gate runs them on a diff under either package. Risks R1. GPU-less browsers. The session runs on WebGL; a machine without it gets the same model3d_unavailable degradation the server reports, drawn as an outlined placeholder in the preview. Nothing else in the timeline breaks. R2. Context limits. One WebGL context per active 3D layer, capped at two, over the existing video cap. A third active 3D clip is reported as a dropped layer with reason model3d_layer_cap. R3. Server preview latency. One Chromium launch per preview_timeline_frame call with 3D layers, about a second, then per-frame render under SwiftShader. Acceptable for a tool that returns a handful of frames; the report carries the render time. R4. Large models. A GLB is fetched and parsed once per asset per session pool and evicted least recently used. The inspector shows a loading state until the session resolves. R5. Bake fidelity. Blender’s lighting presets and three’s are not the same picture. The bake is the intended look, the live layer is the proxy, and the inspector says so. R6. Alpha bakes in the browser. VP9 with alpha decodes in Chromium and Firefox and not in Safari. The bake_undecodable fallback keeps the live layer on screen there, and the export still runs in Chromium. Rollout Schema, defaults, validation, and the render session with tests. Scene model kind, camera channels, orbit preset, browser layer source: a dropped GLB previews and exports. Agent frames on the server, ops and tools, capability suites. Inspector, add menu, drag-and-drop, lane thumbnails, preview orbit gesture. Opaque bakes with Blender, with the orbit-on-animated-model prerequisite. Transparent bakes through the PNG sequence and ffmpeg. Steps 1 and 2 are a shippable slice: a glTF asset on an overlay track, rendered over footage with a transparent background, its animation scrubbed with the playhead, exported.--- # Agent System Improvements Source: https://docs.nodetool.ai/plans/agent-system-improvements.html Agent System Improvements Status: shipped. Every work package here (A1 through A8) has landed, so this document is the record of what shipped rather than work to pick up. The loop it converged on is stated in docs/adr/0002-codeact-is-the-one-agent-loop.md; §6 lists where the result diverged from what was planned. Scope: eight work packages from the agent-system review, each cut into tasks an autonomous agent could take end to end. A6 (eval runs in CI) was cut from the first draft and is back in, as an on-request workflow rather than a gate — see D8. Every task names its files, its acceptance criteria, and the test that must fail before the change and pass after it. Read §1 and §2 before any task: they hold the assumptions every task relies on and the decisions no task may re-open. 0. Where the system is Three loops run agents today, and only one of them is the one the docs describe. Host Loop Constructed at Gate Cost cap Compaction Chat turn (web, Telegram, MCP) one CodeAct session over provider.generateLoop packages/websocket/src/session/chat-turn.ts:1499, loop at :1974 gateTools (:1364), mode from permission_mode none none Chat plan mode create_plan → TaskPlanner.planMultiTask, execute_plan → ParallelTaskExecutor packages/agents/src/capabilities/agents.ts:177, 303 inherited belt none none Workflow AgentNode JSON tool calls over provider.generateLoop, max_turns 100 packages/llm-nodes/src/nodes/agents.ts:1195 none none none CLI nodetool agent run Agent: TaskPlanner → ParallelTaskExecutor → TaskExecutor → CodeActExecutor → CompilerAgent packages/cli/src/commands/agent.ts:278 none none none new Agent( has no other production caller. Everything only Agent wires is therefore CLI-only: the plan approval gate (agent.ts:633-649), the security monitor (agent.ts:377), planningModel/reasoningModel (agent.ts:151-155, set by nobody), the plan cache and checkpoint store (checkpoint-store.ts, supplied by nobody, and FileCheckpointStore.load returns undefined until a loadFromDisk() that Agent never awaits), skill auto-select by word overlap (agent.ts:396-453), and the compiler. CostCappedTurnBudget (packages/runtime/src/turn-budget.ts) is correct and is wired only into the supervisor (supervisor/supervisor-agent.ts:161), app-build, and the runSubAgent/authorGraph options. Nothing user-facing passes one. generateLoop returns silently on a refused turn (base-provider.ts:1155-1157), so a consumer cannot tell a budget stop from the model ending its turn. No code in packages/agents or the websocket runner compacts or windows a transcript. Tool results are cut at 25 000 chars (constants.ts:6; the docs say 20 000). The finish nudge in CodeActExecutor carries only the last assistant message forward (codeact/codeact-executor.ts:962-970), so the model asked to call finish() cannot see the observations it computed from, and the iteration budget resets per nudge round (:900, :956). Both DAG executors dispatch in barrier rounds: every ready node starts, the round waits for the slowest, then the ready set is recomputed (parallel-task-executor.ts:255-272, task-executor.ts:149-205). The concurrency cap is per mergeAsyncGenerators call (utils/merge-generators.ts), not one semaphore, so tasks × steps × sub-agents nest to 8 × 8 × 8 provider conversations. Under Agent, maxSteps is 10 dispatch rounds per task (agent-policy.ts:33), and a step chain deeper than that fails with “step budget exhausted” (task-executor.ts:210-214). CompilerAgent stores the finish_step payload without validating it against outputSchema (compiler-agent.ts:268-312); the step executors validate on the host. On a null compiler result Agent substitutes the last task’s memory value with no error (agent.ts:768-771). Dead or duplicated: TaskPlanner.plan() and CreateTaskPlanTool (task-planner.ts:266-380, 748-825, no caller, passes {} as ProcessingContext); process-mode fan-out (task-executor.ts:344-520, unreachable because PlanBuilder never sets step.mode); StepExecutor artifact/source/control machinery (step-executor.ts:634-778, 1201-1214); TaskExecutor.parallelExecution=false and finalStepId; the skill file stubs (agent.ts:94-114, 188, 403-405); Agent.task = taskPlan.tasks[0] (agent.ts:654-656); Agent creating ~/nodetool_workspace/<ts> on every run (agent.ts:567-570); FilePlanCache and FileCheckpointStore as one class twice; isChunk/isToolCall and the uiEvents/drainUi buffer each written three or four times (step-executor.ts:1221, compiler-agent.ts:488, codeact-executor.ts:1139, task-planner.ts:650). 1. Assumptions, invariants, constraints Assumptions A-1. CodeAct is the action space the product has committed to (packages/agents/AGENTS.md § “Code-shaped orchestration is CodeAct, not a mode”; docs/codeact-design.md). No task here adds a second one. A-2. TaskPlanner + ParallelTaskExecutor + TaskExecutor are a product path, because chat plan mode runs on them. They stay. A-3. The workflow AgentNode stays a provider-native JSON tool loop. Converging it on CodeAct is a product decision outside this plan (§2, D2). A-4. No model-catalog record carries a context-window size (grep for contextWindow/context_length in packages/protocol, runtime, config finds none). A4 must not depend on one. A-5. CostCalculator.estimateTokenCostUsd answers 0 for local providers and null for a model outside the price catalog (cost-calculator.ts:179-186). “Unknown” must never read as “free” (turn-budget.ts commit rule, nodetool costs unpriced convention). A-6. Workspace files, secrets, and the permission ladder are already correct where they exist. This plan widens where they apply, not what they do. Invariants (must hold after every task) I-1. One gate ladder. Every tool call a model makes, on every host, goes through decidePermission (tools/tool-permissions.ts) via CapabilityRun.invoke or the gateTools shim. No second classification table, no host that skips it. tests/capabilities-gate-parity.test.ts keeps the two entrances equivalent. I-2. One budget object per run, shared downward. A child loop (run_subtask, start_subtask, run_search, execute_plan, a step, an AgentNode spawned by run_node) reserves against its parent’s budget, never a fresh one. I-3. A budget stop is visible. When a budget refuses a turn, the consumer receives a distinguishable signal and the user sees a message naming the limit. Silent return is a bug. I-4. Fail closed on the unknown, loud on the refusal. An unpriced model is not free; a headless host with nobody to ask does not silently allow a gated action. I-5. Step failure is terminal, not completion (packages/agents/AGENTS.md § “Step failure is terminal”). Nothing in the scheduler rewrite may treat a failed dependency as satisfied. I-6. Memory contract unchanged: context.memory is the only cross-step store, the step:/task:/input:/shared: namespaces keep their writers (packages/agents/AGENTS.md § “Who writes what”). I-7. Provider prefix stability. Per-turn volatile text goes into the last user message, never into the system message (chat-history.ts:50-60, tests/chat-prompt-stability.test.ts). Compaction must not break this. I-8. Every new check is inverted once and observed failing before it ships (AGENTS.md § “Claims, Checks, and Measurements”). Each task below names the inversion. I-9. No hand-written any, no enum, .js import suffixes, workspace imports only (AGENTS.md § TypeScript Rules). Constraints C-1. Post-change verification is exactly npm run test:affected, npm run typecheck, npm run lint, plus npm run dev:nodetool -- harness gate --base main before a PR. Adding or re-declaring a capability also requires npm run capabilities:check. C-2. A change to packages/agents/src/capabilities/*.specs.ts (a description, a schema) moves the capability’s contract fingerprint; the coverage table in packages/cli/src/harness/capability-table.ts must be synced (npm run capabilities:sync). C-3. packages/agents/AGENTS.md and docs/AGENTS.md are updated in the same PR as the code they describe. No counts, dates, or “currently”. C-4. PRs target under 400 LOC of non-test change where the task allows; the deletion tasks are the exception and say so. C-5. Nothing here touches packages/kernel execution semantics, the supervisor’s fail-closed rules, or the websocket protocol message set except where a task names the message it adds. C-6. Node 22.22.1, nvm use, and the sandboxed-install notes in AGENTS.md apply to every agent taking a task. 2. Decisions D1 and D2 are recorded with their evidence in ADR 0002. D1. CodeAct is the one product loop; Agent is retired. The CLI’s agent run becomes a single CodeAct session over the same belt chat uses, with create_plan/execute_plan reachable the way they are in chat. What Agent alone wired goes with it: CompilerAgent (synthesis is the parent loop’s next turn, which already reads task:<id> from memory), the plan approval gate (the permission gate on execute_plan is the approval), the plan cache and checkpoint store, planningModel/reasoningModel, and skill auto-select (the chat’s catalog + load_skill is the model). TaskPlanner, ParallelTaskExecutor, TaskExecutor, CodeActExecutor, SubAgentTool and the supervisor stay. Rationale: two loops means every fix lands twice or in the wrong one, and the repo already made this call once when it deleted script mode. D2. AgentNode stays a JSON tool loop but joins the shared gate (A2) and the shared budget (A1). A workflow node needs a small deterministic loop with provider-native tool calls; giving it the sandbox and the nodetool.* API is a separate product question. D3. One RunBudget per run: USD cap, wall-clock deadline, a shared concurrency semaphore, and a cumulative turn count, created by the host and threaded down through CapabilityRun/SubAgentRuntime. USD admission is the existing reserve-then-commit TurnBudget; unpriced models fall back to a token ceiling and are recorded as unpriced (never admitted as free). D4. Headless hosts gate in auto with an explicit no-human approver. A kernel workflow run is consent: the user pressed Run on a graph whose node lists its tools. A run_node from chat inherits the chat’s live mode. The CLI asks on a TTY and runs auto with a stderr notice when piped. D5. Compaction is a persisted summary message, cut at a user-turn boundary, triggered by a token estimate and by the provider’s own context-exceeded error. Not a rolling window: a window drops the decisions the user made in turn 3. D6. The DAG executors become one event-driven scheduler over a shared semaphore. Round caps go away; plan size, per-step iteration caps, and the run budget bound the work. D7. A5 is a conditional patch. D1 deletes the code A5 fixes. Ship A5 only if A3’s deletion task cannot land in the same milestone as A1. Resolved: T-A3.3 landed first, so A5 ships against the seams that inherited the defect instead of the compiler (see A5). D8. A6 is in scope, and it runs on request only. The first draft cut eval gating on cost. Cost is real — every case in every suite that matters here drives a live model — and it comes with model-run variance, so a red check would as often mean “the model had an average night” as “someone broke the planner”. Both together rule out a gate: no schedule, no PR trigger, no required check. What A6 ships is a workflow_dispatch workflow a maintainer starts after changing the planner, the CodeAct loop, the sub-agent path or a ui_* contract, with a floor per suite and a report to read. The floors are defaults a dispatch overrides, not merge conditions. Whether the eval runs that already fire on their own (app-build-eval.yml’s nightly, and the Quality Gate leg that runs app-build’s keyless cases) should also become on-request is a separate call, not made here. 3. Work packages Task ids are T-<package>.<n>. “Size” is S (one PR, under a day of agent work), M (one PR, one to two days), L (two PRs or more). “Depends” lists tasks that must be merged first. A1. Budgets: one RunBudget per run, shared downward Rationale. Today the only bounds on a chat turn are iteration counts, and those nest and reset. A user on a paid provider can lose an unbounded amount of money to a loop nobody stops. The budget primitive already exists and is tested; it is not connected. Context. packages/runtime/src/turn-budget.ts (TurnBudget, CostCappedTurnBudget), base-provider.ts:1053-1160 (generateLoop, _admitTurn), codeact/codeact-executor.ts:270-273, 900-906, 956, subagent.ts:83, 156, 465-476, capabilities/agents.ts:258-364 (execute_plan), capabilities/types.ts (CapabilityRun, SubAgentRuntime), chat-turn.ts:1318-1340, 1694, 1974-1985, packages/llm-nodes/src/nodes/agents.ts:964-966, 1195-1201, packages/config/src/setting-catalog.ts, utils/merge-generators.ts, docs/workflow-supervisor-design.md §6 (why reserve-then-commit). Target design. // @nodetool-ai/runtime export interface RunBudget { /** USD admission; reserve before a turn, commit after. */ turns: TurnBudget; /** Absolute deadline; every loop checks it before a turn and before a tool call. */ deadline: Deadline; // { at: number; remainingMs(): number; expired(): boolean } /** Process-wide bound on concurrent provider conversations for this run. */ concurrency: Semaphore; // acquire(): Promise<Release> /** Cumulative model turns across every loop in the run. */ turnCount: Counter; // max, current, increment(): boolean /** Why a stop happened, set once. */ readonly exhausted: BudgetExhaustion | null; // { kind: "cost"|"deadline"|"turns"; detail: string } } export function createRunBudget(opts: { capUsd: number | null; // null = no USD cap (local-only install) maxOutputTokens: number; // the known worst case reserve needs unpricedTokenCeiling: number; // per-turn prompt-token ceiling when the model has no price deadlineMs: number; maxConcurrency: number; maxTurns: number; }): RunBudget; CompositeTurnBudget implements TurnBudget: when estimateTokenCostUsd is a number, behave as CostCappedTurnBudget; when it is null, admit iff inputTokens <= unpricedTokenCeiling, record unpriced: true on the reservation, and log once per model. generateLoop gains an explicit stop signal. Add a ProviderStreamItem variant { type: "stop"; reason: "budget" | "deadline" | "iterations" | "aborted"; detail: string } yielded as the final item when the loop ends for any reason other than the model ending its turn. base-provider.ts:1155 yields it instead of return. Both overrides (claude-agent-provider.ts, openai-provider.ts) do the same at their admission points. The classifyProviderStream consumers (chat-turn.ts, codeact-executor.ts, llm-nodes/agents.ts) surface it as a user-visible message: chat emits an assistant-role notice “Stopped: turn budget of $X reached”; AgentNode raises a node error; the CodeAct step fails with the reason instead of “ended after N action(s) without calling finish()”. CodeActExecutor: maxIterations is the total across nudge rounds (turnsThisRound accumulates into turnsTotal; the next round is given maxIterations - turnsTotal), and exhaustedIterations reflects the total. Every action start and every bridged tool call checks budget.deadline. SubAgentRuntime carries budget: RunBudget; runSubAgent, RunSubtaskTool, StartSubtaskTool, RunSearchTool, execute_plan, and authorGraph read it from there and pass it to every CodeActExecutor they build. Agent-less after A3, so no other spawn site exists. mergeAsyncGenerators accepts a Semaphore in place of the numeric concurrency, so nested fan-outs share one bound. nodetool.batch and parallelMap inside the sandbox keep their own local concurrency (guest CPU is not a provider conversation), but every nodetool.agents.run/ start acquires the run’s semaphore. Hosts create the budget: Chat turn: from settings NODETOOL_AGENT_TURN_COST_CAP_USD (default 5.00), NODETOOL_AGENT_TURN_DEADLINE_MS (default 30 min), NODETOOL_AGENT_MAX_CONCURRENCY (default 8), NODETOOL_AGENT_MAX_TURNS (default 200), NODETOOL_AGENT_UNPRICED_TOKEN_CEILING (default 400 000), all declared in settingCatalog() so the settings page and nodetool settings show list them. A per-thread override is a follow-up. AgentNode: new props cost_cap_usd (default from the same setting) and timeout_s (default 20 min); max_turns feeds maxTurns. When the node runs inside a chat run_node, it reads the chat’s budget off the context (see A2’s context key) instead of creating its own. CLI agent run / nodetool-chat: flags --cost-cap <usd>, --timeout <s>; defaults from the same settings. Spend lands where it already lands (attachRunCostLedger); the budget adds no second ledger. budget.turns.spentUsd is reported in the run’s final log_update and in the CLI summary line. Tasks. T-A1.1 RunBudget primitive + stop signal (runtime). Size M. Files: packages/runtime/src/turn-budget.ts (add CompositeTurnBudget, RunBudget, createRunBudget, Deadline, Semaphore, Counter), providers/types.ts (the stop item), providers/base-provider.ts:1150- 1160 and the two overrides, packages/runtime/src/index.ts exports. Acceptance: a scripted provider run with capUsd below one turn’s worst case yields exactly one stop item with reason: "budget" and makes zero model calls; an unpriced model under the token ceiling is admitted and the reservation carries unpriced: true; over the ceiling it is refused. Tests: extend packages/runtime/tests/turn-budget*.test.ts; inversion: set the ceiling to Infinity and watch the over-ceiling case wrongly pass, then restore. Depends: none. T-A1.2 Thread the budget through CapabilityRun/SubAgentRuntime and every spawn site. Size M. Files: capabilities/types.ts, capabilities/invoke.ts (createCapabilityRun takes budget), capabilities/agents.ts (subAgentRuntime, execute_plan passes the parent budget and the parent maxStepIterations), subagent.ts, tools/run-subtask-tool.ts, tools/start-subtask-tool.ts, tools/run-search-tool.ts, author-graph.ts, parallel-task-executor.ts and task-executor.ts (forward budget into every CodeActExecutor), utils/merge-generators.ts (accept a Semaphore). Acceptance: a parent with capUsd X that spawns three run_subtask children spends at most X in total across all four loops (scripted provider with a fixed per-turn price); a start_subtask fan-out of 20 with maxConcurrency 2 never has more than 2 provider turns in flight (assert with a counter in the scripted provider). Tests: tests/run-subtask-tool.test.ts, tests/background-subtasks.test.ts, tests/merge-generators.test.ts, new tests/run-budget-propagation.test.ts. Inversion: give the child a fresh budget and watch the total exceed X. Depends: T-A1.1. T-A1.3 CodeActExecutor cumulative iterations + deadline checks. Size S. Files: codeact/codeact-executor.ts:885-980. Acceptance: with maxIterations 4 and two nudges, the step makes at most 4 model turns in total; a deadline expiring mid-action fails the step with reason: "deadline" and the action is aborted through the sandbox signal. Tests: tests/codeact-executor.test.ts. Inversion: restore the per-round reset and watch the count reach 12. Depends: T-A1.1. (The observation-history fix for the nudge is T-A4.4, same file; coordinate so the two PRs do not conflict.) T-A1.4 Chat turn creates the budget; settings entries. Size M. Files: packages/config/src/setting-catalog.ts (five entries), chat-turn.ts (create at :1318 next to chatGate, pass to createCapabilityRun, to generateLoop as turnBudget, to subAgentRuntime; handle the stop item at the classifyProviderStream consumer with a persisted assistant notice), packages/websocket/tests/chat-turn-test-harness.ts and a new chat-turn-budget.test.ts. Acceptance: a turn whose scripted provider prices each call above the cap ends after zero calls with a persisted assistant message naming the cap; a turn with a local (price 0) model runs unbounded by USD and bounded by maxTurns. Inversion: set the cap to 0 and assert the notice is absent, watch it fail. Depends: T-A1.1, T-A1.2. T-A1.5 AgentNode budget props and stop handling. Size S. Files: packages/llm-nodes/src/nodes/agents.ts (props at :809-905, loop at :1195), packages/llm-nodes/tests/agent-loop.test.ts. Acceptance: a node with cost_cap_usd below one turn fails with an error naming the cap; a node run under a chat context with a budget on it uses that budget (assert via the shared spentUsd). Depends: T-A1.1, T-A2.2 (the context key). T-A1.6 CLI flags. Size S. Files: packages/cli/src/commands/agent.ts, packages/cli/src/chat-codeact.ts, packages/cli/src/index.ts (flags), docs/cli.md, AGENTS.md § CLI. Acceptance: --cost-cap 0.01 on a priced model prints the stop reason and exits non-zero; help text lists both flags. Depends: T-A1.1, and lands after T-A3.2 rewrites the command (do not add flags to code A3 deletes). Out of scope. Per-thread or per-user budget overrides in the UI; moving spend reservations into the database for multi-instance deployments (the credit gate has the same limitation and documents it). Risks. R1: a stop item breaks a consumer that switches exhaustively on item type. Mitigation: isProviderSessionUpdate-style narrowers exist; add isProviderStop and grep every classifyProviderStream consumer. R2: the Claude Agent SDK override reports usage only on its terminal message, so a cap on that provider is coarse. Accept; the reserve rule already charges the worst case on an aborted session. A2. One permission gate on every host Rationale. The permission ladder is the user’s only control over what a model may do, and it is applied in one host out of four. Concretely: a chat in plan mode may call run_node on an AgentNode, whose tools resolve ungated (agent-utils.ts:497-514 → hydrateBuiltinAgentTool), so the mode that promises “no mutations” can mutate. A JS script run from chat builds its capability run with ungatedCapabilityRun (js-script-sandbox.ts:92). Context. tools/tool-permissions.ts:332-353 (category matrix), :369-401 (PermissionGateOptions), capabilities/invoke.ts (the ladder, UNGATED, createCapabilityRun), capabilities/gate-tools.ts:79, execute-agent-graph.ts:130-145 (the precedent: a child context with setInjectedTools), packages/runtime/src/context.ts:1678-1683 (setInjectedTools/getInjectedTool), :1876-1883 (get/set context variables), agent.ts:787 (PLAN_APPROVAL_CONTEXT_KEY as the pattern for a context-carried callback), chat-turn.ts:1318-1340, packages/llm-nodes/src/nodes/agent-utils.ts:490-514, agent-tool-hydration.ts:84-112, packages/execution/src/service/ workflow-workspace.ts:150-200 (buildWorkspaceExecutionContext), packages/agents/src/js-script-sandbox.ts, packages/cli/src/commands/ agent.ts, packages/cli/src/app.tsx:868-905, docs/tool-class-retirement-design.md § “Where the permission gate lives”, tests/capabilities-gate-parity.test.ts. Target design. A context key PERMISSION_GATE_CONTEXT_KEY (in @nodetool-ai/agents types.ts, beside PLAN_APPROVAL_CONTEXT_KEY) holding a PermissionGateOptions. gateFromContext(context): PermissionGateOptions returns it, or the headless gate when absent: { mode: "auto", sessionAllow: new Set(), requestApproval: headlessDeny } where headlessDeny resolves "deny" with a reason naming the host. In auto the ladder allows read/write/execute/external without asking, so the approver is reached only for a request the mode itself escalates; a headless host then denies rather than hangs (I-4). Hosts set the key: Chat sets chatGate on the turn context. run_node, run_js_script, and every AgentNode spawned under that context inherit the live mode (the getter on chatGate.mode already tracks set_permission_mode). The kernel job runner sets the headless gate in buildWorkspaceExecutionContext (D4). CLI sets a TTY gate: requestApproval prompts y / n / a (allow for this session) on stderr via readline; when !process.stdin.isTTY or --permission-mode auto, the headless gate with a one-line notice. --permission-mode <default|auto|plan> on agent run and nodetool-chat. AgentNode.buildTools wraps the hydrated tools: gateTools(normalizeTools(...), gateFromContext(context)). Control tools (buildControlTools) and the structured submit_result tool are not gated: they are graph wiring, not capabilities. js-script-sandbox.ts builds its run with createCapabilityRun({ context, gate: gateFromContext(context), … }) instead of ungatedCapabilityRun. ungatedCapabilityRun keeps exactly two legitimate users: lazy-tool.ts (the tool is gated from outside by gateTools) and packs.ts (reads a SKILL.md, a read-class call). A test pins that list. The security monitor stays an opt-in PermissionGateOptions.securityMonitor callback with no production host; A3 removes the Agent-side construction and nothing here adds one. Tasks. T-A2.1 gateFromContext + headless gate + pinned ungatedCapabilityRun users. Size S. Files: packages/agents/src/types.ts, tools/tool-permissions.ts (export headlessGate()), new capabilities/gate-from-context.ts, index.ts, new tests/gate-from-context.test.ts (also asserts, by grepping src/, that ungatedCapabilityRun is referenced only from lazy-tool.ts, packs.ts, and invoke.ts, and asserts the grep found those three so it cannot pass on nothing). Inversion: add a fourth reference and watch it fail. Depends: none. T-A2.2 Chat and kernel set the key; AgentNode and JS scripts read it. Size M. Files: chat-turn.ts:1318-1340 (one context.set), packages/execution/src/service/workflow-workspace.ts, packages/llm-nodes/src/nodes/agents.ts:916-920, packages/llm-nodes/src/nodes/agent-utils.ts, js-script-sandbox.ts:92. Acceptance (the bug this exists for): a chat turn in plan mode calling run_node on an AgentNode whose tool list includes delete_workflow gets blocked_in_plan_mode from inside the node, and the workflow row survives; the same node in a kernel job run with the same tool runs it (auto). Tests: packages/websocket/tests/chat-turn-handler-run-node.test.ts, packages/llm-nodes/tests/agent-tool-hydration.test.ts, new packages/agents/tests/js-script-gate.test.ts. Inversion: remove the gateTools wrap in buildTools and watch the plan-mode case delete the row. Depends: T-A2.1. T-A2.3 CLI TTY gate and --permission-mode. Size S. Files: packages/cli/src/commands/agent.ts, packages/cli/src/chat-codeact.ts, packages/cli/src/index.ts, docs/cli.md. Acceptance: piped input runs auto and prints the notice once; a TTY run in default mode blocks on a write tool until answered; a persists for the session. Tests: packages/cli/tests/ with a fake stdin. Depends: T-A2.1, and lands after T-A3.2. T-A2.4 Docs. Size S. packages/agents/AGENTS.md (a “Where the gate is set” table: host → mode → approver), docs/AGENTS.md § Tool System, docs/tool-class-retirement-design.md (one paragraph). Depends: T-A2.2. Out of scope. A new UI for per-node permission modes; changing what any category allows; wiring the security monitor to a host. Risks. R3: shipped example workflows with an AgentNode whose tools the headless approver would deny. Mitigation: auto allows all four categories, so only a future escalation path hits the approver; npm run validate:examples runs in the gate. R4: run_node today may run on a copied context that drops variables. Check context.copy copies variables; if not, that is a one-line fix in context.ts and part of T-A2.2. A3. Retire Agent; CodeAct is the one loop Rationale. D1. The planner→compiler pipeline has one production caller, the CLI, and carries the largest share of the dead code, the unvalidated output, and the unwired features. Keeping it means A1, A2, A4 and A7 each land in two places. Context. agent.ts (whole file), compiler-agent.ts, checkpoint-store.ts, agent-policy.ts, security-monitor.ts:1-40, prompts/security-monitor-prompt.ts, workflow-agent.ts, execute-agent-graph.ts, index.ts:132-143, 430-431, 480-501, 845-854, packages/cli/src/commands/agent.ts:225-300, packages/cli/src/useExecutionState.ts:50, 86 (reads compiler planning_updates), packages/cli/src/chat-codeact.ts (the local CodeAct turn the CLI already runs), codeact/chat-codeact.ts (createChatCodeActSession), capabilities/agents.ts (create_plan, execute_plan), packages/agents/AGENTS.md §§ “Plan Approval Gate”, “Parallel Task Execution”, “One policy per run”, “Final synthesis: CompilerAgent”, “Skills System”, “Tuning Checklist”, “Model Selection”; docs/AGENTS.md §§ Architecture, Planning, Execution, Configuration Reference; docs/agent-cli.md; docs/agent-memory.md. Tests that go with the code: tests/agent.test.ts, compiler-agent.test.ts, compiler-agent-loop.test.ts, checkpoint-store.test.ts, plan-cache-checkpoint.integration.test.ts, plan-approval.test.ts, agent-policy.test.ts, security-monitor.test.ts (keep: the gate callback survives), memory-propagation.test.ts (re-target to execute_plan). Target design. nodetool agent run = one CodeActExecutor single-step session (or the createChatCodeActSession adapter the CLI’s chat already uses; pick the one that gives execute_plan and the direct tools the same way chat gets them, which is the session adapter) over the full belt, with the objective as the user message, --json events as today, and the A1/A2 flags. The planning_ update/task_update events still stream when the model calls create_plan/execute_plan, so useExecutionState.ts keeps rendering them; its compiler phase branch is deleted. The graph branch (agent.ts graph/useGraphPlanner, workflow-agent.ts, execute-agent-graph.ts): keep executeAgentGraph and applyRunPolicy only if something outside Agent imports them (subagent.ts and app-build/build.ts import author-graph.ts, not these). Verify with a grep in the task; delete what has no importer. TaskPlanner loses plan(), CreateTaskPlanTool, maxRetries, reasoningModel. planMultiTask keeps its signature. ParallelTaskExecutor loses checkpointStore/runId; TaskExecutor loses nothing here (A7/A8 cover it). AgentPolicy: after this, execute_plan is the only consumer of the bounds. Fold maxStepIterations/maxConcurrentAgents into the RunBudget-bearing runtime (A1) and delete agent-policy.ts, or keep it as the one constants file the two executors read. Decide in T-A3.3 by whether A1 has merged: if yes, delete; if no, keep and delete in A7. Memory docs: “Final synthesis: CompilerAgent” is replaced by “the calling loop reads task:<id> via read_shared after execute_plan returns”, which is what execute_plan’s description already says. Tasks. T-A3.1 Decision record. Size S. Add docs/decisions/ entry or a section in docs/codeact-design.md stating D1 and D2 with the call graph from §0 as evidence. No code. Depends: none. T-A3.2 Rewrite nodetool agent run on the CodeAct session. Size M. Files: packages/cli/src/commands/agent.ts, packages/cli/src/chat-codeact.ts (share the session builder), packages/cli/src/useExecutionState.ts, docs/agent-cli.md, docs/cli.md. Acceptance: echo "list my workflows" | nodetool agent run -p <p> -m <m> --json emits tool_call_update events for an execute_code action and exits 0 with the final text; a plan-shaped objective can reach create_plan/execute_plan (scripted provider test asserting both appear on the belt). Tests: packages/cli/tests/agent-command .test.ts with ScriptedProvider. Depends: T-A3.1. T-A3.3 Delete Agent and what only it wired. Size L (deletion-heavy, exempt from C-4). Files: delete agent.ts, compiler-agent.ts, checkpoint-store.ts, prompts/ files only the compiler used, the Agent-side monitor construction; prune index.ts; prune task-planner.ts (plan(), CreateTaskPlanTool, maxRetries, reasoningModel, planCache), parallel-task-executor.ts (checkpointStore, runId); delete or re-target the tests listed above; grep web/src and packages/*/src for planning_update phases compile/awaiting_approval and remove dead render branches only if nothing else emits them (the create_plan path emits planning_update, so the type stays). Acceptance: grep -rn "new Agent(\|CompilerAgent\|CheckpointStore\|PlanCache" packages web electron --include=*.ts --include=*.tsx returns nothing outside git history; npm run check green; npm run backend:smoke green (the bundle must not reference a deleted module). Docs: every § named in Context above rewritten in the same PR; docs/AGENTS.md Configuration Reference table replaced by the CodeAct session options. Depends: T-A3.2 merged; A1.T2 merged or the AgentPolicy decision above applied. T-A3.4 Graph branch triage. Size S. Grep importers of executeAgentGraph, applyRunPolicy, resolveAgentGraph, runWorkflowAsAgent; delete the unimported, keep the rest with a one-line note in packages/agents/AGENTS.md naming the importer. Depends: T-A3.3. Out of scope. Making AgentNode run CodeAct; adding a model argument to create_plan/execute_plan (the per-phase-model idea survives only as a follow-up note on those two specs). Risks. R5: a downstream consumer (web, mobile, Telegram) renders a planning_update phase only Agent produced. Mitigation: the grep in T-A3.3; the protocol type is not narrowed. R6: the CLI agent run loses the “plan first, then execute” shape a user relied on. Mitigation: the objective can say “plan first”, and --permission-mode plan gives exactly the chat’s plan mode. A4. Transcript compaction and observation continuity Rationale. A stateless provider is sent the whole thread every turn. A long creative session (hundreds of tool results at up to 25 000 chars each) hits the model’s context limit and the turn fails with a provider error the user cannot act on. Inside a step, the finish nudge asks the model to finish from a transcript that no longer contains what it computed. Context. chat-turn.ts:1160-1262 (history load, the session probe, loadFullHistory), :1702-1745 (messagesToSend, appendContextToLastUser), :1974-1985 (generateLoop call), session/chat-history.ts (role filter at :40-50, repairOrphanedToolCalls, the prefix-stability note at :50-60), packages/models/src/message.ts (role, content, execution_event_type, provider_session), base-provider.ts:260 (estimatePromptTokens, currently module-private), anthropic-provider.ts:1362, 1469 (model_context_window_exceeded), codeact/codeact-executor.ts:940-980 (the nudge), constants.ts:6 (MAX_TOOL_RESULT_CHARS), tests/chat-prompt-stability.test.ts, packages/websocket/tests/session- chat-history.test.ts, packages/websocket/tests/websocket-client-session- chat-resume.test.ts (the session-probe resume this must coexist with). Target design. Compaction record. A persisted Message row, role: "user", execution_event_type: "compaction", content: "[Conversation so far]\n<summary>". History assembly starts at the newest compaction row: Message.paginate from that row forward, then the same convertDbMessages path. Older rows stay in the DB for the UI and for nodetool.threads.*; only the provider view is cut. A compaction row is itself ordinary history afterwards, so the cached prefix is stable (I-7). Cut point. Always a user-message boundary, never between a tool call and its result. Keep the last K user turns verbatim (K = 4, a setting); summarize everything before them. Trigger. Two, both in chat-turn.ts before generateLoop: (1) estimatePromptTokens(messagesToSend) > NODETOOL_CHAT_COMPACTION_TOKENS (setting, default 120 000; A-4 says no catalog value exists to derive it from); (2) reactive: the provider ends with a context-exceeded signal (Anthropic stop reason; OpenAI/Gemini error classes the provider maps to a ProviderError with code: "context_exceeded", add the mapping) → compact, then retry the turn once. Providers that hold the transcript server-side (priorSession set, or the Claude Agent SDK) skip trigger (1); they get trigger (2) only. Summarizer. One generateMessage call on the turn’s own provider/model (or NODETOOL_COMPACTION_MODEL when set), with a fixed prompt that must preserve, as a bulleted list: the user’s standing goals and constraints, decisions taken, every artifact reference verbatim (asset://, workflow and document ids, file paths), open questions, and the last tool results’ conclusions. The summary is capped at 4 000 tokens. A failed summarizer call leaves the thread uncompacted and the turn proceeds (fail open on the summary, since the alternative is a turn that cannot run), with a log_update warning. User visibility. The web renders the compaction row as a collapsed “Earlier conversation summarized” card (one component; the row already reaches the client as a message). A log_update names the token estimate that triggered it. Redaction. The summarizer input goes through the same redaction the provider debug dump uses (base-provider.ts:351 “secrets redacted”) before it leaves the process. Step-level (CodeAct). Collect every ProviderMessageEvent of a round into history before pushing the nudge, so the re-prompted model sees its own observations. No summarization inside a step: step transcripts are bounded by A1’s cumulative iteration cap and the 25 000-char observation cut. Align MAX_TOOL_RESULT_CHARS docs with code. Tasks. T-A4.1 Export the token estimator and add the context-exceeded error class. Size S. Files: base-provider.ts:260 (export estimatePromptTokens), providers/types.ts (ProviderError.code "context_exceeded"), anthropic-provider.ts, openai-provider.ts, gemini-provider.ts (map their signals). Tests: one fixture per provider in the contract-probe offline suite pattern (packages/runtime/tests/provider-contract-probes*), asserting the mapped code. Inversion: unmap one and watch it fail. Depends: none. T-A4.2 Compaction record + history assembly from it. Size M. Files: packages/models/src/message.ts (no schema change if execution_event_type suffices; otherwise a Drizzle migration for a compaction boolean), session/chat-history.ts (historySince Compaction(rows)), chat-turn.ts:1160-1262 (integrate with the session probe: a compaction row newer than the probe session wins). Acceptance: a thread with a compaction row at position 40 of 60 sends the system message, the compaction row, and rows 41–60; no tool call is orphaned. Tests: session-chat-history.test.ts, websocket-client-session-chat-resume.test .ts. Inversion: cut at a tool-result boundary and watch repairOrphanedToolCalls change the count. Depends: none. T-A4.3 Triggers, summarizer, retry-once, settings. Size M. Files: chat-turn.ts (before :1974), new session/chat-compaction.ts (summarizer prompt, cut-point logic, redaction), setting-catalog.ts (NODETOOL_CHAT_COMPACTION_TOKENS, NODETOOL_CHAT_COMPACTION_KEEP_TURNS, NODETOOL_COMPACTION_MODEL). Acceptance: a scripted thread over the threshold produces exactly one compaction row whose content contains every asset:// uri from the summarized region, then the turn runs against the shortened history; a scripted provider raising context_exceeded once causes one compaction and one retry, twice causes a surfaced error. Tests: new packages/websocket/tests/chat-compaction.test.ts; chat-prompt-stability.test.ts extended to assert the system message hash is unchanged across a compaction. Inversion: drop an asset:// uri from the fake summary and watch the assertion fail. Depends: T-A4.1, T-A4.2. T-A4.4 CodeAct nudge keeps observations. Size S. Files: codeact/codeact-executor.ts:940-980. Acceptance: with a scripted provider, the second round’s request contains the first round’s tool-result messages. Tests: tests/codeact-executor.test.ts. Inversion: push only lastAssistant and watch it fail. Depends: none; coordinate with T-A1.3. T-A4.5 Web card for the compaction row + docs. Size S. Files: one component under web/src/components/chat/ using ui_primitives, wired where message rows are rendered by execution_event_type; packages/agents/AGENTS.md § “Tool Result Truncation” rewritten (25 000, and the compaction rule); docs/AGENTS.md. Depends: T-A4.3. Out of scope. Compacting inside a CodeAct step; deriving the threshold from a per-model context window (a follow-up once the catalog carries one); compaction for the AgentNode (its history input is caller-owned). Risks. R7: the session-probe resume and the compaction row disagree about where history starts. Mitigation: T-A4.2 defines the precedence (newest marker wins) and the resume test covers both orders. R8: a summary loses a constraint the user stated in turn 2. Mitigation: the fixed prompt’s required sections and the asset:// invariant test; K verbatim turns; the user can still scroll the full thread. A5. Validate the deliverable Rationale. Agent.getResults() was the one output a CLI caller read, and the only place in the pipeline where a schema was declared and not checked. Outcome. T-A3.3 landed first, so D7’s condition resolved against the original patch: compiler-agent.ts and agent.ts are deleted and there is no compiler finish_step closure to validate. The first acceptance criterion is met by the code that survived — every deliverable now completes through validateAgainstSchema (utils/json-schema-validate.ts), reached from StepExecutor.validateResultPayload and the CodeAct finish() bridge, and a violation round-trips to the model as an error result bounded by maxIterations, MAX_FINISH_NUDGES, and MAX_REPAIR_ROUNDS. The second criterion did not survive the deletion. A null deliverable stayed silent at three seams, each reported as success: The fan-out aggregate. task-executor.ts process mode fills new Array(n) from step_result messages, so a failed item leaves a hole, and the array is stored and the step marked complete regardless. A step declaring both perItemSchema and outputSchema has its own outputSchema checked by nobody: the per-item executor applied the other one. execute_plan’s results map. capabilities/agents.ts reads each task’s deliverable with if (value !== undefined) results[task.id] = value and still reports status: "completed". This is the seam that replaced the compiler, and its description tells the model to write the answer from results that may not be there. nodetool agent run’s exit code. The command’s one output is a transcript scrape; when it finds no final assistant text it writes nothing to stdout and exits 0. Task. T-A5.1 Size S. Carried to the seams above, since D7’s original target is deleted. Acceptance: a fan-out with a failed item fails the step naming the missing indexes, and an aggregate violating a declared outputSchema fails with the violations; execute_plan marks a task whose task:<id> key was never written with result_missing; agent run exits non-zero and names the reason when it produces no answer. Inversion: each check written and observed failing before the fix. Depends: T-A3.3. A6. Eval runs in CI, on request Rationale. packages/agents/src/evals/ scores exactly what the packages here change — the graphs the planner authors, the CodeAct action loop, delegation to sub-agents, the ui_* tool contracts — and reaching any of it means remembering a command, a provider, a model and a threshold. Running the suites from CI writes all four down and turns the result into an artifact anyone can open. They are run on request. A case that drives a model costs real inference and varies run to run, so an automatic trigger buys a check that is expensive when it passes and ambiguous when it fails (D8). One dispatch, rich enough inputs to narrow it, and a report. Context. .github/workflows/agent-eval.yml (this package’s deliverable), .github/workflows/app-build-eval.yml (the report-only shape it borrows from), .github/actions/setup-build, packages/cli/src/commands/eval.ts (EVAL_SUITES, evalGateFailures, the --min-success/--min-score gates), packages/cli/src/commands/jtbd.ts (--min-achieved), packages/agents/src/evals/, packages/cli/tests/eval-command.test.ts. Which cases can run for free is a property of the suite, and it decides what a maintainer can dispatch without spending anything: nodetool eval <suite> --list marks them, and --keyless runs exactly those. Today the whole keyless set is app-build’s deterministic cases, which author from a script rather than a model (scriptedAuthorProvider, evals/app-build-eval.ts) and run on the real kernel. Every other case in every other suite calls provider.generateMessages/generateLoop and fails with `Could not reach ` when there is none — including the `graph-e2e` cases whose answers are pinned, which are deterministic in their *checks*, not in their planning. **Tasks.** - **T-A6.1 The on-request workflow.** Size S. `.github/workflows/agent-eval.yml`: `workflow_dispatch` only, the suites that need a provider plus `jtbd`, one floor per suite recorded next to the reason for the number, per-suite JSON uploaded, and one table across all of them in a summary job. Inputs pick the suites, the provider, the model, the cases, and a floor that overrides every suite's default for one run. `--min-cases 1` everywhere, so a run whose cases were all skipped reads as a run that examined nothing rather than as a quality failure. Acceptance: no `schedule` and no `pull_request` trigger; a suite below its floor reds its own leg and no other; an unselected suite writes nothing and stays green; the summary renders a row for a suite that produced no report. Inversion: run the workflow's own step script with a floor the run cannot meet and observe the non-zero exit, and again with a selection whose every case skips, to see `--min-cases` name it. Depends: T-A6.2. - **T-A6.2 Selection and guards in `nodetool eval`.** Size S. Two flags the workflow needs and a maintainer wants. `--keyless` runs the cases a suite declares free of key and network, asking the suite (`keylessCaseIds`) instead of naming ids at the call site, and is refused by name on a suite that has none. `--min-cases ` fails a run that examined fewer cases than that — the one failure a rate cannot express, since every suite reports 0 over an empty set and 100% over the survivors of a set that shrank. `--list` marks the keyless cases. Acceptance: `evalGateFailures` reports the shortfall in its own words before any rate line; the suite registry is the only place case ids live. Inversion: a unit case for a run whose every rate reads green and whose case count fell, observed failing first. Depends: none. - **T-A6.3 Docs.** Size S. This section, plus `AGENTS.md` § "nodetool eval" and `packages/agents/AGENTS.md` § "Eval suite" saying the model-driven suites run on request from `agent-eval.yml` and gate nothing. No counts, no thresholds copied into prose — the workflow holds the numbers and `--list` holds the case sets. Depends: T-A6.1, T-A6.2. **Out of scope.** - A PR gate over the model-driven suites, and a schedule for them (D8). - The eval runs that already fire on their own: `app-build-eval.yml`'s nightly and the Quality Gate leg that runs `app-build`'s keyless cases. Both are left exactly as they are; whether they should also become on-request is the owner's call to make, not this package's. - Raising a floor on the strength of one run. A default moves when the artifact history says the suite has held above it, and the comment carrying the number moves with it. - A floor on `nodetool jtbd`. It rides the same dispatch, because it asks the same kind of question one rung up, but a job's finding is its transcript and the friction pass over it — which names an owner per signal and wants a person reading it. `--min-achieved` is there for when a baseline says what the number should be. **Risks.** - **On request means it can go unrun.** Nothing here prevents that, and no automation is allowed to. What the workflow can do is make the run cheap to start and the report worth opening; the task's acceptance is about the report, not about coverage. - **A floor set too tight becomes noise** — and on a dispatch-only workflow, noise is worse than elsewhere, because a red run that nobody had to start is a red run nobody will start again. Loose defaults, raised against recorded history. - **A keyless case that stops being keyless** does not fail loudly: it is skipped, and the survivors still pass. `--min-cases` is the whole answer to that, which is why T-A6.2's inversion covers it and not just a broken case. --- ### A7. Event-driven scheduling over one semaphore **Rationale.** Barrier rounds make a plan as slow as its slowest sibling in every round, and the round cap turns depth into failure. With A1's budget the work is bounded without counting rounds. **Context.** `parallel-task-executor.ts:232-272, 322-340, 491-524`, `task-executor.ts:149-215, 253-268 (per-step tools), 272-287 (blocked dependents)`, `utils/merge-generators.ts`, `agent-policy.ts`, `tools/plan-builder-tools.ts:185-215` (the DAG is already validated on the way in), `packages/agents/AGENTS.md` §§ "Parallel Task Execution", "Concurrency Defaults", "Step failure is terminal", `tests/parallel-task- executor.test.ts`, `task-executor*.test.ts`, `merge-generators.test.ts`. **Target design.** ```ts // utils/dag-scheduler.ts export interface DagNode { id: string; dependsOn: readonly string[] } export interface DagScheduleOptions<N extends DagNode, E> { nodes: readonly N[]; run: (node: N) => AsyncGenerator; // the node's event stream settle: (node: N, outcome: "ok" | "failed", error?: string) => E[]; // terminal events concurrency: Semaphore; // A1's shared semaphore signal?: AbortSignal; onBlocked: (node: N, by: N) => E[]; // dependents of a failure } export function scheduleDag<N extends DagNode, E>(opts): AsyncGenerator; ``` - Starts every node whose deps are satisfied, each under the semaphore; on a node's completion, marks it, computes the newly ready set, starts them immediately; a failed node fails its transitive dependents with the blocking id (I-5); ends when every node is settled. Built on a dynamic merge (`createDynamicMerge()` with `add(gen)`/`close()`, extracted from `mergeAsyncGenerators`, which becomes a thin wrapper over it). - `ParallelTaskExecutor.execute` = `scheduleDag` over tasks with `run` = `executeTask`; `TaskExecutor.execute` = `scheduleDag` over steps with `run` = a `CodeActExecutor`. `getExecutableTasks`, `getExecutableSteps`, `failBlockedTasks`, `failBlockedSteps`, `allTasksComplete`, `allStepsSettled`, `maybeDeferFinishStep` (re-expressed as a dependency of the finish step on every other step, added at plan build), and the round loops are deleted. - Caps: `DEFAULT_MAX_TASK_ITERATIONS` (100 rounds) and `maxSteps` (rounds per task) are removed; `maxStepIterations` per step stays; the run budget bounds the rest. A cycle cannot reach the scheduler (PlanBuilder rejects it), and the scheduler still asserts progress: a state with unsettled nodes and nothing running or ready fails them with "unsatisfiable dependency". - Failure semantics unchanged: a real exception from a node generator propagates after the other running generators are drained, as today. **Tasks.** - **T-A7.1 `createDynamicMerge` + `scheduleDag`, pure and tested.** Size M. Files: new `utils/dag-scheduler.ts`, `utils/merge-generators.ts`. Tests: new `tests/dag-scheduler.test.ts` with fake generators and a controllable clock: (a) a 3-node chain behind a slow sibling starts node 2 before the sibling finishes; (b) semaphore of 2 over 10 ready nodes never exceeds 2 running; (c) a failed node fails exactly its transitive dependents, naming it; (d) abort mid-run settles everything and returns the child generators; (e) 20 000-node chain finishes in bounded time (the O(n·m) trap in `AGENTS.md`). Inversion for (a): reinstate a barrier and watch it fail. Depends: T-A1.1 (the `Semaphore`). - **T-A7.2 Port `TaskExecutor`.** Size M. Files: `task-executor.ts`, `tests/task-executor*.test.ts` (keep every assertion about failure and blocked dependents; delete the round-count assertions). Acceptance: the existing step-failure and coverage suites pass; a step chain of depth 15 completes (it fails today under `maxSteps` 10). Depends: T-A7.1; A8's process-mode deletion (T-A8.2) should land first so the port carries no dead branch. - **T-A7.3 Port `ParallelTaskExecutor`, remove round caps, update `execute_plan`.** Size M. Files: `parallel-task-executor.ts`, `capabilities/agents.ts:303-317` (pass the budget's semaphore and `maxStepIterations`), `agent-policy.ts` (delete if T-A3.3 left it), `tests/parallel-task-executor.test.ts`, `packages/agents/AGENTS.md` §§ above. Acceptance: two independent tasks of unequal length — the dependent of the short one starts before the long one ends (scripted provider with per-step delays). Depends: T-A7.1, T-A7.2, T-A3.3. **Out of scope.** Work stealing across runs; priority; re-planning after a failure (a design question of its own; the executor exposes the failed set and `execute_plan` already returns it). **Risks.** R9: an event-ordering assumption in a UI test (`task_update` before the first `step_update` of a task). Mitigation: `settle` and the start events are emitted by the scheduler in the same order the round loop emitted them; the parity is asserted in T-A7.3 against a recorded stream of the old executor for one fixture plan. --- ### A8. Delete the dead paths, consolidate the duplicates **Rationale.** Every dead branch is a place a fix can land without effect and a reader can mistake for a live contract. Do it before A3 and A7 so those ports carry nothing they do not need. **Context.** §0's list; `packages/protocol` for whether `Step.mode` / `perItemInstructions` are protocol types (if they are, leave the type and delete the executor branch and the planner prompt text that mentions fan-out modes); `packages/llm-nodes/src/nodes/agents.ts` already consumes a `classifyProviderStream` helper (find its module first); that helper or the narrowers beside `isProviderSessionUpdate` in `runtime/providers/types.ts` are the target for the `isChunk`/`isToolCall` copies; `subagent.ts:104` (`settleStepResult`) as the one failure-shape detector. **Tasks.** Each is one PR; each PR's description lists the deleted symbols and the grep that proves no importer remains (the grep output is the evidence, C-1's "I checked means you enumerated"). - **T-A8.1 Planner dead path.** Size S. Delete `TaskPlanner.plan()`, `CreateTaskPlanTool`, `maxRetries`, the legacy `plan` prompt strings, and their tests; `MAX_RETRIES` doc row in `packages/agents/AGENTS.md` § "Concurrency Defaults". Depends: none. - **T-A8.2 Process-mode fan-out.** Size S. Delete `task-executor.ts:344-520` (`handleProcessStep` and helpers), the `discover`/`process`/`aggregate` table in `docs/AGENTS.md` § "Fan-Out Execution", the planner prompt's mention if any; keep protocol types if shared (state which in the PR). Acceptance: `tests/task-executor-coverage.test.ts` cases for process mode deleted, everything else green. Depends: none. Lands before T-A7.2. - **T-A8.3 `StepExecutor` residue.** Size S. Delete the artifact/source/ control machinery (`step-executor.ts:634-778, 1201-1214`) and the `getSources`/`getControlEvents` API; confirm `SupervisorAgent` and app-build spec still construct it and run their suites. Depends: none. - **T-A8.4 `TaskExecutor` flags.** Size S. Delete `parallelExecution` (both callers pass `true`) and `finalStepId` (no caller passes it). Depends: none; or fold into T-A7.2 if that lands first. - **T-A8.5 Stream-item and UI-buffer helpers.** Size S. One `isChunk`/`isToolCall`/`isProviderStop` set in `@nodetool-ai/runtime` `providers/types.ts` (narrowers next to `isProviderSessionUpdate`); one `createUiEventBuffer()` in `utils/`; `settleStepResult` replaces `isErrorResult` (`parallel-task-executor.ts:599`) and `detectTaskFailure` (`:448`). Acceptance: `grep -rn "function isChunk\|function isToolCall" packages/agents/src` returns one hit in runtime and none in agents. Depends: none. - **T-A8.6 Docs drift in the same sweep.** Size S. `packages/agents/AGENTS.md` § "Interactive Commands" (`/agent — Toggle agent mode` describes a no-op), § "Tool Result Truncation" (20 000 vs 25 000), `docs/AGENTS.md` Configuration Reference (`maxStepIterations` 15 vs 10 across the two docs; after A1 both read from one constant and the docs point at it). Depends: none. The items §0 lists that belong to `Agent` (skill file stubs, `Agent.task` compat, the unused workspace directory, `FilePlanCache`/`FileCheckpointStore`) are deleted by T-A3.3, not here. --- ## 4. Sequencing ``` T-A8.1 T-A8.2 T-A8.3 T-A8.4 T-A8.5 T-A8.6 (independent, land first) T-A2.1 ──► T-A2.2 ──► T-A2.4 T-A1.1 ──► T-A1.2 ──► T-A1.4 └─► T-A1.3 T-A1.5 (needs T-A2.2) T-A4.1 ┐ T-A4.2 ┴─► T-A4.3 ──► T-A4.5 T-A4.4 (independent; coordinate with T-A1.3) T-A3.1 ──► T-A3.2 ──► T-A3.3 ──► T-A3.4 │ T-A1.6, T-A2.3 (CLI flags, after the rewrite) T-A1.1 ──► T-A7.1 ──► T-A7.2 (after T-A8.2) ──► T-A7.3 (after T-A3.3) T-A5.1 after T-A3.3 (retargeted; see A5) T-A6.2 ──► T-A6.1 ──► T-A6.3 (CI + one CLI flag pair; no runtime code) ``` Parallel lanes for four agents: (1) A8 then A7; (2) A1; (3) A2 then the CLI tasks; (4) A4. A6 shares nothing with them — workflow files and one CLI flag — so it rides whichever lane is free. A3 sits on lane 3 after A2 or on its own lane; T-A3.3 is the merge point everything else must not race, so schedule it when lanes 1 and 2 have merged their runtime changes. ## 5. Per-PR verification Every PR: ```bash nvm use npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base main ``` Plus, where the task says so: `npm run capabilities:check` (any `*.specs.ts` change), `npm run backend:smoke` (T-A3.3, T-A8.5, anything touching module graphs), and the inversion named in the task, with its failing output pasted into the PR's Verification section per `.github/pull_request_template.md`. A task is done when its acceptance criteria are covered by a test that was observed failing before the change, the docs named in its Context are updated, and the PR's verification section shows the four commands green. ## 6. Divergences from the plan Where the landed code differs from what §3 specified. Each is the code's behaviour, not a task left open. - `mergeAsyncGenerators` was deleted rather than made a thin wrapper over the scheduler; `packages/agents/src/utils/dag-scheduler.ts` is the one merge. - The finish step's dependency on its siblings is added where the executor builds its scheduler nodes (`stepsFinishWaitsFor`, `packages/agents/src/task-executor.ts`), not where the plan is built (`packages/agents/src/tools/plan-builder-tools.ts`). - The kernel job runner sets no permission-gate key on the context, so a workflow-run agent takes the headless fallback in `packages/agents/src/capabilities/gate-from-context.ts`. - `context_exceeded` is a `ProviderFailureDetail` code (`packages/runtime/src/providers/provider-error.ts`), not an error class of its own. - The run's semaphore bounds task permits only; each task's steps and its sub-agents fall back to the executors' own bound (`packages/agents/src/constants.ts`, `packages/agents/AGENTS.md` § Executor defaults). - The scheduler that replaced `mergeAsyncGenerators` keeps admitting queued generators after a sibling throws, as `mergeAsyncGenerators` did.--- # Applications as the Only Mini-App Resource Source: https://docs.nodetool.ai/plans/applications-canonical.html Applications as the Only Mini-App Resource Recipe Applications: a Recipe is an Application carrying a versioned recipe manifest in its ApplicationDocument. It is not a separate resource/table. The manifest describes catalogue metadata, inputs, semantic operation intents, outputs, preservation rules and media policy; executable truth remains the ApplicationDocument operation bindings and their pinned workflow/script versions. Because the manifest lives inside the document, Application versions/releases and ApplicationBundle export/import freeze and round-trip the exact Recipe metadata automatically. List/catalogue APIs project isRecipe, recipeSlug and recipeCategory without executing the app. Status: draft v2, 2026-07-26. Scope: retire workflows.app_doc, make the applications table (added 2026-07-25, PR #4479) the single place a mini app lives, and keep apps and workflows orthogonal in the UI: apps get their own sidebar panel and their own tabs, and a workflow bound to an app opens only when the user asks for it from the app surface. A new bundle format ships an app together with its workflows, and the shipped examples become a curated set of app bundles instead of one generated form per template. v2 supersedes the v1 draft: the “view any workflow as an app” on-ramp is dropped, and the plan gains the bundle format and the examples overhaul. Why A mini app is not a workflow. The document schema already says so: ApplicationDocument.operations is a list where every operation carries its own workflowId, so one app can bind several workflows (or the same workflow twice with different mappings). Storing the document on workflow.app_doc collapses that into 1:1, gives the app no identity, no version history, no publish/release cycle, no budget, and no telemetry — all of which exist on the applications table and none of which make sense on a workflow row. The UI inherited the same conflation: the app builder is routed by workflow id (/app-builder/:workflowId), the workspace View tab toggles a workflow into an app, and viewing a bare workflow silently writes a generated app_doc onto it. Users never chose to have an app; every workflow grew one. The result is ~40 shipped templates whose apps are copies of the same form, and a live fork bug: CreateApplicationFromWorkflowButton copies an app_doc into an application row and nothing syncs or marks either copy as canonical afterward. Current state Two storage paths over one document type: workflow.app_doc — reached by the web builder route (/app-builder/:workflowId), the /miniapp/:workflowId runtime and the workspace View tab, mobile (read-only), the CLI nodetool app debug, the REST workflow API, the ~40 shipped templates (scripts/generate-template-apps.mjs), the app-preview/marketing pipeline, and the agent ui_app_* tools. The hostWorkflowId option on parseApplicationDocument exists solely to serve this storage. applications + application_versions — web + tRPC only (packages/websocket/src/trpc/routers/applications.ts): sidebar Apps panel, workspace application tab, Design/Run/Settings surface, publish with pinned workflow_graphs, budgets, invocation metering in unified-websocket-runner.ts. Target state Storage: the applications table is the only persisted home for an ApplicationDocument. workflow.app_doc is accepted on import as a legacy field, lifted into an application row, and never written again. UI orthogonality: no workflow/app switch anywhere. The Apps sidebar panel is where apps are listed and opened — one workspace tab per app. A workflow never presents itself as an app, and an app tab never morphs into a workflow editor. The workflows an app binds are loaded in the background (graphs prefetched for the app’s operations) and surface only through a Linked workflows menu on the app surface; clicking one opens a normal workflow tab. Creation is explicit: an app exists because a user (or an example install) created one — “New app” in the Apps panel, or “Create app from workflow” as a one-way scaffold. No auto-generated per-workflow app. Portability: an ApplicationBundle JSON format serializes an app together with the full graphs of every workflow it binds, so apps ship, export, and import as one artifact. Examples: shipped example apps are curated bundles — each one a real use case carrying one or more workflows. Workflow templates without a good app use case ship no app at all. The ApplicationBundle format One JSON document (zippable for assets), defined in packages/app-runtime: { "schemaVersion": 1, "app": { /* ApplicationDocument — operations reference workflows[].key */ }, "workflows": [ { "key": "draft", "name": "...", "graph": { "nodes": [], "edges": [] } } ] } Inside a bundle, OperationBinding.workflowId holds a bundle-local key; import creates real workflows and rewrites keys to the new ids (the same indirection .nodetool bundles use for bundle:// asset refs). Asset-carrying bundles reuse the existing .nodetool zip machinery: the bundle JSON rides alongside the asset entries, refs stay bundle://. Export pins what application_versions.workflow_graphs pins today — a released app exports reproducibly. Tasks Each task is scoped for one subagent: it names its files, has a checkable outcome, and states its dependencies. Tasks in the same phase are independent and can run in parallel. Every task ends with npm run check green and, where it touches a package with tests, tests for the new behavior. Phase 0 — decisions locked in this doc No agent work; recorded so tasks below don’t re-litigate them. Apps and workflows are orthogonal resources with orthogonal UI. There is no workflow→app view, toggle, or tab. The workspace View tab and the auto-generated app_doc are removed, not made ephemeral (supersedes v1). Application id is the identity on every surface (tabs, routes, agent tools, CLI, API). Workflow-scoped app routes (/app-builder/:workflowId, /miniapp/:workflowId) go away; a redirect covers old links where cheap. ApplicationBundle (above) is the interchange format for examples, export, and import. Examples ship as bundles, not as example.app_doc. workflow.app_doc is deprecated on day one, written by nothing after Phase 2, and dropped in Phase 5. Phase 1 — foundations (parallel) T1. Lift-on-read helper in app-runtime Add liftLegacyAppDoc(workflow) → ApplicationDocument | null to packages/app-runtime: wraps parseApplicationDocument with hostWorkflowId, returns a document ready to insert as an application row. Single implementation shared by the server import path, the migration, and the CLI. Files: packages/app-runtime/src/document.ts (+ tests in __tests__/). Done when: unit tests cover schemaVersion-3 docs, legacy {version,data} docs, and empty-workflowId operations getting the host id. T2. Server: applications REST surface The applications router is tRPC-only. Add REST routes mirroring the tRPC procedures external clients need: list, get, create, update, delete, releasedDocument. Reuse the router’s handlers; do not duplicate logic. Files: packages/websocket/src/http-api.ts, packages/protocol/src/api-schemas/applications.ts (Zod contract exists), tests in packages/websocket/src/__tests__/. Done when: REST and tRPC return identical shapes for the same app. T3. ApplicationBundle format Implement the schema, parser, serializer, and key-rewriting in packages/app-runtime (new src/bundle.ts), with the Zod contract in packages/protocol. Includes bundleFromApplication(app, graphs) and applyBundle(bundle) → { app, workflows } (pure — persistence stays in the server/CLI callers). Files: packages/app-runtime/src/bundle.ts + tests, packages/protocol/src/api-schemas/applications.ts. Done when: round-trip test — export an app binding two workflows, import it, operations point at the newly created workflow ids. T4. Docs: rewrite the mini-apps pages docs/mini-apps.md still says the document “lives on workflow.app_doc” and never mentions the applications table. Rewrite docs/mini-apps.md, docs/mini-apps-guide.md, docs/mini-apps-reference.md, and the stale parts of docs/app-builder.md around the target state: apps are their own resource, opened from the Apps panel, linked workflows on demand, bundles for sharing. Follow docs/WRITING_STYLE.md. Done when: no doc presents app_doc as where apps live or describes a workflow/app toggle. Phase 2 — UI orthogonality (parallel, after Phase 1) T5. Apps panel is the front door Make the existing sidebar Apps panel (web/src/components/applications/ApplicationListPanel.tsx) the only entry point: list, search, “New app”, “Create app from workflow” (explicit scaffold), each app opening one application workspace tab (WorkspaceTabsStore already has the tab type). Remove the workflow-keyed entry points: the /app-builder/:workflowId route and container (AppBuilderPage.tsx), the workspace View tab, and WorkflowAppView.tsx’s generate-and-save path. /miniapp/:workflowId redirects to the app tab when a backing application exists, else 404s. Files: web/src/components/appbuilder/ (delete/reduce AppBuilderPage, WorkflowAppView, generateAppDoc, persistence.ts), web/src/index.tsx routes, web/src/stores/WorkflowManagerStore.ts (drop app_doc from save payloads), PanelLeft.tsx, Jest tests. Done when: no web code path writes app_doc and no route keys an app by workflow id. T6. App surface: linked workflows in the background On the app tab (ApplicationSurface.tsx), prefetch the graphs of every workflow the document’s operations bind (TanStack Query, enabled on tab focus) without opening editor tabs. Add a Linked workflows menu listing them (name, operation(s) using it, released-pin status); clicking opens a normal workflow tab via the workspace tab store. Broken links (deleted workflow) render as errors in the menu, not crashes. Files: web/src/components/workspace/ApplicationSurface.tsx, a new LinkedWorkflowsMenu component (ui_primitives only), hooks in web/src/hooks/useApplications.ts, tests. Done when: opening an app never opens a workflow tab; the menu opens one on click; graphs are fetched once and cached. T7. Agent tools: key ui_app_* by application id The Puck agent bridge keys editors by workflow id because workflow id is currently the app identity. Move the tool contract to application id (no workflow-id fallback — orthogonality applies to agents too; the scaffold tool covers “make an app for this workflow”). Files: web/src/lib/tools/builtin/puck.ts, web/src/components/appbuilder/puck/puckAgentBridge.ts, puck/PuckAgentBinder.tsx, web/src/lib/chat/uiContext.ts; update the app-tools eval suite in packages/agents/src/evals/. Done when: nodetool eval app-tools passes against the new contract. Phase 3 — bundles everywhere (parallel, after Phase 1; T8 also after T2) T8. Server + CLI: bundle export/import API: GET /api/applications/:id/export-bundle, POST /api/applications/import-bundle (creates the app and its workflows, rewrites keys via T3). CLI: nodetool apps export-bundle <id> -o my.app.json and nodetool apps import-bundle my.app.json, direct-DB like the existing workflows export-bundle. Editor command-menu entries mirror the workflow bundle ones. Depends on: T2, T3. Files: packages/websocket/src/http-api.ts, packages/cli/src/commands/ (new apps.ts), web command menu, tests. Done when: export → import on a clean DB reproduces a working app with its workflows. T9. CLI: nodetool app debug accepts application id and bundle files Today the harness loads “the app_doc on a workflow”. Accept an application id (models layer, no server) or an ApplicationBundle JSON file; keep workflow-id input as a legacy path that lifts via T1. Multi-operation apps already work in the harness — this is a loader change. Depends on: T1, T3. Files: packages/cli/src/commands/app.ts, packages/cli/src/app-debug/app-spec.ts, debug/target.ts, Vitest tests. Done when: app debug <application_id> and app debug my.app.json produce the same report shape an app_doc run did. T10. Mobile: read applications Mobile parses workflow.app_doc read-only. Point it at the applications REST API (T2 — mobile has no tRPC client): list apps, open one per screen per mobile/ARCHITECTURE.md § Documents. Remove the in-memory per-workflow generateAppDoc path along with the concept it served. Depends on: T2. Files: mobile/src/components/app_runtime/, data hooks, Jest tests. Done when: an app published on the server renders on mobile without the workflow carrying any app_doc. Phase 4 — examples overhaul (after Phase 3) T11. Curate the example apps Audit the 37 templates (see docs/plans/examples-revamp.md for the shape census) and produce a curation spec: which use cases deserve an app, which templates stay workflow-only, and what each example app contains. Selection criteria — user-centric, in priority order: (1) the app is how a real person would want to use it (a form/dashboard beats opening a node graph), (2) it exercises something the platform is good at (multi-workflow operations, streaming, variables), (3) it runs on a fresh install or degrades clearly without keys. Target ~8–12 apps, several binding more than one workflow (e.g. generate + refine, transcribe + summarize + translate). Deliverable: a table in this file’s companion docs/plans/example-apps.md with per-app operations, workflows, and widgets. Done when: the spec is reviewed and merged; no code. T12. Build the example app bundles Implement the curated apps as ApplicationBundle files shipped with packages/base-nodes (new examples/apps/ dir). Replace the per-template generation in scripts/generate-template-apps.mjs (curation table + Output-node augmentation) with a build step that validates each bundle (nodetool app debug --no-run) and emits the preview bundles in web/public/app-preview/. Server exposes example apps for install; install goes through T8’s import path. Depends on: T8, T11. Files: scripts/ (replace generate-template-apps.mjs), packages/base-nodes/nodetool/examples/, packages/websocket example routes, tests. Done when: installing an example app creates the app + its workflows; no example JSON carries app_doc. T13. Preview + marketing pipeline on bundles Point web/src/app_preview/AppPreviewApp.tsx, web/scripts/screenshot-app-previews.mjs, and marketing/scripts/generate-miniapp-entries.mjs at the bundle-derived previews. /apps/* pages regenerate from the curated set only — fewer, better pages. Depends on: T12. Done when: npm run gen:apps runs clean and every /apps/* page corresponds to a shipped example app. Phase 5 — data migration and removal T14. One-time lift migration for user data A models-layer migration (packages/models/src/migrations/versions.ts) that walks workflows with a non-null app_doc, creates an application row per doc via T1’s helper, and nulls the value. Where a fromWorkflowId import already created a (possibly newer) application, the application wins and the app_doc is archived as a new draft version, not merged. Depends on: T1; lands only after Phase 2 (no writers left). Done when: migration test proves a seeded app_doc workflow ends up with an equivalent application and an empty app_doc. T15. Drop app_doc from schema and API Remove the column (SQLite + PG schemas, raw DDL in db.ts), the Zod/API fields (packages/protocol), the now-unreachable hostWorkflowId lift paths, and the screenshot-server read. Grep-clean: rg app_doc returns only migration history and this plan. Depends on: everything above shipped, and T14 released at least one version earlier so old clients have upgraded. Sequencing summary Phase 1: T1 T2 T3 T4 — parallel Phase 2: T5 T6 T7 — parallel, after Phase 1 Phase 3: T8(←T2,T3) T9(←T1,T3) T10(←T2) — parallel Phase 4: T11 → T12(←T8) → T13 Phase 5: T14(←T1, after Phase 2) → T15 (one release later) Risks Fork already in the wild: users who used create({ fromWorkflowId }) have divergent copies; T14’s application-wins rule covers it, but the archived draft version must be visible in the version history or the user’s edits look lost. Old mobile/CLI clients read app_doc from the API. T15 waits a release after T14 for this reason; T2/T9/T10 must ship first. Removing the View tab regresses a flow people may use today (instant form for any workflow). The mitigation is the explicit scaffold (“Create app from workflow”) being one click from both the Apps panel and the workflow editor — cheap to invoke, but a deliberate act that creates a real resource. Example count drops from ~40 auto-generated apps to ~8–12 curated ones; /apps/* marketing pages shrink accordingly. That is the point, but existing inbound links need redirects (docs/redirects/).--- # Built-in game engine: GPU assessment and extensible 2D design Source: https://docs.nodetool.ai/plans/builtin-game-engine-design.html Built-in game engine: GPU assessment and extensible 2D design Recommendation Build a small TypeScript game runtime with a native 2D renderer on NodeTool’s existing WebGPU infrastructure. Start with sprites, tilemaps, input, collision, animation, audio, and a reproducible playtest loop. Keep the simulation independent of rendering and of workflow execution. Add 3D through explicit 3D scene and physics types later. The built-in engine replaces Godot entirely. The target product has one game runtime, native game templates, and its own player/export path. Godot execution, project generation, packages, bundled templates, and agent tools are removed. Preserve useful generated media and asset-validation logic when replacing them. The initial native 2D vertical slice, in-app player, and standalone web export are implemented. Hardware profiling, atlas packing, tile chunking, an isolated script/render worker, native mobile packaging, multiplayer, and 3D remain later work in this design. The 3D upgrade design defines a shared gameplay lifecycle with separate 2D/3D spatial implementations, plus 3D rendering, asset preparation, and compatibility gates. The recommendation favors shared NodeTool effects and headless rendering. If the priority becomes the shortest path to a broadly compatible browser game editor, choose PixiJS for rendering instead. The project model and simulation design below can remain the same. Existing capabilities Finding Evidence Design implication F1. NodeTool has a portable GPU foundation. GPU context, browser entry, and Node/Dawn adapter. Reuse device integration, shader definitions, texture metadata, scratch textures, pipeline caching, and uniform buffers. F2. It supports fragment effects and compute. Executor encodes full-screen fragment passes or compute dispatches. Shader pool exposes filters, transforms, masks, color operations, sources, and multipass recipes. Game post-processing can use this catalog on the same device. The executor is image-oriented, so sprite geometry needs a separate rendering path. F3. The layer compositor is not a sprite batcher. renderBlendPass samples a layer and an accumulation texture, then draws a full-screen quad into another texture. Using one such pass per sprite scales with layer count and screen area. Build instanced quad batches and reserve compositing passes for effects and groups. This is a structural inference, not a measured game benchmark. F4. Browser fallback and device-loss handling exist in individual surfaces. Sketch initialization, sketch runtime, timeline initialization, and timeline compositor. Follow their lifecycle patterns. A game still needs its own capability report and recovery behavior. These surfaces do not establish one application-wide device owner. F5. The former games were Godot asset-production projects. Game nodes and games router. Reuse asset generation and validation in the native game model. F6. NodeTool already edits and renders 3D assets. glTF scene operations and Three.js rendering. Reuse glTF assets and import/edit operations later. The inspected render session creates a WebGLRenderer, so it is not already a shared WebGPU game renderer. Local GPU evidence The read-only device probe found an NVIDIA GeForce RTX 3060 through lspci. nvidia-smi could not communicate with its driver. A direct Dawn adapter request succeeded using Mesa llvmpipe, reporting architecture: software and isFallbackAdapter: true. This establishes the adapter available to this process, not the adapter used by the user’s browser or Electron renderer. The default requested device exposed an 8192-pixel maximum 2D texture dimension, a 256 MiB maximum buffer, a 128 MiB maximum storage-buffer binding, and 256 maximum compute invocations per workgroup. These are exposed device limits, not VRAM capacity or recommended allocation sizes. The adapter advertised optional features including timestamp queries and shader f16, but the default device did not enable those optional features. The existing checks below passed, with nine tests executed and none skipped: npm run test --workspace=packages/gpu -- \ tests/computeSmoke.test.ts \ tests/headlessComposite.test.ts \ tests/uniformRing.test.ts They verify compute output, compositing, and uniform-buffer behavior on the available software adapter. They do not measure hardware FPS. The shell used Node 22.23.2, while .nvmrc specifies the repository runtime. No browser capability probe or device-driver change was performed. GPU work required before an engine ships The current GPUCapabilities is too narrow for game admission. It hardcodes external-texture support and derives f16Storage from shader-f16. The latter feature enables the WGSL f16 type, so it must not stand in for a format-and-usage capability table. This is a design concern to reproduce in focused tests before changing existing behavior. WGSL extension definition. Use a game capability report that records the selected backend, hardware versus software versus unknown adapter status, requested and enabled features, relevant limits, and a successful minimal render. Request optional features only when supported and used. A GPU visible to CUDA or a cloud generation provider says nothing about the browser’s available graphics device. Probe each execution host. Track device loss, release resources on scene replacement, and rebuild textures and pipelines from retained asset data after recovery. The Node adapter caches a device acquisition promise but does not invalidate it on later device loss. Its Dawn instance retention also keeps one-shot processes alive, as documented in the implementation. Headless game commands should run in a managed subprocess with an explicit completion and teardown policy. The texture pool retains released allocations until disposal. Add game-session accounting and bounded eviction before loading many scenes. Use exact-size screen targets: its existing bucket policy would round a 1920×1080 request to 2048×2048 unless exact is set. One exact RGBA8 target is about 7.91 MiB, while that bucket is 16 MiB. Two RGBA16F targets at exact 1080p require about 31.64 MiB, before textures and other allocations. The executor currently defers strict color-space checking. Reuse of its labels does not establish color correctness. Define sprite import, working-space, effect, and presentation conversions and test them against reference images before enabling shared effects in games. Technology choices Option Benefit Cost or constraint Recommendation O1. Native WebGPU/TypeGPU renderer Direct reuse of NodeTool’s GPU effects, same-device textures, and Dawn rendering. NodeTool owns sprite batching, atlases, clipping, text, resource lifetime, and fallback behavior. Use for the proposed bounded engine, subject to the first rendering milestone. O2. PixiJS renderer with NodeTool simulation Existing sprite renderer and WebGL compatibility reduce renderer work. Shared NodeTool WGSL effects need separate integration. Its current guide recommends WebGL for production and labels WebGPU experimental. Preferred alternative if compatibility or delivery time dominates. O3. Phaser framework Supplies more game behavior and an established 2D programming model. Adopting its scene/runtime model makes a NodeTool-owned simulation and later 3D a larger integration task. Choose if shipping conventional 2D games becomes the entire scope. O4. Embed Godot web builds Reuses a complete engine and existing NodeTool Godot assets. Retains the external engine and its build/export pipeline. Rejected by the product direction to remove Godot entirely. These tradeoffs are architectural judgments. Primary platform references are PixiJS renderers, Phaser’s renderer selection, and Godot web export. Godot’s current stable documentation specifies WebGL 2 Compatibility rendering for web export. Single-threaded export avoids the cross-origin isolation required by its threaded export. Do not assume TypeGPU automatically provides a WebGL fallback for the existing shader pool. The current experimental WebGL module supports a subset of rendering operations and excludes compute, storage buffers, and bind groups. Compatibility with the repository’s installed TypeGPU version would need separate validation. Proposed architecture flowchart TD Editor[Game editor and agent tools] --> Doc[Versioned game document] Workflows[Asset generation workflows] --> Assets[Versioned game assets] Doc --> Session[Game session] Assets --> Session Input[Tick-indexed input] --> Session Session --> Simulation[Fixed-step simulation] Simulation --> Frame[2D render frame] Simulation --> Events[Audio and gameplay events] Frame --> GPU[WebGPU sprite renderer] Frame --> Canvas[Canvas2D fallback] GPU --> Effects[Shared GPU effects] Effects --> View[Canvas or headless capture] Canvas --> View Test[Headless replay harness] --> Session D1. Give games their own persistent identity Add game to the project document model. The current document union has no game entry. A proposed game record holds id, project_id, workspace_id, source_root, and the current immutable revision reference. There is no engine selector. The engine version identifies a version of the built-in runtime. Store authoritative source under games/<full-game-id>/ in the project workspace: game.json, scenes/, prefabs/, behaviors/, and an asset manifest. Store builds under a separate game-builds/<full-game-id>/<revision>/ path. All reads and writes use the existing workspace interface, including virtual workspaces. The manifest declares schema version, engine version, entry scene, input actions, simulation frequency, scene references, and asset dependencies. Asset bindings include source asset ID, immutable content digest, sprite frame metadata, pivot, sampling mode, and generation provenance. Store full resource IDs internally. Use one immutable source revision plus compare-and-swap of the current revision reference to publish an edit atomically. Validate references before publication. This avoids relying on filesystem rename semantics in object storage. Reject stale edits with a conflict result. Undo and restore select a prior revision. Templates seed source once. Attached workflows produce candidate assets, then an explicit installation operation updates selected bindings. Re-running generation must preserve scenes and behavior edits. A play session pins a revision and applies accepted updates only at a tick boundary. Art can update live when its layout is compatible, while behavior or schema changes restart the session first. This retains the project-ownership requirements from the earlier Godot project design question and supersedes its choice of runtime. Existing Godot source can remain as ordinary workspace files or a downloadable archive, without NodeTool execution or export support. Generated images and audio can be installed into new native games. GDScript and scene behavior require explicit reconstruction. Do not present an asset import as a converted, playable game. D2. Keep the simulation independent of the GPU and workflow scheduler Use a scene graph for hierarchy and entity IDs with typed component data for behavior. A compact entity store is sufficient initially. Add dense arrays for hot transforms only when measurement justifies them. The initial component vocabulary is Transform2D, Sprite, Tilemap, Camera2D, Body2D, Collider2D, Animator, AudioSource, and Behavior. These are game data components, not React components or workflow nodes. Use world units with +X right, +Y up, radians, and an explicit pixels-per-unit setting. Convert to downward canvas coordinates only in rendering and picking. Convert authored pixel hitboxes once during import. Validate parent graphs for cycles. Physics bodies own world transforms, so initially prohibit scaled or rotating physics parents instead of silently producing incorrect collisions. The simulation advances at a fixed 60 Hz using tick-indexed input and seeded randomness. Each tick consumes input and prior events, updates behavior, advances physics, produces contacts, and commits spawn/despawn commands. Collision handlers enqueue changes instead of mutating collections during iteration. Rendering interpolates between completed ticks using requestAnimationFrame. Clamp interactive catch-up after a stall and report dropped wall time. Pause on document invisibility and reset the accumulator on resume. A replay executes every requested tick without consulting wall time. Keep simulation state out of React and Zustand’s frame-by-frame update path. Inspector snapshots can update at a lower rate. Use CPU/WASM physics initially. Prototype Rapier 2D for bodies, sensors, collision queries, and a kinematic character controller. Keep its handles private to the simulation module. Pin the version and creation order for replay. Rapier documents determinism for its JS/WASM version under matching initial conditions and version, but application math can still invalidate that guarantee. Rapier determinism. GPU compute is suitable later for cosmetic particles or lighting. Authoritative gameplay remains on the CPU so readback and GPU scheduling do not enter the tick loop. Workflow generation jobs also stay asynchronous and outside that loop. D3. Build a batched 2D renderer and reuse effects selectively Upload sprite assets once, pack texture atlases, and render a unit quad with per-instance transform, UV rectangle, tint, and opacity. Cull against the camera. Preserve painter order and batch consecutive compatible instances. Do not sort transparent sprites globally by texture merely to reduce draw calls. Render visible tilemap chunks through the same quad path. Keep animation as UV frame selection. Use nearest sampling and integer scaling for pixel art, with atlas padding to prevent adjacent frames bleeding under filtering. Start HUD text with a baked glyph atlas, leaving general vector text and text shaping outside the first rendering milestone. Use premultiplied alpha consistently. Start with normal alpha blending through fixed-function GPU blending. Effects that require reading the destination render an isolated group through a compositing pass. Apply shared blur, color grading, outline, or glow to selected groups or the final frame, not a full-screen pass for every sprite. Introduce a game-approved effect subset after color validation. The renderer owns its game-session device and resources. Its sprite pass and shared effect executor receive the same device. Do not depend on sharing textures with a separate sketch device, the server’s Dawn device, or a WebGL context. Prepare pipelines before play and retain CPU asset bytes for re-upload. Avoid per-frame decoding, pipeline compilation, GPU readback, and synchronization waits. Offer two explicit capability levels: Level Supported behavior Core 2D Sprites, tilemaps, cameras, basic HUD, normal alpha blending, and identical simulation. Implement with WebGPU and Canvas2D. GPU effects Approved shared effects and later visual compute. Requires successful WebGPU initialization and the relevant capabilities. Each game declares required and optional capabilities. Omit optional effects with a visible notice. Reject play with an explanation if a required effect is missing. Canvas2D is a functional compatibility path with a lower performance target, not a promise of identical shading. Headless simulation needs no renderer at all. Replace the canvas element when recovery switches its rendering-context type. D4. Use a small runtime interface across editor and tests Proposed modules and entry points: Location Responsibility packages/protocol Versioned game/scene schemas, commands, inputs, reports, resource references. New packages/game-runtime validateGame, createGameSession, fixed ticks, behavior, physics, snapshots, replay. No DOM, React, GPU, or workflow-runner dependency. New packages/game-renderer createGameRenderer, WebGPU drawing and effects, Canvas2D drawing, texture ownership. Browser and Node entries keep platform imports separate. Existing models and websocket packages Authorized persistence, revision publication, build jobs, asset installation. Existing CLI and agents packages Drive validation, simulation, screenshots, and exports through the shared modules. Web game surface Scene tree, viewport, inspector, asset palette, play/pause/step, errors, and measured performance. The session interface should expose step(input), inspect(query), snapshot(), and dispose(). step advances exactly one tick and returns events plus a render frame. createGameSession accepts a validated document, seed, and optional saved snapshot. The render frame contains visible instances and camera data, with no GPU objects. The renderer exposes render(frame, interpolation), resize, and dispose, and owns the previous frame needed for interpolation. Use an event sink for audio and host integration. Browser playback consumes audio events through Web Audio after a user gesture. Headless tests record those same events without requiring an audio device. Save files include game revision, simulation version, tick, RNG state, entity state, and physics snapshot. Browser and headless rendering are real adapters. WebGPU and Canvas2D are another real seam. Do not add a generic plug-in architecture or a universal physics interface before a second implementation is needed. D5. Make game authoring and execution separately controlled Start with declarative behavior built from reviewed modules: movement, patrol, triggers, health, collectible, spawn, and scene transitions. An agent edits data and tests before generating arbitrary runtime code. Authored JavaScript uses one memory-limited QuickJS runtime per game session. A positive AST check permits resident functions only for input-only expressions that cannot retain hidden state. Other sources are evaluated in a fresh context for each call, preserving existing initialization and JSON parsing behavior. Only returned JSON state and the seeded random generator’s state persist as simulation state. Per-call inputs and world query results are private copies. One immutable tick-start host snapshot serves world queries. All sources are validated in temporary contexts during preparation. Resident functions compile on the first active call and remain resident while the behavior is active. The resident path constructs native sandbox input values and exposes legacy input.world lazily. Compatibility calls pass a native string to the guest parser. Host JSON normalization and budget accounting still apply to both paths. Measure context creation, source evaluation, and serialization costs before increasing instance budgets. The implementation lives in packages/game-runtime/src/scripts.ts without depending on the agents package. Game scripts receive state, tick input, seeded random functions, and allowed game commands. They receive no secrets, filesystem, network, toolbelt, DOM, or raw GPU device. Per-call timeouts share a total tick deadline, including host serialization and validation. UTF-8 limits count the logical tick input and complete output envelope. An interrupt stops runaway guest code. A failed tick terminates the session and cannot produce a save of partially updated state. A separate worker remains future work. QuickJS isolation and the restricted host interface are both required. The existing sandbox limits also warn that typed-array and string payloads need accounting beyond the guest heap limit. Start with reviewed shaders. Arbitrary generated WGSL adds GPU resource and execution risks that JavaScript instruction limits do not contain. Treat custom shader admission as separate future work. Agent tools should provide document operations and a bounded playtest request that specifies revision, seed, tick count, input recording, assertions, and capture ticks. Return failed assertions, entity state, script errors, and screenshot asset IDs. The UI uses the same operations. Asset inspection uses the existing media resolution path, while the game texture loader resolves authorized media bytes before upload. No asset:// identifier is passed directly to a DOM src. D6. Extend into 3D without changing game ownership or time Keep scene identity, parent relationships, input actions, events, behaviors, asset references, save/version rules, and fixed-step execution dimension-neutral. Keep spatial types explicit: Transform2D and Body2D remain 2D. Introduce Transform3D, Camera3D, Mesh, Material, Light, and Body3D when needed. Reject invalid mixtures in a physics world. A 2D HUD can overlay a 3D viewport. The first extension can be 2.5D: parallax layers, orthographic cameras, normal-map lighting, and cosmetic particles. Full 3D then imports existing glTF/GLB assets through the model3d module and adds a 3D renderer. Evaluate Three.js at that point instead of implementing a PBR engine from scratch. Its WebGPURenderer supports a WebGL2 fallback, but using it would still require explicit texture/device and effect integration. Three.js renderer documentation. Add Rapier 3D beside 2D when 3D physics is required. Reuse behavioral concepts such as health and triggers while adapting movement and collision data explicitly. A 2D project remains valid without acquiring depth, quaternion, or 3D material fields. Extensibility preserves useful contracts, not automatic 2D-to-3D conversion. D7. Remove Godot throughout the product Retain the game creation experience, but make it create a native game document and attach asset-generation workflows. Templates become native scene, behavior, input, and asset-manifest data. The first template is the top-down vertical slice. Platformer and shooter templates follow when their native behaviors are verified. The inspected removal and replacement surfaces are: Surface Required change Former packages/godot and packages/godot-templates packages Remove both packages, their dependencies and TypeScript references, shipped Godot files, runner, and package-specific tests. Game nodes and former packages/game-nodes/src/project.ts Replace Godot export/project assembly with native game creation, asset installation, and build operations. Retain only nodes that have a defined native purpose. Asset contract, graph construction, and designer Replace Godot-version and script-hook assumptions with native template/schema versions and behavior configuration. Keep useful sprite, tile, image, and audio layout metadata. Image validators and audio validators Preserve byte validation and generation outputs. Remove Godot-specific wording and dependencies in their contracts. Former packages/agents/src/capabilities/godot.ts and packages/system-skills/godot-game/SKILL.md Remove the Godot tools and skill. Register native game authoring, bounded playtesting, and build tools, and update product knowledge and skill discovery. Games router and former web/src/components/setup/game/useGameSetupFlow.ts Serve native templates and open the native editor/player after creation. Replace instructions to download or open a Godot project. Packaged assets, harness registry, examples, and generated DSL bindings Remove Godot packaging and checks, register native equivalents, rebuild examples, and regenerate bindings from the node registry. Search manifests, lockfiles, build configuration, documentation, prompts, fixtures, and agent instructions again during implementation. This table identifies the inspected integration points, not a claim that every reference has been enumerated. Cut over after native creation, playtesting, and web export pass their acceptance checks. Remove the old execution path in the same release that switches the creation flow. Temporary development overlap does not become a supported dual-engine product or a runtime fallback. Preserve existing workspace files and source assets. Old workflows containing removed nodes must fail validation with an explicit migration diagnostic. Never silently map ExportGodotProject to a different operation. A one-time migration can detach the old export stage and attach the surviving generation stages to a new game, but it must report behaviors that still need rebuilding. Removal is complete when a fresh installation can create, playtest, save, reopen, and export the native sample with no Godot executable, package, bundled project, registered tool, or template dependency. Audit the shipped artifact as well as source imports. Historical documentation and preserved user files do not keep a Godot runtime installed. Delivery sequence and acceptance Action Deliverable Evidence required before advancing A1. Prove the renderer choice Instanced sprites, one atlas, animated frames, tile chunks, and one shared effect on the same device. Compare a PixiJS scene using the same assets. Measure submission cost, draw calls, upload bytes, frame times, and texture memory on hardware. Check alpha, UVs, ordering, and effect color. Reconsider O1 if maintaining it is disproportionate. A2. Build a playable vertical slice One small top-down room with movement, wall collisions, collectibles, a win state, camera, HUD, and sound. Use built-in behaviors. Browser and headless replay reach the same asserted state. Core rendering works with WebGPU unavailable. Pause, single-step, reset, and save/load work. A3. Persist and edit the game Game document, revision operations, scene editor, asset installation, and agent playtest tools. Reopen the game, regenerate only its player art, preserve scene changes, reject stale edits, and restore a previous revision. A4. Add scripted behavior and web export Session-owned QuickJS runtime with isolated calls and a version-pinned standalone player. Infinite loops terminate, invalid commands fail, script budgets are measured, and an exported build runs from an HTTP server without NodeTool credentials or backend access. A6. Cut over and remove Godot Switch game creation to the native runtime and perform D7’s removal and migration. This precedes A5 despite retaining the existing action codes. A clean installation and packaged artifact complete the native game lifecycle without Godot. Legacy source assets survive and workflows with removed nodes report migration errors. A5. Add complexity after profiling Platformer controller, additional scenes, approved effects, then 2.5D or 3D. Replay regressions, renderer capability checks, and measured performance remain within the selected target profile. The standalone export copies authorized assets into a content-addressed bundle, rewrites asset bindings to relative URLs, and includes schemas, behavior runtime, physics WASM, and a pinned player. It contains no project tokens, provider keys, or expiring media URLs. Unknown component versions fail validation with a migration message. Cloud build jobs stage through workspace materialization and absorb only declared build outputs. GPU rendering runs on a worker that reports its own capabilities, independently of the browser playing the game. The nodetool game validate|simulate|capture|build CLI family and corresponding agent tools provide headless access. The game surface is registered in the harness registry following Harness-First Engineering. Performance targets and verification limits Begin with a 60 FPS target at 1280×720 on a named integrated-GPU reference machine. Measure a normal scene with 1,000 visible sprites, then a 10,000-sprite stress scene, with atlas count, transparency, and effects fixed. These are proposed test loads, not supported-capacity claims. Reserve 16.67 ms per frame overall, including simulation, command encoding, GPU work, and presentation. CPU submission timing alone does not measure GPU completion. Record p50/p95 frame times, simulation time, draw calls, upload bytes, texture allocation estimates, and GPU time when timestamp queries are enabled. Run a long-lived scene-switching test for retained resources. Test hidden tabs, focus loss and stuck keys, resizing, device loss, missing assets, cancellation, and software adapters. GPU-required tests must report unavailable separately rather than silently passing by skipping. Use exact assertions for simulation and bounded pixel tolerances for visual regressions. Do not promise identical screenshots across GPUs. Add resource-ID tests at tool, persistence, and media-resolution interfaces, including the exact short IDs returned to agents. Once implementation changes code, run the mandatory repository checks. Remaining risks Risk Consequence Resolution R1. No hardware performance measurement yet Software-adapter correctness can conceal unacceptable frame times. Run A1 on working integrated and discrete GPUs, and probe the actual browser/Electron device. R2. Owning the renderer grows the scope Text, clipping, batching, and recovery can consume more work than expected. Keep the first feature set bounded and compare against O2 before building the editor. R3. Per-call script isolation costs CPU time Context creation, source evaluation, and input copying may exceed the frame budget. Benchmark representative scenes and optimize without retaining hidden state outside snapshots. Keep termination and output limits. R4. GPU compatibility differs by host Browser play, cloud capture, and Electron can select different devices and capabilities. Pin required capabilities in the game, expose fallback decisions, and test each supported host. R5. Removing Godot leaves legacy gameplay to rebuild Existing GDScript and scenes cannot be made playable by importing their artwork. Preserve source files and assets, report migration gaps, and remove runtime support without pretending conversion is automatic.--- # Removing redundant Tool/capability wrapping around the permission gate Source: https://docs.nodetool.ai/plans/capability-gate-unwrapping.html Removing redundant Tool/capability wrapping around the permission gate Scope: the conversions between Tool and capability in packages/agents/src/capabilities/adapters.ts, packages/agents/src/capabilities/gate-tools.ts, and packages/agents/src/evals/tool-loop-permission.ts. Background and the migration this finishes are in tool-class-retirement-design.md. The per-host gate table is in packages/agents/AGENTS.md. 1. The problem The permission ladder runs in one place, gatedCall inside CapabilityRun.invoke (capabilities/invoke.ts). Several callers reach it by converting the same operation between representations more than once. The headless eval path converts five times for one call: HeadlessTool (evals/tool-loop-bridge.ts) → HeadlessSurfaceTool extends Tool (evals/tool-loop-permission.ts) → GatedCapabilityTool (capabilities/gate-tools.ts) → capabilityFromTool(inner) (capabilities/adapters.ts), per call → createCapabilityRun(...) one run per call → run.invoke → gatedCall → impl → HeadlessSurfaceTool.process → HeadlessTool.execute gateHeadlessTools then rebuilds a HeadlessTool around the result. Registry capabilities that a host already holds a gated CapabilityRun for take a shorter but still redundant path: toolForCapabilityName(name, run) LazyCapabilityTool, calls impl directly → gateTools(...) GatedCapabilityTool → capabilityFromTool(...) a second spec for the same name → createCapabilityRun(...) a second run beside `run` → gatedCall → LazyCapabilityTool.process → impl(run, args) The sites are buildAuthoringBelt (packages/agents/src/author-graph.ts) and the execute_plan entries in buildCliAgentBelt (packages/cli/src/chat-codeact.ts) and the chat turn (packages/websocket/src/session/chat-turn.ts). 2. Decisions D1. Headless tools become capabilities directly. gateHeadlessTools builds one CapabilityExport per headless tool and one CapabilityRun over all of them, and each gated execute calls run.invoke. No Tool is constructed. This is the completion criterion. D2. A host with a gated run invokes through it. A new belt adapter, toolInvokingCapability(name, run), returns a Tool whose process() is run.invoke(name, args). It replaces gateTools(toolForCapabilityName(...)) where the run already carries the intended gate. The Tool stays because these belts are still Tool[]. D3. Heterogeneous belts keep gateTools and capabilityFromTool. The chat turn, MCP mount, CLI base belt, Code node, and AgentNode (llm-nodes/agent-utils.ts) gate belts that mix registry tools with host-built ones (RunNodeTool, UI bridges, sandbox tools, dynamic Code-node tools). capabilityFromTool is what classifies a non-registry tool through capabilityCategoryFor. Removing it before those belts are capability lists would reach invoke with no category. The design doc records the same constraint in the survey section. D4. Deliberately ungated delegation stays ungated. run_subtask, start_subtask, wait_subtasks, run_search, and create_plan call their implementation directly by design (their children run the gated belt). D2 applies only to tools that are gated today. Routing the others through invoke would add approval prompts that do not exist now. D5. toolFromCapability is left in place. It is a one-line alias of toolFromLazyCapability with two production callers and about twenty test callers. Deleting it is a rename, not a removed layer. It can go in a separate cleanup PR if wanted. Kept unchanged: the ladder in gatedCall (read fast path, mode decision, session allow-set, approval, security monitor, clock suspension), argument validation through validateCapabilityArgs, capabilityCategoryFor lookup order, and the approval request records the eval reports. 3. Tasks A1. Headless eval tools gate without a Tool (D1) Files: packages/agents/src/evals/tool-loop-permission.ts, packages/agents/tests/tool-loop-eval.test.ts. Add a module-private capabilityFromHeadlessTool(tool: HeadlessTool): CapabilityExport: name and description from the tool. inputSchema from zodToJsonSchema(tool.parameters), the same conversion Tool.inputSchema uses today. category from capabilityCategoryFor(tool.name) (capabilities/registry.ts). The ui_* names resolve to their registered spec categories in capabilities/ui.specs.ts, which is what the current path produces. No zodSchema. HeadlessTool.execute already parses with parseWithTypeCoercion, and the current path does not validate at the gate. Adding it would change error shapes the eval cases see. impl: (_run, args) => tool.execute(args). In gateHeadlessTools, build the exports once, then one run: createCapabilityRun({ context, gate, capabilities: exports }). Each returned tool keeps name, description and parameters and sets execute: (args) => run.invoke(tool.name, args). One run for the whole case is equivalent to one run per call: the only mutable state is gate.sessionAllow, which already lives on the shared gate object, and budget is read from the same context either way. Delete HeadlessSurfaceTool and the imports of gateTools, Tool, and ZodType. Rewrite the file header, which currently explains the Tool detour. Tests: The existing permission-gated cases block in tests/tool-loop-eval.test.ts must pass unchanged. It covers plan-mode block, default-mode deny, the ungated case, and default-mode allow with recorded requests. Add one test that proves no Tool path is used: mock ../src/capabilities/gate-tools.js and ../src/capabilities/adapters.js with implementations that throw, then run the plan-mode and deny cases. Confirm the test fails against the current implementation before the change. Add a case with two writes in default mode and assert two recorded requests. This pins that sharing one run across calls did not change approval behaviour. Acceptance: a headless capability is authorized and executed without constructing a Tool, and all permission tests in tests/tool-loop-eval.test.ts pass. A2. Belt tools that invoke through the host’s run (D2) Files: packages/agents/src/capabilities/lazy-tool.ts, packages/agents/src/capabilities/index.ts, packages/agents/src/index.ts, packages/agents/tests/capabilities-gate-parity.test.ts. Add toolInvokingCapability(name: string, run: CapabilityRunSource): Tool beside toolForCapabilityName. It resolves the spec with capabilitySpec and throws on an unknown name, as toolForCapabilityName does. Its process() resolves the run from the source and returns run.invoke(this.name, params). It does not validate itself, because gatedCall validates. Name, description, inputSchema, zodSchema, needsToolCallId, and userMessage come from the spec, sharing the LazyCapabilityTool accessors (extend that class with a mode flag rather than adding a third Tool subclass). Update the lazy-tool.ts header: it currently says a belt is always gated from outside by gateTools. State the two modes and when each applies. Export it from capabilities/index.ts and src/index.ts. Extend tests/capabilities-gate-parity.test.ts so each existing parity case (read, approve, deny, plan block, auto, allow_for_chat, monitor block, clock suspension, _tool_call_id threading, _message stripping) also runs through toolInvokingCapability. The transcripts must match the invoke and gateTools entrances. A3. Move the gated-run call sites to A2 (D2, D4) buildAuthoringBelt (packages/agents/src/author-graph.ts): the run is built with gate. Replace gateTools(names.map((n) => toolForCapabilityName(n, run)), gate) with names.map((n) => toolInvokingCapability(n, run)). Update the function’s doc comment. Cover with the existing author-graph tests, plus one test that a write-class belt tool in plan mode returns blocked_in_plan_mode. buildCliAgentBelt (packages/cli/src/chat-codeact.ts): delegationRun already carries gate. Replace the gateTools([toolForCapabilityName( "execute_plan", delegationRun)], gate) block with toolInvokingCapability("execute_plan", delegationRun). Leave the other delegation tools on toolForCapabilityName. Chat turn (packages/websocket/src/session/chat-turn.ts): its delegationRun uses UNGATED on purpose. Add a second source, planExecutionRun, with the same options and gate: chatGate, and build execute_plan with toolInvokingCapability("execute_plan", planExecutionRun). Do not change the gate of delegationRun. Run the plan-mode tests that already cover execute_plan in the CLI and websocket packages (grep -rn execute_plan packages/*/tests) and confirm they pass without edits. A4. Documentation capabilities/gate-tools.ts header: state that gateTools is for belts holding tools that are not registry capabilities over a gated run, and name toolInvokingCapability as the entrance for the rest. packages/agents/AGENTS.md § Where the permission gate is set: add one sentence naming the two entrances. Run npm run check:agents-docs. docs/tool-class-retirement-design.md § Where the permission gate lives, “What shipped since”: replace the sentence saying every Tool enters through GatedCapabilityTool with the current split, and keep D3’s reason for the remaining capabilityFromTool use. 4. Order and PR shape A1 is independent and meets the completion criterion alone. Ship it first. A2 and A3 go in one PR, because A2 has no production caller without A3. A4 lands with the PR whose code it describes. 5. Verification For each PR, from the repository root, after npm run build:packages: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main A2 and A3 change imports inside packages/agents/src/capabilities/. The tools/ to capabilities/ import cycle that once hung the packaged server is visible only in the bundle, so also run npm run backend:smoke for that PR. toolInvokingCapability must not add an import from tools/ into capabilities/ beyond what lazy-tool.ts already has. 6. Risks R1. Category drift for a headless tool name that is not a registered capability. capabilityCategoryFor falls back to the classification map, then to external, which is the same answer capabilityFromTool gives today. The A1 tests pin ui_add_node as write and ui_get_graph as read. R2. Double validation in A2. toolInvokingCapability must not call validateCapabilityArgs itself. The parity test asserts one invalid_tool_arguments shape, not a nested one. R3. Gating a tool that is ungated today. A3 touches only execute_plan and the authoring belt, both gated now. Any other swap is out of scope (D4). 7. Out of scope Converting heterogeneous Tool[] belts to capability lists, which is what deleting capabilityFromTool and GatedCapabilityTool requires (D3). Deleting toolFromCapability (D5). Moving classification from TOOL_PERMISSION_CATEGORIES into specs.--- # Code Node AI Authoring — Simplified Plan Source: https://docs.nodetool.ai/plans/code-node-ai-authoring.html Code Node AI Authoring — Simplified Plan Goal: Let a user describe the result they want and get a working, typed Code Node — instead of writing JavaScript or hunting the node catalog for a chain of type-compatible nodes. Describing the result is faster than finding and wiring the chain. Nothing is removed. The Code Node keeps every capability it has today. AI authoring is an additional way to write the code, not a restricted variant of the node. Design rule: No new persistence, no new node state, no contract layer. Generation fills the fields the node already has (code, dynamic_inputs, dynamic_outputs, dynamic_properties) and then gets out of the way. A generated node is indistinguishable from a hand-written one. What exists today The platform already has almost everything this feature needs. Two recent changes removed most of what the earlier draft of this plan had to build: Typed dynamic slots are persisted. DynamicSlotMeta (packages/protocol/src/graph.ts) carries type, description, default, and required for each dynamic input; dynamic_outputs carries a type per output. Both round-trip through reactFlowNodeToGraphNode.ts / graphNodeToReactFlowNode.ts. There is nothing left to invent for “stable typed handles” — the generator just writes these fields. The sandbox grew a real standard library. js-sandbox.ts now bridges format.* (Intl number/date/relative-time/list), data.parseCsv (papaparse), data.selectHtml (cheerio), crypto.* (WebCrypto), progress(), binary helpers (toBase64/fromBase64/toHex/fromHex), extended workspace (bytes/stat/mkdir/remove), and per-run limit overrides. The old gap — prompts advertising libraries the sandbox lacked — is now mostly a prompt bug, not a missing capability. Existing capability Where How this plan uses it Universal JS Code Node packages/code-nodes/src/nodes/code-node.ts (nodetool.code.Code) The only runtime node. Unchanged. QuickJS sandbox + bridges packages/agents/src/js-sandbox.ts Export a serializable capability manifest; derive the generation prompt from it. Typed dynamic inputs/outputs DynamicSlotMeta in protocol, persisted by both graph converters Generation output lands here. No new fields. AST handle inference web/src/utils/codeOutputInference.ts Untouched. Handles keep tracking code exactly as today, for generated and hand-written code alike. Monaco editor web/src/components/node_types/CodeBody.tsx Untouched. Ask AI is a button beside it. Snippets web/src/config/codeSnippets.ts, snippetMetadata.ts Untouched. Model preferences ModelPreferencesStore.ts generic defaults map Add a code_model key. No schema change. Planner infrastructure GraphPlanner + SubmitGraphTool, Tool base class, withAgentSpanGen, eval harness (packages/agents/src) Code generation is a mini-planner on this machinery — submit tool, validation-as-tool-result feedback rounds, tracing, evals. Provider abstraction BaseProvider.generateMessageTraced in packages/runtime Reused via the planner loop. Single-node / inline execution useRunSingleNode.ts, runInlineGraphJob.ts Preview runs through the normal execution path. Agent serving path packages/websocket/src/agent/ (runs GraphPlanner today) First choice for serving CodePlanner; a thin tRPC mutation is the fallback. What the earlier draft built that this plan drops The previous version of this plan added an explicit “contract” layer on top of the node: a versioned CodeTransformationArtifact with schema, authoring-policy and sandbox-runtime versions, ui_properties.code_contract with port ordering, ui_properties.code_transform authoring history, contract-precedence rules that disabled AST inference, a contract editor with edge-migration dialogs, runtime contract enforcement in the kernel, and a three-table publish/version/draft store. All of it is dropped: No contract. Typed handles are dynamic_inputs/dynamic_outputs, which already persist. Record insertion order is the display order. If the user edits the code and the returned keys change, handles update through the existing inference path — same as today. No precedence rules, no mismatch dialogs, no “Update Contract / Revert” flow. No artifact versioning. The generation response is transient. Nothing is saved on the node except the standard fields. No authoring history, no policy versions, no original-intent storage, nothing new in workflow export. No authoring policy engine. The prompt is derived from the sandbox manifest, plus a few lines of guidance (below). Static analysis checks syntax and declared outputs — it does not classify code as “transformation” vs “not”. No publishing / reusable definitions. Deferred entirely (see Later). No runtime type enforcement, no saved test cases, no snippet folding. Deferred. Product scope The generator writes code for the shapes users actually ask for: reshape, merge/join, compute/derive, extract/parse, split, format/render, validate, seed. Ordinary conditional logic (if/else, switch, ternaries, early returns) is normal JavaScript and unrestricted. With the extended sandbox, CSV parsing, HTML extraction, date/number formatting, and hashing are all in scope without any library discussion. Prompt guidance (guidance, not enforcement): Prefer pure data reshaping; use fetch, workspace, or getSecret only when the instruction asks for it. Don’t use state or yield unless asked — they exist for streaming and are noise in a one-shot transformation. Emit every declared output on every return path. Conditional emission is workflow branching; point the user at nodetool.control.If / Switch instead of generating it. Preserve media/asset reference objects; never invent base64 conversions. Out of scope (unchanged from before): calling arbitrary NodeTool nodes from JS, replacing subgraphs/workflow nodes, new iteration models, user-installed packages, and AI authoring for the other nodetool.code.* executors (ExecutePython, ExecuteBash, …) — those run real interpreters with a fixed stdout/stderr shape and need their own generator later. Design One generation service, built as a mini-planner Don’t hand-roll a request/repair function. GraphPlanner (packages/agents/src/graph-planner.ts + tools/submit-graph-tool.ts) already implements the exact loop this feature needs: a system prompt, a forced submit tool, validation run inside the tool, and failures returned as tool results so the model resubmits over feedback rounds — with withAgentSpanGen tracing, abort linking, retry caps, and an eval harness attached. Code generation is the same shape with a smaller payload. Add a CodePlanner beside graph-planner.ts with one SubmitCodeTool (the submit_code analog of submit_graph): The planner accepts: instruction, input slots (name, type, description — seeded from connected handles), optional expected output (name + type, when launched from a destination handle), optional current code + slots (for edits), optional user-approved sample values, provider, and model. The prompt is built from the sandbox capability manifest and the guidance above. The model must call submit_code, whose arguments are: interface CodeGenSubmission { title: string; // imperative, becomes the node title summary: string; // one sentence code: string; inputs: { name; type; description?; default?; required? }[]; outputs: { name; type; description? }[]; } SubmitCodeTool validates the arguments with Zod (valid JS identifiers, NodeTool type metadata shapes, unique names, ≥1 output, bounded sizes), parses the code with acorn, and checks that every declared output is assigned on every statically visible return path. Failures come back as the tool result — same feedback-round mechanism as submit_graph — with rounds capped at 3. The planner returns the accepted submission or a discriminated error (provider unavailable, aborted, no valid submission, internal). Free text and code fences are never scraped; only the accepted tool call’s arguments count. Prompt building and analysis stay pure functions in packages/agents/src/code-gen/ so they test without a server, and the Tool base class, provider handling, and tracing come from the existing planner infrastructure rather than new code. The editor chat agent is not the transport: the dialog needs an atomic validated result with no thread state. But because CodePlanner is an ordinary planner, the chat agent can later invoke it as a tool without a second implementation. Telemetry keeps provider, model, tokens, cost, duration, and failure code; it does not record instructions, samples, or generated code. Applying a result The frontend applies an accepted submission as one undoable graph action: data.code ← code, node title ← title. data.dynamic_inputs ← input slots (as DynamicSlotMeta), data.dynamic_outputs ← output slots, defaults into dynamic_properties. When launched from a handle, create the edge(s) after isConnectable passes. On cancel or failure, no node or edge is left behind. That’s the whole persistence story. Reload shows the same typed handles because the protocol already round-trips them. Samples stay local by default Preview runs the node through runInlineGraphJob with the latest connected values (or hand-edited samples) — locally, through the normal execution path. Sample values are only included in the model prompt when the user ticks “include sample values”, and the exact payload is shown before sending. By default the model sees names and types only. This replaces the earlier consent/redaction machinery with a checkbox. Model selection Add code_model to the existing ModelPreferencesStore.defaults map. Resolution, centralized in useCodeAuthoringModel: code_model preference, if set. Active global chat model. Default language_model preference. Application default. Skip any candidate without tool support (the submission is a tool call); thread the existing requireToolSupport filter through LanguageModelSelect → LanguageModelMenuDialog, which don’t accept it yet. Never store the model on the node. UI Ask AI button on the Code Node body (CodeBody.tsx), gated on isCodeNode(nodeType) — nodetool.code.Code only, not isCodeBodyNode, which matches all 19 nodetool.code.* executors. One dialog (CodeGenDialog.tsx): instruction field, detected typed inputs, expected output when known, effective model (with a link to change the preference), Generate, result review (title, summary, handles, collapsed code view), preview panel, Apply. Monaco lazy-loads only when the code view opens. Keyboard operable; aria-live for generation and preview status; focus returns to the opener on close. Entry points: A searchable “Write Code with AI” palette entry (aliases: transform, convert, reshape, merge, join, split, extract, parse, format, compute, validate — following the snippetMetadata.ts search-term pattern). “Transform this output…” on the output-handle context menu. “Create value with AI…” on the input-handle context menu, seeding the expected output name and type. No compact summary body, no contract editor, no mode indicator. A generated node renders and edits exactly like any Code Node. Tasks 1. Sandbox capability manifest Files: packages/agents/src/js-sandbox.ts, prompt construction, useChatIntegration.ts. Export one serializable manifest of the sandbox surface — bridges, guest helpers, limits — built from the module’s own exports, never restating values. Scope it to nodetool.code.Code; the other executors don’t share this runtime. Derive the generation prompt and Code Node help text from the manifest. Fix useChatIntegration.ts, which still advertises _ (lodash), dayjs, and a global cheerio — none exist as guest globals. CSV and HTML work goes through data.parseCsv / data.selectHtml; formatting through format.*. Add a test that fails when authoring instructions name an API the manifest doesn’t contain. 2. Transport schemas Files: packages/protocol/src/api-schemas/code-gen.ts + tests. Zod schemas for request, CodeGenSubmission, and the error union. Identifier-safe port names, NodeTool type metadata shapes, unique names per direction, ≥1 output, size limits on instruction/ports/samples/code. Transport validation only — no JS parser in protocol. 3. Generation core (CodePlanner) Files: packages/agents/src/code-planner.ts, packages/agents/src/tools/submit-code-tool.ts, packages/agents/src/code-gen/{prompt,analyze}.ts + scripted-provider tests; add acorn to the agents package. Follow the GraphPlanner / SubmitGraphTool pattern: extend the Tool base class, validate inside the tool, return failures as tool results, cap feedback rounds at 3, trace with withAgentSpanGen, link aborts with the existing helper. Prompt builder: manifest + guidance + inputs + expected output + optional samples. Only the accepted submit_code call’s arguments count; free text and code fences are ignored. acorn analysis: syntax, every declared output assigned on every visible return path. Ternaries, if/else, partitions into always-emitted outputs all pass; a branch omitting a declared output fails with a pointer to the control nodes. Typed failure codes on the planner result. 4. Serving path Files: packages/websocket/src/agent/ or a thin trpc/routers/code-gen.ts, tests. First choice: serve CodePlanner through the existing agent WebSocket path (packages/websocket/src/agent/), which already handles provider resolution, secrets, cancellation, and progress streaming for GraphPlanner. Check whether a one-shot request/response fits that channel cleanly. Fallback if it doesn’t: a thin protected tRPC mutation that instantiates CodePlanner — glue only, no generation logic in the router. Either way: suppress instruction/sample/code bodies from spans and failure logs while keeping usage metadata; per-user in-flight cap with a stable rate-limited error; no chat thread, no message history writes. 5. Model preference Files: ModelPreferencesStore consumers, DefaultModelsMenu.tsx, LanguageModelSelect.tsx, LanguageModelMenuDialog.tsx, useCodeAuthoringModel.ts + tests. code_model preference labeled “Code Generation”, empty state “Use chat model”; fallback chain in the hook; requireToolSupport threaded through the two select components; blocking selector with a Settings link when no tool-capable model resolves. 6. Dialog and apply Files: web/src/components/node_types/code_gen/CodeGenDialog.tsx, web/src/serverState/codeGen.ts, a NodeStore apply action, CodeBody.tsx integration, component tests. The dialog described above; TanStack mutation; single undoable apply that sets code, slots, defaults, title, and edges; accessible states for loading, error, aborted, and offline. 7. Graph entry points Files: palette config, handle context menus, connection handlers, tests. The three entry points above, seeding slot names/types from handle metadata; edges only after validation; collision suffixes for duplicate port names; nothing left behind on cancel. 8. Preview Files: preview panel in the dialog, sample adapter, tests. Latest connected values where available, type-aware sample editors, “from latest run” vs “manual” labels; run via runInlineGraphJob; show each named output, logs, duration, errors; warn when an output value doesn’t match its declared type; preview state keyed by job id. “Include sample values” checkbox with the exact payload shown before the first send. 9. Verification and rollout Unit tests: schemas, prompt, analysis, repair, model fallback, apply round-trip. E2E: source output → generate → preview → connect → save → reload → run; and the destination-handle variant. A code-gen live-provider eval suite beside graph-planner, registered with nodetool eval (--provider, --model, --cases, --min-success), reporting first-pass and post-repair acceptance separately. Scripted-provider tests cover mechanics, not model quality. Feature flag in the web config layer until the E2E journeys and the eval suite pass ≥90% (post-repair) on two configured providers. npm run typecheck, npm run lint, npm run test; npm run build:packages when decorator packages change. Exit criteria A user can go from “I have these values” to a connected, typed, working node without opening Monaco. Typed handles survive save/reload (already guaranteed by the protocol — covered by the apply round-trip test). No generation path leaves a partially configured node or dangling edge. Existing Code Nodes, snippets, inference, and every other nodetool.code.* node behave exactly as before. Phase 2 — Edit and repair (after Phase 1 ships) Edit: the same planner call with the current code + slots attached; the model returns a full replacement submission. Confirm before applying an edit that removes, renames, or retypes a connected handle (show affected edges: keep, remap, or disconnect). Per-dialog transcript in UI state only. Repair: after a preview or run failure, offer one-click Repair — the edit call with the error and the failing (local) sample attached. Preview before replacing working code; keep the last working code until accepted. Connection adapter: when a connection fails only on type incompatibility, offer “Create adapter with AI” beside the error, and “Insert transformation” on an existing data edge — both pre-fill exact source and destination types. Later, deliberately unscheduled Reusable published definitions with versions (the dropped Phase 3). If demand shows up, it can be added without a contract layer: publish snapshots a node’s code + slots; nothing in this plan blocks it. Saved per-node test cases. Runtime output-type checking in the kernel. Declared per-node capabilities/libraries. AI authoring for ExecutePython and friends (needs per-runtime manifests — the manifest seam from Task 1 — and a typed-output convention for stdout-shaped nodes).--- # Example Apps — Curation Spec Source: https://docs.nodetool.ai/plans/example-apps.html Example Apps — Curation Spec Status: draft, 2026-07-26. Companion to applications-canonical.md T11. Scope: which of the 36 shipped templates in packages/base-nodes/nodetool/examples/nodetool-base/ get a shipped example app (an ApplicationBundle, T12), and what each app contains. No code. Today every template gets a generated app from the CURATION table in scripts/generate-template-apps.mjs — 36 variations of one form. This spec replaces that with 11 curated studio apps binding 23 workflows; the remaining templates ship as workflows only. A second set of 10 single-job apps (below) binds the templates listed below. Selection criteria Applied in priority order, recorded per app as C1/C2/C3: C1 — a real person wants the app, not the graph. A form, dashboard, or gallery is the natural interface for this job. C2 — it exercises the platform. Several workflows behind one surface, variables carrying state between operations, streaming, live property bindings. C3 — it runs on a fresh install, or degrades legibly. Keyless, local, or an explicit note naming the key and the cost. The apps App Workflows bound Operations Key widgets Criteria Needs Photo Studio Image Enhance, Photo Enhancement Suite enhance, batch ImageInput, 5 Sliders, Image, Table C1 C2 C3 none for enhance; FAL key for batch Meeting Room Transcribe Audio, Meeting Transcript Summarizer, Private Assistant transcribe, summarize, ask AudioInput, Markdown, Table, TextInput C1 C2 FAL (Whisper) + OpenAI; ask needs Ollama Concept Studio Concept Art Iteration Board, Pokemon Maker, Image Enhance concepts, creatures, polish Select, Slider, Image gallery, Button C1 C2 OpenAI + FAL; polish keyless Research Desk Research Agent, Hacker News Agent brief, pulse TextInput, Markdown ×2, Progress C1 C2 OpenAI (+ web search tool) Ask Your Documents Chat With Your Documents, Private Assistant ask, askLocal TextInput, Switch, Markdown ×2 C1 C2 C3 Ollama (embeddings, askLocal); OpenAI for ask Brand & Social Brand Asset Generator, Hook & Thumbnail Factory kit, thumbnails ColorInput, Image, Columns C1 C2 OpenAI + FAL Product Launch Kit Product Mockup Generator, Product Video Generator mockups, video ImageInput, Slider, Image, Video C1 C2 C3 OpenAI + FAL; Gemini (Veo) for video, cost note Film Studio Script to Screen, Directed Film to Timeline, Movie Posters produce, cut, poster TextInput, Select, Progress, Video, Image C1 C2 C3 OpenAI + FAL + Gemini (Veo) + Replicate; cost note Study Buddy Flashcard Generator, Prompt Template cards, explain NumberInput, Table, Markdown C1 C2 OpenAI Model Arena Model Arena compare TextInput, Columns, Markdown ×3 C1 C2 OpenAI + Anthropic + Gemini keys Dataset Builder Data Generator generate TextInput, NumberInput, Table C1 OpenAI Nine of eleven bind more than one workflow. Two are usable with no cloud key (Photo Studio’s enhance, Ask Your Documents in local mode). Per-app detail Output names below are the ones in the shipped graphs. Names marked † come from the Output-augmentation table in generate-template-apps.mjs; T12 must bake those Output nodes into the bundled graphs (see Prerequisites). 1. Photo Studio Generate-free, refine-heavy: the flagship for live property bindings. enhance — Image Enhance. Inputs: image from an ImageInput widget; five node properties bound to sliders — denoise-node.radius, tone-node.brightness, tone-node.contrast, color-node.saturation, sharpen-node.amount, all pace: "release". Policy replace. Output enhanced_image† → display. batch — Photo Enhancement Suite. Inputs: photos (ImageListInput), brightness_adjust / color_boost sliders. Output enhanced_photos† → variable batchResults, rendered as a grid. Variables: sourceImage (instance) shared by both operations so the single photo and the batch use the same drop zone. Degradation: enhance is pure GPU filters (lib.image.*) and runs with no keys; batch adds a FAL ImageToImage grade pass and says so in a note. 2. Meeting Room Recording in, minutes out, then question the transcript — the transcribe → summarize → interrogate chain across three workflows. transcribe — Transcribe Audio. Input audio from an AudioInput widget, event on change (auto-run on upload). Output transcript† → variable transcript. Policy replace. summarize — Meeting Transcript Summarizer. Input Transcript from variable transcript (the graph’s IsEmpty/If pair already skips its own ASR branch when a transcript is supplied). Outputs Summary → Markdown, Action Items → Table (it emits a dataframe), Transcript unused. ask — Private Assistant. Inputs document from variable transcript, question from a TextInput, tone from a Select. Output answer → Markdown. Local Ollama, so the follow-up Q&A costs nothing and works offline once the transcript exists. Variables: transcript (instance) is the whole point — one value written by op 1 and read by ops 2 and 3. 3. Concept Studio Generate a gallery, pick one, polish it. The creative iteration loop. concepts — Concept Art Iteration Board. Inputs creative_brief, art_style, mood_keywords (TextInput/Select), variations (Slider 1–8). Output Concept Art → append-disposition gallery. creatures — Pokemon Maker. Inputs animals (TextInput), style (Select fed from the graph’s existing SelectInput options: cel-shaded, 3D render, watercolor, 16-bit, holographic). Output pokemon† → the same gallery. polish — Image Enhance, bound a second time with different mappings: input image from variable picked, sliders as in Photo Studio. Policy replace. Variables: picked (instance) — the gallery selection feeds polish. Both generators use ListGenerator → TextToImage, so results stream in one at a time; a Progress widget shows the batch filling. 4. Research Desk One topic, two sources, two streamed briefings side by side — the case where a dashboard beats a chat box. brief — Research Agent. Inputs topic (variable topic), audience (Select: engineers / executives / general). Output brief → Markdown, streams as the agent works. pulse — Hacker News Agent. Input topic from the same variable. Output analysis → Markdown. Both run parallel from one Button, each into its own panel with its own Progress label. 5. Ask Your Documents RAG with a legible offline fallback. ask — Chat With Your Documents. Inputs question, search, and the three document text areas doc_specs / doc_charging / doc_warranty (shipped with the bundled Aurora sample text). Outputs Answer → Markdown, Retrieved Passages → Markdown (the citation panel). askLocal — Private Assistant. Inputs document from variable pastedDoc, question from the same question variable. Output answer → Markdown. Widgets: a Switch bound to variable localOnly decides which operation the Run button triggers, and visibleWhen swaps the results panel. Note: the vector collection embeds with Ollama nomic-embed-text, so ask needs Ollama running and an OpenAI key; askLocal needs only Ollama. The app note must say this — it is the one place a user can be surprised. 6. Brand & Social The marketing dashboard: one brand identity drives two deliverables. kit — Brand Asset Generator. Inputs brand_name, brand_description, tagline (TextInputs, written into variables), primary_color (ColorInput). Outputs social_assets → Image grid, brand_brief → Markdown. thumbnails — Hook & Thumbnail Factory. Inputs Video Topic (TextInput), Target Audience from variable audience, Number of Hooks (Slider 3–8). Output thumbnail_gallery → gallery. Variables: brandName, audience, voice (user-scoped, persist: true) — fill them once, every operation reuses them across sessions. 7. Product Launch Kit Photo in, mockups and a launch video out; the expensive step is opt-in. mockups — Product Mockup Generator. Inputs product_image (ImageInput → variable productPhoto), product_name, product_description, target_audience, num_scenes (Slider 1–6). Outputs mockup† → gallery, scene† → Markdown shot list. video — Product Video Generator. Inputs image_input_1 from variable productPhoto, campaign_brief, target_audience (shared variable), key_features. Output product_video† → Video. Its own Button with a cost note; policy queue, timeoutMs generous — Veo runs are long. Runs mockups on a fresh install without touching Veo, so the app is useful at the FAL tier alone. 8. Film Studio The showcase run: brief → direction → storyboard → cut → key art. produce — Script to Screen. Inputs Brief (variable brief), Visual Style (Select), Shot Count (Slider 3–8). Outputs direction → Markdown, storyboard → gallery, film → Video. cut — Directed Film to Timeline. Input Brief from the same variable. Output film → Video, presented as “the editable rough cut”. poster — Movie Posters. Inputs Movie Title, Genre, Visual Style (shared Select). Output Poster → Image. Progress widget per operation; policy queue throughout; a standing note that this app spends real money (Veo 3.1 + Replicate MusicGen) and should be run once, deliberately. 9. Study Buddy Structured data an app renders better than a graph does. cards — Flashcard Generator. Inputs topic (variable topic), num_cards (NumberInput). Output Flashcards → Table, with front/back columns; the graph persists them in SQLite, so re-opening the app shows the stored set. explain — Prompt Template. Inputs Topic from the same variable, Audience (Select). Output Explanation → Markdown, shown beside the cards as “the concept behind this deck”. Variables: topic (user-scoped, persisted) is the study session. 10. Model Arena Single workflow, but the app is the point: three answers in three columns is unreadable on a canvas. compare — Model Arena. Inputs brief, context (TextInputs). Outputs openai, anthropic, gemini → three Markdown panels inside a Columns layout, each streaming independently with its own Progress label. Degradation: a missing provider key fails one column, not the run. The note names all three keys. 11. Dataset Builder generate — Data Generator. Inputs topic (TextInput), row_count (NumberInput 5–50). Output generated_data → Table. The smallest app in the set, kept because a table view of a dataframe is a genuinely better surface than a Preview node, and it is the reference for the Table widget. Single-job apps Retired: Product Reshoot, Product Shot Video, Multi-Shot Video, Scene Builder, Video Restyle, AI Spokesperson, Vertical Cut and Upscale Image no longer ship. Each repeated one operation of a larger app (SKU Factory, Trailer Room, Film Studio, AI Video Post House, Dubbing Desk, Viral Ad Engine). Their templates still ship as workflows. The rows below record the original design. The studios above put several jobs behind one surface. These ten are the opposite shape, the one a Runway-style app has: one upload, a few choices, one result. Each is a media job somebody already knows they want, so the app is the whole product and the graph is an implementation detail. C1 applies to every one; C2 comes from variables carrying one upload into several operations, and from Select and Slider widgets driving node properties inside the graph rather than Input nodes. Six jobs in the Runway gallery have no counterpart here because no shipped template does them: video upscaling, removing an object from a video, driving a performance from a reference take, slide design, localizing an ad image, and outpainting or expanding an aspect ratio. Each needs a graph first, so adding one is a template change before it is an app change. Four of the ten are not literally one operation: Product Reshoot, Product Shot Video, Upscale Image and Ad Maker put two or three operations behind a single upload, each with its own button and result panel. The shape still holds, since the choice is which treatment to run, not how to wire a pipeline. App Workflows bound Operations Key widgets Needs Vary Image Edit a Still with Words edit ImageInput, Select (what to change), TextInput (extra directions), Image FAL Product Reshoot Put a Product on a Studio Backdrop, Relight a Product for a Seasonal Campaign, Cut a Product Out of Its Background backdrop, relight, cutout ImageInput, Select (comp.prompt), Select (rl.prompt), Image ×3 FAL Product Shot Video Ad Loop from a Product Photo, Spin a Packshot into a Turntable Clip loop, turntable ImageInput, Select, Select (v.prompt), Video ×2 KIE for loop, FAL for turntable Multi-Shot Video Movie Trailer Generator trailer TextInput, Select, Slider (shot count), Video Gemini (director, Veo) + KIE (keyframes); cost note Scene Builder Editorial Still from a Line, Bring a Still to Life look, motion TextInput, Image, Select, Slider (vid.duration), Video FAL Video Restyle Video Restyle Studio restyle WorkflowInput (video), Select, TextInput, Video Replicate AI Spokesperson AI Spokesperson revoice WorkflowInput (video), TextInput, Video Replicate (voice) + FAL (lip-sync) Upscale Image Upscale a Still, Take a Product Shot to Print Resolution faithful, clarity ImageInput, Slider (up.scale) ×2, Image ×2 FAL Vertical Cut Cut a Landscape Clip for Vertical, Pull a Still from a Clip vertical, cover VideoInput, Slider (frame.time), Video, Image none (ffmpeg) Ad Maker Ad Copy in Three Registers, Five Headlines for a Landing Page, Write the Prompt, Then Make the Image copy, headlines, visual TextInput, Markdown ×3, Image OpenAI + FAL Per-app notes, where the table does not say it all: Vary Image. The Select is bound to the instruction Input, so each option is a complete edit instruction; the first one is the graph’s own default. A second TextInput is bound to the directions Input, which the template’s tpl node appends to the instruction, so a typed line can qualify the chosen option instead of replacing it. Leaving it empty changes nothing. There is no strength control: the edit model declares no such input, and a slider bound to a property the provider drops would promise a dial that does nothing. Product Reshoot. One productPhoto variable feeds all three operations. The setting and season selects drive the prompt property of the edit and relight nodes, so the graph’s Input nodes stay as they are. Scene Builder. look writes its picture output to the still variable, which motion reads as its still input. The still is shown from the variable, so the user sees exactly the frame that will be animated. The duration slider offers only lengths the LTX fast endpoint accepts at the template’s resolution; a value outside its list is dropped and the endpoint falls back to its default, billing a clip the user did not ask for. Vertical Cut. The only keyless app in this set. One clip variable of type video feeds both operations, and one button runs them in parallel. Ad Maker. The offer variable is user-scoped and persisted; all three agents read it and one button starts them in parallel. visual maps the offer to the template’s idea input. Templates that stay workflow-only These templates were considered and left without an app: Template Reason Audio To Image One input, one image; the template card already tells the whole story. Cold Outreach Co-Pilot Preview-only terminals, and the structured draft fields need an app-side email composer this spec does not define. Color Boost Video Per-frame grading of an uploaded video is minutes long with one slider — a bad first-run experience; the sliders belong in the editor. Conditional Logic Engine Teaching example. Its value is the graph. Image To Audio Story One image in, one audio clip out — a Run button with extra steps. Image to Video Animation Text-to-image → Veo; it has no image input, so it cannot be the “animate this” step of any gallery app (see Findings). Music Video Visualizer Strong candidate deferred: one upload, one long paid run, one video. Revisit once a Podcast/Audio app justifies the surface. Podcast Repurposing Studio Preview-only terminals; needs Output nodes and structured quote-card output before an app can render it. SEO Content Engine Preview-only terminals; the typed article fields want a per-article editor, not a results panel. Workflow As A Tool Composition demo for graph authors. An app hides exactly what it teaches. Prerequisites and findings Things T12 must handle, found while reading the graphs: Output augmentation must move into the graphs. Eight bound workflows have no Output node today and rely on the AUGMENT table in generate-template-apps.mjs: Transcribe Audio, Summarize RSS, Pokemon Maker, Image Enhance, Photo Enhancement Suite, Product Mockup Generator, Product Video Generator, Social Media Calendar Filler (2 of its 3 outputs). Bundles carry graphs, not a generator, so those nodes must be committed to the example JSON. Preview-only templates cannot back an app. Ad Creative Factory, Cold Outreach Co-Pilot, Podcast Repurposing Studio, and SEO Content Engine end in nodetool.workflows.base_node.Preview with zero Outputs. All four are in the workflow-only list for that reason; promoting any of them later is a graph change first. Image to Video Animation has no image input. The obvious “animate the concept I picked” operation for Concept Studio is not buildable against the shipped graph (TextToImage → ImageToVideo). Concept Studio uses Image Enhance for its second step instead; adding an ImageInput to that template is a separate, small change if the animate step is wanted. Chat With Your Documents needs Ollama even in “cloud” mode. Its vector.Collection embeds with nomic-embed-text on Ollama while the answer agent is OpenAI gpt-5-mini. Either the app note says both, or the template switches to a cloud embedding model. Multi-binding is real here. Image Enhance is bound by two apps (Photo Studio, Concept Studio) and Private Assistant by two (Meeting Room, Ask Your Documents) — with different input mappings each time. The bundle format’s workflows[].key indirection covers this; the installer must not create duplicate workflow rows when both bundles are installed. Marketing impact. /apps/* drops from 36 generated pages to 11 curated ones. Every retired slug needs a redirect to the app that absorbed it (or to its /templates/<slug> page, which survives) — the redirect map is T13 work, but the mapping is this table.--- # Examples Revamp Plan Source: https://docs.nodetool.ai/plans/examples-revamp.html Examples Revamp Plan Status: draft, 2026-07-17. Scope: the 37 shipped templates in packages/base-nodes/nodetool/examples/nodetool-base/ and their generated mini apps (scripts/generate-template-apps.mjs, web/public/app-preview/). Why The examples are the product’s first impression three times over: the template gallery in the editor, the mini apps on /apps/* marketing pages, and the first thing a new user runs. Today they undersell on all three surfaces. The workflows are mostly linear prompt pipes that don’t justify a node editor, and the generated apps are 37 copies of the same form. Meanwhile the platform just gained the features that would make them compelling — node-property bindings, reactive subgraph runs, run pacing, container-responsive app layout — and no example uses any of them. Analysis Measured across all 37 templates (graph JSON + generated app_doc): Workflow shape 15 of 37 graphs are purely linear (max fan-out 1); median size ~7 nodes. The dominant shape is Input → Prompt template → LLM → Output — a prompt wrapper that a chat box does better. Control flow is nearly absent: 14 nodetool.control nodes and 6 nodetool.boolean nodes across the whole set, almost all in one teaching example (Conditional Logic Engine). Node namespace census: nodetool.input 72, nodetool.text 60, nodetool.agents 24, nodetool.generators 19, image 34, video 11, lib.sqlite 3. No vectorstore/RAG usage, no workflow-as-tool composition, no streaming showcase. Models gpt-5-mini appears 41 times — effectively every text node. Images are all fal-ai/flux/schnell (16). Video is veo-3.1 (5, expensive). Providers: openai 43, fal 22, gemini 6, claude_agent_sdk 1. Zero local models (Ollama/MLX). A fresh install without an OpenAI key has ~30 broken examples and no private/offline example at all. Content hygiene Shipped prompt text includes typos and one-liners (“Create a 3 compelling Pokémons”). No system prompts, no output-format instructions. Prompts are the actual product of these workflows and read as first drafts. Hardcoded content that should be inputs: RSS feed URL (BBC Europe), search topics, folder paths. FolderPathInput appears in two templates and is meaningless in a hosted app. Node titles/descriptions are sparse; comments decorate rather than teach. Outputs 30 of 37 templates have exactly one Output; nearly everything funnels into a text/markdown blob. Flashcards, calendars, SEO fields, and generated data are prose instead of structured objects — which caps what the apps can render (widget census over all 37 apps: 28 Markdown + 37 Text vs 13 Image, 7 Video, 1 Audio, 1 Json). Apps All 37 app docs share one generated shape: heading, tagline, input form, Run button, results panel. Zero Slider/Select/Switch widgets anywhere (71 WorkflowInput widgets, 37 Buttons). 8 templates have no inputs at all — their app is a Run button (Agent Google Search, Color Boost Video, Data Generator, Fetch Papers, Hacker News Agent, Image To Audio Story, Summarize RSS, YouTube Research Agent). Apps open with empty fields (seeded demo values exist only in the marketing preview bundles), and none use pacing, node-property bindings, variables, or an iteration loop (regenerate/variants/gallery). Marketing surface The examples feed the marketing site directly: every template gets a /apps/<slug> landing page (marketing/scripts/generate-miniapp-entries.mjs → miniAppEntries.generated.ts, 37 pages live), a screenshot (web/scripts/screenshot-app-previews.mjs → marketing/public/apps/<slug>.png, 37 present), and a /templates/<slug> page via the generated template catalog. Consequences of the current state: All 37 screenshots show the same empty form layout — a grid of visually identical pages that read as one product with 37 names. Conversion surface, currently underselling. The 8 no-input apps are marketed as apps while being a Run button. Landing pages exist for near-duplicate templates, splitting search intent (three YouTube/research agents, two thumbnail pipelines, three audio summarizers) instead of concentrating it on one strong page each. Root cause The generator (generate-template-apps.mjs) is sound infrastructure with a timid curation table, and the workflows were authored to demo single nodes, not to be products. Both are fixable without new platform features — the platform is now ahead of its own examples. Goals Every shipped example runs on first try and passes nodetool validate. Every example demonstrates at least one thing a chat box can’t do (parallel comparison, media pipeline, control flow, RAG, live parameters, structured data). A keyless install has at least 3 fully working local examples. The featured apps feel like tools: seeded defaults, real controls, visible progress, an iteration loop. Fewer, better: target ~20 curated examples instead of 37 near-duplicates. Plan Phase 0 — Guardrails (do first, half a day) CI job: run nodetool validate over every example JSON; --warnings-as-errors for unknown types, dangling edges, unselected models. This is the rot detector for everything that follows. Add a cheap smoke tier: nodetool debug on 3–4 keyless/local examples. Skip paid-model examples in CI; run them in a manual pre-release pass. Phase 1 — Hygiene pass on all survivors (1–2 days) For each kept template: Rewrite prompts properly: system prompt, task structure, explicit output format, no typos. Prompts are the product; treat them like code review. Parametrize hardcoded values into inputs (feed URL, topic, style). Name and describe every node; make comments explain the why of the graph. Set sensible input defaults so Run works with zero typing. Model pass: keep a cheap fast default for text, but vary providers deliberately where it shows breadth; flag expensive nodes (veo) in the description with a cost note. Phase 2 — Restructure the catalog (2–3 days) Disposition of all 37 (keep = hygiene only; fix = keep + targeted change; merge/retire = fold into another or drop): Template Disposition Change Ad Creative Factory keep showcase; structured variant outputs Agent Google Search merge into Research Agent (topic input, streaming) Audio To Image keep starter; seed demo audio Brand Asset Generator keep hero app Cold Outreach Co-Pilot fix structured outputs (draft objects) Color Boost Video fix VideoInput + slider-bound intensity (live grading) Concept Art Iteration Board keep hero; variant gallery loop Conditional Logic Engine keep teaching example for control flow Creative Story Ideas merge into one “Prompt Template” starter Data Generator fix schema input → dataframe output → table app Fetch Papers merge into Research Paper Summarizer (topic input) Flashcard Generator fix emit card objects, card-flip app rendering Hacker News Agent fix topic input, streaming markdown Hook & Thumbnail Factory keep hero (absorbs YouTube Thumbnail Pipeline) Image Enhance fix FLAGSHIP: sliders bound to node properties, pace: release Image To Audio Story fix add ImageInput Image to Video Animation keep   Learning Path Generator merge into Prompt Template starter Meeting Transcript Summarizer fix action items as dataframe (absorbs Summarize Audio) Movie Posters keep hero Movie Trailer Generator keep showcase; cost note Music Video Visualizer keep showcase Photo Enhancement Suite fix replace FolderPathInput (multi-image) or mark local-only Podcast Repurposing Studio keep showcase Pokemon Maker fix hero; rewrite prompt, style Select, variant gallery Product Mockup Generator keep hero Product Video Generator keep   Research Paper Summarizer fix absorbs Fetch Papers SEO Content Engine fix typed field outputs Social Media Calendar Filler fix calendar as dataframe Story to Video Generator retire overlaps Movie Trailer; FolderPath-bound Summarize Audio merge into Meeting Transcript Summarizer Summarize RSS fix feed URL input; digest by topic Transcribe Audio fix instant tool: auto-run on upload (change event, paced) Wikipedia Agent merge into Research Agent YouTube Research Agent merge into Research Agent YouTube Thumbnail Pipeline merge into Hook & Thumbnail Factory Net: 37 → ~24, minus retire → ~23, plus new below → ~28; prune to ~20 at final curation if the merged families hold up. Marketing follow-through for every merge/retire: keep the old /apps/<slug> and /templates/<slug> URLs as redirects to the absorbing template’s page (the slugs are already indexed), and grep docs/tutorials for retired template names in the same pass. Phase 3 — New differentiator examples (2–3 days) Each maps to a marketing story the catalog currently can’t tell: Chat With Your Documents — RAG: index a folder into the vectorstore, ask questions with cited answers. (First vectorstore usage in examples.) Research Agent — the merged agent family: topic input, real tool use (search + browse), streaming markdown brief with citations. Model Arena — one brief fanned out to 3 providers side-by-side. Earns the graph visually and doubles as a great comparison app. Private Assistant (local) — Ollama/MLX end-to-end, no API keys. Along with 2 other local-capable starters, satisfies the keyless goal. Workflow as a Tool — a workflow exposed via tool_name, consumed by an agent in another workflow. Shows composition, which nothing demos. Phase 4 — App layer (2 days, extends the curation table) Extend generate-template-apps.mjs curation entries with per-template hints instead of hand-editing app docs: sliders: node-property bindings with ranges (autofill already derives min/max/step from metadata), pace: "release" for anything expensive. selects: finite choices (aspect ratio, style, tone, platform) instead of free-text inputs. seeds: shipped default values in the app itself, not just the marketing preview — first Run must work with zero typing. gallery: append-disposition outputs rendered as a grid with a “Make another” button for the creative heroes. featured: flag the ~8 hero apps for the gallery and /apps landing pages (Image Enhance, Pokemon Maker, Movie Posters, Brand Asset Generator, Hook & Thumbnail Factory, Product Mockup Generator, Concept Art Iteration Board, Music Video Visualizer). Progress widget on every template whose run exceeds a few seconds. Regenerate previews and /apps pages (screenshot-app-previews.mjs, marketing/scripts/generate-miniapp-entries.mjs) as the last step. Phase 4b — Marketing page impact What the revamp changes on the marketing site, and the work it implies: Screenshots become the sell. Today’s 37 near-identical empty-form screenshots become ~20 distinct ones showing seeded results: a photo mid-enhancement with sliders, a Pokémon variant gallery, a model-arena comparison, a calendar table. The screenshot pipeline needs seeded values rendered (the preview bundles already seed; verify the new gallery/slider states render before the screenshot settles). Fewer, stronger landing pages. ~20 pages that each answer one search intent instead of 37 splitting it. Merged/retired slugs redirect (see Phase 2); the /apps index orders by the featured flag with heroes first. New pages with real search demand. The Phase 3 examples add landing pages for terms the site can’t currently target: “chat with your documents / RAG”, “compare AI models side by side”, “local private AI assistant”, “AI research agent”. These are stronger queries than most existing template names. Copy pass per page. Landing copy currently describes the workflow; rewrite to describe the outcome and name the differentiator (“drag the slider, watch the photo update — runs the graph live”), reusing each template’s new description field so app, gallery, and landing page say the same thing. Mechanics. All of this flows through the existing generators — the work is the redirects map, the featured ordering, and the copy fields in the curation table; no new marketing infrastructure. Entries without a fresh screenshot stay noindexed by the existing rule, which safely gates half-finished templates out of search. Phase 5 — Verification and release nodetool validate green across the catalog (CI from Phase 0). nodetool app debug per hero app: bindings resolve, run trigger exists, display widgets receive values. Manual pass on paid-model examples before release; screenshot refresh. Sequencing and effort Phases 0–1 are independent of 2–4 and land first (~2 days, pure win, no decisions needed). Phase 2 dispositions need a maintainer sign-off on the merge/retire list, then 2–4 parallelize by template family. Total ~7–9 working days; the flagship (Image Enhance live editor) is worth pulling forward as the proof piece for the whole direction. Risks Cost: veo/fal examples are expensive to smoke-test. Mitigation: CI validates but doesn’t execute paid models; manual pre-release pass only. Model drift: pinned model ids rot. Mitigation: Phase 0 CI catches unknown models at validate time; prefer provider-default aliases where the registry supports them. User familiarity: retired/merged templates may be referenced in docs and tutorials. Mitigation: grep docs for template names during Phase 2; keep slugs redirecting on the marketing site. Scope creep: structured outputs may reveal missing table/gallery rendering in app widgets. The Json/Output widgets cover the minimum; a dedicated Table widget is a fast follow, not a blocker. Success criteria Fresh install, no keys: 3+ examples run end-to-end locally. Every example passes validate in CI; hero apps pass app debug. Every kept example demonstrates a named differentiator (recorded in its description). Featured apps: seeded first run, at least one real control (slider/select), progress on long runs, and an iteration loop on the creative ones. Marketing: no dead /apps or /templates URLs (redirects in place), every indexed page has a distinct seeded-result screenshot, heroes lead the /apps index.--- # Higgsfield provider research and implementation plan Source: https://docs.nodetool.ai/plans/higgsfield-provider-research.html Higgsfield provider research and implementation plan Status: Implemented for the current image and video endpoint catalog, including the Genjutsu and Cinema Studio endpoints. Undocumented artifact families remain out of scope. Question: How should NodeTool integrate the Higgsfield API without losing model-specific capability, reliable asynchronous execution, or media safety? 1. Recommendation Build higgsfield as a manifest-backed media provider and node pack. Use a small, direct REST client for authentication, upload, submit, poll, cancel, estimate, and result download. Do not use @higgsfield/client as the runtime dependency: its documented TypeScript surface is oriented around automatic polling, while NodeTool needs to own request IDs, cancellation, timeouts, telemetry, durable generation lookup, and safe result ingestion. Start with image and video models, including text-to-image, image editing, text-to-video, image-to-video, reference-to-video, video editing, video extension, and motion-transfer endpoints when their individual schemas are verified. The shared API also describes audio and 3D artifacts. Add those only after one public, model-specific schema and a NodeTool output contract are verified for each artifact type. The provider must be manifest-backed, not OpenAPI-backed. Higgsfield’s own documentation says that /docs/openapi.json is supplementary and not an authoritative model catalog. It currently contains only a small subset of the public catalog and omits documented upload and estimate endpoints. The public model explorer exposes endpoint-specific IDs and parameter tables, but no documented machine-readable catalog endpoint was found. The first shipped manifest should therefore be curated from verified model pages and protected by a source-link and schema-drift test. A catalog sync job is appropriate only when Higgsfield documents a stable listing endpoint or supplies a supported export. 2. Verified API surface Area Observed contract NodeTool consequence Authentication Every API call uses Authorization: Key <key-id>:<secret>. Legacy hf-api-key and hf-secret headers exist but are discouraged. Store HIGGSFIELD_API_KEY_ID and HIGGSFIELD_API_KEY_SECRET separately, compose the header only at the request boundary, and redact both from logs and spans. Model invocation POST https://api.higgsfield.ai/<model-id> submits JSON and returns a request controller. Each model has its own endpoint and input schema. Model ID is a manifest field. The factory creates typed node properties from each verified schema rather than maintaining generic, lossy image and video forms. Lifecycle Submission returns request_id, status_url, and cancel_url. States are queued, in_progress, completed, failed, nsfw, and canceled. Only queued work can be canceled. Persist the provider request ID immediately. Use returned controller URLs during the live run. When only the ID survives, recover through the documented ID-based endpoints described in P3. Translate completed to success, failed and nsfw to terminal generation errors, and canceled to NodeTool cancellation. Output Completed responses can include images, video, audio, or audios. Shared docs also name zip, mov, jsx, fbx, and ply artifacts for some video and 3D operations. The transport must extract every known output URL, then the factory must expose all semantically distinct outputs. Do not ship a model whose multiple artifacts cannot be represented by declared slots. Input media A caller can pass public HTTPS URLs, or create a presigned URL at POST /files/generate-upload-url, PUT bytes with all returned headers, then pass public_url. Upload URLs expire after one hour. Documented types are JPEG, PNG, WebP, GIF, WAV, and MP4. Resolve NodeTool AssetRef bytes, select and validate the MIME type, request an upload URL, upload without Higgsfield credentials, and pass the returned public URL. Reuse the package’s safe external-media controls for downloads. Polling Higgsfield recommends a two-second initial delay, 1.5x backoff up to ten seconds, jitter, and an application-level timeout. Retry only polling GETs for transport/5xx failures. Implement abort-aware polling with those bounds. Never retry a submission POST after an ambiguous failure because submissions have no idempotency key. Webhooks hf_webhook is a submission query parameter. Higgsfield POSTs terminal results, may deliver duplicates, retries network/5xx failures for up to two hours, and expects a response within ten seconds. The documentation does not describe a webhook signature. Do not expose webhooks in the first release. Polling is sufficient for in-process workflow runs. Add a public route only after Higgsfield provides an authenticity mechanism or an explicit NodeTool policy accepts an unauthenticated, schema-validated callback. Errors and limits 400 can mean invalid input or concurrency reached. 401, 403, 404, 422, 423, 500, and 503 have documented meanings. Limits are account/model concurrency, with no standard rate-limit or Retry-After headers. Each response carries X-Correlation-ID. Classify errors without parsing message text for durable decisions. Bound concurrent submissions per credential, retry polling 5xx only, and attach correlation ID plus request ID to structured diagnostics. Billing POST /estimate/<model-id> accepts the generation parameters and returns account-specific credits and usd. Successful generations are charged. Failed, NSFW, and successfully canceled queued work are refunded. Output URLs last at least seven days. Add an optional estimate API and cost preview after the execution path ships. Absorb completed media into NodeTool storage immediately, because Higgsfield output is transient. Do not invent static price catalog entries from storefront prices. SDKs Official Python and TypeScript SDKs cover auth, uploads, polling, and cancellation. The documented TypeScript v2 example uses subscribe(..., { withPolling: true }); explicit lifecycle control is documented via Python. Direct REST avoids a runtime dependency and supports the lifecycle NodeTool actually needs. The SDKs remain useful only as contract references and optional test comparators. Sources: documentation index, authentication, request lifecycle, polling, uploads, errors, rate limits, billing and retention, webhooks, and SDKs. 3. Model catalog and feature coverage Higgsfield presents a broad, changing multi-vendor catalog behind the common asynchronous contract. The public explorer currently exposes these product families: Family Verified examples and feature shape Integration treatment Text-to-image Soul 2, Marketing Studio Image, Recraft 4.1, Qwen Image 3, Z-Image Turbo Typed image nodes. Preserve each model’s resolution, aspect ratio, batch, seed, output-format, styling, and prompt-enhancement options. Image editing Marketing Studio Image accepts image_urls. Grok Imagine Image 2.0 accepts optional image_urls and has quality, resolution, and aspect-ratio controls. Expose typed image or list[image] inputs, never raw URL strings. Text-to-video Seedance 2.5, Kling 3.0, MiniMax H3, Wan 3.0, LTX 2.5, Grok Imagine Video, Cinema Studio Typed video nodes with model-specific duration, resolution, aspect ratio, sound/audio, output-format, camera, CFG, multi-shot, and multi-prompt properties. Image/reference-to-video Seedance, Kling, Grok, and related variants expose image or reference inputs. Map images, video, and audio references to typed input fields. When the upstream schema has one heterogeneous reference array, use the AtlasCloud wrapInto approach to preserve media typing. Video transformation Seedance video edit/extend, Kling video edit/reference/motion control, Genjutsu motion transfer/object swap Ship only after verifying every input and every output from that model page. These belong to video_to_video or reference_to_video, not a generic text-to-video node. Marketing workflows Marketing Studio has direct generation/editing and preset-enhanced mode. Enhanced requests require one product image, preset_id, and a live visible-preset catalog. They cost 10% more. Add a dedicated preset-list API/tool and a specialized node or dynamic property source. Never bake CMS-managed preset UUIDs into the manifest. Audio and 3D The common output and webhook contracts mention audio and 3D artifacts, but the public catalog and supplementary OpenAPI inspected here do not provide a complete verifiable model list or typed schema for them. Treat as a deliberately deferred capability, not proof that it is unsupported. Representative schema evidence shows why a generic form would lose value: Seedance 2.5 takes prompt, 4–30 second duration, 480p/720p resolution, six aspect ratios, MP4/MOV output, and optional audio. Kling 3.0 Standard adds sound, element IDs, CFG scale, multi-shot, and multi_prompt. Marketing Studio Image has a paginated live preset list and distinct direct/editing versus enhanced modes. Grok Imagine Image 2.0 supports image editing, quality, 1K/2K resolution, and a distinct aspect ratio set. Recraft 4.1 accepts structured color and background-color objects in addition to image generation controls. The public model explorer is discovery evidence, not a stable API contract. It shows the current direct image and video families and links each family to endpoint-specific variants. The manifest maps those direct endpoints, including the related-model variants listed on the official family pages. Genjutsu motion transfer, Genjutsu object swap, and Cinema Studio 4.0 now publish API-reference pages with the same POST https://api.higgsfield.ai/<model-id> contract and a typed input schema, so the manifest includes them. The explorer’s Product Shots, Graphic Ads, and Marketplace Design workflows route to Marketing Studio Image and need no separate entry. The catalog also lists higgsfiled/genjutsu/motion-transfer/v1.0, a misspelled alias of the Genjutsu motion-transfer workflow, which the manifest omits. See the public explorer, image explorer, and video explorer. Current direct endpoint catalog The checked-in manifest covers every endpoint reachable from the explorer’s catalog families and their related-model tables. Three inferred routes were removed after the official API pages returned 404s: Grok Imagine Video text-to-video and image-to-video siblings, plus Kling 2.5 Turbo Standard text-to-video. Image: Marketing Studio Image with its 2.5 Flare and 2.5 Sunburst variants, Grok Imagine 2.0, Soul 2, Ideogram 4.0, Recraft 4.1 with its Pro, Utility, and Utility Pro variants, Soul Standard, Qwen Image 3 text and edit, and Z-Image Turbo. Higgsfield: Cinema Studio 4.0 and Genjutsu motion transfer and object swap. ByteDance: Seedance 2.5 and Seedance 2.0 text, image, reference, edit, and extension endpoints. Kling: 3.0 standard, pro, 4K, turbo, and motion-control endpoints, O3 first/last-frame, image-reference, video-edit, and video-reference endpoints, 2.6 text/image/motion-control endpoints, Omni first/last-frame, image-reference, video-edit, and video-reference endpoints, and 2.5 Turbo standard image and pro text/image endpoints. Alibaba, MiniMax, Lightricks, xAI, and PixVerse: Wan 3.0 Prime, Wan 3.0, Wan 2.7, Wan 2.6, Happy Horse 1.1, Happy Horse 1.0, MiniMax H3, MiniMax Hailuo 2.3 Standard, LTX 2.5 Fast/Pro, Grok Imagine Video 1.5, and PixVerse V6 across the direct modes exposed by their official related-model tables. The endpoint IDs and field lists are stored in packages/higgsfield-nodes/src/higgsfield-manifest.json, with source links and verified field names in packages/higgsfield-nodes/src/schema-fixtures/catalog.json. The official MiniMax H3 family page, Wan family page, Kling family page, and LTX family page provide the related endpoint tables used for this expansion. 4. Implementation plan P1 — Establish the package and credential boundary Add packages/higgsfield-nodes/ to the root workspace and create its AGENTS.md plus CLAUDE.md pointer. Model it on atlascloud-nodes because both are asynchronous multi-model media APIs. Add HIGGSFIELD_API_KEY_ID and HIGGSFIELD_API_KEY_SECRET to packages/config/src/setting-catalog.ts. Register PROVIDER_IDS.HIGGSFIELD in packages/protocol/src/api-types.ts, the provider registry, built-in pack catalog, node search prefixes, and Settings labels. Create higgsfield-transport.ts in packages/runtime/src/providers/ with exact Zod-validated response shapes for submission, status, upload URL, estimate, error, and media outputs. Use Authorization: Key id:secret only at this boundary. Instrument each external request and retain the response correlation ID. Implement abort-aware helpers: higgsfieldCreateUploadUrl, higgsfieldUploadMedia, higgsfieldSubmit, higgsfieldGetStatus, higgsfieldAwaitResult, higgsfieldCancel, higgsfieldEstimate, and higgsfieldDownloadResult. Retry only idempotent status GETs and billed result downloads, never job-creating POSTs. P2 — Ship manifest-driven image and video nodes Add higgsfield-manifest.json, higgsfield-factory.ts, and higgsfield-base.ts to packages/higgsfield-nodes/. Reuse the current factory conventions: typed AssetRef defaults, metadataOutputTypes, loadMediaRefBytes, and fetchExternalMedia for billed outputs. Define a manifest schema with: stable model ID, title, source documentation URL, modality/task classification, output slots, poll budget, parameter schema, asset-field mapping, and an explicit lifecycle/config version. Seed it with the current source-backed direct endpoint catalog. Every manifest entry must have a source link, a catalog fixture entry, typed input fields, and an output contract. Add future endpoints only after the same evidence is captured. Register the pack in packages/websocket/src/node-registry-setup.ts and add @nodetool-ai/higgsfield-nodes to packages/websocket/package.json. Register { pkg: "@nodetool-ai/higgsfield-nodes", path: "higgsfield-manifest.json" } in PACKAGE_RUNTIME_ASSETS. Copy the manifest into dist, expose it from package exports, and load it through loadPackageAssetJson. Registration is required in development as well as the packaged backend because the loader rejects unregistered assets. P3 — Add the generic provider capability layer Create HiggsfieldProvider in packages/runtime/src/providers/, register it from index.ts, and use the manifest loaders for available image/video models and offline model validation. Implement textToImage, imageToImage, textToVideo, imageToVideo, and referenceToVideo only where manifest evidence establishes the input mapping. Forward model-specific options without dropping them, but reject unrecognized options before submission. Persist the full upstream request_id as provider_request_id at acceptance, before waiting for completion. Use NodeTool’s awaited durable acceptance hook through recordGenerationReceiptAsync so a worker restart after acceptance does not lose the ID. Use terminal error classes that retain both Higgsfield’s error and X-Correlation-ID. Add generated-output ingestion through NodeTool storage during the same run. Retention is not a reason to leave external URLs in persisted workflow state. Implement getGeneration(requestId, options) as a single status lookup using the existing ProviderGenerationLookup interface. For restart recovery or an ID-only lookup, explicitly allow GET https://api.higgsfield.ai/requests/{request_id}/status and POST https://api.higgsfield.ai/requests/{request_id}/cancel, as documented in the status reference and cancellation reference. Validate the upstream UUID and encode it as one path segment. Authenticate with the owning account’s credentials. This is the recovery exception to using returned controller URLs during a live run. Recovery must never submit a replacement generation. Treat a successful cancellation’s empty 202 response as accepted and a 400 after processing starts as a cancellation rejection, not a canceled generation. P4 — Add dynamic catalog features only after the core is sound Implement the Marketing Studio presets client, including cursor pagination, credential forwarding, and a refreshable UI selection. Require it only when enhance_prompt is true. Add an estimate endpoint to generic-node preflight or cost UI. The estimate must use the exact payload that will be submitted, minus NodeTool-only properties, and must never become a static rate table. Add audio, 3D, webhook acceleration, and new model families only with a source-backed schema, typed output mapping, cancellation behavior, and regression fixture. Webhooks remain optional optimization over polling. Create a schema-import or drift-check script only after a documented machine-readable catalog is available. Until then, a failing test should ensure every manifest entry has a reachable source page and a stored schema fixture. 5. Verification plan Layer Required evidence Transport Unit tests for auth composition, request/status parsing, each terminal status, 400 concurrency vs validation handling, 401/403/422 no-retry, 5xx GET retry, ambiguous POST no-retry, cancellation, timeout, and correlation-ID capture. Upload Tests that every returned upload header is sent, credentials are not sent to the presigned host, MIME validation rejects unsupported content, and an uploaded public_url is substituted into the model payload. Factory Tests for scalar, enum, object, array, single media, list media, heterogeneous reference groups, all declared output slots, and models whose output cardinality cannot be represented. Provider Contract tests for manifest model enumeration, each initial generic capability, request-ID lookup, terminal result mapping, and immediate result absorption. Test restart recovery after durable acceptance with all in-memory controller state discarded: recover status and completed outputs using the saved ID and owning credentials, cancel queued work through the ID-based endpoint, preserve running status when cancellation is rejected, and assert that recovery makes no generation POST. Security Add all Higgsfield submit, status, upload, and result hosts to the URL egress inventory. Verify hostile output URLs cannot reach private/local addresses and result downloads retry only after a job is accepted. Packaging Manifest asset is registered in PACKAGE_RUNTIME_ASSETS, copied into dist, exposed from package exports, and loaded through loadPackageAssetJson. Verify registered loading and rejection of an unregistered manifest, plus manifest staging and loading in the bundled backend. Run npm run backend:smoke when adding the manifest asset. Repository gate Run npm run test:affected, npm run typecheck, npm run lint, and npm run dev:nodetool -- harness gate --base origin/main. 6. Risks and decisions needed before coding R1 — Catalog freshness. There is no verified public model-list endpoint. The implementation must not pretend static coverage is the whole Higgsfield catalog. The curated-manifest approach makes coverage explicit and safe, but new models require an update until Higgsfield supplies an official catalog. R2 — Webhook authenticity. The webhook guide specifies delivery and retry behavior but no signing header or verification recipe. A public callback route without verification would expand NodeTool’s attack surface. Poll first. R3 — Timeout ownership. Higgsfield asks clients to choose a model-specific application timeout, but does not publish all expected durations. Store a conservative per-manifest budget and let a workflow-level timeout shorten it. R4 — Exact cost accounting. Estimate responses are account and parameter specific. Use the authenticated estimate when available; do not infer cost from the explorer’s promotional storefront prices. R5 — Model access. A documented model can still return 404, 423, or 503 for a given account. Model visibility in a NodeTool manifest means supported integration, not guaranteed entitlement. 7. Explicit non-goals for the first release Browser-side Higgsfield requests or credentials. Automatic retry of a generation submission. A generic untyped JSON node as the primary user experience. An unauthenticated public webhook endpoint. Static pricing claims based on storefront prices. Shipping unverified audio or 3D model nodes solely because the shared result envelope can represent their artifacts.--- # Marketing recipes: asset production master plan Source: https://docs.nodetool.ai/plans/marketing-recipes/MASTERPLAN.html Marketing recipes: asset production master plan M1. Assignment and boundary Produce the complete asset packages needed to redesign the four marketing recipe pages around NodeTool’s guided flows and editable documents. This plan governs asset production. Page design, React implementation, metadata changes, publishing, and replacing existing samples happen after the asset gate in M12. The instruction that created these plans authorized writing the plans. It did not start production or authorize provider spend. A later execution instruction determines the production authorization and spending limit. Do not treat a hypothetical budget in a plan as an approved budget. The intended execution model is one Sol agent per recipe, with a coordinating agent reviewing shared requirements. Use gpt-5.6-sol when creating the requested execution tasks. Do not spawn those tasks merely because these files exist. ID Recipe plan Primary surface Production responsibility D1 Product ad variants Storyboard, Commercial Vertical ad, hook variants, thumbnails, narrated script, guided-flow demonstration D2 Multilingual video Script Source presenter clip, translated script, voices, subtitles, dubbed versions, guided-flow demonstration D3 Product catalogue assets Storyboard, Commercial Shared product reference, cutout, scene treatments, motion clip, print master, listing copy, guided-flow demonstration D4 Storyboard to Trailer Storyboard Character and location references, screenplay, voiced lines, shots, score, trailer, guided-flow demonstration Keep the existing recipe slugs as identifiers. The display names above describe the proposed page direction and do not authorize route changes. M2. Read before executing Read the root AGENTS.md, docs/DEVELOPMENT_STANDARDS.md, docs/WRITING_STYLE.md, docs/BRAND.md, this master plan, and the assigned recipe plan. Read directory overlays before changing source code in that directory. For production, read .claude/skills/storyboard-core/SKILL.md, including its Tool contract section. Each recipe names its additional skills. Use the callable tool schemas and checked-out implementation to resolve contract drift. Do not copy an obsolete model ID or a stale tool signature out of prose. Use NodeTool to create the work being demonstrated. A generated image from a different service may be an explicitly disclosed source reference, but cannot serve as proof that a NodeTool flow generated that image. Source preparation and the recipe’s resulting work must be distinguishable in the manifest. Do not author, import, or display node graphs to produce these examples. Use guided setup, direct storyboard/script/timeline/sketch operations, and agent media tools. Encoding, probing files, making contact sheets, and packaging deliverables with ordinary local tools are permitted support operations. M3. Execution order and ownership ID Work Dependencies Completion evidence M3.1 Coordinator creates a run ID, checks available NodeTool tools and connected providers, and records the execution authorization None Run directory and authorization record M3.2 D3 creates and reviews the shared product reference package M3.1 Product reference, descriptor, entity export, provenance, checksums M3.3 D4 validates the Storyboard capture sequence while producing its own assets M3.1 Real setup screenshots and capture procedure M3.4 D2 produces its independent source and language assets M3.1 Its recipe package M3.5 D1 produces ads using D3’s accepted product reference M3.2 Its recipe package M3.6 D3 completes its product assets M3.2 Its recipe package M3.7 Coordinator reviews the four handoffs and resolves missing evidence M3.3–M3.6 Acceptance or a specific list of remaining assets M3.8 Marketing page implementation is commissioned separately M3.7 and user direction New page-build task consuming these assets D3 owns the shared product reference. D1 consumes it without changing it. D2 and D4 have no dependency on the product and can proceed independently. Each agent owns its recipe directory, project, documents, and output files. Do not edit another recipe’s files, shared page data, or the existing recipe generator. If an agent needs a change to the common contract, send it to the coordinator. Only one agent may drive a shared browser tab or foreground app at a time. Separate NodeTool projects do not isolate a shared tab’s focus. Coordinate a capture window, or use separate browser sessions. Do not race UI actions. Headless operations must carry explicit document IDs and stay inside the assigned project. M4. Recheck the flows before filming The source inspection that informed these plans found the following paths. Recheck them on the execution revision and record differences rather than silently editing the story to fit. ID Flow Real screens Landing F1 Storyboard Idea, Story selection, Story review, Entities, Look Board with stills queued F2 Script Idea, Format selection, Format review, Voices Script editor with speech queued F3 Video Idea, Beats selection, Beats review, Look Timeline with text-generated video and optional speech F4 Image Idea, Brief selection, Brief review, Look Text-generated variations and sketch editor Sources: web/src/components/setup/storyboard/useStoryboardSetupFlow.ts, web/src/components/setup/script/useScriptSetupFlow.ts, web/src/components/setup/video/useVideoSetupFlow.ts, web/src/components/setup/image/useImageSetupFlow.ts, and web/src/components/setup/SetupFlow.tsx. Story selection and review share one visible stepper label. The Storyboard progress bar is Idea / Story / Entities / Look. Script uses Idea / Format / Voices. Capture both selection and review screens even though they share a progress step. ID Boundary to verify Consequence for the assets F5 Storyboard setup generates stills, not a complete movie Show Render clips and Assemble timeline as editor actions after setup F6 Imported Script words are preserved Translation must happen separately before the translated import. The Voices language selector is not a translation command F7 Generic Image generation uses text-to-image bindings Do not advertise preservation of a real SKU through that path F8 Image’s Image to video handoff opens a workflow canvas Do not use it in these recipes F9 Generic Video generation uses text-to-video bindings Product reference-conditioned production belongs on Storyboard here F10 Video setup’s Music control was disabled Generate or add music through a supported finishing surface. Do not film a working toggle unless it actually works on the execution revision F11 Studio and workspace have different handoffs Pick one edition for each demonstration and record it. Do not splice buttons from different editions into one supposed session F12 Storyboard’s composer carries entity IDs but not arbitrary reference attachments into setup Add new product/character photos in Entities, or pre-create entities and select them. Do not film a dropped photo silently becoming an entity These are product constraints to respect, not authorization to fix the app. A required unsupported operation gets a blocker entry with the exact failed surface, input, and missing capability. Finish independent assets and report the remaining gap. Never replace it with a graph or a fabricated screen. M5. Creative defaults and shared product Use the briefs in the recipe plans without another creative interview unless an execution instruction supplies a different subject. These are original demonstration projects, not client work or endorsements. D3 produces an unbranded Olive Travel Cup for both D3 and D1: a matte muted-olive cylindrical travel cup, slightly tapered toward the base, with a charcoal flat lid and a small rectangular drinking opening. No handle, lettering, logo, measurement marks, badges, or decorations. The reference image is the visual authority. This is a fictional demonstration object. Do not invent capacity, insulation performance, materials certification, pricing, or availability. Keep the product shape readable and consistent. D1 uses a warm kitchen/daylight treatment. D3 uses clean commercial light with one seasonal variation. Their grades may differ, but the cup and lid cannot become different objects. D2 uses a synthetic adult presenter and a short English explanation translated into Spanish. D4 uses the original fictional film The Next Tide. Each plan gives its exact source text and visual direction. M6. Output directories and durable sources Create a new run directory under marketing/recipe-assets/<run-id>/. A run ID is a short unique name chosen at execution, such as a date plus descriptive suffix. Keep all raw assets out of marketing/public/ until the page-build phase chooses the delivery assets. marketing/recipe-assets/<run-id>/ run.json shared/product/ viral-video-ad-engine/ multilingual-video-dubber/ ecommerce-sku-visual-factory/ storyboard-to-trailer/ Each recipe directory uses this structure: <slug>/ sources/ original imports and synthetic source media documents/ exported entities, scripts, boards, timelines, shot map generation/ original model outputs and generation ledger selects/ chosen full-quality takes, with stable names masters/ full-quality finished videos, audio and images web/ delivery encodes, posters, still derivatives captures/raw/ original screenshots and uncut screen recordings captures/steps/ cleaned crops and compressed screenshot derivatives captures/walkthrough/ edited demonstration and poster evidence/ probes, comparison sheets, validation output asset-manifest.json capture-log.json handoff.md qa.md Existing files under marketing/public/recipes/samples/ belong to the old process. They can be inspected for context but must not be overwritten, relabeled as a new run, or cited as proof of the new flow. Store production documents in a dedicated NodeTool project named Marketing recipes / <slug> / <run-id>. Export the documents as well as recording server IDs. A server ID or expiring signed URL alone is not a durable handoff. Download the underlying bytes and checksum them. Keep secrets and signed query parameters out of committed manifests. Do not automatically commit raw binaries, push, upload to a CDN, or deploy. Keep the completed directory available locally and report its absolute path. If files exceed practical repository storage limits, the coordinator chooses a durable artifact destination before removing any local copy. M7. Model selection, authorization, and iteration Before generating, discover callable NodeTool tools and actual connected models. Record the exact provider/model IDs and needed capabilities. Product and character stills must use an image model that accepts the reference images, not merely an unrelated model whose name sounds suitable. Prepare all authorized non-spending work first: source inventory, text, shot plan, project organization, model lookup, and a list of generation calls. Read the actual UI estimate where available and record it as an estimate. If no estimator covers the operation, report quantities and mark cost unknown. Read actual spend after each generation phase where the environment supports it. Honor the execution instruction’s existing authorization. If it explicitly approves the complete production and a spending envelope, proceed within it without repeatedly asking for the same permission. If authorization stops at a phase boundary, present the concrete references or stills for the next decision. The storyboard skills require review between direction, stills, and clips unless the user’s instruction overrides that wait. Quote and link the applicable skill if it causes a pause. ID Review point Check before continuing G1 Source and plan Subject, references, entity assignments, exact words, shot order, chosen models G2 Stills Identity, geometry, scene continuity, framing, readable UI captures G3 Speech and clips Pronunciation, semantic accuracy, motion, lip sync where applicable, measured duration G4 Final exports Editable project matches the exported bytes, audio mix, delivery encodes, manifest completeness Plan one initial generation per required source/still/clip and reserve at most two targeted retries for an unsatisfactory asset. This is an iteration ceiling, not spending authorization. Stop earlier if the authorized budget is reached. Each retry needs a stated defect and a changed instruction. Do not rerender a whole board to fix one shot or silently change a model midway through the set. Keep selected takes explicit. Read current tool support for take selection. Where only the takes gallery can select an older version, select it through that UI and verify the resulting state. Do not claim a previous take is active because its asset ID was copied into notes. On a resumed run, read the existing manifest and NodeTool documents first. Reconcile pending job IDs with their actual state before submitting another generation. A timeout is not evidence that a provider job failed. Reuse accepted files by hash, preserve their source IDs, and generate only missing or rejected assets. Save the ledger after each completed batch so another Sol session can resume without repeating paid work. M8. Actual production and captured experience Create a real project by following the guided flow for the core demonstrated work. Capture selection screens, reviewed text, references, and the generation handoff during that process. Agent tools can refine the real documents and do finishing work. Record those interventions as agent/editor work. One content set must connect the whole demonstration: the same brief, product or cast, shot order, script, selected stills, rendered clips, and final timeline. A beautiful output from a different board is not an acceptable final frame for the walkthrough. Generation waits may be cut from the edited recording. Record the cut in the capture log and mark the sequence time-compressed. Do not add a stopwatch, elapsed-time claim, or an impression of instant rendering. The demo system in demo/README.md can render real document components from recorded state. Use it only when a clean replay is needed and the required surface is supported. A replay must use exported documents and real generated media from this run, be labeled ui-replay, and be separate from the raw recording. Do not invent tool success, cost totals, progress, UI buttons, or output provenance. Extending the demo system or changing product code is a separate scoped task unless explicitly authorized. Prefer existing dedicated capture tools. If computer use is needed, use the available approved computer-use surface. Do not work around a blocked browser extension by injecting code or controlling the user’s browser through another mechanism. A capture blocker does not stop independent headless asset production. M9. Capture and delivery specifications ID Asset Specification E1 Raw screenshots PNG, full application content viewport, target 1600×1000 or larger, browser zoom 100%. Record actual viewport and pixel ratio E2 Step screenshots Lossless source crop plus WebP derivatives at 1600 and 960 pixels wide when the source supports them. Preserve complete relevant controls and step labels E3 Walkthrough 35–60 seconds, 16:9, 1920×1080 where capture resolution supports it, 30 fps. Show the actual setup stages and a short finishing sequence E4 Walkthrough audio Silent by default. Supply accurate WebVTT captions for step explanations and a clean poster. Any added narration must also exist as a script and audio source E5 Finished videos High-quality master, H.264 MP4 with yuv420p and faststart, VP9 WebM, WebP poster. Preserve the recipe’s native aspect ratio E6 Audio Original provider output plus WAV at 48 kHz for editing where conversion is needed. Keep voice and music stems separate E7 Finished stills Original resolution PNG or lossless original, WebP at full delivery size and 960 px wide, with alpha retained where required E8 Recipe card 1600×900 WebP using the real finished work. For portrait ads, use an intentional two/three-frame composition or contain the frame. Do not crop away the product or subtitle E9 Social preview 1200×630 image from the accepted assets, with room for page text to be added later. Do not bake a speculative page design into it E10 Comparison sheet Legible labeled before/after or selected-take contact sheet. Labels are ordinary layout text, not generated pixels inside the footage Do not upscale low-resolution screenshots to meet a nominal size. Record the actual size and explain any exception. Preserve original generation dimensions, native frame rate, and duration in the manifest even if a delivery encode changes them. For speech-led video, target approximately -16 LUFS integrated and true peak at or below -1 dBTP in the web delivery. Measure rather than assert. Keep music below intelligible speech. For silent loops, remove the audio track and mark hasAudio: false. Do not autoplay a sound-bearing file in later page work. Optimize encodes through a visual comparison against the master. Suggested review targets are below 8 MB for a short portrait ad and below 15 MB for a 40-second trailer or walkthrough. They are delivery targets, not reasons to accept visibly damaged gradients, unreadable UI, or distorted speech. Report an exception instead of silently degrading the asset. Title cards, offer text, language labels, and captions must remain editable timeline or composition text. Use a bundled font supported by the actual timeline renderer. Do not ask a video model to draw exact marketing text. M9.1. Local encoding reference Use an installed trusted ffmpeg/ffprobe or the existing project media tooling. Verify the executable before planning a conversion. Do not install a new binary merely to follow an example command. The following commands illustrate the delivery settings for an already finished master, with paths relative to a recipe directory. Use a versioned destination when an accepted file already exists. Probe each master and delivery independently: ffprobe -v error -show_format -show_streams -of json masters/final.mp4 Encode H.264 while preserving the master’s dimensions, frame rate and audio presence: ffmpeg -i masters/final.mp4 -map 0:v:0 -map '0:a:0?' \ -c:v libx264 -preset slow -crf 20 -pix_fmt yuv420p \ -c:a aac -b:a 160k -movflags +faststart web/final.mp4 Encode a WebM sibling: ffmpeg -i masters/final.mp4 -map 0:v:0 -map '0:a:0?' \ -c:v libvpx-vp9 -crf 30 -b:v 0 -row-mt 1 \ -c:a libopus -b:a 128k web/final.webm For a silent delivery, omit the audio map and use -an. These commands do not mix or normalize speech. Finish and measure the audio in the master first. Adjust compression after visual inspection and retain the exact successful command in the derivative’s provenance. Avoid overwriting a source through an input/output path mistake. Extract the poster at a deliberately selected, verified moment rather than automatically using a black first frame. Record its source video and timestamp. A thumbnail crop or contact sheet is a deterministic derivative and must retain its source mapping. If a transform stretches a portrait image into a wide card, reject it and use containment or a composed layout instead. M10. Manifest and handoff contract The coordinator owns run.json. It records run ID, repository commit, app build/edition, NodeTool project host, assigned agents, execution authorization, spending envelope or absence of one, and the shared product revision. Do not put keys or personal account details in it. Each asset-manifest.json contains schemaVersion: 1, runId, recipeSlug, status, documents, assets, generationCalls, blockedItems, and supportedClaims. ID Record Required fields M10.1 Document Stable role, document type, server ID, project ID, export path, SHA-256, linked document IDs M10.2 Asset Stable ID, role, relative path, SHA-256, MIME type, width/height where applicable, duration/fps where applicable, bytes, hasAudio, alpha status, source IDs, producing document and shot/line IDs M10.3 Generation call Call/job ID, tool name, provider/model, input asset IDs, prompt file, generation settings, output IDs, actual cost/currency or null, attempt number, failure reason if any M10.4 Provenance user-supplied, generated-in-nodetool, generated-source, derived, live-ui-capture, or ui-replay. Derived assets list their parents and transformation M10.5 Review Criterion, reviewer, result, evidence path, selected take ID, whether user acceptance is recorded or still pending M10.6 Blocker Required asset ID, attempted operation, observed limitation/error, available partial result, exact next action needed M10.7 Supported claim Proposed factual wording, evidence IDs, limitation, whether safe to use in page copy Use null for unknown facts and explain them. Do not substitute a zero cost for unavailable cost data. Do not mark a reference synthetic merely because it was found in an old sample folder. Trace its actual origin or treat the origin as unknown. capture-log.json records each capture ID, recipe action ID, visible screen label, project/document IDs, app revision, viewport, capture mode, raw path, edited path, chronological order, and any time compression or redaction. Preserve the raw original separately. handoff.md must give the absolute package path, the accepted hero candidate, all playable previews, document links/IDs, asset-role map, screenshot order, exact captions/alt text suggestions, generation totals, known limitations, and supported page claims. Explain how to reopen the project and which selected takes the timeline uses. Keep the next page agent from having to infer these from filenames. qa.md records results against M11 and the recipe-specific checks. Do not fill it with unchecked boilerplate and call the package ready. M11. Verification ID Check Evidence to retain V1 Every required deliverable exists and decodes File inventory, MIME/probe output, dimensions, durations, hashes V2 The final picture belongs to the demonstrated project Document exports, shot-to-asset map, selected-take map, timeline provenance V3 The UI sequence follows the actual flow Raw captures of every required screen and the capture log V4 The whole film plays correctly Watch every second at normal speed, inspect transitions, first/last frames, and listen with headphones V5 Still detail survives inspection Inspect product geometry, faces, hands, reference continuity, alpha edges, and text at full size V6 Spoken text and subtitles agree Final transcripts, reviewed script, cue timing checks, pronunciation/meaning notes V7 Editable documents remain usable Reopen from the recorded IDs or exported copies, verify links, run the available timeline validation, check missing media V8 Page-use derivatives retain quality Side-by-side master/encode inspection, no distorted aspect ratios, no missing audio metadata V9 Claims are supported Each proposed claim maps to actual assets and observed behavior. No speed, identity, cost, or language claims inferred from a still V10 No graph appears in the user path Inspect the full walkthrough and every screenshot, including finishing actions Probe files with the available media inspection tools or ffprobe. Download or open the actual bytes when checking a URI. A job marked complete is not evidence that a file is playable. No application-code changes are requested in this phase. If an explicitly authorized capture change modifies code, read the relevant overlays and run the repository’s four mandatory checks: npm run test:affected, npm run typecheck, npm run lint, and npm run dev:nodetool -- harness gate --base origin/main. An asset-only export does not require running all application tests. M12. Asset gate and later page work A recipe may be marked ready-for-review when all required files exist, pass technical checks, and have complete provenance. Mark accepted only when the coordinator/user acceptance required by the execution instruction has been recorded. Use partial for packages missing a required production or capture result. An optional variant may be omitted only when the recipe plan says it is optional. Do not start page implementation until the user has accepted the asset packages and any reduced outcome caused by a capability gap. A folder of generated clips without guided-flow captures is not a completed asset package. After acceptance, a separate page task will consume the manifests and build the recipe detail pages, recipe index, and homepage recipe cards. It will address the old workflow-derived data contract in marketing/src/data/recipes.ts and marketing/scripts/generate-recipes.mjs, preserve slugs, and use the accepted captions and supported claims. This plan does not authorize those changes now. M13. Dispatch template Use this as the message to each separately requested Sol task. Fill the variables before sending it. Execute asset production for <recipe name> from the repository root. Read docs/plans/marketing-recipes/MASTERPLAN.md and docs/plans/marketing-recipes/<recipe slug>.md completely. Produce the required assets, real editable NodeTool documents, guided-flow captures, manifests, verification evidence, and handoff. Do not build or publish marketing pages. Use guided flows and direct document/agent operations. Do not create or show node graphs. Own only your recipe directory and project. Run ID: <run ID> Output root: <absolute run directory> Shared product package: <accepted path/revision, or not applicable> Execution authorization and spend limit: <actual user authorization> Capture window/session: <coordinator assignment> Resolve routine creative choices using the plan. Reuse accepted assets and finish independent work when a capability is missing. Report a specific blocker instead of inventing a screen, a successful generation, or a cost. Stop at the asset handoff and report its path, previews, checks, and any unaccepted or missing required deliverables.--- # D3. Product catalogue assets: asset production plan Source: https://docs.nodetool.ai/plans/marketing-recipes/ecommerce-sku-visual-factory.html D3. Product catalogue assets: asset production plan SKU0. Assignment Read MASTERPLAN.md first. Produce the asset package for ecommerce-sku-visual-factory when dispatched. Do not edit the marketing pages or run the old workflow-based sample renderer. You own the shared product reference used by this recipe and D1 Product ad variants. Release that reference early, after the required source review, so D1 can proceed independently. Continue your own production after the handoff. Read .claude/skills/product-commercial/SKILL.md and .claude/skills/launch-kit/SKILL.md after storyboard-core. Use a stills-first commercial board, animate only the selected motion shot, and keep the product the same across the entire set. The required result is one fictional SKU shown as an original source, transparent cutout, clean studio scene, seasonal scene, motion-ready still and clip, print-size image, and listing copy. The guided demonstration centers on Storyboard / Commercial / Entities / Look. Cutout, upscale and copy production are explicitly identified as preparation or finishing outside that guide. Owner directories: <run-root>/ecommerce-sku-visual-factory/ and <run-root>/shared/product/. SKU1. Capability check before production ID Required capability Proof to collect SKU-P1 Generate an original unbranded product reference Connected image model and a supported native image size SKU-P2 Create an entity from the accepted asset Callable entity operation and successful returned ID SKU-P3 Generate stills conditioned on the entity image Reference-capable image model, actual reference passed to generation SKU-P4 Produce a true transparent cutout Supported background-removal/editor/media operation that returns an alpha channel SKU-P5 Animate a selected still Image-to-video path available on the storyboard SKU-P6 Upscale the accepted image Supported image-editor/agent media operation and actual output dimensions SKU-P7 Capture the guided flow and export board assets Actual screens and Download ZIP or equivalent verified export Do not represent a white-background image as a transparent PNG. Do not create a node graph when a direct operation is missing. Record the unavailable item, produce unaffected work, and leave the package partial until the required result is available or the user accepts a reduced scope. SKU2. Generate and release the shared product reference The default is a synthetic, unbranded demonstration object. If the execution instruction supplies a real product, stop using the default and make that supplied photo authoritative. Preserve its actual labels, geometry, and source attribution. Reference-generation prompt: Studio reference photograph of one unbranded travel cup. Matte muted-olive cylindrical body, subtly tapering toward a narrower base. A flat charcoal lid with a single small rectangular drinking opening near the front rim. No handle. No logo, lettering, measurement marks, badges, patterns, decorative ridges, or extra accessories. Front three-quarter view at product height, the whole cup including lid and base visible, occupying roughly two thirds of the frame height. Plain very light gray background, soft even studio lighting, small natural contact shadow, accurate straight geometry, sharp product edges. Photographic realism. One object only. ID Action Required result SKU-R1 Discover a connected model and generate a reference at its supported size, preferably at least 2048 px on the long edge Original provider file and generation record SKU-R2 Inspect the object at full size One lid, one opening, clean body, consistent taper, no illegible pseudo-brand marks SKU-R3 If needed, perform a targeted retry naming the defect Accepted source plus rejected attempts retained in generation history SKU-R4 Create a product entity named Olive Travel Cup from the accepted image Entity ID and exported metadata SKU-R5 Record the descriptor from the actual accepted image Geometry and finish only. Do not add inferred specifications SKU-R6 Export the source package to the shared directory and notify the coordinator/D1 Stable paths, checksums, entity ID/export, source status and review record Descriptor starting point, to be corrected to the actual accepted reference: Unbranded travel cup with a matte muted-olive body that narrows slightly toward the base, and a flat charcoal lid with one small rectangular drinking opening. No handle, lettering or decorations. Preserve the body proportions, lid shape, opening orientation and matte finish from the reference image. Required shared files: ID Path under shared/product/ Content SKU-SH1 product-reference.<original-extension> Accepted full-quality original SKU-SH2 product-reference.webp Viewable derivative, not a replacement for the original SKU-SH3 product-entity.json Entity metadata and source asset ID SKU-SH4 product-descriptor.txt Canonical accepted descriptor SKU-SH5 source-manifest.json Hash, origin, provider/model, prompt path, review and version SKU-SH6 product-cutout.png Add after its alpha inspection. D1 need not wait for this optional input A later change to the accepted reference requires a new source revision and notice to D1. Never replace the bytes under the accepted filename while another agent is using them. SKU3. Source cutout Use the accepted source file as input to a supported background-removal operation. Preserve the source object and natural edge detail. The output must have a meaningful alpha channel, not a baked-in checkerboard or a white matte. Inspect the cutout against light gray, charcoal, and a saturated diagnostic background. Look for missing lid corners, erased dark details, olive halos, jagged edges and residual background inside the drinking opening. Preserve the original source separately. A contact shadow may be delivered separately, but cannot obscure whether the object cutout has real transparency. Save a diagnostic comparison sheet and machine-readable alpha inspection: image mode, dimensions, count or proportion of fully transparent pixels, partially transparent edge pixels, and opaque interior pixels. Check a known interior area as well as corners. An image with every pixel transparent must fail. This is source preparation. The later storyboard guide should not show a fake background-removal step. SKU4. Commercial board direction Use one square board with three still treatments. All three shots use the same accepted product entity. No character, generated brand label, or spoken script is needed for this recipe. Board style: Photographic catalogue imagery of Olive Travel Cup. Faithful product geometry, matte muted-olive body and charcoal lid, clear edges, controlled commercial lighting, restrained neutral surroundings, no additional branding or text. The product is the subject in every frame. No hands or people. Treat the source image as the authority for the cup, lid and finish. Scene-specific lighting is specified in each shot. Shot ID Purpose Action for the still Motion instruction SKU-S1 Clean studio scene and print master source Olive Travel Cup centered on a pale limestone plinth against a warm light-gray sweep. Front three-quarter view matching the reference. Entire lid and base visible. Broad soft key from upper left, neutral fill, precise contact shadow. No props. Stills-only. Do not buy a video for this shot SKU-S2 Seasonal campaign scene Olive Travel Cup on the same pale limestone plinth and from the same camera height. Low winter sunlight from the left makes a longer soft shadow. Cool pale background, one blurred bare branch far behind the cup. Product color, lid and geometry stay consistent with the reference. No snow on the product and no festive logos. Stills-only. Do not buy a video for this shot SKU-S3 Motion-ready product hero Olive Travel Cup on an uncluttered neutral studio surface. Three-quarter front view, room around the full silhouette, soft key with readable lid detail, no other props. Match the accepted source proportions. A slow 30-degree camera arc around the stationary cup over approximately 6 seconds. The cup, lid and opening remain rigid. No full spin, opening lid, liquid, floating or morphing Call the motion asset a short product motion clip in handoff copy. Do not claim a full 360-degree turntable or accurate unseen backside reconstruction. This preserves a simple demonstration with a checkable reference. The final still targets are 2048×2048 or the highest practical native square resolution supported by the chosen model. The print derivative target is 4096×4096 from SKU-S1. Label it as upscaled, not native capture or proof of print color accuracy. SKU5. Guided-flow steps and screenshots Capture ID Actual surface Action and content to capture SKU-C0 Source preparation Original and accepted alpha cutout, clearly labeled as preparation SKU-C1 Storyboard → Idea Enter the brief below, click Continue SKU-C2 Story choices Select Commercial and 3 shots, show Generate screenplay SKU-C3 Story review Three shots matching SKU4. Edit away any invented product claims or dialogue, then click Set up entities SKU-C4 Entities Select Olive Travel Cup. Show its reference and explicit assignment to the three shots SKU-C5 Look Choose 1:1, controlled photographic styling and a reference-capable still model. Show the estimate and Generate your storyboard SKU-C6 Board Three finished stills. Open one and compare its product detail to the reference SKU-C7 Shot inspector Select SKU-S3 and generate its clip. Do not run Render clips for all three shots SKU-C8 Editor/agent finishing Real upscaling operation on SKU-S1 and the measured 4096 px result SKU-C9 Agent/text document Listing draft using supplied visual facts and explicitly unknown specifications SKU-C10 Board/export Download ZIP and show the organized output set Brief for SKU-C1: Create three square catalogue images of Olive Travel Cup: a clean studio hero on a pale stone plinth, a winter-light version in the same setting, and an uncluttered three-quarter product view for a short motion clip. Use the product reference I will select in Entities. Keep its geometry, olive body and charcoal lid consistent. No people, logos, lettering, invented features, or dialogue. Do not use the Image guide’s Product shot card as proof of source-preserving edits. Its text-generation path is a different capability. Do not capture the Image to video action, which leads to a node canvas. SKU6. Production sequence ID Action Review before continuing SKU-G1 Produce and release the source package Reference passes SKU2, hash/version frozen SKU-G2 Produce the real alpha cutout Composite against three backgrounds and inspect edges SKU-G3 Complete the actual Storyboard guide and capture its stages Three shots, correct entity and no unnecessary dialogue SKU-G4 Render the three stills Side-by-side comparison against the original. Reject duplicate lids, changed opening, new handle, stretched proportions or materially shifted color SKU-G5 Render only SKU-S3’s motion Inspect the whole clip. A short camera arc must not reveal inconsistent new geometry SKU-G6 Upscale accepted SKU-S1 Verify actual dimensions and compare edges/texture to the original selected still. Do not upscale an unaccepted take SKU-G7 Write listing copy in a saved text document Use the fact sheet below. Do not infer technical specifications from appearance SKU-G8 Export and compose the output overview Each tile points to the actual asset ID and file SKU-G9 Record and edit the demonstration Guide remains central. Preparation and finishing receive their own captions Retry only the defective asset within the master plan’s iteration rules. A newly selected still invalidates any clip or upscale derived from the older take. Update the dependency map and regenerate only those dependent outputs when needed. SKU7. Listing copy and source facts Create documents/product-facts.md and documents/listing-copy.md. The latter contains a short title, a one-paragraph description, three visual feature bullets, and a separate list of facts that would need the merchant’s input. Permitted facts: Fact ID Fact SKU-F1 Product is an unbranded fictional travel cup for this demonstration SKU-F2 Visible body color is muted olive SKU-F3 Visible lid is charcoal and has a rectangular drinking opening SKU-F4 The photographed design has no handle or visible lettering Unknown facts include capacity, exact materials, insulation performance, leak resistance, dishwasher/microwave compatibility, dimensions, weight, certifications, price, warranty and availability. Put those in a merchant-input section. Do not turn plausible assumptions into product benefits. Suggested title: Olive travel cup with charcoal lid. Suggested description: A muted-olive cup with a clean tapered shape and a charcoal lid. The unbranded design keeps the focus on its simple silhouette and contrasting finish. The agent may improve the phrasing while preserving the factual limits. This copy is an output example, not a live listing. SKU8. Required deliverables Asset ID Path or family Requirements SKU-A1 Shared files SKU-SH1–SKU-SH6 Accepted source package and later alpha cutout SKU-A2 sources/product-original.<ext>, product-cutout.png Recipe-local source copy or manifest link to immutable shared bytes SKU-A3 documents/production-board.json, product-entity.json Real document exports and exact source reference SKU-A4 selects/studio-hero.png, winter-scene.png, motion-hero.png Three selected stills at original resolution SKU-A5 masters/product-motion.mp4 Accepted 5–8-second square motion clip, native source retained SKU-A6 web/product-motion.mp4, .webm, product-motion-poster.webp Silent web delivery with no audio track SKU-A7 masters/studio-hero-print-4096.png Actual 4096×4096 upscaled output, with source ID and upscale settings SKU-A8 documents/product-facts.md, listing-copy.md Reviewed copy and explicit unknowns SKU-A9 web/studio-hero.webp, winter-scene.webp, motion-hero.webp and 960 px derivatives Page-ready stills SKU-A10 web/source-to-set.webp Labeled source, cutout, studio, seasonal, motion still, and print detail SKU-A11 web/seasonal-comparison.webp Accepted studio and seasonal images with consistent presentation SKU-A12 web/recipe-card.webp, social-preview.webp Real-output compositions at common master-plan sizes SKU-A13 captures/steps/sku-c0 through sku-c10 Full capture sequence, original PNG and WebP derivatives SKU-A14 captures/walkthrough/catalogue-guided-flow.mp4, .webm, .vtt, poster.webp 40–60-second demonstration SKU-A15 evidence/alpha-report.json, alpha-backgrounds.webp, product-consistency.webp, print-detail.webp Checks that can be independently inspected SKU-A16 masters/catalogue-assets.zip Actual production outputs and listing text, plus a readable contents file Include the common manifest, capture log, handoff and QA files. Record that the product and scenes are synthetic. Avoid a claim that source pixels stayed unchanged during generative scene treatment. SKU9. Walkthrough edit Segment Approximate time Picture Caption SKU-W1 0–5 s Whole output set One product reference, several treatments. SKU-W2 5–10 s Source photo and cutout Prepare the product reference. SKU-W3 10–16 s Idea and Commercial selection Describe the images you need. SKU-W4 16–23 s Actual three-shot review Review each treatment before rendering. SKU-W5 23–30 s Entities stage Choose the same product for every shot. SKU-W6 30–38 s Look and finished still board Set the look. Compare the results. SKU-W7 38–46 s One selected shot animated Add motion to one selected image. SKU-W8 46–55 s Upscale, copy document, export set Finish the print image and listing copy. The short opening source sequence and final finishing sequence must be identified as work outside setup. Do not make a fake five-click catalogue wizard by removing required screens from the record. SKU10. Acceptance and handoff Check ID Pass condition SKU-Q1 Shared reference is accepted, versioned, and usable by D1 without new image generation SKU-Q2 Cutout has real alpha and passes transparent-corner, opaque-interior and edge-background checks SKU-Q3 Selected stills preserve recognizable product geometry and contain no invented labels, logos or features SKU-Q4 Seasonal change affects the scene and light without an unexplained change of product SKU-Q5 Motion clip stays coherent over its full duration and has no unexpected audio SKU-Q6 Print derivative measures 4096×4096, is labeled upscaled, and has no new artifacts that make it worse than the source SKU-Q7 Listing copy contains only supplied/visible facts and leaves unknown specifications explicit SKU-Q8 Capture sequence shows the real Entities stage and keeps cutout/upscale/copy outside the guided steps SKU-Q9 Every SKU-A item exists and the common M11 checks pass In handoff.md, recommend studio-hero as the initial page hero and source-to-set as the main proof image unless actual inspection favors another. Provide exact captions for the alpha, seasonal, motion and print comparisons. Document any remaining limitation before a page agent uses the assets. SKU11. Ready-to-dispatch task Produce D3 Product catalogue assets using the master plan and this recipe plan. First generate and review the shared unbranded Olive Travel Cup reference, create its entity, and release its immutable source package to D1. Then create the cutout, three commercial storyboard stills, one motion clip, a 4096 px print derivative, factual listing copy, real guided-flow captures, web deliveries and evidence. Use the actual Storyboard Commercial flow. Identify preparation and finishing outside it. Do not create node graphs, invent SKU specifications, change the shared reference silently, or build marketing pages. Honor the coordinator's execution authorization and finish with the asset handoff.--- # D2. Multilingual video: asset production plan Source: https://docs.nodetool.ai/plans/marketing-recipes/multilingual-video-dubber.html D2. Multilingual video: asset production plan DU0. Assignment Read MASTERPLAN.md first. When dispatched, produce the asset package for multilingual-video-dubber. Do not change the marketing page, product UI, or old recipe generator. Produce an English presenter source clip, a reviewed Spanish translation and back-translation, editable source and target scripts, voiced Spanish lines, timed subtitles, a Spanish voiceover cut, and a lip-synced Spanish cut. The comparison must use the same source presenter footage. Capture the real Script guided flow for the translated words and identify transcription, translation, and lip sync as separate work. Read .claude/skills/script-video/SKILL.md after storyboard-core. This recipe works on existing picture. Do not derive a new storyboard from the translated script or generate replacement scenes and call that dubbing. Source preparation may use a small storyboard to create the synthetic demonstration footage, but that is recorded as preparation and is not the main recipe walkthrough. Owner directory: <run-root>/multilingual-video-dubber/. This task has no dependency on D1 or D3. DU1. Validate the complete route early Lip sync is a required output for the existing recipe promise. It is not a verified step of Script setup. Test the available route before spending on a long source or final voice set. ID Capability Required confirmation DU-P1 Presenter source A supplied, reusable clip or a connected way to generate an original synthetic source DU-P2 Transcription Actual audio extraction/transcription operation and returned transcript DU-P3 Script import TXT or SRT/VTT import preserves text and speaker/cue structure on the execution revision DU-P4 Spanish speech Actual Spanish-capable voice and readable output from a short audition DU-P5 Lip sync A callable NodeTool agent/media/editor operation accepts the source video and target audio or supported equivalent inputs DU-P6 Audio/timeline finishing Supported import, mute/replace source speech, align new audio, add captions, export video DU-P7 Guided capture Real Script screens and the correct edition’s finishing controls are available Discover the actual tool/model schema for lip sync. Do not assume find_model has a lip_sync capability or invent an endpoint. Query the supported model registry/tool descriptions, then inspect the relevant primary provider documentation if needed. Record the selected route and exact input contract. Run a short pilot through that route with authorized sample media. Compare mouth closure on plosives, sentence starts and stops, jaw stability and face identity. Reuse an accepted pilot segment in the final source if possible. A request ID or completed job without inspected output does not pass this check. If no direct supported route exists, complete the scripts, target audio, subtitles and voiceover cut. Record the lip-sync output as blocked and leave the package partial. Do not build a graph, hide the mouth, substitute unrelated native-audio video, or silently rename the voiceover version as lip-synced. DU2. Source and visual direction Prefer a source clip explicitly provided for this task. If none is provided, create the following synthetic demonstration. Keep its origin disclosed in the manifest and comparison caption. Do not use a real person’s likeness or clone a voice without explicit authorization. Default presenter: an original adult in their thirties with short dark curly hair, a plain muted-blue crew-neck shirt, and a calm conversational delivery. Chest-up, centered, facing the camera, eyes near the top third, mouth unobscured. Soft neutral studio background, diffuse frontal light, no logos, jewelry, microphone crossing the mouth, hands near the face, or moving background. Reference prompt: Photographic casting reference of an original adult presenter in their thirties, short dark curly hair, plain muted-blue crew-neck shirt, chest-up, looking toward the camera with a relaxed neutral expression. Soft frontal studio lighting and an unobtrusive warm-gray background. Face and mouth fully visible. No microphone, hands near the face, text, logos, or resemblance to a named person. Natural skin detail, straight-on composition, ordinary speaking posture rather than a fashion pose. If generated, create an entity named Demo Presenter from the selected reference. The face/hair descriptor comes from that reference. Keep wardrobe and lighting fixed throughout source production. The entity is a source-production asset, not a feature of the Script guide. Source motion: locked camera, small natural head movements, occasional blink, shoulders stable, hands outside the frame. Preserve the full face and mouth. Avoid dramatic gestures that make a dubbing comparison harder to interpret. Target source length is approximately 24 seconds, 1920×1080, 25 or 30 fps according to the actual source. Use the provider’s real supported lengths. If one continuous source is unavailable, create three matching segments with visible straight cuts between complete sentences. Record the segment boundaries and do not call it a single take. The source may be generated with verified native English speech or built from a silent synthetic presenter performance plus English speech and a verified lip-sync operation. Record that entire preparation chain. The final English source must visibly and audibly speak the English script before it becomes the recipe input. DU3. Exact source text and translation draft Use these lines as the source production script. Speak naturally with brief pauses. The times are planning targets, not provider guarantees or final subtitle timings. Line ID Approximate source window English source DU-L1 0–8 s Start with a short brief. Review the words, choose a voice, and listen before you export. DU-L2 8–16 s When one line needs a change, edit that line and record it again. DU-L3 16–24 s Your script stays editable, so the next version starts from work you already have. Spanish target is Spain-oriented neutral Spanish. Use a compatible voice if available. If only another Spanish variety is available, record the choice and adjust any page language accordingly rather than claiming a specific accent. Line ID Spanish working translation Expected meaning DU-L1 Empieza con una breve descripción. Revisa el texto, elige una voz y escucha el resultado antes de exportarlo. Start with a brief description, review the text, choose a voice, listen before export DU-L2 Si necesitas cambiar una frase, edítala y vuelve a grabarla. Edit and record an individual line again when it needs a change DU-L3 Tu guion sigue siendo editable, así que la siguiente versión parte del trabajo que ya tienes. The script remains editable and the next version starts from the existing work These translations are production drafts. Review them against the transcribed actual source, which may differ slightly from the prompt. Do not silently substitute the intended script for words the source never spoke. DU4. Transcription and translation review ID Action Required result DU-T1 Export the final English source and extract its actual audio Original source video, source audio and measured duration DU-T2 Transcribe the source audio Machine transcript with timing and model provenance DU-T3 Listen through and correct the transcript Reviewed source transcript, cue boundaries and difference notes DU-T4 Translate the reviewed source line by line Spanish draft with stable DU-L IDs and speaker mapping DU-T5 Back-translate in a separate pass that receives the Spanish but not the source wording Independent English back-translation, prompt/model recorded DU-T6 Compare source, Spanish and back-translation Saved review table with meaning, omissions, additions, tone and timing notes DU-T7 Accept the target wording under the execution review rules Frozen Spanish import file and source/target hash The review document must have these columns: line ID, actual English source, Spanish target, independent back-translation, timing target, reviewer note, and status. Use a normal saved text/table document or generated comparison sheet, not a fabricated native translation UI. Review the product terms in context: brief means a short description, line means a script line, and editable must remain explicit. Do not add claims about automatic timing, perfect translation, or voice cloning. Record who reviewed language and pronunciation. An agent check is an agent check. Do not label it native-speaker or human approval without that review having happened. DU5. Script guided flow Create the translated script through the actual guide and capture it. Bring the already translated text into Idea. The Language selector in Voices must not be described as translating the source. Capture ID Screen or surface What to do and show DU-C0 Source playback Play the English presenter clip, with a source-language label outside the video DU-C1 Transcription/review preparation Show the actual source transcript and corrected words DU-C2 Translation review document Show one source/Spanish/back-translation row and its real review note DU-C3 Script → Idea Use Paste your script or Import subtitles with the accepted Spanish. Show the imported-words panel DU-C4 Format selection Choose Voiceover narration. Select a custom target length corresponding to the measured source if appropriate. Show the actual preparation button DU-C5 Format review Show the accepted Spanish lines, speaker and imported timing where present. Make a genuine necessary correction if one exists, then Continue to voices DU-C6 Voices Set Spanish, choose an available voice and pace, audition it reading actual target words DU-C7 Voice your script → editor Generate the lines, then show loaded playable takes and the line structure DU-C8 Script editor Revoice one genuinely corrected/adjusted line if needed and export actual SRT DU-C9 Timeline Show original picture, replacement speech, captions and measured source duration DU-C10 Separate lip-sync operation Capture the real agent/editor operation and completed output. Identify it as finishing outside setup DU-C11 Comparison Play English source and Spanish result in a clearly labeled comparison Notes for the writer field: This is an approved Spanish translation of an English presenter clip. Keep the imported words exactly. One presenter reads all lines. Preserve line IDs and subtitle cue timing when the import provides it. Prepare the lines for voice review without rewriting, summarizing, or translating them again. Use an attributed subtitle or script format supported by the actual importer. If IDs cannot be embedded in that format, maintain the DU-L-to-imported-line mapping in documents/line-map.json. An import preserves text, but its target times do not prove that a voice take will fit them. Record and measure every returned take. DU6. Voice, timing, lip sync, and export ID Operation Detailed instruction DU-G1 Audition a Spanish voice Use DU-L1 or a representative segment. Listen for clear pronunciation, natural stress and an appropriate pace. Choose one voice for the entire target script DU-G2 Voice the accepted script Set the speaker’s provider/model/voice explicitly, then generate lines without conflicting global overrides DU-G3 Measure takes against source windows Record duration and leading/trailing silence per line. Place sentence boundaries deliberately. Shorten the translation only through another documented meaning review DU-G4 Assemble a Spanish voice track Keep the speech editable per line. Use natural pauses and small timing adjustments. Do not trim phonemes, stack overlapping words or conceal unnatural time-stretching DU-G5 Export Spanish voiceover cut Keep the original picture unchanged. Mute original speech. Add the new voice and optional room tone without doubling the English audio DU-G6 Run the accepted lip-sync route Use the original English source picture and the accepted Spanish audio as required by the verified tool contract. If segmented, retain exact segment boundaries and consistent settings DU-G7 Inspect synchronization Watch at normal speed and frame-step at starts, stops, plosives and transitions. Compare identity, mouth region, teeth and face edges against source. Retry only the failing segment DU-G8 Generate subtitles from final delivery timing Use the final aligned Spanish track, not approximate source cue windows. Verify every subtitle’s words and timing. Preserve SRT and VTT DU-G9 Finish the lip-synced timeline Use the accepted lip-sync outputs, accepted Spanish speech and editable captions. Avoid accidentally keeping the source’s English audio DU-G10 Export both target versions and comparison Retain clean and captioned versions. Equalize comparison loudness and identify English, Spanish voiceover, and Spanish lip-synced versions correctly Do not create music for this recipe. Clear speech is the comparison. Native source ambience may be retained only when it does not include residual English speech or obscure the target voice. Make the side-by-side comparison a 1920×1080 composition containing two uncropped 16:9 source views scaled to fit, with readable language labels. Play the English source first and Spanish result second, or provide separate sections so the voices never compete. Also deliver the individual files so the page can use an accessible language switch later. DU7. Required deliverables Asset ID Path or family Requirements DU-A1 sources/presenter-reference.<ext>, presenter-entity.json Required for synthetic source production, or supplied-source provenance in their place DU-A2 sources/source-en.mp4, source-en-audio.wav Accepted English presenter source and actual audio DU-A3 documents/source-transcript.md, source-en.srt Corrected transcript and actual cue timing DU-A4 documents/translation-review.md, target-es.txt, back-translation.md Traceable source/target/meaning review DU-A5 documents/script-en.json, script-es.json, line-map.json Editable source/target documents and stable line/take mapping DU-A6 masters/voice-es/, masters/voice-es.wav Original target takes and final aligned voice stem DU-A7 masters/dub-es-voiceover-clean.mp4, dub-es-voiceover-captioned.mp4 Same source picture with replacement Spanish speech DU-A8 masters/dub-es-lipsync-clean.mp4, dub-es-lipsync-captioned.mp4 Inspected lip-synced output. Required, not replaceable by DU-A7 DU-A9 documents/timeline-voiceover.json, timeline-lipsync.json Editable finishing documents and dependencies DU-A10 web/source-en.mp4, dub-es-voiceover.mp4, dub-es-lipsync.mp4 and WebM siblings Web deliveries with sound and individual posters DU-A11 web/dub-es.srt, dub-es.vtt, source-en.vtt Final aligned subtitle/caption files DU-A12 masters/language-comparison.mp4, web/language-comparison.mp4, .webm, poster.webp Labeled before/after with one audible version at a time DU-A13 web/translation-review.webp Legible excerpt from the actual reviewed translation table DU-A14 web/recipe-card.webp, social-preview.webp Real presenter result and language labels, common dimensions DU-A15 captures/steps/du-c0 through du-c11 Screenshots of real preparation, guide, finishing and comparison DU-A16 captures/walkthrough/dubbing-guided-flow.mp4, .webm, .vtt, poster.webp 45–60-second demonstration with explicit stage boundaries DU-A17 evidence/timing-report.json, lipsync-review.md, language-review.md, source-provenance.json Measured timing, visual sync review, actual review status, source chain Include common manifest, capture log, handoff and QA. If the source is synthetic, the comparison caption must say so. Do not describe the voice as cloned or identical to the presenter unless that was actually authorized and performed. DU8. Walkthrough edit Segment Approximate time Picture Caption DU-W1 0–6 s English and Spanish result excerpts The same presenter clip in another language. DU-W2 6–14 s Actual transcript and translation review Review the transcript and translation first. DU-W3 14–21 s Script Idea import Bring the approved words into Script. DU-W4 21–28 s Format and line review Keep the wording editable, line by line. DU-W5 28–37 s Voices, language, actual audition Hear the voice before generating the script. DU-W6 37–44 s Loaded takes and target subtitles Review the delivery and subtitles. DU-W7 44–52 s Actual separate lip-sync operation and timeline Finish the dubbed picture. DU-W8 52–59 s Spanish result Preview the complete version. If DU-P5 is blocked, edit only a clearly titled voiceover demonstration and mark it partial. Do not silently remove DU-W7 and continue to claim the complete recipe is ready. DU9. Acceptance and handoff Check ID Pass condition DU-Q1 English source visibly speaks the transcribed words, and its synthetic/supplied origin is traceable DU-Q2 Spanish preserves every source claim and adds none. Independent back-translation and review notes exist DU-Q3 Imported Spanish was preserved through the guide. Language selection is shown as voice setup, not translation DU-Q4 Voice is intelligible and natural, no clipped syllables or residual English audio in target versions DU-Q5 Subtitles match actual final words and timing. Cue times increase, end after start, and remain within video duration DU-Q6 Lip-synced output uses the same source performance and passes a full viewing plus checks at speech boundaries DU-Q7 No obvious facial identity drift, flickering mouth/teeth, face-edge seams or frame discontinuities remain in the selected output DU-Q8 Before/after comparison is fairly framed and loudness-matched, with no simultaneous competing speech DU-Q9 All DU-A assets are present and the common M11 checks pass. Missing lip sync means partial, not accepted In handoff.md, separate supported claims about translation, voice editing, subtitles and lip sync. State the actual target language/voice variety and language-review status. Recommend the individual language videos as the main page interaction assets and the comparison film as a standalone proof clip. Include a factual synthetic-source caption and a short description of each finishing step outside setup. DU10. Ready-to-dispatch task Produce D2 Multilingual video assets using the master plan and this recipe plan. Check the direct lip-sync route early. Create or use the authorized English presenter source, transcribe and review it, translate it into Spanish, back-translate independently, import the accepted target through Script setup, choose and generate voices, and deliver editable scripts, aligned subtitles, voiceover and lip-synced cuts, comparisons, real guided-flow captures and QA. Keep transcription/translation/lip sync visibly separate from the Script guide. Do not invent a translation button, clone an unauthorized identity, use node graphs, or build marketing pages. If a required operation is unavailable, complete the independent assets and report an explicit partial handoff.--- # D4. Storyboard to Trailer: asset production plan Source: https://docs.nodetool.ai/plans/marketing-recipes/storyboard-to-trailer.html D4. Storyboard to Trailer: asset production plan TR0. Assignment Read MASTERPLAN.md first. Produce the asset package for storyboard-to-trailer when dispatched. Use the actual Storyboard guided flow to create the project, then finish through the script, storyboard and timeline editors. Do not create a node graph or modify the marketing pages. Produce an original 40-second mystery trailer titled The Next Tide, with six generated shots, recurring character/location entities, a linked editable script, three voiced lines, an original score, finished exports, posters, and complete guided-flow captures. Read .claude/skills/short-film/SKILL.md and .claude/skills/script-video/SKILL.md after storyboard-core. Follow the actual guide’s ordering for the recorded production. The screenplay review precedes Entities. Do not skip that visible stage because reference assets were prepared earlier. Owner directory: <run-root>/storyboard-to-trailer/. This recipe is independent of the other production tasks and is the first pilot for the common Storyboard capture procedure. TR1. Fixed creative brief Logline: A lighthouse keeper finds a message in a bottle dated forty years in the future, written in her own handwriting. The trailer is a quiet mystery. It should suggest a specific story through a small number of clear images. No monster reveal, disaster footage, frantic montage, or generic action sequence. There is one visible adult character and one coastal location with an interior. Speech is voiceover, so none of the selected shots requires lip sync. Production brief for the Idea screen: The Next Tide: a 40-second mystery trailer. A lighthouse keeper finds a message in a bottle dated forty years in the future, written in her own handwriting. Six shots: establish the coast, discover the bottle, examine the letter, watch the keeper's reaction, return to the lighthouse beam, end close on her recognition. Blue-hour sea light and a warm practical lamp inside. One keeper, one lighthouse. Quiet, restrained, photographic. Three voiceover lines: "The tide brought it in." "The date is forty years from now." "And the handwriting is mine." No generated on-screen text. Select Mystery and 6 shots in Story choices. The Director may draft a different structure. Review it in the real UI and edit it toward TR4. Record the draft and final reviewed version rather than pretending the Director generated this exact plan unchanged. Board style: Restrained photographic coastal mystery at blue hour. Cold blue-gray sea and sky, weathered stone, damp surfaces with believable reflections, warm amber practical light only in the keeper's room. Natural skin detail, moderate contrast, fine film grain, no fantasy glow, no exaggerated teal-orange grade. Camera movements are slow and deliberate. No generated lettering, titles, logos or subtitles. Lighthouse architecture, keeper identity and wardrobe remain consistent across shots. Final format: 1920×1080, 24 fps, 40 seconds including a two-second title card. These are final edit specifications. Record the native provider output dimensions, lengths and frame rates separately. TR2. Entity package Generate source references only after reviewing authorization. Prepare them before the recorded Entities screen if needed, but select them visibly at that stage. If references already exist and the execution instruction authorizes reuse, inspect their provenance and suitability before creating anything new. Entity ID Name / kind Reference direction Assignment TR-E1 Elin / character Original woman around forty, short dark hair with a few gray strands, lightly freckled face, calm direct expression, neutral soft daylight casting portrait. No resemblance request for a real person TR-S2, TR-S4, TR-S6 TR-E2 North Point Lighthouse / location Modest cylindrical weathered white lighthouse on a low rugged headland, dark lantern roof, a small keeper’s cottage beside it, blue-gray sea and a narrow stone path. Consistent building proportions TR-S1, TR-S2, TR-S5 TR-E3 Keeper's Room / location Small stone-walled room with a weathered wooden table, a simple amber-shaded lamp on the left, one small window showing blue twilight, restrained maritime details without text TR-S3, TR-S4, TR-S6 TR-E4 Message Bottle / prop Small clear weathered glass bottle, simple cork, folded cream paper inside, no label or embossed brand mark, photographed against a neutral background TR-S2, TR-S3 For Elin, keep the canonical descriptor about face, hair and build. Wardrobe is fixed in shot text: a dark navy wool sweater under a faded mustard raincoat, no hat, scarf or visible logos. Use the same wardrobe wording whenever she appears. Do not invent alternate outfits or wet/dry identity changes. Use an explicit entity list on every shot. The exterior location must not automatically apply to the interior shots. The bottle must not appear in the final close-up unless the shot asks for it. Save one accepted visual reference per entity plus its full entity export and descriptor. Reference acceptance checks: recognizably original adult face, consistent lighthouse silhouette, usable room geometry and a simple bottle without implausible glass or paper. Generate at useful inspection resolution, preferably at least 1536 px on the long edge when the model supports it. Do not present a collage of alternate character candidates as a single accepted entity. TR3. Script and voice Use a single speaker, Elin, delivering the lines quietly as voiceover. The pictures should not show her mouth forming these words. Choose an actual available English voice with a natural, measured delivery. Keep a small amount of breath and pause without turning the lines into a theatrical whisper. Line ID Text Intended placement Direction TR-L1 The tide brought it in. Over the discovery, approximately 7–10 s Matter-of-fact, just beginning to question it TR-L2 The date is forty years from now. Over the letter, approximately 14–18 s Slower on “forty years,” with controlled uncertainty TR-L3 And the handwriting is mine. Over the final close-up, approximately 33–37 s Quiet recognition, a short pause before “mine” if natural These windows are editorial targets. Voice the lines, measure the actual takes, and place them without cutting a word or forcing unnatural speed. Keep the lines as editable script resources with a stable mapping to the shots. If the Studio host already creates a linked script on Story review, use it instead of creating a duplicate. In the workspace, use the real Extract script action if the board is unlinked. The final on-screen title is THE NEXT TIDE. The two-second end card contains that title only. Do not invent a release date, distributor, award badge, review quote, or streaming-platform logo. Do not ask a model to draw the letter’s exact date or readable handwriting. The voice establishes the date. The letter image may show plausible marks, but any exact text necessary to the scene must be authored as editable text through a supported editor and recorded as a finishing operation. No claim about accurate generated handwriting belongs on the page. TR4. Shot and final edit plan Use keyframe mode for all six shots. The stills carry the chosen look. Each motion instruction contains physical events rather than an abstract emotional label. Shot ID Final edit slot Action for the still Motion Entities TR-S1 0–6 s Wide view of North Point Lighthouse from the stone path at blue hour. Lighthouse on the left third, gray sea behind it, keeper’s cottage visible, one warm window. No people, signs or lettering Slow 20 cm push along the path. Sea moves gently. The building stays rigid and the beacon rotates slowly without changing architecture TR-E2 TR-S2 6–12 s Medium view from behind Elin at the edge of the headland. She wears the navy sweater and faded mustard raincoat. Message Bottle rests on wet stone near her. Lighthouse is recognizable but distant. Hands positioned naturally and not obscuring the bottle Elin bends slightly and reaches toward Message Bottle once. A small wave recedes. No repeated reaching, disappearing bottle or extra fingers TR-E1, TR-E2, TR-E4 TR-S3 12–19 s Close view of the wooden table in Keeper’s Room. Message Bottle lies on its side beside unfolded cream paper, warm lamp glow from left, blue window reflection in glass. No hands. Marks on paper are not relied on for readable words Slow 5 cm push toward the paper. One corner moves slightly in a draft and settles. Bottle stays still. No self-writing text TR-E3, TR-E4 TR-S4 19–25 s Close side view of Elin in Keeper’s Room, navy sweater and faded mustard raincoat, face lit by warm lamp with blue twilight on the far side. She looks down at the table. Mouth relaxed and closed Elin lifts her eyes once toward the window and holds. Tiny breath movement, no speech, head morphing or dramatic expression change TR-E1, TR-E3 TR-S5 25–32 s Wide view of North Point Lighthouse across the dark water, same tower and cottage silhouette as TR-S1. The lantern beam points into the empty blue-gray horizon. No boats or new buildings Locked camera. One slow lighthouse-beam sweep across the water, low natural wave movement. No flashing, lightning or additional lights TR-E2 TR-S6 32–38 s Frontal close-up of Elin in Keeper’s Room, same face and wardrobe. Warm lamp catches one side of her face, background falls softly out of focus. Her gaze is just below lens, as though reading the paper A small eye movement toward the camera, one restrained inhale, then stillness. Mouth does not speak. Hold the final second with stable identity TR-E1, TR-E3 TR-TITLE 38–40 s Editable timeline title on a dark blue-gray field derived from the film’s palette Quiet fade or cut, fully readable hold No generation Provider clips may run 5, 6, 8 or other supported lengths. Select a model/duration that can cover the intended slot. Keep source handles where possible. Do not stretch a short clip into a seven-second slot without inspecting the result. Prefer a supported longer render or a deliberate editorial adjustment approved within the plan’s total running time. Narration links can alter assembly timing. Set intended manual holds where the current document contract supports them, then inspect the assembled timeline. Trim and place the final clips explicitly to achieve the table. Do not assume requested shot duration controls the generated file length. TR5. Guided-flow production and captures Capture ID Actual surface Required visible content and action TR-C0 New project Storyboard entry, no node canvas TR-C1 Idea TR1 brief or a concise logline entered. Continue TR-C2 Story selection Mystery and 6 shots selected. Generate screenplay and actual estimate if present TR-C3 Story review Six reviewed shots, action/dialogue/duration fields. Make one real creative correction. Set up entities TR-C4 Entities Accepted Elin, lighthouse, room and bottle references. Show per-shot assignments, including different interior/exterior casts TR-C5 Look 16:9, selected art style and reference-capable still model. Generate your storyboard TR-C6 Storyboard All six loaded stills, coherent grade and continuity TR-C7 Shot inspector/takes A genuine single-shot revision if needed, with old and chosen take visible and recorded TR-C8 Script editor Three voiced lines, Elin’s voice and the link to the board TR-C9 Storyboard Real clip generation and playable completed shots. Waits may be cut in the edited demo TR-C10 Timeline Six picture clips, separate speech and music, editable title card, 40-second final duration TR-C11 Preview/export The actual accepted final trailer playing If no still needs a revision, demonstrate an actual review selection or a text change from the first screenplay draft. Do not intentionally create a broken image just to manufacture a dramatic before/after. The capture log states what changed and why. The guide uses four visible stages, even though Story has two screens. Preserve that structure in the edited video and screenshot filenames. Render clips and Assemble timeline belong after setup. TR6. Production sequence ID Action Evidence TR-G1 Discover connected image, image-to-video, TTS and music models Exact IDs, capabilities, estimated/unknown spend and authorized quantities TR-G2 Generate and review four entity references Source images, descriptors, entity exports, review record TR-G3 Create the project through Storyboard setup Real brief, initial screenplay, reviewed screenplay, every guided screen capture TR-G4 Select entities and look, then generate six stills Shot-to-reference map, original stills, selected take IDs TR-G5 Review continuity at full size Character, architecture, room light, bottle and wardrobe comparison sheet TR-G6 Open/extract the linked script and voice three lines Actual script export, voice settings, takes and measured durations TR-G7 Render six clips from selected stills Original clips, native durations and frame rates, visual review notes TR-G8 Assemble and edit the film Timeline export, exact source ranges, deliberate final timing TR-G9 Generate and mix the original score Music source, generation provenance, sound stems and measured final mix TR-G10 Add editable title, validate, and export Validation output, 40-second master and web encodes TR-G11 Finish proof images and walkthrough Capture log, posters, contact sheet, actual result playback Do not regenerate an accepted reference or all six shots to fix one scene. Apply the master plan’s targeted iteration rules. If a selected still changes, invalidate and replace only the dependent clip and derived exports. TR7. Score and sound Original score brief: Instrumental score for a restrained forty-second coastal mystery trailer. Sparse low piano notes, a soft sustained bowed texture, a slow pulse that becomes slightly more insistent near the last third, then a quiet resolved tail for a two-second title. No vocals, no speech, no recognizable melody, no imitation of a named artist or film score, no loud trailer impacts. Leave space for three short, quiet voiceover lines. Discover a real text-to-music route. If the board’s screenplay music field is supported by the active tool family, set it through that contract and verify the resulting draft/generation. Otherwise add an actual generated music asset to a timeline track. Do not use the disabled generic Video Music control or pretend a draft audio binding is rendered sound. Generate the bed as an original source, retain its native length, and trim/mix it deliberately to the final timeline. Create any silence, fades or ducking through timeline edits. Do not expect a prompt to produce an exact silent gap. Add one restrained coastal room-tone/wind/water bed only if available within the approved production. It may be generated separately or taken from authorized source audio. Do not mix competing native audio from six video clips. Mute unwanted Shot Audio clips under the continuous score and voiceover, then listen through the full film for joins and clicks. The score is required for the trailer package. Extra sound effects are optional. If music generation is unavailable, an explicitly authorized original/licensed score may be used with provenance. A silent trailer is partial unless the user accepts that changed outcome. TR8. Required deliverables Asset ID Path or family Requirements TR-A1 sources/entities/elin.<ext>, lighthouse.<ext>, keepers-room.<ext>, message-bottle.<ext> Four accepted references, originals and viewable derivatives TR-A2 documents/entities.json, entity-descriptors.md Entity exports and explicit shot assignments TR-A3 documents/brief.md, screenplay-draft.json, screenplay-reviewed.json, production-board.json Actual guide draft, reviewed plan and finished board TR-A4 documents/script.json, voiceover.txt, shot-line-map.json Three accepted lines, voice settings and board links TR-A5 selects/tr-s1.png through tr-s6.png Six selected stills at original resolution TR-A6 selects/tr-s1-clip.<ext> through tr-s6-clip.<ext> Six accepted original motion takes TR-A7 masters/audio/voice/, score.<original-ext>, voice-stem.wav, music-stem.wav Original and editing audio assets TR-A8 documents/timeline.json, shot-map.json Editable final cut, source in/out points, selected takes, text and sound TR-A9 masters/the-next-tide-clean.mp4, the-next-tide-captioned.mp4 40-second high-quality final versions with editable project sources TR-A10 web/the-next-tide.mp4, .webm, the-next-tide.vtt Web delivery with sound and accurate captions TR-A11 web/trailer-poster.webp, trailer-poster-960.webp A strong actual film frame, with title treatment derived from the editable title TR-A12 web/storyboard-contact-sheet.webp Six selected frames in story order, short shot labels TR-A13 web/entity-continuity.webp Source references paired with relevant selected frames TR-A14 web/recipe-card.webp, social-preview.webp Common 1600×900 and 1200×630 derivatives from the film TR-A15 captures/steps/tr-c0 through tr-c11 Every actual setup and finishing capture TR-A16 captures/walkthrough/trailer-guided-flow.mp4, .webm, .vtt, poster.webp 45–60-second walkthrough using real assets and screens TR-A17 evidence/continuity-review.md, timeline-validation.json, media-probes.json, mix-measurement.txt Full visual, structural and audio checks Retain originals and all required common handoff files. A contact sheet is a derived illustration of selected frames, not evidence of the UI itself. Keep it separate from the board screenshot. TR9. Walkthrough edit Segment Approximate time Picture Caption TR-W1 0–5 s Lighthouse or keeper moment from final trailer From a logline to a trailer. TR-W2 5–11 s Idea screen Start with the story. TR-W3 11–20 s Mystery selection and real screenplay edit Choose the direction. Review the shots. TR-W4 20–29 s Entities and per-shot assignments Keep the keeper and locations consistent. TR-W5 29–36 s Look and loaded still board Set the look before generating clips. TR-W6 36–44 s Script takes and completed clips Voice the lines and animate the shots. TR-W7 44–52 s Timeline with score and title Finish the cut on the timeline. TR-W8 52–58 s Final trailer/title The storyboard, script, and cut stay editable. Use readable holds on the Story and Entities screens. Fast cursor movement cannot substitute for showing the decision. Keep the guide visually dominant and the finished film large enough to judge. TR10. Acceptance and handoff Check ID Pass condition TR-Q1 Six shots communicate the premise, discovery and recognition without needing explanatory page prose TR-Q2 Elin’s face, hair and wardrobe remain consistent. Exterior and interior references are assigned only where appropriate TR-Q3 Lighthouse architecture and keeper-room layout remain recognizable between shots TR-Q4 All clips pass full playback for object drift, hand defects, camera jumps and unwanted text TR-Q5 Three voiceover lines match the script, are intelligible, and are not falsely shown as lip-synced speech TR-Q6 Final cut is 40 seconds to within one 24 fps frame, including a readable two-second title card TR-Q7 Score is rendered, original/authorized and traceable. Speech and music are separate editable tracks with no overlapping unwanted shot audio TR-Q8 Real setup captures include Idea, both Story screens, Entities and Look, followed by clearly separate editor actions TR-Q9 Every TR-A item exists and the common M11 checks pass In handoff.md, identify the strongest wide poster and close-up frame, recommend a card crop, provide a one-sentence story synopsis and accurate captions, and list which document demonstrates each editing capability. Give the coordinator the capture procedure that worked so D1 and D3 can reuse its framing and pacing without reusing this recipe’s content. TR11. Ready-to-dispatch task Produce D4 Storyboard to Trailer assets using the master plan and this recipe plan. Make the original forty-second mystery trailer The Next Tide through the real Storyboard guided flow: Mystery, six shots, reviewed screenplay, character/location/prop entities, look and stills. Then voice the three linked script lines, render clips, edit the timeline, add an original score and title, and deliver masters, web encodes, posters, all setup captures, manifests and QA. Pilot the common Storyboard recording procedure and share it with the coordinator. Do not create node graphs, fake live generation, use unrelated footage, or build marketing pages. Respect the supplied execution authorization.--- # D1. Product ad variants: asset production plan Source: https://docs.nodetool.ai/plans/marketing-recipes/viral-video-ad-engine.html D1. Product ad variants: asset production plan AD0. Assignment Read MASTERPLAN.md first. Execute this plan with Sol when the user dispatches asset production. Deliver assets and evidence only. Preserve the recipe identifier viral-video-ad-engine. Do not edit the marketing page, its data, or the shipped recipe manifest. Produce three finished 15-second vertical product ads, each with a different opening hook, using the same product, remaining footage, voice, and grade. Supply matching thumbnails, an editable script and storyboard, and a recorded walkthrough of the actual Storyboard guided flow. Read .claude/skills/product-commercial/SKILL.md, .claude/skills/launch-kit/SKILL.md, and .claude/skills/script-video/SKILL.md after storyboard-core. The guided flow remains the demonstrated starting path. Use the script skill for voicing and timing after the reviewed board exists, not as a reason to replace the guided start with a different workflow. Owner directory: <run-root>/viral-video-ad-engine/. Project name: Marketing recipes / viral-video-ad-engine / <run-id>. AD1. Required inputs and dependency D3 owns the shared Olive Travel Cup reference package at <run-root>/shared/product/. Wait for its source acceptance before any product generation. You may prepare copy, capture instructions, model selection, and document structure while waiting. ID Input Use AD-IN1 Accepted original cup reference and SHA-256 Visual authority for every frame AD-IN2 Accepted product entity ID or entity export The same cup attached to every product shot AD-IN3 Product descriptor and source disclosure Prompt consistency and handoff provenance AD-IN4 Source cutout, if accepted and available Reference/comparison asset, not an invented new product AD-IN5 Execution authorization, models, and capture session Production controls from the coordinator If running in a different NodeTool instance, import the accepted bytes and recreate the entity from that image. Record the old and new IDs and the unchanged file hash. Do not generate a visually similar replacement. The cup is unbranded and fictional. Use no price, discount, warranty, capacity, heat-retention claim, or implied customer testimonial. The ad demonstrates a production process rather than offering a real item for sale. AD2. Creative direction Setting: a quiet morning kitchen, pale stone worktop, soft window light from the left, warm neutral surroundings, muted olive cup and charcoal lid. There are no people or hands in this first sample. This keeps the cup readable and avoids an unnecessary character-generation branch. Board style: Photographic product commercial in a quiet morning kitchen. Soft warm daylight from a window on the left, pale limestone and warm off-white surfaces, gentle shadow falloff, accurate muted-olive and charcoal colors, restrained contrast, clean product edges. The Olive Travel Cup keeps its reference geometry and finish in every shot. No on-screen type, logo, lettering, watermark, or extra product decoration. Camera and light move gently. The product stays still. Use the accepted entity’s geometry descriptor without embellishing it. The grade belongs on the board once. Shot actions specify composition and physical details that vary. Final format: 1080×1920, 30 fps, exactly 15 seconds per variant after editing. Generate at the provider’s supported resolution/duration and record those originals. Do not claim the provider generated exactly 15 seconds or that a low-resolution render was native 1080p. Typography is added as editable timeline text. Use a bundled font, medium weight, with a small amount of text in a clear area. Keep important text within the middle 80% of width and away from the top/bottom 12% of the portrait frame. Produce a clean master without burned-in captions as well as the captioned delivery. AD3. Copy and script Use a single calm adult narrator. Select an actual available English voice and audition it. Do not request an imitation of a public figure. Preserve one voice selection across all versions. Line ID Section Variant A Variant B Variant C AD-L1 Hook Make a little room for your morning. Your morning, before the noise. A quieter start looks like this. AD-L2 Moment A warm cup. Same accepted take as A Same accepted take as A AD-L3 Moment A moment to yourself. Same accepted take as A Same accepted take as A AD-L4 Close Take it with you. Same accepted take as A Same accepted take as A Exact end-card text: Take it with you. No button, fake URL, price, or made-up brand mark. Keep spoken words as four editable lines. Voice the three alternative hooks separately and reuse the accepted shared lines. Measure takes before cutting. Let pauses and product holds fill the 15 seconds. Do not force a long take into its slot with visibly or audibly excessive speed changes. If a hook cannot fit naturally, first choose a calmer but more concise delivery direction, then report any necessary copy change. Save documents/ad-copy.md, three script exports, original speech files, and final subtitle files. Preserve source line IDs and their relation to each shot. Where a script-linked assembly derives duration from audio, inspect and adjust the final timeline deliberately to reach the 15-second delivery. Do not assume shot duration fields override speech timing. Keep A as the canonical script linked to the production board. B and C are separate variant script documents and cuts. Do not repeatedly relink the same board to three scripts or overwrite A to produce B. Reuse accepted audio through supported copy/take operations when available, or reference the original shared audio assets from the variant timelines and manifest. Never invent a voiced status on a script line whose take was not actually attached. If the current tools cannot copy takes, document that limitation while still reusing the audio bytes in the finished cuts. AD4. Shot plan Create four main shots through the guided flow, review them to this specification, and generate those stills first. Add the two alternative opening shots only after the main ad has a usable cut. Shot ID Final cut position Action for the still Motion instruction Entity AD-S1A 0–3 s, opening A Medium three-quarter view of Olive Travel Cup on a pale stone kitchen worktop. A soft rectangular window shadow falls behind it. Entire lid and base visible, cup on right third, clear space on left for a short caption. Slow 5 cm push toward the cup. Window light remains steady. The cup and lid do not move or change. Product only AD-S2 3–7 s, shared Close product detail of Olive Travel Cup’s charcoal lid meeting the olive body. Both the rectangular drinking opening and matte finish remain recognizable. Shallow depth of field, no invented handle or seam. Gentle rack focus from the lid edge to the front of the cup. No rotation, lid movement, steam burst, or liquid. Product only AD-S3 7–11 s, shared Olive Travel Cup on the worktop beside a closed plain cream paperback. Cup fully visible. The book has no readable text and is smaller in visual emphasis. Warm diagonal light leads to the product. Slow 8 cm lateral slide. The cup and book remain stationary, with stable perspective and contact shadows. Product only AD-S4 11–15 s, shared close Front three-quarter hero view of Olive Travel Cup on pale stone. Cup below center with uncluttered space above. The lid opening faces the same direction as the reference. Very small push for the first two seconds, then a stable hold. No object transformation or background movement. Product only AD-S1B 0–3 s, alternate B Olive Travel Cup on the same surface with its long soft shadow leading into frame. Wider composition than AD-S1A, entire object visible, same window direction. Camera holds while a subtle exposure-stable highlight shifts along the background. The product does not move. Product only AD-S1C 0–3 s, alternate C Close view from slightly above Olive Travel Cup showing the charcoal lid and the upper olive body. Space below for the short hook caption. Same geometry and finish as the reference. Slow 10-degree camera arc. The lid opening remains fixed on the product. No full orbit or newly invented backside detail. Product only The time ranges above are final edit slots. A provider may return longer clips. Trim the accepted usable section in the timeline and record source in/out points. Keep the full original clip. AD5. Guided production and capture sequence Capture each screen before advancing. Use the same project for production and the recording. Do not prefill successful media into a fresh empty guide and portray it as a real generation. Capture ID Surface and action Required visible content Result AD-C0 New project Storyboard entry card and the short brief Establish the actual entry AD-C1 Idea Paste the brief below and click Continue Saved production brief AD-C2 Story selection Commercial and 4 shots selected. Show Generate screenplay and its estimate if available Director writes the initial screenplay AD-C3 Story review Four shots, useful action text, dialogue and durations. Make one real text edit before continuing Reviewed four-shot plan AD-C4 Entities Select the accepted cup entity, or create it from the shared reference. Show per-shot assignments Cup reference attached to each product shot AD-C5 Look 9:16, chosen style, reference-capable still model, actual estimate, Generate your storyboard Stills start in the real board AD-C6 Storyboard Four loaded stills with the same cup. Open one shot for inspection Visible selected takes AD-C7 Script editor Four lines, narrator voice, a selected voice take, and the link to the board Editable spoken copy AD-C8 Storyboard Render clips followed by completed playable clips Real motion assets AD-C9 Timeline Four shots, voice on its own track, editable end-card text Finished 15-second sequence AD-C10 Agent/editor continuation A real request for two alternative openings and the resulting extra shots/copy Variant work is identified as a continuation AD-C11 Finished work Play A, then show the three opening frames together Clear result of the recipe Brief for AD-C1: Make a 15-second vertical product ad for the Olive Travel Cup. Four shots: a quiet morning opening, a close look at the lid and finish, the cup beside a plain book, and a clean hero ending. Warm window light in a pale kitchen. The cup must follow the product reference I will choose in Entities. Use these spoken lines: "Make a little room for your morning." "A warm cup." "A moment to yourself." "Take it with you." No price, logo, or product claims. The Director’s first draft may differ. Review and edit it through the actual Story screen to match AD4. Film that as a normal creative edit. Do not claim the unedited output matched a hand-authored shot list. Save a raw screenshot of each capture ID. AD-C2 and AD-C3 share the Story step but need separate images. The edited walkthrough should use the visible four-stage guide, then a brief script/board/timeline continuation. AD6. Generation instructions ID Operation Constraints and verification AD-G1 Discover image, image-to-video, TTS and optional music models Use connected models and record exact IDs. The still model must accept the cup image AD-G2 Attach the cup to the board and every product shot Verify actual entity IDs and source image. Do not re-describe an imagined cup AD-G3 Generate the four main stills One initial take each. Inspect the entire cup at full size against the source before accepting AD-G4 Extract/open and voice the script Create real editable lines. Set a voice for the narrator before voicing. Generate only required lines AD-G5 Generate the four main clips Keyframe mode. Start from selected stills. Inspect every frame for lid drift, duplicated objects, crawling texture, and moving contact shadows AD-G6 Assemble and finish A Inspect the resulting duration. Use a separate voice track, mute unwanted native shot audio, add editable end-card text, and trim to 15 seconds AD-G7 Create B and C openings Add two new storyboard shots and two hook lines. Reuse shared shots and accepted shared speech. Do not pay for another full board AD-G8 Finish B and C as separate editable cuts Use supported timeline copy/version/create operations. Record source asset IDs and in/out points for the shared footage. Do not reassemble all six board shots into every ad AD-G9 Produce thumbnails and derivatives Derive them from the selected stills, not another image-generation call. Keep clean and captioned versions Music is optional for this recipe. If used, one original understated instrumental bed is reused across all variants, with voice intelligibility checked. Do not add music through a disabled guided control. The ads must be complete with voice and room for silence if no music model is available. AD7. Required deliverables Filenames below are relative to this recipe’s directory. The manifest must enumerate each actual file, including encoded siblings. Do not mark a family complete if only the master exists. Asset ID Path or family Requirements AD-A1 documents/production-board.json Exported board including four main shots and two alternate openings, with source IDs AD-A2 documents/script-a.json, script-b.json, script-c.json, ad-copy.md Exact accepted copy, voice selections, line/take IDs AD-A3 documents/timeline-a.json, timeline-b.json, timeline-c.json Editable 15-second cuts, asset map and source in/out points AD-A4 selects/ad-s1a.png, ad-s1b.png, ad-s1c.png, ad-s2.png, ad-s3.png, ad-s4.png Six full-quality selected stills AD-A5 selects/<shot-id>-clip.<original-extension> Six accepted source clips, original native resolution and duration AD-A6 masters/ad-a-clean.mp4, ad-b-clean.mp4, ad-c-clean.mp4 Clean high-quality portrait masters with voice AD-A7 masters/ad-a-captioned.mp4, ad-b-captioned.mp4, ad-c-captioned.mp4 Same cuts with legible caption/end-card treatment AD-A8 web/ad-a.mp4, ad-b.mp4, ad-c.mp4 and .webm siblings Web deliveries, plus one WebP poster per variant AD-A9 web/thumbnail-a.webp, thumbnail-b.webp, thumbnail-c.webp One portrait thumbnail per hook, source reference and exact caption documented AD-A10 web/hooks-contact-sheet.webp Three hooks and their corresponding opening frames, labels readable AD-A11 web/recipe-card.webp, social-preview.webp 1600×900 and 1200×630 compositions of real output AD-A12 masters/voice/ and web/ad-a.vtt, ad-b.vtt, ad-c.vtt Original voice takes, editable stems, accurate captions AD-A13 captures/steps/ad-c0 through ad-c11 PNG originals and required WebP derivatives AD-A14 captures/walkthrough/ad-guided-flow.mp4, .webm, .vtt, poster.webp 35–60-second setup and finishing demonstration AD-A15 evidence/product-consistency.webp, evidence/variant-reuse.json Reference versus selected frames, and proof of shared footage/takes Also deliver the common manifest, capture log, handoff, and QA files from the master plan. Do not create sample analytics or conversion results. AD8. Walkthrough edit Target a 45–55-second silent demonstration with concise captions. Keep the raw recording complete. Segment Approximate time Picture Caption AD-W1 0–4 s Best moment of ad A A product ad, with openings to compare. AD-W2 4–10 s Idea entry Describe the ad. AD-W3 10–18 s Commercial choice and real Story review Review the shots and words. AD-W4 18–25 s Cup entity and assignments Use the same product reference. AD-W5 25–31 s Look selection and loaded board, with wait cut disclosed Choose the look and review the stills. AD-W6 31–39 s Script takes, completed clips, timeline Voice it and finish the cut. AD-W7 39–47 s Alternate opening shots and three previews Change the opening. Keep the rest. AD-W8 47–52 s Finished ad held large Editable shots, script, and timeline. These are edit allocations, not a claim about production speed. Avoid a montage so fast that the guided choices cannot be read. AD9. Acceptance and handoff Check ID Pass condition AD-Q1 The same accepted cup appears in all six stills and clips. Lid opening, taper, color and proportions agree with the source AD-Q2 All three ads are 15 seconds to within one delivery frame, with no black gaps or clipped words AD-Q3 Only the opening copy/shot differs between variants unless a documented timing adjustment is necessary. The shared asset map proves reuse AD-Q4 The scripts, selected takes, board and timelines can be reopened and relate to the exported videos AD-Q5 Captions remain readable on a phone-size playback and do not cover the cup or lid detail AD-Q6 The walkthrough contains the actual Entities stage and both Story screens, with no node graph AD-Q7 The package describes a fictional unbranded demonstration and makes no sales, performance, virality, or speed claims AD-Q8 Common M11 checks pass and every AD-A asset is accounted for Recommend ad A as the initial hero unless the evidence favors B or C. Explain the choice in handoff.md using visible composition and readability, not predicted advertising performance. Supply one sentence of alt text for each thumbnail and one factual caption for the comparison sheet. AD10. Ready-to-dispatch task Produce the asset package for D1 Product ad variants using the master plan and this recipe plan. Use the accepted shared Olive Travel Cup reference from D3. Create three 15-second portrait ads with different opening hooks, shared remaining shots and voice takes, editable scripts/board/timelines, thumbnails, real guided-flow captures, web derivatives, and the complete evidence handoff. Use Storyboard > Commercial as the guided entry. Do not use node graphs, invent product claims, replace the shared reference, or build marketing pages. Respect the execution authorization supplied by the coordinator. Wait only on the shared source or a required approval/capability, and finish independent work.--- # Motion Design — Implementation Plan Source: https://docs.nodetool.ai/plans/motion-design-tasks.html Motion Design — Implementation Plan Companion to motion-design.md (the technical design). Every task below is written to be executed by an agent with no prior conversation context. Each task’s first step is: read docs/plans/motion-design.md in full — it is the contract; this file adds sequencing, file-level pointers, and acceptance criteria. Ground rules for every task: Repo root is a TypeScript monorepo; run nvm use && npm install if node_modules is missing (sandboxed environments: npm install --ignore-scripts, see AGENTS.md § Install in sandboxed environments). After changes: npm run typecheck && npm run lint, plus the package tests named in the task. npm run dev:nodetool -- affected maps changed files to the minimal workspaces to rebuild/test. packages/timeline is pure TypeScript (no DOM, no GPU, no store imports) — keep it that way. Web UI: primitives from web/src/components/ui_primitives/ only, design tokens per docs/DESIGN.md. Never import raw MUI. Commit per task with a descriptive message; do not reformat unrelated code. Model key (suggested per task): Opus 4.8 (claude-opus-4-8) — design-sensitive core, rendering internals. Sonnet 5 (claude-sonnet-5) — well-specified implementation and integration. Haiku 4.5 (claude-haiku-4-5-20251001) — mechanical plumbing with exact specs. Dependency graph: M1: T1 ──► T2 (schema) T1 ──► T3 (render) T1 ──► T4 (edit ops) T3 + T4 ──► T5 (agent tools) M2: T3 + T4 ──► T6 (inspector) T3 ──► T7 (chips) T3 + T5 ──► T8 (verify) M3: T3 ──► T9 (text clips) ──► T10 (text agent + UI, also needs T5, T6) Milestone 1 — Animation engine and integration T1 — Animation core in packages/timeline · Opus 4.8 Goal: the pure animation engine: types, easings, preset catalog, compiler, sampler, and edit-op semantics hooks. No web code. Read first: docs/plans/motion-design.md §§ Data model, Preset catalog, Editing semantics. Existing style references: packages/timeline/src/types.ts (doc-comment conventions), packages/timeline/src/splitClip.ts (pure-op shape). Create packages/timeline/src/animation/ with: types.ts — AnimationRole, EasingId, ClipAnimation, AnimationPresetId exactly as specified in the design doc. easing.ts — ease(id: EasingId, t: number): number, each easing pure, t ∈ [0,1] → [0,~1] (back/elastic may overshoot; clamp opacity at the composition site, not here). presets.ts — the v1 catalog table from the design doc (fade, slide, pop, spin, pulse, shake, bounce, kenBurns, float, breathe, rotate), each entry { id, roles, defaultDurationMs, defaultEasing, params: {name, default, min?, max?, options?}[], describe: string, curves(params, canvas): PropertyCurve[] }. Export ANIMATION_PRESETS and getAnimationPreset(id). compile.ts — compileClipAnimations(animations, clipDurationMs, canvas) → CompiledAnimation[] (window math per role incl. delayMs, clamping to clip duration, dropping degenerate windows, resolving normalized preset distances to px). kenBurns compiles as a full-clip one-shot with holdAfter: true, not a repeating loop. sample.ts — IDENTITY_SAMPLE, sampleAnimations(compiled, localMs) implementing the fold rules (add offsets/rotation, multiply scale/opacity; hold-before for in, hold-after for out, modulo for loop). Also export hasActiveAnimationWindow(compiled, localMs): boolean. index.ts barrel; re-export everything from packages/timeline/src/index.ts. Modify: packages/timeline/src/types.ts — add animations?: ClipAnimation[] to TimelineClip (after transitionIn), doc comment included. packages/timeline/src/splitClip.ts — left half keeps in, right keeps out, emphasis/loop copied to both, right-half ids regenerated. Follow how the file already handles caption/paragraphId for split-field precedent. packages/timeline/src/defaults.ts — no new factory needed; verify makeClip output stays valid with the field absent. Tests (packages/timeline/src/animation/__tests__/, Vitest — mirror the package’s existing test setup; if the package has no __tests__ yet, add the standard Vitest config used by sibling packages): easings: endpoints 0→0, 1→1; easeOut monotonic; back overshoots. window math per role, incl. out measured from clip end, delays, clamping when duration > clip length. sampler: identity outside windows, hold semantics, fold of in+loop concurrently, loop wraparound continuity (start==end value for every compiled animation with loop: true), delayed fade-in holds opacity 0 during delay, folded opacity clamped to [0,1] under easeOutBack. split semantics as above. determinism: same inputs → deeply equal compiled output. Acceptance: npm run test --workspace=packages/timeline green; npm run typecheck && npm run lint green; no imports from web/ or DOM types. T2 — Protocol schema + persistence round-trip · Haiku 4.5 Goal: animations survives save/load; older documents load unchanged. Depends on: T1 merged (types exist). Context: TimelineDocument persists as a JSON blob (no DB migration), but the wire schema strips unknown fields on PATCH — see the comment on storyboardBoardId in packages/timeline/src/types.ts. Modify: packages/protocol/src/api-schemas/timeline.ts — add to the clip schema: animations: z.array(clipAnimationSchema).optional() where clipAnimationSchema = z.object({ id: z.string(), role: z.enum(["in","out","emphasis","loop"]), preset: z.string(), durationMs: z.number(), delayMs: z.number().optional(), easing: z.string().optional(), enabled: z.boolean().optional(), params: z.record(z.union([z.number(), z.string(), z.boolean()])).optional() }). preset/easing are plain strings on the wire by design (forward compat); the compiler skips unknown presets. Check web/src/hooks/timeline/timelineDocumentPayload.ts and web/src/hooks/timeline/useLoadTimelineIntoStore.ts: if they pass clips through wholesale, no change; if they enumerate fields, add animations. Tests: protocol package round-trip test — a clip with two animations parses through the zod schema unchanged; a clip without the field parses; an animation with an unknown preset string parses (validation is not the schema’s job). Acceptance: npm run test --workspace=packages/protocol green; save → reload in the web store preserves animations (covered by the payload check above); typecheck/lint green. T3 — Preview + export rendering integration · Opus 4.8 Goal: animated transform/opacity visible in the live preview and identical in the MP4 export. Depends on: T1. Read first: design doc § Rendering integration; web/src/components/timeline/preview/sceneModel.ts (whole file — note the change-horizon contract on computeActiveLayersWithHorizon, which must NOT change); web/src/components/timeline/preview/PreviewCompositor.tsx; web/src/components/timeline/render/TimelineRenderer.ts; web/src/components/timeline/preview/gpu/transform.ts (position = canvas px from center, scale multiplies contain-fit, rotation radians). Create in sceneModel.ts (keep it pure): resolveAnimatedLayerProps(layer, currentTimeMs, canvas, cache) → { transform?: ClipTransform; opacity: number } per the design doc’s composition table. Identity fast-path: no enabled animations → return the layer’s existing values without allocating. hasActiveAnimation(layers, currentTimeMs, cache): boolean. An AnimationCompileCache (Map keyed by clip id, invalidated when the animations array reference changes) so compilation never runs in the rAF loop. Wire up: PreviewCompositor.tsx: where each ActiveLayer’s transform/opacity feed the GPU layer list, route through resolveAnimatedLayerProps. Then fix the redraw condition: the compositor skips redraws while the cached layer set is valid (nextChangeMs horizon) and nothing else changed. With animations, any tick where the playhead time changed and hasActiveAnimation(...) is true must redraw even though the layer set is cached — otherwise motion freezes mid-clip. A paused playhead needs only the one draw after the last seek: samples are pure functions of time, so a static frame stays correct. render/TimelineRenderer.ts: apply the same resolver at each stepped frame. The cache can live for the whole render. Caption layers get the same resolution (they already carry transform). Tests (Jest, web/src/components/timeline/preview/__tests__/): composition: static transform {position:{x:100,y:0}} + slide in from left at window midpoint → hand-computed expected transform/opacity. parity: for a seeded clip set, resolveAnimatedLayerProps output at t = {window start, mid, end, past-end} is identical when called the way the compositor calls it and the way the renderer calls it. horizon: nextChangeMs from computeActiveLayersWithHorizon is identical with and without animations on a clip — animations must never shrink the horizon; callers keep reusing cached layers and re-resolve properties per frame instead. Acceptance: cd web && npm test -- --testPathPattern=preview green; manual check via the debug harness or cd web && npm start optional; typecheck/lint green. Do not modify computeActiveLayersWithHorizon’s signature or horizon semantics. T4 — Store patch action + trim/duplicate semantics · Sonnet 5 Goal: clip animation edits flow through the Zustand store with undo/redo, and duplicate/trim behave per the design doc. Depends on: T1. Read first: design doc § Editing semantics; web/src/stores/timeline/TimelineStore.ts (find patchClip(clipId, patch) around the store actions; note the temporal undo middleware and pure-reducer style); packages/timeline/src/trimClip.ts. Do: Add a store action setClipAnimations(clipId, animations: ClipAnimation[]) (thin wrapper over patchClip is fine if patchClip already handles arbitrary fields — verify it does not shallow-merge arrays incorrectly). duplicateClip path (find the existing duplicate logic — store or useTimelineAgentBridge.ts): copy animations with fresh ids (crypto.randomUUID()). Trim: confirm no data rewrite is needed (compile-time clamping from T1 covers it) and add a comment where trimClip is called noting that. Verify split flows through the T1 splitClip change (store calls the pure op — confirm, don’t reimplement). Tests: store-level Jest test: set animations → undo → redo; split a clip with in+out+loop and assert the halves’ animation roles; duplicate regenerates ids. Acceptance: cd web && npm test -- --testPathPattern=timeline green; typecheck/lint green. T5 — Agent tools · Sonnet 5 Goal: an agent can discover the motion vocabulary, apply animations, and inspect them — the primary authoring surface. Depends on: T1, T3 (so results are visible), T4 (store action). Read first: design doc § Agent surface; web/src/lib/tools/builtin/timeline.ts (register pattern, targetParam, error convention); web/src/components/timeline/timelineAgentBridge.ts (TimelineAgentHandler); web/src/hooks/timeline/useTimelineAgentBridge.ts (handler implementation — see setClipParams at ~line 377 for the patch pattern). Do: Extend TimelineAgentHandler with setClipAnimations(target, animations: ClipAnimationInput[], mode: "add" | "replace") and clearClipAnimations(target, role?); implement in useTimelineAgentBridge.ts (resolve target by id/name/”selected” like the existing methods; fill defaults from the preset catalog; validate preset/role and throw descriptive errors that include the valid options). Register in web/src/lib/tools/builtin/timeline.ts: ui_timeline_animate_clip — zod params { target, mode: z.enum(["add","replace"]).optional(), animations: z.array(z.object({ role, preset, durationMs?, delayMs?, easing?, params? })) }. Description must include the compact vocabulary summary (preset names by role, that durations are ms, and the recommended loop: get_state → animate → get_clip_frames at window boundaries → adjust). ui_timeline_clear_animations — { target, role? }. ui_timeline_list_animation_presets — no params; returns ANIMATION_PRESETS mapped to { id, roles, params, defaults, describe }. Include animations in getSnapshot() clip entries (find the snapshot builder in useTimelineAgentBridge.ts). Tests: Jest tests beside existing frontend-tool tests (find the pattern under web/src/lib/tools/): registering is side-effectful on import, so test through FrontendToolRegistry.call with a stubbed handler — animate applies defaults, an unknown preset produces an error whose message lists the valid preset ids, clear with role filter keeps other roles. Acceptance: cd web && npm test -- --testPathPattern=tools green; typecheck/lint green; tool count in the manifest grows by 3. Milestone 2 — Manual UI + verification T6 — Inspector “Animate” section · Sonnet 5 Depends on: T3, T4. Read first: design doc § Manual UI; web/src/components/timeline/Inspector/ClipAdjustments.tsx (section pattern, fold behavior via usePersistedFold.ts), InspectorPrimitives.tsx (labeled rows, sliders, selects), web/src/components/ui_primitives/STRATEGY.md, docs/DESIGN.md. Do: an “Animate” section in the clip inspector: list current animations grouped by role; add-animation flow (role → preset filtered by roles → defaults applied); per-animation controls: duration (ms), delay, easing select, preset params (number params as sliders with catalog min/max, option params as selects); enable toggle; delete. All edits go through the T4 store action (single patchClip per change so undo granularity is per-edit). Constraints: ui_primitives only; SPACING/TYPOGRAPHY/MOTION tokens; no hardcoded px/transition strings; selection comes from the existing inspector wiring (TimelineInspector.tsx). Tests: RTL: renders animations from a stub clip; adding a pop in-animation dispatches the expected store patch; slider change patches params. Acceptance: cd web && npm test -- --testPathPattern=Inspector green; typecheck/lint green. T7 — Clip animation chips in track lanes · Haiku 4.5 Depends on: T3. Read first: web/src/components/timeline/Tracks/Clip.tsx (how fade handles / status badges are drawn at clip edges, zoom-dependent widths). Do: read-only affordances on clips that have animations: a left wedge spanning the in window’s width at current zoom, a right wedge for out, a small loop glyph (⟳ icon from the project’s icon set) when a loop/emphasis animation exists. Clicking them selects the clip (existing click path). Keep it cheap: pure derivation from clip.animations, no new store state; hide wedges below a minimum px width (~6 px) to avoid clutter at low zoom. Acceptance: visual check in cd web && npm start; no new lint/type errors; existing Tracks tests still green. T8 — End-to-end verification pass · Sonnet 5 Depends on: T3, T5. Goal: prove the loop the feature exists for: an agent animates a clip and the export contains the motion. Do: Add a seeded visual test to the existing web visual/debug-harness suite (web/tests/, see yts806379-everything-claude-code-e2e-testing patterns in repo tests): build a two-clip sequence in code (image clip + slide in + kenBurns loop; second clip pop in with delay), render frames at fixed timestamps through the real compositor, snapshot-compare. Export parity: run renderTimeline for the same sequence at a low fps (e.g. 5) for 3 s and assert frame N’s decoded pixels match the preview compositor’s output within tolerance (reuse whatever the existing export tests do — locate them under web/src/components/timeline/render/). Update docs/video-editor.md (and docs/timeline-editor-prd.md if it lists capabilities) with a short “Animations” section; follow docs/WRITING_STYLE.md. Acceptance: new tests green in cd web && npm test; docs updated; full npm run check green at repo root. Milestone 3 — Text clips (phase 2, separately mergeable) T9 — text clip type + rasterized rendering · Opus 4.8 Depends on: T3. Read first: design doc § Phase 2; web/src/components/timeline/preview/captionRender.ts (existing text rasterization — reuse its font/scale handling); web/src/components/timeline/preview/gpu/source.ts (how bitmaps become GPU sources); packages/timeline/src/types.ts. Do: packages/timeline: extend TimelineClip.mediaType union with "text"; add textStyle?: ClipTextStyle ({ text: string; fontFamily?: string; fontSizePx: number; fontWeight?: number; color: string; align?: "left"|"center"|"right"; maxWidthFrac?: number }); protocol schema field (same pattern as T2); makeClip support. Scene model: a text clip contributes a layer with kind: "image" semantics but sourced from a rasterized bitmap — introduce kind: "text" on ActiveLayer and teach both compositors (gpu/compositor.ts, gpu/canvas2dCompositor.ts) and the renderer to draw it: rasterize textStyle to an offscreen canvas at sequence resolution × devicePixelRatio, cache by hash of textStyle, upload like any image source. Text clips are sourceType: "imported" with status: "generated"-equivalent drawability — pick the minimal status handling that makes effectiveAssetId-gated paths treat text as always drawable (they have no asset id; adjust computeActiveLayers’ caption-only guard accordingly). Animations apply unchanged via resolveAnimatedLayerProps. Tests: scene-model unit tests (text layer present, no asset id required); rasterizer cache hit test; visual snapshot of a styled text frame. Acceptance: a text clip with pop in-animation renders in preview and export; npm run check green. T10 — Text clip authoring: agent tool + inspector · Sonnet 5 Depends on: T9, T5, T6. Do: Handler + tool ui_timeline_add_text_clip — { text, trackId?, startMs?, durationMs?, style? }, defaulting onto an overlay track (create one if none, same fallback logic as generateClip’s track selection in useTimelineAgentBridge.ts), default duration 3000 ms. Extend ui_timeline_set_clip_params (or the snapshot + a textStyle patch on setClipParams) so the agent can restyle existing text clips. Inspector: text section (content textarea, font size, weight, color, align) for mediaType === "text" clips, same primitives/tokens rules as T6. AddClipMenu.tsx: manual “Text” entry. Tests: tool test (adds clip on overlay track with defaults); RTL inspector test (edit patches textStyle). Acceptance: agent flow works end-to-end: ui_timeline_add_text_clip → ui_timeline_animate_clip (pop in, float loop) → ui_timeline_get_clip_frames shows animated text; npm run check green. Sequencing summary Order Task Model Parallelizable with 1 T1 animation core Opus 4.8 — 2 T2 protocol schema Haiku 4.5 T3, T4 2 T3 render integration Opus 4.8 T2, T4 2 T4 store + edit ops Sonnet 5 T2, T3 3 T5 agent tools Sonnet 5 — 4 T6 inspector Sonnet 5 T7 4 T7 clip chips Haiku 4.5 T6 5 T8 verification Sonnet 5 — 6 T9 text clips Opus 4.8 — 7 T10 text authoring Sonnet 5 — Ship gate for Milestone 1+2: T8 green. Text (M3) is separately mergeable.--- # Motion Design for the Timeline — Technical Design Source: https://docs.nodetool.ai/plans/motion-design.html Motion Design for the Timeline — Technical Design Status: implemented (July 2026). Companion doc: motion-design-tasks.md (implementation plan, agent-consumable tasks). Goal Jitter-style motion design inside the existing timeline editor: clips animate — slide in, pop, drift, loop — without the user placing keyframes. The authoring unit is a named animation preset attached to a clip (fadeIn, slideIn, kenBurns, …) with duration, delay, easing, and a few parameters. The primary authoring surface is the agent: a user says “make the title pop in and float”, the agent applies animations via ui_timeline_* tools, previews the result frame-by-frame, and adjusts. Manual UI (inspector panel, clip badges) is a thin layer over the same data. Presets compile to an internal keyframe/curve form evaluated by one sampler, so a curve editor or per-keyframe tools can be added later without a schema break. Implementation baseline The timeline supports preset motion on visual clips, including authored text and shapes. The table below records the baseline that this design extended. Concern Where it lives today Data model packages/timeline/src/types.ts — TimelineSequence, TimelineTrack, TimelineClip. Clips carry a static transform?: ClipTransform (position px from canvas center, scale multiplier on contain-fit, rotation radians, anchor normalized), opacity, effects[], transitionIn (crossfade), fadeInMs/fadeOutMs (audio-only, applied in web/src/components/timeline/preview/AudioGraph.ts). Scene resolution web/src/components/timeline/preview/sceneModel.ts — pure computeActiveLayersWithHorizon(tracks, clips, currentTimeMs) returns ActiveLayer[] plus nextChangeMs, the change horizon that lets the rAF loop skip recomputation between boundaries. Live preview web/src/components/timeline/preview/PreviewCompositor.tsx + preview/gpu/ (WebGPU with Canvas2D fallback; affine math in preview/gpu/transform.ts). Export web/src/components/timeline/render/TimelineRenderer.ts — steps in exact 1/fps increments, composites via the same scene model and GPU compositor, encodes MP4 via WebCodecs (mediabunny). Server ffmpeg path (packages/video-nodes/src/nodes/timeline.ts) is an explicit rough cut and stays one. Edit ops packages/timeline/src/splitClip.ts, trimClip.ts (pure); Zustand web/src/stores/timeline/TimelineStore.ts (patchClip(clipId, patch)), temporal undo/redo. Persistence TimelineDocument JSON blob in timeline-sequences (packages/models/src/timeline-sequence.ts); zod wire schema packages/protocol/src/api-schemas/timeline.ts. Fields absent from the zod schema are stripped on PATCH. Agent surface web/src/lib/tools/builtin/timeline.ts (13 ui_timeline_* tools) → TimelineAgentHandler interface in web/src/components/timeline/timelineAgentBridge.ts, implemented by web/src/hooks/timeline/useTimelineAgentBridge.ts. ui_timeline_get_clip_frames already lets the agent see rendered frames. Data model New module: packages/timeline/src/animation/ (pure, no DOM/GPU/store access), re-exported from packages/timeline/src/index.ts. // packages/timeline/src/animation/types.ts export type AnimationRole = "in" | "out" | "emphasis" | "loop"; export type EasingId = | "linear" | "easeIn" | "easeOut" | "easeInOut" // cubic | "easeOutBack" // overshoot (pop) | "easeOutElastic" | "easeOutBounce"; export interface ClipAnimation { id: string; // crypto.randomUUID() at creation role: AnimationRole; /** An AnimationPresetId. Typed `string` on purpose: documents saved by a * newer client may carry ids this build doesn't know — they parse fine and * are skipped at compile time (forward compat). Tool inputs and the UI * constrain to the catalog union. */ preset: string; /** Animation length in ms. For "loop", the period of one cycle. */ durationMs: number; /** * "in" | "emphasis" | "loop": offset from clip start to window start. * "out": offset from clip END back to window end (0 = ends exactly at clip end). * Default 0. */ delayMs?: number; easing?: EasingId; // default per preset /** Default true. Disabled animations are kept but not evaluated. */ enabled?: boolean; /** Preset-specific knobs; unknown keys ignored. See catalog. */ params?: Record<string, number | string | boolean>; } TimelineClip gains one optional field (after transitionIn): /** Motion-design animations evaluated at render time. Evaluation is * order-independent (the fold is commutative — see animation/sample.ts); * array order is presentation order in the UI only. */ animations?: ClipAnimation[]; Compiled form (internal substrate) Presets never render directly; they compile to curves. This is the extension point for a future curve editor and for baked custom motion. // packages/timeline/src/animation/compile.ts export type AnimatedProperty = | "offsetX" | "offsetY" // canvas px, resolved from normalized preset units | "scale" // uniform multiplier on ClipTransform.scale | "rotation" // radians, added to ClipTransform.rotation | "opacity" // multiplier on the layer's resolved opacity | "wipeProgress" // 0 hidden → 1 revealed (drives CompiledAnimation.mask) | "blur" // source px, added to the layer's blur radius | "brightness" // -1..1, added to the color grade's brightness term | "saturation"; // 0..4, multiplies the color grade's saturation export interface Keyframe { t: number; value: number; easing?: EasingId; } // t normalized 0..1 within the animation window; keyframes sorted by t; // first t === 0, last t === 1. export interface PropertyCurve { property: AnimatedProperty; keyframes: Keyframe[]; } export interface CompiledAnimation { role: AnimationRole; windowStartMs: number; // clip-local, resolved from role + delay + duration windowEndMs: number; loop: boolean; // repeat the window until clip end. Set by the // preset's compile, not derived from role alone: // most "loop"-role presets compile loop:true, but // kenBurns compiles loop:false (one-shot full clip) /** Hold curve endpoint values outside the window (in: hold t=0 before window; * out: hold t=1 after window). false → identity outside the window. */ holdBefore: boolean; // true for "in" holdAfter: boolean; // true for "out" curves: PropertyCurve[]; } /** Pure. canvas = sequence {width, height}; clipDurationMs for window math. */ export function compileClipAnimations( animations: ClipAnimation[], clipDurationMs: number, canvas: { width: number; height: number } ): CompiledAnimation[]; Preset position units are normalized (fraction of canvas width/height) inside the catalog; compileClipAnimations resolves them to px so the sampler and GPU stay unit-free. Sampler // packages/timeline/src/animation/sample.ts export interface AnimationSample { offsetX: number; // px, add to transform.position.x offsetY: number; scale: number; // multiply transform.scale.x and .y rotation: number; // radians, add to transform.rotation opacity: number; // 0..1, multiply layer opacity } export const IDENTITY_SAMPLE: AnimationSample; // {0, 0, 1, 0, 1} /** Pure. localMs = currentTimeMs - clip.startMs. */ export function sampleAnimations( compiled: CompiledAnimation[], localMs: number ): AnimationSample; Composition rules (deterministic, order-independent across roles): Evaluate every compiled animation independently, then fold: offsets and rotation add, scale and opacity multiply. After the fold, clamp the resulting opacity to [0,1] — overshoot easings (easeOutBack, easeOutElastic) may push individual samples past 1. Scale is clamped to ≥ 0. Within a window: map localMs to normalized t, find the bracketing keyframes, apply the segment’s easing (the easing stored on the right keyframe of the segment; ClipAnimation.easing overrides every segment), lerp. "in" before its window: hold the t=0 value — a delayed fadeIn keeps the clip invisible during the delay. After the window: identity. "out" after its window: hold the t=1 value (clip stays out). Before: identity. "emphasis" outside its window: identity. "loop": identity before windowStartMs; from there, t = ((localMs - start) % durationMs) / durationMs until clip end. Loop curves must start and end at the same value to avoid pops (catalog invariant, asserted in tests). Speed (speedMultiplier) does not rescale animation time: animations run on the timeline clock, not the source clock. (Matches how fades and crossfades behave today.) Preset catalog v1 packages/timeline/src/animation/presets.ts. Each entry: id, allowed roles, default duration/easing, params with defaults, and a curve generator (params, canvas, role, durationMs?) => PropertyCurve[]. durationMs is the animation’s own window length, for a preset whose motion depends on it (a noise fade envelope, arc-length sample density) — optional, so a preset that ignores it keeps a shorter signature. AnimationPresetId is the union of ids. sampleCountFor(durationMs, frequencyHz, {min, max}) is the shared sample density rule (~8 keyframes per cycle, capped at MAX_CUSTOM_KEYFRAMES) that shake, float, and followPath all use instead of a fixed count. Preset Roles Params (defaults) Motion fade in, out — opacity 0↔1 slide in, out direction: "left"\|"right"\|"up"\|"down" (“left”), distance (0.3 = fraction of canvas) offset from/to direction, opacity 0↔1 pop in, out overshoot (1.08) scale 0.6↔1 with easeOutBack, opacity 0↔1 spin in, out turns (0.25) rotation ±turns·2π, opacity 0↔1 wipe in, out direction: "left"\|"right"\|"up"\|"down" (“left”; the edge the reveal starts from — “up” = top, “down” = bottom), softness (0.05 = feather width as fraction of the wipe axis, 0 = hard edge) directional mask reveal. Added after v1 with per-layer compositor masking: the preset compiles a wipeProgress curve (0 = hidden, 1 = revealed), direction/softness ride on CompiledAnimation.mask, and AnimationSample.mask carries the resolved mask to both compositors. Overlapping wipes fold by keeping the smaller progress (more hidden wins). The mask lives in the layer’s own quad space, so the wipe edge rotates with the clip blur in, out amount (12 = blur radius in source px, 0..40) blur amount→0 with an opacity fade (rack focus); reversed for out colorFade in, out — saturation 0→1 (grayscale → full color); reversed for out pulse emphasis intensity (0.06) scale 1→1+i→1 flash emphasis intensity (0.6, 0..1) brightness 0→+i→0 (brief exposure spike) shake emphasis intensity (0.02, fraction of canvas width/height), frequency (8 Hz), seed (1), fadeInMs/fadeOutMs (0) seeded value noise (mulberry32, deterministic per seed) on both offsetX and offsetY, amplitude-enveloped by fadeInMs/fadeOutMs; the curve’s own first/last keyframes are always forced to 0 regardless of fade settings bounce emphasis height (0.05) offsetY dip with easeOutBounce kenBurns loop zoom (0.12), direction (“in”), driftX/driftY (0.02) slow scale 1→1+z (or reverse) + drift. Compiles loop:false, window = full clip, holdAfter:true — it runs once over the whole clip, so durationMs and delayMs are ignored and the monotonic curve is exempt from the loop start==end invariant float loop amplitude (0.015), frequency (1 cycle/period), seed (0 = pure sine) offsetY sine bob (loops seamlessly — the sine always closes on a whole cycle regardless of a fractional frequency); a nonzero seed blends in seeded circular value noise as organic drift without breaking the loop breathe loop intensity (0.03) scale sine rotate loop rpm via durationMs rotation 0→2π per cycle followPath emphasis, loop d (SVG path, normalized 0..1 like ClipShapeStyle.d), pathX/pathY/pathWidth/pathHeight (0,0,1,1 — normalized placement box), orient (false), startT/endT (0, 1) walks d by arc length from startT to endT, writing positionX/positionY (replace, canvas px) and, when orient is set, rotation from the path’s tangent. Takes the path literally — resolving an existing shape clip’s d into these params (pathClipId) is a follow-up op, since a preset compiles without the document to look one up in. Geometry: pathSampling.ts, reusing render/shapeGeometry.ts’s flattener Defaults: in/out 500 ms, emphasis 600 ms, loop 3000 ms. in defaults to easeOut, out to easeIn, others per table. blur, flash, and colorFade animate GPU effect parameters. They drive blur/brightness/saturation curves that the sampler folds like the effect pre-pass aggregates (blur and brightness add, saturation multiplies). At the render site resolveAnimatedLayerProps composes them with the clip’s static effects: animated blur adds to the static blur radius, animated brightness adds to the grade’s brightness term, animated saturation multiplies the grade’s saturation — emitted as synthesized ClipEffects so both compositors reuse the existing color-grade / Gaussian-blur pipeline, keeping preview/export parity. Rendering integration The layer set stays cacheable; animated properties are resolved per frame. computeActiveLayersWithHorizon and its change-horizon contract are untouched — animation windows never add or remove layers, so nextChangeMs stays correct for set membership. New pure helper (in sceneModel.ts, since it is shared preview/export vocabulary): export interface AnimatedLayerProps { transform?: ClipTransform; opacity: number; } /** Compose a layer's static transform/opacity with its animation sample at t. * Returns the inputs unchanged when the clip has no enabled animations. */ export function resolveAnimatedLayerProps( layer: ActiveLayer, currentTimeMs: number, canvas: { width: number; height: number }, cache?: AnimationCompileCache // keyed by clip id + animations ref ): AnimatedLayerProps; Call sites (the only two, keeping 1:1 preview/export parity): Live preview — PreviewCompositor.tsx, where each ActiveLayer’s transform/opacity is handed to the GPU layer list. While any active layer has an enabled animation whose window (or loop tail) covers the current time, the compositor must redraw every rAF tick even when paused- idle short-circuits would otherwise skip drawing — expose hasActiveAnimation(layers, timeMs) next to the resolver for that check. Export — render/TimelineRenderer.ts, same resolver per stepped frame. No other change: WebCodecs encode path is agnostic. Composition into ClipTransform (identity when clip has none: position {0,0}, scale {1,1}, rotation 0, anchor {0.5,0.5}): position.x += sample.offsetX scale.x *= sample.scale position.y += sample.offsetY scale.y *= sample.scale rotation += sample.rotation opacity *= sample.opacity sample.opacity multiplies the already-resolved layer opacity (base × crossfade), so animations compose with transitionIn rather than fighting it. Caption layers (kind: "caption") get the same treatment — they carry the clip’s transform already. Compilation is memoized per clip (WeakMap on the animations array ref, or a Map<clipId, {ref, compiled}>) — the per-frame cost is sampling only: a handful of lerps per animated layer, no allocation in the steady state (sampler writes into a scratch AnimationSample). Persistence TimelineDocument is a JSON blob — no DB migration. Old documents lack animations; every consumer treats undefined as “no animations”. Required: add animations to the clip zod schema in packages/protocol/src/api-schemas/timeline.ts (see the existing storyboardBoardId comment in types.ts: fields missing from the schema are stripped by PATCH). Use a permissive z.array(clipAnimation) with preset: z.string() (not the enum) so older servers don’t reject documents from newer clients; unknown presets are skipped at compile time with a console warning. Autosave/undo need no changes: patchClip flows through the temporal middleware like any clip field. Editing semantics Implemented where the pure ops live (packages/timeline/src/): splitClip: left half keeps role: "in" animations, right half keeps "out"; "emphasis" and "loop" are copied to both halves (loop windows re-derive naturally from each half’s duration). Ids on the right half are regenerated. trimClip / duration changes: windows are clamped at sample time (compileClipAnimations clamps windowEndMs to clipDurationMs, and drops a window whose start ≥ clip duration), so trims need no data rewrite. When in+out overlap because the clip got too short, both still evaluate; fold rules keep the result continuous. duplicateClip: copies animations with fresh ids. Linked clips / speed / bake: no interaction — animations live on the timeline clock. Agent surface Extend TimelineAgentHandler (web/src/components/timeline/timelineAgentBridge.ts, implemented in web/src/hooks/timeline/useTimelineAgentBridge.ts) and register three tools in web/src/lib/tools/builtin/timeline.ts: ui_timeline_animate_clip — { target, animations: ClipAnimationInput[], mode?: "add" | "replace" } (replace default; add appends). Each input: { role, preset, durationMs?, delayMs?, easing?, params? } — ids and defaults filled in by the handler. Returns the updated clip including resolved animations. On validation errors (unknown preset, role not allowed for the preset) the handler throws with a message listing the valid options — the tool layer surfaces handler throws back to the agent, same as every existing ui_timeline_* tool. ui_timeline_clear_animations — { target, role? }. ui_timeline_list_animation_presets — no args; returns the catalog (id, roles, params with defaults and ranges, one-line description, default duration/easing). This is the runtime vocabulary discovery; tool descriptions carry a compact summary so simple requests skip the extra call. Also: include animations in getSnapshot() clip entries so ui_timeline_get_state shows current motion. The critique loop needs no new tools: ui_timeline_get_clip_frames + ui_timeline_seek already exist. Tool descriptions should state the intended loop: get state → animate → get frames at window boundaries → adjust. Manual UI (thin, after the agent surface) Inspector: an “Animate” section in web/src/components/timeline/Inspector/ (pattern: ClipAdjustments.tsx, primitives from InspectorPrimitives.tsx). Per role: preset select, duration, delay, easing, preset params. ui_primitives only; SPACING/TYPOGRAPHY/MOTION tokens (see docs/DESIGN.md). Clip chips: in web/src/components/timeline/Tracks/Clip.tsx, small in/out zone markers (left/right wedge spanning the window width at current zoom) and a loop glyph — read-only affordance, click selects the clip and opens the inspector section. Phase 2 — text and shape clips Jitter is mostly motion typography; this design animates whatever a clip can draw, so text arrives as a new clip kind, not an animation change: mediaType: "text" on TimelineClip + textStyle?: ClipTextStyle (content, font family/size/weight, fill, alignment, max width fraction). Rendered as a rasterized layer: reuse the caption rasterization approach (preview/captionRender.ts) — draw to an offscreen canvas at sequence resolution × device scale, hand the bitmap to the compositor as an image source keyed by a style hash. Animation then applies for free via the same resolveAnimatedLayerProps. Agent tool ui_timeline_add_text_clip. Shapes (rect/ellipse/line) follow the same rasterize-to-layer pattern later. Non-goals Lottie/GIF export — MP4 via the existing WebCodecs path only. Curve editor UI / dope sheet — the compiled-curve substrate exists for it, but no UI in this effort. Server-side render parity — the ffmpeg rough cut ignores animations, exactly as it ignores effects and transitions today. Per-property keyframe authoring by agents — presets + params only until real demand appears. Testing packages/timeline/src/animation/__tests__/ (Vitest): easings (endpoint + monotonicity), window math for every role incl. delays and clamping, sampler fold rules, loop wraparound and start==end invariant, split/trim semantics, compile determinism. web/src/components/timeline/preview/__tests__/ (Jest): resolveAnimatedLayerProps composition against hand-computed frames; hasActiveAnimation horizon behavior; export/preview parity by sampling both paths at the same t. Visual: one debug-harness / Playwright pass rendering a seeded sequence with each preset at t = window midpoints, snapshot-compared (existing web/tests/ visual patterns). Risks Skipped redraws during playback: the preview skips redraws while the cached layer set is valid (nextChangeMs horizon). With animations, any tick where the playhead moved must redraw even though the set is cached, or motion freezes mid-clip. A paused playhead needs only one draw — samples are pure functions of time. Caught by the T8 visual test and manual check, not by unit tests. Schema strip: forgetting the protocol zod field silently loses animations on save. Caught by a round-trip test in the protocol package. Perf: per-frame sampling is O(active animated layers × curves); with memoized compilation this is negligible next to video decode. Do not compile in the rAF loop.--- # Motion Graphics — Implementation Plan Source: https://docs.nodetool.ai/plans/motion-graphics-tasks.html Motion Graphics — Implementation Plan Companion to motion-graphics.md (the technical design). Every task below is written to be executed by an agent with no prior conversation context. Each task’s first step is: read docs/plans/motion-graphics.md in full. It is the contract; this file adds sequencing, file-level pointers, and acceptance criteria. Finding codes (F1…), assumptions (AS), invariants (I), constraints (C) and decisions (D) refer to that document. Ground rules for every task Repo root is a TypeScript monorepo. nvm use && npm install if node_modules is missing (sandboxed environments: npm install --ignore-scripts, see AGENTS.md § Install in sandboxed environments). Work on a branch named motion/<task-id>-<slug>. One task per PR. Title the PR feat(timeline): <task title> [<task-id>] and list the finding codes the PR resolves in its body. After changes, run and paste the results into the PR: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base main A task that adds a capability also runs npm run capabilities:sync and npm run capabilities:check (I10). Schema mirror (I1). Any field you add to packages/timeline/src/types.ts must exist in packages/protocol/src/api-schemas/timeline.ts and be covered by the round-trip test in packages/execution/tests/ (T1 creates it). If T1 has landed, the field already exists; do not re-add it. packages/timeline root stays dependency-free (AS2). Render code lives under src/render. packages/agents imports @nodetool-ai/timeline/scene only (C5). Every new check proves it can fail (I12). A validator code ships with a fixture that triggers it; an eval predicate ships with a state that fails it. Web UI: primitives from web/src/components/ui_primitives/ only, tokens per docs/DESIGN.md. Never import raw MUI. Prose in docs and skills follows docs/WRITING_STYLE.md. Do not widen a task. If you find an adjacent bug, file it in the PR body under “Found, not fixed” and move on. Commit messages: conventional, imperative, body says why. Model key Opus — design-sensitive core: scene model, compositors, schema landing, agent contracts, skill prose. Sonnet — well-specified implementation against a landed schema. Every task below is written for Opus unless marked otherwise; the ones marked Sonnet-capable have an exact spec and no open design choice. Dependency graph M0 T1 ─┬─► T2 T3 T4 T5 (T2–T5 parallel, need T1 only for T4) M1 T1 ─┴─► T6 ──► T7 ──► T8 M2 T7 ──► T9 ──► T10 T9 ──► T11 T9 ──► T12 T7 ──► T13 M3 T1 ──► T14 T15 T16 T17 (independent) M4 T4+T9+T11+T12+T13 ──► T19 T1 ──► T18 T20 T9+T18 ──► T21 T3 ──► T22 T22 ──► T23 T19+T21+T22+T23 ──► T24 ──► T25 T19 ──► T26 M5 T3 ──► T27 T7 ──► T28 T7 ──► T29 T9..T16 ──► T30 T7..T16 ──► T31 Serial chain through the compositors (R4): T7 → T9 → T10 → T11 → T12 → T13 → T28 each touch sceneModel.ts, canvas2d.ts and frameCompositor.ts. Run them one at a time, rebasing on the previous merge. Everything else can run in parallel with that chain. Files shared by many tasks, land early: packages/timeline/src/types.ts and packages/protocol/src/api-schemas/timeline.ts (T1 only), the validator (T30 collects codes; feature tasks may add their own code when the fixture is trivial, and T30 reconciles), timelines.specs.ts (T18–T23 each add specs; rebase conflicts are additive). Milestone 0 — Unblock and correct T1 — Schema landing · Opus Resolves: prerequisite for every field-adding task. Decision D1. Goal: add every new optional field named in motion-graphics.md § Document model to the TypeScript types and the Zod schema in one PR, with no behavior change and one round-trip test per field. Read first: packages/timeline/src/types.ts, packages/protocol/src/api-schemas/timeline.ts (note the “Without this field Zod strips it” comments; they are the convention), packages/execution/src/timeline-debug/validate.ts (checkFieldStripping). Change: types.ts: parentId, mask: ClipMask, matte: ClipMatte, the widened ClipTransition union, the widened ClipEffect union, timeRemap, compositionId, compositionParams, ClipTextStyle additions, ClipShapeStyle additions, ShapeFill, CaptionStyle on ClipCaption, mediaType: "group". Mirror every new type into the Zod file as a z.object with the same optionality, with a one-line comment on each field that would otherwise be stripped. ANIMATED_PROPERTIES in animation/types.ts: add the channel names from the design (scaleX, scaleY, contrast, hue, temperature, tint, positionX, positionY, anchorX, anchorY, trimStart, trimEnd). Add a parallel ANIMATED_PROPERTY_FOLD: Record<AnimatedProperty, "add" | "multiply" | "replace" | "min">. Do not change sample.ts (T7 does). Do not add behavior. Compilers, samplers, compositors and the validator must ignore the new fields. Tests: packages/execution/tests/timeline-schema-roundtrip.test.ts: build one document that sets every new field to a non-default value, parse through timelineDocument, and assert checkFieldStripping returns no issue. Then delete one field from the Zod schema in a vi.mock and assert the test fails (prove the check bites, I12), restore. packages/protocol type test that z.infer<typeof timelineClip> is assignable to TimelineClip and back for the new fields. Acceptance: typecheck, lint, npm run test --workspace=packages/timeline, --workspace=packages/protocol, --workspace=packages/execution green; the round-trip test enumerates every field in the design’s model section (list them in the PR body as a table: field, TS line, Zod line). T2 — F16 caption fields and F17 dropped-layer report · Sonnet Resolves: F16, F17. Change: Zod captionWord gains kind: z.enum(["word","filler","pause"]).optional() and confidence: z.number().min(0).max(1).optional(). computeActiveLayersWithHorizon returns droppedLayers: { clipId: string; reason: "video_layer_cap" }[] in ActiveLayersResult. preview_timeline_frame (packages/agents/src/timeline-preview/frames.ts) surfaces them per frame under dropped. The validator emits layer_cap_exceeded (warning) when any instant has more than MAX_VIDEO_LAYERS simultaneous video clips; compute from clip windows, not by sampling. Tests: round-trip fixture with kind: "filler" survives parse (packages/execution/tests/); a nine-video-clip fixture reports one dropped layer from the scene model and one layer_cap_exceeded from the validator. Acceptance: web/src/stores/timeline/transcriptOps.ts filler removal works after a save round trip (add a store test). T3 — Server render through Dawn on every profile · Opus Resolves: F10, F14 (progress and cancel half). Decision D9. Read first: scripts/bundle-backend.mjs (DESKTOP_ONLY_EXTERNAL_PACKAGES, REQUIRED_EXTERNAL_PACKAGES), scripts/verify-backend-bundle.mjs (requireWebgpu), Dockerfile (mesa-vulkan-drivers block), packages/gpu/src/node.ts (instance retention, R2), packages/video-nodes/src/nodes/timeline.ts, packages/video-nodes/src/nodes/timeline/compositeRender.ts, .github/workflows/quality-checks.yml (the docker leg, scripts/docker-smoke.mjs). Change: Move webgpu from DESKTOP_ONLY_EXTERNAL_PACKAGES to the common required list; verifyBackendBundle requires it on both profiles. Update the comments in both scripts and the Dockerfile that say the server ships without it, and the header of timeline.ts. RenderTimelineNode.process: pass onProgress that posts { type: "node_progress", node_id: this.id, progress: frame, total: totalFrames } through context.postMessage at most four times a second; pass the run’s abort signal into renderTimelineComposited, which checks it once per frame and throws an AbortError. When the compositor is unavailable, log a warning level job log naming the fallback and the reason, and set output.metadata.render_mode = "rough_cut"; on the composited path set "composited". scripts/docker-smoke.mjs: after the health check, POST a one-node RenderTimeline workflow with a two-clip fixture (a text clip over a solid shape) and assert the job’s render_mode is "composited". This is the check that proves the image renders. Tests: packages/video-nodes/tests/timeline-render.test.ts runs renderTimelineComposited on a fixture with lavapipe (VK_DRIVER_FILES, see AGENTS.md § WebGPU on a headless machine) and asserts progress callbacks arrive in order and an aborted signal stops before totalFrames. Skip only when no adapter exists, and print why. Acceptance: the Quality Gate docker leg passes with the new smoke assertion. npm run backend:smoke passes on the server profile. T4 — Custom animation op for agents · Opus Resolves: F19. Decision D2. Read first: packages/timeline/src/animation/custom.ts, packages/agents/src/custom-animation-bake.ts, packages/timeline/src/ops/apply.ts (animate_clip), packages/agents/src/capabilities/timeline-bridge.ts (host baking), packages/agents/src/capabilities/timelines.ts (edit_timeline), packages/agents/src/capabilities/timelines.specs.ts, docs/timeline-custom-animations.md. Change: Bridge animate_clip accepts preset: "custom" with exactly one of curves (inline {property, keyframes:[{t, value, easing?}]}[]) or code (a JS body). With curves, run normalizeCustomCurves and store custom: {curves, bakedAt}. With code, call bakeCustomAnimation with the clip’s window and canvas size, store custom: {code, curves, bakedAt}. Any other preset keeps today’s catalog check. mask is required when a curve drives wipeProgress, else refused with the message from custom.ts. The browser ui_timeline_animate_clip schema (web/src/lib/tools/builtin/timeline.ts) and the headless edit_timeline op schema (timelines.specs.ts) both gain curves and code. list_animation_presets lists custom with its input contract and the ANIMATED_PROPERTIES names and ranges. Extend docs/timeline-custom-animations.md with the agent path. Tests: packages/agents/tests/timelines-op-input.test.ts cases for curves accepted, code baked, both refused, neither refused, unknown property refused with the property list in the message. timeline-tool-loop eval case keyframed-slide (deterministic assertions on the stored curves). Acceptance: an edit_timeline call with a two-keyframe opacity curve renders through preview_timeline_frame at mid-curve with the interpolated opacity (add to capabilities-timeline-preview.test.ts, read the pixel). T5 — Frame preview in the CodeAct object model · Sonnet Resolves: F22 (namespace half). Change: packages/agents/src/codeact/nodetool-api.ts: add "preview_timeline_frame" to the timelines namespace list and timelines.preview(target, opts) to the object model, taking an id or an inline document like validate. Update the generated typings and the sandbox manifest (packages/agents/src/code-gen/sandbox-manifest.ts) where the namespace is documented. Tests: the existing object-model test that enumerates namespaces; add preview to it. Milestone 1 — Animation depth T6 — Easing grammar: cubic bezier and spring · Opus Resolves: F2. Decision D3. Read first: packages/timeline/src/animation/easing.ts, animation/types.ts (EasingId), animation/compile.ts (where easing is applied per segment), packages/timeline/tests/animation.easing.test.ts. Change: easing.ts: parseEasing(id: string): EasingFn | null. Accepts the seven named ids, cubic-bezier(x1,y1,x2,y2) (solve x by Newton with bisection fallback, 1e-4 tolerance, x1 and x2 clamped to [0,1]), and spring(stiffness,damping,mass) (closed-form damped oscillator normalized so f(0)=0, f(1) within 1e-3 of 1; overshoot allowed). ease(id, t) keeps its signature and delegates. Parsing is memoized per string. Keyframe-level easing on custom curves and preset segments accept the grammar. normalizeCustomCurves validates easing strings and reports the offending one. Validator code unknown_easing (warning) for a string that does not parse; the sampler falls back to linear (I2). Preset catalog: pop’s default easing becomes spring(180,12,1) only if the existing animation.wipe/pop tests still pass at their sampled points; otherwise leave the catalog alone and note it. Tests: analytic anchors: cubic-bezier(0,0,1,1) equals linear at ten points; cubic-bezier(0.42,0,0.58,1) matches easeInOut within 1e-3; spring with critical damping never overshoots; malformed strings return null. Fixture triggering unknown_easing. T7 — Channels and fold metadata · Opus Resolves: F1 (channel half). Serial chain start (R4). Read first: animation/sample.ts (the fold), animation/compile.ts, render/sceneModel.ts (resolveAnimatedLayerProps, composeAnimatedEffects), render/transform.ts (buildTransformMatrix), render/canvas2d.ts, render/frameCompositor.ts, render/effects.ts. Change: sample.ts: fold by ANIMATED_PROPERTY_FOLD instead of the property switch. replace channels take the last enabled animation in document order; record replacedBy so the validator can warn on overlap. resolveAnimatedLayerProps: scaleX/scaleY multiply the respective axis (existing scale multiplies both); positionX/Y replace transform.position; anchorX/Y replace transform.anchor; contrast/hue/temperature/tint fold into the synthesized color effect with identities matching ClipColorEffect defaults; trimStart/trimEnd land on shapeStyle for the shape rasterizer (T16 draws them; until then they are carried and ignored). Presets: keep the existing catalog unchanged. Add two presets that exercise the new channels: squash (emphasis, scaleX up while scaleY down and back, easeOutBack) and hueShift (loop, hue 0→360). Document them in the catalog. transform.ts: buildTransformMatrix accepts non-uniform scale. Tests: animation.sample.test.ts fold cases per kind; a replace overlap case; render.transform.test.ts non-uniform scale; canvas2d pixel test that scaleX: 2 doubles a shape’s width and not its height; preview parity test in packages/agents/tests/capabilities-timeline-preview.test.ts. T8 — Character and line stagger · Sonnet Resolves: F4 (stagger half). Read first: docs/plans/motion-typography.md, animation/compile.ts (stagger span), render/sceneModel.ts (resolveTextStaggerContext, clipStaggerCount), render/draw.ts (drawStaggeredText), packages/timeline/tests/animation.stagger.test.ts. Change: unit: "character" counts grapheme clusters (Intl.Segmenter with a code-point fallback) per wrapped line and draws per glyph with measureText advances; unit: "line" counts wrapped lines. The span-compression rule stays and now reports compressed: true from the compiler so T30 can warn. Whitespace characters are counted for timing and drawn nothing. Tests: a five-character word with offsetMs: 100 produces five windows 100 ms apart; a two-line clip with unit: "line" produces two; an emoji grapheme counts once. Rasterizer snapshot at mid-stagger through canvas2d. Milestone 2 — Structure T9 — Groups with transform inheritance · Opus Resolves: F5 (parenting half). Decision D4. Serial chain. Read first: render/sceneModel.ts end to end, render/transform.ts, both compositors, packages/timeline/src/splitClip.ts, trimClip.ts, linked.ts (how linked clips move together), the web store web/src/stores/timeline/TimelineStore.ts (patchClip, move/trim actions). Change: Scene model: resolve group clips first into resolvedGroups: Map<clipId, {matrix, opacity, window}> (a group may itself have a parent; resolve depth-first, refuse cycles). A child layer outside its parent’s window is not emitted. ActiveLayer gains parentMatrix?: Mat4 and the emitted opacity already includes the parent’s. buildTransformMatrix(layer, canvas, parentMatrix?) composes parent × own. Both compositors pass it through; no other change to them. Pure ops in packages/timeline/src: moveClip of a group moves its children by the same delta; deleteClip of a group unparents children (does not delete them); splitClip refuses a group (message: split the children); trimClip on a group clamps children to the new window. Web store: route move/delete/trim of a group through those ops. Groups render on the track lane as a bracket over their children’s span; no inspector work beyond a read-only “Parent” row. Validator codes parent_missing, parent_cycle, parent_not_group. Tests: scene-model tests for composition order (parent × child position, rotation about the parent’s anchor), window clipping, nested groups, cycle refusal; op tests for move/delete/trim; canvas2d pixel test that rotating a group rotates a child about the group’s anchor. T10 — Precomposite: group effects and blend · Opus Resolves: F5 (precomp half). Serial chain. Read first: T9’s scene model, render/frameCompositor.ts (HeadlessFrameCompositor), packages/gpu/src/compositor/compositor.ts (WebGPULayerCompositor), render/canvas2d.ts. Change: a group with effects or a blendMode other than normal composites its children into an intermediate frame-sized texture (GPU: a second WebGPULayerCompositor pass into an offscreen texture; 2D: an OffscreenCanvas or @napi-rs/canvas canvas through RasterContext2D’s factory hook, added to the interface), runs the group’s effect chain on it, then blends the result once at the group’s z. Children of such a group are removed from the main stack. Without those fields nothing changes (T9 path). Tests: a group with opacity: 0.5 over two overlapping children reads as one 50% layer (no double darkening in the overlap) on both compositors; a group blur blurs the composite, not each child; unsupported group effects on 2D are reported through unsupportedEffectTypes (I7). T11 — Two-clip transitions · Opus Resolves: F6. Decision D5. Serial chain. Read first: render/sceneModel.ts (crossfadeOpacity, the auto-dissolve rule), the motion-graphics skill’s transition section, both compositors’ wipe handling, animation/sample.ts (AnimationSampleMask). Change: resolveTransition(clip, prevOnTrack, timeMs) returns {type, progress (eased), role} for the incoming clip and, when a partner exists, the complementary record for the outgoing clip. ActiveLayer.transition carries it. crossfadeOpacity is folded into it; the auto-crossfade on overlap stays the default. Draw rules per type, identical on both compositors: dipToColor draws a full-frame solid whose opacity peaks at progress 0.5 while both clips fade; wipe masks the incoming layer with the existing feathered wipe and leaves the outgoing beneath; push offsets outgoing by -progress × frame and incoming by (1-progress) × frame along direction; slide moves only the incoming; zoom scales outgoing up and incoming from 0.8 with a crossfade. Bridge op set_transition ({target, transition | null}); the browser ui_timeline_set_clip_params refusal of transitionIn stays, the new op replaces it. transition_exceeds_duration covers the new types. Validator unknown_transition (I2). Tests: per-type scene-model tests for both roles at progress 0, 0.5, 1; canvas2d pixel tests for push (a red-over-blue frame is half red, half blue at 0.5) and dipToColor (mid-frame is the color); preview parity. T12 — Masks and track mattes · Opus Resolves: F7. Decision D6. Serial chain. Read first: packages/gpu/src/shaders/mask/{apply,fromImage,invert}, render/effects.ts (how a shader step is invoked), render/draw.ts, render/canvas2d.ts wipe mask path (destination-in gradient), T9/T10 offscreen hooks. Change: draw.ts: drawMask(ctx, mask, w, h) rasterizes rect | ellipse | path in normalized layer space with featherPx (2D: blur through a gradient ring for rect/ellipse, ctx.filter blur for path where available) and invert. Path data parsed by a small SVG path parser in packages/timeline/src/render/svgPath.ts (M, L, C, Q, Z, absolute and relative; anything else is mask_path_invalid). GPU: the mask raster is uploaded as a texture and applied with maskApplyV1 before the layer blend. 2D: ctx.clip for hard masks, offscreen destination-in for feathered. Mattes: a layer with matte receives matteLayer from the scene model (source removed from the stack); the compositor renders the source to an offscreen texture and applies it as alpha or luma via maskFromImageV1 (2D: destination-in, luma converted through a grayscale pass). Bridge ops set_mask, set_matte. Validator matte_source_missing, mask_path_invalid. Tests: ellipse mask on a solid reads transparent at the corners and opaque at center on both compositors; inverted path mask the reverse; a luma matte from a linear-gradient shape produces a ramp; a matte source never draws itself. T13 — Clip effects from the shader catalog · Sonnet Resolves: F12 (reach half). Decision D7. Serial chain end. Read first: render/effects.ts, packages/gpu/src/shaders/index.ts (exports for glow, dropShadow, vignette, sharpen, chromaKey, curves, levels, liftGammaGain), packages/image-nodes/src/nodes/lib-image-effects.ts (parameter mapping already done once for images), render/canvas2d.ts (unsupportedEffectTypes, filter mapping). Change: map each new ClipEffect type to its shader step with the same parameter conventions the image nodes use. 2D implements dropShadow through shadowColor/shadowBlur/shadowOffsetX/Y on the layer draw and lists every other new type in unsupportedEffectTypes. Bridge op set_effects ({target, effects[]}) replaces the list. Validator unknown_effect (I2). Track effects are untouched. Tests: per-effect GPU test under lavapipe asserting a pixel property (glow spreads beyond the source rect; levels with inBlack: 0.5 clips a mid-gray to black; chroma key removes the keyed color); 2D reports the unsupported set exactly; the headless preview returns them in effects_not_applied. Milestone 3 — Text, shape, fonts T14 — Text style: stroke, shadow, background, spacing · Sonnet Resolves: F4 (styling half), part of F15’s fallback. Read first: render/draw.ts (drawText, drawStaggeredText, the line-height and vertical-center constants), web/src/components/timeline/preview/textRender.ts (cache key), packages/agents/src/timeline-preview/rasterize.ts. Change: honor fontStyle, letterSpacingPx (manual advance per glyph; ctx.letterSpacing where the context supports it), lineHeight, verticalAlign, stroke (drawn before fill), shadow, background (rounded rect behind the wrapped block with padding), fill gradient (T16’s ShapeFill helper). Extend the raster cache key with every new field. Stagger draws per unit with the same style. Tests: pixel tests through canvas2d for stroke width, shadow offset, background bounds, verticalAlign: "top"; the cache-key test enumerates the style fields via Object.keys of a fully populated style so a new field cannot be forgotten. T15 — Caption style · Sonnet Resolves: hard-coded caption look (F4 family). Read first: render/draw.ts (drawCaption, the #FFD60A and 5% constants), web/src/components/timeline/preview/captionRender.ts, packages/timeline/src/subtitles.ts. Change: drawCaption reads caption.style with today’s values as defaults. ui_timeline_set_clip_params and the set_clip_params op accept captionStyle. The inspector’s caption section exposes the fields with primitives. SRT/VTT export is unchanged. Tests: default render is pixel-identical to before (snapshot); a custom activeColor changes only the active word. T16 — Shapes: paths, polygons, gradients, dashes, trim · Opus Resolves: F8. Read first: render/draw.ts (drawShape, the RasterContext2D interface at the top), T12’s svgPath.ts, web/src/components/timeline/preview/shapeRender.ts. Change: RasterContext2D gains bezierCurveTo, quadraticCurveTo, arc, setLineDash, lineCap, createLinearGradient, createRadialGradient (both OffscreenCanvasRenderingContext2D and @napi-rs/canvas satisfy them, I6). drawShape handles path (through svgPath.ts), polygon, star, cornerRadius, fillStyle gradients, dash, lineCap/Join, trimStart/trimEnd (walk the flattened path by arc length and stroke the sub-range). ShapeFill helper shared with T14 text fills. Tests: pixel tests for a star’s point count (sample along a circle), a trimEnd: 0.5 path stroking half its length, a dashed line’s gap, a linear gradient’s two ends; svgPath.ts unit tests for relative commands and Z. T17 — Font pipeline · Opus Resolves: F15. Decision D8. Independent of the chain. Read first: packages/websocket/src/trpc/routers/fonts.ts, packages/config/src/package-asset-registry.ts (PACKAGE_RUNTIME_ASSETS, loadPackageAssetJson), scripts/bundle-backend.mjs and scripts/verify-backend-bundle.mjs (asset staging), render/draw.ts font string, packages/agents/src/timeline-preview/rasterize.ts, packages/video-nodes/src/nodes/timeline/rasterizers.ts, web/src/components/timeline/preview/textRender.ts, Dockerfile font packages. Change: packages/timeline/fonts/: Inter (variable), Space Grotesk, Playfair Display, JetBrains Mono, Bebas Neue, Lora, with their OFL files (C7). packages/timeline/src/fonts/catalog.ts lists family, weights, style, relative path, license. Register the directory in PACKAGE_RUNTIME_ASSETS so the bundle stages it and the verifier checks it (C6). resolveFontFamily(name) → { family, portable }. draw.ts builds its font string from it; the fallback stack ends in sans-serif. Registration: registerBundledFonts() in packages/timeline/src/fonts/register-node.ts calls GlobalFonts.registerFromPath once per process; the agent preview and the server rasterizer call it before drawing. Web: a generated @font-face stylesheet under web/src/components/timeline/fonts.css pointing at the same files served from /api/assets/packages/timeline/fonts/<file>, loaded by the timeline editor and awaited (document.fonts.load) before a text raster. Fonts endpoint returns {name, source: "bundled" | "system", portable}. Inspector font picker lists bundled first with a “portable” mark. Validator font_not_portable (warning) when a text clip names a family not in the catalog. Tests: the server rasterizer and the agent preview draw the same word in Bebas Neue and the bitmaps match within a tolerance (same library, same face); a system-only family is flagged; verify-backend-bundle fails when a font file is removed from the staged bundle (prove it, restore). Milestone 4 — Agent surface T18 — set_timeline_document · Opus Resolves: F18. Read first: packages/agents/src/capabilities/timelines.ts (CAS retry pattern in edit_timeline, snapshot helpers), packages/websocket/src/trpc/routers/timeline.ts (update, baseUpdatedAt), packages/execution/src/timeline-debug/validate.ts. Change: capability set_timeline_document {timeline_id, document, fps?, width?, height?, expected_updated_at?, snapshot_name?}: validate first (errors refuse the write and return the issues), snapshot as a manual version, CAS write, return the post-write validation. Add timelines.setDocument to the CodeAct object model and a capability-table row. Tests: capabilities-timelines.test.ts: happy path, validation refusal, CAS conflict, snapshot exists after write. T19 — Structural ops in the bridge · Sonnet Resolves: wiring for T9, T11, T12, T13, T29 into edit_timeline and ui_timeline_*. Read first: packages/timeline/src/ops/apply.ts (shared edit engine), packages/agents/src/capabilities/timeline-bridge.ts (host adapter), packages/agents/src/capabilities/timelines.ts (op normalization, MAX_OPS), web/src/lib/tools/builtin/timeline.ts, packages/agents/src/capabilities/timelines.specs.ts. Change: ops add_group {name, startMs, durationMs, trackId?}, set_parent {target, parentId | null}, set_transition, set_mask, set_matte, set_effects, set_time_remap, each with a Zod input in the op schema and a browser twin. Each op’s error message on a bad target lists valid ids. Skip any whose feature task has not merged; list them in the PR. Tests: timelines-op-input.test.ts per op; the bridge continues past a failing op and records it. T20 — Markers and beat snapping · Opus Resolves: F23. Read first: packages/agents/src/capabilities/analysis.specs.ts (detect_audio_events output shape), web/src/lib/tools/builtin/timeline.ts (ui_timeline_add_marker), packages/timeline/src/snap.ts and resolveSnap.ts, buildSnapPoints.ts. Change: Ops add_marker {timeMs, label, color?}, delete_marker {id}, set_markers_from_beats {onsets_ms[] | (bpm, offset_ms, count), label?}. Op snap_to_beats {targets[] | "all", grid: onsets_ms[] | (bpm, offset_ms), tolerance_ms (default 60), mode: "start" | "end" | "both", action: "move" | "trim"}: for each target, find the nearest grid time within tolerance to the chosen boundary and move or trim to it; report per clip {clipId, before, after, delta} and skip out-of-tolerance clips with a reason. Pure function snapClipsToGrid in packages/timeline/src/beats.ts, used by the op. Browser twins for the marker ops. Tests: beats.test.ts analytic cases (a 120 BPM grid, clips 30 ms off snap, 90 ms off do not; trim keeps startMs); op tests. T21 — Compositions · Opus Resolves: F20. Decisions D11, AS6. Read first: packages/agents/src/capabilities/entities.ts (asset marker pattern, ENTITY_METADATA_KEY), T9 groups, packages/base-nodes/nodetool/examples/storyboards/ and scripts/build-example-storyboards.mjs (shipped example pattern), packages/system-skills/caption-titles/SKILL.md (the five tiers to encode). Change: packages/timeline/src/composition.ts: TimelineComposition {id, name, description, params: Record<name, {type: "string" | "number" | "color" | "boolean", default, path: string}>, group: TimelineClip, children: TimelineClip[]} where child times are relative to the group start and path is a JSON pointer into a child (/1/textStyle/text). instantiateComposition(comp, {startMs, trackId, params}) → clips[] with fresh ids, compositionId and compositionParams stamped. extractComposition(doc, groupId, params) → TimelineComposition. Capabilities list_compositions, get_composition, save_composition {timeline_id, group_target, name, params} (writes a JSON asset with metadata.nodetool_composition), delete_composition; op insert_composition {composition_id, startMs, trackId?, params}. Shipped compositions in packages/base-nodes/nodetool/examples/compositions/: title-card, lower-third, caption-bar, callout, cta-end-card, logo-sting, built by scripts/build-example-compositions.mjs from a spec, validated with validate_timeline on a scratch document, registered in PACKAGE_RUNTIME_ASSETS, listed by list_compositions as source: "shipped". Tests: instantiate then extract is identity modulo ids; a param path that misses is refused; inserting lower-third into the JTBD fixture validates and previews with the name visible (pixel test on the text bounds). T22 — render_timeline as a job · Opus Resolves: F21, F14 (job half). Decision D12. Read first: packages/execution/src/service/workflow-run.ts (runWorkflow), packages/agents/src/capabilities/jobs.specs.ts (get_job, get_job_logs), how start_background_job builds and submits a graph, T3’s node props. Change: capability render_timeline {timeline_id, format?, alpha?, video_codec?, bitrate?, motion_blur_samples?, shutter_angle?, preview_scale?, wait?: boolean, timeout_ms?} builds a one-node graph and submits it as a job; returns {job_id} or, with wait, {job_id, asset_id, render_mode, duration_ms}. preview_scale renders at a fraction of the sequence size for drafts (R1). Add timelines.render to the object model, a capability-table row, and a line in the chat prompt’s timeline section (packages/websocket/src/session/chat-prompt.ts) telling the agent to render and look before declaring a cut done. Tests: capability test with the execution service in-process on a two-clip fixture under lavapipe; render_mode === "composited"; understand_video-free assertion by decoding the middle frame with Mediabunny and reading a pixel. T23 — Preview range, contact sheet, and diff · Sonnet Resolves: F22. Read first: packages/agents/src/timeline-preview/frames.ts, timelines.specs.ts (PREVIEW_TIMELINE_FRAME_SCHEMA, the caps), packages/agents/src/sandbox-media.ts or the image.grid helper the sandbox exposes. Change: preview_timeline_frame gains range {from_ms, to_ms, count ≤ 24} (evenly spaced, inclusive) and sheet: boolean (tile every frame into one image with a timecode label per cell, columns = ceil(sqrt(n))); with sheet the response carries one handle plus the per-frame layer reports. New capability compare_timeline_frames {a: timeline_id | document, b: timeline_id | document | {version}, times_ms[] | range} returning per-frame mean absolute pixel difference (0..1) and a side-by-side sheet handle. Caps: 24 frames, 1280 px width. Tests: sheet dimensions and cell count; a diff of a document against itself is 0; against a shifted clip is > 0 only at the affected times. T24 — Skill and prompt updates · Opus Resolves: the teaching half of F19–F23. Read first: packages/system-skills/motion-graphics/SKILL.md, packages/system-skills/caption-titles/SKILL.md, packages/websocket/src/session/chat-prompt.ts (timeline lines), docs/WRITING_STYLE.md, every tool spec landed in T4, T18–T23. Change: motion-graphics gains sections: keyframes and the easing grammar (when a preset is enough, when to write curves, the fold rules for replace channels); groups (rig a lower third as a group, animate the group); transitions (which type for which cut, the auto-crossfade rule); masks and mattes; compositions (insert first, override params, extract a new one when the user approves a look); beat cutting (detect_audio_events → snap_to_beats, tolerance guidance); the render loop (preview while iterating, render_timeline with preview_scale before the final, compare_timeline_frames after a change the user did not ask for); the validator codes and what each means. Keep the existing timing and easing guidance. caption-titles maps each tier to a shipped composition. Chat prompt: two lines, render-before-done and snapshot-before-edit. Tests: packages/system-skills has a lint test for skill frontmatter; run it. Read the skill aloud against the tool specs: every tool and op it names exists (write a test that greps the skill for `[a-z_]+` names and checks them against the capability registry and op list). T25 — Motion evals and a JTBD job · Opus Resolves: F25. Read first: packages/agents/src/evals/surfaces/timeline.ts (cases, TIMELINE_SYSTEM_PROMPT), packages/agents/src/evals/tool-loop-eval.ts, packages/agents/src/jtbd/registry.ts (caption-titles-picture-locked), packages/agents/tests/jtbd-friction.test.ts (the untouched-world rule), packages/cli/src/harness/capability-table.ts. Change: The eval system prompt embeds the motion-graphics skill body. New timeline-tools cases with deterministic predicates: kinetic-title-staggered (stagger span fits the clip: durationMs + offsetMs × (units−1) ≤ clip.durationMs), lower-third-layered (a shape clip on a higher-index track than the text, both within the picture clip’s window), entrance-decelerates (every in animation’s easing is an ease-out family or a spring), beat-cut (given onsets in the objective, every picture boundary within 60 ms of one), looked-before-done (preview_timeline_frame called after the last edit), keyframed-slide (from T4). JTBD job motion-title-sequence: a vertical timeline with music and three shots; objective in the user’s words (a title sequence that lands on the beat with a lower third and an end card); outcomes: markers or clip boundaries on the grid, a group or composition used, every text clip animated in and out with a fitting stagger, a preview call in the transcript. Must fail on the untouched world. Capability-table rows for the new capabilities name these cases; npm run capabilities:sync. Tests: each predicate has a passing and a failing hand-built final state in timeline-tool-loop.test.ts. T26 — Browser batch tool · Sonnet Resolves: F26. Read first: web/src/lib/tools/builtin/timeline.ts, web/src/components/timeline/timelineAgentBridge.ts. Change: ui_timeline_edit {timeline_id, ops[]} in web, dispatching to the same handler methods the single tools use, continuing past failures with a per-op record (mirror edit_timeline). Mobile stays view-only and uses server agent tools for edits. Tests: web tool tests for the batch path. Milestone 5 — Render quality T27 — Alpha export and output formats · Opus Resolves: F13. Depends on T3. Read first: packages/video-nodes/src/nodes/timeline/rawFrames.ts (openFrameEncoder), compositeRender.ts, render/frameCompositor.ts and render/canvas2d.ts seed color, web/src/components/timeline/render/TimelineRenderer.ts (mediabunny output options), web/src/hooks/timeline/useTimelineExport.ts. Change: alpha: true seeds both compositors transparent and keeps straight alpha through to the encoder. Server: webm → libvpx-vp9 -pix_fmt yuva420p; mov → prores_ks -profile:v 4444 -pix_fmt yuva444p10le; png_sequence → PNGs zipped as one asset with a manifest.json (fps, count, size). mp4 with alpha is refused with the list of formats that carry alpha. Browser export gains a format and alpha choice in the export dialog (WebM VP9 alpha through mediabunny, PNG sequence as a zip). Node props and the render_timeline inputs from T22 pass through. Tests: a transparent frame region decodes with alpha 0 (Mediabunny on the WebM; PNG read on the sequence); mp4 + alpha refused; ProRes path skipped with a message when ffmpeg lacks prores_ks. T28 — Motion blur · Opus Resolves: F11. Decision D10. Serial chain end. Read first: both compositors’ frame entry points, T3/T22 options, TimelineRenderer.ts stepping loop. Change: renderFrame(timeMs, {samplesPerFrame = 1, shutterAngle = 180}) composites N samples at timeMs + (i + 0.5)/N × shutterAngle/360 × frameMs and averages them (GPU: accumulate into an rgba16float texture and resolve; 2D: draw each sample with globalAlpha = 1/N onto a cleared canvas). Applies to browser export, server render and the frame preview (preview_timeline_frame gains motion_blur_samples for checking a setting). Default off. Tests: a fast-moving shape with 8 samples produces a smear whose width matches the analytic travel over the shutter window within one pixel; with 1 sample the output is unchanged (snapshot). T29 — Time remap · Opus Resolves: F9. Decision D13. Read first: packages/timeline/AGENTS.md (source-time rules), packages/timeline/src/sourceRate.ts, splitClip.ts, trimClip.ts, render/sceneModel.ts (clipSourceTimeSec), the preview’s video pool (web/src/components/timeline/preview/PreviewCompositor.tsx) and the server decoder’s forward-only constraint (rawFrames.ts). Change: clipSourceTimeSec evaluates timeRemap.keyframes (piecewise, eased) when present, else today’s rate. splitClip and trimClip refuse a remapped clip with a message naming bake_time_remap; a new pure bakeTimeRemap is out of scope (say so) and the refusal is the contract (D13). The server decoder handles a backwards source seek by reopening the stream (slow, correct); the browser pool seeks. Validator time_remap_not_monotonic (t must ascend; sourceMs may descend). Tests: a two-keyframe ramp 0→2× reaches the analytic source time at 0.25/0.5/0.75; a reverse curve decodes descending frames on the server path; split refused. T30 — Validator codes · Sonnet Resolves: F24 and the codes named across M1–M5. Read first: packages/execution/src/timeline-debug/validate.ts, report.ts, markdown.ts, the motion-graphics skill’s list of failures it owns, T6–T29 PR bodies for codes they added. Change: implement or reconcile: animation_exceeds_clip (window longer than clip after delay), stagger_compressed (compiler flag from T8), text_illegible (font size below 2.5% of frame height; contrast under 3:1 against background.color when set, else against a declared shape clip fully behind it on a higher-index track), replace_curves_overlap, unknown_easing, unknown_transition, unknown_effect, parent_*, matte_source_missing, mask_path_invalid, layer_cap_exceeded, font_not_portable, time_remap_not_monotonic. Severity: warnings except parent_cycle, matte_source_missing, time_remap_not_monotonic (errors). Update docs/cli.md § timeline validate and the skill’s code table. Tests: one fixture per code that triggers it and one that does not; nodetool timeline validate --json on the shipped example storyboards’ assembled timelines stays clean. T31 — Inspector UI for the new fields · Sonnet Resolves: AS4’s thin UI layer. Read first: web/src/components/timeline/Inspector/ (ClipAnimations.tsx, ClipAdjustments.tsx, InspectorPrimitives.helpers.ts), web/src/components/ui_primitives/STRATEGY.md, docs/DESIGN.md. Change: sections for transition type and params, mask, matte source picker, effects list (add/remove/reorder with per-type fields), text style fields (T14), shape fields (T16), parent (read-only from T9 plus a “group selected clips” action), curve list for custom animations (property, keyframe table, easing string with the grammar hint), font picker (T17). No graph editor. Every control writes through the store’s patchClip. Tests: RTL tests per section that a change reaches the store; a11y labels on every control. Running the plan Suggested wave order for a team of agents: Wave 1 (parallel): T1, T3, T5, T17. Wave 2 (after T1): T2, T4, T6, T14, T15, T18, T20, T23 in parallel; T7 starts the compositor chain. Wave 3: T8, T9 (chain), T16, T22 (after T3), T26. Wave 4: T10 → T11 → T12 → T13 (chain, one at a time), T19 as each lands, T21 (after T9), T27 (after T3). Wave 5: T28 → T29 (chain), T24, T25, T30, T31. Each wave ends with npm run dev:nodetool -- harness gate --base main on the merged tree and nodetool eval timeline-tools -p claude_agent_sdk -m claude-sonnet-5 --min-success 0.8 (keyless, see AGENTS.md § nodetool eval).--- # Motion Graphics — Technical Design Source: https://docs.nodetool.ai/plans/motion-graphics.html Motion Graphics — Technical Design Status: implemented. Companion doc: motion-graphics-tasks.md (the agent-consumable task list, sequencing, and acceptance criteria). Predecessors: motion-design.md (preset animation engine, implemented), motion-typography.md (per-word stagger, implemented), ../timeline-custom-animations.md (JS baked to curves, implemented). Goal Two outcomes, in this order: The timeline can render and direct motion graphics at a professional level. Keyframes on any property with parametric easing, groups with transform inheritance, real two-clip transitions, masks and mattes, the shader library on clips, styled text with per-character motion, vector shapes with paths and gradients, reproducible fonts, and a server render that produces the picture the editor previews, with alpha, image-sequence and motion-blur options. An agent can direct that work efficiently. Write a whole document in one call, author keyframes and curves without a UI, instantiate stored compositions (lower third, title card, logo sting, callout), cut to a beat grid, render the finished cut headlessly and look at it, and be measured on motion quality rather than on tool-call shape. Findings this plan resolves Codes are stable across this document, the task list, and PR descriptions. Every finding below was verified against the tree; file references are the evidence. Animation model F1 No first-class keyframes. A clip carries animations: ClipAnimation[] naming a preset; nine animatable channels (ANIMATED_PROPERTIES, packages/timeline/src/animation/types.ts). Position animates only as an additive offset; scale only uniformly. Anchor, non-uniform scale, contrast, hue, temperature, tint, text and shape attributes, border radius and volume are static. F2 Seven fixed easings (animation/easing.ts), no cubic bezier, no spring, no tunable overshoot. F3 No expressions and no parenting. Custom animations bake once (animation/custom.ts). This is by design and stays. F4 Stagger is word-only (compile.ts, unit: "character" declared, unimplemented). No range selectors, no per-glyph geometry, no text on path. Structure and compositing F5 No groups, parents, precomps, nulls or adjustment layers. No parentId anywhere in packages/timeline/src/types.ts or packages/protocol/src/api-schemas/timeline.ts. F6 One transition type, crossfade, linear, incoming clip only (types.ts ClipTransition, sceneModel.ts crossfadeOpacity). F7 The only mask is the wipe animation’s four-direction feathered edge (animation/sample.ts AnimationSampleMask). No shape masks, no track mattes. packages/gpu/src/shaders/mask/* exist and are unused by the timeline. F8 Shapes are rect | ellipse | line, rasterized, no paths, gradients, dashes, trim, or SVG import (types.ts ClipShapeStyle, draw.ts drawShape). F9 Time is one scalar speedMultiplier, clamped positive (sourceRate.ts). No ramps, remap, reverse, or frame blending. Renderer F10 The production server render is the rough cut. bundle-backend.mjs stages webgpu (Dawn) on the desktop profile only (DESKTOP_ONLY_EXTERNAL_PACKAGES), while the Dockerfile already installs mesa-vulkan-drivers (lavapipe) for the image nodes. RenderTimeline therefore hits CompositorUnavailableError and concatenates clips, dropping transforms, effects, text, shapes and overlays (packages/video-nodes/src/nodes/timeline.ts). F11 No motion blur. Export samples one instant per frame (web/src/components/timeline/render/TimelineRenderer.ts); one bilinear tap per layer, no supersampling. F12 sRGB 8-bit throughout (frameCompositor.ts, effects.ts rgba8unorm), yuv420p hardcoded on the server (rawFrames.ts). The timeline uses five of the shader modules in packages/gpu/src/shaders/; glow, drop shadow, curves, levels, lift/gamma/gain, film look, corner pin, displace and LUT-style grades are unreachable from a clip. F13 No alpha export and no image-sequence export. Both compositors seed with opaque black (canvas2d.ts drawTimelineFrame, frameCompositor.ts). F14 No render job. renderTimelineComposited accepts onProgress and RenderTimelineNode never passes it. No cancel token reaches the frame loop, no checkpoint, no frame cache. Browser export dies with the tab. F15 No font pipeline. packages/websocket/src/trpc/routers/fonts.ts lists system font names; nothing bundles, downloads or registers a face. draw.ts falls back to Inter, Arial, sans-serif, which resolves differently on every host. Correctness F16 CaptionWord.kind and confidence exist in packages/timeline/src/types.ts and not in the Zod captionWord (api-schemas/timeline.ts), so every save strips them; filler-word removal reads kind. F17 MAX_VIDEO_LAYERS = 8 drops layers silently (sceneModel.ts). Agent surface F18 No declarative write. validate_timeline and preview_timeline_frame accept an inline document; nothing writes one back. F19 Custom animations are built end to end (bakeCustomAnimation, POST /api/timelines/animations/bake) and no capability exposes them; the bridge throws Unknown animation preset "custom" (packages/agents/src/evals/surfaces/timeline.ts). F20 No stored compositions. packages/system-skills/caption-titles describes five text tiers in prose; nothing instantiates one. F21 No headless render of the finished cut for an agent to judge. F22 Feedback is at most eight stills per call, no contact sheet, no range, no diff. preview_timeline_frame is missing from the CodeAct nodetool.timelines namespace (packages/agents/src/codeact/nodetool-api.ts). F23 detect_audio_events returns onsets and tempo; no op snaps clips to them and there is no headless marker op. F24 validate_timeline omits animation-longer-than-clip, compressed stagger, and legibility; the skill hands them to the agent. F25 Nothing measures motion quality. timeline-tools cases check tool shape; the eval prompt carries none of the motion skill; no JTBD job loads motion-graphics; preview_timeline_frame has no eval case. F26 The browser ui_timeline_* tools need a batch form. Mobile timelines are view-only and have no client editing tools. Assumptions AS1 The one-model architecture stays: packages/timeline/src/render/ is the single scene model, and the live preview, browser export, server render and agent frame preview all consume it. No feature lands in one host only. AS2 packages/timeline stays pure TypeScript with no runtime dependencies at the package root. Render code lives under src/render, reached through ./render (GPU) or ./scene (no GPU), never the root export. AS3 Dawn plus lavapipe is an acceptable production render device. Lavapipe is a CPU rasterizer; a 1080p30 minute at the current per-frame effect load is minutes, not seconds. Throughput is a later concern; parity is the goal of this plan. AS4 The agent is the primary author. Every feature is specified as data plus a headless op first; the inspector UI is a thin layer added after, and a task that cannot fit the UI in scope says so rather than shrinking the data model. AS5 Motion written as JavaScript continues to bake once at author time. There is no render-time expression engine and there will not be one. AS6 A composition is a document fragment, not a new renderer. Templates are groups with parameters, stored as JSON assets, instantiated by copying clips into the target document. AS7 Storage of new document fields is JSON-blob only. No new columns are needed for anything in this plan except none; compositions use the asset table with a metadata marker, the way entities do. AS8 Frame determinism is required across hosts at the scene level (same layers, same transforms, same sampled values) and not at the byte level (decoders and encoders differ). Invariants These hold before and after every task. A task that must relax one says so in its PR and updates this list. I1 Schema mirror. Every field on TimelineClip, TimelineTrack, ClipAnimation and their children exists in both packages/timeline/src/types.ts and the Zod schema in packages/protocol/src/api-schemas/timeline.ts. The Zod schema strips unknown fields on every save; a field missing there is data loss. packages/execution/src/timeline-debug/validate.ts (field_stripped) is the check, and every new field gets a round-trip test. I2 Forward compatibility by string. preset, easing, curve property, transition type, effect type, mask kind and stagger unit are typed as string in the document and narrowed at compile or validate time. A document from a newer build parses; the unknown item is skipped with a validator warning, never a thrown error. I3 Pure sampler. sampleAnimations is a pure function of compiled curves and time. Its fold is commutative across animations on one clip: offsets and rotation add, scale and opacity multiply, wipes take the minimum. New channels declare their fold in ANIMATED_PROPERTIES metadata. I4 No JS at render time. A custom animation’s body runs at bake and nowhere else (normalizeCustomCurves is the single gate). I5 Source-time through one helper. Every conversion from timeline time to source time goes through clipSourceTimeSec and sourceRate. Split, trim, merge, the preview and the render never reimplement it. I6 Draw code takes RasterContext2D. draw.ts never types against OffscreenCanvas or @napi-rs/canvas; both must satisfy the interface. I7 Canvas 2D reports what it drops. unsupportedEffectTypes names every effect the 2D path cannot apply; the headless preview returns them as effects_not_applied. Silent divergence between hosts is a bug. I8 No WebGPU globals at module scope. Under Node they exist only after the Dawn device installs them. I9 Layer order is track order. trackZ(index) = 1000 - index; lowest index on top. Groups do not change this: a child’s z is its own track’s z. I10 A capability has a spec, an implementation, a table row, and a check. New capabilities go in *.specs.ts, the implementation module, packages/cli/src/harness/capability-table.ts (regenerated with npm run capabilities:sync), and either a suite or an eval case. npm run capabilities:check fails otherwise. I11 Headless and browser tools share one implementation. edit_timeline and ui_timeline_* edits use applyTimelineOp in packages/timeline/src/ops/apply.ts. The server adapter lives in packages/agents/src/capabilities/timeline-bridge.ts. Add document edit operations to the shared engine and keep host I/O in the adapters. I12 A new check proves it can fail. Every validator code, eval predicate and JTBD outcome ships with a fixture that fails it (docs/HARNESS_FIRST.md rule 7, AGENTS.md § Claims, Checks, and Measurements). Constraints C1 Node 22.22.1, ESM everywhere, @nodetool-ai/<package> imports only. C2 Web UI uses web/src/components/ui_primitives/ and the design tokens in docs/DESIGN.md. No raw MUI. C3 Every PR runs npm run test:affected, npm run typecheck, npm run lint, plus npm run dev:nodetool -- harness gate --base main. C4 Mobile is not a root workspace; npm --prefix mobile. C5 packages/agents imports @nodetool-ai/timeline/scene, never /render. TypeGPU must not enter the agents package. C6 The server bundle is the deploy artifact. Anything the server loads at runtime is staged by scripts/bundle-backend.mjs and verified by scripts/verify-backend-bundle.mjs; an unstaged file is a production failure, not a dev failure. C7 Fonts shipped in the repo are OFL or Apache licensed and their license files ship next to them. C8 No count, date or status word in AGENTS.md. Plan docs may carry a status line. C9 PRs stay reviewable: one task per PR, target under 400 changed lines of non-test code, schema landing excepted. C10 Every outbound fetch of a caller-chosen URL (a Google Fonts URL) goes through safeFetch and is inventoried in docs/url-egress-inventory.md. Decisions D1 Schema lands first, in one PR. Every new optional field in this plan is added to types.ts, the Zod schema and the validator’s round-trip fixture in a single task (T1) before any feature task starts. Feature tasks then touch behavior files only, which keeps eight parallel agents off the same two files. D2 Keyframes are curves, and curves already exist. A first-class keyframe animation is preset: "custom" with custom.curves set directly and no code. The compiler already accepts this shape. The agent op adds curves as an input alongside code; the UI gets a curve list, not a graph editor, in this plan. D3 Parametric easing by string grammar. easing stays a string. "cubic-bezier(x1,y1,x2,y2)" and "spring(stiffness,damping,mass)" are parsed by parseEasing; unknown strings fall back to linear with a validator warning (I2). D4 Groups are transform parents on their own clip. A group is a clip with mediaType: "group" and no media. A child names it by parentId. The child’s placement matrix is parent × child, opacity multiplies, and the child is clipped to the parent’s window. Children stay on their own tracks, so I9 holds. Group-level effects and blend modes require an intermediate texture and land in a separate task (precomposite) after parenting. D5 Transitions are owned by the incoming clip and resolved for both. transitionIn keeps its place; the scene model finds the outgoing partner (the previous clip on the same track overlapping the window) and applies the complementary curve. New types: dipToColor, wipe, push, slide, zoom, each with easing. D6 Masks are rasterized once per frame, mattes are layers. A clip mask (rect | ellipse | path, feather, invert) rasterizes to a mask texture through draw.ts and is applied with the existing mask/apply shader or a Canvas 2D clip path. A track matte names a source clip; the compositor renders that layer to a texture and applies it as alpha or luma. D7 Effects on clips reuse the shader catalog by id. ClipEffect grows glow, dropShadow, vignette, sharpen, chromaKey, curves, levels, liftGammaGain. Each maps to one existing module in packages/gpu/src/shaders/. Canvas 2D implements dropShadow through ctx.shadow* and reports the rest (I7). D8 Fonts ship with the product. A bundled corpus (Inter variable plus a small set of OFL faces) is registered on every host by one packages/timeline/src/fonts/ table. System fonts remain available and are marked as non-portable in the fonts endpoint and the validator. D9 The server renders through Dawn on every profile. webgpu moves to the common staged externals; the Dockerfile already installs lavapipe. A host with no adapter still falls back to the rough cut, and the fallback is logged as a warning on the job, not swallowed. D10 Motion blur is sub-frame accumulation. samplesPerFrame and shutterAngle render options; the compositor averages N samples inside the shutter window. Costs N× render time and is off by default. D11 Compositions are JSON assets with a marker. metadata.nodetool_composition on a JSON asset, the entity pattern. Shipped compositions live in packages/base-nodes/nodetool/examples/compositions/. D12 Render is a workflow job. render_timeline builds a one-node graph (nodetool.timeline.RenderTimeline) and runs it through the existing execution service, so it gets the job table, logs, progress messages and cancel for free. D13 Time remap is a curve, and a remapped clip cannot be split. Mirrors speedBaked: split and trim refuse a clip carrying timeRemap and tell the caller to bake first. Reverse is a remap whose curve descends. Desired solution Document model All fields optional unless stated. Names are final; tasks reference them. // packages/timeline/src/types.ts type TimelineTrackType = "video" | "audio" | "overlay" | "subtitle"; type ClipMediaType = "image" | "video" | "audio" | "overlay" | "text" | "shape" | "group"; interface TimelineClip { // ... existing ... /** Group this clip belongs to. Transform composes, opacity multiplies, * window clips. Must name a clip with mediaType "group". */ parentId?: string; mask?: ClipMask; matte?: ClipMatte; transitionIn?: ClipTransition; // union widened, see below effects?: ClipEffect[]; // union widened, see below timeRemap?: ClipTimeRemap; /** Composition provenance, set by insert_composition. */ compositionId?: string; compositionParams?: Record<string, number | string | boolean>; } interface ClipMask { kind: string; // "rect" | "ellipse" | "path" /** Normalized 0..1 in the layer's own space. */ x?: number; y?: number; width?: number; height?: number; /** SVG path data, normalized 0..1 space, for kind "path". */ d?: string; featherPx?: number; invert?: boolean; } interface ClipMatte { sourceClipId: string; mode: string; // "alpha" | "luma" invert?: boolean; } type ClipTransition = | { type: "crossfade"; durationMs: number; easing?: string } | { type: "dipToColor"; durationMs: number; color: string; easing?: string } | { type: "wipe"; durationMs: number; direction: string; softness?: number; easing?: string } | { type: "push"; durationMs: number; direction: string; easing?: string } | { type: "slide"; durationMs: number; direction: string; easing?: string } | { type: "zoom"; durationMs: number; easing?: string }; type ClipEffect = | ClipColorEffect | ClipBlurEffect | { type: "glow"; radius: number; intensity: number; color?: string } | { type: "dropShadow"; offsetX: number; offsetY: number; blur: number; color: string; opacity?: number } | { type: "vignette"; amount: number; softness: number } | { type: "sharpen"; amount: number; radius?: number } | { type: "chromaKey"; color: string; tolerance: number; softness: number; spill?: number } | { type: "curves"; master: CurvePoint[]; r?: CurvePoint[]; g?: CurvePoint[]; b?: CurvePoint[] } | { type: "levels"; inBlack: number; inWhite: number; gamma: number; outBlack: number; outWhite: number } | { type: "liftGammaGain"; lift: [number, number, number]; gamma: [number, number, number]; gain: [number, number, number] }; interface ClipTimeRemap { /** Monotonic in t; sourceMs may descend (reverse). t normalized 0..1 over the clip. */ keyframes: { t: number; sourceMs: number; easing?: string }[]; } interface ClipTextStyle { // ... existing text, fontFamily, fontSizePx, fontWeight, color, align, maxWidthFrac ... fontStyle?: string; // "normal" | "italic" letterSpacingPx?: number; lineHeight?: number; // multiplier, default 1.2 verticalAlign?: string; // "top" | "middle" | "bottom" stroke?: { color: string; widthPx: number }; shadow?: { color: string; blurPx: number; offsetX: number; offsetY: number }; background?: { color: string; paddingPx: number; radiusPx?: number }; fill?: ShapeFill; // gradient text } interface ClipShapeStyle { kind: string; // "rect" | "ellipse" | "line" | "path" | "polygon" | "star" // ... existing fill, stroke, strokeWidthPx, x, y, width, height, x2, y2 ... d?: string; // kind "path", normalized 0..1 sides?: number; innerRadius?: number; // polygon / star cornerRadius?: number; fillStyle?: ShapeFill; // wins over fill when set dash?: number[]; lineCap?: string; lineJoin?: string; trimStart?: number; trimEnd?: number; // 0..1, animatable } type ShapeFill = | { type: "solid"; color: string } | { type: "linear"; angle: number; stops: { offset: number; color: string }[] } | { type: "radial"; stops: { offset: number; color: string }[] }; interface ClipCaption { // ... existing words ... style?: CaptionStyle; } interface CaptionStyle { fontFamily?: string; fontSizeFrac?: number; color?: string; activeColor?: string; outline?: { color: string; widthPx: number }; bottomMarginFrac?: number; background?: { color: string; paddingPx: number; radiusPx?: number }; } interface CaptionWord { word: string; startMs: number; endMs: number; kind?: "word" | "filler" | "pause"; // F16: add to Zod confidence?: number; // F16: add to Zod } Animation channels grow to: offsetX offsetY scale scaleX scaleY rotation opacity wipeProgress blur brightness saturation contrast hue temperature tint positionX positionY anchorX anchorY trimStart trimEnd Each channel carries fold metadata (add | multiply | replace | min) in ANIMATED_PROPERTIES, replacing the implicit switch in sample.ts. positionX/Y and anchorX/Y are replace channels: the last enabled animation in document order wins, and the validator warns when two replace curves overlap in time. Stagger gains unit: "character" and unit: "line". The text rasterizer already draws per-unit; resolveTextStaggerContext counts units per the declared unit. Scene model computeActiveLayersWithHorizon changes in four places: Groups. Group clips are resolved first. A layer whose clip has parentId gets parentMatrix and parentOpacity from the group’s own resolved animated props, and is dropped when the query time is outside the group’s window. Cycles and missing parents are validator errors and are ignored at render (the child renders unparented). Transitions. For each clip with a non-crossfade transitionIn, the outgoing partner is found and both layers carry transition: {type, progress, role: "in" | "out", params}. crossfadeOpacity becomes one case of resolveTransition. Mattes. A layer with matte carries the matte source’s resolved layer under matteLayer, and the source is removed from the normal stack (a matte source never draws itself). Layer cap. Dropped video layers are returned as droppedLayers: {clipId, reason}[] (F17); the preview and the validator report them. resolveAnimatedLayerProps applies fold metadata and adds scaleX/Y, positionX/Y, anchorX/Y, trimStart/End and the extra color channels to the synthesized ClipEffect. buildTransformMatrix accepts a parent matrix. clipSourceTimeSec reads timeRemap when present (I5). Rendering Both compositors (frameCompositor.ts, canvas2d.ts) gain: parent matrix composition; transition draw rules (push and slide move the outgoing layer, wipe uses the existing feathered mask with the outgoing layer beneath, dipToColor draws a solid between, zoom scales both); clip masks (GPU: rasterize through draw.ts into a mask texture and maskApplyV1; 2D: ctx.clip with a feathered gradient approximation); mattes (GPU: render the matte layer to a texture, maskFromImageV1 + apply; 2D: destination-in with an offscreen canvas); the new clip effects; and a transparent seed when alpha: true. Precomposite (group-level effects and blend): a group carrying effects or a non-normal blendMode renders its children into an intermediate texture sized to the frame, applies the group’s effect chain, and blends the result once. Without those fields a group is free. Motion blur: renderFrame(timeMs, {samplesPerFrame, shutterAngle}) averages N composites at timeMs + i × (shutterAngle/360) × frameMs / N. Implemented in the two compositors’ frame entry points, so browser export, server render and the frame preview all support it. Server render (packages/video-nodes/src/nodes/timeline/): webgpu staged on the server profile (D9); onProgress wired to node_progress; the abort signal checked per frame; new node props format (mp4 | webm | mov | png_sequence), alpha, video_codec, bitrate, motion_blur_samples, shutter_angle. Alpha routes: webm → VP9 with yuva420p; mov → prores_ks -profile:v 4444; png_sequence → a zip asset of PNGs with straight alpha. Fonts: packages/timeline/src/fonts/catalog.ts lists the bundled faces (family, weights, style, file, license). packages/timeline/fonts/ holds the files. Registration: @napi-rs/canvas GlobalFonts.registerFromPath in the agent preview and the server rasterizer, @font-face rules in the web timeline. resolveFontFamily(name) returns the bundled family or the literal name with portable: false. The fonts endpoint returns both sets tagged by source. Agent surface New and changed capabilities, all in packages/agents/src/capabilities/timelines*.ts unless stated, all dispatched through the one bridge (I11): Capability / op Purpose set_timeline_document Replace the whole document, CAS on updated_at, validated first, snapshot before write. F18. edit_timeline op animate_clip Accepts preset: "custom" with curves (inline keyframes) or code (baked server-side through bakeCustomAnimation). F19. op add_group, set_parent Create a group clip; parent or unparent clips. op set_transition, set_mask, set_matte, set_effects, set_time_remap Structural fields. op add_marker, delete_marker Headless markers. op snap_to_beats Given onsets_ms[] or bpm + offset_ms, move or trim named clips so boundaries land on the nearest beat within tolerance_ms. F23. op insert_composition Instantiate a stored composition at a time on a track with params. list_compositions, get_composition, save_composition Composition library over JSON assets. F20. render_timeline Run RenderTimeline as a job; returns job_id, and with wait: true the output asset. F21. preview_timeline_frame gains range {from_ms, to_ms, count} and sheet: true A tiled contact sheet as one image; a dense sample of a window. F22. compare_timeline_frames Two documents (or a document and a version), same timecodes, per-frame pixel difference score plus a side-by-side sheet. ui_timeline_edit (web) The batch twin of edit_timeline in the browser. F26. nodetool.timelines in codeact/nodetool-api.ts gains preview, render, setDocument, compositions. Validation New codes in packages/execution/src/timeline-debug/validate.ts: animation_exceeds_clip, stagger_compressed, text_illegible (font size below 2.5% of frame height, or contrast ratio under 3 against a declared background), parent_missing, parent_cycle, parent_not_group, matte_source_missing, mask_path_invalid, layer_cap_exceeded, font_not_portable, unknown_easing, unknown_transition, unknown_effect, time_remap_not_monotonic, replace_curves_overlap. Each ships with a failing fixture (I12). Skills, evals, JTBD packages/system-skills/motion-graphics/SKILL.md gains sections for keyframes and easing grammar, groups, transitions, masks, compositions, beat cutting, the render loop, and the new validator codes. The eval system prompt for timeline-tools embeds the skill. New eval cases score motion outcomes (stagger span fits, entrance easing decelerates, scrim below text, beat alignment within tolerance, a preview call before the summary). A new JTBD job motion-title-sequence loads motion-graphics and grades the world it leaves behind. Non-goals A graph or curve editor UI. Curves get a list editor. An expression language (AS5). 3D layers, cameras, lights. Distributed rendering. Progress and cancel land; sharding does not. Linear-light compositing and HDR output. Recorded as R6 for a later plan. Particle systems. Lottie or Rive import. Risks R1 Lavapipe throughput. A minute of 1080p with three effects may take several minutes. Mitigation: progress messages, cancel, and a preview_scale render option for drafts. R2 Dawn instance retention keeps the event loop alive (packages/gpu/src/node.ts). The server is long-lived so this is fine; the render job must not be run in a one-shot child process that expects to exit on drain. R3 Schema landing (T1) is a large PR. Mitigation: fields only, no behavior, one round-trip test per field, reviewed as a table. R4 Parallel tasks touching sceneModel.ts, canvas2d.ts and frameCompositor.ts will conflict. Mitigation: the task list marks the serial chain through those files; everything else is parallel. R5 Contrast-based legibility is a heuristic. Ship it as a warning, not an error. R6 Compositing in sRGB makes glow and additive blends band. Accepted for this plan. R7 Bundled fonts add megabytes to every artifact. Cap the corpus at eight families and subset where the license allows. Milestones M0 Unblock and correct T1 schema · T2 F16/F17 · T3 server Dawn · T4 custom animation op T5 preview in CodeAct API M1 Animation depth T6 easing grammar · T7 channels + folds · T8 character stagger M2 Structure T9 groups · T10 precomposite · T11 transitions · T12 masks/mattes T13 clip effects M3 Text, shape, fonts T14 text style · T15 caption style · T16 shapes/paths · T17 fonts M4 Agent surface T18 set_document · T19 structural ops · T20 markers/beats T21 compositions · T22 render_timeline job · T23 preview sheet/range/diff T24 skills · T25 evals + JTBD · T26 browser batch M5 Render quality T27 alpha + formats · T28 motion blur · T29 time remap T30 validator codes · T31 inspector UI Sequencing and per-task specs: motion-graphics-tasks.md.--- # Motion Typography — Per-Word Staggered Text Animation Source: https://docs.nodetool.ai/plans/motion-typography.html Motion Typography — Per-Word Staggered Text Animation Status: implemented (July 2026). Companion doc: motion-design.md (the preset/curve/sampler substrate this builds on). Goal Jitter is mostly motion typography, and the signature move is the stagger: words of a title animating in sequence, each offset in time from the previous — a headline whose words pop in one after another, a kicker that slides away word by word. The motion-design system animates a text clip as one block; this design splits the block into words without breaking the one-clip-one-layer model. Authoring stays preset-shaped: a stagger is not a new animation type but a modifier on an existing ClipAnimation — “run this pop-in once per word, 120 ms apart”. The agent (ui_timeline_animate_clip) is the primary authoring surface; the inspector gets a thin stagger row on text clips. v1 unit is words. Characters and lines can follow behind the same stagger.unit field without a schema break. Architecture: stagger inside the text rasterizer Two candidates were on the table: A (chosen). Time-dependent text raster. Keep one layer per text clip. The TextRasterizer already measures every word to wrap lines, so per-word geometry is one refactor away. At draw time each word evaluates its own animation sample (same compiled curves, per-word time offset) and is drawn into the block’s canvas with its own translate/rotate/scale/alpha — 2D canvas transforms per word are cheap. The compositor, layer model, and computeActiveLayersWithHorizon’s change-horizon contract are untouched. Preview/export parity is free because both paths already rasterize through the same class. B (rejected). One sub-layer per word. Explode the clip into N word layers in the scene model, each animated via resolveAnimatedLayerProps with a time offset. GPU-composited words stay crisper under extreme scale/rotate, but it breaks the layer-set caching and horizon assumptions, multiplies layer count by word count, and touches every layer consumer (compositors, gizmo, placeholder DOM, video-slot bookkeeping). That price buys quality headroom v1 does not need — a word rotating inside a sequence-resolution raster is more than good enough for titles. The cost of A is cacheability: while a stagger window is active the raster is a function of time, not just style. That is exactly the caption situation (karaoke highlight), and the same remedy applies (see Caching). Data model ClipAnimation gains one optional field (engine animation/types.ts, wire schema packages/protocol/src/api-schemas/timeline.ts — the zod field is mandatory or PATCH strips it): export interface AnimationStagger { unit: string; // "word" — unknown units compile un-staggered (forward compat) offsetMs: number; // delay between successive words; must be > 0 from?: "start" | "end" | "center"; // which word leads; default "start" } export interface ClipAnimation { // …existing fields… stagger?: AnimationStagger; } from ordering: "start" = first word first, "end" = last word first, "center" = middle words first, rippling outward (delay = |i − (N−1)/2| × offsetMs). Stagger is only meaningful on text clips. The agent bridge rejects it on any other clip; a hand-edited document that carries it elsewhere (or an unknown unit) compiles as a plain block animation — same degrade-gracefully posture as unknown presets. Window and duration semantics durationMs is the per-word duration; the total span stretches. Word i runs the full preset curve over durationMs, delayed delay(i) from the span start, so the span is durationMs + maxDelay where maxDelay = maxFactor(from, N) × offsetMs. This is the reading an agent setting it expects — “each word pops for 400 ms, 120 ms apart” — and it keeps durationMs meaning the same thing with and without a stagger. The alternative (squeeze N word-windows inside durationMs) makes the per-word motion speed depend on word count, which reads as a bug when the text changes. Anchoring per role: in / emphasis: span grows forward from delayMs; the first word starts where the un-staggered window would have. out: span grows backward, so the last word to leave still finishes at clipEnd − delayMs — the clip ends empty, like an un-staggered out. loop: no span change; delay(i) is a phase shift on the cycle. Short clips shrink the offset, never the word duration. When durationMs + maxDelay does not fit the clip, the effective offsetMs is scaled down so the last word still completes inside the clip. If no positive offset fits (the clip is shorter than one word’s duration), the stagger drops and the animation compiles as a plain block window, clamped as before. Compiled form: compileClipAnimations takes options.staggerCount (the word count — countStaggerUnits(text), whitespace-split, matching the rasterizer’s wrap tokenization). A compiled stagger rides on the animation as CompiledAnimation.stagger (count, effective offsetMs, from, unitDurationMs, maxDelayMs), and windowStartMs..windowEndMs covers the full span. That one choice keeps the rest of the system ignorant of staggering: hasActiveAnimationWindow (and therefore the preview’s redraw gate and the compile cache) see one window that happens to be longer. Sampling: block/word split A staggered animation’s curves are split across two samplers: sampleAnimations (block level, unchanged signature) folds only its effect/mask curves (blur, brightness, saturation, wipeProgress), over the full span. Transform/opacity curves are skipped — they belong to the words. sampleStaggeredAnimations(compiled, localMs, wordIndex, out?) (new, pure) folds only transform/opacity curves of staggered animations, each word evaluated at its own window [spanStart + delay(i), + unitDurationMs] with the role’s usual holds (in: hold t=0 before, identity after; out: identity before, hold t=1 after; loop: phase shift). Un-staggered animations are skipped — they already applied at the layer. So on a text clip, per-word offset/scale/rotation/opacity come from the raster; block-level animations (a Ken Burns drift, a wipe, an added fade) compose on the layer exactly as before, and a staggered blur preset still blurs the whole block while its opacity fade runs per word. Per-word masks and effect parameters are explicitly out of scope for v1 (see Non-goals). Rendering integration TextRasterizer.rasterize gains an optional stagger argument ({ compiled, localMs }). Without it the block draw path is byte-identical to before. With it, the rasterizer lays out word boxes (line breaks use the same measured-candidate rule as the block path, so a staggered title wraps exactly like its un-staggered self) and draws each word about its own center: translate(wordCenter + sample.offset) → rotate → scale → alpha → fillText The per-frame context is resolved by one shared helper, resolveTextStaggerContext(clip, timeMs, canvas, cache) in sceneModel.ts, called from the only three text draw sites — PreviewCompositor, TimelineRenderer, and rasterClipFrames (the agent’s ui_timeline_get_clip_frames path) — so preview, export, and agent frames render per-word motion from one code path. The preview’s redraw gate needs no change: hasActiveAnimation reads the compiled full-span window, so the compositor keeps redrawing every rAF tick until the last word lands. Caching The style-hash bitmap cache cannot serve a mid-stagger frame. Policy, per rasterize call: Active (clip-local time inside any staggered animation’s span): draw fresh, uncached — a new bitmap per frame, like captions during a karaoke highlight. The rasterizer keeps the latest active bitmap per style key and closes the previous one on the next call (the caller has composited it by then), so playback does not accumulate full-frame bitmaps. Per-frame cost is one offscreen canvas fill of a few dozen words plus a texture upload — the price captions already pay, negligible next to video decode. Time- bucketed caching was rejected: 64 cached 1080p frames is ~0.5 GB. Held (before/after the span): the frame is static (hold or identity) but differs from the plain raster, so it caches under the normal key plus a suffix of the compiled-array identity and a per-animation phase signature (before/after). Steady state after a stagger completes is one cached bitmap again — the GPU re-uses the uploaded texture. Agent surface ui_timeline_animate_clip accepts stagger: { unit: "word", offsetMs, from? } per animation, documented in the tool description with the motion-typography use case. Validation lives where preset/role validation already lives: buildClipAnimation checks the unit and offset, the bridge handler rejects stagger on non-text clips with a corrective message. Ids, defaults, and the returned clip node are unchanged. ui_timeline_list_animation_presets is untouched — stagger is orthogonal to the catalog and composes with any non-full-clip preset. Manual UI Text clips get two rows inside each animation’s editor in the Animate section: a “Stagger words” toggle and, when on, a word-offset pill (ms). from stays agent/JSON-only until demand appears. Hidden for non-text clips and full-clip presets (kenBurns ignores stagger). Editing semantics Nothing new to store: splitClip/duplicateClip copy animations wholesale (stagger included), and trims re-clamp at compile time exactly like plain windows — the compile cache keys on clip duration and word count, so editing the text re-derives the span. Non-goals Character and line units — the schema (unit) and the sampler (unit index/count) are ready; the rasterizer would need per-glyph layout. Per-word effect/mask properties — blur/brightness/saturation and wipes stay block-level; per-word canvas filters would cost a save/filter/restore per word per frame for little visible gain at title sizes. Caption stagger — captions have real word timings; their karaoke path stays separate. Per-word easing/curve overrides — one curve, offset in time, is the Jitter look; anything richer belongs to a future curve editor. Risks Stale raster mid-stagger: a cached bitmap served during the window would freeze the motion. Covered by the active-window cache bypass and a regression test; the phase-suffixed keys cover the held frames. Word-count drift: the engine counts words by whitespace split, the rasterizer by the same split during wrap. A divergence would mis-time later words. Both sides share the tokenization rule (countStaggerUnits is documented against the rasterizer) and the compile cache re-keys on count. Raster churn during playback: while a stagger is active over a playing video, every frame re-rasterizes and re-uploads the text. Accepted (caption precedent); if profiling ever flags it, quantizing localMs to the frame rate inside resolveTextStaggerContext is a local fix.--- # Native game artistic roadmap Source: https://docs.nodetool.ai/plans/native-game-artistic-roadmap.html Native game artistic roadmap Purpose Give creators bloom, music, ambient motion, scrolling backgrounds, typography, and lighting without writing a script for each visual detail. Preserve the JSON document, deterministic simulation, standalone export, and agent playtest model in the engine design. This is a proposed implementation plan. The first release delivers an effect chain and bloom. Later phases can ship independently after their prerequisites. The findings below retain the identifiers from the architecture review. Starting points Finding Current implementation Consequence F9 packages/protocol/src/game.ts permits one brightness/contrast effect. The GPU package already has packages/gpu/src/shaders/filters/glow/v1/module.ts and packages/gpu/src/shaders/color/cubeLut/v1/module.ts. Integrate the existing modules, including their different color-space contracts. F10 packages/game-renderer/src/webgpu.ts composites HUD before effects. packages/game-renderer/src/node.ts has no post-processing. Define HUD order and capture capability before exposing bloom. F11 packages/game-renderer/src/frame.ts covers sprite position only. packages/game-renderer/src/canvas2d.ts retains a bitmap for each color. Smooth tweens require more interpolation and bounded tint memory. F12 web/src/components/workspace/GameSurface.tsx and packages/game-renderer/src/standalone-player.ts use different playback implementations. Share audio lifecycle behavior before adding persistent music. F13 packages/game-nodes/src/nodes/game.ts copies bytes and selects nearest sampling. packages/protocol/src/game-slot-prompt.ts already accept shared style entities. Add preparation and visual-reference propagation to the existing pipeline. F14 There is no light component or font asset kind in the document. Ship embedded fonts and simple point lights separately from advanced lighting. Shared decisions Decision Contract D1 Visual effects never affect collision, randomness, score, or simulation event order. D2 Animation and logical audio timing derive from ticks. Rendering interpolates between simulation samples. Hardware audio clocks do not enter snapshots. D3 Preserve existing documents and one-effect behavior. New fields have explicit defaults. Introduce a version boundary before changing persisted semantics that older engines would misinterpret. D4 Invalid effect parameters or assets fail validation. An unsupported optional effect may be omitted with a diagnostic. An unsupported required effect fails before play or capture. D5 Export contains every required asset and works without backend credentials. Browser and headless paths report their capabilities and omissions. D6 Each phase includes bounded resource use, lifecycle cleanup, authoring guidance, and a reproducible fixture. Performance claims require measurements on a named device and viewport. Delivery order Action Deliverable Dependency A1 Ordered effects, bloom, HUD compositing, and capture behavior First release A2 Asset preparation and shared visual references Can proceed alongside A1 A3 Looping music, volume, and shared playback lifecycle Independent of A1 A4 Declarative visual animation and interpolation Independent of A1 A5 Scrolling layers and parallax Reuse A4 camera interpolation A6 Embedded fonts Extend A2 asset handling A7 LUT grading A1 effect chain and asset validation A8 Ambient light and colored point lights A1 compositing and capability handling A7 can follow A1 immediately when color grading is the priority. It is a separate change because LUT assets and color conversion need their own validation. A1: Effect chain and bloom Extend renderEffects to an ordered, strict union of brightness/contrast and bloom. Keep existing effect fields valid. Start with a maximum of eight effects, finite parameter ranges, and rejection of unknown kinds. Expose threshold, softness, radius, intensity, and the existing required capability policy. Replace the renderer’s single-effect API with a chain API. Update the browser wrapper, workspace player, standalone player, and capture path together. Reuse the GPU executor and glow recipe. Use reusable intermediate targets, count scratch allocations in renderer statistics, and release them on resize, replacement, device loss, and disposal. Record the actual color encoding of uploaded images and render targets. Convert between linear and sRGB where modules require it. Relabeling a texture is not a conversion. Preserve premultiplied alpha through the chain. The first bloom implementation uses the existing 8-bit recipe and makes no HDR claim. For new bloom chains, composite world content, then effects, then HUD. Preserve legacy brightness/contrast output with an explicit compatibility policy. Make the choice visible in the schema when both orders are supported. Keep Canvas2D capture available and report omitted optional effects. Add an explicit GPU capture mode that uses the same effect implementation as the browser, with deterministic resource teardown. Required effects must never produce a silently degraded capture. Include culling margins or overscan so bright objects just outside the viewport can contribute visible glow. Acceptance criteria: A1.1: Existing documents validate and retain their prior rendering behavior. A1.2: Two noncommuting effects produce different pixels when their order is reversed. An empty chain matches the existing path without effect targets. A1.3: A bright sprite produces a halo outside its opaque bounds. Transparent edges retain correct alpha, and HUD text stays unaffected in the new mode. A1.4: Unsupported optional effects produce a visible diagnostic. Required effects fail in fallback, capture, and device-loss scenarios. A1.5: Repeated resize and chain replacement do not accumulate targets. Browser and GPU capture agree within an explicit pixel tolerance. A1.6: Record frame time and target memory for effects disabled and enabled, including the configured maximum chain. Set default radius and quality from that evidence before shipping. A2: Prepare generated assets Extend staging with explicit preparation settings for alpha trimming, target size, crop policy, pivot, sampling, and optional mirror tiling. Preserve original dimensions and crop offsets so trimming does not shift placement. Verify how pivots are consumed by each renderer before exposing new pivot behavior. Transform bytes before calculating their digest. Bind the prepared asset to those exact bytes and validate decoded dimensions. Preserve the existing installation digest check. Share one selected style image across generation calls using supported provider inputs, building on the existing style entity. Record the reference and preparation settings in provenance. Acceptance criteria: A2.1: Repeating preparation with identical inputs and settings produces the same output digest. Changed output bytes cannot pass the old digest check. A2.2: Trimming preserves the sprite’s intended anchor. Mirror tiling has no visible seam under the selected sampling mode. A2.3: Linear sampling survives staging, installation, and standalone export. A2.4: Inspect the generation request to verify the shared image reference is supplied. Do not treat matching prompt text as proof of visual consistency. A3: Music and audio lifecycle Move browser playback behavior into a shared module used by both players. Extend the audio contract with stable voice identity, start/stop, loop, volume, and fades. Define scene-owned music and its logical start tick. Retain one-shot events for effects and allow overlapping voices within a documented voice cap. Pause suspends playback. Reset stops voices and starts the entry scene’s music once. Scene replacement stops departing scene voices. Save/load restores logical music position without replaying old one-shot events. Derive resume offset from ticks and decoded duration. Browser gesture requirements and decoding failures produce actionable status messages. Acceptance criteria: A3.1: Workspace and standalone players pass the same lifecycle scenarios. A3.2: Pause, reset, load, and repeated scene changes never duplicate music. A3.3: Gain and fade scheduling are tested through the playback boundary. A browser test verifies playback after a user gesture. A3.4: Snapshots and headless tests verify logical music state. They do not assert identical samples across audio devices. A4: Declarative visual animation Add bounded visual tracks for rotation, scale, opacity, and tint, with tick duration, delay, repeat, ping-pong, and a small easing union. Add a constant rotation-rate option for continuous spins. Evaluate tracks from entity age, including authored entities after scene entry and spawned instances after load. Tracks affect visuals only. Use a documented composition order: authored value, track result, explicit script override, then lifetime fade/scale. Scripts keep their existing collision and position semantics. Reject conflicting tracks on the same property unless composition is explicitly represented. Interpolate visual properties and the camera in the shared frame path. Retain unwrapped rotation for deliberate full turns. Reset interpolation across teleports and scene changes. Bound Canvas2D tint caching before enabling tint animation, and apply the same color interpolation convention to every backend. Acceptance criteria: A4.1: Save/resume matches continuous playback at delay, loop, and ping-pong boundaries, including spawned entities. A4.2: Intermediate render fractions produce intermediate visual values and camera positions without changing simulation snapshots. A4.3: Script overrides and lifetime effects follow the documented order. A4.4: A long tint animation has bounded cache memory. A5: Scrolling backgrounds Add a visual layer component with per-axis parallax, scroll rate, repeat/mirror mode, and origin. Define parallax zero as screen-fixed and one as world-fixed. Compute scroll from ticks and interpolate it with the same camera sample as sprites. Keep repeating layers independent of collision and entity spawning. Acceptance criteria: A5.1: Camera motion, zoom, and viewport resize preserve the documented parallax behavior in all render backends. A5.2: Repeat boundaries have no seams, including negative offsets and long sessions. Atlas frames cannot sample neighboring frames. A5.3: Save/resume and arbitrary-tick captures reproduce layer positions without a script per layer. A6: Embedded fonts Add a font asset kind and logical font reference on HUD labels. Extend export validation and load the same bytes through browser font loading and the headless canvas font registry. Wait for required fonts before first capture or play. Use stable family identifiers to avoid collisions between games. Acceptance criteria: A6.1: An exported game loads its font offline with the existing CSP policy. A6.2: Required missing or invalid fonts fail clearly. Optional fallback is reported and does not depend on load timing. A6.3: Browser and headless text use matching metrics within a declared tolerance. Do not promise identical font rasterization across platforms. A7: LUT grading Add a LUT effect referencing a content-addressed asset. Initially use the packed 2D layout expected by the existing cube LUT module. Validate dimensions against cube size, opaque data, intensity, and domain bounds. Preserve lookup values during image loading and export. Add explicit color conversion around the module and retain bloom-before-grade as an authoring preset, not a forced order. Acceptance criteria: A7.1: An identity LUT preserves test colors within a declared tolerance. A7.2: Invalid dimensions and missing assets fail before rendering. A7.3: Grading works offline and respects chain order, alpha, and HUD policy. A8: Basic 2D lighting Add scene ambient light and point lights with position, color, intensity, radius, and falloff. Render a bounded light set into a reusable light texture, then multiply it over world color. Apply post-processing afterward and HUD last. Define an unlit/emissive sprite option so glows can remain visible in dark scenes. Use the same required/optional capability policy as effects. Acceptance criteria: A8.1: Multiple colored lights combine predictably without affecting gameplay. A8.2: Emissive sprites, HUD, camera movement, and scene replacement follow the documented compositing and lifecycle rules. A8.3: Validate light-count limits and measure target memory and frame time at the configured maximum. Verification and delivery Use public document, session, renderer, staging, and export boundaries. Extend packages/game-renderer/tests/frame.test.ts, packages/game-renderer/tests/webgpu-effect.test.ts, packages/game-renderer/tests/build.test.ts, packages/game-runtime/tests/session.test.ts, and packages/game-nodes/tests/native-game-nodes.test.ts. Reuse the GPU package’s recipe, alpha, and color-grading tests for module contracts. New assertions must fail on deliberately invalid fixtures. Each phase supplies a small deterministic game fixture, browser verification, and headless coverage where supported. Include device loss, scene replacement, and disposal in tests for persistent resources. Measure software-adapter correctness separately from hardware performance. For code changes, build packages first, then run the AGENTS.md mandatory post-change verification. Update CLI guidance and the packages/system-skills/native-game/SKILL.md when their contracts change. Keep each action reviewable independently and split shared protocol changes from later features when necessary. Deferred scope Custom document shaders, HDR bloom, normal maps, shadow casting, positional audio, and generated frame-consistent sprite sheets require separate designs. Multiply/screen sprite blending remains a separate renderer extension. Shared script storage, a collision spatial grid, and same-tick script reactions belong to the simulation roadmap and are not prerequisites for these artistic tools.--- # Native game benchmark baselines Source: https://docs.nodetool.ai/plans/native-game-baselines.html Native game benchmark baselines These measurements use Linux x64, an AMD Ryzen 5 3600 CPU, Node 24.18.0, seed 1 and the headless simulation backend. The CPU runs have no browser. Runs execute sequentially without concurrent repository tests or captures. Reproduce From the repository root: mise exec node@24.18.0 -- npm run dev:nodetool -- game bench packages/game-runtime/bench/bench-2d-500.json --ticks 1200 --warmup 300 --json The fixture generator creates the workload documents and their checked-in image and model assets. The 2D workload has 500 entities, a tilemap and 32 active scripted behaviors. The original plan’s 50 scripts exceed the runtime’s limit of 32. The fixture uses the supported limit, as agreed for B1. CPU measurements The initial short sampling window warms up for 300 ticks and measures 1,200 ticks. The scale workload also uses the longer window documented below. Tick time includes simulation and render-frame construction. Script time comes from the runtime’s script batch statistics. The allocation profiler runs separately for 10 ticks, with a 32 KiB sampling interval. Its byte estimate includes sampled allocations that have already been collected. It excludes WASM allocations and is not an allocation count. Heap delta is measured across the timing pass and can vary with garbage collection. The original CPU measurements below were collected without per-system instrumentation. The benchmark now enables recordTimings and returns perSystemMs, a map of stage names to p50, p95 and p99 durations. The existing perSystemUnavailableReason field is null. Warmup and allocation-profiler ticks do not contribute stage samples. Missing measured timings fail the run. Instrumented runs include timing clock calls and records in the tick and allocation measurements. Their results are recorded separately from the original untimed baselines. Stage percentiles do not sum to the outer tick percentile. Default runtime sessions keep timing disabled. 2D workload All latency columns are milliseconds. Both runs execute 38,400 script calls. Run Tick p50 Tick p95 Tick p99 Script p50 Script p95 Script p99 Heap delta bytes Sampled allocation bytes 1 16.790 23.918 25.355 16.333 19.281 24.857 349435264 36312968 2 17.230 24.409 26.952 16.782 23.509 26.349 386243392 38342192 Tick p95 differs by 2.05%, within B1’s 10% repeatability requirement. 3D workload support probes The planned 1,000-entity fixture with 30 scripts cannot complete its first simulation tick. Its script input is 132,746 bytes, exceeding the runtime’s 64 KiB logical-tick input budget. No latency baseline exists for that workload. Smaller temporary workloads retained all 30 scripts and the same entity order. The 400-entity and 128-entity probes passed the input-size boundary but failed the 50 ms script batch execution budget at tick 0 during isolated runs. A 64-entity probe completed 120 ticks with seed 1 and verified replay from tick All 30 script state counters reached 120. This demonstrates supported execution for the smaller probe, rather than latency repeatability. The agreed fixtures separate the workloads: bench-3d-1000 has 1,000 entities, 100 dynamic bodies, 10 models and no scripts. bench-3d-64-scripted has 64 entities, 61 dynamic bodies, 10 models and 30 scripts. Runtime budgets remain unchanged. 3D scale workload Run Tick p50 Tick p95 Tick p99 Script p50 Script p95 Script p99 Script calls Heap delta bytes Sampled allocation bytes 1 10.655 11.564 11.991 0.000 0.000 0.000 0 17505624 53457264 2 11.224 12.883 14.113 0.000 0.000 0.000 0 13737696 51084272 3 10.988 12.336 13.052 0.000 0.000 0.000 0 17959360 53769088 The initial pair differed by 11.4% in tick p95, outside the 10% target. An additional sequential run differs from run 2 by 4.25%. This does not establish repeatability for the original sampling window. All runs are retained above, and a longer fixed sampling window is measured separately. 3D scale workload with a longer sampling window This fixed configuration warms up 3,000 ticks and measures 12,000 ticks per run. Both consecutive runs use the same fixture, seed, runtime and idle machine. mise exec node@24.18.0 -- npm run dev:nodetool -- game bench packages/game-runtime/bench/bench-3d-1000.json --ticks 12000 --warmup 3000 --json Run Tick p50 Tick p95 Tick p99 Script calls Heap delta bytes Sampled allocation bytes 1 11.030 12.262 13.231 0 408115576 51511632 2 11.056 12.317 13.240 0 407080312 53461296 Tick p95 differs by 0.44%, within the 10% repeatability target. 3D scripted workload Run Tick p50 Tick p95 Tick p99 Script p50 Script p95 Script p99 Script calls Heap delta bytes Sampled allocation bytes 1 30.868 37.466 39.779 29.727 36.320 38.605 36000 327839544 43125312 2 30.859 37.546 39.444 29.734 36.205 38.225 36000 328527016 44374904 Both runs execute 36,000 script calls, confirming 30 active scripts per measured tick. The scripted workload tick p95 differs by 0.21%, within the 10% target. Instrumented CPU measurements These paired runs use the same Linux x64 machine, AMD Ryzen 5 3600 CPU, Node 24.18.0, seed 1 and headless simulation backend described above. Each pair ran consecutively without competing tests, builds or captures. The benchmark enables recordTimings. These results include its timing overhead and are separate from the original untimed tables. Runtime budgets and the three approved fixtures are unchanged. Every producer exited successfully. 2D: 500 entities and 32 scripts Each run warms up 300 ticks and measures 1,200 ticks. Latency columns are milliseconds. Run Tick p50 Tick p95 Tick p99 Script calls Heap delta bytes Sampled allocation bytes 1 17.029 24.756 27.250 38400 -63221592 37912264 2 16.852 23.990 26.080 38400 515282096 37630344 Tick p95 differs by 3.19%, within the 10% requirement. Stage Run 1 p50 Run 1 p95 Run 1 p99 Run 2 p50 Run 2 p95 Run 2 p99 input 0.000812 0.001272 0.001653 0.000712 0.001113 0.001552 scripts 16.619243 23.997221 26.796242 16.459934 23.085603 25.331304 physics 0.046388 0.065453 0.080281 0.045846 0.056185 0.068328 contacts 0.028654 0.039004 0.050004 0.028563 0.036990 0.047510 gameplay 0.031790 0.041678 0.049843 0.031319 0.040156 0.050405 presentation 0.270109 0.382591 0.787022 0.268976 0.315885 5.111738 3D scale: 1,000 entities and no scripts Each run warms up 3,000 ticks and measures 12,000 ticks. Latency columns are milliseconds. Run Tick p50 Tick p95 Tick p99 Script calls Heap delta bytes Sampled allocation bytes 1 11.068 12.259 13.340 0 382235136 54619056 2 11.052 12.227 13.265 0 379609568 52293896 Tick p95 differs by 0.26%, within the 10% requirement. Stage Run 1 p50 Run 1 p95 Run 1 p99 Run 2 p50 Run 2 p95 Run 2 p99 input 3.281411 3.797092 4.347820 3.268066 3.849762 4.317332 scripts 0.015459 0.036379 0.048011 0.015529 0.036919 0.050105 character 0.532874 0.639965 1.283036 0.526472 0.635818 1.284519 physics 0.030497 0.038703 0.044524 0.029615 0.037792 0.043361 contacts 0.248678 0.300656 0.381238 0.249390 0.298072 0.372131 gameplay 0.053370 0.068048 0.083678 0.051367 0.066004 0.081915 animation 0.117551 0.144482 0.798464 0.116480 0.143240 0.794156 presentation 6.728073 7.588223 8.385344 6.733403 7.594915 8.343425 3D scripted: 64 entities and 30 scripts Each run warms up 300 ticks and measures 1,200 ticks. Latency columns are milliseconds. Run Tick p50 Tick p95 Tick p99 Script calls Heap delta bytes Sampled allocation bytes 1 31.509 37.868 41.056 36000 512130912 41705824 2 30.650 37.019 39.055 36000 466517464 44029192 Tick p95 differs by 2.29%, within the 10% requirement. Stage Run 1 p50 Run 1 p95 Run 1 p99 Run 2 p50 Run 2 p95 Run 2 p99 input 0.203724 0.251975 0.293192 0.194046 0.218371 0.271141 scripts 30.392567 36.749200 39.759770 29.693920 35.835438 37.745516 character 0.230043 0.285247 0.339780 0.176072 0.265681 0.328619 physics 0.031650 0.039715 0.046908 0.023344 0.034936 0.041728 contacts 0.113213 0.158128 0.191210 0.101231 0.140164 0.176563 gameplay 0.018906 0.024487 0.029406 0.014707 0.021561 0.026940 animation 0.041658 0.052058 0.064361 0.030988 0.049273 0.056626 presentation 0.435951 0.524288 0.626871 0.423799 0.468332 0.598518 S1 script execution comparison These measurements compare baseline commit 0d460e01d6573b1332411e72a6f89e8c684146ed with the S1 candidate in /home/mg/nodetool-game-s1-persistent, based on 40c814ce. Both ran on the Linux x64 Ryzen 5 3600 host with Node 24.18.0, using the headless simulation backend and instrumented stage timings. Each command warmed up 300 ticks and measured 1,200 ticks with seed 1. All six commands exited 0, with no retries. The fixed order was all three baseline fixtures, then all three candidate fixtures, in the table order. The fixture SHA-256 values matched between trees. For each fixture, run this command in each tree: npm run dev:nodetool -- game bench packages/game-runtime/bench/bench-3d-64-scripted.json --ticks 1200 --warmup 300 --seed 1 --json The preserved evidence directory is /home/mg/native-game-verification/s1-bench-v2/. plan.json records each command and tree, reports.json holds the unrounded benchmark output, and results.json and individual .exit files record producer exit status. Individual .log files retain raw output. fixture-sha256.json records the fixture comparison. The tables below round latency to six decimal places. Tick and scripts-stage latency All latency columns are milliseconds. The scripts-stage values are perSystemMs.scripts, including host work in that stage, rather than the narrower scriptMs batch metric. Fixture Version Tick p50 Tick p95 Tick p99 Scripts-stage p50 Scripts-stage p95 Scripts-stage p99 bench-2d-500 Baseline 16.856454 23.725450 25.552892 16.466530 23.019691 24.980584 bench-2d-500 S1 4.558845 11.679936 13.684731 4.127222 11.227555 13.261465 bench-3d-1000 Baseline 11.184954 12.932535 14.244866 0.017333 0.046387 0.067407 bench-3d-1000 S1 11.291975 12.958364 13.835095 0.018144 0.044194 0.053621 bench-3d-64-scripted Baseline 30.709824 37.006212 40.006171 29.605855 35.893166 38.881433 bench-3d-64-scripted S1 4.887033 11.771258 13.144434 3.927657 10.556241 11.939486 The scripted 3D candidate’s scripts-stage p99 is 11.939486 ms, below the S1 acceptance target of 25 ms. The no-script scale fixture still performs scripts-stage bookkeeping, so its stage time is nonzero while its script batch time is zero. Script batches and allocation samples Fixture Version Script p50 ms Script p95 ms Script p99 ms Script calls Heap delta bytes Sampled allocation bytes bench-2d-500 Baseline 16.387742 22.901769 24.764216 38400 350683704 37384608 bench-2d-500 S1 4.029318 11.051893 12.922686 38400 382348640 42770456 bench-3d-1000 Baseline 0.000000 0.000000 0.000000 0 -4624184 52636320 bench-3d-1000 S1 0.000000 0.000000 0.000000 0 -3901520 50575096 bench-3d-64-scripted Baseline 29.548737 35.843954 38.820819 36000 336398792 40364256 bench-3d-64-scripted S1 3.882923 10.503922 11.887397 36000 349136928 37639104 Allocation samples use the separate 10-tick V8 sampling pass with a 32,768-byte interval. They include collected allocations and exclude WASM allocations. Heap delta varies with collection timing. These measurements show mixed allocation and heap changes, not a general reduction in memory use. Scope of the comparison This is one sequential before/after pair per fixture, not a repeatability study or a claim about all authored scripts. No concurrent repository build, test, or capture ran during the measurements, but the host was not idle. The preserved *.load-before files show one-minute system load averages from 2.13 to 3.34. The fixed run order does not remove background load or thermal effects. The approved scripted fixtures use the positive AST fast path. Only input-only expressions proven unable to retain hidden state qualify. Sources are validated in temporary contexts during preparation. A qualifying behavior gets its resident function on the first active call, then reuses it while active. Arbitrary sources retain fresh-context execution and existing guest JSON parsing through native string arguments. The resident path passes native objects and lazily copies legacy input.world. Both paths retain JSON normalization and existing budgets. These warm measurements exclude first-call compilation from their measured ticks. Fresh-context follow-up The external-review follow-up combines trusted helper initialization into one evaluation, removes a discarded fallback input copy, and skips the full 2D query snapshot when no scripts are active. The unchanged canonical commands above each exited 0 on this candidate. Scripts-stage p99 was 11.633391 ms for 2D 500/32, 0.051327 ms for 3D 1000/0, and 11.697422 ms for 3D 64/30. These are additional candidate observations against the historical baseline, not a newly paired before/after comparison. A separate serial probe measured one fresh-context call per batch using Math.abs and a local binding, an empty world, seed 1, 100 warmup calls, and 1,000 measured calls. Three rounds rotated the original baseline, published S1 commit ea8ca86abc, and the follow-up candidate. All nine commands exited 0 with the original 20 ms call and 50 ms batch limits. Runtime Mean batch time range across rounds, ms Original baseline 0.623996–0.637125 Published S1 0.712999–0.731253 Follow-up 0.671325–0.710250 The follow-up reduced mean cost by 2.9–5.8% relative to published S1 in this probe. It remained 7.6–11.5% above the original baseline. This small empty-world workload therefore does not establish fallback performance parity. It also does not reproduce the external review microbenchmark, whose exact command was not available. The canonical scripted fixtures exercise the persistent path. Raw samples, commands, actual exits, load readings, and the candidate source manifest are preserved in /home/mg/native-game-verification/s1-external-review-v1/. Browser rendering measurements These runs use Chromium 148.0.7778.96 on the same machine, the WebGL2 backend, and a 960 by 540 viewport. Each warms up 60 frames and measures 600 renders of the same tick-0 frame. gl.finish() completes GPU work before timing ends. The browser measurements isolate rendering and do not execute scripts. mise exec node@24.18.0 -- node --conditions=nodetool-dev --import=tsx packages/game-renderer/scripts/benchmark-effects.ts --game packages/game-runtime/bench/bench-3d-1000.json --assets-dir packages/game-runtime/bench/assets mise exec node@24.18.0 -- node --conditions=nodetool-dev --import=tsx packages/game-renderer/scripts/benchmark-effects.ts --game packages/game-runtime/bench/bench-3d-64-scripted.json --assets-dir packages/game-runtime/bench/assets Fixture Run Frame p50 ms Frame p95 ms Frame p99 ms Draw calls Triangles bench-3d-1000 1 52.500 62.800 108.600 997 11844 bench-3d-1000 2 52.400 63.000 108.700 997 11844 bench-3d-64-scripted 1 36.900 73.500 76.600 62 624 bench-3d-64-scripted 2 37.100 74.400 78.000 62 624 The scale workload frame p95 differs by 0.32%. The scripted companion differs by 1.22%. Both satisfy the 10% rendering repeatability target.--- # Native game editor audit Source: https://docs.nodetool.ai/plans/native-game-editor-audit.html Native game editor audit Scope: the 2D and 3D game editor in web/src/components/game/, the draft store and merge code in web/src/stores/game/, and the games API (packages/websocket/src/trpc/routers/games.ts, packages/models/src/game.ts). Line numbers refer to main as of 2026-10-03. Confidence labels: Reproduced means a test or script showed the failure. Traced means the code path was read end to end. Plausible means the mechanism is in the code but the bad outcome depends on timing. Fixed in PR #6040 ID Finding Evidence F1 Undoing the addition of a parent and its child generated remove_entity for both. The parent’s removal deletes the child, so the child op failed, and every later 2D draft save failed until a reload. Reloading discards the edits. Reproduced: diffGameDocuments.test.ts failed with “Entity child does not exist” before the fix. F2 Ctrl+V kept a pasted child attached to the original parent. Pasting into another scene left a dangling parentId. Traced. gameClipboard.test.ts covers the new behavior. F3 The toolbar showed “Unpublished changes” for both unsaved edits and failed saves. Traced. A test was added. Open findings, highest priority first Data loss and stuck saves F4. High, plausible. One rejected batch blocks every later save (2D and 3D). Where: GameDraftStore.ts:54-59, GameEditor.tsx:155-185, games.ts:349-353. Cause: after a merge, applyMerged diffs the merged document into pendingOps without validating it. Scenario: the user adds a sprite on slot hero while the agent unbinds hero. The merge has no conflict, the server rejects the result with “references missing asset”, and the client resends the same ops on every edit. Plain Errors from trackGameAuthoringEdits behave the same way and are reported as INTERNAL_ERROR. Fix direction: validate after a merge, and when the server rejects with an unchanged token, drop the bad ops or reload the draft instead of retrying. F5. High, plausible. The 3D editor can silently lose edits made while a save is in flight. Where: GameEditor3D.tsx:51-69. Cause: pull() does not wait for the save in progress. The 2D editor does, at GameEditor.tsx:103. Scenario: the save’s own resource_change echo triggers a merge. applyMerged resets savingCount. acknowledge then drops the edits made during the save. F6. Medium, traced. restore, and publish with an explicit document, delete unpublished draft work without a draft token. Where: games.ts:198-228, games.ts:431-439, game.ts:465-506. After the publish, the server deletes the draft file and the whole change history, so the overwritten draft cannot be recovered. F7. Medium, traced. A missing draft version file makes the game uneditable and unpublishable. Where: game.ts:215-221. readDraft throws, and every API path calls it first. Triggers are plausible: change-history pruning or post-publish cleanup can delete a hash-named file that a concurrent writer has just made current again (game.ts:429-442, game.ts:498-503). F8. Medium, traced. After a failed save, the 2D save stays “error” until the next edit. Where: GameEditor.tsx:155-185. If the user keeps their version on every conflict, nothing triggers autosave, and the red error stays visible. The 3D editor has a similar problem (GameEditor3D.tsx:83-88): a merge that resolves cleanly is overwritten with “error”. an offline pull leaves the status at “saving”. Accepting a conflict also moves baseUpdatedAt back to the token captured when the conflict was raised (2D and 3D). The next save is then guaranteed to be rejected as stale. Editing correctness F9. High, reproduced. Dragging, scaling or rotating a child entity in the 2D viewport moves it to the wrong place. Where: GameViewport.tsx:284, :339, :342-344. Cause: the viewport reads world coordinates from the rendered sprite and writes them into transform2d, which is local to the parent. Reproduced: with the parent at x=5 and the child at local x=2, the sprite is at x=7, and a drag writes local x=7+d, so the child jumps to world x=12+d. The same local/world mix-up affects: icon hit-testing, marquee selection, the collider and camera overlays, and the F (frame selection) key, which all treat transform2d as world (viewportGeometry.ts:34-38, GameViewport.tsx:117-167, :352-354). reparenting in the scene tree and inspector, which keeps local values, so the entity jumps. F10. Medium, traced. Delete and arrow-key nudge act only on the first selected entity, while copy takes every selected entity. Where: GameEditor.tsx:319-346. Dragging also moves only one entity. A multi-delete must skip descendants of other removed entities, the same rule as F1. F11. Medium, traced. In the 3D editor, Ctrl/Cmd+Z is captured for every text field in the main row. Where: GameEditor3D.tsx:158-163. This includes the assistant chat box and inspector inputs. The keystroke undoes the game document instead of the text, and it also fires during play. F12. Medium, traced. The inspector’s Parent list offers the entity’s own descendants. Where: GameInspector.tsx:204-206. Choosing one creates a cycle. Validation flags it, but the editor preview then fails with “Missing world transform”. F13. Medium, traced. While the script pane shows “draft changed”, typing is silently not saved. Where: GameScriptPane.tsx:63-67. The only option offered is “Use draft version”, which discards the typed text. F14. Medium-low, traced. In the 3D editor, concurrent flush() callers can send the same ops twice. Where: GameEditor3D.tsx:72, :91. Publish, model install or authoring preview can then fail with “modified concurrently”. Performance F15. High, traced. Every edit rebuilds the whole play stack. Where: useGamePlaySession.ts:156-235 (2D) and useGamePlaySession3D.ts:122-227 (3D). After a 100 ms debounce, each edit recreates: the session. the GPU renderer (WebGPU or three.js). the font loader. the audio player, which in 3D means a new AudioContext and audio re-preload. In 3D every GLB is re-fetched and re-parsed, the toolbar flickers to “Initializing”, and a gizmo drag started right after an edit is cut off. F16. Medium, traced. Each frame during play re-renders the whole 2D editor. Where: GameEditor.tsx:72, :352. Each render runs a JSON diff of the full play and draft documents (diffGameDocuments) and validateGame(document). The 3D editor does the same work at about 10 Hz (GameEditor3D.tsx:137, GameInspector3D.tsx:76). Fix direction: memoize both on document. F17. Medium, plausible. 3D change history grows without bound. Every 3D undo, redo and merge is a single set_document op holding the full document. Each one becomes a change row, up to 500 rows are kept, and draftChanges returns all of them every 5 s. saveDraft has no limit on the number of ops. Publish F18. Low-medium, traced. Publish sends a stale baseRevision from the React Query cache. Where: GameEditor.tsx:256, GameEditor3D.tsx:120. After another session publishes, publish keeps failing until the query refetches. F19. Low-medium, traced. Publish ships the server’s draft, not the document the user validated. Agent edits that land between flushDraft and the publish call are published unseen. The 3D publish has no validation, no change list and no loading state, so a double click publishes twice. F20. Medium, traced. Publish can report failure after it has committed. Where: games.ts:225-228. The cleanup after the commit is not wrapped in try/catch, so a storage hiccup returns a 500 although the revision is live. Retrying then fails as a concurrent modification. Smaller issues F21. In the 3D editor, operationError never clears after a later success (GameEditor3D.tsx:39). F22. 3D mouse input differs from the standalone player: pointer down never sends Mouse0, so the “fire” action cannot fire from the mouse. look works only while the right button is held, with no pointer lock (GameViewport3D.tsx:223-234). F23. 3D input pressed while play is paused piles up and fires on resume. F24. In the 3D editor, the authoring preview’s “highlight” callback clears the selection, and hovering a change in the Changes list highlights nothing (GameEditor3D.tsx:154-156). F25. Restoring a revision to the draft has no confirmation step and clears the undo history. F26. restoreDraft and saveDraft turn validation errors into 500s instead of INVALID_INPUT. F27. Orphaned draft files are left behind when a compare-and-set fails. Revision files are never pruned. F28. The replay history (inputHistoryRef) grows by one entry per tick for the whole play session. F29. “Save play state” calls localStorage.setItem without a try/catch, so a full quota throws an uncaught error. Reported by another thread and not reproduced F30. Unconfirmed. A crash after Stop in the 2D editor. The “Game Editor UI redesign” thread reported that web/tests/journeys/native-game-editor.spec.ts crashed after Stop on main in its container. It gave no error text. In the audit’s cloud container the same journey passed 5 times out of 5 on main’s game code, and once with PR #6040 applied. This container differs in two ways: it has a Vulkan ICD (mesa-vulkan-drivers), and its headless Chromium is aliased from build 1194. Possible cause, inferred and not verified: the crash could depend on the WebGPU fallback path. Stop rebuilds the renderer (see F15). The error text is needed before this can be diagnosed. The 3D journey’s save conflict while installing a model matches F5 and F14. Feature gaps in 3D compared with 2D No Delete, nudge, duplicate, copy or paste shortcuts. No multi-select. No Revisions panel. No script error tools: no replay, no “Run 10 s”, no “Ask the assistant”. No hierarchy or reparenting UI. No mobile layout. The script pane follows the behavior index, not the entity. Selecting another entity silently retargets the open pane. Neither the 2D nor the 3D editor has UI to add or remove scenes. Checked and found sound Draft tokens strictly increase, and the compare-and-set saves are atomic. Every games API procedure checks ownership. The 3D merge adapter fields match the schemas. Overlay, gizmo and session disposal in 3D.--- # Native game editor: assistant, drafts, and direct editing Source: https://docs.nodetool.ai/plans/native-game-editor.html Native game editor: assistant, drafts, and direct editing Purpose Make the native game surface an agent-first editor. The assistant does the large changes: it builds scenes, writes scripts, and generates art. The user reviews those changes and makes precise edits directly in the viewport and the inspector. This plan covers five parts: Part Deliverable A1 Assistant panel and the agent tools it needs A2 A mutable draft with reviewable agent changes A3 Direct editing in the viewport A4 An inspector generated from the game schema A7 A script editor The asset panel (A5), replay and debug tools (A6), and preview and share (A8) are later work. They build on this plan without changing it. The base is codex/native-game-artistic-features, which has schema version 2 (effect chain, lighting, backgrounds, visual tracks, music, fonts). Keep the engine contracts in the engine design and the artistic roadmap. Visual edits never change simulation determinism. Starting points Finding Current implementation Consequence F1 web/src/components/workspace/GameSurface.tsx is one component. Its inspector edits only transform2d.x and y. “Save scene” publishes a new immutable revision on every save. Each small edit makes a revision. Most of the document cannot be edited in the UI. F2 GameSurface.tsx hard-codes KEY_ACTIONS with the arrows and WASD. packages/game-renderer/src/standalone-player.ts also maps Space. An action that uses space works in the export but not in the editor. F3 packages/agents/src/capabilities/game.ts has create, get, publish (whole document), install, playtest, and build. Playtest reads only published revisions and stops at MAX_PLAYTEST_TICKS = 600 (10 s). No tool returns an image. The agent sends the full document for each edit, cannot test unpublished work, and cannot see the game. F4 install_native_game_asset and games.installCandidate map audio to .wav and images to .png. The build’s MEDIA_EXTENSIONS has no font types. Code reading shows that the agent path cannot install MP3 music or JPEG backdrops, and a build can omit installed fonts. This is not reproduced yet. S8 reproduces it first. F5 Game.publish (packages/models/src/game.ts) calls ModelObserver.notify without ops. SyncedDocumentType in web/src/stores/documentSync.ts has no game type. An open editor does not merge external game edits. F6 UiSurfaceType and ChatSource (packages/protocol/src/api-types.ts) have no game entries. UiContext.selection has no entity ids. The agent does not know which game or entity the user is looking at. F7 validateGame returns strings. Most strings start with a path such as scenes.0.backgrounds.1.assetId:, but not all of them. The UI cannot put an error next to its field reliably. Parts that already exist and that this plan reuses: AssistantChatPanel and ResizableSideDock (web/src/components/chat/assistant/), as used by JsScriptAgentPanel. The three-way merge engine web/src/stores/documentMerge.ts, ConflictStore, and the merge hook in documentSync.ts (document sync plan). persistOutput (packages/agents/src/tools/asset-persist.ts), the imageHandle shape, and composeContactSheet (packages/agents/src/timeline-preview/sheet.ts). With these, preview_timeline_frame returns images that the agent reads with view_image. captureGameFrame with backend: "webgpu" (packages/game-renderer/src/node.ts), which applies the effect chain in headless capture. web/src/stores/temporal.ts for undo history, and Monaco (@monaco-editor/react), which JsScriptEditorPane already uses. Decisions Decision Contract Reason D1 Each game has one mutable draft and immutable published revisions. Autosave and agent edits write the draft. Only “Publish” creates a revision. Autosave must not create a revision for each change. The other editors use the same model: a document plus versions. D2 The agent edits through one server capability, edit_native_game, with an ordered op list. There are no ui_game_* frontend edit tools. Edits then work headless (CLI, evals, closed editor), as for storyboard, sketch, and timeline. The ops also feed the merge engine. D3 One pure op reducer, applyGameOps(document, ops), lives in @nodetool-ai/game-runtime. The capability, the web draft store, and the eval bridge all call it. Agent edits and UI edits then have the same meaning, and one test suite covers both. D4 The agent sees the game only through rendered frames. The capture tool returns image handles, and view_image shows the pixels. Text state cannot show visual defects. The showcase games found every visual bug through captures. D5 The server logs each draft write with its actor and chat turn. The UI groups agent writes into change cards. “Undo” on a card reverses that turn through the three-way merge. The user can reverse one agent turn without losing later edits by hand. D6 Edit mode shows the authored state at tick 0 through an editor camera. Play mode runs a session from a copy of the draft. Stop discards the session. The simulation never writes into the draft, so a play session cannot corrupt authored data. D7 The inspector is generated from the Zod schemas through z.toJSONSchema, with a small table of widget overrides. New engine fields then appear in the inspector without a new form. D8 Scripts stay function expressions. Editor typings come from the script command schema, and a test detects drift. The script contract in packages/game-runtime/src/scripts.ts does not change. A1: Assistant panel and agent tools A1.1 Register the surface Add "game" to UiSurfaceType, and add game: "native game" to UI_SURFACE_LABELS in packages/websocket/src/session/chat-prompt.ts. Add "game_assistant" to ChatSource and to CHAT_SOURCE_LABELS. Add entity_ids to UiContext.selection. The prompt formatter already prints each selection key. In formatUiContext, add a game block like the timeline block. It tells the agent to: call get_native_game with view: "outline" first; edit with edit_native_game; look at the result with capture_native_game_frame before it reports a visual change as done; leave publishing to the user unless the user asks. A1.2 Panel Add web/src/components/game/GameAgentPanel.tsx, modeled on JsScriptAgentPanel: chatSource: "game_assistant"; focused: { type: "game", id, title }; getSelection returns { entity_ids } from the editor selection. Add gameAssistantPrompt(gameId) in web/src/components/game/gameAssistantPrompt.ts. It states the loop (outline, edit, capture, playtest) and the script contract. It links to the native-game skill and does not repeat it. Dock the panel with ResizableSideDock (storageKey: "game_assistant"). On mobile, show it in MobileBottomSheet, as in SketchEditor. Welcome examples: “Add a boss that appears at wave 5”, “Make the forest darker and add fog”, “Why does the player pass through this wall?” A1.3 Read the game compactly Extend get_native_game: Argument Values Default source "draft" or "revision" "draft" revision a full revision (with source: "revision") current view "outline", "full", or "entity" "outline" entity_id required with view: "entity" none view: "outline" returns: the game settings; for each scene, its backgrounds, lights, music, and a list of entities (id, name, parent, components present, behavior kinds, script byte size); the asset slots with kind and size. It does not return script sources or tile arrays. The showcase documents are mostly script source, so the outline keeps a read small. view: "entity" returns one entity in full. The response always includes draft_updated_at for compare-and-swap. A1.4 Edit with ops edit_native_game { game_id, base_updated_at?, ops[] } applies all ops to the draft, validates the result, and writes it atomically. When validation fails, it writes nothing and returns each issue with its op index and path (A4.5). When the caller omits base_updated_at, the tool uses the current draft and retries once on a conflict, as in the document sync plan. An entity target is { entity_id, scene_id? }. scene_id is required only when the id occurs in more than one scene. Op Arguments Effect add_entity scene_id, entity (partial), index? Adds an entity. Missing fields get schema defaults. update_entity target, set Deep-merges fields into the entity. A null component removes it. remove_entity target, children?: "remove" \| "reparent" Removes the entity. Rejects the op if a behavior still references it. duplicate_entity target, new_id, offset? Copies the entity and moves the copy by offset. add_behavior / update_behavior / remove_behavior / move_behavior target, index, behavior Edits the behavior list. set_script target, index, source, max_commands?, max_tick_ms? Replaces a script’s source. add_scene / update_scene / remove_scene scene_id, fields Edits name and music. remove_scene rejects the op if a transition still references the scene. set_lighting scene_id, lighting or null Replaces the scene’s lighting as one value. add_light / update_light / remove_light scene_id, index, fields Edits one point light. add_background / update_background / remove_background / move_background scene_id, id, fields Edits background layers by id. set_effects effects, hud_effect_order? Replaces the effect chain. set_game pixels_per_unit?, input_actions?, entry_scene_id? Edits the game settings. bind_asset / unbind_asset slot, binding Binds an installed asset to a slot, or removes the binding. unbind_asset rejects the op if something still references the slot. The reducer lives in packages/game-runtime/src/document-ops.ts, with tests for every op, for rejection paths, and for a list of 1,000 ops, which checks for accidental quadratic work. A1.5 See the game capture_native_game_frame: Argument Meaning game_id, source, revision? What to render. The default is the draft. ticks Up to 8 ticks. The default is [0]. inputs Run-length input, for example [{ pressed: ["right"], ticks: 30 }] seed The simulation seed. The default is 1. camera? { x, y, zoom } to override the game camera overlays? Any of colliders, ids, lights, camera_bounds scale?, sheet? Output scale, and a single contact sheet in place of separate images Run the session to each tick, and capture with backend: "webgpu" when the game has effects. When WebGPU is unavailable, fall back to Canvas2D and return the diagnostic. Return one imageHandle for each frame. Each frame also lists the visible entities with their screen boxes, the HUD texts, and the capture diagnostics. Draw the overlays in a pass after the capture, so they never enter the game pixels. A1.6 Playtest the draft Extend playtest_native_game: Add source: "draft" and use it as the default. Accept run-length inputs and raise the limit to 3,600 ticks (60 s). A run still stops at a wall-time budget, and the budget is reported. Add assertions, for example { at_tick, entity_id, near: { x, y, tolerance } }, { event: "win", before_tick }, and { no_script_errors: true }. Return each failed assertion with the state it saw. Add capture_ticks, which reuses A1.5 and returns image handles. Return a contact summary with counts per entity pair and phase, in place of the first 50 raw events. A1.7 Generate and install an asset generate_game_asset { game_id, slot, kind: "image" | "audio" | "music", prompt, reference_slot?, preparation?, provider?, model? }: Generate through the existing media paths (generate_image, generate_speech, generate_music). Use a background receipt when the model is slow. Prepare the bytes with the binding’s preparation settings, then compute the digest. Stage the candidate, install it, and bind it to the draft slot. Record referenceAssetId and provenance as the artistic roadmap (A2) specifies. Fix F4 in the same slice. Install keeps the source format (png, jpg, webp, wav, mp3, ogg, ttf, otf), and the build accepts fonts. The reproduction test comes first. A1.8 Publish from the draft publish_native_game { game_id, base_revision, message? } without document publishes the current draft. The document form stays for compatibility. The tool description says to publish only when the user asks. A1.9 Registration and guidance Update the following: game.specs.ts and packages/agents/src/capabilities/registry.ts; packages/cli/src/harness/capability-table.ts and packages/websocket/src/trpc/sandbox-coverage.ts; packages/system-skills/native-game/SKILL.md; the native game pipeline section. Add an eval surface, packages/agents/src/evals/surfaces/game.ts. It calls the real reducer (D3), so the eval tests real edit semantics. A1 acceptance A1.a: The prompt names the focused game and the selected entity ids. A1.b: An agent turn that adds an entity and captures tick 0 returns an image that shows the new sprite. A1.c: An invalid op list writes nothing, and the tool returns the op index and path for each issue. A1.d: The Lumen showcase document passes a draft playtest with { event: "win", before_tick: 1500 }. A2: Draft and change review A2.1 Persistence Add the columns draft_updated_at and draft_base_revision to games, with a migration. Store the draft at games/<id>/draft.json through the workspace interface. A game without a draft file reads its current revision as the draft. Add Game.updateDraft(user, id, expectedUpdatedAt, ops), which does a compare-and-swap on draft_updated_at. It calls ModelObserver.notify(game, UPDATED, { ops }), so resource_change carries the ops. Add games.getDraft and games.saveDraft({ id, baseUpdatedAt, ops }) to packages/websocket/src/trpc/routers/games.ts. The web saves ops, not whole documents, so the server log (A2.4) is the same for both actors. games.publish takes the draft when no document is given. It sets draft_base_revision to the new revision. A2.2 Merge into an open editor Add "game" to SyncedDocumentType, and route it in resourceChangeHandler.ts. Add a merge adapter, web/src/stores/game/merge.ts, with these units: Unit Key Rule Scene scene.id The scalars name and music are last-write-wins. Entity (scene.id, entity.id) The fields transform2d, each component, and behaviors are separate fields of the unit. A move and a sprite change do not conflict. Script (scene, entity, behavior index) One value, never merged as text. Background (scene.id, layer.id) One unit for each layer. Lighting scene.id One unit, because point lights have no ids. Asset binding slot One unit for each slot. Effects the document One unit. Game settings the document Last-write-wins for each field. An entity whose parentId names an entity that the draft deleted is dangling. The merge drops it and lists it. A2.3 Web draft store Add web/src/stores/game/GameDraftStore.ts (Zustand with temporal() undo, one instance for each game). Every UI edit calls applyGameOps and queues the ops for autosave, with a 500 ms debounce. A drag or a text entry becomes one history entry. The store registers with registerDocumentSync("game", id, { reload, merge, isDirty, localRevision }). It applies merged results with history paused, as the timeline slice does. A2.4 Change log Add a table, game_draft_changes (id, game_id, actor, thread_id, message_id, ops, summary, before_updated_at, created_at). edit_native_game writes actor: "agent" with the chat thread and message ids from the run context. saveDraft writes actor: "user". Keep the most recent 500 rows for each game, and delete rows older than the current revision after a publish. The server computes summary from the ops, for example “Added 2 entities; changed the script on wisp; moved 3 lights”. A2.5 Change cards Show a “Changes” list above the viewport, with one card for each agent message since the last publish. Each card shows the summary and the affected entities. Hover highlights the entities in the viewport. “Undo” on a card runs the merge engine with these inputs: base is the draft after the turn; draft is the current draft; server is the draft before the turn. The result reverts the agent’s units. A unit that the user edited after the turn stays, and the result lists it as a conflict. The undo is a normal user edit, so it goes through autosave and lands in history. Snapshots for undo: before each agent write, store the previous draft at games/<id>/drafts/<sha256>.json. Delete snapshots together with their change rows. The tool-call view in the chat links to the card. A2.6 Publish The toolbar shows “Unpublished changes” with a count. “Publish” opens a dialog with the change summaries since the last revision and an optional message. The revision list replaces the current “Revisions” column. It shows the message, the time, and “Restore to draft”. A restore writes that revision into the draft and does not publish it. A2.7 A draft change during play In edit mode, a draft change rebuilds the tick 0 frame after a 100 ms debounce. In play mode, a change to an asset binding updates the art at the next tick boundary, as engine design D1 allows. Any other change shows “The draft changed. Restart to apply.” It does not restart the session without the user. A2 acceptance A2.a: A hundred inspector edits in one minute create no revisions. A2.b: The agent moves entity A while the user edits entity B. Both edits stay, with no conflict. A2.c: The agent rewrites a script while the user has unsaved edits in the same script. The draft keeps the user’s text, and the conflict banner shows the agent’s version. A2.d: “Undo” on an agent card reverts only that turn’s units. A unit that the user edited later stays and shows a conflict. A3: Viewport editing A3.1 Structure Split GameSurface.tsx into web/src/components/game/: GameEditor (layout and mode); GameViewport (canvas and overlay); GameSceneTree; GameInspector (A4); GameScriptPane (A7); GameToolbar; useGamePlaySession (the current session, audio, and font code, unchanged in behavior). TabContent.tsx keeps loading one lazy component. A3.2 Modes and camera Edit mode renders session.frame() from a fresh session at tick 0, with an editor camera { x, y, zoom } and a free viewport size. The frame is data, so the override replaces camera, width, and height before render. Pan with space-drag or the middle mouse button, and zoom with the wheel toward the pointer. “F” frames the selection. “Home” returns to the game camera. Play mode runs the draft copy with the game camera (D6). Pause shows the runtime entities. Selecting a runtime entity shows its session.inspect() state as read-only. Move the key map into one shared module in @nodetool-ai/game-renderer, which both players use. This fixes F2. A3.3 Overlay and picking Draw an HTML canvas above the game canvas. It uses the same world-to-screen transform as the renderer (projectedCamera and pixels per unit), exported from packages/game-renderer/src/frame.ts. Picking uses frame.sprites (entity id, box, rotation, layer), and the topmost layer wins. Entities without a sprite (colliders, cameras, empty parents) show small icons, which can also be selected. Alt-click cycles through the stacked entities under the pointer. Overlays, each with a toggle: selection boxes; collider outlines, in a color for each collision category; the camera bounds; point lights, drawn as a dot with a radius ring; background layer origins; a grid. A3.4 Gizmos and commands Input Result Drag the selection Move. Snaps to the grid (default 0.25 units, toggle “G”). Shift locks the axis. Drag a corner handle Scale scaleX and scaleY. Shift keeps the aspect ratio. Drag the rotation handle Rotate. Shift snaps to 15 degrees. Drag a light dot or its ring Move the light, or change its radius. Arrow keys Nudge by one grid step. Shift nudges by ten. Marquee, shift-click Multi-select. Delete, Cmd+D, Cmd+C/V Remove, duplicate, copy, paste (as add_entity ops with new ids). Cmd+Z, Shift+Cmd+Z Undo and redo in the draft store. Every gizmo commits update_entity or light ops at the end of the drag. During the drag, only the overlay and a local frame override update, so the drag does not rebuild the session. A3.5 Scene tree The scene tree shows the scenes, with their entities under each scene: nested by parentId, with drag-to-reparent and reorder; templateOnly entities in a “Prefabs” group; search by name or id; a “+” menu with presets: sprite, static wall, collectible, trigger, camera, empty. Selection is shared between the tree, the viewport, and the assistant (entity_ids). A3 acceptance A3.a: Drag, scale, and rotate each produce one undo entry and one autosave, and a capture shows the same result. A3.b: Picking is correct for rotated sprites and at zoom levels 0.25 and 4. A3.c: An action mapped to Space works in the editor and in the export. A3.d: Play, then Stop, leaves the draft byte for byte unchanged. A4: Inspector A4.1 Generation web/src/components/game/inspector/schemaForm.ts converts gameEntity, gameScene, and the game-level fields with z.toJSONSchema. It renders them with ui_primitives fields: numbers use NumberInput; enums use Select; booleans use Switch; objects use collapsible sections; arrays use lists with add, remove, and reorder. Defaults come from the schema. A4.2 Widget overrides Path pattern Widget Any #rrggbb string Color picker *.assetId Slot picker, filtered by media kind (image for sprites, backgrounds, and LUTs, audio for music and sources, font for HUD) collider2d.category, collider2d.mask 32-bit checkbox grid with category names visualAnimation.tracks[] Track row with property, from/to, timing, easing, and a small curve preview behaviors[] with kind: "script" Summary row with “Edit script” (A7) transform2d.rotation Degree input that stores radians renderEffects[] Ordered list with an “Add effect” menu (bloom, LUT, brightness and contrast) A4.3 Sections With an entity selected, the inspector shows: identity (id, name, parent, prefab flag); Transform; one section for each component, with an “Add component” menu of the absent components; Behaviors, with an “Add behavior” menu for each kind. With nothing selected, it shows the scene: name, music, lighting (ambient and the light list), and backgrounds. A “Game” tab shows pixels per unit, input actions, entry scene, effects, and HUD effect order. Each field change becomes one reducer op (A1.4), so UI edits and agent edits share the same log. A4.4 Collision category names Add an optional collisionLayers: string[] (at most 32 entries) to the schema 2 document. It is additive and needs no version change. The grid shows these names, and bit numbers when a name is absent. The agent sets the names through set_game. This is the only schema addition in this plan, and question Q2 asks whether to include it. A4.5 Validation Change validateGame to return issues: { path: (string | number)[]; message: string }[] and to keep errors as formatted strings for compatibility. Field errors come from the Zod parse, and reference errors come from validateGame (unknown slot, missing prefab or scene, LUT dimensions). The inspector shows each issue under its field. The scene tree marks entities that have issues. Autosave writes a draft only when it validates. The store keeps a field that is not valid as local state until the user fixes it. A4 acceptance A4.a: Each component and behavior in game.ts can be added, edited, and removed without JSON. A4.b: A new optional schema field appears in the inspector without a change to the inspector code (a test adds a field to a fixture schema). A4.c: A LUT with the wrong dimensions shows the error under assetId of that effect. A7: Script editor A7.1 Pane “Edit script” opens GameScriptPane, which is a Monaco editor. It is a bottom split below the viewport on desktop and a full sheet on mobile. One tab for each open script shows the entity and the behavior index. Edits go to the draft store as set_script ops after a 500 ms debounce, with one undo entry for each pause. A7.2 Typings Export GAME_SCRIPT_TYPES (a .d.ts string) from packages/game-runtime/src/script-types.ts. It declares the argument object (tick, pressed, justPressed, events, entity, world, state, random) and the return type { state, commands }, with each command kind from scripts.ts. A test checks that the command kinds in the .d.ts match the Zod command union, so the typings cannot drift (D8). Monaco gets the typings as an extra lib. It checks the source as a function expression with JSDoc type checks enabled. A7.3 Limits and errors The status bar shows the source size against 16 KiB, and maxCommands and maxTickMs for the behavior. The scene tree marks the entities of a script that fails to prepare (“could not be prepared” or “Source must be a function expression”). A play session that stops with a script error at tick N shows the error in the pane, with the tick and the entity id. “Replay to tick N − 1” restarts the session with the recorded input and pauses before the error. “Run 10 s” runs the draft headless in the browser with no input. It shows the script call time for each entity and the first error. A7.4 Assistant link A selection in the pane sends entity_ids and the behavior index with the next chat message. “Ask the assistant” on an error puts the error text, the tick, and the script key in the composer. A7 acceptance A7.a: A wrong command kind shows a type error in Monaco, and a test proves that the typings match the runtime schema. A7.b: A script that throws at tick 120 shows the error at tick 120 in the pane, and the replay pauses at tick 119. A7.c: The agent’s set_script updates an open, clean pane at once. For an open pane with unsaved edits, it produces a conflict (A2.c). Delivery order Slice Content Depends on S1 Split GameSurface (A3.1), shared key map (A3.2 step 5) none S2 Op reducer (A1.4 reducer), draft persistence and routes (A2.1), edit_native_game, get_native_game views (A1.3), publish from draft (A1.8) none S3 Web draft store, sync and merge adapter (A2.2, A2.3) S1, S2 S4 Assistant panel and prompt (A1.1, A1.2), capture tool (A1.5), draft playtest (A1.6), registration and skill (A1.9) S2 S5 Change log and cards, publish dialog, revision list (A2.4 to A2.7) S3, S4 S6 Viewport editing and scene tree (A3.2 to A3.5) S3 S7 Inspector and structured validation (A4) S3 S8 generate_game_asset and the F4 media fixes (A1.7) S2 S9 Script editor (A7) S7 Each slice is one PR. S2 and S4 give the agent a useful loop before any UI work. S6, S7, and S8 can go in parallel after S3. Verification Each slice runs the mandatory checks. Reducer and merge: unit tests for each op and each merge rule, plus a test with 1,000 entities. Capture: a test in packages/agents renders a fixture draft with an overlay and checks a known pixel. WebGPU tests need the Vulkan ICD, as the renderer suite does. Web: component tests for the gizmos (drag to ops), the inspector generation (A4.b), and the change cards (A2.d). Run only the related Jest suites. End to end: one Playwright test opens a game, sends an assistant message against a mocked provider, sees a card, undoes it, and publishes. The showcase games in ~/workspace/nodetool-games are a manual regression. Import both as drafts, edit them in the UI, and compare captures before and after. Risks Risk Effect Mitigation R1 Server capture with WebGPU is slow or unavailable on some hosts. Fall back to Canvas2D with a diagnostic. Cache the GPU device for each process. Limit a call to 8 frames. R2 Large documents fill the agent context. The outline view is the default. Scripts are read one at a time. R3 Point lights have no ids, so the merge treats all lighting in a scene as one unit. Accept this for now. Add light ids in a later schema change if conflicts occur in real use. R4 Draft snapshots and change rows grow without limit. Cap the rows for each game, and delete them after a publish. R5 Editor-camera frames differ from game-camera frames in culling. Pass the overscan to visibleItems, as capture already does. Not in scope The asset panel (A5), replay scrubbing and debug overlays beyond A3.3 (A6), and the phone preview and share (A8). Tile painting for tilemap. Presence, cursors, and simultaneous editing by several users. 3D scenes. Questions Question Recommendation Q1. When the editor is closed, should agent edits still go to the draft, or publish directly? Use the draft always. Direct publishing hides changes from review. Q2. Add collisionLayers names to the schema (A4.4)? Yes. The mask grid cannot be understood without names. Q3. Should installing an asset still publish a revision, as it does today? No. Installing binds the asset in the draft, and “Publish” makes the revision.--- # Game engine and editor gap analysis Source: https://docs.nodetool.ai/plans/native-game-gap-analysis.html Game engine and editor gap analysis Scope: the native game engine (packages/game-runtime, packages/game-renderer, packages/protocol/src/game*.ts) and the 2D and 3D editors (web/src/components/game/), compared with Unity 6 and Unreal Engine 5. Code was read on main at b1337258 (2026-10-06), which includes the Unity-style layout from PR #6038. This file does not repeat the bugs in the audit (F1 to F30). Gaps here are labelled G1 onwards. How to read the priorities NodeTool games are authored by people and agents together, built from generated assets, and shipped to the browser. That shapes the ranking: P0 blocks the stated goal (state-of-the-art graphics, physics, audio, professional UX) or blocks generated content from working well. P1 is expected by anyone who has used Unity or Unreal and is achievable in a browser engine. P2 is useful polish or depth. Each gap says what exists today, what is missing, and why it matters for NodeTool. Deliberately out of scope These are core to Unity and Unreal but do not fit a browser-delivered, agent-authored engine. They are left out on purpose. Console, mobile-native and VR/XR platform targets and their certification tooling. Nanite-style virtualized geometry, Lumen-class dynamic GI and hardware ray tracing. WebGPU has no ray-tracing extension, and the budget cost is wrong for the web. World Partition and open-world streaming at the scale of tens of kilometres. Blueprint-style visual scripting. NodeTool’s agent writes scripts, and the workflow graph already covers visual authoring. A second visual language would split authoring. C++ or C# native scripting, hot-reloadable native modules and plugin SDKs. Dedicated-server multiplayer, replication graphs and matchmaking. Deterministic lockstep or rollback is mentioned once under G44 because the engine is already deterministic. Terrain sculpting with height-map painting tools, foliage painting and landscape layers (generated meshes and tilemaps cover NodeTool’s use). Cinematic sequencer at Unreal’s level. NodeTool already has a timeline and storyboard product. What exists today Area 2D (engine “1”, schema 1–2) 3D (engine “2”, schema 3) Renderer WebGPU instanced sprite batches, Canvas2D fallback. Tilemaps, parallax backgrounds, 2D point lights (max 32), bloom, LUT, brightness/contrast. three.js WebGLRenderer 0.185 on WebGL2. MeshStandardMaterial PBR, ACES tone mapping, one shadow-casting directional light (PCF soft), point and spot lights without shadows, linear fog, solid-colour background. Physics Custom swept AABB, static and kinematic bodies only, one-way platforms, tile solids. Rapier 0.19 with static, kinematic and dynamic bodies, box, sphere, capsule, convex hull and triangle mesh colliders, layers, sensors, CCD, character controller, ray and shape queries. Audio Web Audio one-shots and looping music with fades, 32 voices, no spatialisation. Same player. Animation Sprite-sheet clips, tweens on rotation, scale, opacity and tint. glTF clips with one crossfade between consecutive clips. Scripting QuickJS, one pure function per behavior, returns commands, per-call and per-tick budgets. Same model with a 3D command set and physics queries. Determinism Fixed 60 Hz tick, seeded RNG, snapshots, replay. Same, with Rapier state serialized into the snapshot. Editor Scene tree, inspector, 2D viewport with move/rotate/scale gizmos and grid snap, collider overlay, script pane, revisions, changes list, agent panel, save/load play state, replay. Hierarchy, schema-driven inspector, orbit and fly camera, translate/rotate/scale gizmo with snap, overlays, script pane, agent panel. Export Static web build. Static web build with hashed assets. Engine gaps Rendering (3D) G1. P0. The 3D renderer is WebGL2 only. The 2D path already runs on WebGPU, and three.js 0.185 ships WebGPURenderer with TSL node materials and a WebGL2 fallback. Moving to it is the prerequisite for compute-based particles (G10), GPU culling, modern post-processing (G3) and the shared GPU device with packages/gpu. Unity’s URP and HDRP, and Unreal, are both built on a compute-capable pipeline. G2. P0. No image-based lighting or sky. The background is a solid colour (gameEnvironment3D.background) and ambient light is a flat colour. PBR materials look flat without an environment map. Missing: HDRI skybox, PMREM-filtered reflection and irradiance, a procedural sky with sun direction linked to the directional light, and reflection probes. This is the single largest visual gap per hour of work, and NodeTool can generate equirectangular HDRIs. G3. P0. No 3D post-processing stack. 2D has bloom and LUT. 3D has ACES tone mapping only. Missing, in rough order of value: exposure control, bloom, SSAO or GTAO, colour grading and LUT (reuse the 2D LUT asset), TAA or SMAA, depth of field, vignette, motion blur, SSR. Unity’s Volume and Unreal’s Post Process Volume make this a per-scene or per-region profile, which suits a document field on gameEnvironment3D. G4. P1. Shadows are limited to one directional light. No cascaded shadow maps (a single 30 m extent shadow map at up to 2048 px makes larger scenes blurry or clipped), no point or spot light shadows, no contact shadows. Cascades and per-light shadow toggles are expected. G5. P1. Materials are a fixed subset of glTF PBR. gameMaterial3D has colour, metalness, roughness, opacity, emissive and alpha mode. Primitives have no texture slots, so a generated texture cannot be applied to a box or plane. Missing: base colour, normal, ORM and emissive maps for primitives, UV tiling, KHR_materials_* extensions beyond unlit, emissive strength and texture transform (clearcoat, transmission, sheen, specular, IOR), and custom or toon shaders. Stylised looks (toon, outline, unlit pixel) are common in generated games and need a shader or material-preset path. G6. P1. No instancing, batching or LOD. Every entity is its own mesh and draw call. There is no InstancedMesh or BatchedMesh for repeated props, no LOD switching and no HLOD. With 4,096 entities per scene allowed, draw-call count, not triangle count, is the first wall. G7. P1. Generated models can hit the budget and cannot be compressed. Preparation rejects Draco, meshopt and KTX2/Basis (renderer3d/preparation.ts:148). Generated models (Meshy, Tripo, Hunyuan, TRELLIS) are often 10–50 MB with 4K textures. Without mesh simplification, meshopt compression and KTX2 texture transcoding in the prepare step, web load times and GPU memory are poor. Unity and Unreal do this at import. G8. P2. Lighting baking. No lightmaps or baked light probes. For static scenes this is how Unity and Unreal ship good GI cheaply. A server-side bake (headless Blender already exists in blender-nodes) would fit NodeTool better than an in-browser baker. G9. P2. No decals, no volumetric fog, only linear fog. Exponential height fog and decals (bullet holes, footprints, generated graffiti) are cheap wins once G1 lands. Rendering (2D) G10. P0. No particle system in either engine. Effects today are spawned prefabs with a lifetime tween, one entity per particle, capped by MAX_GAME_SPAWNED_INSTANCES = 1024. Unity’s Shuriken and VFX Graph and Unreal’s Niagara are central to game feel. Needed: an emitter component with rate and burst, lifetime curves for size, colour and velocity, gravity, collision with the world (optional), sub-emitters, and GPU simulation on WebGPU. Particles must be presentation-only, so they stay outside the deterministic snapshot. G11. P1. 2D lighting has no normal maps or shadows. Point lights are additive radial falloff. Unity’s 2D URP has normal-mapped sprites, shadow casters and freeform lights. Generated sprites can get normal maps from a depth-estimation model, so this fits the asset pipeline. G12. P2. No 9-slice sprites, sprite masks or 2D custom shaders. Needed for UI panels (G21) and stylised effects (dissolve, outline, palette swap). Physics G13. P0. 2D physics has no dynamic bodies. body2d.type is static or kinematic. There is no mass, no stacking, no rigid-body response, no rotation of colliders and only boxes. Rapier has an official 2D build (@dimforge/rapier2d-compat) with the same API and determinism story as the 3D one already in use. Moving 2D onto it would give circles, capsules, polygons, slopes, joints, raycasts and dynamic bodies, and replace the custom sweep code. G14. P1. 2D broad phase is quadratic. solidsNear scans every obstacle state for every moving body (session.ts:588-605). This is fine at tens of entities and slow at hundreds. G13 removes it. If G13 is deferred, a uniform grid fixes it. G15. P1. No joints in 3D. Rapier supports fixed, revolute, prismatic, spherical and rope joints and motors. None is exposed in gameEntity3D. Doors, chains, vehicles, ragdolls and physics puzzles need them. G16. P1. No physics materials or combine rules. Friction and restitution are per collider, with no shared material asset and no combine mode (min, max, average, multiply). Unity and Unreal both use physics material assets. G17. P2. No vehicles, ragdolls, cloth or destruction. Ragdolls follow from G15 and skinned models. Vehicle physics is a Rapier DynamicRayCastVehicleController. Cloth and destruction are lower value. G18. P2. One physics step per tick, no substeps or solver settings. Fast objects rely on CCD only. Exposing solver iterations and substeps per scene would match Unity’s and Unreal’s project physics settings. Audio G19. P0. No spatial audio. Every voice connects straight to context.destination (audio.ts:153). There is no PannerNode, no listener, no distance attenuation, no doppler, and 3D audioSource positions are ignored. This is the largest audio gap for a 3D engine. G20. P1. No mixer, buses or effects. Unity’s Audio Mixer and Unreal’s MetaSounds and Submixes give music, SFX, voice and UI buses with volume, ducking (music under dialogue), snapshot transitions, reverb zones, low-pass occlusion and a master limiter. None exists. Also missing: pitch and volume randomisation per play, random containers (pick one of N footsteps), and priority-based voice stealing (today the oldest one-shot is stolen). G21. P2. No streaming for long audio. Music is fully decoded with decodeAudioData, which for a five-minute track is about 50 MB of PCM. A MediaElementAudioSourceNode path for music fixes it. Animation G22. P0. No animation state machine or blending. animator3d maps names to clips and the runtime crossfades from the previous clip to the current one. Missing: a state graph with parameters and transitions (Unity’s Animator Controller, Unreal’s AnimGraph), 1D and 2D blend spaces (idle, walk, run by speed), layers with masks (upper-body shoot while running), additive clips, root motion and animation events (footstep sound on frame). Characters generated with Mixamo-style rigs cannot look professional without blend spaces. G23. P1. No retargeting or shared animation libraries. Clips are bound to the model they came in. Generated characters usually arrive without animations, so a humanoid retargeting step that applies a shared clip library to any humanoid rig would unlock most of them. Unity’s Humanoid Avatar and Unreal’s IK Retargeter do this. G24. P2. No IK. Foot placement on slopes and look-at are the two that matter. three.js has CCD IK, and two-bone IK is small. G25. P2. No general tween or curve tracks in 3D. 2D has visualAnimation tracks. 3D has setVisual commands only. Moving platforms and doors need keyframed transform tracks without scripts. Gameplay runtime and scripting G26. P0. Script cost grows with scripts × entities. Each script call serialises the full input, including the whole world array, into a source string and re-evaluates it (scripts.ts:243-246, session3d.ts:272). With 50 scripted entities in a 500-entity scene that is 50 × 500 entity records compiled as code every tick. Needed: pass world once per tick as a shared value, let scripts query nearby entities instead of receiving all of them, and keep compiled functions resident. G27. P1. Scripts see too little of the world. The 3D world entry has position, velocity and grounded only. There is no rotation, no tags or groups, no custom properties per entity and no way to read another entity’s script state. Unity’s tags, layers and GetComponent and Unreal’s actor tags and gameplay tags are how gameplay code finds things. A tag field on entities and an entity.props bag the inspector can edit would cover most uses. G28. P1. No timers, coroutines or lifecycle hooks. Scripts are a stateless per-tick function. Common patterns (wait 2 s then spawn, on start, on destroy, on enter scene) must be hand-written with tick counters. A small set of declarative helpers would cut script size and agent errors. G29. P1. No AI navigation. No navmesh, pathfinding or steering. Enemies in generated games patrol on a line. Recast via recast-navigation-js (WASM) generates navmeshes from colliders and provides crowd simulation, which matches Unity’s NavMesh and Unreal’s Navigation System. For 2D, grid A* over tilemaps is enough. G30. P1. No player save system. Save play state is an editor tool backed by localStorage. Shipped games cannot persist progress, settings or high scores. The deterministic snapshot is already a full save format, so this is mostly API and UI. G31. P2. No scene additive loading or persistent objects. A scene transition replaces everything. A persistent player or HUD across levels needs a “don’t destroy on load” equivalent or additive scenes. G32. P2. No events between entities with payloads. emit sends a name with no data and no target. Input G33. P0. Bindings are derived from action names and cannot be configured. actionKeys3D("fire") returns Mouse0 and KeyF, and any other action maps to Key<ACTION> (input3d.ts:4-7). There is no input map in the document, no gamepad support (getGamepads is never called), no rebinding UI for players, and no touch controls in the 3D standalone player. Unity’s Input System and Unreal’s Enhanced Input both separate actions from bindings. A inputBindings section with keyboard, mouse, gamepad and touch per action is the fix, and it also fixes F22. UI and HUD G34. P0. The HUD is text labels only. gameHudLabel supports text, position, size, colour and font. There are no images, health bars, buttons, panels, layout or anchors, and no menus (title screen, pause, game over, settings). Every finished game needs these. Unity’s UI Toolkit and Unreal’s UMG are full layout systems. A minimal retained HUD tree with image, nine-slice panel, bar, text and button nodes, anchored to screen edges, would cover most games. NodeTool’s mini-app widgets could be a reference for layout. G35. P2. No localisation or rich text. Text has no string tables, no markup for colour or icons and no right-to-left support. Asset pipeline and content G36. P1. No asset variants or overrides at import. Model import has scale, forward axis and origin. Missing: per-material overrides at import, collider generation options (convex decomposition, not one hull), LOD generation, pivot editing and texture max-size per platform. Unity’s import settings and Unreal’s static mesh editor are where these live. G37. P2. No sprite atlas packing at runtime build. 2D draws batch by texture. Packing small sprites into atlases at build time reduces draw calls. Performance and scale G38. P0. No frustum, occlusion or distance culling control and no per-system budgets. three.js frustum-culls meshes, but there is no distance culling, no culling of simulation for far entities, and nothing reports CPU time per system (physics, scripts, render). Unity’s Profiler and Unreal Insights exist because performance work is impossible without them. G51 is the editor half. G39. P1. Simulation runs on the main thread. Physics, scripts and rendering share one thread, and the tick clock drops time after five catch-up steps (fixed-tick-host.ts:20). Running the deterministic simulation in a worker and sending render frames to the main thread would keep rendering smooth under script load. The snapshot boundary already separates the two. G40. P2. No asset streaming within a scene. All models in the document load before play. Per-scene asset lists and progressive loading with a loading screen would shorten time to first frame. Build and delivery G41. P1. No quality tiers or runtime settings. There is one quality level. Unity’s Quality settings and Unreal’s scalability groups let a game drop shadows, post-processing and resolution on weak devices. Needed: low, medium and high tiers, dynamic resolution, and a frame-rate cap. G42. P2. No PWA or offline packaging, no embed SDK. The static build is close to a PWA. A manifest and service worker would make games installable and playable offline. An embed snippet with postMessage events would let sites host games. G43. P2. No crash and telemetry hooks in shipped games. NodeTool’s error tracing covers the editor, not exported games. G44. P2. Multiplayer. Out of scope as a full system, but the deterministic tick, seeded RNG and serialised Rapier state already allow rollback netcode for small two-to-four-player games. Worth keeping the determinism guarantee intact for that reason. Editor gaps Viewport and scene manipulation G45. P0. No multi-select and group transform in 3D, partial in 2D. Covered for keyboard actions by F10 and the audit’s 3D feature-gap list. The missing piece beyond those is a pivot mode (selection centre or individual origins) and transforming a multi-selection with one gizmo. G46. P1. Gizmo and snapping controls are minimal. Missing: local versus world space toggle, configurable snap increments for move, rotate and scale, vertex and surface snapping (drop on floor), align to ground, and a scene-view orientation gizmo with axis views and orthographic toggle. Unity’s scene gizmo and Unreal’s snapping bar are expected by any user arriving from those tools. G47. P1. No drag-and-drop placement. Assets and prefabs cannot be dragged from a panel into the viewport to place them on the surface under the cursor. In Unity and Unreal this is the primary way scenes are built. G48. P1. No physics and lighting debug views. 3D overlays show colliders. Missing: contact points and normals, sleeping bodies, raycast visualisation, navmesh (with G29), light range and cone gizmos, shadow cascade view, wireframe, unlit and overdraw view modes. Unreal’s view modes and Unity’s scene view draw modes. G49. P2. No camera bookmarks or multiple viewports. A game camera preview inset while editing (Unity’s camera preview) is the most useful one. Content browser and assets G50. P0. There is no asset browser. 3D shows “Model assets” as a collapsed footer in the hierarchy with an install button. 2D lists slots in the scene tree. Unity’s Project window and Unreal’s Content Browser are the centre of both editors. Needed: a docked panel with thumbnails for every asset, prefab and scene, search and type filters, drag-to-viewport (G47), usage references, and replace-in-place. For NodeTool specifically, each slot should offer “generate”, “regenerate with a new prompt” and “pick from candidates” in place, so the generation pipeline is visible in the editor rather than only through the agent. Play mode, debugging and profiling G51. P0. No profiler or performance overlay. The status bar shows tick, score and backend. Needed: FPS and frame-time graph, draw calls, triangles, texture memory (the renderer already collects GameRendererStats3D), per-system CPU time (physics, scripts, render, audio), script time per entity, and physics body counts. Unity’s Stats window and Profiler and Unreal’s stat commands. G52. P1. No console or log panel. Script errors surface as status text. Needed: a console panel with errors, warnings and script log output, filters, entity links (click to select), collapse of repeated messages, and “Ask the assistant” on any line. G53. P1. Limited live editing while playing. Unity lets the user select runtime objects during play, inspect live values and tweak them, then decide whether to keep changes. Here edits rebuild the whole play session (F15). Needed: a live runtime inspector for any entity, pause-and-edit, and “copy runtime values to the document”. G54. P1. No breakpoints or script debugging. The script pane has error tools in 2D only (audit 3D gap list). Beyond that, missing: stepping a single entity’s script, watching its state over ticks, and a timeline scrubber over the existing replay history (the replay already exists, so scrubbing is mostly UI). Inspector and authoring G55. P1. The 3D inspector is a schema form with no component-specific editors. Every component renders as raw fields from Zod. Missing: colour pickers with HDR intensity, curve editors (G10, G25), asset pickers with thumbnails, collider “fit to mesh” buttons, camera preview on camera components, light colour temperature, and a material editor with a live preview sphere. Unity’s custom inspectors and Unreal’s details panel customisations are what make components approachable. G56. P1. Prefabs have no edit mode. Prefab instances and overrides exist (GameOverrideFields). Missing: open a prefab in isolation, nested prefabs, prefab variants, apply and revert overrides per field, and a visual indicator of overridden fields. Unity’s prefab workflow is one of its defining features. G57. P1. No animation tooling. No clip preview scrubber on a selected model, no state-machine graph editor (with G22) and no event markers. NodeTool’s node graph editor could host the state machine graph, which would keep one graph language. G58. P1. No environment and lighting presets. Scene settings are raw fields for background, ambient, fog and shadows. Unity’s Lighting window and Unreal’s environment light mixer give sky, sun and post-processing presets. One-click presets (“sunny day”, “night”, “overcast”, “neon interior”) would suit both people and the agent. G59. P2. No terrain or level-blocking tools beyond primitives. ProBuilder-style or Unreal Modeling Mode box editing is out of scope at full depth, but extrude and resize-by-face handles on box primitives would speed up greyboxing. Professional UX polish G60. P1. No keyboard shortcut map or command palette. Shortcuts exist (F, Delete, Ctrl+Z, arrows in 2D) but are undiscoverable and differ between 2D and 3D. Unity’s Shortcut Manager and Unreal’s keyboard settings provide a list and rebinding. A command palette would also expose agent actions. G61. P1. 2D and 3D editors diverge. They have different panels, shortcuts, toolbars and feature sets (see the audit’s 3D gap list). Unity and Unreal use one editor for both. A shared shell with mode-specific viewport and inspector sections would halve future UI work and remove the inconsistency. G62. P1. Layout is fixed. Panels can be toggled, not docked, tabbed, resized and saved as layouts. Unity’s and Unreal’s docking layouts are expected for a professional tool, especially with more panels arriving (asset browser, console, profiler). E1 adds tabbed, resizable, drag-to-dock regions, per-user layout persistence and named layouts. G63. P2. No in-editor onboarding. No empty-scene guidance, no tooltips that explain components, no sample-scene starter beyond templates. Coverage against the reference feature spec Matthias supplied a technology-agnostic engine feature spec on 2026-10-06. Each item is mapped below. “Have” means the code implements it. “Partial” and “Missing” link to the gap that covers it. G64 to G71 are new gaps the spec surfaced. 1. Scene and entity-component architecture Spec item Status Notes Composition-based scene graph Have parentId hierarchy with world transforms, entity components as optional fields. Component registry and reflection Have Zod schemas drive validation and the inspector (gameSchemaFields). Components are a closed set, not user-extensible. That is intended for agent safety. Prefab and spawning pipeline Partial Runtime spawn, despawn and prefab instances exist. Prefab edit mode, nesting and variants are missing (G56). Scene serialization format Have JSON game documents with schema versions, op-based edits, revisions and merge. 2. Editor infrastructure and UX Spec item Status Notes Docking workspace and layout persistence Partial Tabbed, resizable, drag-to-dock regions with per-user persistence and named layouts (G62, E1). No Game view separate from Scene view (G49), no Asset Browser (G50), no Profiler (G51). Project and file management Have, by design NodeTool projects and workspace tabs own this. A separate launcher is out of scope. Visual asset browser Missing G50. Thumbnails, breadcrumbs, metadata tooltips and drag-and-drop are all part of it. Flycam, orbit, pan, frame selection Have (3D) Orbit, fly and F to frame exist. Orbit pivots on the last target, not the selection (G46). Raycast picking Have renderer3d/index.ts:334. Local vs world gizmos with snapping Partial Snapping is a fixed on/off toggle. No local/world toggle (G46). Command-pattern undo/redo with labels Partial Undo and redo exist, unlabelled (G64). Debug overlays Partial Collider wireframes, camera frustums and grid exist. Selection outline and light bounds are missing (G48, G65). 3. Rendering and post-processing Spec item Status Notes Modern RHI Partial 2D on WebGPU, 3D on WebGL2 (G1). PBR with full map set Partial glTF models get all maps. Primitives get none (G5). Shadows and SSAO Partial One directional shadow. No point shadows, no SSAO (G3, G4). Post-processing chain with tone-mapping curves and AA Missing in 3D Fixed ACES and MSAA only (G3). Particle and VFX subsystem Missing G10. Fixed-aspect viewport scaling Partial Aspect ratio and letterboxing exist. Internal render resolution and scaling modes are missing (G70). 4. Physics and simulation Spec item Status Notes Rigid body dynamics Have (3D), Missing (2D) 3D has mass, damping, impulses and velocity. No continuous force or torque command (G68). 2D has no dynamic bodies (G13). Joints Missing G15. Collision layers and layer matrix Partial Named layers and category/mask bits per collider. No layer-collision matrix editor (G66). Triggers with enter, stay, exit Have Sensor colliders emit contact events with phases in 2D and 3D. Determinism controls Partial Fixed tick and explicit seed exist. Tick rate is hard-coded to 60 and the editor has no seed field (G69). 5. Scripting and runtime reflection Spec item Status Notes Embedded scripting VM Have QuickJS with budgets. Live hot reloading Partial A script edit restarts the play session from tick 0 (F15). Reload that keeps state is missing (G67). Two-way property binding Missing G27 covers entity properties. Script-declared parameters are missing (G71). Lifecycle hooks Partial Update is the only hook. Contact events arrive as data. G28. 2D canvas and UI layout Missing G34. 6. Audio Spec item Status Notes Positional 3D audio Missing G19. Multi-format streaming Partial Compressed one-shots and music work. Music is fully decoded, not streamed (G21). 7. Deployment, tooling and CI Spec item Status Notes Headless verification Have nodetool game simulate runs ticks headlessly with input files, assertions and replay verification. nodetool game capture renders frames. The exported build itself is not smoke-tested (G72). Standalone distribution Have nodetool game build writes a static web player with hashed assets and no editor code. CI test harness Have Vitest suites in packages/game-runtime/tests cover sessions, physics regressions, scripts, replay and validation. New gaps from the spec G64. P1. Undo has no labels or history list. GameDraftStore keeps an undo stack with no action names. The spec and both reference engines show “Undo Move Player” and a browsable history. 3D undo is also one full-document op per step (audit F17). Labelled, op-level commands fix both. G65. P1. No selection outline or light gizmos. The selected entity is shown only by the transform gizmo. An outline pass for the selection and range or cone gizmos for point and spot lights are standard. This extends G48. G66. P1. No layer-collision matrix. Collision filtering is per collider category and mask bits. A project-level matrix of named layers, as in Unity’s Physics settings, is easier to reason about, and the agent can edit it safely. G67. P1. Script hot reload restarts play. Editing a script during play should swap the function and keep the session, entity states and script state where the shape still fits. Today the session is rebuilt from tick 0 (F15). The deterministic snapshot makes “reload and resume from the current tick” straightforward. G68. P2. No continuous force or torque commands in 3D. Scripts can set velocity, apply an impulse or teleport. Thrusters, wind and buoyancy need addForce and addTorque, which Rapier supports directly. G69. P2. Tick rate and seed are not configurable. tickRate is the literal 60 in both schemas. Physics-heavy games want 120 Hz, and slow board games want less. The editor also always plays with one seed, so seed-dependent bugs are hard to reproduce from the UI. G70. P2. No internal render resolution or scaling modes. Missing: a fixed internal resolution with integer scaling for pixel art, a render-scale factor for 3D, and a choice between letterbox, crop and stretch. G71. P1. Scripts cannot declare inspector-editable parameters. A script’s tunables (speed, damage, a target entity, an asset) must be edited in source. Unity’s serialized fields and Unreal’s UPROPERTY let designers tune these in the inspector. A declared params schema per script behavior, rendered by the existing SchemaFields, would cover numbers, colours, entity references and asset references with no editor code per script. It also gives the agent a safe surface for tuning. G72. P2. Exported builds are not smoke-tested. simulate and capture run against the document, not the files build produces. Loading the built player headlessly for N frames and checking for errors would catch missing assets and bundling breaks. Top priorities If the goal is the biggest visible step toward state-of-the-art graphics, physics and audio plus professional UX, this is the order I recommend. Order Gaps Why first 1 G2, G3 (IBL, sky, 3D post-processing) Largest visual improvement per day of work. Works on the current WebGL2 renderer, so it does not wait on G1. 2 G50, G47 (asset browser with in-place generation, drag to place) Makes NodeTool’s main advantage, generated assets, visible in the editor. Also the most-missed Unity panel. 3 G34, G33 (HUD widgets and menus, input map with gamepad) Every shipped game needs menus, bars and gamepad support. Today none can be built. 4 G13 (Rapier 2D) Removes the custom 2D physics, unlocks dynamic bodies, joints and slopes, and fixes G14. 5 G19, G20 (spatial audio, mixer) Audio is the cheapest area to bring to modern standards. Web Audio already has panners, compressors and convolution. 6 G10 (particles) Game feel. Best built after G1 to use GPU compute, but a CPU emitter can come first. 7 G22, G23 (animation state machine, retargeting) Required for generated characters to move convincingly. 8 G51, G52, G26, G71, G67 (profiler, console, script cost, script parameters, hot reload) Needed before scenes grow. G26 is a contained fix with a large payoff. 9 G1, G7 (WebGPU 3D renderer, model compression) Foundation for the next tier of rendering and for loading generated models quickly. Larger change. The audit’s open data-loss and performance findings (F4, F5, F9, F15) should still go before any of these, because F15 makes every editor feature above feel slower than it is.--- # Native game engine and editor: parallel implementation plan Source: https://docs.nodetool.ai/plans/native-game-implementation-plan.html Native game engine and editor: parallel implementation plan This plan turns the gap analysis (G1 to G72) and the open findings in the audit (F4 to F30) into work for autonomous coding agents running in parallel. Each agent takes one workstream and works through its task cards in order. The plan was written against main at b1337258 (2026-10-06). File and line references are starting points. Re-check them before editing. 1. How to use this plan Run Wave 0 first. Its tasks create the extension seams and the ownership boundaries that make parallel work safe. Wave 0 has three streams that can themselves run in parallel (K, W and B). After Wave 0 is merged, start up to 12 workstreams at once. Each agent gets the launch prompt in section 9 with its stream letter filled in. An agent works through its stream’s task cards top to bottom. A card may start only when every card in its Depends on line is merged to main. One task card is one pull request. Large cards list sub-PRs explicitly. When a card needs a file owned by another stream, follow the shared-file protocol in section 4. Never edit another stream’s owned files directly. 2. Rules every task must follow These come from the repository’s AGENTS.md, docs/DEVELOPMENT_STANDARDS.md and the existing game design documents in docs/plans/ (builtin-game-engine-design.md, native-game-3d-upgrade.md, native-game-editor.md, native-game-artistic-roadmap.md, retained-game-authoring.md). Read those before the first task. 2.1 Environment Run scripts/setup-agent-env.sh, then npm run build:packages, at the start of every session. In the cloud container, a plain npm install fails because the lockfile pins xlsx to cdn.sheetjs.com. Swap xlsx to its npm-registry version, install, then restore package.json and package-lock.json before committing. Never commit the swap. The runtime and renderer suites need ffmpeg and a Vulkan ICD (mesa-vulkan-drivers). Install them before suite-wide runs. Do not skip a test that reports “No WebGPU adapter available”. Headless Chromium 3D capture can fail in the cloud container. If a capture test cannot run there, say so in the PR and point to the CI run that executed it. Never mark it skipped. 2.2 Definition of done for every card A card is done only when all of these hold: The four mandatory checks pass: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main New behavior has tests in the package’s existing test directory. Bug fixes ship the reproduction test. Determinism holds (section 2.3). Schema changes follow section 2.4. The agent can author the feature (section 2.5). The native-game system skill section owned by the stream is updated when authoring changes (packages/system-skills/native-game/SKILL.md). UI follows docs/DESIGN.md and web/src/components/ui_primitives/STRATEGY.md: primitives only, no raw MUI outside ui_primitives/ and editor_ui/, design tokens only, font weights 400, 500 or 600. Performance-sensitive cards record before and after numbers from the benchmark harness (card B1) in the PR body. The PR body uses the repository template if one exists. It names the gap and finding IDs in the title, for example game: image-based lighting and sky (G2). 2.3 Determinism contract The engine is deterministic: 60 Hz fixed tick, seeded RNG, snapshots, replay (nodetool game simulate --verify-replay). Every change must keep that. Simulation state is anything that changes events, scores, positions, physics or script results. It lives in the snapshot and must be reproducible from the document, the seed and the input frames. Presentation state includes particles, audio mixing, post-processing, IK, cosmetic animation blending and camera smoothing. It must never feed back into simulation. It may read simulation state. A change to simulation behavior for an existing document requires either a new engine contract version or proof that the existing fixtures produce identical snapshots and event streams (packages/game-runtime/tests/fixtures/engine1-*.json, engine1-characterization.test.ts, session3d.test.ts). A change to Rapier usage that alters stepping must update GAME_PHYSICS_BUILD_3D in packages/game-runtime/src/spatial3d/world.ts so old 3D snapshots are rejected rather than silently diverging. 2.4 Schema change rules Schemas live in packages/protocol/src/game.ts (2D, schema 1–2, engine “1”) and game3d.ts (3D, schema 3, engine “2”). After card W1 they are split into per-component modules (section 4). New fields on schema 3 are optional, or have defaults that reproduce today’s behavior exactly. Existing stored documents must parse and play identically. native-game-3d-upgrade.md (D1) requires that future 2D changes allocate a deliberate schema version. Any 2D change that alters simulation (card P1) uses a new 2D schema version and engine version. It does not reuse 1, 2 or 3. Every new field gets: a Zod definition with bounds, validation in validate.ts or validate3d.ts when it references ids or assets, document ops coverage in document-ops.ts or document-ops3d.ts, and a diff and merge unit (web/src/stores/game/) when it is not covered by the generic entity diff. Scripts are typed from the schemas (script-types.ts, script-types3d.ts). New script commands update those declarations, which feed Monaco and the agent. 2.5 Agent and harness reachability NodeTool follows harness-first engineering (docs/HARNESS_FIRST.md). Every feature must be drivable without the editor UI. The in-product agent edits games through the game capability in packages/agents/src/capabilities/game.ts and the document ops. Confirm the new field or command is reachable through them. Add a case to packages/agents/src/evals/surfaces/game.ts when the feature changes what an agent can author. nodetool game validate | simulate | capture | build (packages/cli/src/commands/game.ts) must accept documents that use the feature. Add a simulate or capture fixture. Selfchecks are wired through packages/cli/src/harness/registry.ts (game-flow entry). Add new test filters there when a new test file is not already selected. 2.6 Dependencies New npm dependencies need a short justification in the PR (size, license, maintenance, WASM or not) per docs/DEVELOPMENT_STANDARDS.md#21-dependencies--versions. Pin exact versions for anything that affects simulation, as Rapier is pinned today. Already in the repository and preferred: three 0.185.1, @dimforge/rapier3d-compat 0.19.3, @gltf-transform/core and @gltf-transform/functions, meshoptimizer, @xyflow/react. 3. Parallelisation model 3.1 Why the foundation wave exists A few files are touched by almost every feature: Hotspot Size Touched by packages/protocol/src/game3d.ts, game.ts 301, 263 lines every engine feature packages/game-runtime/src/session3d.ts, session.ts 475, 1111 lines physics, scripting, animation, audio events, input packages/game-renderer/src/renderer3d/index.ts 622 lines every 3D rendering feature, HUD, particles packages/game-renderer/src/webgpu.ts 844 lines 2D rendering, particles, HUD web/src/components/game/GameEditor.tsx, GameEditor3D.tsx 491, 236 lines every editor feature packages/system-skills/native-game/SKILL.md 323 lines every authoring change If twelve agents edit these at once, most PRs conflict. Wave 0 splits each hotspot into modules with a registration point. After that, a feature adds new files plus one registration line. One-line conflicts in a registration list are trivial to resolve. 3.2 Waves Wave Streams Gate to start 0 K (stabilisation), W (seams), B (baselines) none 1 R, V, P, A, N, S, U, E, C, D (first cards), M (milestone games) W1–W5 merged, K1–K3 merged 2 Same streams, later cards, plus G (navigation) per-card dependencies 3 R8 (WebGPU 3D migration) and cards that depend on it R1–R7 and V3 merged, renderer freeze declared (section 3.4) Waves are not global barriers after Wave 0. A stream moves to its next card as soon as that card’s dependencies are merged. 3.3 Stream overview Stream Scope Gaps and findings Cards K Stabilisation of existing editor and API F4–F29 K1–K7 W Foundation seams G61 (shell), G64 (command bus) W1–W6 B Benchmarks, golden images, showcase fixtures G38 (measurement), G72 B1–B3 R 3D rendering G1–G6, G8, G9, G41, G58, G65, G70 R1–R11 V 2D rendering and VFX G10–G12, G37 V1–V8 P Physics G13–G18, G66, G68, G69 P1–P9 A Audio G19–G21 A1–A4 N Animation G22–G25, G57 N1–N7 S Scripting and gameplay runtime G26–G28, G30–G32, G67, G71 S1–S8 U Input and game UI G33–G35 U1–U4 G Navigation and AI G29 G1n–G4n E Editor UX and tooling G45, G46, G48, G49, G51–G55, G60, G62, G63 E1–E11 C Content pipeline and assets G7, G36, G40, G47, G50, G56, G59 C1–C7 D Performance architecture and delivery G38, G39, G42, G43 D1–D5 M Milestone games and integration acceptance across streams M1–M3 Navigation cards use the suffix n (G1n to G4n) so they are not confused with gap IDs. 3.4 Renderer freeze for R8 R8 moves the 3D renderer from WebGLRenderer to WebGPURenderer. While R8 is open, streams R, V and U must not merge changes to packages/game-renderer/src/renderer3d/. R8’s agent announces the freeze by opening a draft PR titled game: renderer freeze for WebGPU migration (R8) and lifts it on merge. Other streams continue on non-renderer cards during the freeze. 4. Ownership and the shared-file protocol 4.1 Ownership map after Wave 0 Path Owner packages/protocol/src/game3d/components/<component>.ts (created by W1) The stream that owns the component (table 4.2) packages/protocol/src/game3d/index.ts, game2d/index.ts (registration lists) W, then shared by registration line packages/game-runtime/src/systems/<system>.ts (created by W2) Stream that owns the system packages/game-runtime/src/session3d.ts, session.ts after W2 W during Wave 0, then S. Other streams register systems and do not edit the core loop packages/game-runtime/src/spatial3d/ P packages/game-runtime/src/scripts*.ts, script-types*.ts S packages/game-runtime/src/physics.ts and 2D collision code in session.ts P packages/game-renderer/src/renderer3d/passes/, environment/, materials/, shadows/ (created by W3) R packages/game-renderer/src/renderer3d/assets/ (loaders, created by W3) C packages/game-renderer/src/renderer3d/animation/ (created by W3) N packages/game-renderer/src/renderer3d/hud/ and src/ui/ U packages/game-renderer/src/webgpu/ passes (created by W3 from webgpu.ts) V, except webgpu/hud.ts owned by U packages/game-renderer/src/particles/ V packages/game-renderer/src/audio/ (created by W3 from audio.ts) A packages/game-renderer/src/input*.ts, touch-controls.ts U packages/game-renderer/src/build*.ts, standalone-player*.ts D packages/game-renderer/src/renderer3d/preparation.ts, packages/game-nodes/ C web/src/components/game/shell/ (created by W4) E web/src/components/game/panels/<panel>/ Stream that owns the panel web/src/components/game/viewport3d/, viewport2d/ (created by W4) E web/src/components/game/inspector/editors/<component>.tsx Stream that owns the component web/src/stores/game/ K during Wave 0, then E web/src/components/game/useGamePlaySession*.ts K during Wave 0, then S packages/cli/src/commands/game.ts D, with other streams adding options through the shared-file protocol packages/agents/src/capabilities/game.ts, evals/surfaces/game.ts Shared by append (protocol below) packages/system-skills/native-game/SKILL.md One section per stream, created by W6 4.2 Component ownership Component or field Owner primitive, model.material, gameMaterial3D, gameEnvironment3D, light3d, camera3d rendering fields, render quality settings R particles (new), 2D sprite rendering fields, light2d, renderEffects V body2d, collider2d, tilemap solidity, body3d, collider3d, character3d, joints (new), physicsMaterials (new), collisionMatrix (new), simulation settings (new) P audioSource, music, audio settings (new) A animator, animator3d, animationGraphs (new), visualAnimation, tracks3d (new) N behaviors (script and built-in), tags and props (new), save-data settings S inputBindings (new), ui (new HUD tree), strings (new) U navigation (new), navAgent (new) G prefabs, asset bindings, import settings C 4.3 Shared-file protocol When a card needs a change in a file it does not own: If the change is a registration line in a list created for that purpose, make it directly. Keep it to that line. Otherwise, write the smallest interface the owner should expose, as a TypeScript signature plus a test, in the PR description of a request PR titled game: interface request for <stream> (<card>). Open it against the owner’s files with only that interface and a test. The owning stream’s agent reviews and merges it ahead of its own queue. While waiting, continue with work that does not need the interface. Never copy the owner’s code into your own files as a workaround. Append-only shared files (capabilities/game.ts tool descriptions, evals/surfaces/game.ts, harness/registry.ts filters): add to the end of the relevant list and rebase before merge. 5. Wave 0: foundation Stream K: stabilisation K runs first because several fixes touch files that W4 restructures. K1. Save path correctness (F4, F5, F8, F14). Depends on: none. Owns: web/src/stores/game/, the save and merge sections of GameEditor.tsx and GameEditor3D.tsx, packages/websocket/src/trpc/routers/games.ts save handlers. Do: validate the merged document before diffing into pendingOps. When the server rejects an op batch with an unchanged token, retain the local document, pending operations and undo history, and resend once as a whole-document save (games.saveDraftDocument). Batches above MAX_GAME_DRAFT_OPS save as a whole document directly. If the whole-document save fails, report the error and stop automatic retries until another edit or explicit save action. Make the 3D pull() wait for the in-flight save as 2D does. Serialise concurrent flush() callers. Clear “error” when a later merge or save succeeds. Keep baseUpdatedAt at the latest server token after accepting a conflict. Accept: reproduction tests for each of F4, F5, F8 and F14 in web/src/stores/game/__tests__/ fail before and pass after. The 3D journey web/tests/journeys/native-game3d-editor.spec.ts passes without the model-install save conflict. K2. Play session reuse (F15, F16, F23, F28). Depends on: none. Can run in parallel with K1. Owns useGamePlaySession.ts, useGamePlaySession3D.ts. Do: keep the renderer, font loader, AudioContext and decoded audio across edits. On document change, rebuild only the session, and reload only assets whose bindings changed. Memoise diffGameDocuments and validateGame on the document. Drop paused input. Cap the replay input history with a ring buffer and keep the replay feature working inside the cap. Accept: a test proves the renderer instance survives 10 edits. B1’s editor benchmark (once merged) shows no “Initializing” state on edit. Play-mode React renders per second drop to the HUD’s update rate. K3. 2D child transforms (F9) and selection operations (F10, F12). Depends on: none. Owns GameViewport.tsx, viewportGeometry.ts, GameInspector.tsx. Do: convert between world and parent-local transforms at the viewport seam. Hit-testing, marquee, overlays and frame selection use world transforms. Reparenting preserves world transform. Delete, nudge and drag act on the whole selection and skip descendants of removed entities. The Parent list excludes descendants. Accept: the F9 reproduction (parent at x=5, child at local x=2) passes. Multi-delete of a parent and child produces a valid op list. K4. Publish and restore safety (F6, F7, F18, F19, F20, F25, F26, F27). Depends on: K1. Owns packages/models/src/game.ts, games.ts publish and restore handlers, revisions UI. Do: require the draft token for restore and explicit-document publish. Recover a missing draft version automatically only from a mirror with the same digest. Otherwise preserve its token and history, report the unavailable source, and require a confirmed revision selection to restore the draft. Read baseRevision fresh at publish. Publish the validated document by digest. Wrap post-commit cleanup. Confirm before restore. Return INVALID_INPUT for validation errors. Prune orphans and old revision files. Accept: one test per finding. K5. 3D editor input and parity bugs (F11, F21, F22 partial, F24, F29). Depends on: none. Owns the keyboard and status sections of GameEditor3D.tsx, GameViewport3D.tsx. Do: scope undo shortcuts to the viewport and hierarchy, not text fields. Clear operationError on success. Fix highlight. Wrap localStorage. Leave mouse bindings for U1, which replaces them. K6. Script pane draft conflict (F13) and history growth (F17). Depends on: K1. F17 is solved properly by W5. K6 only caps draftChanges payload size and adds a limit to saveDraft op count. K7. F30 investigation. Capture the error text of the reported crash after Stop on a container without a Vulkan ICD. Fix it or record it as not reproducible with evidence. Stream W: seams W cards are pure refactors. Each must prove no behavior change: the full game selfcheck (game-flow in the harness registry) passes, replay fixtures are byte-identical and golden captures from B2 match. W1. Split protocol schemas into component modules. Depends on: none. Do: move each 3D component schema into packages/protocol/src/game3d/components/<name>.ts and compose gameEntity3D, gameScene3D and gameDocument3D in game3d/index.ts from a list. Do the same for 2D under game2d/. Re-export everything from the existing module paths so imports keep working. Accept: a test serialises z.toJSONSchema (or the repository’s existing schema-to-declaration helper) of every exported schema before and after and asserts equality. Script type declarations in script-types*.ts are byte-identical. W2. Runtime system pipeline. Depends on: W1. Do: restructure the 3D tick in session3d.ts into an ordered list of systems in packages/game-runtime/src/systems/: input, scripts, character, physics step, contacts, gameplay rules, animation state, presentation events. Each system has init(document, scene), step(context), snapshot() and restore(). Add a presentation-event channel that is not part of the snapshot (consumed by particles, audio and debug draw). Record per-system CPU time in a GameStepTimings structure returned with each step, behind a flag so release builds pay nothing. Apply the same structure to the 2D session where it does not change ordering. Accept: replay and characterization fixtures are identical. A test asserts system order. Timings appear in GameStepResult3D when enabled. W3. Renderer module split. Depends on: none (can run with W1). Do: split renderer3d/index.ts into scene-sync.ts, materials/, lights/, environment/, shadows/, animation/, assets/ (GLTF loading and caches), hud/ and a RenderPipeline3D with an ordered pass list (scene pass, overlay pass, HUD pass). Passes register by name and order. Split webgpu.ts into webgpu/sprites.ts, webgpu/lighting.ts, webgpu/effects.ts, webgpu/hud.ts and a pass list with the same idea. Split audio.ts into audio/player.ts and audio/voices.ts. Accept: golden captures from B2 match pixel-for-pixel on the same backend. GameRendererStats3D output is unchanged. W4. Unified editor shell and panel registry (G61 foundation). Depends on: K1, K2, K3, K5. Do: create web/src/components/game/shell/GameEditorShell.tsx used by both 2D and 3D. It owns the toolbar, the left, right and bottom dock regions, the status bar and the keyboard scope. Panels register in shell/panelRegistry.ts with id, title, icon, dimension support (2D, 3D, both) and default region. Move hierarchy, inspector, script pane, changes, revisions, agent panel and viewport into panels/ and viewport2d/ and viewport3d/. Port the 2D-only features listed in the audit’s 3D gap list where the port is mechanical (revisions panel, script error tools). Leave the rest to E cards. Accept: both editors render through the shell. Existing journeys pass (native-game-editor.spec.ts, native-game3d-editor.spec.ts, native-game-audio.spec.ts). A test registers a dummy panel and finds it in the layout. W5. Labelled command bus for undo and redo (G64, F17). Depends on: W4. Do: replace the anonymous undo stack in GameDraftStore with commands { label, ops, inverseOps, mergeKey? }. Consecutive commands with the same mergeKey within one gesture coalesce (a gizmo drag is one command). 3D edits produce op-level commands, not set_document. Expose a history list for E cards. Accept: dragging an entity and undoing once restores it. Undo labels read “Move Player”, “Change Light Intensity”. A 3D edit produces ops other than set_document (fixes F17’s growth). W6. Skill and docs scaffolding. Depends on: none. Do: add one heading per stream to packages/system-skills/native-game/SKILL.md (empty sections allowed) so streams edit disjoint sections. Add docs/plans/native-game-parallel-roadmap.md that links this plan’s card IDs to PRs as they land. Run npm run check:agents-docs. Stream B: baselines B1. Benchmark harness. Depends on: none. Do: add nodetool game bench <file> (or a vitest bench in packages/game-runtime/bench/) that runs N ticks headless and reports p50, p95 and p99 tick time, per-system time (after W2), script time and allocations. Add fixtures: bench-2d-500.json (500 entities, 32 scripted, tilemap), bench-3d-1000.json (1,000 entities, 100 dynamic bodies, 0 scripted, 10 models), and bench-3d-64-scripted.json (64 entities, 61 dynamic bodies, 30 scripted, 10 models). Add a browser bench that renders 600 frames and reports frame time, draw calls and triangles, using capture’s browser path. Extend packages/game-renderer/scripts/benchmark-effects.ts rather than duplicating it. The 2D fixture uses the runtime limit of 32 scripted behaviors. The 3D workloads separate scale from scripting because the combined 1,000-entity script input exceeds the 64 KiB budget. The scripted companion completed a 120-tick replay probe within existing budgets. Accept: running it twice on the same machine gives p95 within 10%. Baselines saved to docs/plans/native-game-baselines.md with machine, browser and backend recorded. B2. Golden image set. Depends on: none. Do: capture fixed-tick frames of every shipped example game and the bench fixtures with nodetool game capture. Store with a perceptual-diff tolerance test (SSIM or per-pixel threshold) so later cards can show visual change deliberately. B3. Exported build smoke test (G72). Depends on: none. Do: after nodetool game build, serve the output, load it in headless Chromium, run 300 frames with scripted input, fail on console errors, missing assets or a stalled tick. Add as nodetool game smoke <dir> and to the game-flow selfcheck. 6. Wave 1 and later: workstreams Each card lists dependencies on other cards. Cards without cross-stream dependencies start as soon as Wave 0 is merged. Stream R: 3D rendering Targets: 60 fps at 1280×720 on a mid-range desktop GPU for bench-3d-1000 with all R1–R4 features on at the “high” tier. Software rendering in CI proves correctness only. R1. Image-based lighting and sky (G2). Do: add environment.sky to gameEnvironment3D: { kind: "color" } (today’s default), { kind: "hdri", assetId, rotation, intensity }, { kind: "procedural", sunEntityId?, turbidity, rayleigh, groundColor }. Generate a PMREM environment map for both. Use it as scene.environment with environmentIntensity. Link the procedural sun to a directional light. Add an hdri asset media kind (equirectangular .hdr or .exr, size budget) with preparation in stream C’s pipeline through an interface request. Accept: golden captures with each sky kind. A PBR sphere grid fixture shows reflections. Default documents render identically to before. R2. 3D post-processing stack (G3). Depends on: R1. Do: add postProcessing to the scene (and later volumes) with ordered effects: exposure, tone mapping choice (ACES, AgX, Neutral), bloom, SSAO or GTAO, colour grading with LUT (reuse the 2D LUT asset format), SMAA or FXAA, vignette, depth of field, chromatic aberration. Implement as passes in RenderPipeline3D using three’s EffectComposer on WebGL2. Keep each effect’s parameters bounded. Reuse the parameter vocabulary of 2D renderEffects where meanings match. Accept: each effect has a golden capture. B1 browser bench shows the cost per effect. Disabling all effects reproduces R1 output. R3. Shadows (G4). Do: cascaded shadow maps for the directional light (2 to 4 cascades, configurable split), shadows on point and spot lights with a per-light toggle and a scene-wide budget (for example at most four shadowed local lights), shadow bias and normal bias fields. Accept: a 200 m scene fixture shows sharp near shadows and continuous far shadows. Budget overflow produces a diagnostic, not a crash. R4. Material system (G5). Do: give primitives texture slots (base colour, normal, ORM, emissive) with UV tiling and offset. Support glTF extensions KHR_materials_clearcoat, transmission, sheen, specular, ior, volume in preparation’s allow-list (preparation.ts is owned by C, so request the allow-list change). Add material presets as named document-level materials (materials: Record<id, Material3D>) that entities reference, plus a toon shading model with outline option. Keep arbitrary shader code out of scope. Accept: generated textures can be applied to a box via the agent. Toon preset golden capture. R5. Environment and lighting presets (G58). Depends on: R1, R2, R3. Do: ship preset bundles (sky, sun, ambient, fog, post) as data in packages/protocol/src/game3d/presets.ts. Expose them through an “apply preset” document op and an editor control registered as an inspector editor for scene settings. Accept: each preset has a capture. The agent can apply one by name. R6. Height fog and decals (G9). Do: exponential height fog alongside linear fog. Decal component projecting a texture onto receivers, bounded count per scene. R7. Selection outline and editor overlay passes (G65, G48 rendering part). Do: an editor-only outline pass for selected ids, light range and cone helpers, and a wireframe, unlit and overdraw view mode switch in the pipeline. These are editor passes, excluded from player and capture builds. Accept: E6 consumes the view modes without touching renderer files. R8. WebGPU 3D renderer (G1). Depends on: R1–R7, V3, U2, N1 renderer parts merged. Declares the renderer freeze. Do: switch to three’s WebGPURenderer with automatic WebGL2 fallback. Port post effects to TSL node post-processing. Keep GameRendererCapabilities3D.backend truthful (webgpu or webgl2). Share the GPU device with packages/gpu where possible. Accept: all R and V golden captures match within tolerance on both backends. Fallback path tested by forcing WebGL2. B1 numbers recorded for both. R9. Instancing, batching and LOD (G6). Depends on: W3. Better after C4 (LOD generation). Do: automatically instance entities sharing a model or primitive and material when they have no per-instance animation. Use BatchedMesh for static mixed geometry. Support LOD levels from model bindings, with screen-size thresholds. Accept: bench-3d-1000 draw calls drop by at least 5× with the same capture. R10. Quality tiers and render scale (G41, G70). Do: document-level quality tiers (low, medium, high) that map to shadow, post and resolution settings. Dynamic resolution that holds a target frame time. Fixed internal resolution with integer scaling option for pixel-art 3D. Player-facing setting surfaced through U3’s settings screen. R11. Lightmap baking (G8). P2. Do: server-side bake via the existing headless Blender (packages/blender-nodes) for static geometry, stored as a prepared asset, applied as a second UV lightmap. Light probes for dynamic objects. Stream V: 2D rendering and VFX V1. Particle core (G10). Do: packages/game-renderer/src/particles/ with a CPU simulator: emitters with rate and bursts, shape (point, circle, sphere, cone, box, edge), lifetime, start speed, size, rotation and colour ranges, curves over lifetime, gravity, drag, sub-emitters on death, world or local space, max particles per emitter and per scene. Particle state is presentation only. Seed per emitter from the entity id so captures are stable. Add the particles component schema and emitParticles script command (a presentation event from W2). Accept: unit tests for curve evaluation and emission counts. Particle simulation never appears in snapshots (test). V2. 2D particle rendering. Depends on: V1. Do: render particles in the WebGPU sprite path as instanced quads with additive or alpha blend, sprite sheets, and lighting opt-out. Canvas2D fallback with a lower cap. V3. 3D particle rendering. Depends on: V1, W3. Do: billboard, stretched-billboard and mesh particles as a RenderPipeline3D pass. Soft particles with depth fade when depth is available. V4. Curve and gradient editor primitives. Do: ui_primitives/CurveEditor and GradientEditor usable by inspector editors (V5, N, A, E10). Keyboard accessible. V5. Particle inspector and presets. Depends on: V2, V3, V4. Do: inspector editor for the particle component with live preview, and a preset library (fire, smoke, sparks, dust, magic, rain, snow, explosion). V6. 2D normal maps and shadows (G11). Do: optional normal-map binding per sprite (generated through stream C from a depth model), normal-mapped lighting in the light pass, and 2D shadow casters from colliders. V7. Nine-slice, masks and sprite effects (G12). Do: nine-slice sprites (also used by U2), sprite masks, and a bounded set of sprite effects (outline, dissolve, flash, palette swap) as parameters, not custom shaders. V8. Build-time atlas packing (G37) and GPU particles. Depends on: R8 for GPU particles. Do: pack small sprites into atlases in nodetool game build. Add a compute-shader particle path on WebGPU for emitters above a threshold. Stream P: physics P1. Rapier 2D engine (G13, G14). Sub-PRs: P1a schema and engine version, P1b spatial adapter, P1c migration tool, P1d editor support. Do: introduce a new 2D schema version and engine version (per native-game-3d-upgrade.md D1) whose physics runs on @dimforge/rapier2d-compat pinned exactly. Reuse the spatial seam the 3D upgrade introduced (native-game-3d-upgrade.md D2) with a 2D adapter. Support static, kinematic and dynamic bodies, box, circle, capsule and convex polygon colliders, rotation, one-way platforms (via contact modification or solver hooks), tilemap colliders merged into chains, sensors, and a 2D character controller with the same settings as character3d plus platformer features (coyote time, jump buffer). Old 2D documents keep running on engine “1” unchanged. Add nodetool game migrate --to <version> that converts a document and reports behavior differences found by simulating both. Accept: the platformer example migrated plays its scripted input to the same win tick within a stated tolerance. bench-2d-500 physics time is lower than engine “1”. P2. Joints (G15). Do: a joints list on the scene: fixed, revolute (hinge with limits and motor), prismatic (with limits and motor), spherical, rope and spring, connecting two entity ids with anchors. Joint state in the snapshot via Rapier serialisation. Script commands to set motor targets. Accept: door hinge, chain and piston fixtures replay deterministically. P3. Physics materials and combine rules (G16). Do: document-level physicsMaterials with friction, restitution and combine modes. Colliders reference a material id or keep inline values. P4. Collision layer matrix (G66). Do: document-level named layers and a symmetric collision matrix. Colliders pick a layer by name. Category and mask bits are derived. Keep raw bits as an advanced override for existing documents. Accept: existing documents produce the same bits. The inspector editor (registered by P) shows the matrix. P5. Forces and torques (G68). Do: addForce, addTorque, addForceAtPoint script commands, accumulated for one tick. P6. Simulation settings (G69, G18). Do: document-level simulation with tick rate from an allowed set (30, 60, 120), physics substeps, solver iterations, and a default seed. Every consumer of tickRate: 60 reads the setting. Hosts pass the seed. Accept: a 120 Hz fixture runs at real time in the player. Existing documents are unchanged. P7. Physics debug data. Do: expose contacts, normals, sleeping state and query rays as presentation events or a debug snapshot for E6. P8. Vehicles and ragdolls (G17). P2. Depends on: P2, N1. Do: a raycast vehicle component on Rapier’s vehicle controller. Ragdoll generation from a humanoid skeleton with joints, switchable from animation. P9. Interest management for simulation (G38 part). Do: optional per-entity simulation distance and sleeping of scripts and bodies outside it, with determinism preserved (activation decided by simulation state only). Stream A: audio A1. Mixer and buses (G20). Do: a bus graph in audio/: master (with limiter), music, sfx, voice, ui, and user-defined buses. Per-bus volume, mute, low-pass, and send to reverb. Ducking rules (music ducks under voice). Snapshot transitions between mixer states on events. Document-level audio.mixer schema. Accept: unit tests with an OfflineAudioContext render known gain envelopes. A2. Spatial audio (G19). Depends on: A1, W2 (presentation channel carries emitter transforms). Do: per-voice PannerNode (HRTF or equal-power by quality tier), listener from the active camera, distance models and rolloff curves, cone settings, doppler. audioSource gains spatial, minDistance, maxDistance, rolloff. Positions update every rendered frame from interpolated transforms. Accept: offline-rendered test shows left-right panning for an emitter at x=±5. A3. Streaming music (G21). Do: long tracks play through MediaElementAudioSourceNode routed into the music bus, with loop points and crossfades. Sync offset to snapshot tick on resume as today. A4. Sound design features. P2. Do: random containers (one of N), pitch and volume variance, priority-based voice stealing, reverb zones (convolution with bundled impulse responses), simple occlusion by raycast against static colliders (presentation-only query). Stream N: animation N1. Animation graph runtime (G22). Do: animationGraphs at document level, referenced by animator3d.graph. States with clips or blend spaces (1D and 2D), transitions with conditions on parameters (float, bool, trigger), exit time and duration, layers with bone masks and additive mode. Graph state (current state, transition progress, parameter values) is simulation state in the snapshot. Pose sampling stays presentation. Script commands setAnimParam. Keep playAnimation working by mapping it to a direct state change. Accept: idle, walk and run blend by speed in a capture sequence. Replay is identical. N2. Animation events and root motion. Do: events at clip times emitted deterministically from graph state (footstep sound, spawn hit box). Optional root motion applied to the character controller as simulation input. N3. Humanoid retargeting and shared clip libraries (G23). Do: at preparation (interface request to C), detect common humanoid rigs (Mixamo, VRM, Unreal mannequin naming) and map to a canonical humanoid. A shared library asset of clips on the canonical rig. Runtime retargets library clips onto any mapped model. Accept: a generated rigged model with no clips plays library idle, walk, run and jump. N4. IK (G24). Do: two-bone IK for feet with ground raycast (presentation-only query) and look-at for head. Toggles per animator. N5. 3D tracks (G25). Do: keyframed transform, light and material tracks per entity with easing, like 2D visualAnimation, plus a kinematic mode that moves physics bodies deterministically (moving platforms without scripts). N6. Clip preview in the editor. Do: inspector editor for animator3d with a clip list, scrubber and play in the viewport without entering play mode. N7. Animation graph editor (G57). Depends on: N1, W4. Do: a bottom-dock panel that edits the state graph with @xyflow/react, matching the visual language of the workflow editor. Parameter list, transition inspector, live state highlight during play. Stream S: scripting and gameplay runtime S1. Script input cost (G26). Do: stop embedding input JSON in evaluated source per call. Keep a function resident per behavior instance only when a positive AST check proves an input-only expression cannot retain hidden state. All other sources keep fresh-context execution and existing guest JSON parsing behavior through native call arguments. Build one immutable host world snapshot at tick start, before movement behaviors or script commands. Add global world.get(id) and world.query({source?, tag?, near?, radius?, limit?}), returning private entity copies and ordered IDs respectively. Limit combined query attempts to 64 per call and results to 1024 IDs. Legacy input.world remains a private mutable copy, created lazily on the resident path, with its existing shape and 2D collider/camera population. Keep the pinned declaration strings, document schemas, and 3D contractVersion: 3 unchanged. Compose additive world declarations in Monaco. Tags remain unauthored until S2, so tag filters match nothing. JSON normalization and budget accounting remain in both paths. Accept: B1 records script-stage p99 below 25 ms for bench-3d-64-scripted, half the batch budget. Report before/after nodetool game bench measurements on the same host for all three approved fixtures, with fixture bytes unchanged: bench-2d-500 (500 entities, 32 scripted), bench-3d-1000 (1,000 entities, none scripted), and bench-3d-64-scripted (64 entities, 30 scripted). CI runs 120 ticks with seed 1 and replay from tick 60 for all three. Existing limits, script tests, compatibility assertions, and goldens remain unchanged. Budgets still interrupt runaway scripts. S2. Tags and entity properties (G27). Do: tags: string[] and props: Record<string, JSON> on entities, both 2D and 3D. Scripts read entity.tags, entity.props, other entities’ tags and props, rotation and active state. Props can be changed by script commands and are snapshot state. Integrate authored tags with S1’s tick-start world projections and ScriptWorldSnapshot in both dimensions. Cover tag filtering combined with source, radius, document ordering, and result limits. Until S2’s schema and runtime integration land together, S1 tag queries intentionally return no matches. S3. Script parameters bound to the inspector (G71). Depends on: S2, W4. Do: a script behavior declares params as a JSON-schema-like object (number with range, boolean, colour, enum, entity reference, asset reference, vector). The behavior stores values. The inspector renders them through SchemaFields. Scripts receive input.params. Validation checks entity and asset references. Accept: changing a param in the inspector changes play behavior without editing source. The agent can set params through ops. S4. Lifecycle hooks and timers (G28). Do: a script may export an object with onStart, onUpdate, onFixedUpdate (alias of update at tick rate), onContact, onTriggerEnter, onTriggerExit, onDestroy, onSceneEnter. Keep the plain-function form working. Add deterministic timers (after(ticks, name), every(ticks, name)) stored in snapshot state. Accept: hook order test. Timer replay identical. S5. Typed events (G32). Do: emit gains an optional payload (bounded JSON) and target (entity id, tag, or broadcast). Hooks receive payloads. S6. Script hot reload during play (G67). Depends on: S1, K2. Do: on script edit during play, recompile the changed script and continue the session from the current tick with existing script state. If compilation fails, keep the old function and show the error in the console (E3). Offer “restart from tick 0” separately. Accept: edit a speed constant while playing and see the change without the tick resetting. S7. Player saves (G30). Do: a save/load script command and player API backed by a per-game storage adapter (IndexedDB in the standalone player, project storage in the editor). Save data is a bounded JSON object plus optional full snapshot. Settings saves are separate from progress saves. S8. Persistent entities and additive scenes (G31). P2. Do: persistent: true entities survive scene transitions. Additive scene load and unload commands. Stream U: input and game UI U1. Input bindings (G33, F22, F23). Do: document-level inputBindings: per action, a list of bindings (key, mouse button, gamepad button, gamepad axis with dead zone and inversion, touch control). Axes composite from keys or sticks. Defaults generated from today’s naming rules so existing games behave the same. Gamepad polling per tick via navigator.getGamepads. Pointer lock for look in 3D. 3D touch controls in the standalone player. Player-facing rebinding stored through S7 when available. Accept: existing example games play identically with generated default bindings. A gamepad simulated in Playwright drives the 3D fixture. U2. HUD widget tree (G34). Do: a ui tree on the document and scene: nodes panel (colour or nine-slice image from V7), image, text, bar (value bound to a script-set variable or entity health), button (emits an event), stack and grid containers. Anchors and pivots relative to the HUD rectangle, safe-area aware. Script commands to set text, values and visibility. Render in both the WebGPU 2D HUD pass and the 3D HUD module with the same layout code (shared layout engine in src/ui/). Keep hud labels working as a compatibility layer. Accept: a health bar, score panel and pause button fixture renders identically in 2D and 3D captures. Buttons work with mouse, touch and gamepad focus navigation. U3. Screens and menus. Depends on: U2, S7. Do: screens (title, pause, settings, game over, custom) built from U2 nodes, with transitions and pause semantics. A default settings screen exposes volume buses (A1), quality tier (R10) and bindings (U1). U4. Localisation and rich text (G35). P2. Do: string tables keyed by locale, t("key") in UI and scripts, simple markup for colour and inline icons, right-to-left layout. Stream G: navigation and AI G1n. Navmesh bake (G29). Do: bake navmeshes from static colliders and walkable surfaces with recast-navigation-js (WASM, pin version) at publish and on demand in the editor. Store as a prepared asset with digest. Agent radius and height profiles. G2n. Path queries and agents. Depends on: G1n. Do: navAgent component with speed and avoidance, moveTo and findPath script commands, crowd simulation run deterministically inside the simulation tick. Accept: replay identical with 20 agents. G3n. 2D grid pathfinding. Do: A* over tilemaps and a grid derived from colliders, same findPath command. G4n. Navmesh debug overlay. Register an overlay for E6. Stream E: editor UX and tooling E1. Docking and saved layouts (G62). Depends on: W4. Do: resizable, tabbed, drag-to-dock regions with layout persistence per user and named layouts (Default, Scripting, Animation, Wide). Prefer existing workspace panel primitives. A docking library is allowed after dependency review. Implemented without a docking library. GameEditorShell renders shell/GameDockCanvas.tsx, which lays out the left, right, bottom and viewport regions as tabbed groups (GameDockGroup.tsx) with resize handles, drag-to-dock and a keyboard-accessible Move control. GameLayoutMenu.tsx applies the four presets, shows hidden panels, and saves, renames and deletes named copies. Each panel’s content stays mounted in one StableGamePanelHost element that is moved between group slots, so editor state, focus and the 3D canvas survive a move. The host carries data-game-panel="<panel id>" for shortcut routing (for example [data-game-panel="viewport"]) and data-game-undo-scope for panels that accept editor shortcuts. An inactive or detached host is hidden and inert and releases a pointer lock held inside it. stores/game/GamePanelLayout.ts validates layouts with Zod and applies layout actions. An action that changes nothing returns the same layout object, and the store then skips the update, the render and the storage write. stores/game/GamePanelLayoutStore.ts persists the current layout and up to 32 named copies in browser storage per signed-in user, or for the anonymous user, under nodetool.game-layout.v1:<scope>. On load, an invalid current layout falls back to Default and each invalid or duplicate named copy is dropped on its own, so valid copies survive. Placements of panels that are not registered yet are kept, so extension panels return to their place when they register. Registered panels missing from a layout are added to their default region. Invalid names and a full list are reported as readable messages in the layout menu. E2. Shortcut manager and command palette (G60). Do: one registry of editor commands with default shortcuts shared by 2D and 3D, a rebinding view, and a command palette (Ctrl/Cmd+K) that also lists agent actions. E3. Console panel (G52). Depends on: W2 (diagnostic channel). Do: errors, warnings and script log output with tick, entity link, collapse of repeats, filters, and “Ask the assistant” per line. E4. Profiler panel (G51). Depends on: W2, B1. Do: frame-time graph, per-system CPU time, script time per entity, draw calls, triangles, texture and geometry memory, physics body and contact counts, audio voices. Capture and export a profile. E5. Gizmos, snapping and multi-select (G45, G46). Do: local or world toggle, pivot mode, snap increments per mode, surface snap and align to ground (raycast), multi-select with one gizmo in 3D, orbit around selection, scene orientation gizmo with axis views and orthographic toggle. Implemented in viewport3d/GameViewport3D.tsx with the math in stores/game/gameGizmo3D.ts and the preferences in stores/game/GameViewportSettings3DStore.ts. The gizmo drives one pivot object. A drag reads the pivot’s change in its own frame and replays it on each selected root, so a child of a selected parent is not moved twice, and the drag commits every root in one op batch, which is one undo step. Surface snap and Align to ground raycast against the rendered entity objects outside the selection and fall back to the ground plane at height zero. The settings are per-user browser preferences under nodetool.game-viewport3d.v1:<scope>, not document state, so they need no schema change or agent capability. Vertex snapping (G46) is not covered. E6. Debug view modes (G48). Depends on: R7, P7. Do: viewport menu for wireframe, unlit, overdraw, colliders, contacts, sleeping bodies, light bounds, navmesh (G4n), audio emitter ranges. E7. Scene view and game view (G49). Do: a separate game view panel showing the active camera at the game’s aspect ratio while the scene view stays free, plus camera preview inset on camera selection and camera bookmarks. E8. Live runtime inspection (G53). Depends on: K2. Do: select runtime entities during play, see live state (transform, velocity, script state, animation state), edit values on the live session, and “apply to document” for selected fields. E9. Script debugging and replay scrubber (G54). Depends on: E3. Do: a timeline scrubber over replay history, step a single tick, watch an entity’s script state over ticks, and per-entity script error tools in 3D (port from 2D). E10. Component editors (G55). Depends on: V4. Do: inspector editors registered per component: colour picker with HDR intensity, asset picker with thumbnails (from C1), collider “fit to mesh”, light colour temperature, camera preview, material editor with preview sphere (with R4). E11. Onboarding (G63). P2. Do: empty-scene guidance, component tooltips sourced from schema descriptions, starter scenes. Stream C: content pipeline and assets C1. Asset browser (G50). Depends on: W4. Do: a bottom-dock panel listing assets, prefabs and scenes with thumbnails (rendered with the game renderer for models, generated for audio as waveforms), search, type filters, references (“used by”), replace-in-place, and slot actions: generate, regenerate with a new prompt, pick from staged candidates. Generation goes through the existing slot prompt and staging flow (packages/game-nodes, game-slot-prompt.ts). C2. Drag to place (G47). Depends on: C1, E5 (surface raycast). Do: drag assets and prefabs into the viewport to create entities on the surface under the cursor, with snapping. C3. Model optimisation at preparation (G7). Do: in renderer3d/preparation.ts, use @gltf-transform/functions and meshoptimizer to weld, simplify above budget, quantise, compress with meshopt, and transcode textures to KTX2 with a size cap per tier. Accept Draco and meshopt inputs by decoding at preparation. Register MeshoptDecoder and KTX2Loader in the loader module (owned by C). Bump preparationVersion. Accept: a 40 MB generated model prepares to under a stated size with an SSIM-bounded capture. Existing prepared assets still load. C4. Import settings (G36). Do: material overrides at import, collider generation options including convex decomposition (V-HACD or CoACD WASM after dependency review), LOD generation for R9, pivot editing and texture max size. C5. Prefab editing (G56). Depends on: W4, W5. Do: open a prefab in isolation in the viewport, nested prefabs, variants, per-field apply and revert of overrides, and override markers in the inspector. C6. Per-scene asset loading (G40). Do: scene asset manifests, progressive loading with a loading screen (U3), and unloading on scene exit. C7. Greybox editing (G59). P2. Do: face handles on box primitives to extrude and resize, with UV-preserving world-space texturing. Stream D: performance architecture and delivery D1. Simulation in a worker (G39). Depends on: W2, S1. Do: run the session in a dedicated worker in the player and editor play mode. The main thread sends input frames and receives render frames and presentation events through transferable buffers. Keep a main-thread fallback. Accept: B1 shows render frame time unaffected by a deliberately slow script up to the script budget. D2. Frame budget and distance culling (G38). Do: render-side distance culling per entity and per layer, and budgets that warn in the console when exceeded (draw calls, triangles, particles, voices). D3. PWA and embedding (G42). Do: manifest and service worker in standalone builds, offline play, and an embed snippet with a postMessage API (start, pause, events, score). D4. Shipped-game diagnostics (G43). Do: optional error reporting hook in standalone builds that respects the project’s error-tracing redaction rules (docs/error-tracing.md). D5. Release checks. Do: add B3’s smoke test to the publish flow, and size budgets for builds. Stream M: milestone games M builds showcase games that exercise many streams at once. Each milestone is the integration test for the cards it lists. M works through the agent and the editor, not by hand-editing JSON, and files bugs back to the owning stream. M1. 3D third-person showcase. After R1–R4, U1, U2, N1, A2, S1–S4. A generated character with library animations, physics props, a HUD with health and score, spatial audio and post-processing. Must hit the R stream frame target. M2. 2D physics platformer. After P1, V2, V5, U1, U2, A1. Dynamic crates, slopes, particles and menus. M3. Full loop. After U3, S7, C1–C3, B3. Title screen to win screen with saved progress, built and smoke-tested as a standalone build. 7. Dependency graph Arrows read “must merge before”. K1 ─┬─> K4, K6 └─> W4 K2 ─┬─> W4 ├─> S6 └─> E8 K3 ─> W4 K5 ─> W4 W1 ─> W2 ─┬─> S1 ─> S6, D1 ├─> E3 ─> E9 ├─> E4 (also needs B1) ├─> A2 (also needs A1) └─> V1 (presentation channel) W3 ─┬─> R1 ─> R2 ─> R5 (also R3) ├─> R3, R4, R6, R7, R9 ├─> V3 (also V1) ├─> U2 └─> C3 (loader module) W4 ─┬─> W5 ─> C5 ├─> E1, E2, E5, E7, E10, C1 ─> C2 ├─> S3 (also S2) └─> N7 (also N1) R1–R7 + V3 + U2 + N1 ─> R8 ─> V8 P2 + N1 ─> P8 R7 + P7 ─> E6 U2 + S7 ─> U3 ─> M3 G1n ─> G2n 8. Suggested allocation for twelve agents Agent Wave 0 After Wave 0 1 K1, K4, K6 R1, R2, R5, R8 2 K2, K5 S1, S2, S3, S6 3 K3, K7 E5, E6, E7, E8 4 W1, W2 S4, S5, S7, S8 (S takes over session ownership) 5 W3 R3, R4, R6, R7, R9, R10 6 W4, W5 E1, E2, E3, E4, E9, E10 7 W6, B1, B2, B3 D1, D2, D3, D4, D5 8 (starts after Wave 0) P1 (all sub-PRs), P4, P6 9 (starts after Wave 0) P2, P3, P5, P7, P9, then G1n–G4n 10 (starts after Wave 0) V1–V8 11 (starts after Wave 0) U1–U4, then A1–A4 12 (starts after Wave 0) C1–C7, then N1–N7 M1–M3 run on whichever agent frees up first after their dependencies land. With fewer agents, keep the stream boundaries and give one agent several streams in sequence. Do not split one stream across two agents at the same time. 9. Launch prompt for a workstream agent Fill in <STREAM> and give the agent this prompt, this plan and the gap analysis. You are the implementation agent for stream <STREAM> of the NodeTool native game engine roadmap. Repository: nodetool-ai/nodetool. Read AGENTS.md, docs/DEVELOPMENT_STANDARDS.md, docs/HARNESS_FIRST.md, docs/plans/native-game-3d-upgrade.md and the implementation plan at docs/plans/native-game-implementation-plan.md before writing code. Work through the task cards of stream <STREAM> in order. For each card: 1. Check that every card in its "Depends on" line is merged to main. If not, take the next card whose dependencies are met. If none is available, stop and report what you are blocked on. 2. Create a branch named game/<stream>-<card id>, for example game/r-r1. 3. Re-check the file and line references in the card against current main. 4. Write failing tests first for new behavior and for bug reproductions. 5. Implement the card within the files your stream owns (section 4). For anything else, use the shared-file protocol (section 4.3). Never edit another stream's files beyond a registration line. 6. Keep determinism (section 2.3) and schema rules (section 2.4). Presentation state never enters snapshots. 7. Make the feature reachable by the in-product agent and the nodetool game CLI (section 2.5). 8. Update your stream's section of packages/system-skills/native-game/SKILL.md when authoring changes. 9. Run the four mandatory checks and fix every failure before opening the PR. 10. Open one PR per card, titled "game: <summary> (<gap and finding ids>)". In the body, list what changed, the tests added, benchmark numbers for performance cards and any capture diffs. 11. Drive the PR to green CI. Address review comments. Then move to the next card. Do not start cards from other streams. Do not widen a card's scope. If a card's design turns out to be wrong, stop, write down why with evidence in the PR description, and propose the change instead of silently implementing something else. 10. Risks and how the plan handles them Risks use X codes so they do not collide with stream R card IDs. ID Risk Handling X1 Seams in Wave 0 take longer than expected and block everyone. W1, W3 and W6 have no dependencies and run in parallel. K runs alongside. Streams P and G can start design spikes (no merges) during Wave 0. X2 Replay divergence from runtime changes. Section 2.3 plus fixtures in every runtime PR. GAME_PHYSICS_BUILD_3D and new engine versions for behavior changes. X3 Visual regressions slip through parallel renderer work. B2 golden captures with tolerance, required in every R, V and U PR. Renderer freeze for R8. X4 Schema growth makes documents hard for the agent to author. Every new field needs an eval case in evals/surfaces/game.ts and a skill section update. Defaults reproduce existing behavior. X5 WebGPU support varies across browsers. R8 keeps automatic WebGL2 fallback. All features except GPU particles must work on WebGL2. X6 Rapier 2D migration changes the feel of existing 2D games. Engine “1” stays. Migration is opt-in with a behavior report (P1c). X7 New WASM dependencies (Rapier 2D, Recast, decomposition) increase load size. Lazy-load per feature, record size in the PR, and include them in B3’s size budget. X8 Agents duplicate work across streams. Ownership map in section 4 and interface-request PRs. One agent per stream at a time.--- # K7: Stop with no host Vulkan driver Source: https://docs.nodetool.ai/plans/native-game-k7-investigation.html K7: Stop with no host Vulkan driver The crash reported in F30 did not reproduce in one run of the 2D editor journey on the current implementation working tree, based on commit 81288ab79aa0ff90391ad1618108e80eb793b8f7. No application code changed for this investigation. Reproduction Run from the repository root after building packages. Ports 3000 and 7777 must be available. Verify that the driver path does not exist before running. test ! -e /tmp/native-game-missing-vulkan-icd.json env \ VK_DRIVER_FILES=/tmp/native-game-missing-vulkan-icd.json \ VK_ICD_FILENAMES=/tmp/native-game-missing-vulkan-icd.json \ DEBUG=pw:browser \ mise exec node@24.18.0 -- npm run test:journeys --workspace=web -- \ native-game-editor.spec.ts --workers=1 \ > /tmp/k7-no-vulkan.log 2>&1 native_game_journey_status=$? printf '%s\n' "$native_game_journey_status" > /tmp/k7-no-vulkan.exit exit "$native_game_journey_status" The environment overrides reach the seeded backend and Chromium through the existing journey setup. DEBUG=pw:browser captures browser-process diagnostics that the journey’s normal error filter can omit. Observed result The command exited with status 0 and reported 1 passed (26.3s). The journey reached Play, observed advancing ticks, clicked Stop, verified the saved document, undid the change, and published a revision. The seeded backend became ready before the journey and stopped during normal teardown. Chromium used chromium_headless_shell-1223 with the existing configuration’s --use-angle=swiftshader flag. Its stderr included these errors before the journey completed: ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED Exiting GPU process due to errors during initialization The evidence is in /tmp/k7-no-vulkan.log and the captured command status is in /tmp/k7-no-vulkan.exit. These temporary artifacts are local evidence, not required inputs to the reproduction. Interpretation and limits The observed GPU initialization errors did not prevent Stop or the remaining journey actions. Stop resets client play state. It does not call a backend Stop endpoint. The browser renderer can fall back to Canvas2D when a WebGPU adapter is unavailable. This source behavior is consistent with the passing result, but this run did not instrument the chosen renderer backend. The overrides force an absent host driver manifest. Chromium can also use its bundled SwiftShader implementation. This does not recreate every property of the original container, whose crash error text remains unavailable. The test used the current implementation changes rather than an untouched baseline. The result closes K7 as not reproduced in this tested configuration and does not establish that the historical report was incorrect.--- # Native game implementation links Source: https://docs.nodetool.ai/plans/native-game-parallel-roadmap.html Native game implementation links Use the implementation plan for card scope, ownership and dependency gates. The editor audit and gap analysis define the findings each card addresses. Add a pull request link beside its card when it is opened. A card starts only after its dependencies have merged, as specified in the plan. Stream Cards Pull requests K K1–K7 K1: draft save correctness, K2: play resource reuse, K3: child transforms and selection, K4: publication and restore safety, K5: input and parity fixes, K6: script conflicts and bounded draft history, K7: missing-Vulkan investigation W W1–W6 W1: component schemas, W2: runtime systems, W3: renderer modules, W4: editor shell and panel registry, W5: labelled undo commands, W6: workstream scaffolding B B1–B3 B1: supported workload benchmarks, B2: example and benchmark goldens, B3: exported build smoke test R R1–R11 R1: image-based lighting and sky V V1–V8   P P1a–P1d, P2–P9   A A1–A4   N N1–N7   S S1–S8 K4 publication interface request U U1–U4   G G1n–G4n   E E1–E11   C C1–C7   D D1–D5   M M1–M3  --- # Project View — Implementation Plan Source: https://docs.nodetool.ai/plans/project-view/PLAN.html Project View — Implementation Plan Make the project the unit of the workspace instead of the document. Every document row already carries a project_id column (packages/models/src/db.ts — timeline_sequences, image_documents, applications, and siblings) and every creation site hardcodes projectId: "default". This plan spends that column: a project is a tab group plus one overview tab inside the existing WorkspaceShell. No new window, no mode. The fifteen document editors keep the center area untouched. Mockups Four 1440×900 screens in mockups/ (self-contained HTML, inline styles; open in any browser). Also published as an editable canvas: https://claude.ai/code/artifact/9fde1641-9965-4f71-bb23-2543ef4bc4db File Screen Referenced by mockups/storyboard-board.html Storyboard editor restyled as a shot grid, inside a project tab group Phase 0, A1–A4; Phase 2, B1–B2; Phase 5, E1 mockups/projects-list.html Projects list tab Phase 2, B3; Phase 3, C4 mockups/project-overview.html Project overview tab, populated (Aurora Launch Spot) Phase 3, C1–C3; Phase 4 (spend) mockups/new-project.html Full-view “Start a project” surface Phase 4, D1–D3 The mockups draw the shipped chrome from real values: 40px tab bar (c_app_header #0A0B0D), 50px full-height rail, 32px bottom strip, divider rgba(255,255,255,0.08), fonts Inter 22/18/15/13/11 + JetBrains Mono, radii from theme.rounded. Anything in a mockup that differs from a token in web/src/components/themes/ThemeNodetool.tsx or docs/DESIGN.md — the tokens win. Type glyph colors come from colorForType in web/src/config/data_types.ts (the mockups hardcode the same hex values). Project identity color everywhere: info.light cyan #67E8F9 — rail icon, scope chip, group underline, ◆ glyph. Ordering Phase 0 ships alone, first (explicit request: the restyled storyboard is wanted regardless of the project layer). Phases 1→5 build on each other; 1 and 0 are independent. Rules that apply to every phase: ui_primitives only (no raw MUI), design tokens only (SPACING / BORDER_RADIUS / MOTION / Z_INDEX / var(--fontSize*)), media through ResponsiveImage/VideoPlayer with locator, and after each phase npm run test:affected && npm run typecheck && npm run lint must pass. Phase 0 — Restyle the storyboard board as a shot grid Reference: mockups/storyboard-board.html, center area. Today (web/src/components/storyboard/StoryboardBoard.tsx) the board is a large settings Panel (Screenplay + Direction FormGrid, then a wide button row) followed by a vertical list of ShotCards, each a two-column card (shotCardGridSx: minmax(220px, 300px) minmax(0, 1fr)). The mockup replaces that with a dense grid the user approved. Keep all existing behavior (store wiring, Direct/re-direct dialog, generate hooks, undo/redo, preview, zip, script link); this phase is presentation. A1 — Compact board toolbar. Collapse the settings Panel into one toolbar row at the top of the surface: board title (18px, weight 600), a 13px text.secondary meta line (“8 shots · night-minimal style · entity: Aurora lamp” — shot count, style field, entity names), then right-aligned actions: Render stills, Render clips (outlined) and Assemble timeline (contained primary). Move the full Screenplay/Direction form (title, brief, style, entities, the three model pickers, aspect ratio, shot count, Direct) into a collapsible section or popover opened from the toolbar — it must stay reachable, but not permanently occupy the top of the board. Undo/redo, Preview, Download ZIP, and ScriptLinkControl move into the same toolbar row or its overflow. Files: StoryboardBoard.tsx. A2 — Shot grid. Replace the vertical ShotCard list with a responsive CSS grid: repeat(4, minmax(0, 1fr)) at desktop width, collapsing to 3/2/1 columns. Each card (restyled ShotCard.tsx): thumbnail on top (fixed-height media area, background.paper card, 1px divider border, BORDER_RADIUS.lg), action text below (13px, 1.45 line-height, 2-line clamp), overlays on the thumbnail: top-left SH NN · Ns mono label on a scrim, bottom-right status pill. The per-shot detail editing that no longer fits the card (prompt, render mode, takes gallery, per-shot model) moves to a selected-shot inspector or expand-in-place — decide against ShotCard.tsx’s current feature list and keep every existing control reachable. Files: ShotCard.tsx, StoryboardBoard.tsx, tests in web/src/components/storyboard/__tests__/. A3 — Status vocabulary. Pills are JetBrains Mono 11px, pill radius: done = success.main #50FA7B on rgba(80,250,123,0.12) with 0.5-alpha border (“clip · 38s”); rendering = video-violet #9460FF border on rgba(148,96,255,0.16) plus a 3px progress bar along the thumbnail’s bottom edge, and the whole card takes a 1px #9460FF border (mid-render border, per the app’s visual language); still-only/queued = neutral rgba(255,255,255,0.06)/0.15. Selected card: 1px primary.main border + 1px spread ring, as in the mockup’s SH 05. A4 — Selection footer. A 44px bar docked under the grid when a shot is selected: SH NN selected, then “Appears in” chips (see E1 for where the links come from; in this phase render the chips only for what the board already knows — the linked script line via ScriptLinkControl data), right-aligned Revise take and Re-render clip actions bound to the existing generate hooks. Until E1 lands the timeline chip is omitted, not faked. Acceptance: board renders 8+ shots as a grid matching the mockup’s geometry; all pre-existing storyboard tests pass (updated where they assert the old layout); no raw MUI imports; npm run test:affected green. Phase 1 — Projects data layer No mockup; this is the substrate. Backend only. P1 — projects table + model. packages/models: id, user_id, name, kind (free text: “spot”, “trailer”, “report”, …), created_at, updated_at. Follow an existing small model (e.g. packages/models/src/script.ts) for shape, migrations in both sqlite and pg schema paths in packages/models/src/db.ts / schema-pg. P2 — tRPC router. projects.list / get / create / update / delete plus projects.documents(id) — one query per document table filtered on project_id, returned as a typed union {type, ref, name, updated_at} matching WorkspaceTabType. Wire where the other routers live (packages/websocket). P3 — Status + spend rollup. projects.get returns a derived status line and spend. Status from document state: storyboard (shot count, stills n/m, clips n/m), script (lines voiced/stale via needsVoicing from @nodetool-ai/timeline), timeline (clip count, rendered or not). Spend from the prediction/cost ledger (nodetool costs reads it; see attachRunCostLedger in @nodetool-ai/execution) — requires cost rows to carry a project attribution: add project_id to the prediction record where it is written, nullable, backfilled as null. Per-document and per-category (stills/clips/voice/pipeline) sums, as shown in mockups/project-overview.html’s spend bar. P4 — Creation sites take a project. Replace the hardcoded projectId: "default" at every web creation site (grep projectId: "default" under web/src) with the active project id from the store added in B1, falling back to "default" when no project is active. "default" remains the loose-documents bucket; no migration of existing rows. Phase 2 — Shell: project group, rail entry, projects list References: mockups/storyboard-board.html and mockups/project-overview.html (tab bar anatomy), mockups/projects-list.html (list surface). B1 — Active-project state. web/src/stores/WorkspaceTabsStore.ts: tabs gain an optional projectId; store gains activeProjectId plus open/close-project actions. Opening a project opens its overview tab and restores its document tabs as one contiguous group. B2 — Tab bar group rendering. WorkspaceTabBar.tsx: tabs sharing the active projectId render contiguously behind a scope chip — ◆ <project name> ▾ on rgba(103,232,249,0.06) with inset 0 -2px 0 #67E8F9; grouped tabs carry inset 0 -2px 0 rgba(103,232,249,0.35). The chip’s menu: switch project, close group, open overview. Loose tabs render exactly as today. Active tab keeps its existing background treatment. B3 — Projects rail entry + list tab. PanelLeft.tsx: a Projects section at the top (diamond icon, cyan when a project surface is active). It opens a project-list tab rendering mockups/projects-list.html’s center: header row (title, search, + New project), 3-column card grid — thumbnail montage from real shot stills (via ResponsiveImage + locators), name row with ◆ and relative time, status line and spend from P3, status pill styles as in A3 — then a “Not in a project” strip of loose documents (chips, drag onto a card sets project_id). Ghost card opens the D1 surface. B4 — Tab type registry. WorkspaceTabsStore/TabContent/OpenMenu glyph+color tables gain the new types (project-list, project, project-new): glyph ◆, color #67E8F9. Update SUPPORTS_BOTH_MODES, TYPE_GLYPH, TYPE_COLOR in WorkspaceTabBar.tsx. Phase 3 — Project overview tab Reference: mockups/project-overview.html. C1 — Layout. New project tab type, first tab of its group. Header strip: name + status pill, derived status line, spend block (“$4.12 / so far · provider rates, no markup”), primary next-step action (label derived from status: “Render master”, “Render clips”, …). Below: 460px left column + fluid right column, split by divider. C2 — Left: the project agent. The chat thread that built the project (chat document bound to the project), rendered compactly: user turns as bubbles, agent turns as text + tool-call chips (mono 11px on Paper.overlay), status pills for renders it kicked off. Input box at the bottom feeds the same thread. Reuse the existing chat components where they fit; this is a narrow rendering of an existing thread, not a new chat system. C3 — Right: document cards. One card per project document: type-specific thumbnail (storyboard = 4-still strip, script = first lines with voiced/stale dots, timeline = track miniature, workflow = node sketch), name, status pill, meta line with per-document spend. Click opens the document tab in the group. Card geometry per the mockup: 2-column grid, 120px media area, 8px radius. C4 — Spend bar. Bottom of the right column: stacked bar split by category with a mono legend, totals from P3. Categories and colors: stills #E838FF, clips #9460FF, voice #08B8FF, pipeline #6690d4. Unpriced spend shows as an unpriced segment, not dropped (ledger convention). Phase 4 — New-project surface Reference: mockups/new-project.html. Full center-view surface, itself a tab (project-new), opened from the project selector, the projects list ghost card, and the rail. D1 — Layout. Centered 860px column: “What do you want to make?” (22px), one-line promise (13px secondary), prompt card (background.paper, 1px primary.main border, 12px radius) with context chips (ref images, voice, entities), cost estimate line (“est. $3–6 · provider rates, no markup” — from the shape preset’s historical range, or omitted when unknown; never a made-up number), Start primary button. Shape shortcut chips below; the selected shape renders its document chain (Board → Script → Cut → Render pipeline) as a glyph row. Foot of the view: “Blank document” strip — the full existing OpenMenu catalog as a 6-column grid, opening loose tabs exactly as today. D2 — Start wiring. Start creates the project row (P1), opens its group with the overview tab (C1), and posts the prompt as the first turn of the project’s agent thread. The agent builds documents through the existing headless tools (create_storyboard, voice_script_lines, assemble_storyboard_timeline, …) with the project id applied via P4. D3 — Start a project. The project selector on the tab bar owns “Start a project…”, which opens the D1 tab. + New keeps its document catalog. ProjectSelector.tsx, OpenMenu.tsx. Phase 5 — Cross-document context Reference: mockups/storyboard-board.html, selection footer. E1 — “Appears in” links. Given a selected shot, resolve where it lands in sibling documents: timeline placement via buildStoryboardTimeline’s shot→clip mapping (@nodetool-ai/timeline) against the project’s assembled sequence, script line via the existing script↔storyboard link (get_storyboard link state). Render as chips (violet ▤ timeline at timecode, sky 🎙 script line); clicking opens the sibling tab and selects the clip/line there. Requires a “select this clip/line on open” param on the timeline and script tab-open paths. E2 — Reverse direction. Timeline clip selection shows a “from Board · SH NN” chip that jumps back. Same mechanism, other direction. Out of scope Mobile (documents open one-per-screen there; project grouping needs its own design), Electron-specific chrome, migrating existing "default" rows into projects, and any change to the fifteen editors beyond Phase 0’s storyboard restyle and E1/E2’s selection deep-links. Verification per phase npm run test:affected, npm run typecheck, npm run lint. Phase 0 also: existing storyboard suites updated, nodetool harness gate --base main if the diff touches surfaces the registry maps. Phases 1–2: model tests beside the new table, router tests, WorkspaceTabsStore unit tests for grouping. UI phases: RTL tests for the new surfaces (behavior, not layout snapshots).--- # Do we need more provider task types? Source: https://docs.nodetool.ai/plans/provider-task-types-research.html Do we need more provider task types? Status: Research, for decision Question: Does ProviderCapability need new entries, and which? Related: packages/runtime/src/providers/base-provider.ts, packages/runtime/src/providers/manifest-models.ts, packages/runtime/src/context.ts, packages/execution/src/cost-ledger.ts 1. Answer Yes, for three: text_to_sound_effects, audio_to_audio, video_to_audio. Everything else on the candidate list is either already covered by an existing capability, reachable through a per-endpoint node, or too thin on provider support to pay for the ten surfaces a capability touches. The gap is not “the product cannot call these endpoints”. Every FAL, Replicate and KIE manifest entry becomes a node class at runtime (packages/fal-nodes/src/fal-factory.ts), so all of them are usable on a canvas. The gap is that they are invisible to the generic capability layer: model pickers, the chat composer’s media modes, ui_search_models, and every flow that takes a provider:model_id pair instead of a node type. 2. What exists today ProviderCapability (base-provider.ts:191) has 22 entries. A provider advertises one by overriding the matching BaseProvider method; providerCapabilities() reads the prototype to derive the set. Provider coverage of the media capabilities, counted by which methods each *-provider.ts overrides: Capability Providers implementing text_to_image / image_to_image aki, atlascloud, evolink, fal, gemini, huggingface, kie, minimax, nodetool, openai, python, replicate, reve, together, xai (+codex, openrouter text-only) text_to_video / image_to_video atlascloud, evolink, fal, gemini, huggingface, kie, minimax, nodetool, openai, python, replicate, together, xai reference_to_video atlascloud, fal, kie, minimax, nodetool, python, replicate video_to_video fal, replicate lip_sync fal, replicate inpainting fal, kie, openai, replicate upscale_image fal, replicate, topaz remove_background / relight_image fal, replicate segment_image fal vectorize_image fal, replicate text_to_speech elevenlabs, gemini, huggingface, minimax, openai, python, together text_to_music fal, kie, minimax, python, replicate automatic_speech_recognition gemini, huggingface, minimax, openai, python generate_embedding cohere, gemini, huggingface, jina, minimax, mistral, node-llama-cpp, ollama, openai, python, voyage text_to_3d / image_to_3d meshy, rodin 3. Findings F1 — 50 of 127 FAL audio endpoints are in no model list, on purpose. isAudioTransformNode (manifest-models.ts:1159) drops 43 endpoints matching audio-to-audio, speech-to-speech, video-to-audio, isolation, voice-changer, inpaint/outpaint, extend-audio, super-resolution. isSoundEffectNode drops the rest; 7 of those are pure text→SFX generators (fal-ai/elevenlabs/sound-effects/v2, cassetteai/sound-effects-generator, mirelo-ai/sfx1.6/text-to-audio, sonilo/v1.1/text-to-sound-effects, beatoven/sound-effect-generation, two stable-audio-3 SFX checkpoints). isTTSNode and isMusicNode both return false for them, so they appear in neither the TTS nor the music picker. The exclusions are correct given the capability set — a voice changer offered as a text-to-music model fails at call time with nothing to work on. They are the symptom, not the bug. F2 — SFX already costs the product a workaround. The game flow takes image_model and music_model as provider:model_id, but takes sfx_node_type as a raw registry node type (packages/agents/src/capabilities/workflows.specs.ts:751), because there is no SFX model list to pick from. That asymmetry is the missing capability showing through the API. F3 — audio_to_video is half-plumbed. inferVideoTasks already emits the string (manifest-models.ts:298), MediaGenerationMode and MediaMode both list it, and MediaModeMenu ships the entry with enabled: false. capabilityForMode("audio_to_video") returns null, asserted by web/src/components/chat/composer/__tests__/useModeProviderSetup.test.tsx:48. 23 FAL endpoints classify into it and can never be selected. F4 — video upscalers ride video_to_video. inferVideoTasks routes upscal/super-resolution/vsr/enhancer into video_to_video with a comment saying why (keeping them out of the generation pickers). That puts 11 FAL and 5 Replicate upscalers next to restyle models in one picker, offering a prompt and strength the upscalers ignore. Topaz’s 2 video models (packages/topaz-nodes/src/topaz-manifest.json) are unreachable at all — topaz-provider.ts implements upscaleImage only. F5 — rerank is advertised in settings and implemented nowhere. The Cohere and Jina setting descriptions (setting-catalog.ts:382,387) both name reranking, hf-models.ts:341 has a reranker model type, and api-types.ts:1134 groups Voyage and Jina under “Embeddings / reranking”. No provider method, no capability, no node. Three providers already configured (cohere, jina, voyage) ship a rerank API. F6 — two unions share one name. ProviderCapability is declared twice: base-provider.ts:191 (what a provider implements, 22 entries) and context.ts:292 (what a generation record can carry, 24 — adding render_model3d and bake_model3d_clip, both local Blender work, not a provider call). The split is intentional; the shared name is not, and nothing keeps the 22 in sync. F7 — capability names and model-task names diverge for four tasks. The capability is upscale_image / relight_image / segment_image / vectorize_image; the model task inferImageTasks emits and useModelsByProvider.ts:227 filters on is upscale / relight / segment / vectorize. Every new task type has to pick a side, and today the choice is made per surface. F8 — a new capability touches ten places. base-provider.ts (union, providerCapabilities(), method stub), context.ts (union, dispatchCapability case, generationMime), manifest-models.ts (classifier + model loader), cost-ledger.ts (UNIT_BILLED_CAPABILITIES), a generic node in {image,video,audio}-nodes, protocol/toolSchemas.ts (MODEL_SEARCH_KINDS), web/hooks/useModelsByProvider.ts, web/components/properties/ModelProperty.tsx, web/core/chat/mediaPrediction.ts, plus the composer mode map when it gets a chat surface. None of it is hard; all of it is the real cost. 4. Recommendation R1 — add text_to_sound_effects. 7 dedicated FAL generators plus ElevenLabs’ own SFX API, and F2 shows the product already paying for its absence. Same shape as text_to_music (prompt + duration → encoded audio), so it reuses runEncodedGeneration and the textToMusic param bag. Lets the game flow take sfx_model like it takes music_model, and gives the timeline a foley source. R2 — add audio_to_audio. 43 FAL endpoints, ElevenLabs voice changer and audio isolation, Replicate’s separator and enhancer. It is one method taking audio bytes plus a prompt, and it unblocks voice conversion, denoise, stem separation and audio super-resolution as one family rather than four. Fold speech_to_speech (2 endpoints) into it — the call shape is identical and a separate capability buys nothing. R3 — add video_to_audio. 6 FAL endpoints (kling-video/video-to-audio, two mirelo-ai/sfx versions, sam-audio/visual-separate). Small, but it is the one capability the storyboard→timeline pipeline cannot express at all: score a rendered shot from the picture. Ship it after R1 — it produces the same kind of output and can share the SFX node’s surface. R4 — do not add audio_to_video yet; finish or delete the stub. F3 says the plumbing is half built and the UI entry is disabled. Either wire capabilityForMode and a provider method, or drop the mode and the inferVideoTasks branch so 23 endpoints stop classifying into a dead string. Leaving it as is costs a test that asserts the null. R5 — split video upscaling out of video_to_video. F4 is a picker bug, not a missing capability: the same upscale_image capability could take video bytes, or a new upscale_video could. Either way Topaz’s video models need topaz-provider to implement it. Lower priority than R1-R3 — these models are reachable today, just badly presented. R6 — treat rerank as its own decision. F5 is a retrieval capability, not a media one, and its natural consumer is packages/vectorstore RAG, not a model picker. Worth doing, worth scoping separately from this list. R7 — do not add 3d_to_3d (13 FAL + Meshy retexture/remesh; the model3d editor covers the in-house path and no flow asks for it), vision / image_to_json / video_to_text (38+3+3 endpoints, all served by generate_message with image or video content), or training (70 FAL LoRA endpoints, a different product). R8 — before any of the above, rename one of the two ProviderCapability unions (F6) and pick one vocabulary for task names (F7). Both are cheap now and get more expensive with every capability added. 5. How the numbers were measured Endpoint counts come from the checked-in manifests, tallied by moduleName: packages/fal-nodes/src/fal-manifest.json 1600 entries packages/replicate-nodes/src/replicate-manifest.json 662 packages/kie-nodes/src/kie-manifest.json 163 packages/topaz-nodes/src/topaz-manifest.json 11 FAL modules with no matching capability: audio_to_audio 42, audio_to_video 23, 3d_to_3d 13, video_to_audio 6, speech_to_speech 2, plus vision 38 and training 70 (both out of scope per R7). The audio exclusion counts in F1 were produced by running the AUDIO_TRANSFORM_KEYWORDS and SOUND_EFFECT_KEYWORDS lists from manifest-models.ts against every manifest entry with outputType: "audio". The provider table in §2 was produced by grepping each *-provider.ts for overrides of the 19 optional BaseProvider methods — the same signal providerCapabilities() reads at runtime.--- # Reference-to-Video Provider Contract — Technical Design Source: https://docs.nodetool.ai/plans/reference-to-video-provider-design.html Reference-to-Video Provider Contract — Technical Design Status: Draft for review Related: providers.md, models-and-providers.md, media-generation-tracking-design.md 1. Summary NodeTool needs a separate reference_to_video capability for video models that use images and clips as identity, style, scene, motion, or audio references. Those inputs do not define the first output frame. The current image_to_video contract cannot represent that distinction. Its public method accepts an image list, but its documented meaning is “animate the first image.” Some providers send only images[0]; others send every image to a list field. Manifest discovery also classifies endpoints named reference-to-video as image_to_video. The same workflow therefore changes meaning when its provider changes. This design adds: a provider capability and method named reference_to_video / referenceToVideo; role-named reference image and reference video inputs; a generic nodetool.video.ReferenceToVideo node; task discovery that keeps reference models out of the image-to-video picker; direct-provider mappings for providers whose existing manifests describe reference inputs; and a Python bridge operation that stages multiple inputs for command-backed providers such as WanGP. image_to_video remains the start-frame operation. No provider falls back from one operation to the other. 2. Current behavior The current path is: ImageToVideo node -> ProcessingContext.runProviderPrediction(capability=image_to_video) -> BaseProvider.imageToVideo(images, params) -> direct provider OR PythonProvider direct provider: provider-specific image mapping PythonProvider: sends only images[0] -> provider.image_to_video MsgPack request -> nodetool-core stages one input-image file -> command adapter receives image_path -> encoded video bytes The contract diverges at three points: ImageToVideoParams says the first image drives the animation. FAL, KIE, and Replicate can forward several images when a manifest declares a list input. AtlasCloud and manifest task inference route reference-to-video endpoints through image_to_video, although those endpoints also accept reference videos and assign no start-frame role to their images. The Python path loses every image after the first. The WanGP adapter then maps that image to image_start with image_prompt_type = "S", even when the model metadata declares reference-image support. 3. Goals Give start frames and reference media different provider contracts. Preserve input order within each media kind. Support several reference images and several reference videos. Let model discovery expose only models that implement the requested task. Use one contract across direct TypeScript providers and Python workers. Stage remote-worker media without exposing host paths to the worker. Keep generation cancellation, progress, tracking, and chunked output working. 4. Non-goals Reference audio as a separate input. A reference video’s audio may be used through one boolean option, but standalone audio references need their own later contract. First/last-frame and keyframe authoring. Those are ordered frame constraints, not reference media. Video-to-video editing, extension, retake, or motion control. A new chat composer mode or agent media capability in the first change. The workflow node and ProcessingContext provider seam are the initial surfaces. Provider-specific controls that have no common meaning. Changing upstream WanGP code. 5. Operation semantics Capability Required media Media role text_to_video none Prompt creates the clip. image_to_video one start image The image defines the first output frame. reference_to_video at least one reference image or video Media guides identity, style, composition, motion, or soundtrack. video_to_video one source video The source clip is transformed. The operation name describes intent. A model may support more than one row, but each call enters through one method and gets one input interpretation. D1. Add a method instead of extending imageToVideo Adding referenceVideos to ImageToVideoParams would leave the image role ambiguous. Providers that use only the first image would keep producing different results from schema-driven providers. A separate method makes an unsupported mapping fail before spending money. The same change makes the provider-level imageToVideo input singular: imageToVideo( image: Uint8Array, params: ImageToVideoParams ): Promise<Uint8Array>; This removes the unused multi-image interpretation from the start-frame contract. ImageToVideoNode keeps its existing serialized list[image] property so saved workflows still load, but resolves and sends only the first non-empty image. Reference workflows move to ReferenceToVideo. D2. Use a typed input object The provider method receives media in an object so the two ordered lists cannot be swapped accidentally: export interface ReferenceToVideoInputs { images: Uint8Array[]; videos: Uint8Array[]; } export interface ReferenceToVideoParams extends TextToVideoParams { /** Let a provider use audio tracks carried by reference videos. */ useReferenceVideoAudio?: boolean | null; } referenceToVideo( inputs: ReferenceToVideoInputs, params: ReferenceToVideoParams ): Promise<Uint8Array>; prompt stays required at the provider contract. Providers whose API permits an empty prompt may receive "", matching textToVideo behavior. ProcessingContext filters empty buffers and rejects the call when both resulting lists are empty. It does not infer reference media from entities; entity descriptors may still join the prompt through the existing path. D3. Keep provider limits out of the shared type Providers differ on image count, video count, file size, duration, format, and whether video audio can be used. Each provider validates its model’s declared limits before upload or submission. Model metadata may expose limits later, but the common method will not encode one provider’s limits as global rules. 6. Runtime contract 6.1 Capability and model metadata Add reference_to_video to: ProviderCapability in runtime and ProcessingContext; protocol task unions used by model search and prediction records; VideoModel.supportedTasks values produced by provider discovery; model filtering for video_model properties; and the task vocabulary accepted by model-search tools. providerCapabilities(instance) advertises reference_to_video only when the provider overrides BaseProvider.referenceToVideo. A provider having video models is not enough. 6.2 ProcessingContext dispatch The provider prediction request uses snake_case with binary arrays: { provider: "wangp", capability: "reference_to_video", model: "minimax_h3_ref2va_pruned_pdd", params: { reference_images: [imageBytes1, imageBytes2], reference_videos: [videoBytes1], prompt: "Keep the person from image 1 and the motion from video 1", use_reference_video_audio: true, duration_seconds: 5, aspect_ratio: "16:9", resolution: "768p" } } ProcessingContext.dispatchCapability converts this to: provider.referenceToVideo( { images: coerceByteList(params.reference_images), videos: coerceByteList(params.reference_videos) }, { model, prompt, useReferenceVideoAudio, negativePrompt, durationSeconds, aspectRatio, resolution, timeoutSeconds, signal } ); coerceByteList accepts only non-empty Uint8Array values. Unlike the current image helper, it has no singular compatibility field because this is a new operation. 6.3 Generic workflow node Add nodetool.video.ReferenceToVideo with these properties: Property Type Notes reference_images list[image] Ordered identity, style, subject, or scene references. reference_videos list[video] Ordered motion, scene, edit, or soundtrack references. model video_model Filtered by reference_to_video. prompt str Uses “image 1” and “video 1” numbering in list order. use_reference_video_audio bool Requests soundtrack conditioning where supported. negative_prompt str Passed only when the provider declares it. aspect_ratio str Existing video values. resolution str Existing video values. duration int Existing video duration values. timeout_seconds int Existing provider timeout behavior. The node resolves every media ref with imageBytesAsync or videoBytesAsync, rejects an empty combined input, and returns a normal VideoRef. It uses the existing generation tracking and asset-save path. ImageToVideoNode keeps its serialized shape. Its description changes to say that only the first non-empty image is the start frame and directs reference-media workflows to ReferenceToVideo. 7. Model task discovery D4. Classify reference endpoints before image-conditioned endpoints inferVideoTasks checks these reference forms before the generic image-conditioned rules: reference-to-video reference to video referencetovideo ref-to-video r2v They map to reference_to_video, not image_to_video. For manifest entries without a descriptive id, video-output entries qualify when they declare one of: reference_images as an image list; reference_videos as a video list; or a mixed reference field such as refers, reached through manifest wrapInto metadata. Explicit supportedTasks remains authoritative. A model may explicitly list both image_to_video and reference_to_video when its API assigns semantics by input count. narrowTasksByRequiredInputs treats a required reference video as evidence for reference_to_video. It must not reclassify that endpoint as video_to_video, because the reference clip is not the transformed source. 8. Direct-provider mappings D5. Map from manifest roles, not field order Schema-driven providers need a reference-specific selector. The current primary-image selector intentionally skips fields containing reference, so it must not be reused. Provider Mapping FAL Upload each buffer through FAL storage. Map image URLs and video URLs to manifest fields whose semantic name contains reference, preserving list order and list shape. KIE Generalize the current image uploader to typed media uploads. Use manifest upload records for reference_images and reference_videos, including their API parameter names. Replicate Encode or upload each reference using the existing SDK input conventions. Select reference fields rather than the primary image/video field. AtlasCloud Convert resolved media to correctly typed data URLs, then map reference_images and reference_videos. When manifest fields share wrapInto, append ordered {url, type} records to that mixed array. MiniMax Advertise S2V-01 as reference_to_video. It accepts one reference image, no reference video, and maps it to subject_reference. Other MiniMax models keep their existing tasks. NodeTool delegation Forward the complete typed input object to the delegated provider. OpenAI, direct Gemini Veo, Together, Evolink, and xAI keep image_to_video only until their direct adapters implement a verified reference-media request shape. A provider endpoint available through FAL or KIE does not imply that the direct provider implements the same contract. Provider tests assert the complete outgoing request for at least one image-only, video-only, and mixed reference model. Tests also assert that a single-frame model is absent from the reference picker. 9. Python bridge protocol 9.1 TypeScript bridge Add this bridge method: providerReferenceToVideo( providerId: string, inputs: ReferenceToVideoInputs, params: Record<string, unknown>, secrets?: Record<string, string>, signal?: AbortSignal ): Promise<Uint8Array>; It sends one MsgPack request: type: provider.reference_to_video request_id: <uuid> data: provider: wangp reference_images: [bin, ...] reference_videos: [bin, ...] params: {...} blob_transfer: chunked-v1 blob_transfer still describes the result. The worker returns the video through the existing blob.start, blob.chunk, blob.end, and terminal result frames. Adapter progress frames keep using the request id. D6. Bound inline inputs before adding an upload protocol The current bridge already sends provider input media inline, and the worker’s maximum frame is 256 MiB. The first version keeps that transport and rejects a reference request whose combined input exceeds 192 MiB. This leaves room for MsgPack keys, parameters, and framing without relying on the hard limit. The error reports the measured size and limit before opening a provider job. A request-side chunk protocol is separate work because it needs staging state, expiry, cancellation, and reconnect behavior. It should serve every binary provider operation rather than exist only for this method. 9.2 Python worker staging nodetool-core handles provider.reference_to_video through the existing command-adapter path: Validate that the provider advertises reference_to_video. Validate the arrays and combined byte limit. Create one request-scoped temporary directory. Sniff each image/video suffix from its bytes. Write unique files such as reference-image-0.png and reference-video-0.mp4. Invoke the adapter with JSON containing reference_image_paths and reference_video_paths. Read the result path and return the encoded video through chunked output. Remove all staged inputs on success, failure, or cancellation. The adapter JSON contains paths, not media bytes: { "operation": "reference_to_video", "provider": "wangp", "params": { "model": "minimax_h3_ref2va_pruned_pdd", "prompt": "...", "useReferenceVideoAudio": true }, "reference_image_paths": ["/tmp/nodetool-provider-.../reference-image-0.png"], "reference_video_paths": ["/tmp/nodetool-provider-.../reference-video-0.mp4"] } The bridge protocol version increments, but its minimum compatible version does not. Older workers omit the capability from provider.list; the model picker therefore cannot select the operation. A forced call receives an unsupported operation error. 10. WanGP adapter mapping WanGP remains isolated in its own interpreter. Only the combined-image adapter imports WanGP. The adapter advertises reference_to_video only for metadata that declares a reference image or reference video input. It maps: NodeTool input WanGP setting reference_image_paths image_refs first reference video video_guide second reference video video_guide2 one reference video metadata-supported video_prompt_type containing VG two reference videos metadata-supported video_prompt_type containing V+G useReferenceVideoAudio = true metadata-supported audio prompt mode for guide-video audio The adapter reads choice values from WanGP model metadata. It does not hardcode a prompt-mode value that the selected model does not advertise. It validates the model’s counts before session.run_task and returns a clear error for an unsupported combination. image_to_video keeps mapping its one input to image_start and image_prompt_type = "S". No upstream WanGP files change. 11. Errors and validation Errors must identify the provider and model and occur before submission when NodeTool can detect them. Condition Result No non-empty reference input reference_to_video requires at least one reference image or video Provider does not override the method <provider> does not support referenceToVideo Model lacks the task Model is filtered from selection; a forced call fails before upload. Provider accepts images but not videos Error names the model and unsupported reference kind. Too many references or invalid format Provider validation reports the declared limit. Combined bridge input exceeds 192 MiB Bridge rejects before sending the frame. Worker disconnects or call is cancelled Existing abort and adapter termination paths run; staged files are removed. Adapter returns no file Existing provider-adapter result error is used. No prompt or media bytes are logged or attached to traces. Logs may include provider id, model id, media counts, total input bytes, duration, and request id. 12. File-level change map nodetool packages/runtime/src/providers/types.ts: input and parameter types. packages/runtime/src/providers/base-provider.ts: capability and base method. packages/runtime/src/context.ts: prediction union and dispatch. packages/runtime/src/providers/python-provider.ts: Python bridge mapping. packages/runtime/src/python-bridge-types.ts: bridge interface. packages/runtime/src/python-bridge-base.ts: wire request and size check. packages/runtime/src/swappable-python-bridge.ts: forwarding method. packages/runtime/src/providers/manifest-models.ts: reference task inference and reference-field helpers. Direct provider files for FAL, KIE, Replicate, AtlasCloud, MiniMax, and NodeTool delegation, plus the mechanical change to a singular imageToVideo input in every current provider. packages/protocol/src/api-types.ts and model-search schemas: task vocabulary. packages/video-nodes/src/nodes/video.ts: generic node. packages/video-nodes/src/index.ts and packages/base-nodes/src/index.ts: exports. web/src/components/properties/ModelProperty.tsx: node-to-task mapping. nodetool-core src/nodetool/worker/provider_handler.py: operation dispatch, byte checks, media suffix detection, staging, cleanup, and adapter payload. Worker protocol schema/version files: new request type and capability value. Provider handler and frame-contract tests. nodetool-wan2gp combined/provider_adapter.py: discovery and WanGP setting mapping. Dockerfile.combined: advertised adapter capability. Combined-adapter tests and contract documentation. 13. Test plan Runtime ProcessingContext preserves the order of both reference lists. ProcessingContext sends one start image to imageToVideo. Empty buffers are removed; two empty lists fail. PythonProvider forwards every reference instead of selecting index zero. Bridge tests assert the full MsgPack frame and chunked video result. The 192 MiB check fails without sending a frame. Cancellation reaches direct and Python providers. Model discovery Reference names infer only reference_to_video. First-frame, keyframe, and video-to-video names keep their existing tasks. Reference media fields infer the capability when the id is ambiguous. Explicit task metadata overrides inference. Provider-level capability discovery requires a method override. Direct providers FAL and KIE preserve image/video list order and use declared reference fields. Replicate maps list and singular fields without using a primary source slot. AtlasCloud builds separate arrays and a mixed refers array correctly. MiniMax S2V-01 maps one image to subject_reference and rejects videos. Unsupported models fail before the network call. Workflow node Asset, file, and inline references resolve to bytes. Mixed image/video input reaches runProviderPrediction unchanged. No-reference input fails with an actionable message. Output bytes become a typed VideoRef and retain auto-save behavior. Python worker and WanGP The worker stages several media files with unique names and detected suffixes. A fake adapter receives ordered path lists and emits progress plus a video. Cancellation terminates the adapter and removes the temporary directory. WanGP settings tests cover images only, one video, two videos, mixed media, guide-video audio on/off, and unsupported model combinations. The package boundary check continues to find no WanGP or mmgp imports under the public node package, tests, or scripts. 14. Rollout Add the additive worker operation and publish nodetool-core. Add the WanGP adapter mapping, pin the new core revision, and build the combined image. Add the NodeTool runtime contract, direct-provider mappings, model discovery, and workflow node. Test one image-only, one video-only, and one mixed WanGP request on the deployed worker. Test one schema-driven cloud provider with mixed references. Capability discovery is the rollout gate. An older worker or adapter does not advertise reference_to_video, so its models stay out of the new node’s picker. The text-to-video path does not change. Image-to-video keeps its wire operation and start-frame behavior, but every direct provider now receives one image. A saved ImageToVideo workflow that selected a reference-only model is not rewritten automatically because the new node has different input roles. Static validation reports that the model lacks image_to_video; the workflow author replaces the node and reconnects the references explicitly. 15. Acceptance criteria ReferenceToVideo can submit ordered image and video references to a direct provider and to WanGP through a remote Python worker. Model selection shows reference-capable models and excludes start-frame-only models. ImageToVideo still maps its first image to the provider’s start-frame field. No reference input is silently dropped or assigned a different role. Unsupported media kinds fail before provider submission. Remote generation reports progress, supports cancellation, and returns the video through chunked output. WanGP remains unmodified and isolated from the public NodeTool environment.--- # Retained game authoring work plan Source: https://docs.nodetool.ai/plans/retained-game-authoring.html Retained game authoring work plan Extend native game construction with retained authoring and safe rebuilds. Keep preparation, construction and simulation separate. The delivery proof is the second and third edit of a generated game, including preserved property edits and deleted-instance suppression. Scope and decisions D1. A versioned authoring model is the saved authority for generated content. Native runtime documents are validated build outputs. Retained source belongs to explicit procedural units. Source and semantic data cannot become competing authorities. D2. Ship stable authored identity from the first construction milestone. Separate definition, authored-instance and runtime-instance identity. Display names and insertion order cannot determine identity. Keep project resource IDs consistent with repository ID rules. D3. Preview creates a candidate without changing drafts or sessions. Apply checks the exact base revision and source/dependency digests atomically. D4. Initial procedural units have explicit inputs, literal preparation results and pinned asset bindings. Rebakes have no external capabilities. Automatic timeline-style capture is optional follow-up work. D5. Property overrides and deletion suppression are required for the delivery proof. Conservative entity conflicts are an intermediate milestone only. D6. Retained-project sessions pin definitions and asset dependencies. Adopt rebuilds on restart. General live state migration is deferred. The contract milestone must choose metadata placement in the existing saved document versus a versioned envelope, and document its compatibility behavior. Authoring versioning must remain independent of simulation engine versioning. Sub-agent schedule Use at most three workers plus the coordinator. Waves follow dependency gates. Wave Assignment Exclusive ownership Exit gate A1 Coordinator: freeze contracts Proposed protocol game-authoring.ts, protocol exports and integration, saved metadata decision Schemas, identity rules, override paths, limits, candidate/apply contract and legacy compatibility specified A2 Worker 1: retained builder packages/sandbox-packs/sandbox-game/sandbox/index.js, shipped pack skill, sandbox-game tests Explicit build units, declared inputs, stable keys, deterministic authoring seed and duplicate rejection A2 Worker 2: isolated bake Proposed packages/agents/src/game-code-bake.ts and tests Bounded candidate-only bake rejects external operations and reproduces validated native output A2 Worker 3: identity and reconciliation Proposed game-runtime authoring modules and tests Collision-safe identities and baseline/current/new reconciliation preserve manual entities and expose conflicts A3 Worker 1: prefab and parameter compilation Authoring prefab/parameter modules and tests Retained definition-instance relationships compile into existing 2D/3D representations A3 Worker 2: persistence and capabilities packages/models/src/game.ts, game capabilities/specs and related tests Preview is read-only, stale candidates fail, valid apply is atomic and persisted metadata round-trips A3 Worker 3: override edit operations document-ops.ts, document-ops3d.ts and related tests Property edits record overrides, deletes record suppression, reset/detach are explicit, replacement preserves ownership A4 Worker 1: editor integration Game editors, inspectors, tree and preview controls with adjacent tests Entity/dependency changes and conflicts are reviewable, inherited/generated/overridden fields are visible A4 Worker 2: session policy useGamePlaySession.ts, useGamePlaySession3D.ts and tests Active session retains definition/assets, restart uses rebuilt definition, runtime state never enters authoring A4 Worker 3: proof fixture and replay Proposed retained-game fixture/tests, existing CLI game replay integration if needed Second and third edits pass document assertions, fixed-input replay and captured-frame review A5 Coordinator: integration and verification Shared exports, registry wiring, documentation, final diff Required checks pass and fixture evidence is recorded The coordinator owns shared exports and registry changes. Workers must request shared-file changes through the coordinator. Freeze A1 before A2 implementation. A3 starts after the bake and reconciliation interfaces pass focused tests. Editor presentational work may use typed fixtures earlier, but integration waits for A3. Session tests and proof inputs may be prepared independently. Acceptance gates A6. Identical retained inputs and pinned resources reproduce identical output. Insertion, reorder and rename preserve IDs and references. Duplicate keys, ambiguous bindings and invalid references fail. Exercise large collections. A7. Deliberately forbidden network, generation, installation, secret and workspace calls fail during bake. Assert no writes or external calls occurred. Authoring and simulation RNG seeds remain separate. A8. Preview does not mutate draft, published revision or session. Concurrent editor changes and competing candidates cause stale apply rejection. Atomic failure leaves the saved draft intact. A9. Move one generated collectible and delete another. Change the layout and enemy parameters. Rebuild twice. Preserve the moved position, propagate other generated changes, retain suppression and hand-authored objects, and preserve references. Removing an overridden or referenced generated entity reports a conflict rather than transferring its identity. A10. Resetting an override restores its generated value. Detach converts generated ownership into manual ownership. Duplicates initially become detached copies. Suppression survives generator removal and later reappearance. A11. Legacy 2D and 3D documents remain readable without inferred ownership. Unsupported authoring versions fail clearly. Existing published revisions remain immutable. A12. Replay a fixed input route against the rebuilt definition with explicit win/state assertions and midpoint snapshot restoration. Review captured frames. Reuse the existing game replay harness rather than adding another simulator. After code changes, build backend packages, then run npm run test:affected, npm run typecheck, npm run lint, and npm run dev:nodetool -- harness gate --base origin/main. Read the native game harness reference before using the game harness. Instruction changes also require npm run check:agents-docs. Risks and deferred work R1. Existing 3D prefab operations clone and remap entities but do not retain authoring relationships. Runtime snapshot prefab data cannot substitute for authoring metadata. R2. The 2D play hook reads live asset bindings. Pinning retained-project assets requires an explicit change and regression tests. R3. Behavior source identity currently includes array position. Treat behavior arrays atomically until stable behavior IDs have compatibility handling. R4. Whole-document edits could discard ownership. Define their behavior before enabling retained saves and validate every edit path that touches generated data. After the delivery proof, schedule explicit behavior parameter/state bindings. Reusable presentation follows with transform ownership and cancellation rules. Defer arbitrary JavaScript-to-graph conversion, expression AST editing, nested prefab overrides and unrestricted hot reload. Inspected foundations Timeline retained capture and isolated bake. Game builder. 2D schema and 3D schema. 2D edit operations and 3D edit operations. Game persistence. Game CLI and harness registry. The resulting interface is documented in Retained game construction.--- # Timeline rendering performance measurements Source: https://docs.nodetool.ai/plans/timeline-rendering-performance-results.html Timeline rendering performance measurements These measurements use the generated fixtures in web/tests/benchmarks/. They describe specific paths on an Apple M4 Pro Mac. Browser, codec, resolution, cache state, and sample counts matter; the figures do not predict performance for other projects. Preview scheduling The 24 fps steady-playback fixture ran for five seconds in Chromium. Before decoded-frame scheduling, the cold run observed 115 hidden source-video frame callbacks with 515 source uploads and 519 compositor submissions. Afterward, the matching cold run observed 117 callbacks with 117 uploads and 122 submissions. The warm run changed from 119 callbacks, 515 uploads, and 518 submissions to 122 callbacks, 122 uploads, and 127 submissions. This measures redundant preview work. The callbacks do not measure preview-canvas presentations or cut latency; those presentation counters remain unavailable in the report. The final headed L3 matrix used fixture version timeline-preview-l3-v1. Each cold and warm run passed all 11 added named scenario oracles: one and four streams at 1080p and 4K, dissolve, animated text, matte, effect-heavy video, scrub, reverse remap, and motion blur. Its separate trimmed-cut oracle also passed in both runs against an independently decoded FFmpeg frame at the paused playhead; the preview matched 21 of 25 sampled pixels in the focused cut check. The effect-heavy scenario confirms rendering but does not isolate GPU effect cost, so it does not justify source-effect downscaling yet. A review reproduced stale motion-blur shutter textures: seeked could fire before the decoded-frame callback updated the upload generation. The shutter path now marks the sought frame ready before compositing and ignores late callbacks from the old frame. The focused headed motion-blur scenario passed after requiring every completed shutter seek to have a matching source upload and subsequent compositor submit. This remains a signal diagnostic; it does not compare each shutter sample’s output pixels to an independent reference. The prepared-cut check started before the 30-second lookahead, observed the incoming source prepared, and verified that the same element reached the cut without another load(). Headed runs matched the first visible incoming frame to the decoded source and its playhead-derived frame index. Boundary-to-matching-screenshot upper bounds were 157.9 and 190.6 ms in two runs; this is not a p95 cut-latency estimate. Headless alpha resolve The single-sample alpha path now resolves straight-alpha pixels on the GPU before readback. A resident premultiplied RGBA8 texture with alpha values 0, 64, 128, and 255 produced byte-identical output to the old CPU scan. Each path had one warmup and ten alternating timed samples per run. Timing includes GPU resolve where applicable, copy, map, and row copy, plus the old CPU scan. It excludes upload, scene composition, and encoding. Resolution Run CPU scan p50 / p95 GPU resolve p50 / p95 1920 × 1080 1 4.78 / 6.32 ms 1.24 / 3.08 ms 3840 × 2160 1 18.57 / 20.61 ms 4.10 / 5.20 ms 1920 × 1080 2 5.87 / 8.18 ms 2.26 / 4.89 ms 3840 × 2160 2 18.74 / 21.07 ms 4.15 / 5.14 ms An earlier 4K four-stream server fixture completed 24 frames in 3.11 s (7.7 fps), with sampled Node process RSS of 2.39 GB. That fixture used four identical full-frame clips, so it could not detect missing layers. The corrected fixture uses four distinct clips in visible quadrants and checks independent first, middle, and last frames. Its 24-frame runs took 408 ms for one 1080p stream, 903 ms for four 1080p streams, 1215 ms for one 4K stream, and 2642 ms for four 4K streams. An alpha ZIP case took 910 ms and passed RGBA pixel checks. A deliberately missing-layer renderer failed the corrected oracle. The corrected timings establish a new baseline and are not directly comparable with the earlier fixture. FFmpeg child RSS was unavailable. Two-frame server pipeline decision The proposed two-frame headless pipeline did not improve the representative export with real codecs. In isolated comparisons on the same machine, serial then pipeline took 3.53 s and 3.61 s; reversing run order gave 3.07 s and 3.19 s. The pipeline was removed. Encoder drain/error settlement and serial cancellation fixes were retained and covered by abort tests. Browser export decoding The headed Chromium 1080p H.264 fixture exported 48 frames in ABBA order. Seek decoding took 2305 / 791 ms; sequential timestamped decoding took 398 / 355 ms. All 48 sequential MP4 output frames were nearest their source oracle at the same time. The seek path was nearest the previous frame for 32 of 48 frames. Headless Chromium produced visually blank MP4 output for both decoder paths, so the visual claim uses the headed run. Native decoder memory was unavailable; peak JS heap was about 20.4 MB sequential and 15.7 MB seek. The fallback element decoder now waits for decoded data when the requested time is zero and the element has metadata but no current frame. A failing test reproduced the premature return. Export pool disposal also aborts concurrent pending seeks; its failing test had left two promises unsettled until timeout. The final browser PNG export produced 48 frames at 1920 × 1080 in 2.36 s. An independent FFmpeg reference matched the first, middle, and last output frames at checked pixels. Reverse server decoding The 320 × 180, 60-frame GOP-15 fixture changed from 59 FFmpeg reopens and 2833 ms to one reopen and 156 ms using bounded reverse windows on aligned source frames. The final figure includes the source-grid eligibility probe. Peak retained frame bytes were 13.6 MB. A 4K five-frame window retained 132,710,400 bytes under the 128 MiB configured cap. A second backward descent after a forward jump exposed a stale forward-only decoder cursor; its failing test was added, then the cursor fix passed. A review found that a reverse window starting between source frames could select the next frame at fractional timestamps. A lossless 30 fps fixture reproduced this at 1.69 s, where the cached path returned frame 52 and a fresh FFmpeg seek returned frame 51. Another fixture found wrong frames when a 24 fps source was sampled on a 30 fps timeline, plus near-boundary differences from FFmpeg seek rounding. The decoder now uses bounded windows only for verified regular, matching-rate source grids and frame-aligned requests. Other requests use fresh FFmpeg seeks; both independent review probes passed all 31 comparisons. The compositor keys uploads to decoded array identity so adjacent source times cannot collide through rounded version numbers. Fractional reverse requests consequently do not receive the one-reopen speedup. Preview bitmap cache Text and shape bitmaps now use byte budgets and remain pinned until the current frame consumes them. A headed preview fixture with two 4096 × 4096 shape layers held 134,217,728 bytes while rendering. The cache reported zero resident bytes after frame release. The 687 × 687 preview matched the top track’s red shape at the checked pixels. Browser export of the same fixture produced one 4096 × 4096 PNG in 1.53 s with the expected pixels; cache residency again returned to zero. Caption bitmaps use the same frame scope. A failing 65-caption test had shown the first bitmap closing before the frame consumed it; the scoped version keeps it alive until release. Rapid scrub baseline The headed 1080p fixture requested source times 417, 833, 417, 833, and 1250 ms through the preview’s keyboard controls. Each of three runs completed five seeks and uploads, and the final 5 × 5 pixel grid matched the last request. Last-request-to-matching-frame latency was 137.3, 127.9, and 148.9 ms (p50 137.3 ms, p95 148.9 ms). Requests were about 58 ms apart, which allowed each seek to complete; a denser request sequence is needed to measure the value of coalescing while decode is pending. With the seek coordinator, three matching runs again completed five seeks and passed the final pixel oracle. Latencies were 131.1, 128.7, and 128.9 ms (p50 128.9 ms, p95 131.1 ms). The sample is too small and the request spacing too wide to attribute the difference to coalescing. In a denser five-request sequence (10–15 ms spacing), native Chromium seeking completed one seek and passed all three final pixel oracles, with latencies 166.0, 175.4, and 153.3 ms. The added coordinator completed three seeks per run and also passed the pixel oracles; observed latencies ranged from 154.3 to 165.8 ms. The browser already coalesced more work than the coordinator, and the latency samples did not establish a gain. The coordinator was removed under the plan’s retention gate. Preview quality A headed 1080p visual fixture includes crop, placement, an ellipse mask, and a timed caption. Its Full preview measured 1219 × 685 backing pixels. Half measured 609 × 342 and matched the Full 5 × 5 pixel grid at all 25 points; Quarter measured 304 × 171 and matched 19 points. Paused Auto restored the 1219 × 685 backing and matched all 25 points. The Full reference also passed an independent decoded-source geometry check at 15 of 25 grid points. A broad 4K browser transition run timed out at 120 s. A smaller headed 4K probe confirmed the Auto backing changed from 1219 × 685 while paused to 609 × 342 while playing and returned to 1219 × 685 when paused again. The probe observed a 4K source upload, so the smaller backing does not imply source effects run at half resolution. Pending decision gates Source-effect downscaling remains deferred until an effect-heavy trace isolates its cost and a reduced-resolution implementation passes visual parity. A repeated cut-latency distribution is still needed before setting a latency target. After the final code changes, npm run test:affected, npm run typecheck, npm run lint, and npm run dev:nodetool -- harness gate --base origin/main all passed.--- # Timeline Rendering Performance Plan Source: https://docs.nodetool.ai/plans/timeline-rendering-performance.html Timeline Rendering Performance Plan Outcome and evidence Make ordinary timeline playback, cuts, scrubbing, and export measurably faster without changing rendered frames. The starting analysis inspected commit 6cb1a6d1d3c78e5bc354baa3d070fa43aa87a748. Its M4 Pro component measurements found 4K four-layer GPU compositing at 5.60 ms with resident textures and 25.46 ms with upload and readback. A two-frame headless experiment reduced one-layer 4K throughput time from 16.91 to 14.18 ms per frame when sources changed. These numbers exclude codec work, audio, browser presentation, and editor contention. They are not playback results or a claim of parity with Premiere or Final Cut. This plan targets four separate costs: browser frame scheduling, decoder readiness at cuts, headless readback, and preview memory. Keep preview, browser export, server export, and agent frame preview on the shared scene and compositor rules in packages/timeline/AGENTS.md. Preserve the transient playback clock, scene change horizon, source texture reuse, batched frame compositing, and existing multi-sample GPU alpha resolve. Delivery rules Start each task from the current branch and read AGENTS.md, the relevant area overlays, and docs/DEVELOPMENT_STANDARDS.md. This plan describes work, not an exception to those rules. Use one agent per task and one small PR per task. Agents may work concurrently only when the ownership table gives them disjoint files. Integrate in dependency order, rebase, and rerun checks on the integrated branch. Before fixing an observed bug, make the reproduction fail. Before enforcing a new metric check, prove it fails on invalid input. Use the canonical playback or export path to confirm a harness finding, as required by docs/HARNESS_FIRST.md. Every code PR runs npm run test:affected, npm run typecheck, npm run lint, and npm run dev:nodetool -- harness gate --base origin/main. Add the focused render parity or browser scenario named in its task. Attach before and after measurements from the same machine, fixture, browser, resolution, and cache state. Keep benchmark thresholds out of CI until the browser baseline is repeatable. Correctness gates can enter CI first. Document unsupported browser paths and preserve a working fallback. Measurement contract L0 produces a repeatable browser scenario using checked-in or locally generated media with known frame timestamps and colors: steady 24 fps playback and a 60-second lead clip with a trimmed incoming clip. L3 extends the same fixture to 1080p and 4K, one and four H.264 streams, a dissolve, animated text, a matte, scrubbing, reverse playback, motion blur, and browser and server export. Record cold and warm cache runs separately. Compare each change with the matching baseline on the same machine, source files, codec, and preview quality. A separate manual comparison with commercial editors can follow; it is not a delivery gate for this plan. The machine-readable report records source frame generation, uploads, composites, presented frames, intentionally held frames, dropped or duplicate source frames, cut-to-correct-frame and scrub-to-correct-frame latency, p50/p95/p99 presentation interval, main-thread time, GPU completion time where available, audio drift, queue depth, estimated allocated bytes, peak process memory, export frames per second, and cancellation time. Mark unavailable GPU or memory counters as unavailable, never zero. Count presentation separately from source decoding so a 24 fps clip on a 120 Hz display is not reported as dropping four frames out of five. The fixture checks pixel identity at known cut and matte frames and includes alpha output. A report must state its sample count and fail if no frames or no relevant counters were observed. Performance results remain diagnostic until repeated runs establish useful thresholds. Correctness and resource bounds are release gates now: no wrong or reordered frames, no unbounded queue, no lost cancellation, no alpha or audio drift regression. Tasks and ownership ID Owner Depends on Owned implementation files Deliverable L0 Sol none New fixture/report files under web/src/components/timeline/ or web/src/perf/, plus minimal diagnostic hooks in preview/gpu/compositor.ts Core baseline, failure reproductions, report contract L3 Sol L0 The L0 fixture and report files Extended scenario matrix and export baseline S1 Sol L0 web/src/components/timeline/preview/PreviewCompositor.tsx, preview/gpu/compositor.ts Decoded-frame driven preview scheduling and uploads S2 Sol L0, S1 preview/PreviewCompositor.tsx, preview/videoSlotPool.ts Live-clock lookahead and prepared decoder promotion S3 Sol none packages/timeline/src/render/frameCompositor.ts GPU straight-alpha conversion for the single-sample headless path S4 Sol S3, L3 frameCompositor.ts, packages/video-nodes/src/nodes/timeline/compositeRender.ts, rawFrames.ts Bounded two-frame render/readback and ordered encoding L1 Sol L3, S1, S2 preview/textRender.ts, preview/shapeRender.ts, preview/PreviewCompositor.tsx, render/TimelineRenderer.ts, cache tests Byte-bounded bitmap caches with frame-safe eviction L2 Sol S2, L1 preview/PreviewArea.tsx, preview/PreviewCompositor.tsx, web/src/stores/timeline/TimelineUIStore.ts Auto/Full/Half/Quarter control and correct backing resolution S5 Sol L3, L2 Shared compositor and preview source adaptation files, assigned after trace Reduced source processing cost at lower preview quality S6 Sol L3, S2, L1 Browser export and decode files, assigned after codec probe Bounded timestamped decode for sequential browser export S7 Sol L3, S6 Browser scrub request files, assigned after trace Coalesced scrub requests S8 Sol L3, S4 packages/video-nodes/src/nodes/timeline/rawFrames.ts Bounded server reverse GOP windows Sol agents own the fixture, cache, UI, scheduler, decoder, and GPU changes. Test files belong to the task that changes their behavior. The L0 diagnostic hook merges before S1 edits preview/gpu/compositor.ts. L1 waits for S2 because it touches PreviewCompositor.tsx; L2 waits for L1 for the same reason. S8 waits for S4 because both edit rawFrames.ts. S3 can run alongside L0 because its files do not overlap, but its performance claim waits for L3’s export baseline. A workable two-agent order is L0 with S3, L3 with S1, S2, L1 with S4, then L2 with S6. L0 — Browser baseline and reproductions Build a runnable scenario against the real preview, with a JSON report and a short command documented next to it. Use requestVideoFrameCallback metadata and getVideoPlaybackQuality where available. Add minimal source-upload and compositor-submission diagnostic hooks at the existing sites in preview/gpu/compositor.ts, behind a flag; S1 will retain these hooks when changing that file. Capture the current redundant-work behavior during steady 24 fps playback on a high-refresh display, then capture the 60-second cut and trimmed in-point. Keep the metric collector outside React state and disabled for ordinary playback. Acceptance: The command runs the same fixture twice and produces nonempty, schema-checked reports. Deliberately invalid input fails the check. The cut case shows whether the first displayed incoming frame matches its requested source in-point. Report browser/OS, GPU adapter if available, codec, media dimensions, display refresh, preview dimensions, and warm or cold cache status. Do not assert a speedup before S1 or S2 lands. L3 — Extended fixture and export baseline Extend L0 without changing its report schema. Add the four-stream, 4K, matte, animation, scrub, reverse, motion-blur, and export cases in separate named scenarios. Include a correct-frame oracle at the cut and an alpha reference for export. Record audio drift where the fixture has sound. Keep test media generation repeatable and report scenario-specific sample counts. Acceptance: Each scenario runs independently and fails if it observes no frames. The browser and server export baselines include total elapsed time, frame count, and peak memory when available. A changed fixture records its own version in the report so numbers from different media are not compared as one experiment. S1 — Decoded-frame scheduling Track a monotonically increasing decoded-frame generation per active preview HTMLVideoElement using requestVideoFrameCallback, with callback cleanup on slot reuse and unmount. Give the uploader a source-version interface for tracked preview elements. Browser export uses the same uploader with sought elements; keep a deterministic seek-based version path for those elements, including source replacement at the same timestamp, until S6 changes decode. Reuse a GPU texture when a new bitmap has the same dimensions and format, but upload its changed pixels. Composite when a video generation changes or when scene changes, animation, transitions, motion, caption timing, or the playhead requires it. Keep an explicit fallback for browsers without the callback. A seek requests redraw when the correct decoded frame arrives, not merely when currentTime changes. Acceptance: A held video frame causes no repeat upload or composite solely because rAF ticked. Animated overlays still composite on their own clock. Pause, seek, source replacement, loop, cut, matte, and callback-unavailable cases show a correct frame. Browser export still uploads each sought frame in sequential, retimed, matte, and motion-blur cases, including a replaced source at the same timestamp. L0 reports fewer uploads and submissions in steady video playback without more wrong, dropped, or duplicate source frames. Add focused tests near preview/gpu/__tests__/compositor.test.ts, timeline/__tests__/TimelineRenderer.test.tsx, and the preview tests. S2 — Live lookahead and decoder promotion Use getTimeMs() at each preload tick, not the frozen sceneTimeMs, to select upcoming sources. Keep a prepared (clipId, assetUrl) element through activation by moving its binding into the active pool without setting src or calling load() again. Seek a prepared element to the incoming source in-point as metadata allows. Include matte sources. Keep the active slot cap and define eviction when a dissolve occupies multiple slots. Cancel stale listeners and pending seeks when a slot changes ownership. Acceptance: A clip beginning at 60 seconds is prepared inside the configured lookahead while the lead clip plays. The prepared element is the one used after the cut; its source is not reloaded. The first correct frame, trimmed source in-point, matte, and overlapping dissolve are verified in a browser run. Compare p95 cut latency with L0 and report it even if decoder promotion brings no gain. Extend preview/__tests__/videoSlotPool.test.ts. S3 — Headless alpha resolve Reuse the existing GPU unpremultiply shader pattern from the multi-sample motion-blur resolve for the single-sample headless path. Keep the public renderFrame output as straight-alpha RGBA8 for both opaque and alpha export. Preserve zero-alpha RGB behavior and match current rounding closely enough for the repository’s pixel parity tolerance. Do not skip conversion solely because the output format is opaque: chroma key or another effect can lower alpha after the ground is seeded. Acceptance: The single-sample output matches reference frames for opaque, translucent, zero-alpha, chroma key, masks, and generated mattes. Motion blur remains unchanged. The per-pixel CPU scan leaves the hot path. Report standalone readback and end-to-end export timings, separately. Run packages/timeline/tests/render.parity.gpu.test.ts, packages/video-nodes/tests/timeline-alpha-render.test.ts, and output-format checks. S4 — Two-frame headless pipeline After S3, split render submission/readback completion from encoder write through a small ordered frame result interface. Allow at most two frames in flight and one ordered encoder consumer. In rawFrames.ts, make a pending encoder write() settle when FFmpeg fails or abort() destroys stdin, even if it was waiting for drain. Honor encoder backpressure; on abort or error, stop scheduling, release mapped buffers and sources, abort encoding, and surface the original failure. Do not decode two samples from one forward-only FFmpeg stream concurrently. Verify whether the shared compositor’s ping-pong textures are safe while a prior readback is pending before allowing a second submit. If they are not, add the required target ownership inside HeadlessFrameCompositor rather than exposing GPU buffers to the caller. Acceptance: Alternating-color frames prove order and data ownership at 1080p and 4K. Cancellation works during decode, GPU mapping, and blocked encoder writes. Tests force abort and FFmpeg failure while write() awaits drain, assert prompt settlement, and preserve the original error. Memory and queue depth stay bounded. Alpha, masks, changing scene content, and motion-blur output match the serial path. Measure total export throughput with real codec decode and encode as well as synthetic frame throughput; retain the pipeline only if the end-to-end result is non-regressing on the representative fixture. L1 and L2 — Preview cost controls L1: Replace entry-count-only limits for sequence-sized text and shape bitmaps with eviction by estimated bytes. Count both TextRasterizer.cache and TextRasterizer.activeBitmaps, plus shape bitmaps. Use an explicit frame scope at preview and browser-export call sites: pin every bitmap returned during source resolution until that frame has consumed or uploaded it, then release pins. Close only unpinned evictions. Define an oversized-live-frame policy: the live frame may temporarily exceed the cache budget, report those bytes, and evict after frame release. Measure resident cache bytes and texture allocations in the L3 scenarios. S1 owns same-size GPU texture reuse in preview/gpu/compositor.ts; L1 verifies that changed bitmap content still uploads after S1 merges. A later cropped-raster change needs explicit offsets and visual parity for crop, placement, and source-pixel effects. L2: Add Auto/Full/Half/Quarter preview quality using the existing UI primitives and design tokens. Size the preview backing surface to the selected scale and device pixel ratio; preserve sequence-space placement and restore a sharp paused frame where practical. Auto may choose a lower scale from measured pressure, but must have a deterministic initial rule and a visible current setting. This task does not claim to reduce 4K source effects until S5 changes source processing. Acceptance: L1 demonstrates a byte cap between frames, closes evicted bitmaps only after frame consumption, and does not serve a stale bitmap or texture on a same-size content change. A preview and browser-export fixture with one live frame larger than the budget must render every layer correctly, then return below the cap after release. L2 demonstrates correct crop, placement, captions, masks, and paused-frame sharpness at every setting. Browser export and server export are unaffected. L3 reports backing and source texture sizes separately. S5–S8 — Decode and source-cost track Start these only after L3 shows which cost dominates on the target machines. S5, source cost: At lower preview quality, downscale before expensive effects when their units and sampling semantics can be preserved. Compare with full-quality output at matched display size. If quality differs for an effect, keep its full-resolution path or label the preview approximation. Evaluate actual proxy files separately; a smaller canvas by itself is not a proxy. S6, browser export: Probe VideoDecoder support, decoder latency, queue behavior, and hardware acceleration on supported platforms. Build timestamped decoded-frame reuse and sequential decode behind the existing browser export interface. Keep a fallback to the current seek path for unsupported codecs or environments. Preserve exact frame times, trim, retime, matte, and motion-blur samples. Do not replace deterministic export with approximate playback frames. S7, scrub: Coalesce obsolete seeks during rapid browser scrubbing; a stale decode must never present over the latest request. S8, reverse: Decode bounded GOP windows with byte and duration limits instead of reopening FFmpeg for every descending server request. Preserve the current frame-at-time contract. Acceptance: Each task has a before and after latency distribution from L3, bounded memory and queues, frame-accurate fixtures, cancellation tests, and a working fallback. Keep a change only when the representative end-to-end scenario improves without a correctness regression. Integration order and decision gates Merge tasks in order within each track below. Preview and export tasks with disjoint files may interleave after their listed dependencies are satisfied. Baseline: Merge L0, then L3. Record present bottlenecks and set machine-specific performance targets from repeated runs. S3 may be developed concurrently and measured after L3. First preview release: Merge S1, then S2, then L1 and L2. Compare frame work, cut latency, and memory against L0. Release only when correctness fixtures and the four mandatory checks pass. First export release: Merge S3, then S4. Compare pixel parity, queue depth, cancellation, synthetic throughput, and real export throughput. S3 and S4 are separate rollback points. Second wave: Choose S5 through S8 from measured traces. Profile CPU effects and GPU pass cost again before considering premultiplied internal passes, effect-result caching, source-over fast paths, external textures, hardware codecs, workers, or background-rendered ranges. Each of those needs its own measurement and parity task. Do not make scene-resolution rewrites or generic React memoization part of this effort without a new trace: the inspected simple-scene resolution was already below the measured media and CPU costs. Do not promise broad Premiere or Final Cut parity without same-machine, same-media editor comparisons across supported codecs and effects.--- # Tutorial focus and animation plan Source: https://docs.nodetool.ai/plans/tutorial-focus-animation.html Tutorial focus and animation plan Make each tutorial direct attention to the component being explained. The camera should arrive before an action, hold while the viewer reads or watches the result, and widen when the relationship between components matters. This plan covers every tutorial registered in demo/src/tutorials.ts, chatTutorials.ts, timelineTutorials.ts, and docTutorials.ts, cross-checked against web/src/components/tutorials/tutorialsData.ts. It proposes implementation and editorial changes. No video changes have been implemented or visually verified by this plan. D1. Scope and starting point The graph composition already uses demo/src/camera.ts. It frames nodes from their position and width, assumes a 1920×1080 frame, moves for 750 ms after a step begins, and pulls back at the end. It does not frame individual controls or account for node height. The other tutorial compositions render their players without a camera. TutorialShell already owns title, step indicator, captions, and outro timing. Preserve that shared entry point. Cookbook and workflow-gallery compositions also use the graph tutorial, so preserve their existing focus behavior when introducing the new shot format. Their editorial redesign, the promo, and the landing-page hero are outside this tutorial rollout. Use the real production components and existing synthetic casts. Reframe and retime their demonstrated behavior. Add cast actions only where they are necessary to demonstrate what the tutorial already promises. D2. Shared motion rules Code Rule Starting treatment M1 Establish location Hold the full surface for about 1 second before the first close-up. M2 Arrive before the action Move for 500–750 ms, then settle for 200–300 ms before the relevant event. M3 Hold for comprehension Keep framing fixed during typing, streaming, and result inspection. Allow at least 2 seconds for a short result, longer for code or multiple lines. M4 Fit the subject Fit the complete target bounds inside the available picture area, with padding. Keep the label and immediate context visible. M5 Explain relationships Frame both endpoints for connections and both the control and affected region when their relationship is the lesson. M6 Limit attention cues Use one subtle outline or spotlight at a time. Fade it in after the camera settles and out before the next move. M7 Keep text stable Captions and step indicators stay in screen coordinates. They do not scale or pan with the surface. M8 Close on evidence Hold the completed image, answer, board, or test result before a short outro. Widen only if the wider view explains something. These are initial editing values, to be tuned on the pilot renders. Do not force every shot into the same duration. Avoid camera bounce, repeated pulses, continuous drifting, and zooming on every streamed token. Prefer smooth easing without overshoot. For a long move between distant panels, briefly widen or use a short cut instead of sliding across a magnified interface. Do not animate the camera and the editor’s own timeline zoom simultaneously. Use a reading-budget check when authoring captions, initially allowing roughly 3 words per second plus a short settling interval. Shorten or split text that cannot fit. Extend the replay when the lesson needs more time. Preserve the result hold instead of using it to absorb an overlong caption. D3. Shot data and timing Separate instructional steps from camera shots. One step can require several views, such as a connection’s source, both handles, and its output. Do not create extra numbered steps solely to move the camera. Each authored shot should declare: Field Purpose id Stable name for inspection and frame exports. fromMs, toMs Presentation-time interval, relative to the replay. target Discriminated target: overview, node, component, or group. moveMs Transition duration within the interval. padding, maxZoom Framing limits, with shared defaults. emphasis Optional outline or dimmed surroundings. actionAtMs Optional alignment point for a demonstrated action. Keep a separate replay-time mapping for holds and compressed waiting. Camera, captions, cursor cues, and step labels use presentation time. Cast playback and its media use mapped cast time. Both must be pure functions of the requested frame, including backward seeks. Begin with identity mapping. Add hold or speed segments only where a tutorial needs them. Do not shift assistant messages without their associated document patches. If timing affects a cast invariant, retime both tracks together and update its checks. Existing steps[].focus and zoom remain supported when a composition has no explicit shots. Migrate the tutorial entries to shots without requiring cookbook or gallery edits. D4. Resolve component bounds reliably Introduce stable focus identifiers through the demo surfaces. Prefer an existing semantic identifier, a component ref, or a narrow demo attribute over selectors based on text, styling classes, or DOM position. Targets should resolve to rectangles in the untransformed surface coordinate system. A group resolves to the union of its member rectangles. Graph targets also retain graph coordinates so the existing controlled viewport can frame them without applying a second zoom. Measure only after the frame’s replay state, fonts, and required media are ready. Capture bounds at an authored shot anchor, then hold them through the shot to avoid chasing streaming text or growing cards. If content outgrows the view, author a deliberate scroll or another shot. Scrolled and virtualized targets must be mounted and brought into view before measurement. The fixed-layout document, chat, and timeline surfaces should render at a known base size inside a clipped camera wrapper. Transform the wrapper without changing the layout size during zoom. Keep the caption and step layers outside it. Use composition dimensions and reserved overlay rectangles to calculate the available picture area. Clamp zoom when a small control would otherwise become excessively large. Include the control’s label in its focus target. Missing targets should produce a visible diagnostic in Studio and fail final-render validation with tutorial, shot, target, and time. Validate that the checker resolves actual targets and deliberately test a nonexistent one. D5. Focus overlays and action cues Render the highlight from the same transformed bounds as the camera. Use shared design tokens for its color, radius, and motion. A modest dimming layer should preserve the surrounding interface’s location cues. A cursor appears only for a represented pointer action. Its click pulse aligns with the state change. Assistant edits use a highlight on the changed component, without a cursor pretending to operate it manually. Several existing casts start with populated inputs or connected graphs. Do not add typing or dragging decoration over an unchanged state. Either add a deterministic replay of the advertised interaction through the real surface or revise that beat to explain the existing state. In particular, the connection tutorial needs an explicit decision in A3 about showing graph construction. Tutorial shot plans The sequences below describe the proposed edit in order. Durations are initial total-video targets, including a roughly 1–1.5 second introduction and 1.5–2 second closing card. They are not final event timestamps. During implementation, map each beat to the cast’s actual events and lengthen the film if the content cannot be read at the proposed pace. T1. Build your first workflow (first-workflow) Target length: 26–30 seconds. Start wide with the full input → LLM → image → preview path. Frame the Text Input’s value and label, then widen just enough to show its outgoing connection. Move to the LLM’s generated text before streaming begins and hold through a short readable passage. Frame the image-generation node’s status and preview area before generation completes. Finish on the completed image, then reveal the full connected workflow. Add an explicit final preview shot. The existing step list ends on generation. Keep the provider/model title visible when it helps identify which node is running. Acceptance: viewers can identify the starting input, intermediate text, and final image without following a moving camera while reading. T2. Connect & run (connect-run) Target length: 20–24 seconds. Show the source value close up. Frame the source output handle and Uppercase input together. Show the connection being established only after adding a real replay action for it. Briefly frame the run control if the player exposes the genuine control and can demonstrate it. Hold on the Uppercase node’s running-to-completed state, follow the second connection, and finish on HELLO NODETOOL in Preview. The existing cast starts with nodes and edges already present and replays execution updates. Add the minimum graph-edit playback needed for its advertised construction lesson, or explicitly narrow the captions to tracing and running an existing graph. Do not imply that an edge activation is a drag interaction. Acceptance: source handle, destination handle, execution, and changed output are individually legible. T3. Generate a list (list-generator) Target length: 22–26 seconds. Frame the topic field. Show its connection to the list-generating node, then settle on the list as its first items arrive. Keep the camera stationary while items accumulate. Move once to Preview when the list is complete, using enough vertical space to show the output is a collection. End wide enough to show that Preview is downstream. Choose a concise list that fits the close-up. If the cast’s list overflows, add one deliberate scroll after a reading hold. Acceptance: the viewer can distinguish the input topic, individual generated items, and downstream list output. T4. Ask the AI (ask-ai) Target length: 20–24 seconds. Frame the question field, then show the question-to-LLM connection. Settle on the answer region before text starts appearing. Hold until a useful sentence is readable. Move to the completed Preview and finish with the LLM and Preview together. Keep provider controls secondary because this lesson is about data flow. Acceptance: the completed answer stays readable before the closing card and the output’s location is clear. T5. Combine two inputs (combine-inputs) Target length: 22–26 seconds. Open on both source nodes. Highlight the name and topic in sequence while keeping their framing stable. Show both connections entering the template node. Zoom into the template placeholders with their surrounding sentence. Move to the merged result and briefly highlight the substituted text. Close on the branch-and-merge layout. Replace the first shot’s single-node focus with a group target for both inputs. Acceptance: viewers can map each source value to its corresponding place in the finished sentence. T6. Summarize a document (summarize-text) Target length: 22–26 seconds. Frame the source field with enough visible text to establish that it is a long passage. Do not make the viewer read the whole source. Move to the summarizer’s output region before streaming starts. Hold on the key points, then show the complete summary in Preview. End with source and summary visible together if both remain readable. Avoid panning down the source merely to show volume. Acceptance: the final summary receives the longest reading hold and visibly differs in length from the input. T7. Describe an image (describe-image) Target length: 22–26 seconds. Open on the source image, framed without cropping its subject. Widen to show its connection into the Agent node. Move to the description area as text arrives. Finish with image and completed description together when the layout permits, otherwise hold each in consecutive shots. Choose the description shot’s bounds from the text region, rather than applying the image node’s zoom to every node. Acceptance: viewers can compare the visible image subject with the words produced about it. T8. Ask the chat agent (chat-agent-qa) Target length: 22–26 seconds. Establish the whole chat panel. Frame the submitted question. Move to the web-search tool card before it changes state, showing the tool name and status. Hold through completion, then move to the answer and its source information where the cast provides it. Close with the question, completed tool call, and answer in context. Extend the first caption’s reading interval. If the composer interaction is not represented, describe the submitted question rather than showing a fake typing sequence. Scroll at shot boundaries so tool cards and messages do not slide underneath a fixed close-up. Acceptance: the viewer sees what the tool did and where the answer appears. T9. Cut a scene together (timeline-trim-arrange) Target length: 30–36 seconds. Establish preview, tracks, and inspector. Frame the opening clip as it is selected. Move to the inspector briefly to connect selection to settings. Return to the clip’s right edge before its duration changes from eight to five seconds. Hold on the resulting gap and then the second clip landing at the five-second boundary. Frame the caption track as the caption is added. Widen and settle before the editor’s own timeline zoom. End with playback showing the preview and relevant tracks together. Give trimming its own beat, since it is a concrete cast event at 2200 ms that the existing steps do not isolate. A moving trim cursor requires a deterministic interaction track aligned to the actual duration change. Preserve the full playback interval. Acceptance: the trim, adjacency of clips, caption placement, and played result are all visible. T10. Edit a sketch by asking (sketch-assistant) Target length: 26–30 seconds. Open on canvas and assistant together. Frame the vignette request, then the relevant tool call. Move to the layers panel before the new layer appears. Keep its name and selection visible. Frame blend mode and opacity as their values change. Return to the canvas for a held final image, then briefly restore the complete editor. Use shared framing for the before and after canvas views. Acceptance: viewers see that a new editable layer was added and understand which controls changed its appearance. T11. Write and voice a script (script-assistant) Target length: 30–34 seconds. Frame the request’s length, speaker, and tone requirements. Move to the cast/speaker area as speakers appear. Frame the script lines with their speaker assignments. Hold on one line as its draft changes to a voiced take. Move to a second completed take only if it adds information. End on the populated script with speaker and take state visible. Split the existing combined lines-and-voices beat. Show playback only if playable media is actually present and synchronized. Acceptance: speaker assignment, line text, and take availability can each be read. T12. Board a shot list (storyboard-assistant) Target length: 32–38 seconds. Frame the teaser brief, then widen to the board as planned shots arrive. Zoom into one shot’s direction and camera description. Hold its planned state before moving to generation. Show that same card changing from pending to a completed still. Widen to include subsequent cards as they complete. End on the finished board. Keep card positions stable across planned and ready states. Do not zoom individually into every still. Acceptance: the viewer sees direction written before rendering and can connect a planned shot to its generated image. T13. Write a JS script (jsscript-assistant) Target length: 30–34 seconds. Frame the requested input/output behavior. Move to the declared input and output ports. Show their names and types together. Frame the relevant code body with enough surrounding lines to understand its purpose. Move to the saved test’s input and expected result, then hold on its completed status. Finish with ports and the successful test in context. Split body and test into separate shots. Disable caret blinking or incidental scrolling through the demo’s frame-driven mechanism. Acceptance: the ports, meaningful code excerpt, and verification result are readable at the app’s video-player size. T14. Build a mini app (app-assistant) Target length: 28–32 seconds. Frame the request, then the assistant’s workflow-binding tool result. Use the actual tool card for binding details because this cast renders AppRuntimeView, not a configuration inspector. Move to the topic input as it appears with its saved value. Show the button and output region arriving. End on the assembled app with its full input-to-output layout visible. The cast builds the app but does not execute its bound workflow. Keep the closing shot about the assembled interface. Do not fabricate a generated headline or show a click that has no replayed result. Acceptance: the app’s components and their roles are clear, and the camera never targets an editor panel that is not mounted. T15. Correct it without starting over (sketch-correction) Target length: 30–34 seconds. Frame the first request, then hold the strong wash on the canvas. Move to the correction message and allow time to read it. Frame the existing layer row and opacity control as the assistant adjusts them. Return to the canvas with precisely the same framing as before. Finish on a view that includes the unchanged layer stack and corrected image. Use sequential before/after holds. Avoid a transition that obscures the actual visual change. Acceptance: viewers see the same layer being amended, a weaker effect, and no extra layer appearing. T16. It asks before it spends (storyboard-ask) Target length: 28–32 seconds. Frame the underspecified request. Move to the assistant’s clarification question and keep it still long enough to read. Briefly widen to show that the board remains empty while the question is pending. Frame the answer, then move to the board as planned shots appear in the chosen format. Close on the planned state and unrendered cards. Retain a visible waiting interval. Do not show a generation pulse or completion effect during the pause. Acceptance: the question precedes the answer, the board stays empty while waiting, and the final cards are visibly planned rather than rendered. T17. A test catches it (jsscript-repair) Target length: 34–40 seconds. Frame the suspected edge case. Move to the saved case before any code change. Hold the failing result with its NaN reason and a readable failure label. Frame the repaired code region without traversing the whole file. Return to the same saved cases for rerun and hold both successful results. Close with the retained cases and tool history in context. Use the same camera framing for failure and success. Keep textual status alongside color. Acceptance: viewers can identify the original failure, the changed code, and the same cases passing afterward. Implementation sequence Code Work Dependencies Reviewable output A1 Audit casts and target surfaces for T1–T17. Map each proposed shot to actual events, components, and scroll requirements. Record any missing demonstrated action. None Per-tutorial shot/event table and baseline contact sheets. A2 Add shot types, pure time mapping, bounds fitting, safe-area handling, and validation. Preserve legacy graph focus. A1 Deterministic camera tests and a Studio debug view showing target rectangles. A3 Build graph and fixed-layout camera adapters, target registration, and minimal action playback required by T2/T9. A2 T1 and T9 pilot renders, plus a T2 construction feasibility check. A4 Add frame-driven focus overlays and integrate captions, steps, and result holds. A3 Before/after pilots reviewed at full and embedded size. A5 Author and render T1–T7. A4 Complete graph tutorial set with readable connection and result shots. A6 Author and render T8–T14, including assistant-to-document moves. A4 Chat, timeline, and document tutorials with resolved component targets. A7 Author and render T15–T17. A6 Correction, clarification, and repair sequences preserving their causal order. A8 Verify every render, refresh posters and durations, and stage the complete asset set. A5–A7 Validated MP4s, matching posters, and catalog metadata. Keep implementation changes focused on the tutorial system and the demo surfaces needed to expose targets. Read applicable component overlays and design rules before touching production UI files. Shared component edits must follow the repository’s primitives and token rules. File ownership Area Expected changes demo/src/types.ts or a focused sibling Shot and target types independent of StepIndicator. demo/src/camera.ts Bounds-aware framing, dimension inputs, timeline validation, and legacy adaptation. demo/src/tutorialTiming.ts Presentation-to-cast time mapping and duration calculation. demo/src/components/TutorialShell.tsx Camera surface slot, fixed overlays, and explicit time delivery. demo/src/components/ Focus overlay and optional pointer cue driven by frame time. demo/src/{Tutorial,ChatTutorial,TimelineTutorial,DocTutorial}.tsx Connect each player to its camera adapter. demo/src/{tutorials,chatTutorials,timelineTutorials,docTutorials}.ts Shot plans, revised captions, and timing for the full catalog. web/src/demo/ Target exposure, deterministic scroll/action playback where required, and retimed casts. demo/package.json and render scripts Explicit complete-catalog rendering and revised poster frame selection. web/src/components/tutorials/tutorialsData.ts Duration labels matching the produced files. docs/assets/tutorials/ and web/public/tutorials/ Rendered videos and synchronized posters. Verification and delivery Code Check Passing evidence V1 Framing geometry Tests cover tall nodes, small controls, groups, edge targets, overlay exclusions, and composition dimensions. V2 Time boundaries Tests cover first/last frame, short replay windows, overlapping or unordered shots, holds, and backward seeking. Key times remain strictly ordered. V3 Target coverage Every authored target resolves at its shot anchor. A deliberately invalid identifier fails validation. The audit asserts it inspected tutorials and shots. V4 Determinism Selected frames match when rendered sequentially and sought directly, including streaming, scroll, and media readiness. V5 Visual review Contact sheets show each shot’s settled frame and result. Every video is watched for transitions and reviewed at the actual embedded-player size. V6 Lesson fidelity Each action cue matches a represented event. Correction preserves the layer, clarification precedes planning, and failure precedes repair. V7 Shared consumers Representative cookbook and gallery renders retain their legacy framing when no shots are authored. V8 Repository checks Run npm run test:affected, npm run typecheck, npm run lint, and npm run dev:nodetool -- harness gate --base origin/main after implementation. Resolve failures before declaring completion. V9 Asset delivery Each catalog video and poster exists, duration labels match encoded duration, and app posters match the docs copies. Before final encoding, export inspection frames at transition start, midpoint, settlement, action, and result. Check that text remains legible, targets do not clip, highlights track the target, and overlays do not cover controls. Watch full renders to catch uncomfortable movement that still frames cannot reveal. Use the existing renderer groups for graph, chat, timeline, documents, and steering tutorials. render:tutorials alone only invokes the graph group. A complete-catalog render command should enumerate the tutorial registries and preserve the special chat and timeline output filenames. Choose poster frames from settled result shots after retiming, rather than retaining the old numeric frame offsets. Render final videos into docs/assets/tutorials/, regenerate JPGs there, then run npm run sync:posters from demo/. Update displayed durations from the encoded files. Keep MP4s out of the app bundle. Publishing is a separate deployment action after the implementation and assets are ready. Risks to resolve in the pilots Code Risk Planned response R1 Streaming or lazy layout causes camera jitter Measure after readiness, freeze framing for the hold, and use authored scroll boundaries. R2 Zoom makes captions readable but leaves product text too small Inspect at embedded size and tighten the target or split the shot. R3 Extra movement makes tutorials harder to follow Keep M3 reading holds and remove any move that teaches nothing. R4 Retiming breaks assistant/document causality Use one mapped cast clock for both tracks and retain cast invariant checks. R5 Existing casts do not contain the advertised interaction Resolve it explicitly in A1/A3 through minimal action playback or precise captions. R6 Camera changes alter cookbook and gallery output Preserve the legacy path and verify representative existing compositions before rollout. Completion requires T1–T17 to have authored shots, resolved targets, reviewed renders, synchronized posters, accurate durations, and passing implementation checks. An animation framework alone does not complete this plan.--- # Fixed-surface tutorial focus audit Source: https://docs.nodetool.ai/plans/tutorial-focus-fixed-audit.html Fixed-surface tutorial focus audit This audit maps T8–T17 to their real cast events and mounted focus targets. The catalog test derives transition, settlement, action, result, and poster inspection frames from these shots. Final contact sheets and encoded assets belong to A8. Tutorial Cast events Authored focus sequence Scroll or action finding T8 Ask the chat agent question 300; search 1300–3600; answer 4000–10600 panel, submitted question, search result card, answer, full thread The player scrolls the requested message or tool target into view at each shot anchor. Direct and backward seeks therefore measure the same mounted target. T9 Cut a scene together select 600; trim 2200; second clip 3600; caption 5800; zoom 7200; playback 7600–15600 editor, opening clip, clip and inspector, opening clip bounds, adjacent clips, caption clip, editor, preview and tracks The nonlinear map places each authored action on its real cast event. The camera settles wide before the editor’s own zoom and stays still through the full eight-second playback. T10 Edit a sketch by asking request 400; add layer 5200; blend and opacity 9400 editor, request, add-layer tool, new layer row, blend and opacity controls, canvas The layer row and property controls are mounted production targets. One cast clock drives the dock and document patches. T11 Write and voice a script request 400; cast 3800; lines 7200; voiced takes 13200 editor, request, real cast panel, line rows, voice tool and first take, populated lines The cast panel shows the two named speakers before the tutorial moves to lines. Lines and voiced takes have separate shots. T12 Board a shot list request 400; plan 4600; first still 8600; remaining stills through 19600 editor, brief, planned board, direction table, generation tool, first completed card, finished board A 30000 ms presentation map holds the real shot edit table long enough to read description and camera movement, then maps still actions to the existing shot identities. T13 Write a JS script request 400; ports 4200; code 9000; saved case and report 13600 editor, request, ports, code body, real test console, code and console The real read-only console shows saved inputs, expected output, and the green report from the same cast clock. T14 Build a mini app request 400; operation 4400; setting and input 8600; button 11400; output region 14000 editor, request, binding tool, setting tool and document, widgets, assembled app The binding focus is the mounted tool result. The cast never runs the workflow, so there is no click or generated headline. T15 Correct it without starting over first request 400; layer 4800; blend 6600; correction 10400; opacity repair 14200 editor, first request, strong canvas, correction message, same layer and opacity control, corrected canvas, layer stack and canvas The repair targets the existing layer-warm-wash; the layer id and layer count stay unchanged. T16 It asks before it spends request 400; question 1600–5400; answer 7200; planning 8700–12200 editor, request, clarification, empty-board wait, answer, planning tool and board, planned board The 5400–7200 wait stays visible with an empty board. No generation or completion cue occurs during the pause. Final cards remain planned. T17 A test catches it request 400; save case 4200; red report 7000; repair 14000; green report 16800 editor, edge-case request, test console, failing console, repair and code, passing console, console and assistant history Red and green use the same jsscript-tests bounds. The console shows the same saved cases plus a textual failed/pass status; the red report shows expected 3, actual NaN, and its reason. Backward seeks before 7000 clear the report. Stable target exposure is narrow and semantic: chat messages, tool calls, editor controls, document regions, timeline regions, clip ids, and test results use data-focus-id. Group shots resolve every member before framing. Chat and assistant docks scroll deterministically at the shot anchor before measurement, including on backward seeks. Code shots measure the intrinsic text in the production editor’s read-only fallback. This removes lazy editor loading and caret motion from the replay. The saved CSV fixtures are checked with Papa Parse: clean input produces 7.5, the original edge case produces NaN, and the repaired sum produces 3. Test reports remain timed synthetic cast events. T8 starts from a submitted message rather than simulated typing. Every authored action beat maps to an existing chat, document patch, test report, timeline, zoom, or playback event. The catalog test also pins T9’s nonlinear presentation times to its six cast events.--- # Graph tutorial focus audit Source: https://docs.nodetool.ai/plans/tutorial-focus-graph-audit.html Graph tutorial focus audit The graph tutorials use the production DemoPlayer. Shot anchors measure the rendered ReactFlow nodes in graph coordinates, including their live heights. The camera passes one controlled viewport back to ReactFlow. It does not scale the graph a second time. T1. Build your first workflow Cast events: Text Input runs at 300 ms and completes at 1,400 ms; edge e1 activates at 1,600 ms; Enhance Prompt runs at 2,200 ms, streams from 2,600 ms, and completes at 9,600 ms; Text To Image runs at 10,600 ms and completes with an inline image at 15,500 ms. Shots: overview, prompt, input and LLM relationship, stationary LLM stream, image generation, completed image, completed graph. Fidelity: the cast has no separate Preview node. The final evidence is the image content in Text To Image, so the captions and final close-up say that directly. T2. Connect and run Cast events: the source completes at 1,300 ms; e1 activates at 1,700 ms; Uppercase runs at 2,600 ms and completes at 4,200 ms; e2 activates at 4,700 ms; Preview receives HELLO NODETOOL at 6,000 ms and completes at 6,600 ms. Shots: overview, source value, source and Uppercase relationship, Uppercase execution, Uppercase and Preview relationship, completed Preview, completed graph. Fidelity: both edges exist in the initial graph and there are no node-add, edge-add, drag, or Run-control events. This edit teaches tracing and running the existing graph. It does not claim or depict graph construction and uses no cursor. T3. Generate a list Cast events: topic completes at 1,400 ms; e1 activates at 1,700 ms; Generate List runs at 2,400 ms; five item/index pairs stream from 2,900 ms; the list completes at 10,800 ms; Preview receives the list at 12,500 ms and completes at 13,100 ms. Shots: overview, topic, topic and generator relationship, stationary list stream, completed collection in Preview, downstream pair. Fidelity: all five authored items fit the list body. No synthetic scroll or pointer action was added. T4. Ask the AI Cast events: question completes at 1,500 ms; Agent runs at 2,500 ms; answer chunks stream from 3,000 ms; Agent completes at 9,600 ms; Preview receives the answer at 11,300 ms and completes at 11,900 ms. Shots: overview, question, question and Agent relationship, stationary answer stream, completed Preview, Agent and Preview context. Fidelity: provider controls remain secondary. The cast has no question typing action, so the tutorial describes the existing submitted question. T5. Combine two inputs Cast events: the name and topic source nodes run and complete separately; both existing edges activate into Prompt; Prompt completes with the substituted sentence; Preview receives the merged value. Shots: overview, both source nodes, each source value, both incoming relationships, template, merged Preview, branch-and-merge overview. Fidelity: the opening source view is a group. Existing connections are explained without fabricated dragging. T6. Summarize a document Cast events: the source completes before Summarizer starts; summary chunks stream in the Summarizer body; Preview receives the completed short summary after streaming. Shots: overview, long source, source and Summarizer relationship, stationary summary stream, long completed-summary hold, source and summary comparison. Fidelity: the source is shown only long enough to establish its size. There is no decorative pan through the passage. T7. Describe an image Cast events: Image Input completes before Agent starts; description chunks stream in Agent; Preview receives the completed description after streaming. Shots: overview, uncropped image node, image and Agent relationship, stationary description stream, completed Preview, image and description comparison. Fidelity: the final group view compares the source and description. No pointer action is implied. Caption and duration check Captions were shortened to fit their authored intervals at roughly three words per second plus settling time. Replay windows include a one-second establishing hold and a final result hold. Total composition length is intro + replay + outro; the catalog values stay within the plan’s initial ranges or use the closest duration needed for readable evidence.--- # Tutorial focus verification Source: https://docs.nodetool.ai/plans/tutorial-focus-verification.html Tutorial focus verification The focus plan is implemented across T1–T17. Shot/event decisions are recorded in the graph audit and fixed-surface audit. Check Evidence V1 Framing Camera tests cover target height, small controls, groups, edges, overlay exclusions, and composition dimensions. V2 Timing Tests cover interval boundaries, invalid ordering, short windows, holds, backward seeks, action alignment, and result reading budgets. V3 Targets Every catalog entry rendered with required target validation. Browser tests deliberately request a missing DOM target and assert render failure. V4 Determinism Browser fixtures compare direct, sequential, and backward-seek frames with changing text, scroll, and media. The real chat composition’s frame 584 also matched a sequential render byte for byte. V5 Visual review Inspection frames cover shot starts, midpoints, settlement, actions, and results. Encoded videos were reviewed through frame sequences at 920px, with denser transition samples. This was frame-based review, not realtime playback. V6 Fidelity Cast tests preserve correction on the existing layer, clarification before planning, and failure before repair. CSV fixtures reproduce the displayed clean sum, NaN failure, and repaired sum. V7 Shared consumers Representative Cookbook-image-enhancement and Workflow-transcribe-audio frames rendered through the retained legacy camera path. V8 Repository npm run test:affected, npm run typecheck, npm run lint, and npm run dev:nodetool -- harness gate --base origin/main passed. Typechecking used Node’s 8GB heap setting. V9 Assets All 17 MP4s fully decoded. Each is H.264, 1920×1080, 30fps, with the catalog frame count. Encoded durations match app labels. All 17 posters match their app copies byte for byte. No MP4 is bundled under web/public/tutorials/. Videos and posters are in docs/assets/tutorials/. The complete render command is npm run render:tutorials:all from demo/, followed by npm run sync:posters. It derives filenames, inspection frames, posters, and durations from the complete catalog. The casts remain synthetic demonstrations using production components. T2 traces an existing graph, T9 represents the recorded trim without a fabricated drag, and T14 ends on the assembled app without claiming a workflow run. Publication is separate from these committed assets.--- # TypeScript 7 migration plan Source: https://docs.nodetool.ai/plans/typescript-7-migration.html TypeScript 7 migration plan This document is the implementation contract for moving NodeTool’s command-line compiler to TypeScript 7 while retaining TypeScript 6 for tools that use the compiler API. Problem statement NodeTool’s root dependency range resolves to TypeScript 5.9, so package builds, application type-checks, and most local tsc commands use the JavaScript-based compiler. TypeScript 7 provides a native tsc, parallel checking, and lower build latency, but TypeScript 7.0 does not expose the compiler API. NodeTool scripts and development tools import that API. A direct replacement of typescript with version 7 would break API consumers, make the compiler selected by bare tsc commands depend on npm bin-link order, and bypass assumptions in the current compiler runners. The migration needs a side-by-side compiler topology, a single executable-selection boundary, and a staged cutover that keeps every merged step green. Goal Use TypeScript 7.0.2 for NodeTool’s command-line builds and type-checks. Keep @typescript/typescript6 available through the package name typescript for compiler-API consumers and an explicit fallback command. Apply the same policy to the root workspaces and the separate mobile, marketing, and Chrome-extension dependency trees where their toolchains require it. Completion means: normal repository build and type-check commands run the TypeScript 7 native compiler. scripts, linters, test transformers, and other API consumers load TypeScript 6 without invalid peer dependencies. clean, incremental, project-reference, watch, and declaration builds retain their required behavior. every TypeScript surface has an explicit version policy and lockfile. CI identifies the compiler it ran and can select TypeScript 6 as a rollback. the required post-change verification commands pass. Current state The implementation task must re-run these inventories. Do not rely on a stored count because manifests and compiler entry points change. rg -n '"typescript"\s*:' --glob 'package.json' --glob '!**/node_modules/**' rg -n '\btsc\b|typescript/bin|typescript/lib' \ --glob '!**/node_modules/**' --glob '!electron/backend-bundle/**' rg -n "(?:from|require\(|import\()\s*['\"]typescript['\"]|typescript/lib" \ --glob '!**/node_modules/**' --glob '!electron/backend-bundle/**' rg -n '"(baseUrl|moduleResolution|esModuleInterop|ignoreDeprecations)"\s*:' \ --glob 'tsconfig*.json' --glob '!**/node_modules/**' npm ls typescript --all Known compiler-API consumers include: scripts/check-workspace-deps.mjs scripts/sync-base-node-metadata.mjs scripts/generate-node-docs.mjs packages/base-nodes/scripts/classify.ts packages/telegram/tests/dependency-cone.test.ts typescript-eslint, @typescript-eslint/*, ts-jest, ts-node, and TypeScript-based documentation tooling Known executable assumptions include: scripts/build-typescript-workspace.mjs executes node_modules/typescript/bin/tsc through Node. scripts/run-tsc.mjs searches for the same JavaScript entry point and adds a V8 heap limit. package scripts invoke bare tsc for lint, build, type-check, and watch. mobile, marketing, and the Chrome extension have independent package locks. Known TypeScript 7 configuration blockers include: web/tsconfig.json sets esModuleInterop to false. web/tsconfig.node.json and electron/tsconfig.test.json use legacy moduleResolution: "node". electron/tsconfig.test.json, demo/tsconfig.json, and examples/chat_app/tsconfig.json use baseUrl. some configs depend on implicit @types inclusion, while TypeScript 7 defaults types to an empty list. Next.js configs declare language-service plugins that TypeScript 7.0 cannot host through a public API. Generated contents under electron/backend-bundle/ are build artifacts, not migration inputs. Fix their source or bundling step instead of editing bundled package manifests. External constraints TypeScript 7.0 is the native compiler release and documents the side-by-side installation. TypeScript 6.0 is the compatibility bridge. Its stableTypeOrdering mode exposes ordering differences before the TypeScript 7 cutover. @typescript/typescript6 exports the TypeScript 6 API and a tsc6 executable. TypeScript 7 intentional changes define accepted differences between the JavaScript and native compilers. Re-check the stable patch releases before implementation. If a newer 7.0 patch exists, use it only after reviewing its release notes. Do not adopt a 7.1 prerelease as part of this migration. User stories US1: Faster command-line feedback As a NodeTool developer, I can run the existing build and type-check commands through the native compiler without learning a second command set. Acceptance criteria: npm run build:packages, npm run typecheck, and package-local compiler commands identify TypeScript 7 as their command-line compiler. cold and warm measurements are captured for TypeScript 6 and 7 on the same machine and input. the native path does not inherit obsolete V8 heap configuration. US2: Existing compiler integrations keep working As a maintainer of NodeTool scripts and test tooling, I can continue importing typescript and receive the supported TypeScript 6 API. Acceptance criteria: every direct typescript import resolves to @typescript/typescript6. typescript-eslint, ts-jest, ts-node, and documentation scripts run without peer-dependency warnings caused by TypeScript 7. no API consumer imports @typescript/native. US3: Builds remain reproducible As a release maintainer, I can install dependencies from each lockfile and get the same compiler roles on supported operating systems. Acceptance criteria: compiler and compatibility-package versions are exact pins during the migration. root and standalone lockfiles contain the intended platform packages. a repository check fails when a workspace reintroduces an unapproved TypeScript range or direct compiler path. US4: The cutover is reversible As a CI or release maintainer, I can rerun a failing compiler step with TypeScript 6 without editing manifests or regenerating the lockfile. Acceptance criteria: the shared launcher accepts an explicit compiler selection. CI and local scripts expose a documented TypeScript 6 fallback command. selection errors name the missing package and corrective install command. Implementation decisions D1: Separate the compiler from the compiler API The target root topology is: { "devDependencies": { "@typescript/native": "npm:typescript@7.0.2", "typescript": "npm:@typescript/typescript6@6.0.2" } } @typescript/native owns command-line compilation. The package name typescript remains the TypeScript 6 compatibility API because third-party tools import that name and declare a TypeScript 6 peer range. Use exact versions for the initial cutover. A later dependency update can restore a compatible range after CI has exercised the native compiler. D2: Select compilers through one launcher Create a shared executable resolver used by scripts/build-typescript-workspace.mjs and scripts/run-tsc.mjs. It must: select TypeScript 6 or 7 explicitly. read package metadata instead of assuming typescript/bin/tsc. spawn JavaScript compiler entry points through Node and native executables directly. prefer a standalone project’s local installation before the repository root. forward arguments, environment, signals, stdout, stderr, and exit status. add NODETOOL_TSC_HEAP_MB only to the TypeScript 6 JavaScript process. produce a clear error when the selected compiler is absent. Use one environment variable, NODETOOL_TSC_VERSION=6|7, for low-level selection. Repository scripts provide named commands so developers do not need to set it for normal work. Keep version 6 as the default until the cutover task. D3: Treat the migration as expand, migrate, contract The launcher and dual-compiler dependencies are the expansion. Backend, application, and standalone surfaces migrate in independently verifiable cohorts. The final cutover changes the default to 7 and contracts direct compiler paths. TypeScript 6 remains as the API package and rollback compiler, so removing it is not part of this work. D4: Fix configuration behavior under TypeScript 6 first Run TypeScript 6 with --stableTypeOrdering and without ignoreDeprecations. Fix errors rather than suppressing them. Do not leave stableTypeOrdering in normal commands after parity is established because it exists for migration diagnosis and slows the JavaScript compiler. Adopt TypeScript 7-compatible settings before switching the default: replace moduleResolution: "node" with bundler for bundled applications or nodenext with a matching module for Node applications. remove baseUrl and make each paths target explicitly relative. remove esModuleInterop: false without disabling synthetic default imports. declare each project’s required global types. enable noUncheckedSideEffectImports explicitly and add declarations for valid asset imports instead of weakening the option. retain explicit rootDir on emitting projects. remove deprecated or ignored options instead of setting ignoreDeprecations. Do not change strict, skipLibCheck, exactOptionalPropertyTypes, runtime targets, or module formats unless a TypeScript 7 incompatibility requires a targeted change. D5: Keep fixes local to the exposed incompatibility TypeScript 6 or 7 may report new inference errors. Add an explicit type argument, annotation, import, or module declaration at the narrowest boundary. Do not introduce any, non-null assertions, broad casts, @ts-ignore, or weaker compiler settings. D6: Compare behavior, not byte ordering TypeScript 7 can change declaration ordering. A byte-for-byte .d.ts diff is diagnostic, not an acceptance gate. Review differences and require the same exports, types, runtime module format, source-map availability, package files, and consumer test behavior. D7: Keep editor plugins on TypeScript 6 where required TypeScript 7.0 does not expose the language-service API used by some Next.js and embedded-language plugins. CLI builds and type-checks use TypeScript 7. Editors may use the TypeScript 6 workspace service for those projects until TypeScript 7.1 and the relevant plugins are supported. Document the split. Do not remove plugin configuration merely to make the TypeScript 7 CLI quiet. D8: Tune native parallelism only from measurements Turbo already runs package builds concurrently. TypeScript 7 also parallelizes checking and project-reference builds. Measure the combinations before setting --checkers or --builders. Start package-local Turbo builds with conservative native concurrency, then raise it only when wall time improves without exceeding CI memory. Keep the selected values configurable rather than tied to a developer workstation’s core count. D9: Include standalone dependency trees The root lockfile does not control mobile/, marketing/, or chrome-extension/. Each has its own migration and verification step. Standalone projects that have compiler-API tooling use the same dual-package topology. A project with no API consumer may depend directly on TypeScript 7, but the task must prove that with npm ls and its full checks. Milestones and dependency graph M1 Compatibility foundation T1 Baseline and inventory └─> T2 Compiler launcher expansion └─> T3 TypeScript 6 and configuration bridge └─> T4 Dual-compiler topology and shadow lane M2 Cohort migration T4 ──> T5 Backend packages and project references ──┐ ├──> T6 Web, Electron, demo, and examples ─────────┼──> T8 Default cutover and CI └──> T7 Mobile, marketing, and Chrome extension ──┘ M3 Completion T8 ──> T9 Contract checks, documentation, and final verification T5, T6, and T7 can run in parallel after T4. All other blocking edges are intentional. Tickets Each ticket is self-contained for an implementation agent. The implementation notes name the expected path. The checkboxes are the merge criteria. T1 — Capture the compiler inventory and baseline Blocked by: None — can start immediately. Status: ready-for-agent What to build: Produce reproducible evidence for compiler selection, current behavior, and performance before changing dependencies. Implementation notes: Run the inventory commands in Current state. Classify every result as command-line compiler, compiler-API consumer, configuration, generated artifact, or documentation. Record the actual version used by root, package-local, web, Electron, mobile, marketing, and Chrome-extension commands. Run clean and warm baselines for build:packages, the root type-check, build:packages:tsc, and representative package builds. Capture exit status, wall time, and peak memory where the runner supports it. Save command output as PR evidence. Do not add transient timings or counts to repository guidance. Identify existing failures before the migration. Do not attribute them to the compiler change. Acceptance criteria: Each TypeScript surface has a named owning manifest, lockfile, tsconfig, and verification command. Every direct compiler path and API import is classified. All baseline commands have recorded exit statuses. The task changes no production behavior. T2 — Add a version-aware compiler launcher Blocked by: T1 — Capture the compiler inventory and baseline. Status: ready-for-agent What to build: Make JavaScript and native compiler execution explicit through one launcher while the default remains unchanged. Primary files: scripts/run-tsc.mjs scripts/build-typescript-workspace.mjs scripts/__tests__/run-tsc.test.mjs the matching build-workspace script tests root and workspace npm scripts that invoke bare tsc scripts/create-pack.mjs, which generates compiler scripts for new packages Implementation notes: Extract package and executable resolution into one small module. Support TypeScript 6’s tsc or tsc6 entry and TypeScript 7’s native tsc entry without assuming either is a Node script. Add NODETOOL_TSC_VERSION validation and named script entry points. Route repository-owned root and workspace package scripts through the launcher so npm bin-link order cannot select a compiler. Include build, no-emit, lint, watch, Electron, examples, and generated package templates. Preserve nearest-install behavior for mobile and other standalone trees. Preserve exit codes and signal handling. Apply the V8 heap limit only when launching the JavaScript compiler. Update tests to execute a JavaScript stub and a native-style executable stub. Prove arguments, environment, working directory, exit status, and missing-package errors. Keep compiler 6 as the default in this task. Acceptance criteria: Existing compiler commands still select the current compiler. Both executable kinds are covered through subprocess behavior tests. A non-zero compiler exit remains non-zero at the npm command boundary. Mobile-local resolution still wins over the root installation. No owned root or workspace command invokes a direct compiler path or bare tsc. Documented user examples and standalone projects assigned to T7 are reported separately. T3 — Make all configurations TypeScript 6 and 7 compatible Blocked by: T2 — Add a version-aware compiler launcher. Status: ready-for-agent What to build: Remove deprecated configuration and surface TypeScript 7 defaults under the TypeScript 6 compiler. Primary files: tsconfig.base.json web/tsconfig*.json electron/tsconfig*.json demo/tsconfig.json examples/*/tsconfig.json standalone project tsconfigs manifests and lockfiles needed to run the TypeScript 6 bridge Implementation notes: Install the supported TypeScript 6 bridge version and regenerate the relevant lockfiles. Run every type-check with --stableTypeOrdering and no deprecation suppression. Replace legacy module resolution according to D4. Remove baseUrl and preserve every alias with project-relative paths. Remove the false interop setting and verify runtime imports through existing tests and builds. Add explicit types lists based on actual global usage. Enable noUncheckedSideEffectImports. Add narrow *.css, media, worker, or other asset declarations where required. Audit for removed import assertion syntax and deprecated namespace syntax. Fix new inference errors without weakening repository TypeScript rules. Run TypeScript 6 twice from a clean state to check stable diagnostics and declaration output. Acceptance criteria: TypeScript 6 passes every existing compiler command with stable ordering. No tsconfig contains ignoreDeprecations, baseUrl, legacy node/node10 module resolution, or esModuleInterop: false. Every emitting project has an explicit rootDir. Required global and asset types are explicit. TypeScript 5 is no longer required to pass any in-scope surface. T4 — Install both compilers and add a TypeScript 7 shadow lane Blocked by: T3 — Make all configurations TypeScript 6 and 7 compatible. Status: ready-for-agent What to build: Make TypeScript 7 runnable everywhere without changing the default compiler. Primary files: root and workspace package.json files package-lock.json compiler runner and version-contract tests root npm scripts Implementation notes: Set typescript to the exact @typescript/typescript6 npm alias wherever a root-workspace tool or direct import needs the API. Add the exact TypeScript 7 package through the @typescript/native alias at the root. Align workspace declarations so npm does not install nested TypeScript 5 copies. Regenerate the root lockfile and inspect platform-specific native packages. Add explicit typecheck:ts6, typecheck:ts7, build:tsc6, and build:tsc7 migration commands, or equivalent named entry points. Add a contract test proving command selection and API selection: tsc role is 7, tsc6 role is 6, and import "typescript" exposes the TypeScript 6 API. Run npm ls and resolve invalid compiler peer dependencies without using --force or --legacy-peer-deps. Keep normal build and type-check commands on compiler 6 until T8. Acceptance criteria: A clean npm ci installs both compiler roles. Named TypeScript 6 and 7 commands report the expected major versions. Direct API imports and type-aware lint/test tooling still use TypeScript 6. The lockfile has no unintended nested TypeScript 5 installation owned by a NodeTool workspace. The existing default commands remain green on TypeScript 6. T5 — Migrate backend packages and project-reference builds Blocked by: T4 — Install both compilers and add a TypeScript 7 shadow lane. Status: ready-for-agent What to build: Make all backend package compilation and declaration emission pass through TypeScript 7. Primary boundaries: tsconfig.base.json and tsconfig.build.json packages/*/tsconfig.json scripts/build-typescript-workspace.mjs scripts/build-stale-websocket-workspaces.mjs scripts/clean-ts-build.mjs package build, test, and packaging commands Implementation notes: Run the root project-reference build through TypeScript 7 from a clean checkout. Run Turbo package builds through TypeScript 7 with conservative checker concurrency. Fix diagnostics by dependency order. Keep each fix in the owning package. Exercise decorator-heavy node packages and packages loaded from dist/. Compare TypeScript 6 and 7 JavaScript, declarations, declaration maps, source maps, and package entry points. Classify each difference against the official intentional-change list. Reproduce and verify incremental behavior for deleted sources, missing dist/, stale build info, --force, and build stamps under TypeScript 7. Verify parallel package tests cannot observe partially missing outputs. Run package tests against TypeScript 7-produced dist/ output. Update comments and tests that claim behavior was measured only on TypeScript 6. Acceptance criteria: Clean and incremental package builds pass with TypeScript 7. npm run build:packages:tsc and Turbo package builds produce consumable package output. Orphan pruning and build-stamp tests pass against the native compiler path. Package tests pass using TypeScript 7 output. Reviewed declaration differences preserve public API semantics. npm pack --dry-run for representative published packages contains the required files. T6 — Migrate web, Electron, demo, and examples Blocked by: T4 — Install both compilers and add a TypeScript 7 shadow lane. Status: ready-for-agent What to build: Run application type-check and build entry points through TypeScript 7 while API-based test and lint tooling remains on TypeScript 6. Primary boundaries: web/ electron/ demo/ examples/chat_app/ examples/nextjs-vercel/ examples/nextjs-cloudflare/ Implementation notes: Run each application’s no-emit type-check through TypeScript 7. Fix diagnostics at the application boundary without changing runtime module format or strictness. Run Vite, Next.js, Remotion, and Electron builds that consume the checked source. Run Jest and ts-jest suites and prove they still load the TypeScript 6 API. Run type-aware linting and prove typescript-eslint sees its supported peer. Keep Next.js language-service plugin configuration and document editor use of TypeScript 6 until native plugin support is available. Measure web type-check memory without the JavaScript compiler heap override. Verify Electron backend preparation and bundle smoke tests when emitted backend output changed. Acceptance criteria: All application type-checks pass with TypeScript 7. Web, Electron, demo, and example builds pass. Jest, ts-jest, ESLint, and documentation tooling pass with TypeScript 6 as their API provider. Next.js plugin behavior is not silently removed. The TypeScript 7 application path does not execute the native binary through Node or apply NODETOOL_TSC_HEAP_MB to it. T7 — Migrate standalone dependency trees Blocked by: T4 — Install both compilers and add a TypeScript 7 shadow lane. Status: ready-for-agent What to build: Apply an explicit compiler policy to mobile, marketing, and the Chrome extension, including their independent locks. Primary boundaries: mobile/package.json and mobile/package-lock.json marketing/package.json and marketing/package-lock.json chrome-extension/package.json and chrome-extension/package-lock.json their tsconfigs and npm scripts Implementation notes: Inventory direct API consumers and peer ranges inside each standalone tree. Use the dual-package topology for mobile and marketing if Expo, Next.js, ESLint, or another tool imports typescript. Use TypeScript 7 directly for the Chrome extension only if its complete dependency tree has no compiler-API requirement. Otherwise use the same dual-package topology. Update each local compiler command to select the intended executable explicitly. Regenerate each lockfile with its own install command. Run mobile type-check, lint, and tests. Run marketing type-check, build, and non-network checks. Run Chrome-extension type-check, build, and tests. Confirm the root runner still prefers a standalone project’s local compiler. Acceptance criteria: Each standalone project reports its command-line and API compiler roles. Clean installs from all standalone lockfiles succeed without peer overrides. Mobile, marketing, and Chrome-extension checks pass. The root typecheck:mobile path selects the intended mobile compiler. No standalone manifest retains an unconstrained TypeScript 5 range. T8 — Switch defaults, update CI, and tune concurrency Blocked by: T5 — Migrate backend packages and project-reference builds. T6 — Migrate web, Electron, demo, and examples. T7 — Migrate standalone dependency trees. Status: ready-for-agent What to build: Make TypeScript 7 the normal compiler with a tested TypeScript 6 rollback. Primary boundaries: root and package npm scripts compiler launcher defaults CI workflows and cache inputs Turbo configuration when measurements require it Implementation notes: Change the compiler launcher’s default from 6 to 7. Make ordinary build, type-check, lint-time compiler, and watch commands use the native compiler. Keep named :ts6 fallbacks. Add an early CI version-contract step after npm ci. Run native package installation and compiler smoke checks on every operating system used for release builds. Compare TypeScript 6 and 7 cold and warm measurements on the same revision. Measure Turbo concurrency together with TypeScript 7 --checkers and --builders. Set conservative configurable defaults only when they improve the measured commands within CI memory. Verify watch mode observes edits, additions, deletions, and renames. Run one CI-equivalent job with the TypeScript 6 fallback to prevent rollback drift during the migration window. Document the rollback command in developer and CI troubleshooting guidance. Acceptance criteria: Ordinary compiler commands identify TypeScript 7. CI fails before compilation when compiler roles are wrong or a native binary is missing. All supported CI operating systems can execute the native compiler. TypeScript 7 is faster than TypeScript 6 on the agreed representative cold or warm command, measured on the same host and revision. Native concurrency stays within CI memory and does not multiply Turbo’s parallelism without a measured benefit. The TypeScript 6 fallback remains green. T9 — Add drift checks, update guidance, and run final verification Blocked by: T8 — Switch defaults, update CI, and tune concurrency. Status: ready-for-agent What to build: Prevent regression to ambiguous compiler selection and close the migration with repository evidence. Primary boundaries: dependency and lockfile checks under scripts/ compiler runner tests docs/DEVELOPMENT_STANDARDS.md, docs/dev-environment.md, and relevant AGENTS.md files only where commands or behavior changed this plan Implementation notes: Add a repository check that enumerates in-scope manifests and rejects unapproved TypeScript ranges, direct native-API imports, and compiler paths outside the shared launcher. Assert that the audit inspected files. Add lockfile assertions for the compiler roles and standalone locks. Update comments and documentation that describe V8 heap requirements, compiler locations, version selection, watch behavior, or rollback. Do not hard-code package counts or transient timing results in guidance. Run documentation link checks and npm run check:agents-docs if agent instructions change. Run all migration checks and the mandatory post-change verification. Record any accepted TypeScript 6/7 diagnostic or declaration differences in the PR with the test that proves runtime behavior. Acceptance criteria: The drift check fails on a deliberately invalid manifest or direct compiler path, then passes after the fixture is restored. Documentation describes TypeScript 7 as the CLI compiler and TypeScript 6 as the API compatibility package and fallback. No repository command depends on npm bin-link collision order. All commands in Final verification pass. Testing decisions Boundary Evidence Compiler selection Subprocess tests cover TypeScript 6 JavaScript and TypeScript 7 native executables, arguments, environment, signals, and exit codes. API compatibility A Node test imports typescript, checks the expected major version, and exercises a small API such as createSourceFile. Dependency topology Manifest and lockfile audits reject TypeScript 5 workspace ranges, invalid peers, and missing native platform packages. Configuration TypeScript 6 stable-ordering and TypeScript 7 shadow commands compile every tsconfig without deprecation suppression. Backend emit Clean, incremental, forced, and project-reference builds are exercised, followed by package tests and representative npm pack --dry-run checks. Applications Existing web, Electron, demo, and example type-check, build, lint, and test commands run with the intended compiler roles. Standalone projects Each independent lockfile receives a clean-install check and its local type-check/build/test commands. Watch mode An automated script edits, adds, deletes, and renames a temporary source file and observes the expected rebuilds without touching real source. Performance TypeScript 6 and 7 run on the same revision, host, input, and clean/warm state. Capture producer exit status with each timing. Rollback CI or an equivalent local job runs the named TypeScript 6 fallback after TypeScript 7 becomes the default. Tests must distinguish compiler diagnostics from failures in Vite, Jest, Electron Builder, Next.js, or runtime execution. Passing a bundler build does not replace an explicit TypeScript 7 check. Final verification Run from the repository root unless the command states otherwise: npm ci npm run check:lockfile npm run build:packages:clean npm run build:packages:tsc npm run typecheck npm run typecheck:examples npm run test:affected npm run lint npm run dev:nodetool -- harness gate --base origin/main npm run backend:smoke Run the compiler-role assertions and fallback commands defined by T4 and T8. Then run the independent project checks: npm --prefix mobile ci npm --prefix mobile run typecheck npm --prefix mobile run lint npm --prefix mobile test npm --prefix marketing ci npm --prefix marketing run seo:typecheck npm --prefix marketing run build npm --prefix chrome-extension ci npm --prefix chrome-extension run typecheck npm --prefix chrome-extension run build npm --prefix chrome-extension test If a standalone script name differs when implementation starts, use the manifest’s current equivalent and update this plan in the same change. Rollout and rollback The migration lands as an expand-and-contract sequence. T1 through T4 add capability without changing normal compiler behavior. T5 through T7 prove the native compiler by cohort. T8 changes the default only after every cohort is green. T9 adds enforcement. Rollback after T8 sets NODETOOL_TSC_VERSION=6 or invokes the named TypeScript 6 script. It does not change dependencies or lockfiles. If TypeScript 7 emits incorrect production output, switch the affected build command to the fallback and open a minimal reproduction before attempting a source workaround. Do not delete the TypeScript 6 compatibility package during rollback. It is still required by compiler-API consumers. Risks R1: Compiler API absence A dependency can import typescript indirectly even when NodeTool source does not. Mitigation: retain the alias, inspect peer dependencies with npm ls, and test API-based lint and test tooling. R2: Native executable launched through Node The current runners assume a JavaScript file. Mitigation: T2 distinguishes entry types and exercises real subprocess behavior. R3: New defaults change ambient types or asset imports TypeScript 7 defaults types to empty and enables unchecked side-effect import checking. Mitigation: expose both under TypeScript 6, then add explicit project configuration and asset declarations. R4: Parallelism exceeds CI memory Turbo and TypeScript 7 can multiply concurrency. Mitigation: start with conservative checker counts, measure peak memory, and keep overrides configurable. R5: Incremental output behavior differs NodeTool has custom orphan pruning and build stamps around tsc --build. Mitigation: reproduce each invariant under TypeScript 7 before changing the default. R6: Declaration output changes Stable type ordering and the native emitter can produce text differences. Mitigation: compare public semantics and consumer tests rather than requiring byte identity. R7: Editor plugin loss TypeScript 7.0 cannot host TypeScript 6 language-service plugins. Mitigation: keep the TypeScript 6 workspace service for affected projects and limit the initial native cutover to CLI commands. R8: Native platform package missing from a lockfile The native compiler installs platform-specific packages. Mitigation: exercise clean installs and a version smoke check on every release operating system. Out of scope migrating compiler-API consumers to the future TypeScript 7.1 API. adopting a TypeScript 7.1 prerelease. enabling exactOptionalPropertyTypes. changing ECMAScript targets or runtime module formats without a demonstrated TypeScript 7 blocker. replacing Jest, ts-jest, TypeScript ESLint, Next.js, Expo, or bundlers for unrelated reasons. broad source refactors prompted only by new diagnostics. removing the TypeScript 6 rollback during the initial rollout. changing generated files under electron/backend-bundle/ by hand. Open questions Q1: How long should CI keep the TypeScript 6 fallback lane? Default: keep it through the initial TypeScript 7 rollout and remove the CI lane in a separate change after the team has a release-cycle policy. Keep the TypeScript 6 API dependency until its consumers migrate. Q2: Should standalone projects use one shared launcher? Default: use the root launcher where the project already calls root scripts. Otherwise keep a small local npm command that selects its local package. Do not make standalone builds depend on an incidental root installation. Q3: Should TypeScript 7 checker counts be fixed in CI? Default: fix them only if repeated measurements show unstable memory or diagnostics. Prefer the compiler defaults for single application checks and a lower configurable value for externally parallel Turbo package builds.--- # ui_primitives overhaul — verified brief, tech design, implementation plan Source: https://docs.nodetool.ai/plans/ui-primitives-overhaul.html ui_primitives overhaul — verified brief, tech design, implementation plan Status: planned. Verified against main @ 2cf277b (2026-07-18). All file:line references below were re-checked against that commit; when the code drifts, re-verify before implementing. 1. Problem statement web/src/components/ui_primitives/ is a naming layer over MUI, not a design system. The import policy is enforced (never import raw MUI outside ui_primitives/ and editor_ui/) and holds in practice — a sweep found zero raw TextField/Select imports in app code. But the visual decisions were never made, so a screen built 100% from primitives still looks amateur by default: fields with mismatched variants, invisible panels, four label treatments, and per-component invented heights. The measure of success: a bare <Panel><FlexColumn gap={3}><TextInput label="Name" /><SelectField label="Kind" options={...} /></FlexColumn></Panel> with no styling props renders as a professional form — visible panel, two identically-styled fields, labels above controls, one control height. Reference implementation of the target look: the storyboard header in web/src/components/storyboard/StoryboardBoard.tsx (as of 2cf277b). It hand-builds everything this plan systematizes: a local Field wrapper (StoryboardBoard.tsx:69) putting labels above controls, uniform 36px control heights via CSS override (StoryboardBoard.tsx:131-133), an uppercase .group-label (:112-116), a responsive .header-grid (:102-111), and an overflow: visible workaround for Panel clipping the floating label (:96-98). Note it does not solve everything — its Panel is still the invisible default background. Once the primitives are fixed, migrate this file to the shared versions and delete the local workarounds (Fix 4). 2. Defect inventory (verified) Every claim below was verified against the codebase; corrections from the original audit draft are marked ⚠. 2.1 No control-level tokens tokens.ts exports FONT_WEIGHT, FONT_SIZE_SANS/FONT_SIZE_MONO, TYPOGRAPHY, MOTION, reducedMotion, Z_INDEX, BORDER_RADIUS, and scrollbar helpers — but no control height, control padding, or field radius. Every primitive invents its own: Component Value Source EditorButton 24 / 28 raw px editor_ui/EditorButton.tsx:55 Editor node controls 28px theme.editor.heightNode (ThemeNodetool.tsx:70), used at themes/components/editorControls.ts:108 Inspector controls 32px theme.editor.heightInspector (ThemeNodetool.tsx:71), used at editorControls.ts:129 TagInput 48px ui_primitives/TagInput.tsx:51 SearchInput radius 8px ui_primitives/SearchInput.tsx:48 Editor control radius 6px theme.editor.controlRadius (ThemeNodetool.tsx:76) MuiButton radius theme.rounded.buttonLarge = 6px ThemeNodetool.tsx:241 Consequences: a compact EditorButton (24px) next to a node-canvas select (28px) misaligns by 4px; next to an inspector select by 8px. Three independent radius sources for controls. A second, parallel control scale lives on theme.editor that EditorButton does not read. EditorButton.tsx:65 does height: size ? undefined : height — passing size silently discards the density height. Off-grid values inside the primitives whose own docs ban them (spacing.ts:23 forbids 0.25/0.75/1.25/2.5/5 steps; canonical scale is [0, 0.5, 1, 1.5, 2, 3, 4, 6, 8] units of 4px): Slider.tsx:45 — 13px vertical padding (normal density) NodeSlider.tsx:56 — 3px margin-top (compact) Panel.tsx:39 — comfortable: 2.5 units; Panel.tsx:151 computes paddingValue / 2 ⚠ (a computed halving, not a literal 1.25 — yields 1.25 units = 5px for comfortable) ⚠ Additional offenders the original draft missed (same class of defect, fold into the Fix 1 sweep): File Defect TabGroup.tsx:86,105 minHeight: isSmall ? "36px" : "42px" — 42 is off-grid Chip.tsx:64 height: "22px" — off-grid UndoRedoButtons.tsx:51-52,68,73 padding: 6px, border-radius: 6px, 24px box MenuItemPrimitive.tsx:46,114,131,135 border-radius: 4px literals, min-height: 32px, min-width: 28px MobileBottomSheet.tsx:45-46,62-63 16px corner radii, 40px/4px grab handle WarningBanner.tsx:73 border-radius: 8px Breadcrumbs.tsx:56 border-radius: 4px ZoomControls.tsx:34 minWidth: "45px" — off-grid NotificationBadge.tsx:56 font-size: 10px — off the type scale 2.2 Unrelated defaults across primitives TextInput.tsx:52-53 → variant="outlined", size="medium" SelectField.tsx:98-99 → size="medium", variant="standard" A no-props TextInput above a no-props SelectField renders a boxed field above an underline-only field — the default composition is broken. Of 29 real SelectField call sites, 17 rely on the standard default and 12 pass variant="standard" redundantly (7 of those in SettingsMenu.tsx alone). Surfaces: Panel.tsx:89-90 → bordered=false, background="default". Dark background.default is #08090A (paletteDark.ts:268) — same as the page. A default Panel is invisible. (Corrected during PR 3: the “42 call sites” here was a <Panel prefix-grep artifact matching PanelLeft, PanelHeadline, etc. — the primitive is actually used at 13 sites across 10 files. Also found: Panel’s numeric borderRadius: 6 went through the sx multiplier and rendered 36px, not 6px.) Surface.tsx:64-68 → background="paper", bordered=false, elevation=0 with explicit boxShadow: "none". Different background AND a different radius source (theme.rounded[...], Surface.tsx:92-95) than Panel (theme.shape.borderRadius, Panel.tsx:135). size semantics diverge: TextInput maps compact → MUI small (TextInput.tsx:73); SelectField’s standard variant barely responds to size; EditorButton discards its height when size is passed (see 2.1). Flex containers: FlexRow.tsx:55, FlexColumn.tsx:53, Stack.tsx:61 all default gap/spacing to 0. ~374 of ~1010 usages (37%) rely on that default, so it cannot be changed — form spacing must come from the Fix 4 composition primitives instead. 2.3 Four label treatments, none shared MUI floating label — TextInput, hand-correcting MUI’s transform (TextInput.tsx:95-96 translate(14px, 17px)), sized via theme.fontSizeNormal || "15px" ⚠ (token with fallback, not a hardcoded 15px), opacity: 0.6 (TextInput.tsx:89-92). MUI InputLabel — SelectField.tsx:123-129, same 15px, no opacity, no transform fix. Renders differently from TextInput’s at rest. Label.tsx — Typography component="label" (:81), 13px, weight 500, text.secondary, baked marginBottom: theme.spacing(0.5) = 2px (:88). FormField.tsx:85-96 — reimplements the same thing with its own Typography (does NOT use Label), duplicating the required-asterisk logic (FormField.tsx:98-108 vs Label.tsx:95-104). The theme sets a fifth treatment on MuiFormLabel (ThemeNodetool.tsx:249-262): capitalize, 13px, padding: 0 0 8px 0 — which TextInput/SelectField then override back to 15px, defeating the theme’s label token exactly where labels matter. The floating-label choice is also a bug class: the shrunk label renders above the input box, and Panel.tsx:136 is overflow: hidden, so a TextInput at the top of a Panel gets its label clipped. This shipped (the storyboard title) and is worked around in StoryboardBoard.tsx:96-98 with overflow: visible. Escape hatches are asymmetric: SelectField has hideLabel (SelectField.tsx:48); TextInput has none — the storyboard Field wrapper exists partly because you cannot stop TextInput from floating its label. 2.4 No usable form composition primitive FormField exists but has exactly 1 consumer (components/collections/CollectionForm.tsx); STRATEGY.md:47 counts 14+ places doing label+input+helper by hand. FormField double-labels: it renders its own label while TextInput/SelectField render theirs. No FormRow, FormSection, FieldGroup, or grid primitive. Only flex helpers — which default to gap=0 (2.2). Spacing ownership is incoherent: SelectField.tsx:117 comments “spacing is the parent’s job” while Label.tsx:88, FormField.tsx:120, AutocompleteTagInput.tsx, EmptyState, ProgressBar, SectionHeader, and Panel bake their own margins. Stacking three fields yields four different gaps. TextInput additionally inherits MUI’s default helper-text margin (3px 14px 0) which nothing overrides — off-grid and different from SelectField’s mt: 0.5 (SelectField.tsx:157). 2.5 The type scale is fake Text.tsx:14 advertises 8 sizes; ThemeNodetool.tsx:39-46 collapses them to 4 real values: giant = bigger = big = 18px; normal = 15px; small = 13px; smaller = tiny = tinyer = 11px. <Text size="tiny"> and <Text size="smaller"> are indistinguishable. Usage counts across web/src (for migration sizing): small 277, normal 71, big 42, smaller 39, tiny 36, bigger 17, tinyer 4, giant 2 — 488 total. The aliased names (bigger, tiny, tinyer, giant) cover 59 call sites. Nothing consumes the legacy CSS vars (--fontSizeTiny etc.) directly. 2.6 Zero-opinion passthroughs masquerading as primitives Box.tsx — 2-line re-export. (Acceptable as an import-policy shim, but should say so.) muiReexports.ts:107-130 re-exports Accordion/AccordionSummary/ AccordionDetails, Tabs/Tab, ToggleButton/ToggleButtonGroup, Modal, Fade, Toolbar, LinearProgress under a header comment (:8-10) claiming “the MUI theme overrides in ThemeNodetool already style them”. ⚠ The theme’s actual override set is larger than the original draft claimed — MuiCssBaseline, MuiTypography, MuiButton, MuiFormLabel, MuiFormControl, MuiPopover, MuiModal, MuiToolbar, MuiDialogTitle, MuiTooltip, MuiDivider in ThemeNodetool.tsx, plus MuiOutlinedInput, MuiSelect, MuiPaper, MuiMenu, MuiMenuItem, MuiSwitch from editorControls.ts — but the core claim holds: there is no MuiTabs, MuiToggleButton, MuiAccordion, or MuiLinearProgress override anywhere in themes/. Those four render stock MUI. Contrast: MuiTooltip (ThemeNodetool.tsx:301-319) gets backdrop blur, a divider border, and a layered shadow — more surface craft than any panel, card, or input receives. Inputs have no default field background: only editor-scoped controls get Paper.overlay, gated on the nt-editor-control marker class (editorControls.ts:19-23, class defined in constants/editorUiClasses.ts:10). SearchInput.tsx:49 gives itself action.hover — so search boxes read as fields and plain text inputs don’t. Paper.overlay exists in both schemes (#17181B dark, #F0EDE6 light), so a default field background is safe in both. 3. Tech design 3.1 Control tokens (CONTROL) Add to ui_primitives/tokens.ts: /** * Control-chrome tokens: the five control heights, the two horizontal * paddings, and the single field radius. Every interactive control in * ui_primitives/ and editor_ui/ derives its box from these. */ export const CONTROL = { /** px, all on the 4px grid */ height: { xs: 24, // toolbar-density buttons (EditorButton compact) sm: 28, // node-canvas controls, EditorButton normal md: 32, // inspector controls lg: 36, // default form controls (TextInput, SelectField, SearchInput) xl: 44, // touch targets (MobileBottomSheet actions, TagInput) }, paddingX: { compact: 8, normal: 12 }, // px, on-grid radius: BORDER_RADIUS.md, // 6px — matches editor controlRadius + buttonLarge } as const; Rationale for five steps rather than the three the draft proposed (28/36/44): the editor’s two densities (28 node, 32 inspector) are real, shipped, and correct for a dense canvas — collapsing them into a 3-step scale would either restyle the whole editor or leave theme.editor as a second unreconciled scale, which is the defect we’re fixing. Five named steps, one source. Radius: BORDER_RADIUS.md (6px) — the value the editor controls and MuiButton already use, so the flip is churn-free for the editor and only moves SearchInput (8→6). (Alternative considered: sm/4px, rejected — it would restyle every editor control and button for no gain.) Wiring: theme.editor.heightNode / heightInspector / controlRadius (ThemeNodetool.tsx:69-79) become derived: `${CONTROL.height.sm}px`, `${CONTROL.height.md}px`, CONTROL.radius. ThemeNodetool.tsx may import from ui_primitives/tokens.ts (tokens has no theme dependency beyond types — no cycle). theme.d.ts needs no change; the shape of theme.editor is unchanged. EditorButton: density === "compact" ? CONTROL.height.xs : CONTROL.height.sm; the size prop maps small → sm, medium → md instead of discarding the height (EditorButton.tsx:65 bug). TextInput / SelectField: compact/small → CONTROL.height.sm, default → CONTROL.height.lg, applied as minHeight on the input root (leave multiline free to grow, as the storyboard CSS does at StoryboardBoard.tsx:135). SearchInput: radius → CONTROL.radius; height → lg (or sm when its compact variant is set). TagInput: 48px → CONTROL.height.xl (44) as minHeight (chips can wrap taller). Off-grid fixes in the same pass: Slider 13px→12px, NodeSlider 3px→4px, Panel comfortable: 2.5→3 (= SPACING.lg) and replace the paddingValue / 2 computed bottom padding with snapSpacing, plus the 2.1 ⚠-table sweep (TabGroup 42→40, Chip 22→24 or pill height token, radius literals → BORDER_RADIUS.*, font literals → FONT_SIZE_SANS.*). 3.2 One label treatment Standardize on label above the control: fontSizeSmall (13px), weight 500, text.secondary, no text-transform, 4px gap to the control. This is what Label.tsx already implements and what the storyboard Field wrapper hand-rolls. Label becomes the single renderer. Add htmlFor support (already has the asterisk); change its baked marginBottom from 2px to 4px (theme.spacing(1)) — the one component allowed to own that gap. TextInput: stop forwarding label to MUI. Render <Label htmlFor={id}> above the field; generate id via React.useId() when the caller passes none, so label/input association survives. Delete the floating-label correction sx (TextInput.tsx:88-101). Add hideLabel for parity with SelectField (renders the label visually hidden but keeps aria-label). SelectField: drop the InputLabel path (SelectField.tsx:123-129); render the shared Label above. hideLabel keeps its contract. FormField: use Label; delete its own Typography block and asterisk duplicate (FormField.tsx:85-108). Theme: remove textTransform: "capitalize" and the 8px padding from MuiFormLabel (ThemeNodetool.tsx:249-262) — labels are no longer MUI-managed; keep the 13px/500 rules for any residual MUI-internal labels. Outlined inputs no longer need the label notch; MUI renders no legend when no label is passed, so nothing to patch. This kills the clipped-label bug class. The overflow: visible workaround in StoryboardBoard.tsx:96-98 and its .field-label CSS go in Fix 4’s migration. Call-site impact: ~56 TextInput label= sites (17 files) and 29 SelectField label= sites keep their code unchanged — the prop’s meaning changes from “floating label” to “label above”, which is the point. Visual diff at every one of those sites is expected and desired. 3.3 Coherent defaults SelectField: default variant="outlined". Remove the 12 redundant explicit variant="standard" props in the same PR (they only restated the default; keeping them would fork the look). The standard variant stays supported but undocumented-discouraged; if post-flip audit finds no legitimate use, delete it in a follow-up. Inputs get a default field background: Paper.overlay on the input root (both TextInput and SelectField, outlined variant), matching the editor controls’ treatment. Change SearchInput from action.hover to Paper.overlay so all fields read identically. Panel: default background="paper" and bordered=true; radius source moves to theme.rounded (aligning with Surface — one radius source for surfaces). Risk: 42 call sites, none pass background. Docked workspace panels (PanelLeft, PanelBottom, WorkspaceShell, PanelToolbar) plausibly want the flush look — audit all 42 during the flip and pin background="default" bordered={false} (or introduce variant="docked") where flush is intentional. Everything else inherits the visible default. FlexRow/FlexColumn/Stack: keep gap=0 (37% of ~1010 usages rely on it). Form spacing is owned by the Fix 4 primitives. EditorButton size fix lands with Fix 1 (same file). 3.4 Form composition primitives FormField rework: owns the label (via Label), the 4px label gap, and a helper/error line with a fixed on-grid margin (theme.spacing(1)), overriding MUI’s 3px 14px 0 helper margin. Provides a FormFieldContext { controlId, labelless: true }; TextInput and SelectField consume it and suppress their own labels + adopt the id — no double labels by construction, no convention to remember. FormSection: optional group label (uppercase, smaller/11px, letterSpacing: "0.08em", text.disabled — the storyboard .group-label treatment, StoryboardBoard.tsx:112-116) above a FlexColumn gap={3} body. FormGrid: n-column CSS grid (gridTemplateColumns from a columns prop, default minmax(0, 1fr) 260px two-column like .header-grid), gap from SPACING, stacking to one column below a stackBelow breakpoint prop. Proof migrations: StoryboardBoard.tsx (delete local Field, the .header-grid/.group-label/.field-label/36px CSS, and the overflow: visible workaround), CollectionForm.tsx (the one existing FormField consumer), and WorkflowForm.tsx. 3.5 Honest type scale Five real sizes, five names: name px change giant 22 gets a real value (was 18; 2 call sites, both want display size) big 18 unchanged normal 15 unchanged small 13 unchanged smaller 11 unchanged Codemod the aliases with ast-grep (see §4 tooling note; pattern <Text size="tiny" $$$REST>$$$C</Text> verified to match): bigger→big (17 sites), tiny→smaller (36), tinyer→smaller (4) — mechanical, zero visual change since the values were already identical. Then narrow TextProps["size"] to the five names and delete fontSizeGiant/fontSizeBigger/fontSizeTiny/fontSizeTinyer from the theme after adding fontSizeGiant: "22px" — grep confirmed nothing consumes the legacy CSS vars directly. Text.tsx’s size map shrinks to five entries. Review the 2 giant call sites for the 18→22 bump. 3.6 Passthrough honesty For each re-export in muiReexports.ts:107-130 that the theme does not style, add the missing override with tokens (preferred over relabeling — the primitives-first policy funnels all usage through these re-exports, so a theme override fixes every consumer at once): MuiTabs/MuiTab: minHeight: CONTROL.height.lg, label 13px/500, 2px indicator (match TabGroup’s look so the two tab systems agree). MuiToggleButton: CONTROL height/radius/padding. MuiAccordion: Paper.paper background, divider border, radius theme.rounded.md, no default elevation shadow. MuiLinearProgress: 4px track, BORDER_RADIUS.pill, primary bar (match ProgressBar). Rewrite the muiReexports.ts header comment to state which exports are themed and which are deliberate raw escape hatches (MuiAutocomplete, MuiDialog at :132-141 already are). Document Box.tsx as an import-policy shim. 4. Implementation plan Codemod tooling: the mechanical rewrites in PRs 3 and 5 (and future large migrations — this repo refactors heavily) use ast-grep, driven by the official agent skill vendored at .claude/skills/ast-grep/ (plus ast-grep-outline for structural surveys). Run it via npx --yes --package @ast-grep/cli ast-grep <args> — it is not a repo dependency. For multi-step, test-gated migrations beyond single-pattern rewrites (e.g. the raw-MUI → primitives sweep STRATEGY.md tracks), evaluate the Codemod CLI + MCP stack (npx codemod ai), which layers AST inspection and codemod tests on top of ast-grep. Six PRs, in order. 1–3 change no call-site code (PR 3 removes 12 redundant props); every existing screen improves without migration. Each PR: cd web && npm run typecheck && npm run lint && npm test, plus npm run lint:design, plus the visual-suite step below. PR 1 — control tokens (mechanical) Add CONTROL to tokens.ts; export from the barrel; unit test alongside tokens.test.ts (all heights on 4px grid, radius is a --rounded-* var). Wire: EditorButton (incl. the size-discards-height fix), theme.editor derivation in ThemeNodetool.tsx, TextInput, SelectField, SearchInput, TagInput. Off-grid sweep: Slider, NodeSlider, Panel padding, plus the ⚠ table in 2.1 (TabGroup, Chip, UndoRedoButtons, MenuItemPrimitive, MobileBottomSheet, WarningBanner, Breadcrumbs, ZoomControls, NotificationBadge). Add a DesignTokens.Control Storybook story (heights/paddings/radius swatches) next to the existing DesignTokens.* stories. Acceptance: an EditorButton (normal), a compact TextInput, a compact SelectField, and a node-canvas select in one FlexRow share one height (28px), one radius, one horizontal padding; grep gates (§5) pass. PR 2 — one label treatment Label: add htmlFor; margin 2px→4px. TextInput: label-above via Label + useId; delete floating-label sx; add hideLabel. SelectField: drop InputLabel; label-above via Label. FormField: render via Label; delete duplicate asterisk logic. Theme: trim MuiFormLabel override. Tests: update TextInput.test.tsx (label association via getByLabelText must still pass — it proves the htmlFor wiring); add the missing SelectField.test.tsx (render, a11y, ref, label association, hideLabel). Acceptance: TextInput, SelectField, and a Field-wrapped model select render pixel-identical labels; no label renders outside the control’s box (Panel overflow: hidden no longer clips anything). PR 3 — coherent defaults SelectField default variant="outlined"; delete the 12 redundant variant="standard" props (SettingsMenu.tsx ×7, LayerRow, GetVariableBody, PropertiesPanel, WorkersPanel, SearchProviderSetupDialog). Default field background Paper.overlay for TextInput/SelectField; switch SearchInput from action.hover; verify both schemes (dark #17181B, light #F0EDE6). Panel defaults background="paper", bordered=true; radius from theme.rounded; audit all 42 call sites and pin the docked panels (PanelLeft, PanelBottom, PanelToolbar, WorkspaceShell, timeline/ sketch inspectors as judged) to the flush look explicitly. Update Panel.test.tsx defaults; visual-QA the 17 SelectField sites that relied on standard. Acceptance: the bare no-props composition from §1 renders as a visible bordered panel containing two identically-styled fields. PR 4 — form composition primitives Rework FormField (context-based label suppression); add FormSection, FormGrid; barrel exports; tests per the STRATEGY.md new-primitive checklist (render + axe + ref for each); Storybook stories. Migrate StoryboardBoard.tsx header — delete local Field, the .header-grid/.group-label/.field-label/minHeight: 36px CSS, and the overflow: visible workaround. Migrate CollectionForm.tsx and WorkflowForm.tsx as proofs. Acceptance: storyboard header is visually unchanged (within visual-suite tolerance) but contains zero local form CSS. PR 5 — honest type scale Theme: fontSizeGiant: "22px"; delete fontSizeBigger/fontSizeTiny/ fontSizeTinyer keys (and their CSS-var emission). Codemod size="bigger|tiny|tinyer" → canonical (59 sites incl. giant review); narrow TextProps["size"]; update Text.test.tsx. Review the 2 giant sites at 22px. PR 6 — passthrough honesty Theme overrides for MuiTabs/MuiTab, MuiToggleButton, MuiAccordion, MuiLinearProgress per §3.6. Rewrite the muiReexports.ts header comment; document Box.tsx as a shim. 5. Verification Per PR: cd web && npm run typecheck && npm run lint && npm test and npm run lint:design. Manual before/after screenshots per PR: storyboard header (/workspace storyboard tab), node inspector, SettingsMenu, CollectionForm — in both color schemes. Grep gates after PR 1 (run in web/src/components/ui_primitives/ and editor_ui/, excluding __tests__/ and *.md): no minHeight: "2[248]px" / height: "2[24]px" control literals no border-?[rR]adius: "?[468](px)?" literals (use BORDER_RADIUS/CONTROL.radius) no 13px/3px/5px/42px/45px box values (exception: the 7px/3px vertical paddings inside TextInput/SelectField/SearchInput — they are derived from the CONTROL heights to make MUI’s content box land exactly on 36/28px, not layout spacing) no font-size: 1[04]px literals After PR 2: grep -rn "MuiInputLabel" web/src/components/ui_primitives/ returns nothing. After PR 4: grep -n "header-grid\|group-label\|field-label" web/src/components/storyboard/StoryboardBoard.tsx returns nothing. 6. Risks and open decisions Panel default flip — resolved in PR 3. The audit found 13 real sites (not 42); 10 docked inspector/transcript sites were pinned to the flush look explicitly, content panels (cost estimate, storyboard header) inherit the new visible default. Rendered radius corrected from the accidental 36px to a true 6px (theme.rounded.md). Label-above changes every labelled field’s geometry (~85 sites gain ~17px of height). Dialogs and dense inspectors may need spot re-layout; the visual suite will surface them. standard-variant removal is deferred until post-flip audit; the variant remains functional in PR 3. TagInput at 44px vs 36px: chosen 44 (xl) to keep the touch-friendly chip row; revisit if it reads oversized next to 36px fields in forms. giant at 22px restyles 2 call sites; trivially reviewable. Line numbers in §2 are pinned to 2cf277b; PRs land sequentially and each shifts the next PR’s targets — re-grep, don’t trust stale lines.--- # Privacy and Data Handling Source: https://docs.nodetool.ai/privacy This page states what NodeTool does with your data, based on the code. It describes behavior, not contractual terms. For the hosted service at app.nodetool.ai, read any terms that service publishes separately. Short version Workflows, assets, chat history, and settings are stored on the machine that runs the NodeTool server. Provider API keys are stored encrypted in the local database. Data leaves the machine when a node calls a provider you configured, when you install or download something, and when you turn on an opt-in feature. The desktop app contains no analytics. Error traces stay in your own database. Where data is stored The NodeTool server keeps its state in a data directory. The desktop app runs this server on 127.0.0.1, so it is reachable only from your machine. What Default location Database (nodetool.sqlite3): workflows, chat threads, memory, jobs, settings, encrypted secrets ~/.local/share/nodetool/ on macOS and Linux, %APPDATA%\nodetool\ on Windows Assets (assets/) Same directory Vector store (vectorstore.db) Same directory Log file ~/.local/share/nodetool/logs/nodetool.log on macOS and Linux, %LOCALAPPDATA%\nodetool\logs\nodetool.log on Windows DB_PATH, DATABASE_URL, ASSET_FOLDER, STORAGE_PATH, and VECTORSTORE_DB_PATH move these. The full list of locations is in Desktop App and Configuration. Other storage backends are in Storage. Desktop users can keep separate data sets in vaults. Each vault has its own database, assets, and collections. See Desktop App. NodeTool does not set a retention limit on your workflows, assets, or chat messages. History such as autosaves, old versions, finished jobs, and run events is trimmed only when you ask, or on a schedule if you set NODETOOL_STORAGE_AUTO_CLEANUP. The windows are listed in Configuration. How secrets are encrypted Provider keys and other secrets are saved in the database as ciphertext. NodeTool encrypts each secret with AES-256-GCM, using a per-user key derived from a master key with PBKDF2-SHA256. The master key comes from the first of these that exists: The SECRETS_MASTER_KEY environment variable. The operating system keychain: macOS Keychain, Windows Credential Manager, or the Linux Secret Service. A new key that NodeTool generates and saves to the keychain. In the Docker image, if SECRETS_MASTER_KEY is unset the entrypoint generates a key and saves it to /workspace/.secrets_master_key on the data volume. A database copied to another machine can be decrypted only where the master key has the same value. Losing the master key means losing the stored secrets, and you re-enter them. Details are in Configuration. What leaves your machine NodeTool sends data only for these reasons. Provider calls When a node or agent uses a cloud model, the server sends the prompt, input media, and the parameters to that provider with your key. This covers chat models, image, video, and audio generation, speech and transcription, embeddings, and web search tools such as SerpAPI. The provider’s own policy then applies. NodeTool does not route these calls through a NodeTool server and does not mark up prices. See Models and Providers. Nodes that take a URL, such as a web fetch or an image URL, contact that URL. The server applies SSRF checks to URLs supplied by callers, providers, or models. See URL Egress Inventory. Downloads you start Installing a node pack, a runtime package, a Python environment, or a model downloads files from their sources, such as GitHub, Hugging Face, or Ollama. Your account data is not part of those requests, apart from a Hugging Face token if you set HF_TOKEN. Desktop updates Automatic updates are off by default. When you turn them on in Settings, the desktop app checks the GitHub releases of nodetool-ai/nodetool. With the setting off, the app makes no update check. See Upgrading. Opt-in diagnostics Feature Default What it sends Error trace sync (NODETOOL_ERROR_TRACE_SYNC_URL and NODETOOL_ERROR_TRACE_SYNC_TOKEN) Off Redacted error traces, to the server URL you name OpenTelemetry export (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, or TRACELOOP_API_KEY) Off Run spans, to the backend you name Bug report dialog Your action A report you submit, which can attach the last hour of redacted server errors Run spans include token counts and cost. They include prompt content only when NODETOOL_TRACE_INCLUDE_CONTENT=1. Analytics The desktop app and self-hosted installs load no analytics. The web app loads the Plausible analytics script only when it is served from app.nodetool.ai. The code skips it on localhost, other hosts, and in the desktop app. The code comment describes it as cookieless. Error traces NodeTool records server failures, failed workflow runs, and client crashes as error traces in its own database. It sends nothing to a third-party error tracker. A trace stores the error type, message, stack, app version, platform, and ids such as job and workflow ids. It does not store prompts, node inputs, outputs, or request bodies. Redaction removes credentials, tokens, email addresses, IP addresses, and home-directory names before a trace is saved. Traces older than 30 days are pruned. Set NODETOOL_ERROR_TRACES=off to stop capture. The whole mechanism is described in Error Tracing. Local models Local models keep your prompts and media on the machine. NodeTool can run them through: Ollama, llama.cpp, vLLM, and LM Studio servers that you run yourself An in-process llama.cpp provider Transformers.js and whisper.cpp in the server Python nodes that use Hugging Face and MLX models, through a local process connected over stdio The model files download once, then run offline. Mixing is allowed in one graph, so a single cloud node sends only its own inputs to its provider. Setup is in Models and Providers and Hugging Face. Local, self-hosted, and hosted mode The server picks its mode from its environment. Both SUPABASE_URL and SUPABASE_KEY must be set for Supabase mode.   Local mode Supabase mode Login None. Loopback requests, plus networks you list in NODETOOL_TRUST_LOCAL_NETWORKS, run as one user A valid Supabase JWT is required on every request Who sees the data Whoever can reach the server as that user Each signed-in user, scoped to their own records Storage Local disk by default NODETOOL_STORAGE_BACKEND selects file, s3, or supabase Workspaces Folders on disk Object storage under workspaces/<user>/ by default when NODETOOL_ENV=production A self-hosted Docker server from docker-compose.yml runs in Local mode with no login. Do not expose it to the internet as it is. See Self-Hosted Deployment, Authentication, and Security Hardening. On NodeTool’s hosted cloud, a generation can run on a curated model catalog with NodeTool’s own provider keys and a credit balance. That provider is cloud-only. See Costs and Credits. Export and erase your data A server holds an export and an erase endpoint for the signed-in identity, even in Local mode: Endpoint Effect GET /api/account/export Downloads everything the server holds about you as JSON DELETE /api/account Erases your data. The body must be {"confirm": "DELETE MY ACCOUNT"} See API Reference. Workflows and apps also export as bundles you can keep. Related pages Configuration for storage, secrets, and retention variables Error Tracing for what is recorded and how to turn it off Security Hardening for deployments--- # Plan: MCP over HTTP and the Python bridge on a production server Source: https://docs.nodetool.ai/production-mcp-python-flags-plan.html Plan: MCP over HTTP and the Python bridge on a production server Status: implemented — 2026-08-22. Problem The Docker image (ghcr.io/nodetool-ai/nodetool), fly.toml, and docker-compose.yml set NODETOOL_ENV=production. Two surfaces turn off in that mode: The /mcp streamable-HTTP mount (packages/websocket/src/server.ts, the if (!isProduction) block around the app.all("/mcp", …) route). An MCP-aware agent cannot reach a self-hosted server over HTTP. The Python bridge (packages/runtime/src/python-stdio-bridge.ts, _assertCanConnect and hasPython). Python nodes are unavailable. A self-hoster asked for an opt-in flag for each, modeled on NODETOOL_ENABLE_EXTENSION_BRIDGE=1 (packages/websocket/src/plugins/websocket.ts). Their setup: one shared instance behind a VPN, humans on the web UI, an agent on MCP with a bearer token. Findings The Python flag already exists. NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 lifts the production refusal in python-stdio-bridge.ts and is documented in docs/configuration.md. Two things are missing around it: The boot log in server.ts says “Python bridge disabled in production” and does not name the flag, so an operator cannot find it from the log. The Docker image ships no Python worker (nodetool-core). The flag alone does not make Python nodes work in the image. The MCP mount cannot be a bare flag flip. The mount binds every session to agentToolsScope: { userId: "1", source: "local-dev-http" }. In production that is an anonymous, full-access surface. Two more facts: McpServerOptions.agentToolsScope.source already has an unused "http-session" member (packages/websocket/src/mcp-server.ts), reserved for exactly this: a mount that binds the authenticated user. The auth hook’s static-asset exemption in server.ts skips auth for any GET outside /api, /ws, /v1, /trpc when a static app is served. GET /mcp is the SSE stream. It would bypass auth. Decision Add NODETOOL_ENABLE_MCP=1. The mount registers in production only with the flag, and always binds the user the auth hook resolved — never "1". Add no Python flag. Fix the log line and the docs so the existing flag is discoverable, and document the worker-image requirement. Steps 1. NODETOOL_ENABLE_MCP — packages/websocket/src/server.ts Define mcpHttpEnabled = !isProduction || process.env["NODETOOL_ENABLE_MCP"] === "1", next to the other production gates. Register the /mcp route when mcpHttpEnabled. When disabled in production, log one line that names the flag, the same way the extension bridge does. Build the scope per request: req.userId ? { userId: req.userId, source: isProduction ? "http-session" : "local-dev-http" } : undefined. A missing scope already returns the 401 scopeRefusal from mcp-server.ts at initialize. Add /mcp to the prefix list the static-asset exemption in the auth hook does not cover. 2. Bind a session to its owner — packages/websocket/src/mcp-server.ts Keep sessionOwners: Map<sessionId, userId> next to sessionTransports. Record the owner in onsessioninitialized; delete it wherever the transport is evicted. On POST, GET, and DELETE with a session id: when the recorded owner differs from the request’s scope user, answer 404 Session not found. A 404 rather than a 403, so the response does not confirm the id exists. Tests in packages/websocket/tests/mcp-server.test.ts: the same user resumes a session; a different user gets 404; initialize with no scope gets 401 (already covered — keep it). 3. Python bridge diagnostic — packages/websocket/src/server.ts Replace the production branch of the “Python nodes will not be available” log with two cases: flag unset: name NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1; flag set and no interpreter: “Python not found”, the same line the non-production branch prints. No change to python-stdio-bridge.ts. 4. Docs docs/configuration.md: a table row for NODETOOL_ENABLE_MCP, mirroring the NODETOOL_ENABLE_EXTENSION_BRIDGE row. State that the mount inherits the server’s auth mode and binds the authenticated user. docs/self-hosted-deployment.md: a section “MCP over HTTP and Python nodes” covering: both flags; how an agent authenticates — a Supabase bearer token when SUPABASE_URL is set; NODETOOL_TRUST_LOCAL_NETWORKS=<vpn cidr> in local mode; or a delegated token minted through NODETOOL_INTEGRATION_TOKEN; that the image ships no Python worker: derive an image that installs nodetool-core, and set NODETOOL_PYTHON to that interpreter. docker-compose.yml: commented examples of both flags in the environment block. 5. Verification npm run test --workspace=packages/websocket, npm run typecheck, npm run lint. Prove the gate can fail. Boot the image with NODETOOL_ENV=production NODETOOL_ENABLE_MCP=1 and: POST /mcp initialize without a token → 401; with a token → a mcp-session-id; GET /mcp with that session id and no token → 401. This is the check for the static-asset exemption fix in step 1.4; POST /mcp with the session id and a second user’s token → 404. Boot without the flag and confirm /mcp answers 404 and the log names the flag. Out of scope The tRPC mcpConfig router stays disabled in production. It edits MCP client config files on the server’s filesystem, which has no meaning on a shared host. /ws/download and the other local-only websocket routes keep their production gates. Size About 80 lines of code, 40 of tests, 60 of docs. One PR.--- # Project resource and entry-point inventory Source: https://docs.nodetool.ai/project-resource-inventory.html Project resource and entry-point inventory This inventory records the project boundary found in source inspection for T1. The original findings are retained below; the final verification result is at Final verification. Scan evidence The inventory was produced from the repository with these checks: set -euo pipefail project_tables=$(for f in packages/models/src/schema/*.ts; do if rg -q 'project_id' "$f"; then printf '%s\n' "$f"; fi done) test -n "$project_tables" printf '%s\n' "$project_tables" surface_files=$(rg -l 'projectId|project_id|useProjects|trpcClient\\.(project|asset|entity|workflow|thread|workspace)' \ web/src electron/src mobile/src packages/agents/src packages/websocket/src || true) test -n "$surface_files" printf '%s\n' "$surface_files" The first scan returned these 12 project-bearing tables: applications, assets, image_documents, jobs, js_scripts, predictions, scripts, storyboards, threads, timeline_sequences, workflows, and workspaces. The schema directory also contains dependent tables without their own project column, including messages, run_events, run_inbox_messages, workflow_versions, and workflow sharing tables. Those rows inherit ownership through their parent thread, job, or workflow and must be covered by the parent lifecycle. Historical T1 resource inventory Resource Storage and current ownership References and creation paths List/search paths Delete behavior Planned project boundary and migration Storyboards storyboards.project_id in packages/models/src/schema/storyboards.ts; Storyboard model JSON shot refs include asset refs. Create/update via packages/websocket/src/trpc/routers/storyboards.ts, REST timeline/example helpers, and agent capabilities/storyboards.ts Storyboard.find, router procedures, listProjectDocuments and summarizeProject Model delete and router delete Project-owned. Keep existing IDs, assign null/legacy rows to Personal. Reads and writes must require the owning project. Copy must remap referenced assets. Scripts scripts.project_id; Script model Script sections and takes reference media and entities. Create/update via routers/scripts.ts and agent capabilities/scripts.ts Script router/model and listProjectDocuments Model/router delete Project-owned. Migrate unassigned rows to Personal. Moving requires checking storyboard/timeline links and preserving or rewriting referenced dependencies. Timelines timeline_sequences.project_id; TimelineSequence and version tables Timeline document contains clip asset refs, script links, and generated outputs. Create/update via routers/timeline.ts, REST routes/timelines.ts, and agent capabilities/timelines.ts Timeline router/model and listProjectDocuments Model/router delete; versions follow the sequence’s document lifecycle Project-owned. Migrate the sequence and versions together. Copy must include referenced assets and linked project documents or return an unsupported-dependency error. Image documents / sketches image_documents.project_id; ImageDocument model Canvas document and versions reference assets. Create/update via routers/sketch.ts and agent capabilities/sketches.ts Sketch router/model and listProjectDocuments Model/router delete; versions follow the document Project-owned. Migrate document plus versions. Copy needs asset remapping. Applications / mini-app documents applications.project_id; application document may reference workflows/assets. routers/applications.ts, routes/applications.ts, web/src/hooks/useApplications.ts, mobile services/api.ts and useApplications.ts, agent capabilities/apps.ts Application router/REST and mobile listApplications(projectId?) Application service/model delete Project-owned under D3. Migrate existing application rows. Mobile currently supports an optional project filter but defaults to default; pass explicit project context on every creation and list. JavaScript scripts js_scripts.project_id; JsScript model and versions Create/update via routers/js-scripts.ts, REST routes/js-scripts.ts, web sync hook, and agent capabilities/js-scripts.ts Router/model and agent capability Router/model delete; versions are document-owned Project-owned. Migrate rows and versions. Agent and web calls need an originating project rather than a default fallback. Entities Entity marker stored in an asset’s metadata; entity ownership is assets.project_id. Entity creation/update in agent capabilities/entities.ts and packages/models/src/entity.ts; mention picker useEntities.ts, entity capability; entity search currently has a shared-library path Asset delete; project move uses moveDocumentToProject(..., "entity", ...) Project-owned per D3. Migrate marked assets only. Search, mentions, and agent queries must filter by project. Do not treat unmarked assets as entities. General assets and folders nodetool_assets; parent_id forms the asset folder tree. project_id defaults to default, but current code documents that only entities are shown per project. Upload/finalize through asset tRPC, REST routes/assets.ts, web AssetStore, workflow outputs and agent media/binary-output tools. Folder operations use asset parent IDs. Asset router list, recursive, search, web AssetStore and mobile asset screens Asset delete removes the asset row and stored file according to asset service behavior Project-owned under D3, including folders. Migrate every asset and preserve parent_id relationships. Define whether generated assets inherit the run’s project and make every asset list/search project-scoped. Workflows nodetool_workflows.project_id owns saved workflows; workflow versions/shares/collaborators inherit through workflow_id and have no project column. Web workflow manager/editor, mobile graph editor and workflow screens, Electron api.ts/preload/tray, REST routes/workflows.ts, tRPC workflows router, agent capabilities/workflows.ts, imports/templates Workflow model/router, REST examples/public/tools/name endpoints, mobile and Electron workflow lists Workflow.deleteOwned removes the workflow and access grants; workspace deletion refuses linked workflows Project-owned under D3, while reusable templates and examples remain global under D5. Migrate default workflows to Personal and preserve parent ownership for versions and sharing rows. Public/collaborative access must not bypass the owning project boundary. Threads and messages nodetool_threads.project_id owns thread containers; nodetool_messages.thread_id inherits that ownership. A project row still has one optional thread_id for its legacy agent conversation. Thread/message models and packages/websocket/src/trpc/routers/threads.ts; web GlobalChatStore, chat hooks/components; mobile ChatStore, ThreadsScreen; project ensureThread creates one agent thread Thread.paginate filters user and optional workflow but not project; message list is thread-scoped Thread router deletes messages and thread; agent tools are read-only for thread deletion Chats are project-owned. Migrate legacy projects.thread_id associations before claiming default threads, then add project-aware list and operation checks for multiple threads per project. Runs and background jobs nodetool_jobs.project_id owns the execution record; run_events and inbox rows inherit through run_id. Job.create in packages/websocket/src/headless-job-runner.ts and session/job-execution.ts, workflow runner, job control routes, agent capabilities/jobs.ts, web WorkflowRunsStore/LiveRunStore, mobile Jobs screens Job model/router and live WebSocket events; prediction ledger queries Job cancellation stops execution, but Project.deleteOwned does not currently cancel or reassign jobs, events, or inbox rows Project-owned execution context. Keep the project ID through creation, events, retries, and activity routing. Deletion must wait for or cancel active runs and prevent retries from writing before removing or retaining their history. Predictions / generated outputs predictions.project_id and document_id are nullable; asset_ids records generated asset IDs. Outputs are normally asset rows plus chat/message output refs. packages/execution/src/generation-tracker.ts, cost ledger, provider generation paths, agent media and binary-output tools, chat turn Prediction model/router/cost ledger and asset list; project summary only includes spend, not output resources Prediction rows are history. Project delete currently leaves rows with the deleted project ID; assets are reassigned by the broad asset update Project-owned when created by a project run or document. Migrate nullable legacy rows by thread/job/document where unambiguous, otherwise Personal. Keep output asset ownership and prediction ownership consistent; never leave a completed output pointing only at a deleted project. Workspace files and folders nodetool_workspaces.project_id owns the workspace row; files live behind runtime Workspace (local directory or cloud object prefix), not in project tables. Web useWorkspaces, workspace components, tRPC routers/workspace.ts, REST download, agent capabilities/files.ts, CodeAct workspace.*, workflow workspace service, mobile app-runtime consumers Workspace.paginate currently filters only by user; listFiles/readFile operate through the selected workspace; agent workspace.list Workspace row delete is blocked for default and linked workflows; file delete is available through workspace APIs Project-owned files must be explicitly associated with a project. Preserve local/cloud parity through Workspace.read/write/list/stat/copy/move/delete, add project-aware workspace listing and cleanup, and do not use localDir as the ownership mechanism. Migrate existing default workspaces and their files to Personal. Global resources These remain outside project ownership under D5: account settings and personal-data tables (settings, credentials, secrets, OAuth tokens), installed model/provider metadata, and reusable workflow/template examples. A template import creates a new project-owned workflow/document. Dependent rows such as messages, run events, and workflow versions are not global just because they lack project_id; they inherit the parent resource’s project. Cross-project references and move behavior Q10 is partially resolved by the current migration. Project.migrateToPersonal claims default or null rows in all 12 project-bearing resource tables, restores legacy project-thread links, and assigns jobs from already-owned workflows before claiming remaining rows. It reports dangling non-default project ids without moving them. Document JSON and asset metadata can still name assets, workflows, timelines, scripts, or entities without a database-level project foreign key, so ambiguous cross-project references need an explicit repair result. Do not create a shared project as an accidental result. Q11 is resolved as a copy boundary even though the implementation is still absent. A project-document copy must recursively include referenced assets and entity assets, plus linked project documents where the source format names one: storyboard shots, script media and entities, timeline clips and linked scripts, sketch canvas assets, application resources and workflows, JavaScript script dependencies, and workflow graph resources. Repeated references reuse one destination identity, all internal ids are rewritten before commit, and unsupported or missing dependencies fail the operation without a partial copy. Threads, jobs, predictions, and workspace files are not implicit document dependencies; copying a workspace file is an explicit Workspace.copy operation with the destination project’s workspace. Q13 has a concrete current entry point: packages/models/src/project-membership.ts and packages/websocket/src/trpc/routers/projects.ts allow moving storyboard, script, timeline, sketch, application, JavaScript script, and entity resources. The move changes ownership without updating references and does not check dependents. Workflows, jobs, threads, and workspaces have project columns but are not in that move union. Future moves need either dependency-aware rewrites or a refusal with the affected references; a move must not silently break another project’s document. Historical T1 gaps outside the project summary union ProjectDocumentType covers storyboard, script, timeline, sketch, application, and JavaScript script. It does not cover workflows, folders/general assets, threads/messages, workspace files, jobs/runs, or generated output assets. Project.deleteOwned reassigns only the seven membership tables in project-membership.ts to default, including assets. It does not archive, cancel active work, delete workflows, threads, or workspace files, or reconcile prediction/output ownership. Project also has one thread_id, which cannot represent D9’s multiple independent conversations. Electron had workflow-only consumers and no project selector or project argument in its workflow fetch/tray paths. Mobile had an application project filter and project-shaped app documents, but its document backends and most screens used default or unscoped lists. These clients needed the same project-scoped API contracts as web rather than separate ownership rules. Final verification The scan’s 12 direct ownership tables are all present in both Project.migrateToPersonal and Project.deleteOwned: applications, assets, image documents, jobs, JavaScript scripts, predictions, scripts, storyboards, threads, timeline sequences, workflows, and workspaces. Messages, run events, run inbox rows, workflow versions, and sharing rows inherit their parent’s ownership and lifecycle. The scoped entry points are web project selection and resource queries, tRPC resource/document routers, agent capability runs, job and chat execution, Electron’s shared workspace shell, and mobile document transports. Mobile has no project selector in this release, but documentBackend(kind, projectId) forwards its supplied scope through every list and creation request. Unscoped callers retain their existing behavior. mobile/src/documents/__tests__/backends.test.ts verifies every document kind’s scoped list and creation path. No additional project-owned type was omitted from this final scan. Existing mobile callers without project navigation retain their unscoped legacy path; they can pass ownership through the documented transport contract without a mobile navigation redesign. Global settings, credentials, installed models, and reusable templates remain deliberately outside the project boundary.--- # Project scoping PRD Source: https://docs.nodetool.ai/project-scoping/prd.html Project scoping PRD Status: agreed product scope, implementation pending. Companion: Implementation tasks. Problem A creator working on multiple deliverables needs to know which work belongs together and resume each deliverable without rebuilding their working context. NodeTool groups project documents, but its shared tab list and inconsistently scoped resource surfaces allow unrelated work to appear together. A project selector must control resource ownership and execution context as well as navigation. Product outcome One project represents one deliverable, such as a film, campaign, or app. Selecting it brings back its working session and makes its resources the default context for browsing, creation, and agent work. Personal provides the same capabilities for experiments and work without a named deliverable. Agreed decisions The decision IDs preserve the product discussion’s references. ID Decision Required behavior D1 Project selection A persistent selector sits on the tab bar, left of New. It supports finding, switching, creating, and managing projects. Lists, search, pickers, and mentions use the selected project. D2 Project session Each project remembers its open tabs, order, active document, and selected chat. Switching preserves drafts. First opening shows the overview. Closing every tab leaves the project selected. D3 Resource ownership Documents, workflows, assets, entities, project files, chats, and generated outputs belong to a project. D4 Creation and execution New work inherits its originating project. Running generations and agents continue there after a switch. The tab bar shows background activity. D5 Global resources Account settings, credentials, installed models, and reusable templates remain global. Creating from a template produces project-owned work. D6 Personal and migration Personal is permanent and a project is always selected. A one-time migration assigns only unassigned resources to Personal. Existing project assignments remain intact. D7 Project meaning A named project represents one deliverable. Nested projects and client hierarchies are outside this release. D8 Independent reuse Copying into another project creates independent copies and includes referenced assets and entities. Edits never propagate between projects. D9 Multiple conversations Each project supports multiple chat threads. New threads receive project context and can work with its resources. D10 Lifecycle Finished projects can be archived. Deleting a project deletes its contents after confirmation. Personal cannot be deleted. Experience [ Project name v ] [ Account ] ----------------------------------------------------------- Project navigation | This project's open document tabs | Active document | Project chat / inspector Switching and resuming The project name remains visible above the working area. Switching restores the destination’s session rather than opening every document it owns. Returning to a project restores its drafts and focus. Reopening the application restores the selected project and its saved session, with Personal as the fallback. Tabs from other projects do not appear in the selected project’s tab bar. Opening a link to a document in another project switches to its owning project before displaying it. A failed or superseded switch must not leave the header, tabs, and resource panels showing different projects. Creating and finding work Creating documents, importing files, uploading media, generating outputs, and starting chats use the originating project’s identity. Asset browsing, entity selection, mentions, resource search, and agent resource tools use that same scope. Scope must be enforced through resource operations, not just hidden rows in the interface or instructions in an agent prompt. Each supported resource type needs an explicit ownership path, including resource types missing from the existing project overview. Project files must use the workspace interface on both local and cloud storage. Background work and conversations Switching projects does not cancel work. An upload, generation, or agent turn started in A continues to read and write A even while B is selected. Activity indicators identify the owning project and provide a route back to its work. Projects have multiple independent conversation histories. Project context does not mean inserting another thread’s full conversation into a new thread. Selecting a thread cannot silently redirect its resource operations to the project most recently selected elsewhere. Copying between projects An explicit copy action selects a destination project. The copy includes the referenced assets and entities needed to use the document independently. References in the copied content point to the destination copies. Multiple references to the same dependency within one copy operation reuse its copied identity. Source resources remain unchanged. The destination must remain usable after the source project is deleted. Missing dependencies or a failed copy must produce a clear failure rather than report a complete copy with broken references. Global prerequisites, such as installed models and credentials, remain global. Migration and lifecycle On the first startup using this feature, create or resolve Personal for the resource owner and assign unassigned resources to it. Preserve IDs, content, references, and existing project assignments. Repeated startup or an interrupted migration must not duplicate Personal or move subsequently assigned resources. Existing tabs and conversation history remain reachable. Archive retains project contents and supports restoring the project. Archived projects remain discoverable through project management without crowding the normal selector. Delete confirmation identifies the project and explains that its contents will be removed. Deletion must not remove another project’s copies or permit background operations to recreate deleted contents. Release acceptance criteria ID Scenario Pass condition AC1 Switch A to B and back A restores its tab order, active document, selected chat, and unsaved draft. Only A’s tabs appear. AC2 First open and close all tabs A new project opens its overview. Closing all tabs leaves its project context active. AC3 Browse and select resources Lists, search, pickers, mentions, and agent queries return the selected project’s resources plus explicitly global resource types. AC4 Create through different entry points UI, API, and agent creation assign the intended project consistently. Invalid project ownership is rejected. AC5 Switch during work A generation and agent turn started in A finish in A after switching to B. An upload started in B belongs to B. Activity links return to the correct project. AC6 Multiple chats A can create and reopen multiple threads. Switching to B shows B’s threads and uses B’s resources. Returning to A restores its selected thread. AC7 Copy with dependencies Copy a document with repeated asset and entity references into B, edit the copies, and delete A. B still works and its internal references resolve. AC8 Migrate existing content Assigned resources keep their projects. Unassigned resources enter Personal without losing content or references. Rerunning migration produces no additional changes. AC9 Archive and delete Archive preserves contents and can be reversed. Confirmed deletion removes the project’s contents without affecting independent copies. Personal cannot be deleted. AC10 Interrupted navigation and restart Rapid switches, failed loads, direct document links, and application restart keep the selector, session, and resource scope consistent. AC11 Complete coverage A resource and entry-point inventory has no unhandled project-owned type or unscoped creation path. Local and cloud workspace behavior follow the same ownership rules. Scope boundaries ID Outside this release N1 Team membership, project sharing permissions, and collaboration roles. N2 Nested projects, client hierarchies, and synchronized shared asset or entity libraries. N3 A redesign of individual document editors or migration of global credentials and model installations into projects. Implementation questions These details need resolution during the tasks below. They do not change the agreed ownership model. ID Question Resolve in Q10 How should legacy references spanning assigned projects be preserved without breaking documents or introducing new shared ownership? T1, T2 Q11 Which linked document dependencies must also be copied, and how should unsupported external dependencies be reported? T1, T8 Q12 Should deletion wait for running work or explicitly cancel it, and how are retries prevented from writing afterward? T9 Q13 What existing move actions remain valid when moving a resource could break another document’s references? T1, T8 Starting points in the codebase These findings came from source inspection, not a live visual walkthrough. ID Finding Source F1 Project opening adds all project documents to a shared tab list and selects the overview. WorkspaceTabsStore, useProjects F2 The entity library query searches assets without a project filter, while the overview has a project-specific entity section. useEntities, ProjectEntitiesSection F3 Assets carry a project field, but the inspected upload payload does not carry project context. AssetStore, asset schema F4 The project document summary union omits workflows, and project metadata has one agent-thread pointer. project-summary, project schema--- # Project scoping tasks Source: https://docs.nodetool.ai/project-scoping/tasks.html Project scoping tasks Companion: PRD. All tasks are pending. Task IDs identify dependencies, not separate agent assignments. Delivery order Phase Tasks Outcome A1 T1–T3 Complete ownership, Personal migration, and operations bound to their originating project. A2 T4–T7 Project selection, isolated sessions, scoped resource surfaces, and multiple chats. A3 T8–T10 Independent copying, archive and deletion, and release verification. T1 — Inventory ownership and resource entry points Enumerate document types, workflows, assets and folders, entities, workspace files, threads, runs, and outputs. Record storage, ownership, references, creation paths, list/search paths, and deletion behavior. Cover web, Electron renderer, API, agent tools, background jobs, and existing mobile consumers of affected contracts. Classify each resource as project-owned or explicitly global using D3 and D5. Identify gaps beyond the existing project summary union. Resolve legacy cross-project references and dependency-copy coverage (Q10, Q11), and identify existing move behavior needing constraints (Q13). Depends on: none. Acceptance: a concrete file and entry-point inventory supports AC11. Every project-owned type has a planned owner, migration path, and resource boundary. T2 — Complete project ownership and Personal migration Add missing ownership fields and protocol contracts for the T1 inventory. Keep backend schemas and supported database variants consistent. Create or resolve one permanent Personal space per resource owner. Implement an idempotent, restartable migration of unassigned resources. Preserve existing assignments, IDs, content, and references. Preserve access to existing tabs and threads after migration. Handle malformed or dangling legacy membership explicitly without silently moving valid assigned resources. Add scoped model operations and project ownership validation. Preserve compatibility of existing resource operations during rollout. Depends on: T1. Acceptance: AC8 and ownership portions of AC4/AC11. Verify empty accounts, mixed assigned/unassigned content, repeat startup, and interrupted migration. T3 — Bind creation and execution to the originating project Carry project identity through document creation, imports, uploads, generation requests, runs, and agent sessions. Capture scope when work starts. Persist enough context for delayed completions and retries to use the original project. Scope agent resource operations through backend contracts rather than relying on prompts or the foreground tab. Apply project file ownership through the workspace interface on local and virtual storage. Keep global prerequisites outside project storage. Expose project-attributed activity for the selector and navigation back to running work. Depends on: T2. Acceptance: AC4, AC5, and AC11. Start work in A, switch to B before completion, and verify writes and resource reads still belong to A. T4 — Persist separate project sessions Store each project’s open tabs, order, active document, and selected chat independently from its full resource list. Switch sessions without discarding drafts or reopening every document. Define first-open overview, empty-tab state, restart restoration, and Personal fallback. Reconcile deleted or unavailable documents without changing another project’s saved session. Migrate existing persisted tab state. Make rapid switches and failed loads leave one consistent active scope. Depends on: T2. Acceptance: AC1, AC2, and session portions of AC10. Verify switching with dirty editors and restart with multiple saved project sessions. T5 — Put project selection above the tabs Replace the tab-group scope control with a persistent top-level selector. Add project search, creation, management access, and Personal. Render only the selected project’s tabs and project navigation. Show activity from T3 and link it to the owning project’s work. Resolve direct document links to their owning project before opening. Cover loading, failure, empty project, keyboard navigation, narrow layouts, and Electron window controls using existing UI primitives. Depends on: T3, T4. Acceptance: D1, AC2, AC5, and AC10. Project identity remains visible and agrees with the tabs and content during every navigation transition. T6 — Scope resource browsing and selection Apply project scope to every T1 list, search, folder browser, asset picker, entity picker, mention source, and project overview resource type. Include project identity in query/cache and selection state where required. Clear or restore panel state when switching projects. Keep explicitly global resources available with clear scope. Ensure entity membership and its backing/reference assets follow the agreed ownership and migration rules. Depends on: T2, T3, T5. Acceptance: AC3 and AC11. Fixtures with identically named resources in A and B prove that browsing, selection, and mentions resolve the intended resource. T7 — Support multiple chat threads per project Replace the single-thread assumption with project-owned thread listing, creation, selection, and history loading. Preserve existing project conversations through migration. Supply project context to new threads without merging thread histories. Restore the selected chat through T4 and retain the thread’s project identity through background agent turns. Depends on: T3, T4, T5. Acceptance: AC6 and chat portions of AC1/AC5. Two threads in A and one in B retain their histories, selected state, and correct resource access. T8 — Copy documents with independent dependencies Implement destination selection and dependency discovery from the T1 reference inventory. Resolve Q11 and Q13 before finalizing behavior. Copy required assets and entities, assign destination ownership, and remap references. Handle repeated references and cycles without duplicating the same dependency within one operation. Preserve independent lifetimes even if immutable storage bytes are internally deduplicated. Source deletion must not remove copied media. Define failure and retry behavior so incomplete copies are not presented as successful. Report unavailable or unsupported dependencies clearly. Verify dependency traversal on a large resource graph. Depends on: T3, T6. Acceptance: AC7. Edit destination copies and delete the source project, then open and use the destination document with its assets and entities intact. T9 — Archive and delete projects Add archive and restore actions plus archived-project discovery. Add confirmation naming the project and explaining content deletion. Replace the existing return-to-unassigned deletion behavior with the agreed content deletion behavior for the complete T1 inventory. Protect Personal from deletion through both UI and backend operations. Resolve Q12 and prevent active jobs, delayed responses, and retries from writing into a deleted project. Reconcile selected project, sessions, and caches after deletion. Preserve independent copies and global resources. Depends on: T5, T7, T8. Acceptance: AC9 and deletion portion of AC7. Include deletion with active work, repeated deletion requests, and attempts to delete Personal. T10 — Verify the complete project experience Map every PRD acceptance criterion to a deterministic check or an explicit UI walkthrough. Extend the existing relevant verification surfaces and registry where needed. Run the A/B scenario: restore A’s tabs and chat, start work in A, switch to B and upload, return to A, and verify session and output ownership. Verify migration, dependency copying followed by source deletion, archive/restore, direct links, rapid switching, and failed requests. Check the final resource inventory for omitted types and entry points. Perform web and Electron UI checks, including keyboard operation and narrow layouts. Verify affected mobile contracts retain ownership behavior without expanding this into a mobile navigation redesign. Depends on: T6, T7, T8, T9. Acceptance: AC1–AC11 have recorded evidence. Unresolved implementation questions are closed or explicitly returned for product decision before release. Verification for implementation changes After each code change, run the repository’s required checks: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main Use meaningful regression fixtures for the boundaries each task changes. Prove any new validator can fail with a deliberately invalid fixture, and verify inventories contain actual entries. Follow the repository’s additional requirements when a change crosses a dependency seam its affected-test selection cannot detect.--- # Project scoping release verification Source: https://docs.nodetool.ai/project-scoping/verification.html Project scoping release verification This record maps each release acceptance criterion to the regression check that exercises its boundary. Run the named test files with npm run test:affected; the repository gate selects the affected workspaces and their dependent apps. Acceptance criterion Deterministic evidence UI walkthrough evidence AC1: A → B → A restores the session web/src/stores/__tests__/WorkspaceTabsStore.test.ts restores independent tab orders and selected chats. Switch Aurora → Beacon → Aurora with a dirty script. Confirm Aurora’s tab order, active document, draft, and selected chat return, with no Beacon tab visible. AC2: first open and empty state WorkspaceTabsStore.test.ts covers first overview creation and closing the last tab while retaining project scope. Create a project, confirm its overview opens, close every tab, and confirm the selector still names it. AC3: scoped browsing packages/websocket/tests/trpc-assets.test.ts, packages/agents/tests/project-scope.test.ts, and web/src/components/workspace/__tests__/OpenMenuProjects.test.tsx cover scoped assets, agent defaults, and project-aware opening. Give A and B identically named assets and entities. Check search, picker, mention, and agent resource results in each scope. AC4: scoped creation packages/agents/tests/project-scope.test.ts, packages/websocket/tests/trpc-workspace.test.ts, and packages/websocket/tests/chat-project-spend.test.ts cover agent, workspace, and chat project attribution. Create from the UI and API in A. Attempt creation against an unavailable project and confirm it fails. AC5: switching while work runs WorkspaceTabsStore.test.ts runs the A/B round trip with A’s selected chat/output and B’s upload; packages/websocket/tests/chat-project-spend.test.ts attributes an asynchronous A chat turn; packages/models/tests/project.test.ts keeps A’s completed output separate from B’s upload and rejects delayed writes after deletion. Start A’s generation and chat, switch to B, upload an asset, return to A, and confirm A owns its outputs while B owns the upload. Use each activity link to return to its project. AC6: independent chats packages/websocket/tests/trpc-projects.test.ts and WorkspaceTabsStore.test.ts cover project thread creation and selected-chat restoration. Create two A threads and one B thread. Switch between projects and reopen each thread without cross-project history or resource results. AC7: independent copy packages/websocket/tests/project-document-copy.test.ts covers duplicate references, remapping, cycles, source deletion, missing dependencies, and large graphs. Copy a document from A to B, edit the B copy, delete A, then open B and confirm its assets and entities still resolve. AC8: Personal migration packages/models/tests/project.test.ts covers mixed ownership, dangling legacy rows, and repeatable migration. Upgrade an account with assigned and unassigned content. Restart twice and confirm one Personal project, preserved assignments, and preserved content. AC9: archive and deletion packages/websocket/tests/trpc-projects.test.ts and packages/models/tests/project.test.ts cover archive/restore, idempotent deletion, Personal protection, owned-resource removal, and write tombstones. Archive a project, find it in management, restore it, then confirm deletion after its named confirmation leaves an independent B copy intact. AC10: interrupted navigation and restart web/src/hooks/__tests__/useProjects.test.tsx rejects stale and failed project loads; WorkspaceTabsStore.test.ts covers persisted-session migration and unavailable documents. Rapidly select A then B, open a direct A document link, and restart. Confirm selector, tabs, and resource scope agree after each transition. AC11: inventory completeness docs/project-resource-inventory.md records resource ownership and entry points. packages/models/tests/project.test.ts deletes the owned document, asset, workflow, thread, job, workspace, and prediction inventory. Repeat the A/B resource scan in web and Electron. On mobile, use existing document backends with distinct project ids and confirm requests retain their supplied ownership. UI release pass Run the walkthroughs above in both web and Electron. At a narrow viewport, open the selector with Enter, filter with the keyboard, select with Enter, and confirm the selected project remains visible left of New on the tab bar. The deterministic keyboard check is web/src/components/projects/__tests__/ProjectSelector.test.tsx. Resolved implementation questions Q10 is returned for product decision: legacy cross-project references are preserved during migration and dangling membership is reported rather than reassigned. The release needs an explicit repair policy before promising automatic correction of those legacy references. Q11: copy traversal remaps supported asset and linked-document dependencies; unavailable or unsupported dependencies fail before a visible copy exists. Q12: deletion removes owned resources and leaves a tombstone that rejects delayed or retried writes. Q13: supported moves are explicit document membership changes. Copy is the dependency-safe cross-project reuse path.--- # Projects Source: https://docs.nodetool.ai/projects A project is a name over the documents that belong to one job: a storyboard, a script, a cut, key art, a mini app. It owns no content of its own. A document belongs to a project because it carries that project’s id, and the project card reads status, rendered stills, and spend back from those documents. Projects are not workspaces. A workspace is a folder of files a run reads and writes. A project groups the documents you work on. The Personal project Every account has a Personal project. It is where documents land when you have not switched to a named project. You cannot archive or delete it, and it does not appear as a card in the project list. The project list Open Projects in the left rail to see every named project as a card, with a Search projects field and a + New project button. Each card shows: Up to three rendered stills from the project’s documents, or “No stills yet”. A pill over the media, such as clips 3/8 for a storyboard or cut · 0:30 for a timeline. The project name and when it last changed. A status line, such as 8 shots · stills 8/8 · voiced 12/14 · cut 6 clips · 0:30. What the project has cost, labelled “provider rates”. A figure like $4.12 · 2 unpriced means two calls had no price. A leading ≥ marks a lower bound. See Costs & Credits. Click a card to open the project. The dashed Start a project card does the same as + New project. Start a project + New project, on the Projects page or at the top of the Project selector, asks for a project name. Create project adds the project to the list, makes it the active project, and opens its home surface, titled “What do you want to make?”. Describe the work in the prompt. Type / to pick a skill, or @ to attach an asset or an entity. Optionally add Reference images and pick Entities. A picked entity goes to the agent as an entity:// reference. Pick a model with Select a model. If no language model is connected, NodeTool opens provider setup first. Click Send to chat. A starter is a skill: the saved instructions a chat turn loads when it reads /<name>. The pills under the prompt list skills NodeTool ships, such as product-commercial and short-film, and skills you wrote. Skills you used on earlier projects come first, and the rest fold behind a N more pill. Picking a pill writes /<name> into the prompt, and deleting the command from the prompt clears the pill. You can also start with no skill, and then the prompt is the whole ask. Send to chat opens a chat tab in the active project. The agent plans the documents the work needs and builds them. See Global Chat for how chat tabs work and Entities for the library. The estimate line, for example est. $3.10–$5.80 · from 2 past projects · provider rates, no markup, appears next to the send button once at least two of your own past projects of the same kind finished with fully priced spend of $1 or more and a cut with clips. Until then there is no estimate. Below the prompt, Start with a guided flow offers short step-by-step setups that each end in a document, and examples for apps, workflows, storyboards, timelines, sketches, 3D models, and games. See the Templates Gallery. Switch projects The Project selector at the left of the tab bar lists Personal and your named projects, with a search field. Choosing one opens its tabs as you left them. When a project’s tabs are grouped in the bar, the project chip at the start of the group offers Open editor home, Close group, and Switch to each other project. Where new documents land Every new document goes into the active project: the one shown in the selector. With no project open it goes into Personal. This holds for the + New menu, guided flows, and chats. The Documents panel lists the active project’s documents by kind: Workflows, Apps, Creative documents, and Agents & code. File a document into a project Documents that belong to no project appear under Not in a project at the bottom of the project list, newest first (12 at most, with a count when there are more). Drag one onto a project card to move it in. This works for storyboards, scripts, timelines, sketches, apps, games, and JavaScript scripts. Copy a document to another project On a document card, Copy opens a dialog with a Destination project field. The copy brings the referenced assets, entities, and supported documents with it, and the source stays unchanged. Workflows are listed but cannot be copied this way. Export and import a project Export on a project card downloads the whole project as one .nodetool-project file. The file holds the project’s documents, workflows, mini apps, games, and JavaScript scripts with their version history, its conversations, its assets with their media, and every file in its workspaces. Jobs, run history, and spend records are not included. Import project at the top of the project list reads such a file and creates a new project from it, then opens it. Importing never changes an existing project. Every document, asset, and conversation gets a new id, and references between them are rewritten to point at the new copies, so the same file can be imported twice. Imported workflows are private, and a project exported from Personal arrives as an ordinary named project. The same files move through the command line with nodetool projects export and nodetool projects import. See the CLI reference. Archive, restore, and delete Each card carries Export, Archive, and Delete buttons. They are absent for Personal. Archive hides the project from the selector and deletes nothing. Archived projects are listed under Archived projects at the end of the list, where Restore brings one back. Delete asks for confirmation in a dialog titled “Delete name?”. It removes the project and all of its documents, conversations, generated outputs, assets, jobs, and project files. This cannot be undone. There is no rename control in the project list. Ask the agent to rename a project, as described next. The agent and projects A project’s conversations are chats filed under it, and the server can also hold one agent thread per project, created the first time it is asked for and titled with the project’s name. Agents can manage projects with four tools: Tool What it does list_projects Lists your projects, newest first. Pass archived to list archived ones. search_projects Finds projects by name or kind. Every word must match. update_project Renames a project, changes its kind, or archives and restores it. delete_project Permanently deletes a project. It refuses the Personal project and the project the conversation belongs to. The agent is told to confirm with you before deleting and to prefer archiving. For the rest of the interface, see the user interface guide.--- # Provider contract probes Source: https://docs.nodetool.ai/provider-contract-probes.html Provider contract probes A cassette proves NodeTool still handles a response a provider gave us once. It cannot notice that the provider changed the response today — replay never reaches the network, so a renamed field or a dropped usage block is invisible until a user hits it. The contract probes close that gap for the providers whose wire shapes a run depends on most: Anthropic, OpenAI, Gemini, fal, KIE, and Replicate. A probe has two halves, and both run the same production decoder: Offline — packages/runtime/tests/providers/provider-contract-probes.test.ts decodes a checked-in raw HTTP response fixture, then deletes each declared required field and requires the decoder to reject the result. Runs on every PR that touches packages/runtime/src/providers/ (nodetool harness gate). Live — npm run probe:providers makes one real request per provider and decodes today’s response. Runs nightly from .github/workflows/provider-contract-probe.yml. This is not cassette replay and never writes a cassette. A cassette records a normalized conversation; a probe pins one wire shape at the client boundary. The manifest packages/runtime/src/providers/contract/probe-manifest.ts is the checked-in list. Each entry names the provider, the model or endpoint, the production decoder, the raw fixture, the required fields the positive controls delete, and — where a live call is cheap enough — the single request that fetches it. Entry Decoder Live anthropic.context-window-exceeded anthropicContextExceeded fixture only anthropic.context-window-exceeded-stop anthropicContextExceeded fixture only openai.chat-completion decodeChatCompletion 1 request, ≤ USD 0.05 openai.models-list decodeOpenAIModelList fixture only openai.context-length-exceeded openAIContextExceeded fixture only gemini.generate-content decodeGeminiGenerateContent 1 request, ≤ USD 0.05 gemini.models-list decodeGeminiModelsPage fixture only gemini.context-window-exceeded geminiContextExceeded fixture only fal_ai.language-catalog decodeFalLanguageCatalog 1 request, no key, free fal_ai.image-result extractImageUrls fixture only fal_ai.video-result extractVideoUrl fixture only fal_ai.audio-result extractAudioUrl fixture only kie.error-envelope kieEnvelopeError 1 request, free kie.create-task decodeKieTaskSubmission fixture only kie.record-info decodeKieRecordInfo + decodeKieResultUrls fixture only replicate.prediction-output decodeReplicateOutput fixture only The budget is one request and USD 0.05 per provider per run, enforced by runProbes and asserted by the manifest test. A fixture-only entry must say in liveGap why there is no live probe; most say the same thing — a live result would mean paying for a generation. Raising the budget is how those become live. The context-window entries pin a different kind of shape: the refusal a run gets when the prompt outgrew the model’s window. That one is recoverable — the caller can shorten the transcript and retry — so the recognizer that identifies it is as much a decoder as the ones that read a successful response, and its check asserts which of the provider’s signals fired rather than only that some did. Probing a different model The two live chat probes send a fixed model id: gpt-5.4-mini for openai.chat-completion, gemini-3-flash for gemini.generate-content. NODETOOL_PROBE_OPENAI_MODEL and NODETOOL_PROBE_GEMINI_MODEL override them, so you can ask whether a model you are about to adopt answers in the shape the decoder reads, without editing the manifest. NODETOOL_PROBE_OPENAI_MODEL=gpt-5.4 \ npm run probe:providers -- --only openai.chat-completion The budget does not follow the override. runProbes charges the entry’s declared estimatedCostUsd — USD 0.001 for both — not what the request actually cost, so an override onto an expensive model spends more than the ledger says it did. The request cap still holds at one per provider. Network failures are not schema failures runProbes separates them, and the nightly job treats them differently: network-failure — no body reached the decoder (DNS, timeout, a 5xx, an HTML gateway page). Reported, does not fail the job. A nightly that cries wolf on every provider blip stops being read. schema-failure — the provider answered and the response no longer decodes into what NodeTool needs. Fails the job. This is the contract breaking. --strict-network fails on network failures too, for a manual run where the network is known good. What a run retains Nothing of the body. summarizeShape reduces the response to its structure — every string becomes string(<length>), every number number — and only a small allowlist of enum-like keys (role, finish_reason, state, code, content_type, …) keeps a literal value. Free text a probe keeps, such as an error message, goes through redactText, which strips credential-shaped tokens and rewrites every URL to its origin. That is a redaction by construction rather than a denylist of field names, which is wrong the first time a provider adds a field. The uploaded artifact therefore carries no credential, prompt, model output, request id, or signed URL, and provider-contract-probes.test.ts asserts it on a payload seeded with one of each. Adding an entry Extract the decoder if it is still inline in a provider method, and have the provider call it. A decoder no test can reach without a network call is the problem this is fixing. Add the raw response fixture under packages/runtime/tests/fixtures/provider-contract/<provider>/. Hand-write it, or capture one and strip it — no credential, no real signed URL, no user content. Add the manifest entry with a check that asserts what a caller needs, and list the fields whose removal must break it. Run the test: a required field whose removal still passes is not required, and the test says so. Add a live block only if one request fits the provider’s budget; otherwise write the liveGap.--- # Connect OpenAI, Anthropic, Gemini & Ollama to NodeTool Source: https://docs.nodetool.ai/providers.html A provider is the adapter between a NodeTool node and an AI service: OpenAI, Anthropic, Gemini, a local Ollama daemon, or one of the 40+ others below. Every node that calls an LLM or a media model exposes a model property backed by a provider id. Pick a different provider from that same dropdown and the rest of the graph stays the same. This is bring-your-own-key (BYOK). NodeTool does not mark up a provider’s price, and cloud usage is billed directly by the provider. The one exception is the nodetool provider, which exists only in NodeTool’s hosted cloud (see NodeTool managed models). Add a key in Settings → Models & Providers, or skip keys and run everything through local models (Ollama, vLLM, LM Studio, llama.cpp). New to models and providers generally? Start with Models & Providers for the local-vs-cloud overview, or Supported Models for the full model catalog. This page is the provider reference: what each one does, which key it needs, and where to read more. Capability matrix Checked against each provider’s implementation in packages/runtime/src/providers/. A blank cell means that provider doesn’t expose the modality through NodeTool’s generic nodes, even where the underlying service might. Provider Text Image Video TTS ASR Embeddings 3D OpenAI ✅ ✅ ✅ ✅ ✅ ✅   Anthropic ✅             Azure OpenAI ✅ ✅   ✅ ✅ ✅   Amazon Bedrock ✅ ✅           Google Gemini ✅ ✅ ✅ ✅ ✅ ✅   xAI (Grok) ✅ ✅ ✅         DeepSeek ✅             Groq ✅             Mistral ✅         ✅   Cerebras ✅             Alibaba Cloud ✅             GMI Cloud ✅             OpenRouter ✅ ✅ ✅ ✅ ✅ ✅   Requesty ✅             Opper ✅             API Route ✅             Together AI ✅ ✅ ✅ ✅ ✅ ✅   Moonshot (Kimi) ✅             Meta AI ✅ ✅     ✅     MiniMax ✅ ✅ ✅ ✅       Codex (OpenAI OAuth) ✅ ✅           Claude Agent SDK ✅             Antigravity ✅             Evolink ✅ ✅ ✅         kie.ai ✅¹ ✅ ✅ ✅       AKI ✅ ✅           Higgsfield   ✅ ✅         UseAPI   ✅ ✅         vgenv     ✅         Replicate ✅ ✅ ✅ ✅ ✅ ✅   FAL ✅ ✅ ✅ ✅       Comfy Cloud   ⁵ ⁵         HuggingFace ✅ ✅ ✅² ✅ ✅ ✅   Ollama ✅         ✅   vLLM ✅       ✅ ✅   LM Studio ✅             llama.cpp ✅             llama.cpp local ✅         ✅   Transformers.js ✅     ✅ ✅ ✅   ElevenLabs       ✅ ✅³     Topaz   ✅⁴           Reve   ✅           AtlasCloud   ✅ ✅ ✅ ✅   ✅ Cohere           ✅   Voyage AI           ✅   Jina AI           ✅   Meshy AI             ✅ Rodin AI             ✅ ¹ Chat only for a fixed gateway list of GPT, Claude, Grok, Kimi, DeepSeek, and Gemini models (KIE_CHAT_MODELS in kie-provider.ts). Most kie.ai models are image, video, or audio. ² Text-to-video only; no image-to-video. ³ Via a dedicated Speech-to-Text node, not the generic ASR picker. ⁴ Upscale and enhancement, not text-to-image generation. ⁵ No models in the generic pickers — the Comfy API enumerates none. What comes out is whatever the ComfyUI graph you submit produces, through the lib.comfy.RunWorkflowOnCloud node. Provider capabilities Each provider lists the operations it implements by overriding BaseProvider.declaredCapabilities() (packages/runtime/src/providers/base-provider.ts). getCapabilities() always adds the two chat methods, so providers with no chat models (Meshy, Rodin, Higgsfield) throw when asked to chat. Some providers declare more than the matrix shows. Video-to-video, lip sync, upscaling, and music are examples, and they surface through agent tools and dedicated nodes, not the generic pickers. To make a model usable outside the node graph (an agent, the generate CLI command, the generation API), implement the matching method on the provider and declare its capability. The contract test packages/runtime/tests/providers/provider-capabilities-contract.test.ts fails when a declared capability has no implementation or a model in the provider’s catalog offers an undeclared task. OpenAI OpenAI covers chat (GPT), image generation (GPT-Image), Sora 2 Pro video, TTS, Whisper transcription, and embeddings — six of the seven modalities in the matrix above. Cloud only, keyed by OPENAI_API_KEY. Chat models are fetched live from the OpenAI API; image models are a maintained static list. See the OpenAI provider guide. Anthropic Anthropic runs Claude chat models with tool calling and image input for vision tasks. It doesn’t generate images, video, or audio. Cloud only, keyed by ANTHROPIC_API_KEY; models are fetched live from the Anthropic API. See the Anthropic provider guide. Azure OpenAI Azure OpenAI runs OpenAI models deployed in your own Azure OpenAI or Microsoft Foundry resource: chat, GPT Image generation and editing, TTS, Whisper transcription, and embeddings. Requests use OpenAI’s wire format, so each call behaves like the OpenAI provider with the deployment name as the model. Cloud only, keyed by AZURE_OPENAI_API_KEY and AZURE_OPENAI_ENDPOINT (for example https://<resource>.openai.azure.com, with or without a trailing /openai). Azure does not report what a deployment serves, so the models are the names in AZURE_OPENAI_DEPLOYMENTS, separated by commas. A name containing image or dall-e is an image model, tts a speech model, whisper or transcribe a transcription model, and embedding an embedding model. Any other name is a chat model. Add :chat, :image, :tts, :asr, or :embedding to a name to set its kind, for example my-painter:image. For custom deployment names, append =underlying-model, for example speech-prod=gpt-4o-mini-tts or transcription-prod=whisper-1. The underlying model controls media capabilities, including speech instructions and word timestamps, while requests still route to the deployment name. An explicit kind may precede the mapping, such as speech-prod:tts=gpt-4o-mini-tts. Without AZURE_OPENAI_API_VERSION the provider calls the v1 API at <endpoint>/openai/v1. Set a dated version such as 2025-04-01-preview to use the /openai/deployments/<name> routes. Under the cloud profile the endpoint must be a public https host. A deployment named after its model, such as gpt-5.5, is priced from the Azure rates in @pydantic/genai-prices. Amazon Bedrock Amazon Bedrock runs chat through the Converse API, which covers every Bedrock chat model: Claude, Llama, Mistral, Amazon Nova, and others. It streams text and tool calls, accepts images and documents, and passes Claude extended thinking through. Image generation runs Amazon Nova Canvas, Titan Image, and Stability models. Cloud only, keyed by an IAM access key in BEDROCK_ACCESS_KEY_ID and BEDROCK_SECRET_ACCESS_KEY, with BEDROCK_SESSION_TOKEN for temporary credentials. The keys are separate from the AWS_* variables so an S3 storage key is never used for Bedrock. BEDROCK_REGION picks the region (default us-east-1). Models are listed from that region: on-demand foundation models plus the system-defined cross-region inference profiles such as eu.anthropic.claude-sonnet-4-6, which Claude needs in most regions. The IAM user needs bedrock:ListFoundationModels, bedrock:ListInferenceProfiles, bedrock:InvokeModel, and bedrock:InvokeModelWithResponseStream. Set BEDROCK_MODEL_IDS to a comma-separated list of model or inference profile IDs to offer only those and skip the listing calls. Costs are priced from the AWS rates in @pydantic/genai-prices. Google Gemini Gemini handles chat with native multimodal input (images, audio, video as Blobs), Nano Banana image generation (the gemini-*-image models), Veo video, Gemini TTS, Lyria music, audio transcription, and text embeddings. Cloud only, keyed by GEMINI_API_KEY. Text models auto-fetch. The image, video, TTS, and music models are static lists. Imagen is not available because Google shut it down in the Gemini API. See the Gemini provider guide. xAI (Grok) xAI runs Grok chat models plus Grok Imagine for text-to-video, image-to-video, and text-to-image, all classified from the same /v1/models response. Cloud only, keyed by XAI_API_KEY. See the xAI provider guide. DeepSeek DeepSeek is chat only — DeepSeek-V3 and the R1 reasoning line — reached through an OpenAI-compatible endpoint. Cloud only, keyed by DEEPSEEK_API_KEY. See the OpenAI-compatible providers guide for how its models are fetched. Groq Groq runs chat models on its LPU inference hardware for low-latency responses. Text only — no image, video, or audio generation. Cloud only, keyed by GROQ_API_KEY. See the OpenAI-compatible providers guide. Mistral Mistral runs chat models (Mistral, Mixtral) plus one embedding model, mistral-embed. Cloud only, keyed by MISTRAL_API_KEY. See the OpenAI-compatible providers guide. Cerebras Cerebras runs chat models on its high-throughput inference hardware. Text only. Cloud only, keyed by CEREBRAS_API_KEY. See the OpenAI-compatible providers guide. Alibaba Cloud Alibaba Cloud Model Studio serves the Qwen chat models — qwen3-max, qwen-plus, qwen-flash, qwen-turbo, and the qwen3-vl vision line — with streaming and tool calling. Text only. Cloud only, keyed by DASHSCOPE_API_KEY, over the DashScope OpenAI-compatible endpoint (default https://dashscope-intl.aliyuncs.com/compatible-mode/v1, the international/Singapore region); models are fetched live from its /models endpoint. Model Studio keys are region-scoped — set DASHSCOPE_BASE_URL to your region’s /compatible-mode/v1 endpoint when the key was created outside Singapore. Tool calling is gated per model family: Qwen-Math, QVQ, OCR, translation, and audio models are outside Alibaba’s function-calling list and are not sent tools. Get a key at Model Studio. See the OpenAI-compatible providers guide. GMI Cloud GMI Cloud is an OpenAI-compatible chat endpoint for open-weight models — Llama, DeepSeek, and Qwen variants. Text only. Cloud only, keyed by GMI_API_KEY. OpenRouter OpenRouter proxies 300+ chat models, image generation, Gemini image editing, and text-to-video and image-to-video models (Veo, Seedance, Kling, Wan) through a single key. Reference-to-video is not supported because OpenRouter accepts reference media only as public HTTPS URLs. Cloud only, keyed by OPENROUTER_API_KEY. See the OpenAI-compatible providers guide. Requesty Requesty routes chat models from OpenAI, Anthropic, Google, DeepSeek, xAI and others through one OpenAI-compatible endpoint, https://router.requesty.ai/v1. Text only. Cloud only, keyed by REQUESTY_API_KEY. The model picker lists the managed models from /v1/models/managed (short ids such as gpt-5.4-mini) followed by the full vendor/model catalog from /v1/models (for example openai/gpt-4o-mini). Get a key at Requesty. See the OpenAI-compatible providers guide. API Route API Route serves hosted chat models through https://global.api-route.com/v1. Set API_ROUTE_API_KEY in Settings > API Keys or the environment. The model picker reads /v1/models, keeps OpenAI-compatible routes, and uses the shared model classifier to exclude image/video entries by metadata or model ID. Chat IDs without endpoint metadata remain available for older catalogs. This integration exposes text chat and tool calls. Get a key at API Route, then select a model available to that key. API Route is a paid service. See its setup guide and pricing. Opper Opper is an EU-hosted AI gateway that routes chat models from 50+ providers, including OpenAI, Anthropic, Google, DeepSeek and Moonshot, through one OpenAI-compatible endpoint, https://api.opper.ai/v3/compat. Text only. Cloud only, keyed by OPPER_API_KEY. The model picker lists Opper’s pools from /v3/compat/models?type=pool (bare ids such as claude-sonnet-4-6, routed across providers) followed by the full catalog from /v3/compat/models, which adds provider/model ids that pin one route. Get a key at Opper. See the OpenAI-compatible providers guide. Together AI Together AI is one of the broadest providers in NodeTool: chat, manifest-driven image and video generation, TTS (Orpheus, Kokoro, Cartesia Sonic), ASR (Whisper Large v3, Voxtral, Parakeet), and one embedding model. Cloud only, keyed by TOGETHER_API_KEY. See the Together provider guide. Moonshot (Kimi) Moonshot runs Kimi chat models over its OpenAI-compatible endpoint (https://api.moonshot.ai/v1). Text only. Cloud only, keyed by KIMI_API_KEY. See the OpenAI-compatible providers guide. MiniMax MiniMax covers chat, image (Image-01), video (Hailuo 2.3), TTS, and music generation in one provider. ASR and embeddings exist in MiniMax’s API but are disabled in NodeTool’s provider (embeddings need a GroupId NodeTool doesn’t manage) — use another provider for those. Cloud only, keyed by MINIMAX_API_KEY. See the MiniMax provider guide. Meta AI Meta AI serves the Muse family at https://api.meta.ai/v1. Cloud only, keyed by META_API_KEY. Chat. Muse Spark runs on Meta’s Responses API with tool calling. A web_search tool is answered by Meta’s hosted search grounding, and tools marked deferLoading are loaded on demand by Meta’s hosted tool search. Messages can carry images, video and audio, which Muse Spark reads natively. If the model list cannot be fetched, the picker falls back to four built-in Muse Spark ids. Images. Muse Image (muse-image-1.0) generates and edits images, with up to 10 source images per edit. Speech to text. Muse Voice Transcribe (muse-voice-transcribe-1.0) returns the transcript and timed turns. Audio that is not mono 16-bit WAV at 16 or 24 kHz is converted with ffmpeg first. It has no word-level timestamps. Segmentation. SAM 3.1 (sam-3.1) segments an image from a short noun phrase, a box, or points, and returns one full-frame mask per object. Codex (OpenAI OAuth) Codex reaches GPT chat models and GPT Image generation (GPT Image 2 and 1.5) through your ChatGPT/Codex OAuth session instead of an API key, and usage bills against that subscription. Sign in with the Codex card in Settings → Models & Providers. The provider reads the stored CODEX_ACCESS_TOKEN, so no OPENAI_API_KEY is needed. Claude Agent SDK Claude Agent SDK reaches Claude by running your local, logged-in claude CLI in headless mode instead of calling the Anthropic API directly, billing against your Claude subscription rather than per-token API spend. Install Claude Code first. NodeTool finds claude on your PATH or in the usual install locations, or set CLAUDE_CLI_PATH in Settings → Integrations to point at it. It supports tool calls, which run inside NodeTool while the CLI drives the loop. Images in the prompt are forwarded when they are inline PNG, JPEG, GIF, or WebP data. A remote image URL is rejected, so resolve media references before the call. There is no API key. Sign in with the Claude Code card in Settings → Models & Providers or run nodetool auth claude login (add --manual on a headless host). The provider is not available in NodeTool’s hosted cloud because it needs a local executable. See the Anthropic provider guide. Antigravity Antigravity reaches the models in Google’s Antigravity CLI (agy), including Gemini, by running the CLI in headless mode on your signed-in Antigravity account. There is no API key. Install agy and run it once in a terminal to sign in. NodeTool also looks for the CLI in ~/.local/bin, where the official installer puts it. The model list is whatever agy models prints, and effort is part of each model id (for example gemini-3.1-pro-high). Each turn spawns agy in an empty scratch directory under the system temp folder, so the CLI’s own file tools cannot touch your files. Shell commands stay blocked because NodeTool never passes --dangerously-skip-permissions. NodeTool tools are not offered to the model, and prompts must be text: a message with images fails instead of losing them. A forced single tool call (structured output) uses the CLI’s --json-schema. Follow-up turns in a chat resume the same Antigravity conversation with --conversation. The provider is not available in NodeTool’s hosted cloud because it needs a local executable. Evolink Evolink is an OpenAI/Anthropic-compatible gateway: one key for GPT, Claude, Gemini, and DeepSeek chat, plus image (GPT Image 2, Nano Banana 2, Seedream) and video (Seedance, Wan, Veo, Sora, Grok) generation. Cloud only, keyed by EVOLINK_API_KEY. kie.ai kie.ai is a multi-model aggregator: manifest-driven image, video, TTS, and music models (Seedance, Runway, Wan, Kling, FLUX.2, Suno, and more), plus chat for a fixed list of gateway models (GPT, Claude, Grok, Kimi, DeepSeek, and Gemini families). One key covers all of it. kie.ai often lists lower prices than the upstream provider, so compare before you commit. Cloud only, keyed by KIE_API_KEY. See the KIE provider guide. AKI AKI is an OpenAI-compatible gateway for chat plus text-to-image and image-to-image generation. Cloud only, keyed by AKI_API_KEY. Higgsfield Higgsfield generates image and video from a manifest of models (text-to-image, image-to-image, text-to-video, image-to-video, reference-to-video, video-to-video, extend). No chat. Cloud only. It needs two secrets, HIGGSFIELD_API_KEY_ID and HIGGSFIELD_API_KEY_SECRET. The Higgsfield card in Settings accepts them as one pasted key-id:secret value. UseAPI UseAPI generates image and video through your own Google Flow and Dreamina accounts on useapi.net: Nano Banana and Veo models through Google Flow, Seedream, Seedance, and Sora 2 through Dreamina. No chat. Cloud only, keyed by USEAPI_API_TOKEN. Set USEAPI_GOOGLE_FLOW_EMAIL (Google Flow account email) and USEAPI_DREAMINA_ACCOUNT (region:email) so reference image uploads use the right account. vgenv vgenv is an independent API for MiniMax H3 video, not affiliated with MiniMax. It runs text-to-video, image-to-video with a first and optional last frame, and reference-to-video with up to nine ordered images. Models are the Fast, Standard, Enhanced, and Quality tiers at 480p or 768p, 4 to 15 seconds. Prompts are sent as written, with vgenv’s prompt rewriting turned off. No chat. Cloud only, keyed by VGENV_API_KEY. vgenv bills each video from a prepaid balance at creation and releases the hold when a preparing or queued job is cancelled or the platform fails it. NodeTool cancels the job when a run is stopped or times out. Replicate Replicate runs chat, image, video, and music, plus curated TTS, ASR, and embedding models. Chat calls replicate.run()/.stream() on whatever model id you pass, so it isn’t limited to a fixed model list the way most other providers are; image, video, and music nodes are generated from Replicate’s schemas. Cloud only, keyed by REPLICATE_API_TOKEN. See the Replicate provider guide. FAL FAL generates image, video, TTS, and music from models whose nodes are generated straight from FAL’s OpenAPI schemas. Chat goes through fal’s OpenAI-compatible route, which is OpenRouter’s router, so chat model ids are OpenRouter’s catalog. Cloud only, keyed by FAL_API_KEY (FAL_KEY also works). See the FAL provider guide. Comfy Cloud Comfy Cloud runs a ComfyUI workflow graph on cloud.comfy.org — no local ComfyUI install, no GPU of your own. It is reached through the lib.comfy.RunWorkflowOnCloud node rather than a model dropdown: the graph you paste in names its own checkpoints and LoRAs, so the provider lists no models and doesn’t appear in the generic image or video pickers. Cloud only, keyed by COMFY_API_KEY from https://platform.comfy.org. The same key also authenticates partner API nodes inside the submitted graph. See ComfyUI. HuggingFace HuggingFace routes chat, image, text-to-video, TTS, ASR, and embeddings to Hugging Face Inference Providers. The model lists are the most-liked warm inference models for each task, fetched from the Hub and cached for 10 minutes. Tool calling is off for this provider. A HF_TOKEN is required: paste a token with the Inference Providers permission into the HuggingFace card, or set HF_TOKEN in the environment. The 19 huggingface.* task nodes use the same token. See HuggingFace Integration. Ollama Ollama runs chat and embedding models locally, with no API key and no per-token cost. It is a separate program: install it from ollama.com and keep it running, because NodeTool does not ship or start it. Pull a model with ollama pull <model> and it appears in NodeTool. The server URL comes from OLLAMA_API_URL (default http://127.0.0.1:11434), set in Settings → Integrations → Local Model Servers or the environment. OLLAMA_CONTEXT_LENGTH and OLLAMA_KEEP_ALIVE are in Configuration. See the Ollama provider guide. vLLM vLLM points NodeTool at a self-hosted, OpenAI-compatible vLLM server for chat, embeddings (/v1/embeddings), and transcription (/v1/audio/transcriptions). Models appear from its /v1/models endpoint once the URL is set. /v1/models does not report a model’s task, so every served model is offered for each task and the server rejects an unsupported call. Set VLLM_BASE_URL (no default) and VLLM_API_KEY if your deployment requires one. See Local Inference. LM Studio LM Studio connects to the local server that LM Studio’s desktop app exposes. Enable it in LM Studio → Local Server and NodeTool picks up loaded models automatically. The default URL is http://127.0.0.1:1234, overridden with LMSTUDIO_API_URL. LMSTUDIO_API_KEY is optional. See Local Inference. llama.cpp llama.cpp points NodeTool at a local llama-server instance for chat. llama-server constrains generation with a grammar built from the tool schemas, so tool calling works for any model it serves. Set LLAMA_CPP_URL (required, no default). See Local Inference. llama.cpp local llama.cpp local (provider id node_llama_cpp) runs GGUF models inside the NodeTool backend through the node-llama-cpp binding, with no separate server. It serves chat and embeddings. Models are GGUF files in NODE_LLAMA_CPP_MODELS_DIR, which defaults to the shared llama.cpp cache. NODE_LLAMA_CPP_GPU_BACKEND selects auto, metal, cuda, vulkan, or cpu. No key. Local only. whisper.cpp (whisper_cpp) transcribes audio inside the backend through the optional @fugood/whisper.node@1.1.3 package. Install it from Package Manager. GGML models use the Hugging Face hub cache. WHISPER_CPP_MODELS_DIR adds a directory, and WHISPER_CPP_GPU_BACKEND selects auto, metal, cuda, vulkan, or cpu. Restart the backend after changing the GPU backend. whisper_cpp.LiveTranscription transcribes streaming PCM16 audio, with Silero VAD or fixed windows when no VAD model is installed. Local only. whisper.cpp server (whisper_cpp_server) posts audio to a user-run whisper-server at WHISPER_CPP_SERVER_URL. Its model id is default. It requires no native package. Local only. Transformers.js Transformers.js (provider id transformers_js) runs small ONNX models in-process: chat, TTS, ASR, and embeddings. Models download from the Hugging Face Hub on first use into <data-dir>/transformers-js-cache, or the directory in TRANSFORMERS_JS_CACHE_DIR. Tool calling is off. No key. Local only. The matching workflow nodes are the transformers.* nodes in HuggingFace Integration. Local Python providers When the Python worker runs with the matching pack installed, it registers its own providers over the stdio bridge, such as MLX on Apple Silicon and local Hugging Face execution. Install the packs from Package Manager → Python packs, or see Python nodes without the desktop app. With NODETOOL_PYTHON_ON_DEMAND=true, these providers appear only after a workflow has started the worker. The worker’s huggingface provider appears as huggingface-local because the TypeScript runtime already owns the huggingface id for the hosted API. These providers offer image, video, TTS, music, ASR, and embedding models from the worker’s pack. NodeTool managed models The nodetool provider runs a curated model catalog on NodeTool’s own platform keys and meters usage against a credit balance. It exists only in NodeTool’s hosted cloud. Desktop and self-hosted installs do not register it, and they use your own keys for every provider here. Custom OpenAI-compatible endpoints Any endpoint that speaks the OpenAI API can be a provider. In Settings → Models & Providers, use Add endpoint and enter a name, a slug, a base URL, and an optional API key. Chat models come from GET <base_url>/models unless you list ids yourself. Image models (/images/generations) and video models (/videos) are detected from the list, and you can add ids the detection misses. The provider id is custom_<slug>. On a server you can skip the UI and set CUSTOM_<SLUG>_BASE_URL and CUSTOM_<SLUG>_API_KEY as environment variables. See Custom Providers for the fields, model discovery, and editing. ElevenLabs ElevenLabs covers text-to-speech, with a large static voice and model catalog, and speech-to-text, plus realtime WebSocket variants of both. Cloud only, keyed by ELEVENLABS_API_KEY. See the ElevenLabs provider guide. Topaz Topaz enhances existing images and video — upscale, sharpen, denoise, restore — rather than generating from a prompt. Only the enhance and enhance-gen endpoints appear in the generic image-model picker; the rest (sharpen, denoise, lighting, matting, restore) are dedicated nodes. Cloud only, keyed by TOPAZ_API_KEY. See the Topaz provider guide. Reve Reve creates, edits, and remixes images through three dedicated nodes (CreateImage, EditImage, RemixImage); the runtime provider also exposes create and edit to the generic image picker. Cloud only, keyed by REVE_API_KEY. See the Reve provider guide. AtlasCloud AtlasCloud generates image, video, speech, music, and 3D assets from a hand-maintained manifest (Seedance, GPT Image 2, Nano Banana, and more), transcribes audio, and lists chat models from its API. Cloud only, keyed by ATLASCLOUD_API_KEY. See the AtlasCloud provider guide. Cohere Cohere provides text embeddings — embed-v4.0 and the English/multilingual v3 line. No chat and no reranking in NodeTool’s current provider. Cloud only, keyed by COHERE_API_KEY. Voyage AI Voyage AI provides text embeddings only. Cloud only, keyed by VOYAGE_API_KEY. Jina AI Jina AI provides text embeddings only. Cloud only, keyed by JINA_API_KEY. Meshy AI Meshy generates textured 3D meshes from text or a reference image, through the generic nodetool.model3d.TextTo3D / ImageTo3D nodes. Cloud only, keyed by MESHY_API_KEY. Rodin AI Rodin generates 3D assets from text or a reference image, through the generic nodetool.model3d.TextTo3D / ImageTo3D nodes. Cloud only, keyed by RODIN_API_KEY. Generic nodes: provider-agnostic workflows Nodes in the nodetool.* namespace take a model property and route to whichever provider owns that model — this is what makes switching providers a dropdown change, not a rewiring job. Node Switches between nodetool.agents.Agent OpenAI, Anthropic, Gemini, xAI, DeepSeek, Ollama, any chat provider nodetool.image.TextToImage FLUX.2, Nano Banana 2.0, GPT Image 2, Ideogram V3, Z-Image, HuggingFace, MLX nodetool.image.ImageToImage HuggingFace, local servers, cloud services nodetool.video.TextToVideo Sora 2 Pro, Veo 3.1, Seedance 2.0, Runway, Grok Imagine, Wan 2.6, Hailuo 2.3, Kling 3.0, HuggingFace nodetool.video.ImageToVideo Sora 2 Pro, Veo 3.1, Seedance 2.0, Runway, Luma, Grok Imagine, Wan 2.6, Hailuo 2.3, Kling 3.0 nodetool.model3d.TextTo3D / ImageTo3D Meshy AI, Rodin AI, AtlasCloud nodetool.audio.TextToSpeech OpenAI TTS, ElevenLabs, HuggingFace, local TTS nodetool.text.AutomaticSpeechRecognition OpenAI Whisper, HuggingFace, local ASR A generic node drops parameters the selected provider doesn’t support instead of erroring. Negative prompt, guidance scale, and seed apply mostly to diffusion-style backends, and GPT-Image and similar API-only models ignore them. Reach for a provider-specific node only when you need something the generic interface doesn’t carry, such as MiniMax text to speech with its emotion and pitch controls. Getting keys in Wherever a missing provider blocks you (a model dropdown, a node warning, the getting-started checklist), NodeTool opens the Connect an AI provider dialog in place. It groups providers under Sign in with your account (Claude subscription, OpenAI, Hugging Face) and Use an API key. On desktop and localhost it also points at Ollama for running locally at no cost. A pasted key is checked against the provider before it is saved. If the provider rejects it, the dialog does not save it unless you choose to. If the provider cannot be reached, the key is saved and the dialog says it is unverified. Settings → Models & Providers is the full list, grouped as Popular, Language Models, Media Generation, Gateways & Hubs, Web Search, Compute & Local, and Services & Advanced. Each configured provider has a Test button that calls the provider to confirm the key still works, and a Manage button to replace it. Codex and Claude Code offer sign-in with your account instead of a key. The dialog’s OpenAI sign-in stores a ChatGPT Codex token, so it enables the codex provider and leaves openai unconfigured until you paste an API key. Claude Code sign-in is hidden on hosted deployments because it finishes on the server’s own machine. NodeTool stores every key encrypted (AES-256-GCM) in a local SQLite database, not in a plaintext config file. See Configuration for how the encryption key itself is managed. From the CLI: nodetool secrets store OPENAI_API_KEY # prompts for the value, stores it encrypted nodetool secrets list # list stored keys (values are never shown) nodetool secrets get OPENAI_API_KEY # print a stored value Or set the variable directly in .env.development.local or your shell environment. A secret stored in the database wins over an environment variable of the same name, and the environment variable is the fallback when nothing is stored (the exceptions are SUPABASE_URL, SUPABASE_KEY, SUPABASE_SERVICE_ROLE_KEY, and SERVER_AUTH_TOKEN, where the environment wins). FAL_KEY is accepted in place of FAL_API_KEY. See Configuration for the full load order. Local providers (Ollama, vLLM, LM Studio, llama.cpp) don’t need a key. Point NodeTool at the server’s URL instead. For vLLM (VLLM_BASE_URL), llama.cpp (LLAMA_CPP_URL), and LM Studio (LMSTUDIO_API_URL), set it under Settings → Integrations → Local Model Servers or as an environment variable. Ollama reads OLLAMA_API_URL from the environment or the secret store. In NodeTool’s hosted cloud these local providers are not registered, unless the server runs with NODETOOL_NODE_PROFILE=full. Tracking spend Every cloud call records its token counts and cost. The Costs page (/costs, or Costs under the app pages in the left panel’s More tab) shows them over 7, 14, 30, or 90 days, grouped by execution, node type, workflow, provider, or model, so you can see which pipeline is expensive before the invoice does. The same records are readable from the terminal: nodetool costs summary, costs list (filter with --provider and --model), costs by-provider, and costs by-model. Each takes --json. For estimates before a run, spend limits, and hosted credits, see Costs and credits. Adding a new provider Every provider is a class in packages/runtime/src/providers/ extending BaseProvider (or OpenAIProvider / AnthropicProvider for OpenAI- or Anthropic-compatible APIs), registered in provider-registry.ts with the environment variables it needs. The right approach differs by provider — live model discovery, a hand-maintained manifest, or codegen from upstream schemas — so follow the Provider Guides runbook for your case rather than reverse-engineering it from an existing provider. Before opening a PR, run npm run test:affected, npm run typecheck, and npm run lint. See also Models & Providers — local vs. cloud, mixed workflows, getting started Supported Models — the full model catalog and local inference engines Installation — install NodeTool on Windows, macOS, Linux Comparisons — NodeTool vs. ComfyUI, n8n, Dify, Flowise, Langflow, Weavy Chat API — WebSocket API for chat interactions and provider routing Global Chat and Agent Mode — using providers interactively and through the planning agent Workflow API — building workflows with providers Provider Guides — runbooks for adding models and nodes to each provider--- # Python Bridge Protocol Source: https://docs.nodetool.ai/python-bridge-protocol.html NodeTool runs Python nodes and local-compute providers in a separate Python worker process. The TypeScript runtime talks to it over a stdio RPC protocol (a locally spawned worker) or the same frames over a WebSocket (a worker that is already running, such as a Docker container). Why this protocol exists NodeTool uses Python for: Python node execution local ML providers such as MLX and HuggingFace ComfyUI and Blender jobs on workers that front them media processing and model-specific dependencies The TypeScript runtime uses stdio instead of a localhost socket for the worker because it is simpler to supervise, avoids port-management issues, and keeps the worker strictly parent-scoped. Transport createPythonBridge() (packages/runtime/src/python-bridge-factory.ts) picks the transport. If NODETOOL_WORKER_URL is set to a ws:// or wss:// address, the runtime uses the WebSocket transport. Otherwise it uses stdio. Stdio Parent process spawns: python -m nodetool.worker --stdio stdin / stdout: binary protocol traffic stderr: worker logs and startup diagnostics The interpreter is the first match of: the pythonPath option or NODETOOL_PYTHON, the active CONDA_PREFIX when it looks like a NodeTool environment (named nodetool or conda_env), then the NodeTool-managed conda environments for the platform. In production (NODETOOL_ENV=production) the stdio bridge refuses to start unless NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1. The WebSocket bridge is allowed in production. WebSocket The bridge never spawns a process. The worker lifecycle is owned elsewhere. One WebSocket binary message is exactly one MessagePack frame, with no length prefix. NODETOOL_WORKER_TOKEN (or the workerToken option) is sent as Authorization: Bearer <token> on connect and on every reconnect. If the socket drops, in-flight requests are rejected and the bridge reconnects with exponential backoff (1s, doubling, capped at 30s). After a reconnect it repeats discover and worker.status. Stdio framing Each message is encoded as: [4-byte big-endian payload length][MessagePack payload] The payload is a MessagePack object with this envelope shape: { "type": "discover", "request_id": "uuid-or-null", "data": {} } Lifecycle TypeScript spawns the worker Python loads node packages and providers Python prints NODETOOL_STDIO_READY on stderr (stdio only) TypeScript sends discover Python responds with node metadata, protocol version, and any load errors TypeScript requests worker.status, which sets the capability gates described under Versioning Workflow execution uses execute / execute.stream, cancel, provider.*, and, when the worker supports them, models.*, job.*, comfy.*, and blender.* messages Message types discover Returns the worker’s executable node inventory. Response data: protocol_version nodes — each entry may carry requires_vram_gb (v4+): the approximate VRAM that node type’s weights need, in GiB. The JS side echoes it back on execute so the worker’s reclaim pass has a real number to target instead of a percentage threshold. load_errors Example: { "type": "discover", "request_id": "d1", "data": { "protocol_version": 6, "nodes": [...], "load_errors": [ { "module": "nodetool.nodes.mlx.text_generation", "phase": "module_import", "error": "No module named 'mlx_lm'", "error_type": "ModuleNotFoundError" } ] } } worker.status Returns a structured worker health snapshot. Response data: protocol_version node_count provider_count namespaces load_errors transport max_frame_size model_prepare_backends (v5+): the backends models.prepare accepts comfy (v3+): present when the worker fronts a ComfyUI server blender: present when the worker can run Blender execute Executes a Python node. Request data: node_type fields secrets blobs blob_transfer: "chunked-v1", sent only to workers that report v5 or newer (see Chunked blob transfer) Run identity, added in v4. Every field is optional, and a field the JS host cannot name is omitted rather than sent as null: node_id — the graph node id. Becomes the constructed node’s _id. Before v4 the worker built every node with no id, so self._id was "" for all of them: a node calling set_model(self._id, …) registered into a single shared bucket, and release_nodes() had nothing meaningful to release. node_id is what makes the worker’s node → model map real. job_id — pairs the execution with the job.start / job.end boundary. workflow_id, user_id — populate the worker’s WorkerContext, which previously had neither. requires_vram_gb — the hint the worker itself reported for this node type at discover. These are extra dict entries, so the JS side sends them unconditionally rather than behind the v4 capability gate: a pre-v4 worker reads the four keys it knows and ignores the rest, and gating would only starve a worker that does understand them whenever its worker.status has not landed yet. Possible response types: progress result error execute.stream Executes a Python node that streams results. Same request data as execute (including the v4 identity fields), but the worker emits zero or more chunk messages (each carrying partial outputs/blobs) followed by a terminal result or error. cancel Requests cooperative cancellation for an in-flight execute or streaming provider request. provider.* Used for Python-only providers. The message families the TS bridge implements: provider.list provider.models provider.generate provider.stream provider.text_to_image provider.image_to_image provider.text_to_video provider.image_to_video provider.reference_to_video provider.text_to_audio provider.tts provider.tts_encoded provider.asr provider.embedding Streaming providers (provider.stream, provider.tts) emit zero or more chunk messages followed by a terminal result or error. models.* Worker model management (HuggingFace cache). These were introduced in bridge protocol v2 and are gated by supportsModelManagement() — a v1 worker simply does not expose them (see Versioning). models.list_cached — list models cached on the worker’s HF_HOME (cache-only, no network) models.download — download a model onto the worker cache, streaming ordered progress frames then a terminal result models.delete — delete a cached model; returns whether it existed models.prepare (v5, gated by supportsModelPreparation(backend)) — prepare an image-owned model through a backend the worker lists in worker.status.model_prepare_backends. Request data: backend, model_type, repo_id, and an optional Hugging Face token. It streams progress frames like models.download. models.evict (v4, gated by supportsJobLifecycle()) — drop loaded model weights. Optional request data narrows the scope: node_ids, job_id, target_vram_gb (stop once that many GiB are reclaimed). Response data is {evicted: string[], freed_vram_gb?: number}. This is the path for what only the JS side knows — the user switched workflows, another process wants the GPU, the worker is idle. Without it the worker only ever reclaims reactively, on its own thresholds. Calling it against a pre-v4 worker resolves to {evicted: []} instead of erroring, so a host asking to free memory never has to branch on the worker’s version. job.* The run boundary. Introduced in bridge protocol v4 and gated by supportsJobLifecycle(). job.start — opens a run. Request data: job_id, plus optional workflow_id / user_id. Optional in the sense that the worker needs no job.start to attribute an execution (every execute carries its own job_id); it exists as the one place to do a single reclaim pass per run instead of one per node. job.end — closes a run. Same data plus reason (completed | failed | cancelled | abandoned). The job’s nodes are retired and their models become eligible for release. This is the caller release_nodes() never had: without it the worker’s model cache grows across runs and is only ever trimmed reactively under memory pressure. job.end must fire on abnormal termination too — cancelled, client disconnected, run abandoned — or the leak simply moves to the failure path. On the JS side ExecutionSession sends it from the same finally that closes the bridge, so completion, failure, cancellation and timeout all reach it. Both calls are fire-and-forget and swallow their own failures: a boundary is bookkeeping, and a job.end that fails against a worker already tearing down must not turn a finished run into a failed one. A host that owns a long-lived shared bridge (the WebSocket runner) and injects its own executor resolver must pass that bridge to the session as jobLifecycleBridge — that host’s bridgeFactory deliberately returns null, so without it nothing closes the boundary for the exact deployment the shared bridge exists for. comfy.* ComfyUI proxy. Introduced in bridge protocol v3, gated by supportsComfy() — a worker offers these only when it fronts a co-located, loopback-only ComfyUI server AND reports worker.status.comfy.enabled: true. Route comfy.* requests only to such workers. The full field-level reference lives in docs/comfy-proxy.md in nodetool-core; ComfyUI covers the nodes that sit on top of these messages. comfy.execute — submit an API-format workflow; streams its lifecycle as dedicated comfy.event frames (see below), then a terminal result/error. Cancel with the standard cancel frame. comfy.queue — {queue_running, queue_pending} comfy.interrupt — global stop of the running job (admin-only) comfy.cancel — best-effort per-prompt cancel ({prompt_id}); the safe user-facing cancel comfy.upload / comfy.view — stage a file into / fetch a file from ComfyUI’s input dir comfy.object_info — full ComfyUI node catalog comfy.system_stats / comfy.status — health/capacity; comfy.status adds worker-level {enabled, url, reachable} comfy.free — unload models from VRAM without a cold restart comfy.models.list / comfy.models.download / comfy.models.delete — manage model files on the worker’s persistent volume. comfy.models.download streams generic progress frames (NOT comfy.event) then a terminal result. comfy.event comfy.execute does not stream progress frames — ComfyUI’s events don’t fit the {progress, total, message} shape. Instead it emits a dedicated frame: { "type": "comfy.event", "request_id": "<the execute request's id>", "data": { "event": "executing", "prompt_id": "p1", "node": "3" } } data.event is the discriminator, in emission order: queued → queue (repeatable) → started / cached → executing (per node) → progress → node_output → preview (only if previews: true) → completed or cancelled. result is always the last frame. blender.* Blender jobs. Gated by supportsBlender(): a worker offers blender.execute only when worker.status.blender.enabled is true. There is no protocol-version floor for this family, so the flag alone decides. blender.execute — request data is {job, inputs} plus optional blobs and timeout. It streams blender.event frames, then a terminal result or error. Cancel with the cancel frame. blender.event carries data.event (today only progress) with frame and total, taken from Blender’s Fra:<n> output during animation renders. Result, error, chunk, and progress result { "type": "result", "request_id": "e1", "data": { "outputs": { "text": "hello" }, "blobs": {} } } error { "type": "error", "request_id": "e1", "data": { "error": "Unknown node type: foo.Bar", "traceback": "..." } } chunk Used by streaming providers and streaming provider-like operations. progress The worker forwards NodeProgress messages from Python execution as protocol-level progress events: { "type": "progress", "request_id": "e1", "data": { "progress": 32, "total": 100, "message": "Downloading model" } } Binary data MessagePack allows binary payloads directly. NodeTool uses that for: input blobs in execute.data.blobs output blobs in result.data.blobs audio/image chunks for streaming provider APIs Chunked blob transfer From protocol v5 the JS side sets blob_transfer: "chunked-v1" on execute and on provider calls that return media. A worker that honors it sends each result blob as three frame types instead of inline bytes: blob.start — {name, size} blob.chunk — {name, offset, bytes} blob.end — {name, size, sha256} The bridge reassembles the blob and checks it against the final size and SHA-256. A mismatch rejects the request and sends cancel. A blob that declares a size above NODETOOL_PYTHON_MAX_RESULT_BLOB_BYTES (default 2 GiB) is rejected, as is an out-of-order or oversized chunk. Older workers keep returning the inline blobs map. Diagnostics and failure handling The bridge now surfaces worker problems in-band: discover.load_errors lists node import and metadata extraction failures worker.status.load_errors provides the same information on demand error.traceback is returned for failed requests recent stderr lines are still retained on the TS side for debugging startup failures This matters because metadata may exist for a Python node even if its module failed to import in the worker. In that case the worker is connected, but the node is unavailable. load_errors is the authoritative signal for that situation. Limits and timeouts The worker and TS bridge enforce a maximum frame size via NODETOOL_BRIDGE_MAX_FRAME_SIZE (default 256 MiB) Both transports enforce a startup timeout (startupTimeoutMs, default 20s) execute times out after NODETOOL_PYTHON_EXECUTE_TIMEOUT_MS (default 12 minutes) worker.status times out after NODETOOL_PYTHON_STATUS_TIMEOUT_MS (default 30s) Model downloads fail after NODETOOL_PYTHON_DOWNLOAD_IDLE_TIMEOUT_MS (default 5 minutes) without a progress frame Cancellation is cooperative NODETOOL_VALIDATE_BRIDGE_FRAMES=1 validates every inbound frame against the Zod schemas in packages/protocol/src/bridge-frames.ts. It defaults to on under tests and off otherwise. Versioning The JS runtime and Python worker each report a BRIDGE_PROTOCOL_VERSION. Two distinct numbers govern compatibility (see packages/protocol/src/bridge-protocol.ts): BRIDGE_PROTOCOL_VERSION — the protocol the JS runtime currently speaks (presently 6). MIN_BRIDGE_PROTOCOL_VERSION — the hard floor (presently 1). The JS runtime rejects a worker only if it reports a protocol below this floor. Compatibility rules: Older worker, at or above the floor — connects normally. It does not fail startup. Additive features the worker predates are gated per-capability: a v1 worker connects fine and simply doesn’t expose the models.* family (gated by supportsModelManagement(), which requires v2+), comfy.* (gated by supportsComfy(), which requires v3+ and comfy.enabled), job.* / models.evict (gated by supportsJobLifecycle(), which requires v4+), models.prepare (gated by supportsModelPreparation(), which requires v5+ and a matching model_prepare_backends entry), or blender.* (gated by supportsBlender(), which requires blender.enabled). Workers that predate the protocol_version field are treated as v1. The v4 identity fields on execute are the exception that proves the rule: they are extra keys on an existing message, not a new message, so they are sent to every worker and ignored by the ones that predate them. Only new message types need a capability gate, because those are what a worker answers with Unknown message type. Worker below MIN_BRIDGE_PROTOCOL_VERSION — rejected at discover with an actionable error (reinstall the Python environment). This is the only startup-failing case, reserved for genuine wire breaks. Newer worker protocol than the JS runtime — the runtime warns and assumes backward compatibility. Notes for contributors If you change the protocol: update the JS and Python protocol version constants together document the schema change here add or update end-to-end tests in nodetool-core/tests/ keep discover and worker.status authoritative for diagnostics Related Architecture Developer Guide nodetool-core/README.md--- # Ad library Recipes Source: https://docs.nodetool.ai/recipes/ad-library.html Ad library Recipes Each concept on the marketing site’s /ad-library pages ships as a Recipe mini app with the slug ad-<concept slug>. These apps use the shared plan/finish operations and build an editable vertical Timeline. scripts/example-apps/ad-library-recipes.mjs reads marketing/src/data/adLibrary.json. Each beat becomes a shot with the beat’s duration and composition. Each copy slot becomes an exact-text input. The file adds only the image inputs per concept and the beats that show them. Every shot shows the brand color as a background. The last shot adds the logo. Concepts that call for footage take still images, because the shared operations build still motion graphics and generate no video. The app is a sequence of guided steps: Start shows every beat’s illustration with its timing, the finishing model, the optional image model, and a choice: Fill from a website or Fill in myself. One to three content steps group the fields by the material the user has, such as “The offer” and “Your products”. Each group shows only the illustrations of the beats its fields fill. An image and the line shown with it share one card. Ending holds the last copy slot, the logo and the brand color. Its button plans the storyboard. While the plan runs, the layout agent’s activity shows below the fields. Review, Build and Result are the shared steps. Review shows the frames the layout agent composed, and Build adds the motion. Planning lays out every frame with the finishing model, so pick a model that reads images. With an image model, the agent can also generate backgrounds and decoration. See the layout pass. Plan and Build run as bundled workflow jobs, because the agents take longer than a script operation allows. ad-library-steps.mjs holds the steps, labels, placeholders and hints of each concept. A step label also renames the Recipe input. The build fails when a step plan leaves an input without a field. The illustrations ship as package://nodetool-base/ad-library/<beat>.webp. Each app’s gallery thumbnail is ad-<slug>.jpg in the same assets directory. The gallery uses an app’s <slug>.jpg before the art of its first bound workflow. Fill from a website opens a panel on Start. Its only input is a website URL. The agent uses the finishing model selected on Start. ad-library-ai.mjs holds two operations: Suggest options runs a small workflow bundled with the app, so it is a server job with no 120-second script limit. Its Code node has a 30-minute limit, because a Code node timeout of 0 falls back to the sandbox’s 30-second default. The node reads the page text with browser and collects the page’s image candidates from its HTML: social preview images, icon and logo links, and each <img> with its alt text, declared size and widest srcset entry. One run_agent call with the browser and image_search tools and up to 40 turns returns three different options. Each option has a title, an angle, the copy, the brand color and an image URL for each image field. The model may use only facts the website or the pages it reads state, and a missing fact becomes a bracketed placeholder. The panel’s Agent Activity widget shows the agent while it runs. The options appear as Choice Cards with the first image as a preview. Picking a card runs apply, a script operation. It fills every copy field and the brand color from that option, and saves each image with save_asset, which fetches it through safeFetch. It sets an image field only when the asset is an image of at least 400 pixels on its short side, or 100 for a logo. A report under the cards names the image fields left empty, so the user uploads those. Both operations write the same variables the inputs edit, and apply resets approval, so planning still preserves exactly what the fields hold. Stage the illustrations and thumbnails after a change to the marketing illustrations, then rebuild the bundles. The first command needs ffmpeg. node scripts/example-apps/ad-library-assets.mjs node scripts/build-example-apps.mjs node scripts/run-vitest.mjs run scripts/__tests__/ad-library-recipes.test.mjs--- # Product Price Drop Source: https://docs.nodetool.ai/recipes/product-price-drop.html Product Price Drop Product Price Drop is a normal Application available in the example app catalog. Installing it imports two pinned JavaScript operations with the Application. This vertical slice follows epic #5979 and the foundations in PRs #5988, #5989, #5990, and #5991. The Recipe is compiled from its manifest through the shared plan/finish operations and compiler, corresponding to issues #5985 and #5987. Design-frame review and finishing work is tracked in issues #5984 and #5986. There is no separate Recipe resource or executor. Supply a stored product image, logo, headline, old price, new price, CTA, and brand color. Choose a creative direction, run Plan, review the composed cut and exact layer list, approve it, then run Build editable ad. Changing a production input requires planning and approving again. Planning resets approval. The plan has a three-second hook and a three-second CTA. Product and logo use the original stored assets. Copy remains exact text. The brand background is a shape. Finishing creates the normal editable vertical Timeline, without calling an image or video generation provider. Use Open editable timeline on the result to enter the normal Timeline editor and render or export there. Rerunning the plan updates the existing shots by their stable semantic labels. Planning reads the linked Timeline revision, so manual placement edits made before planning can be reconciled. An edit after planning produces a revision conflict and requires a new plan. Finishing updates the layers identified by board, shot, and graphics-element identity. Manual placement edits survive. Manual replacement of protected sources or copy causes an explicit conflict instead of silently overwriting it. Finishing preserves unrelated Timeline metadata. Existing global camera, code-authored source, or media tracking returns an explicit conflict because this finishing slice cannot reconcile those features. Ordinary Storyboard assembly refuses to replace finished semantic layers. Use finish_storyboard to update them through the same reconciliation rules. The operation’s source permissions come from the Recipe manifest’s preservation rules. Both the Recipe and its executable scripts ship in the normal Application bundle. Rebuild it with: node scripts/build-example-apps.mjs --app product-price-drop --skip-validate node scripts/build-example-apps.mjs --app product-price-drop --check The integration fixture installs the bundle, saves a UI-only document edit, runs its actual sandbox scripts, creates a layered Timeline, renders through the existing compositor and encoder, reopens it, and repeats finishing after a manual placement edit and a price change. To preserve the render and sampled frames for inspection: PRICE_DROP_PROOF_DIR=/tmp/recipe-price-drop-proof \ npm run test --workspace=packages/websocket -- tests/product-price-drop.test.ts The frontend interaction test exercises the same Recipe’s normal input, ChoiceCards, and Approval bindings. The Playwright Recipe builder journey opens the real App Builder, edits its title through Puck, saves, reloads and confirms the original Recipe manifest survives. It then uploads both assets, fills every protected input, plans, reviews the composed preview before approval, approves, builds, verifies the inline preview, and opens the editable Timeline. It also changes the live script heads to prove that operations still execute their pinned versions. Run it with: npm run test:journeys --workspace=web -- recipe-builder.spec.ts The actual Timeline validator mechanically checks asset identity, copy, declared transforms and provenance. It does not claim pixel-level semantic recognition of every rendered object. finish_storyboard also accepts strategy: "agentic" in a provider-backed agent session, or with an explicit model: {provider, id} in a Mini App operation or headless script. An explicit model is used exactly, with no silent fallback. The default remains deterministic and needs no model. The bounded invocation requests medium effort with extended thinking disabled. The model still authors composition and performs mandatory pixel review. The opt-in pass receives the whole board, preservation rules and editable scaffold. A new agentic cut must author editable layout or motion before submission. Read-only, no-op and metadata-only edits do not satisfy this requirement. An existing cut may stay unchanged after visual review. This authoring check proves a composition change, while the visual review assesses its quality. Agentic reruns reuse accepted layer presentation and refresh Storyboard-owned source identity, exact copy, protected colors and timing. Typography and permitted manual placement survive input changes. Existing layer names stay user-owned. Manual typography, opacity or animation changes produce an explicit conflict before model dispatch. Retained presentation must pass the current production policy before model dispatch. It renders design references derived from the expected Storyboard revision and compares actual composited cut frames with those references. These are derived references, not a separate historical pixel approval. They show content, hierarchy and intent, not approved layout. The deterministic layout fits each image of known pixel size between the copy above and below it. Before visual review, a layout check renders each layer alone at every shot’s hold frame. That is the frame after the last entrance and before the first exit. The check returns copy that overlaps or crowds another layer, or reaches the frame edge, to the author. Copy wholly inside an image or plate passes as an overlay. The reviewer receives the collisions the references themselves contain, and a candidate that repeats one fails. Each hold frame is labeled in the review. The agent uses existing Timeline operations and cannot invoke media generation or replace accepted media. A batch that changes an accepted layer’s start or duration is rejected and rolled back. Motion delay and duration belong in Timeline animations within that window. Policy, structural and layout checks run before visual review. Findings require a changed draft and another render and review, with at most three candidates. An unreviewed, unresolved, cancelled or stale cut is not saved. Additional decorative layers need explicit stable semantic names within a shot. Rerunning an addition with that name reuses its existing identity. Use set_clip_params to change it. Extra layers carry board, shot and element provenance. Manual edits or deletion of agent-owned decorative layers produce an explicit conflict on rebuild. Existing manual placement edits to source layers retain the deterministic materializer’s preservation behavior. Protected layers cannot inherit an unproven group transform. Unsupported protected animation, mask or effect paths fail policy validation rather than silently weakening source fidelity. A visual review checks sampled frames, not every frame or perfect visual equivalence. The returned review report contains frame hashes, timecodes and derived-reference fingerprints. Agentic results report the provider’s measured cost delta as costUsd. Normal JS script execution scopes both imported capability calls and native nodetool aliases to the script deadline. Finishing checks that signal before provider calls and the final save. Returning from the script also revokes unawaited finishing calls without cancelling the caller’s processing context.--- # Shared Recipe operations and compilation Source: https://docs.nodetool.ai/recipes/shared-operations.html Shared Recipe operations and compilation A Recipe remains a normal Application. compileRecipeApplication in @nodetool-ai/app-runtime turns a manifest and concrete operation bindings into the document edited by the App Builder. It returns either that document or diagnostics identifying the unsatisfied manifest field or operation. Compilation executes no provider, workflow or script. Operation contracts RecipeOperationContract declares the semantic ID and positive version, typed named inputs and outputs, spend boundary, resource writes, preservation guarantees, supported media strategies, idempotency and stale-input behavior. RecipeOperationSpec.intent resolves that semantic ID. version defaults to one for older manifests. id and bindingId retain their existing meanings. The compiler requires one matching concrete OperationBinding with a pinned Script or Workflow target. It does not introduce another execution graph. The supplied contract describes an implementation’s guarantees. It is not a replacement for runtime enforcement. Source ownership and actual Timeline fidelity still run through the existing Storyboard and finishing capabilities. The compiler rejects unsupported preservation or media policy before execution. The first shared library is scripts/recipe-operations.mjs: Semantic operation Inputs and outputs Resource writes Spend and rerun behavior plan_storyboard@1 Typed Recipe inputs and a constant manifest, optional Storyboard ID, then Storyboard ID/revision, visual preview, plan, source fingerprint and approval state Updates source Asset entity metadata and creates or updates a normal Storyboard No generation spend. Creates shots by stable semantic label. Refreshing unchanged inputs adopts current Storyboard edits, regenerates the preview and resets approval. Changed bound inputs are reconciled separately from creative direction. A changed shot order, set or aspect ratio causes an explicit conflict. finish_storyboard@1 The same sources, approved fingerprint and Storyboard revision, optional Timeline ID/revision, then Timeline reference/revision and validation Creates or updates a normal Timeline and its Storyboard link Default deterministic mode produces an explicitly unreviewed editable draft with no model spend. Agentic mode declares model spend and requires rendered visual review before reporting a reviewed finished cut. Requires approval, rejects stale source or policy changes, then uses existing revision-checked finishing and reconciliation. Both use exact source assets, text and colors. Both support still_motion_graphics. Neither dispatches generative video or image editing. The shared planner reads semantic shot and graphics intent from manifest data. It does not contain Recipe-specific prices, copy or product selection. The finisher invokes the existing finish_storyboard capability. Planning tags product and logo assets through the existing create_entity capability when they are not already canonical entities. Entities are Asset metadata, so this does not duplicate uploaded media. Existing entity metadata is reused. A changed entity reference image causes an explicit conflict. Graphics and protected inputs retain their exact asset_id plus the entity relationship. Graphics-only entities are not cast into generation prompt lists. Without a finishing model, the planner invokes the read-only preview_storyboard_design operation and exposes its inline Timeline envelope through the existing Timeline widget before Approval. That is a composed visual preview, with no persisted Timeline or new renderer. Its textual source list remains available alongside it. With a finishing model, the planner invokes layout_storyboard instead. A vision model positions and scales every image and copy layer at each shot’s hold frame. The pixel layout check and a rendered-frame review gate its work, as in agentic finishing. With an optional imageModel, it can also generate backgrounds and decoration. Each generated image joins the Storyboard as a decorative element marked origin: "layout_agent", so a rerun of the plan keeps it. The layout is saved as the board’s Timeline with the materialization stage layout, and the review shows that Timeline. The next agentic finish keeps the placement and must author the motion before it reports a finished cut. A Recipe can author the placement of a graphic element. An element with a frame has an authored box. The scaffold places it there, and the agent keeps that placement. An element can also carry typography, a lock (position, scale or crop) and limits (x, y and scale). The agent checks every edit against the scaffold, never against the previous candidate. An edit that changes a locked property or exceeds a limit is rejected with the element, the property, the authored value and the limit. A parent group cannot move a locked child. In the finish pass, a locked layer takes only opacity animations. The reviewer receives each shot’s review_rules and a note that the locked placements are not defects. An element without a frame is composed freely. The layout takes at most three candidates. It stops early when two rounds in a row fail the same frames. When the mechanical checks pass but the visual review still has findings, layout_storyboard saves the best candidate and returns status: "needs_review" with findings. The plan outputs them as layoutFindings, and the review step shows them in a warning. The person can edit the cut, plan again or approve it. A candidate that fails a mechanical check never saves. The finish pass still fails after three candidates. To opt into agentic finishing, declare strategy: "agentic" and an existing model: {provider, id} on the Recipe’s finishing operation. Both become normal constant input mappings. Compilation rejects an absent model. Execution rejects a missing or changed strategy/model and stale approval before provider dispatch. This allows normal Mini App script execution without requiring an ambient chat session. Omitting the strategy produces an unreviewed draft. Structural and fidelity checks do not certify legible rendered pixels. Only agentic finishing that passes the rendered review gate reports a reviewed finished cut. Agentic script execution uses the protocol’s bounded 120-second timeout. Deterministic planning and finishing retain 60-second timeouts. A layout or finishing agent that reviews rendered frames takes longer, so the Recipe apps built by scripts/example-apps/recipe-app.mjs run both operations as bundled workflow jobs. A Code node runs the same code with a 30-minute limit. Each Recipe specializes the shared scripts’ declared input ports to its own typed inputs. The implementation code and semantic operation version remain the same. Normal script versions are carried in ApplicationBundle and pinned by ordinary import and release machinery. No separate Recipe executor or mutable runtime registry exists. Model, generation and rendering operations supplied by an author must declare their spend boundary and a separate approval input. Their implementation must reject stale inputs or resource revisions. The compiler never folds them into an automatic planning action. This library does not claim to supply generation or export operations that have not been bound to actual implementations. Compiler mapping Manifest input Existing widget Variable type Text TextInput str Text with declared choices ChoiceCards str Number NumberInput float Boolean Switch bool Image, video or audio ImageInput, VideoInput or AudioInput Matching media type Color ColorInput str Entity, generic Asset, Storyboard and Timeline inputs currently fail with an unsupported-widget diagnostic. The existing ResourcePicker selects a resource collection and does not write an input variable. The compiler does not emit a control whose selection never reaches the operation. Defaults preserve original values, including text whitespace. Named input groups become normal headings and controls. Choice values must be unique and their defaults must resolve. Timeline outputs use Timeline. Typed image, video and audio outputs use their media preview widgets. Other asset outputs use Download. Storyboard and structured value outputs use Json for review. Outputs are placed after the operation that produces them, before the next approval or action. Operations remain separate Buttons with ordinary run events and execution error bindings. Shared production scripts also expose the existing Stepper state and Approval control. Input changes invalidate approval, and Finish appears after approval. UI and variables remain editable through the App Builder. Semantic shot intent declares an ID, title, duration, source input references and asset/text/shape roles. It can select an existing Storyboard aspect ratio. It contains no tracks, clip IDs, keyframes, easing, masks or transforms. Shared planning requires an appropriate exact-preservation rule for every visible source. Compilation rejects protected inputs with no visible element. Storyboard IDs and Timeline references remain ordinary durable resources. The generated Application persists its inputs, resource references and revision-bound approval through normal user-scoped variable persistence. A failed refresh preserves the previous checkpoint. Approval is checked against current inputs, the Storyboard revision and any linked Script fingerprint before building. Refresh the Storyboard and approve again after editing it. Script-backed Recipes require NodeTool on the web or desktop. Mobile displays this limitation before collecting inputs because it cannot execute those production steps or edit the resulting Timeline. Authoring and verification Call compileRecipeApplication(manifest, {operations, title}) with contracts and actual bindings. compileSharedRecipeBundle provides the shipped shared plan/finish bindings and script documents. Its result passes ordinary document and executable-bundle validation. The manifests in recipe-manifests.mjs compile without hand-authored UI: O1 — Product Price Drop has two vertical shots, separate product/logo assets and exact headline, prices and CTA. O2 — Testimonial Card has a portrait, exact quotation and attribution in one 4:5 shot. It does not fabricate a generated spokesperson. O3 — Catalogue Visual Set has three different product assets and names across three square product cards in an editable Timeline. The existing example build script regenerates the shipped Price Drop bundle: npm run build --workspace=packages/app-runtime node scripts/build-example-apps.mjs --app product-price-drop --skip-validate node scripts/build-example-apps.mjs --app product-price-drop --check The compiler tests reject invalid contracts and bindings. Script tests run the real app debug --no-run simulator validation for all three bundles, prove determinism, verify normal imported pins, execute shared planning and reject stale copy, asset and preservation-policy approvals. The Price Drop integration test executes the actual sandbox scripts, renders the layered result and checks reconciliation: npm run test --workspace=packages/app-runtime node scripts/run-vitest.mjs run scripts/__tests__/recipe-operations.test.mjs npm run test --workspace=packages/websocket -- tests/product-price-drop.test.ts Adding a new generative operation, new widget strategy or arbitrary layout generator is separate work. It must provide an actual executable binding, runtime policy enforcement and a headless proof before a manifest can use it. An optional live provider proof uses the same installed Application and pinned Script operations. It declares agentic finishing with {provider: "claude_agent_sdk", id: "sonnet"} and requires an authenticated Claude Agent SDK environment. CI skips this provider-spending test by default. It validates protected media and exact copy, requires rendered visual-review evidence and exports the actual layered cut. Normal media generation remains forbidden. The proof does not require a fixed decoration count from the agent. RECIPE_LIVE_FINISH=1 PRICE_DROP_PROOF_DIR=/tmp/recipe-live-proof \ npm run test --workspace=packages/websocket -- tests/product-price-drop.test.ts The proof directory includes the agentic MP4, composited sample frames, Timeline JSON, review evidence and ffprobe dimensions, duration and frame count.--- # RFC: Generation Events — generation_complete, re-scoped output_update, autosave cutover Source: https://docs.nodetool.ai/rfc-generation-events.html RFC: Generation Events — generation_complete, re-scoped output_update, autosave cutover Status: Draft for sign-off · Owner: runtime/web Every line/file reference below was re-verified against the working tree before this revision. Findings that were wrong-as-written (the double-save rationale, the actor job_id field, “_messages authoritative for autosave”) are corrected in-body and catalogued in the Addressed review notes appendix. The scope was widened from “kernel→websocket→web-editor” to all message consumers (browser runner, mobile, CLI, mini-apps, timeline/sketch editors, chat) after a consumer census. 1. Summary We split the one overloaded “a node produced something” signal into three orthogonal channels: node_update — node lifecycle only (running / completed / error, timing, cost). One per node-run. Stops being the autosave/generation source. generation_complete (NEW) — a generator committed one complete artifact: { node_id, node_name, node_type, index, outputs } (plus job_id/workflow_id stamped downstream). Emitted once per process() result (once at stream-end for genProcess). Authoritative; drives liveGenerations + autosave; never suppressed. output_update — demoted to ephemeral display-only live feed (text tokens, progressive preview, realtime audio). Carries an explicit disposition: "append" | "replace". Never persisted, never creates a generation. Edge-suppression becomes harmless. The kernel change is small and TS-only. The breadth is in the consumers: every surface that today reads node_update.result or output_update.value as an artifact must move to generation_complete.outputs or accept a documented, graceful degradation to one artifact. 2. Problem & evidence The kernel collapses multi-execution runs into one node_update. _emitNodeStatus("completed", …) fires exactly once per node-run at actor.ts:440-443, carrying this._latestResult ?? {}. There are only 4 _emitNodeStatus call sites (actor.ts:278 running, :421 suspended, :426 error, :440 completed). _latestResult (declared actor.ts:157) is overwritten on every execution — never accumulated: actor.ts line mode write object identity 357 streaming-input run() = nodeOutputs.collected() fresh from collector 369 streaming-input legacy process() = outputs same ref as executor return 987 genProcess per-yield = {...collected} fresh spread each yield 994 genProcess stream-end = {...collected} fresh spread 1000 correlated process() = outputs same ref as executor return 1290 controlled-loop process() = outputs same ref as executor return (The object-identity column is load-bearing for §7 — see the corrected double-save analysis.) So a ListGenerator.item → TextToImage.prompt run that fires TextToImage 6× (the correlated while (isReady(key)) drain at actor.ts:716-720, each call overwriting _latestResult at :1000) emits one node_update{completed} carrying run #6 only. Autosave inherits the collapse. autoSaveAssets (unified-websocket-runner.ts:428-571) is called from the single site at unified-websocket-runner.ts:1932-1953, gated on outbound.type === "node_update" && status === "completed" && result != null && meta?.auto_save_asset. It walks outbound.result (= _latestResult, run #6) → persists 1 asset/run, 5 lost. (Idempotency guard at :468 if (assetValue.asset_id) continue skips already-asset_id‘d values by mutating that same object in place at :518-520; node_id/job_id stamped from opts at :506-507. Crucially, Asset records carry node_id+job_id but no index — there is no persisted-side dedupe key beyond the in-place asset_id mutation. This matters for replay; see §15.) output_update is the accidental sole carrier of the lost variants — and it’s overloaded three ways. Emitted at runner.ts:1286-1294, once per emit (per yield / per key / per control-run — NOT collapsed). But suppressed at runner.ts:1280-1284 for any handle with an outgoing data edge (findEdges(sourceNodeId, handle).some(isDataEdge) unless always_emit_output_updates). So an intermediate generator feeding a Preview emits nothing; only the terminal Preview re-emits and buffers all N. output_update simultaneously serves: (1) live incremental display (text tokens, progressive preview → setOutputResult(append=true)), (2) realtime audio transport (~50/s, coalesced to a worklet bus — the reason suppression exists, runner.ts:1257-1264), and (3) the accidental multi-execution variant carrier. It also can’t distinguish a chunk (append) from a whole value (replace) — workflowUpdates.ts:515 hardcodes append=true (latent progressive-preview bug). Net: a 6-image run produces 6 artifacts; the generation timeline and persistence see 1. 3. Goals / Non-goals Goals Every committed artifact of a multi-execution run is persisted (N assets/run) and appears as a navigable variant — on both the server and in-browser execution paths. Lifecycle, artifacts, and live display travel on three independent channels with clear contracts. output_update becomes correctness-irrelevant: edge-suppression can drop it freely. Zero nodetool-core / Python-worker change for the targeted multi-execution collapse (see §10). No double-save and no no-save during cutover. Every TS message consumer (web editor, browser runner, mobile, CLI, mini-apps, timeline/sketch, chat) is explicitly migrated or assigned a documented graceful-degradation decision. Non-goals Changing the realtime-audio fast-path (runner.ts:45-52 / bus). It stays byte-for-byte. Changing the TS↔Python bridge protocol. Reworking the salvaged UI navigator (groupByRun/NodeHistoryViewer). It is the correct consumer; it only needs a correct feed. A separate generation_started / progress channel (out of scope; node_update{running} + node_progress already cover it). Persisting N artifacts for multi-artifact-per-output-slot streaming generators (no such node exists today; see §10 — explicitly out of scope, with a guard test). 4. The three channels Channel Carries Rate Persisted? Creates Generation? Suppressible? node_update lifecycle: running/completed/error, timing, cost, property echo 1/run no no (changed) no generation_complete one complete artifact + stable variant index N/run (1 per process() result) yes (drives autosave) yes never output_update live display feed: text tokens, progressive preview snapshot, audio chunks high (≤~50/s audio) no no yes (harmless) 4.1 node_update (unchanged shape, narrowed role) messages.ts:166-178 stays as-is. After this change it no longer drives autosave or generations. Everything else (status/timing/cost/property-merge/error-notification/trace) is unchanged. The result field becomes advisory/back-compat only (still the last result); consumers must not read it for multi-execution artifacts. It remains the graceful-degradation path for consumers that cannot be fully migrated this cycle (mobile shows 1 artifact via result; see §8.6) and for skip-result nodes’ values (see Decision 7). 4.2 generation_complete (NEW) export interface GenerationComplete { type: "generation_complete"; node_id: string; node_name: string; node_type: string; /** k-th generation of this node in this run (backend-assigned, monotonic, stable). */ index: number; /** The complete result dict for this artifact (same shape as a process() return). */ outputs: Record<string, unknown>; /** Stamped downstream by the runner relay, NOT by the actor. */ job_id?: string | null; workflow_id?: string | null; } index is a per-actor monotonic counter (_generationIndex++), NOT the correlation lineage index (those reset per (root, parentKey) and are not a stable global variant id — see §5). A fresh actor is created per _processGraph and a fresh runner per job, so index does not collide across runs. job_id / workflow_id are optional in the actor emit and stamped uniformly downstream — see §5 and the corrected note below. outputs is the whole result dict, passed through unflattened. List-valued handles (num_images=N) stay as arrays; consumers flatten via runVariantValues / recursive autosave collect (Decision 1). Emitted from inside the serial actor paths, so per-node index ordering is deterministic. 4.3 output_update (re-scoped, + disposition) export interface OutputUpdate { type: "output_update"; node_id: string; node_name: string; output_name: string; value: unknown; output_type: string; metadata: Record<string, unknown>; /** NEW. "append" = value is a chunk to concatenate; "replace" = value is a whole snapshot. */ disposition?: "append" | "replace"; /** NEW (optional). Marks the final chunk of an append stream. */ done?: boolean; workflow_id?: string | null; } disposition is optional for back-compat: absent ⇒ treat as "append" (today’s behavior). Streaming chunks (text tokens, audio, genProcess yields) → "append"; whole non-generation values (a string into an Output node, a progressive-preview full-frame) → "replace". 5. generation_complete emission points Add to the actor, alongside _emitNodeStatus. The actor does NOT carry job_id — verified: packages/kernel/src/actor.ts has zero jobId/job_id/_jobId references, and _emitNodeStatus (actor.ts:1361-1380) emits node_update without a job_id field. job_id/workflow_id are backfilled for every outbound message at unified-websocket-runner.ts:1855-1861 (msg.job_id ?? active.jobId, msg.workflow_id ?? active.workflowId). A new type inherits this for free. (The browser path must apply the same backfill; see §8.5.) node_name mirrors node_update: this.node.name ?? this.node.type — not this.node.data?.title. private _generationIndex = 0; // new field near _latestResult (actor.ts:157) private _emitGenerationComplete(outputs: Record<string, unknown>): void { this._emitMessage({ type: "generation_complete", node_id: this.node.id, node_name: this.node.name ?? this.node.type, // mirror _emitNodeStatus node_type: this.node.type, index: this._generationIndex++, outputs // NO job_id here — runner relay stamps it (unified-websocket-runner.ts:1855-1861) }); } Wire it at each completed process()-result boundary — the same sites where _latestResult is assigned: # actor.ts site mode rule 1 after :1000 (correlated process()) one per ready key → N/run for a generator the primary 6×-collapse fix 2 after :1290 (controlled-loop process()) one per "run" control event   3 after :369 (streaming-input legacy process()) once also the path Python is_streaming_input nodes take (see §10) 4 after :994 (genProcess stream-end) ONCE per stream, using final _streamingCollectedOutputs — NOT per yield at :987 (yields are chunks → stay output_update) if also correlation-driven, reached once per key (correct). Load-bearing invariant — see below. 5 after :357 (streaming-input run()) once, nodeOutputs.collected() TS-only; Python nodes never reach run() (see §10). Parity-complete spot for filters/transforms; low priority Load-bearing invariant for site #4 (genProcess stream-end). _streamingCollectedOutputs is an overwrite-merge, not a concatenation: actor.ts:984 does Object.assign(this._streamingCollectedOutputs, routed), and :994 takes {...this._streamingCollectedOutputs}. Therefore the stream-end dict holds the last value written to each output slot, not all N values streamed through that slot. The “once at stream-end” rule is correct iff the generator’s final accumulated dict holds the complete artifact — true for nodes that emit a final whole-value yield (Summarizer-style: a terminal {text, output} consolidating yield; audio-with-done). It is wrong for a hypothetical generator that streams N distinct savable artifacts on the same output slot with no consolidating final yield — that would commit only artifact N (the exact collapse bug this RFC kills, reappearing on the streaming path). No such node exists today (e.g. ListGenerator yields per-item {item, index}, but its items propagate downstream as iteration-correlation envelopes that fire consumers N times → the N generation_complete land on the consumer at site #1, not on the generator, whose own timeline is not a savable sink). We scope multi-artifact-per-slot streaming generators OUT (§3 non-goal, Decision 1 corollary) and add a guard test (§13) asserting the documented limitation so a future node author hits a failing test, not a silent data loss. Constraints: Never routed through the runner.ts:1280 edge-suppression — generation_complete goes straight out via _emitMessage → _emit (runner.ts:999-1001, 1688-1708). Not audio-dropped — do NOT route it through the isAudioChunkOutputUpdate drop (runner.ts:1696-1703). It is pushed into _messages (which is truncated past MAX_RETAINED_MESSAGES = 10_000 via slice at runner.ts:1697-1700 — see §15 on what that does and doesn’t guarantee). No new backpressure: same fire-and-forget path as node_update, rate bounded by result count (low), does not touch inbox flow-control. Skip-result parity: for nodetool.constant.* / nodetool.input.* (the skipResult set at actor.ts:437-439), do not emit generation_complete — those aren’t generators. How their values still reach display sinks is specified in Decision 7. 6. output_update re-scope disposition drives append vs replace. runner.ts:1286-1294 sets disposition: "append" for streaming chunks (genProcess yields, audio), "replace" for whole-value emits. Every frontend consumer branches on it instead of hardcoding append (see §8.7 — there are 5 setOutputResult call sites, not 1), fixing the progressive-preview latent bug. Ephemeral clearing lifecycle. The outputResults buffer (ResultsStore.ts:609-694, keyed ${wf}:${job}:${node}) is cleared (a) on run start (existing clearResults/clearOutputResults at :424-443) and (b) on generation_complete for that (wf, job, node) artifact — once the artifact is committed as a generation, its live scratch buffer is stale. This keeps the display buffer from leaking partial chunks into the settled view. Edge-suppression is now harmless. No correctness depends on output_update reaching anyone — artifacts travel on generation_complete. The runner.ts:1280-1284 data-edge skip and the always_emit_output_updates opt-out remain as-is; they now only affect cosmetic live display, never persistence or variant counts. Audio path unchanged. isAudioChunkOutputUpdate (runner.ts:45-52), the live-stream-without-retention fast-path (runner.ts:1704-1706), queueAudioAppend / publishRealtimeAudioChunk (workflowUpdates.ts:513-523), realtimeAudioChunkBus.ts, AudioOutBody.tsx, useRealtimeAudioPlayback.ts — all untouched. Audio chunks are disposition: "append" and never become generations. Do not narrow value to chunk-only. Display sinks legitimately render whole non-generation values (a string into an Output node, chat string-streaming). Keep value: unknown. 7. Autosave cutover (server path) Move autosave from node_update{completed} to generation_complete — a hard switch, not dual-write: Add generation_complete to the processing gate at unified-websocket-runner.ts:1889-1892 so it gets node-type resolution, constant/input skip, and normalization. New branch (parallel to the deleted one), autosave before normalize (normalize strips inline bytes): if (outbound.type === "generation_complete") { const meta = this.getNodeMetadata?.(nodeType); if (meta?.auto_save_asset && outbound.outputs != null) { await autoSaveAssets(outbound.outputs as Record<string, unknown>, { userId: this.userId ?? "1", workflowId: active.workflowId, jobId: active.jobId, nodeId: String(outbound.node_id ?? ""), textOutputName: primaryTextOutputName(meta) }); } // then: normalizeOutputValue(outbound.outputs) — mirror :1958 on .outputs } Delete the node_update autosave at unified-websocket-runner.ts:1932-1953 in the same commit. Why hard-switch and not dual-write (corrected rationale). The original draft justified the hard switch by claiming the last-result object is a different instance on the two channels and would therefore double-save. That is inverted. Verified at actor.ts:1000/:1290: this._latestResult = outputs assigns the same object reference that _emitGenerationComplete(outputs) would carry — so on the process()/correlated path, node_update.result and the run-#N generation_complete.outputs are the same instance, and the reference-based idempotency guard (autoSaveAssets:468 keys on the in-place asset_id mutation at :518-520) would make a dual-write a no-op for that shared object, not a double-save. The genuine fresh-object case is the genProcess path (:994 does {...spread}), where the guard cannot see the prior save across channels. The hard switch is still required, for two real reasons: (a) Under-save, not double-save, is the dominant risk. Runs 1..N-1 of a multi-execution node exist only on generation_complete — node_update collapses to the last. A dual-write therefore under-saves the early runs regardless of object identity; the new channel must own autosave outright. (b) Cross-channel idempotency is reference-based and partial. It protects only the shared last-result object on the process() path; it does not protect the genProcess fresh-spread path, where dual-write would double-save the final stream-end artifact. Atomic swap is the only clean option. (If a dark-launch is mandated, gate both with one flag, never both ON.) Asset-volume / parity. A 6-image run now persists 6 assets (was 1), each tagged node_id, job_id. The consumer de-dupes by jobId: mergeGenerations (nodeGenerations.ts:74-82) drops live gens whose jobId already persisted, and all N variants share one jobId, so live N + persisted N collapse to N (not 2N). useNodeResultHistory.lastJobAssets (useNodeResultHistory.ts:75-81) already returns N assets for the last job → N tiles. The job_update{completed} asset reload (workflowUpdates.ts:712-725) now surfaces all N correctly. Note: persisted assets carry no index (§2), so mergeGenerations groups purely by jobId; this is sufficient for the live/persisted collapse but means replay idempotency must be addressed at the autosave layer (§15 / Decision 8). Storage volume note: flagged in §15 — runs that previously discarded intermediates now persist them; confirm desired for high-N generators (e.g. a 100-frame batch). 8. Frontend rewiring (all consumers) A census of every node_update / output_update consumer (grep across packages/, web/, mobile/) yields the surfaces below. The original draft covered only §8.1–§8.4; §8.5–§8.10 are the surfaces the consumer census surfaced. 8.1 workflowUpdates.ts reducer (web editor — primary) node_update branch (:866-1044): keep all lifecycle (error notification/state/ErrorStore/trace at :881-920 except the generation write; status/timing/cost at :931-974; property-merge at :1016-1043). DELETE the completed → upsertLiveGeneration write at :986-998. Keep a lightweight running → upsertLiveGeneration({status:"running"}) placeholder (:982-985) so a node shows a spinner before its first artifact (generation_complete only fires at commit). The node_update{error} → generation write (:894-901) stays on node_update (lifecycle owns errors; generation_complete represents committed artifacts only — see Decision 1). Both the running placeholder and the error settle are index-less patches — see §8.2 for how they route in the index-keyed store, and §9 for silent-scrub gating that must cover the running write too. output_update branch (:499-547): becomes display-only. Branch on disposition instead of hardcoded append=true at :515. Audio fast-path + logging + trace unchanged. NEW generation_complete branch: the sole generation driver. Appends/replaces by index, absorbs the silent-scrub gate (§9), clears the ephemeral outputResults buffer for the artifact. 8.2 ResultsStore.upsertLiveGeneration (:496-558) — DELETE the variant-inference, define index-less routing DELETE the Tier-2 “second completed appends a variant” hack: startsNewVariant (:522-525) + the synthesized-variant-id append branch (:527-548, the ${jobId}#${variantCount} scheme at :535). DELETE the isSilentJob-in-upsert special case (:519 + its comment :514-518). Silence moves to the message handler (§9). REPLACE the matcher with append/replace-by-index keyed ${workflowId}:${nodeId}. generation_complete{index:k} → set/replace slot k, id k===0 ? jobId : ${jobId}#${k} (preserve the back-compat id scheme — selected_generation values and tests depend on it). getLiveGenerations (:563-564) read API unchanged. Index-less-patch contract (NEW — addresses the under-specified seam). The surviving node_update{running} placeholder and node_update{error} settle carry no index. Specify: a patch without index targets the newest slot for that jobId — i.e. retain the existing findLastIndex(g => g.jobId === jobId) fallback (:507) only for index-less patches, while any generation_complete patch supplies an explicit index and uses index-keyed set/replace. This guarantees a node that errors before committing any artifact (no generation_complete, correct per Decision 1) still has its running placeholder settled by the index-less error patch — otherwise the index-keyed store has no slot for it and leaves a stuck running generation. 8.3 Preview / Output / content-card (web editor) PreviewNode.tsx (:235-298): DELETE the “accumulate output_updates as a variant array” reliance (the contract documented at :235-242). Preview reads its source node’s generations via the value edge — useNodeResultHistory(incomingValueEdge.source) (already wired at :277-284) / useNodeGenerations of the source — for discrete artifacts; streamBuffer (output_update) only for live text/audio. The sourceFallbackValue path becomes primary. OutputNode.tsx (:239-249): structurally unchanged — already layers streamBuffer ?? settledValue. Stream for live, generation (useNodeGenerations().current) for settled. ContentCardBody.tsx (:600-617): already generation-driven via useNodeResultValue/useNodeResultHistory. Its resolvePreviewValue array-flatten (:281-301) is now redundant for settled artifacts (runVariantValues covers N tiles); it still services the live in-flight buffer, so it stays. 8.4 SALVAGED UI — fed correctly, zero code change utils/nodeGenerations.ts (groupByRun/getCurrentRun/RunGroup/runVariantValues/mergeGenerations/assetToGeneration), useNodeGenerations ({runs, currentRun}), NodeHistoryViewer (the runs/variants/single navigator, defaultView flips to grid at variants.length > 1, :269-270), useNodeIO, nodeGenerationAccessor, useNodeExecState. These already expect index-keyed variants under a stable jobId; they just receive a clean feed instead of the node_update collapse. 8.5 In-browser execution path — normalize + autosave branches REQUIRED (was omitted) The dual-run-path architecture runs the same packages/kernel in a Web Worker, so generation_complete is emitted in-browser too. Two gaps: Normalize. browserRunner.worker.ts:187-203 (attachPreviewBitmaps / resolveImageRefForTransport) and browserWorkflowRunner.ts:435-441 (materializeBrowserOutputs / resolve) currently materialize only node_update.result and output_update.value. Add a generation_complete branch in both that applies the same resolve + materialize to .outputs (GPU read-back, raw-RGBA→PNG, inline-bytes→uri), or in-browser generation_complete carrying GPU refs / inline bytes renders broken and bitmaps are not transferred. The :432 “render identically” guarantee depends on it. Autosave. autoSaveAssets exists only in unified-websocket-runner.ts — the entire web/src/lib/workflow/ directory calls it nowhere (verified: zero matches). So in-browser generative runs do not persist via this code. Decision required (Decision 9): either (a) add a browser-side autosave hook keyed on generation_complete mirroring the server branch, or (b) confirm+document that browser-path persistence already round-trips to the server (the job_update{completed} → invalidateQueries(['assets']) + loadWorkflowAssets reload at workflowUpdates.ts:712-725 assumes the server already saved). The “every committed artifact is persisted” goal is unproven for the browser path until this is resolved; the rollout’s “each step shippable and green” claim is false for browser-eligible workflows until §8.5 lands. The silent-scrub case runs in-browser (§9), so this is on the critical path. Backfill. Confirm the browser relay applies the same job_id/workflow_id stamping the server does at unified-websocket-runner.ts:1855-1861; the bare actor emit (§5) relies on it. 8.6 Mobile (mobile/src/stores/WorkflowRunner.ts) — add a case, decide degradation Independent reducer: reads node_update.result (:354-359) and output_update.value (:384-396) into one nodeResults map; reads node_update only for errors otherwise. After this change, mobile keeps working at 1-artifact-per-node via node_update.result (the advisory last-result), which is acceptable graceful degradation. Required: (1) add a generation_complete case to the WorkflowRunner.ts switch — minimally store .outputs into nodeResults so multi-execution shows at least the latest, ideally a variant list if mobile surfaces variants; (2) regenerate mobile/src/api.ts message-type table (it currently has a generated output_update/node_update union with no generation_complete, :5005+); (3) state explicitly (Decision 10) that mobile’s continued read of node_update.result is an intended degradation, not an accident. Mobile is intentionally NOT a root workspace; build protocol first before its typecheck. 8.7 setOutputResult append/replace plumbing — 5 call sites, not 1 disposition must thread through ResultsStore.setOutputResult’s boolean append param. Verified call sites (excluding tests): workflowUpdates.ts:515, core/chat/chatProtocol.ts:514, hooks/timeline/useGenerateClip.ts:186, hooks/sketch/useGenerateLayer.ts:227. The original draft named only the first two. The timeline/sketch forwardWorkflowMessage paths call setOutputResult(..., true) unconditionally, so a disposition:"replace" value there would still append — re-introducing the exact progressive-preview bug elsewhere. Fix: change setOutputResult to take disposition (or derive append from it) and thread it through all four runtime call sites. (utils/imageRef.ts:51 is only a doc comment; update it too.) 8.8 Timeline & Sketch generation (useGenerateClip.ts, useGenerateLayer.ts) — migrate asset extraction Both share an identical pattern that makes output_update load-bearing for production asset extraction: on each message they jobOutputs.set(jobId, normalizeOutputUpdateValue(...)) gated on node_id === context.selectedOutputNodeId (useGenerateClip.ts:204-209, useGenerateLayer.ts:248-253), then at job_update{completed} call extractAssetId(jobOutputs.get(jobId)) (useGenerateClip.ts:228-235) and mark the whole job FAILED (“finished without producing an output asset”) if absent. Output nodes are terminal (no outgoing data edge) so output_update survives suppression today, but the new contract makes this a freely-droppable display artifact. Migrate the selectedOutputNodeId extraction from output_update to generation_complete.outputs (authoritative, never suppressed); keep output_update only for live preview inside the editors. This is the production path for both the video timeline and the sketch editor — it must move, not just honor disposition. 8.9 Mini-apps (web/src/components/miniapps/hooks/useMiniAppRunner.ts) — migrate result tiles Mini-app result tiles are built entirely from output_update: :84-103 creates one MiniAppResult per output_update (id node_id:output_name:timestamp) → upsertResult; node_update is read only for errors (:105-119). Under the new contract output_update is no longer the artifact carrier, so mini-app lists become incomplete/empty for edge-connected generators. Switch MiniAppResult construction to generation_complete.outputs (flatten per output handle for committed artifacts); keep output_update only for live-streaming display within a result. Mini-apps are a first-class surface. 8.10 Chat / agent path (web/src/core/chat/chatProtocol.ts) chatProtocol.ts independently consumes node_update (:1263) and output_update (:1294 → applyOutputUpdate at :492-521); it has no upsertLiveGeneration / generation path (only content_metadata.media_generation chunk handling at :430). Two requirements: (1) applyOutputUpdate must honor disposition (same demotion as §8.1; chat string-streaming at :523+ stays as legitimate whole-value display); (2) explicitly decide (Decision 11) whether the chat surface creates variants for a multi-execution generator invoked from a workflow tool, or intentionally ignores generation_complete (display-only). Recommend the latter for this cycle — document it — rather than leaving it implicit in a one-line note. If chat later needs variants it gets its own generation_complete branch. 8.11 CLI (packages/cli/src/websocket-client.ts) Protocol consumer used by nodetool workflows run --json and piped chat: handles node_update (:398-403, surfacing result) and output_update (:405-411, surfacing value); no generation_complete case. After the change a 6-image CLI run reports 1 collapsed result via node_update and the variants exist only on a type the CLI ignores. Decide (Decision 12): surface generation_complete events in the JSON stream (one event per artifact) or aggregate them; add the message-type entry to the CLI union (:36-38). Lowest-priority consumer but must not silently drop variants from --json output. 9. Silent-scrub handling useLiveSliderWriter reuses one jobId for an entire scrub, marks it silent (markJobSilent), and emits a running→completed arc per frame (runBrowserGraphJob({ jobId: previewJobIdRef.current })). Under the new model each frame emits generation_complete with incrementing index → without a gate, a 60-frame scrub = 60 variants. This runs in-browser (§8.5), so the browser-path normalize/autosave branches must exist for it. The gate relocates from ResultsStore.upsertLiveGeneration:519 into the new generation_complete handler in workflowUpdates.ts (frontend gate, Decision 2). Both writers that fire per scrub frame must honor it: // in the generation_complete handler: const slot = isSilentJob(jobId) ? 0 : index; // pin scrub frames to slot 0 upsertLiveGeneration(wf, node, jobId, { index: slot, outputs, status: "completed" }); // in the node_update{running} placeholder write (§8.1) — also fires per scrub frame: const slot = isSilentJob(jobId) ? 0 : undefined; // index-less for non-silent (newest-slot) upsertLiveGeneration(wf, node, jobId, slot === 0 ? { index: 0, status: "running" } : { status: "running" }); isSilentJob is already imported in workflowUpdates.ts:37. previewJobs.ts and useLiveSliderWriter are unchanged. Both halves matter: §8.2 deletes the isSilentJob special-case from upsertLiveGeneration, so if the per-frame running placeholder does not also pin slot 0, it would append/churn variants via the index-less newest-slot fallback. The covering test ResultsStore.variants.test.ts:154-183 (running→completed PER frame, 8 frames → 1 generation) relocates to a generation_complete-handler test that drives running + generation_complete per frame for 8 frames and asserts 1 live gen. Keep the kernel dumb (always increment index); silence is a pure slider-preview UI concern, so the gate lives frontend-side. The browser runner already owns the jobId; we do not push a silent flag into the kernel. 10. Python parity — VERDICT: TS-only for the targeted collapse. Zero nodetool-core change. Python nodes execute through the same NodeExecutor interface as TS nodes (node-executor.ts:127-171). PythonNodeExecutor implements only process() (python-node-executor.ts:223-237) and genProcess() (:239-258) — never run(). process() does the bridge RPC, calls materializeOutputs(), and returns a fully-materialized plain JS dict to the actor’s await; genProcess() yields materialized partials. For Python the actor assigns these to _latestResult at sites :369 (legacy single process()), :994 (genProcess stream-end), :1000 (correlated), :1290 (controlled-loop) — identically to TS. Two corrections to the “identical sites” claim (it was over-stated): Site #5 (run() / nodeOutputs.collected(), :357) is TS-only. PythonNodeExecutor has no run(); the actor’s run() path requires this._executor.run (actor.ts:286). A Python node flagged is_streaming_input therefore falls into the legacy single process() branch (actor.ts:358-371, site #3), never site #5. So for Python, site #5 is dead and site #357 is never reached. The zero-Python-change conclusion is unaffected; the coverage claim is corrected. Streaming-output multi-artifact nodes are out of scope (§5 site #4 invariant). Python streaming-output nodes deliver every artifact as a per-yield chunk frame (worker protocol.py:126-132), and the terminal result is deliberately empty (protocol.py:141-145, result_data={'outputs':{},'blobs':{}} for execute.stream). So for a Python generator that yields N distinct artifacts on the same slot, the only place those artifacts exist distinctly is the per-yield boundary — which we route to display-only output_update. The genProcess stream-end generation_complete would carry only artifact N. We scope this out (§3, Decision 1 corollary) rather than fix it, because the faithful fix would require the wire to distinguish a “complete artifact” yield from a progressive chunk — a final/complete marker on chunk frames — which would require a nodetool-core (worker/protocol.py) change, contradicting the zero-Python-change goal. No such node exists today; the §13 guard test documents the limitation. The bridge is pure request/response RPC keyed to request_id (python-bridge-base.ts:172-267) — it emits no node_update/output_update/generation/autosave concept. Therefore generation_complete is emitted entirely in the TS NodeActor. Python nodes get it for free for the targeted multi-execution collapse (the 6× ListGenerator→TextToImage case is a pure TS-actor correlated-drain phenomenon). No packages/protocol Python mirror, no bridge-protocol.ts change, no worker change. Python-side scope: none. 11. Protocol / serialization changes & full touch-point list msgpack is schemaless on both ends — server encodes with msgpackr pack (unified-websocket-runner.ts:7,1229), client decodes with @msgpack/msgpack decode (WebSocketManager.ts:242/249). No addExtension / Packr registry / type table anywhere. Any new type string round-trips for free; zero msgpack change. There is no zod schema for the streaming ProcessingMessage union (the api-schemas zod is REST chat-thread only) — zero zod change. Touch points (expanded to all consumers): packages/protocol/src/messages.ts: add interface GenerationComplete (after :178); add to the ProcessingMessage union (:633-653) — auto-flows into MessageType/MessageOfType, re-exported via index.ts:5. Add disposition? + done? to OutputUpdate (:201-210). packages/kernel/src/actor.ts: _emitGenerationComplete (bare, no job_id; node_name = node.name ?? node.type) + the 4–5 emit sites (§5); _generationIndex field. packages/kernel/src/runner.ts: set output_update.disposition at :1286-1294; ensure generation_complete is retained (not audio-dropped) in _emit (:1688-1708). packages/websocket/src/unified-websocket-runner.ts: add generation_complete to the processing gate :1889-1892; new autosave + normalize branch; delete node_update autosave :1932-1953. (Backfill at :1855-1861 already stamps job_id/workflow_id — no change.) web/src/lib/workflow/browserRunner.worker.ts + browserWorkflowRunner.ts: add generation_complete normalize/materialize branches (§8.5); confirm job_id backfill; add/confirm browser-path autosave (Decision 9). web/src/stores/workflowUpdates.ts: new generation_complete branch + silent-scrub gate (both writers); demote output_update to disposition-aware; delete completed→upsertLiveGeneration from node_update. web/src/stores/ResultsStore.ts: rewrite upsertLiveGeneration to append/replace-by-index with the index-less-patch contract; change setOutputResult to disposition-aware. web/src/core/chat/chatProtocol.ts: applyOutputUpdate honors disposition; documented generation_complete decision (Decision 11). web/src/hooks/timeline/useGenerateClip.ts + web/src/hooks/sketch/useGenerateLayer.ts: migrate asset extraction to generation_complete.outputs; thread disposition through their setOutputResult calls. web/src/components/miniapps/hooks/useMiniAppRunner.ts: build result tiles from generation_complete.outputs. mobile/src/stores/WorkflowRunner.ts + mobile/src/api.ts: add generation_complete case; regenerate message-type union; document node_update.result degradation (Decision 10). packages/cli/src/websocket-client.ts: add generation_complete to the union and the JSON event stream (Decision 12). 12. Rollout plan Ordered, each step shippable and green on both run paths (the original draft was green only for the server path): Protocol additive (no behavior change). Add GenerationComplete to the union + OutputUpdate.disposition?/done?. Build protocol. Nothing emits or consumes it yet. Back-compat: optional fields, additive union member — old clients ignore the new type. Kernel emits bare generation_complete at the 4–5 sites — no job_id and no index (per D8, both are stamped downstream: the server persist/relay seam derives index from DB ordering; the browser relay assigns an arrival-order index). Set output_update.disposition. Relay it through both the websocket (gate :1889-1892, backfill job_id+index, normalize .outputs) and the browser worker (assign arrival-order index, normalize .outputs, §8.5) without autosave yet. No consumer reacts → no behavior change. Add kernel emission tests (incl. the genProcess stream-end guard test). Frontend consumes generation_complete (new reducer branch + upsertLiveGeneration rewrite incl. index-less-patch contract + silent-scrub gate on both writers). Delete the node_update{completed}→upsertLiveGeneration write and the ResultsStore Tier-2 hack in the same commit. Migrate PreviewNode, mini-apps, timeline/sketch asset extraction, and mobile in the same wave (each is independently testable but must not be left reading the demoted channel for artifacts). Now the timeline shows N variants live. Autosave hard-switch. Add the generation_complete autosave branch and delete the node_update autosave atomically on the server (§7); land the browser-path persistence decision (Decision 9) in the same step. Now N assets persist on whichever paths are in scope. Demote output_update display (disposition-aware append/replace across all 5 setOutputResult sites + chatProtocol.ts; ephemeral-clear on generation_complete). Cosmetic. Cleanup: remove the now-dead node_update.result-as-artifact reads where a consumer fully migrated; tighten comments. (node_update.result survives as the documented mobile/skip-result degradation path.) Safe cutover sequence: steps 1–2 are invisible (additive + emit-only on both paths). Step 3 swaps the generation driver in one commit. Step 4 swaps the autosave driver in one commit. Never run both drivers of either {generations, autosave} simultaneously. Client/server version skew (NEW). Steps 1–2 (old-client/new-server) degrade safely — an old client ignores the new type and shows the old 1-variant behavior. The genuine risk is new-client/old-server: step 3 deletes the client’s node_update{completed}→upsertLiveGeneration write, but an old server never emits generation_complete → generations disappear entirely. Electron bundles client+server in lockstep (verified: web and websocket both 0.7.0-rc.23), so in-process is safe. For any remote nodetool serve deployment where web and the websocket server version independently, gate the step-3 deletion behind a server-capability probe, or keep the node_update generation write until the server is known to emit generation_complete. The optional NODETOOL_GENERATION_EVENTS flag stages a same-process cutover but does not cover client/server version skew. (Open — see §15.) 13. Test plan Kernel emission (packages/kernel): Correlated 6× generator (ListGenerator→TextToImage) emits exactly 6 generation_complete with index 0–5, each distinct outputs, one node_update{completed}. Controlled-loop node emits one per "run" event. genProcess node with a final whole-value yield: N output_update{disposition:"append"} yields + exactly one generation_complete at stream-end carrying the consolidated value (assert NOT per-yield, assert value is complete). Guard test (documented limitation): a genProcess node whose last yield does not carry the full value (overwrite-merge on one slot) loses prior content in the single generation_complete — assert this so a future multi-artifact-per-slot streaming node author hits a failing test, not silent loss. genProcess under correlation (multiple keys): one generation_complete per key. nodetool.constant.* / nodetool.input.* emit no generation_complete. generation_complete is never edge-suppressed (intermediate generator feeding a Preview still emits it) and never audio-dropped. Actor emit carries no job_id; assert the runner relay stamps it (server and browser paths). Autosave (packages/websocket): 6-gen run → 6 Asset rows, each with correct node_id/job_id. No double-save: assert node_update{completed} no longer triggers autoSaveAssets. Replay idempotency: re-emitting a fresh generation_complete object (asset_id unset, as a real reconnect would) is a no-op — requires the (job_id, node_id, index) guard from Decision 8, NOT just per-object asset_id. (Today’s per-object guard would duplicate.) Text-output autosave still fires off generation_complete.outputs. Browser path (web): In-browser ListGenerator→TextToImage→Preview end-to-end: 6 normalized variants render (assert generation_complete.outputs materialized — no broken images), and persistence matches Decision 9. Reducer / store (web): generation_complete{index:0,1,2} → 3 variants (rewrite ResultsStore.variants.test.ts:34-66). Index-less patch routing: node_update{running} then node_update{error} with no intervening generation_complete settles to exactly one errored generation (no stuck running). disposition:"append" concatenates; "replace" overwrites the display buffer — assert across workflowUpdates, chatProtocol, useGenerateClip, useGenerateLayer. node_update{completed} no longer mutates liveGenerations. mergeGenerations collapses live N + persisted N (same jobId) → N. Silent-scrub: 8 scrub frames, each driving node_update{running} and generation_complete under one silent jobId → 1 live generation (relocated from ResultsStore.variants.test.ts:154-183; covers both writers). Consumer surfaces: mini-app result list shows N tiles for a multi-execution generator; timeline/sketch extract the asset from generation_complete.outputs (not output_update); mobile shows ≥1 artifact via the new case; CLI --json emits N generation_complete events. 14. DECISIONS TO LOCK Status — locked 2026-06-20 D1 ✅ A — per-process() boundary; multi-artifact-per-output-slot streaming generators out of scope. D8 ✅ B (refined) — index from DB ordering, assigned at the persist/relay seam (not the actor). D9 ✅ Provider-generations only, always server-side; no browser autosave; browser normalize mandatory. D10 🔶 Provisional — mobile gets the generation_complete case + data now; full variant UI fast-follow (awaiting confirm). D11 ✅ B — chat shows variants. D2, D3, D4, D5, D6, D7, D12 ✅ accepted as recommended. Decision 1 — Generation boundary & list handling. Options: A) one generation_complete per process() result; list-valued handles carried as arrays, consumers flatten (runVariantValues / recursive autosave collect). B) fan a list-valued result into one generation_complete per element in the actor. Recommend: A — the actor already hands whole result dicts to _sendOutputs; passing the same dict avoids teaching the actor per-handle list semantics, and runVariantValues + autoSaveAssets’s recursive collect (:448-464) already flatten. Corollary (errors): keep node-level errors on node_update{error}; generation_complete represents committed artifacts only (no error variant in the shape). A run that errors before committing emits no generation_complete — correct. Corollary (streaming): the genProcess stream-end generation_complete carries the overwrite-merged _streamingCollectedOutputs, so multi-artifact-per-output-slot streaming generators are out of scope (§5/§10); guard-tested in §13. DECIDED: A — confirmed, including the streaming scope-out (multi-artifact-per-output-slot streaming generators are not covered this cycle; documented + guard-tested). Decision 2 — Silent-scrub gate location & mechanism. Options: A) backend pins index=0 for silent jobs (kernel must learn “silent”). B) frontend gate in the generation_complete handler and the node_update{running} placeholder: isSilentJob(jobId) → pin slot 0, else append by index (or newest-slot for index-less running/error). Recommend: B — silence is a pure slider-preview UI concern; the kernel stays dumb. The gate must cover both per-frame writers (generation_complete and the running placeholder), since §8.2 removes the isSilentJob special-case from upsertLiveGeneration. isSilentJob already imported in workflowUpdates.ts:37. Decision 3 — Autosave cutover. Options: A) dual-write (both node_update and generation_complete autosave during migration). B) hard switch (delete old, add new in one commit). Recommend: B — corrected rationale: runs 1..N-1 exist only on generation_complete (node_update collapses to the last), so dual-write under-saves the early runs regardless; and the cross-channel asset_id idempotency guard is reference-based — it protects the shared last-result object on the process() path (same instance at actor.ts:1000/1290, so dual-write would be a no-op there, NOT a double-save as the draft claimed) but does not protect the genProcess fresh-spread path (:994), where dual-write would double-save the final artifact. Atomic swap is the only clean option. (If a dark-launch is mandated, gate both with one flag, never both ON.) Decision 4 — output_update.disposition + ephemeral lifecycle. Options: A) add disposition: "append" | "replace" (+ optional done); clear the buffer on run-start and on generation_complete for that artifact; thread disposition through all 5 setOutputResult call sites. B) keep hardcoded append; clear only on run-start. Recommend: A — fixes the progressive-preview latent bug (workflowUpdates.ts:515 always-append) and makes whole-value snapshots correct. Absent disposition defaults to "append" for back-compat. Note the call-site count: workflowUpdates.ts:515, chatProtocol.ts:514, useGenerateClip.ts:186, useGenerateLayer.ts:227 — missing any re-introduces the bug in that consumer. Decision 5 — Keep the name output_update vs rename. Options: A) keep output_update. B) rename to display_update / output_chunk. Recommend: A — renaming churns the protocol union, both web reducers, chatProtocol.ts, mobile, CLI, the relay, and every display-sink component for a cosmetic gain; the disposition field already encodes the honest semantics. Revisit only if a future cleanup pass touches all sites anyway. Decision 6 — Do connected content-card/text nodes still emit output_update for live streaming? Options: A) yes — keep emitting for live token/preview display (now purely cosmetic). B) stop emitting for edge-connected handles (rely only on generation_complete). Recommend: A — live token streaming and progressive previews are real UX that generation_complete (fires only at commit) can’t provide. Edge-suppression already trims the firehose where it matters; the surviving emits are harmless display. Keep them. Decision 7 — How do constant/input/skipResult node values reach display sinks? Context: §5 skips generation_complete for nodetool.constant.* / nodetool.input.*; node_update.result is demoted to advisory; output_update is suppressible on edge-connected handles. That risks a value on none of the three authoritative channels. Options: A) these nodes’ downstream consumers are what emit generation_complete (the constant/input value flows as an input, and the consuming node commits its own artifact) — the constant/input node itself has no savable artifact and needs none; client-authored constants are already in the graph state, and server-resolved inputs surface via their consumer. Keep node_update.result as the explicit display fallback for the rare standalone constant/input preview. B) emit generation_complete for these too. Recommend: A — but state it explicitly rather than leaving the “client already holds the values” hand-wave: a constant/input feeding a connected node has its value carried into the consumer’s generation_complete; a standalone constant/input previewed directly reads node_update.result (the one sanctioned result read, scoped to skip-result node types). This reconciles “don’t read result for artifacts” (multi-execution generators) with “skip-result nodes use result” (they have no generation). Decision 8 — Replay/duplicate-asset idempotency. Options: A) per-object asset_id guard only (today). B) add a persisted-layer idempotency key on (job_id, node_id, index) — skip autosave for an already-persisted tuple. Recommend: B — a real reconnect/replay re-emits a fresh generation_complete (asset_id unset on a new object), so the reference-based guard does not fire → duplicate Asset rows sharing job_id but new ids. Since persisted assets carry no index today, add index to the autosave dedupe (either a stored column or an in-run (node_id, index) set per job) so replay is a true no-op. Without B, the §13 “replay is a no-op” test fails for genuine reconnects. DECIDED: B, refined — index is assigned from DB ordering at the server persist/relay seam, not by the actor. The unified-websocket-runner interception that backfills job_id (:1855-1861) persists the asset, derives index from the DB ordering of (job_id, node_id)’s assets, stamps it onto generation_complete before relay, and dedupes autosave on (job_id, node_id, index) → replay is a true no-op. Live ordering uses the stamped DB index (available because stamping happens in the same pre-relay pass). The browser path (no persist) falls back to an arrival-order index. Decision 9 — Browser-path persistence. Options: A) add a browser-side autosave hook keyed on generation_complete mirroring unified-websocket-runner’s branch. B) confirm + document that browser-path generative runs already round-trip persistence to the server (relying on the job_update{completed} asset reload), and that generation_complete in-browser is display-only. Recommend: decide before step 4 ships. The browser path calls autoSaveAssets nowhere (verified). If browser jobs do not reach the server autosave gate, option A is required or the “every committed artifact is persisted” goal fails for browser-eligible (and silent-scrub) workflows. Regardless of A/B, the browser normalize branch for generation_complete.outputs is mandatory (§8.5) for correct rendering. DECIDED: B — persistence is provider-generations only, always server-side; no browser autosave hook. Browser/non-provider generations are display-only and not persisted. Provider calls (fal/replicate/LLM/etc.) already execute server-side, so every persistable artifact hits the server autosave gate by construction. Option A (browser autosave) is rejected. The browser generation_complete.outputs normalize branch (§8.5) remains mandatory for rendering; only the persistence half is dropped from the browser path. Decision 10 — Mobile degradation. Options: A) mobile adds a generation_complete case but continues to show 1-artifact-per-node via node_update.result (graceful degradation, no variant UI). B) mobile renders full variant lists from generation_complete. Recommend: A for this cycle — state it as an intended degradation, regenerate mobile/src/api.ts types, and add the case so multi-execution at least shows the latest. Variant UI on mobile is a follow-up. DECIDED (provisional): add the generation_complete case + regenerate types now; full variant UI as a fast-follow. Mobile shows the latest immediately and does not block the cutover. Awaiting confirmation on whether mobile variant UI is wanted in this cycle. Decision 11 — Chat/agent path & generation_complete. Options: A) chat ignores generation_complete (display-only via output_update/media_generation); document why. B) chat gets its own generation_complete → variant handling. Recommend: A for this cycle — chat surfaces live media via existing chunk handling; workflow-tool multi-execution variants are not a chat UX today. Document the decision; don’t leave it implicit. applyOutputUpdate still honors disposition regardless. DECIDED: B — chat shows variants. Chat gets full generation_complete → variant handling (overrides the degrade recommendation). applyOutputUpdate still honors disposition. Decision 12 — CLI semantics. Options: A) surface each generation_complete as its own event in the --json stream. B) aggregate into the final result. Recommend: A — preserves the N-artifact information the kernel now exposes; add the type to the CLI union. Lowest priority but must not silently drop variants. 15. Risks & open questions Storage volume. N-per-run autosave means previously-discarded intermediates now persist (a 100-frame batch generator writes 100 assets). Confirm desired; consider a per-node auto_save_asset cap or an explicit “save intermediates” flag for very-high-N generators. (Open — needs product call.) Browser-path persistence (Decision 9). The single biggest open item: browser jobs call autoSaveAssets nowhere. Until Decision 9 lands, N-asset persistence is unproven for browser-eligible and silent-scrub runs. The browser normalize branch is mandatory independent of the persistence decision. Replay duplicate assets (Decision 8). Persisted assets carry no index; the only dedupe is the per-object asset_id mutation. A reconnect re-emits fresh generation_complete objects → duplicate rows. Needs (job_id, node_id, index) idempotency at the autosave layer. _messages truncation vs reconnect replay. generation_complete is pushed into _messages, but _messages is truncated to MAX_RETAINED_MESSAGES/2 past 10,000 (runner.ts:1697-1700) and is not the autosave source (autosave consumes the live relay stream in unified-websocket-runner, not _messages). Drop the draft’s “_messages authoritative for autosave” claim. For a very-high-N reconnect, early generation_complete entries can be evicted from replay; persisted assets recover them via the job_update{completed} reload — verify the reconnect replay path preserves index ordering for what survives, and document that very-high-N reconnects may drop early live variants from replay (persisted side recovers them). running placeholder + index-less patches. generation_complete fires only at commit, so the pre-first-artifact spinner relies on the node_update{running}→upsertLiveGeneration placeholder; a node that errors before any artifact must be settled by the index-less node_update{error} patch via the newest-slot fallback (§8.2). Tested in §13. job_id/node_name in the actor. Resolved (was asserted-resolved-but-wrong): the actor has no job_id; the runner relay stamps it at unified-websocket-runner.ts:1855-1861. node_name = node.name ?? node.type, not node.data?.title. The browser relay must apply the same stamping. Client/server version skew (§12). New-client/old-server erases generations (client deletes the node_update write; old server never emits generation_complete). Lockstep in Electron (verified 0.7.0-rc.23); for remote serve, gate the step-3 deletion behind a capability probe or keep the node_update write until the server is known to emit the new event. mergeGenerations jobId coupling. The live/persisted collapse depends on all N variants and N assets sharing one jobId (nodeGenerations.ts:74-82, assetToGeneration at :58-60). index must not leak into the jobId match. Verified the merge groups purely by jobId; confirm assetToGeneration keeps grouping intact when N assets land. All consumers migrated. The consumer census (§8.5–§8.11) must be complete; a missed reader of node_update.result/output_update.value as an artifact silently loses variants. Re-run the grep before sign-off. Appendix — Addressed review notes Findings verified against the working tree; each is either incorporated above or marked corrected-as-wrong with why. Double-save rationale inverted (high) — INCORPORATED + CORRECTED. Verified actor.ts:1000/:1290 assign _latestResult = outputs (same ref as the would-be generation_complete.outputs), and the autoSaveAssets guard (:468/:518-520) is reference-based. The draft’s “different instances → double-save” was inverted for the process() path (it’d be a no-op there); the real double-save risk is the genProcess fresh-spread path (:994). Rationale rewritten in §7 and Decision 3 around (a) under-save of runs 1..N-1 and (b) the genProcess-only double-save. genProcess stream-end overwrite-merge (high/critical) — INCORPORATED. Verified _streamingCollectedOutputs is Object.assign overwrite-merge (:984) then {...} (:994). Stated the load-bearing invariant in §5, scoped multi-artifact-per-slot streaming out (§3, Decision 1 corollary, §10), added a guard test (§13). Index-less running/error patches into an index-keyed store (medium) — INCORPORATED. Specified the index-less-patch contract in §8.2 (newest-slot-by-jobId fallback retained for running/error; explicit index only from generation_complete) + a §13 test for running→error-with-no-generation settling to one errored gen. 4./21. Actor job_id/node_name wrong-as-written (high/medium) — INCORPORATED + CORRECTED. Verified zero job_id in actor.ts; backfill at :1855-1861; _emitNodeStatus uses node.name ?? node.type. §5 emit snippet now omits job_id and uses node.name; §15 open question resolved. 5./15./16. Browser path autosave + normalize gaps (critical/high) — INCORPORATED. Verified autoSaveAssets only in unified-websocket-runner; browser normalize only handles .result/.value (browserWorkflowRunner.ts:435-441, browserRunner.worker.ts:187-203). Added §8.5, Decision 9, touch-point #5, browser test, and rollout/§3 corrections. 6./7. (Duplicate of 4 from a second reviewer.) — Same correction; consolidated. Silent-scrub running placeholder (high) — INCORPORATED. §9 now gates both the generation_complete write and the per-frame running placeholder on isSilentJob; §13 test drives both halves per frame. _messages retention vs “authoritative for autosave” (medium) — INCORPORATED + CORRECTED. Verified MAX_RETAINED_MESSAGES=10_000 slice (:1697-1700) and that autosave reads the live relay, not _messages. Dropped the “authoritative for autosave” claim; §15 documents truncation + reconnect implications. Replay duplicate assets (medium) — INCORPORATED. Verified Asset has node_id/job_id but no index; guard is per-object. Added Decision 8 ((job_id, node_id, index) idempotency) and a fresh-object replay test. New-client/old-server skew (medium) — INCORPORATED. Verified web and websocket are lockstep 0.7.0-rc.23. §12 documents Electron lockstep vs remote-serve risk and the capability-probe mitigation. 12./13. Python streaming-output + streaming-input site coverage (high/medium) — INCORPORATED. Verified PythonNodeExecutor implements only process()/genProcess() (no run()), so site #5 is TS-only and Python streaming-input uses site #3. Verified the empty terminal result for execute.stream. §10 corrected; streaming-output multi-artifact scoped out. Critical (mobile) — INCORPORATED (§8.6, Decision 10, touch-point #11). Critical (timeline/sketch) — INCORPORATED (§8.8, touch-point #9). Verified the selectedOutputNodeId/jobOutputs/extractAssetId/fail-job pattern. High (browser worker) — INCORPORATED (folded into §8.5). High (mini-apps) — INCORPORATED (§8.9, touch-point #10). Verified result tiles built entirely from output_update. High (browser autosave) — INCORPORATED (Decision 9, §15). Medium (CLI) — INCORPORATED (§8.11, Decision 12, touch-point #12). Medium (chat generation) — INCORPORATED (§8.10, Decision 11). Medium (constant/input leak) — INCORPORATED (Decision 7). Verified the skipResult set (actor.ts:437-439) and reconciled the result-read rules. Medium (setOutputResult call sites) — INCORPORATED (§8.7, Decision 4). Verified 4 runtime call sites + 1 doc comment. No findings were dropped as wrong; the two that were partially wrong (the double-save rationale and the _messages-authoritative claim) are corrected in-body rather than silently removed.--- # runpod-deployment-noext.md Source: https://docs.nodetool.ai/runpod-deployment This page has moved. Please go to Deployment Guide.--- # runpod-deployment-html.md Source: https://docs.nodetool.ai/runpod-deployment.html This page has moved. Please go to Deployment Guide.--- # Runtime Install Design: One Model for Desktop, CLI and Server Source: https://docs.nodetool.ai/runtime-install-design.html Runtime Install Design: One Model for Desktop, CLI and Server Status: proposed, revision 2. Scope: how NodeTool installs and finds optional software on every host: npm runtime libraries, the conda environment, Python packages and uv tools. It extends the Runtime Package Interface, which is implemented for the desktop app only. TypeScript node packs stay under the CLI Pack System. This design does not change its D2. Codes are stable: findings F1 to F44 (F9 to F44 come from two reviews), decisions D1 to D16, risks R1 to R9, questions Q1 to Q4. Section 12 maps every review finding to its resolution. 1. Summary Each host owns its install locations. The desktop app and the CLI keep separate npm roots, uv tools and micromamba binaries. They share only the conda environment, which is the largest item and already recorded in settings.yaml. One resolver in @nodetool-ai/config computes the locations of a host. One startup step puts them on the process environment. One registry in @nodetool-ai/protocol declares every runtime package. One installer package does the install work for the desktop Package Manager and nodetool runtime add. It follows the installer rules of the CLI pack spec: scripts off, plan check, verification, declared rebuilds only, one lock per target. A missing runtime produces one error. Its text names a command that works on the host that failed, and a bring-your-own command. 2. Current state 2.1 Install kinds The table gives defaults. The precedence rules follow it. Kind Default location Installer How the backend finds it Conda env macOS ~/nodetool_env. Linux ~/.local/share/nodetool/conda_env, or /opt/nodetool/conda_env when SUDO_USER is set. Windows %ALLUSERSPROFILE%\nodetool\conda_env, or %APPDATA%\nodetool\conda_env when ALLUSERSPROFILE is unset (electron/src/config.ts:34) micromamba create with ca-certificates only (electron/src/installer.ts:584). Python, uv, ffmpeg, pandoc, pdftotext, yt-dlp and rust install later as separate runtime packages the PATH that getProcessEnv builds (electron/src/config.ts) Python packages the env’s interpreter installRequiredPythonPackages (electron/src/python.ts:293) installs nodetool-core and caller additions. packageManager.ts installs Python node packs, with torch backend selection, CPU fallback and the requirements of packs already installed (:359, :970, :1113) PythonStdioBridge, see below uv tools userData/uv-tools/{tools,bin,python} (uvToolDirs.ts) uv tool install with the env’s uv (UvToolRuntimePackage.ts) the desktop PATH npm root userData/optional-node (NpmRuntimePackage.ts, nodePackManager.ts) the npm bundled with the app, system npm in an unpackaged build (config.ts:457). Cache in userData/npm-cache NODETOOL_OPTIONAL_NODE_MODULES, set at backend spawn (config.ts:352) Precedence rules: The desktop records the env path as CONDA_ENV in settings.yaml, under ~/.config/nodetool on macOS and Linux and %APPDATA%\nodetool on Windows (electron/src/settings.ts:70). micromamba comes from an explicit path, the bundled binary or PATH before the app downloads the pinned 2.3.3-0 into userData/micromamba (installer.ts:314). The Python bridge takes the pythonPath option or NODETOOL_PYTHON. Next it takes an active CONDA_PREFIX whose name or path marks a NodeTool env and whose interpreter exists. Last it tries a default list per platform, which includes Miniconda, Anaconda and /opt paths (packages/runtime/src/python-stdio-bridge.ts:466). userData is ~/Library/Application Support/nodetool on macOS. getNodetoolDataDir() is $XDG_DATA_HOME/nodetool (default ~/.local/share/nodetool) on macOS and Linux and %APPDATA%\nodetool on Windows. The database and assets have further overrides (packages/config/src/paths.ts:47). Other hosts: CLI (uncommitted work). nodetool runtime add installs npm libraries into <dataDir>/optional-node. It spawns npm from PATH, through a shell on Windows (packages/cli/src/commands/runtime.ts:98). It covers npm ids only. install.sh downloads micromamba 2.3.3-0 into <dataDir>/micromamba and creates the env at <dataDir>/conda_env. NODETOOL_HOME and --prefix change both paths (install.sh:537). Source checkout. Most runtime libraries are devDependencies of the packages that import them, so npm install provides them. node-llama-cpp and @fugood/whisper.node are peers only. npm run dev:server starts packages/websocket/src/server.ts directly and never applies the CLI’s optional root (scripts/dev-server.mjs:79). Docker image. It installs a curated SERVER_OPTIONAL_PACKAGES list with --ignore-scripts at build time (scripts/bundle-backend.mjs:225) and sets NODETOOL_OPTIONAL_NODE_MODULES (Dockerfile:163). /opt/venv is a tool venv for execute_bash (yt-dlp, PDF libraries) without nodetool-core. The image has no Python-node env, and the server disables the bridge in production unless NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 (packages/websocket/src/server.ts:2130). 2.2 Findings F1. The CLI can install only one of the four kinds. nodetool runtime add covers npm libraries. An npm user without the desktop app gets Python, ffmpeg or manimgx only through install.sh, a manual conda setup, or the system package manager. F2. No backend package reads settings.yaml. The CLI never sees a custom CONDA_ENV. F3. The Python bridge misses the install.sh env on macOS. Its macOS list is ~/nodetool_env, ~/miniconda3/envs/nodetool and ~/anaconda3/envs/nodetool. installation.md works around it by telling macOS users to set NODETOOL_PYTHON. F4. A library installed in the desktop app is invisible to the CLI. D2 keeps it that way on purpose. F5. The install rules for conda and uv live in Electron. electron/src/runtime/packages/definitions.ts holds them. Only the npm specs are shared (packages/protocol/src/runtime-npm-packages.ts). runtime-package-interface.md records this as “Electron-only for v1”. F6. Binary, Python and uv tool failures name no command that works outside the desktop. MissingBinaryError (video-nodes) and host-binaries.ts point at the Package Manager. The bridge throws a plain error (F38). The npm error names nodetool runtime add, which works on the installed CLI only (F37). F7. Install coordination is incomplete. runExclusive serializes only matching keys inside the desktop main process (electron/src/exclusive.ts:3). Nothing coordinates two processes. F8. The micromamba version is pinned twice. electron/src/installer.ts and install.sh both say 2.3.3-0. Nothing keeps them equal. 3. Prior art NodeTool is a runtime with plugins that ships on npm, not a library used inside a project. Tools of that kind own an install directory and run npm or a downloader into it, whatever way the tool itself was installed: Node-RED installs nodes as npm packages into ~/.node-red, records them in that directory’s package.json, and needs a restart to load them. (Node-RED: adding nodes) n8n installs community nodes into ~/.n8n/nodes with npm i <package>, then restarts. (n8n: manual installation) oclif CLIs (Heroku, Salesforce) install plugins at runtime with npm into the CLI’s data directory. (oclif plugin-plugins, oclif config) Playwright and Puppeteer download browsers into a user cache (~/.cache/ms-playwright, ~/.cache/puppeteer) and fail with one command to run: npx playwright install. (Puppeteer configuration) Libraries used inside a project (ESLint, Prettier) do the opposite: the user’s package.json owns the plugins. That model needs a project, which a NodeTool user does not always have. 3.1 Measured npm resolution A test with a stub CLI package on npm 11.16 showed where Node resolves a library that is installed after the CLI. Section 11 adds the fixture to the repo. How the CLI runs Library installed with Loads npm install -g cli npm install -g lib yes project install npm install lib in the project yes npx cli npm install -g lib no npx -p cli -p lib (same command) yes A global sibling library survives a CLI upgrade, because npm changes only the global packages a command names. It fails under npx. A library installed inside the CLI’s own package directory loads in every mode but disappears on a global CLI reinstall (pack spec E4). An owned directory reached through NODETOOL_OPTIONAL_NODE_MODULES needs one command in every mode and survives upgrades. npm 11.16 does not enforce allowScripts by default. It runs the install scripts of packages nobody reviewed and prints a warning. Only an explicit false entry or strict-allow-scripts blocks a script. The managed installer therefore turns scripts off and runs only declared rebuilds (D6). node-llama-cpp needs its postinstall. 4. Goals and non-goals Goals: Every host computes its locations through one function. nodetool runtime add <id> installs every kind that the desktop Package Manager installs, through the same installer code. Every missing-runtime failure names a command that works on the host that failed, plus a bring-your-own command. Two processes never change one install target at the same time. Non-goals: No shared npm root between the desktop app and the CLI (D2). No change to how TypeScript node packs install. The pack spec owns that. No file moves of existing installs. No install from serve, mcp serve, the job runner, chat or NodeTool agent tools (pack spec 13.1). An external agent that runs nodetool runtime add in the user’s shell acts as the user, as with any CLI command. The Docker image never installs at run time. No change to which packages exist. webgpu, node-web-audio-api, mediabunny with @mediabunny/server, @hyzyla/pdfium and chart.js are regular dependencies (pack spec D16), not runtime packages. No new Python packaging model. Python packs stay PyPI packages installed into the env’s interpreter. install.sh stays a standalone Python installer (D13). 5. Design 5.1 Hosts The host identity is the value of NODETOOL_HOST (D14). The entry point that starts the process sets it. A value that is already set wins. Host Set by npm root uv tools micromamba Conda env desktop Electron, in the backend environment userData/optional-node userData/uv-tools userData/micromamba CONDA_ENV, else the desktop default cli CLI entry, in global or local mode (pack spec 11.1) <dataDir>/optional-node <dataDir>/uv-tools <dataDir>/micromamba CONDA_ENV, else <dataDir>/conda_env npx CLI entry, when its path has an _npx segment as cli as cli as cli as cli source scripts/dev-server.mjs, and the CLI entry in dev mode as cli as cli as cli as cli docker ENV NODETOOL_HOST=docker in the Dockerfile /app/backend/optional-node, filled at build time none none none CONDA_ENV comes from settings.yaml (section 2.1). NODETOOL_CONDA_ENV overrides it, for tests and containers. A new env that the CLI creates goes to <dataDir>/conda_env on every platform, so a fresh Windows CLI has a per-user destination (F18). New macOS desktop installs use <dataDir>/conda_env too (Q1). Existing envs stay at their recorded path. 5.2 Layout of a CLI host <dataDir> ~/.local/share/nodetool, %APPDATA%\nodetool ├── nodetool.sqlite3 (unchanged) ├── optional-node/ │ ├── package.json {name, private, type: "module"} │ ├── .nodetool-install.lock (D10) │ └── node_modules/ ├── uv-tools/{tools,bin,python} ├── micromamba/ └── conda_env/ new envs only └── nodetool-env.json ownership marker (D3, F17) npm and uv use their default caches on the CLI host. The desktop keeps userData/npm-cache and userData/uv-cache. 5.3 The resolver and the startup step (@nodetool-ai/config) type NodetoolHost = "desktop" | "cli" | "npx" | "source" | "docker"; interface RuntimeLocations { host: NodetoolHost; optionalNodeRoot: string | null; // null on docker: the image sets its own uvTools: { tools: string; bin: string; python: string } | null; micromambaRoot: string | null; condaEnv: string | null; // null on docker } function runtimeLocations(env?: NodeJS.ProcessEnv): RuntimeLocations; function runtimeBinPath(locations: RuntimeLocations): string[]; function applyRuntimeEnv(env?: NodeJS.ProcessEnv): void; The desktop locations take userData as an argument from Electron. Backend packages never compute an Electron path. applyRuntimeEnv sets NODETOOL_OPTIONAL_NODE_MODULES and prepends runtimeBinPath to PATH. It never replaces a value that is already set. It sets the paths also when they do not exist yet, so a later install of an npm library or a binary loads without a restart. Three entry points call it: the CLI entry (it replaces applyOptionalNodeModules), the websocket server entry (which covers the dev server and the bundled server), and Electron’s getProcessEnv. The bridge tries interpreters in this order (Q2): The pythonPath option, then NODETOOL_PYTHON. On the source host only, an active NodeTool CONDA_PREFIX. runtimeLocations().condaEnv. On the other hosts, an active NodeTool CONDA_PREFIX. The current default lists, kept for envs that nothing recorded (F12). 5.4 The registry (@nodetool-ai/protocol) One declarative entry per runtime package. runtime-npm-packages.ts grows into it, and definitions.ts in Electron becomes an adapter that builds RuntimePackage instances from it. interface NpmComponent { name: string; version: string; // exact pin, installed by `add` range: string; // what a host accepts as installed } type RuntimeInstallSpec = | { kind: "npm"; components: NpmComponent[]; installScripts?: string[] } | { kind: "conda"; condaPackages: string[]; binaries: string[]; windowsBinSubdir?: string } | { kind: "uv-tool"; toolSpec: string; python: string; binaries: string[] } | { kind: "provided"; binary: string }; // nodejs: the host supplies it interface RuntimePackageSpec { id: RuntimePackageId; name: string; description: string; category: "language" | "tool" | "library"; platforms?: readonly NodeJS.Platform[]; dependsOn?: readonly RuntimePackageId[]; approxSizeMB?: number; install: RuntimeInstallSpec; postInstall?: "python-core"; // python: install nodetool-core bringYourOwn: { kind: "path"; binary: string } | { kind: "env"; variable: string } | { kind: "npm" }; } The micromamba bootstrap is not a runtime package. The installer runs it before the first conda install. Python node packs stay in PYTHON_NODE_PACKS. 5.5 The installer (@nodetool-ai/runtime-install) A Node package with no Electron imports. The Electron main process and the CLI command layer import it. No package that websocket, agents or runtime depends on imports it, and it refuses to run in a process that serve or mcp serve marked (pack spec 13.2). Operation Implementation npm libraries The pack spec 14.1 invocation: npm-cli.js beside process.execPath, never npm.cmd through a shell, npm 10.9.0 or later. Plan with --dry-run, install with --ignore-scripts --save-exact, check versions and integrity, run npm rebuild <name> --ignore-scripts=false for installScripts entries only, load-check in a child process. micromamba bootstrap explicit path, bundled binary, PATH, else download MICROMAMBA_VERSION and check the SHA-256 for the platform before the first run (D11) conda env micromamba create -p <env> when the env is absent, then write nodetool-env.json. micromamba install -p <env> for packages. A reinstall or removal of the env requires the marker (F17). Python core and packs uv pip install --python <env python>, with the policy moved from packageManager.ts: torch backend detection, CPU fallback, torch family reinstall, the requirements of installed packs, and a check after install (F15) uv tools uv tool install with the environment from UvToolRuntimePackage.ts status per component: installed, missing, incompatible (outside its range) or broken (present but fails the load check) uninstall refuses when another installed package lists it in dependsOn. Never touches an env without the marker. Reports a file that a running process holds (Windows) with the process name. Every operation emits RuntimeProgress events and runs under the lock of its target (D10). dependsOn takes the locks in a fixed order: micromamba, then the env, then the target. 5.6 CLI commands Command Status Does nodetool runtime list [--json] exists, npm only every package: id, name, status, version per component, location, install command nodetool runtime add <id...> [--reinstall] exists, npm only. conda, uv and Python in step 4 install the running host’s pins, with dependsOn first. --reinstall replaces the repair operation. nodetool runtime remove <id...> exists, npm only uninstall nodetool runtime path exists print the npm root add is also the update: it installs the pins of the running CLI. There is no update or repair command. add prints the restart line of section 5.7 when the package needs one. 5.7 Error contract One error type replaces MissingBinaryError, the host-binaries.ts message and the plain bridge errors. It keeps the MissingRuntimePackageError name and the missingRuntimePackageOf lookup, so the kernel actor and the editor’s install offer keep working. interface MissingRuntimeDetails { runtimePackage: RuntimePackageId; missing: string; // the specifier or binary that failed reason: "missing" | "incompatible" | "broken"; managed: string | null; // the command for this host bringYourOwn: string | null; restart: string | null; // set when the fix needs a restart } missingRuntimeText(details) builds the message from NODETOOL_HOST. No host registers anything. The npm example is pdf-js, which loadLiteParse throws for @llamaindex/liteparse (packages/document-nodes/src/lib/pdf-text.ts:24). Strings marked “new” need code from section 7. Host managed for pdf-js bringYourOwn for pdf-js desktop Install PDF Libraries in Settings › Packages. npm install --prefix "<userData>/optional-node" @llamaindex/liteparse@1.5.3 pdf-parse@2.4.5 pdf-lib@1.17.1 cli nodetool runtime add pdf-js the same npm command with <dataDir>/optional-node npx (new) npx -p @nodetool-ai/cli@<running version> nodetool runtime add pdf-js as cli source (new) npm run dev:nodetool -- runtime add pdf-js as cli docker (new) The server image does not include PDF Libraries. Rebuild it with --build-arg NODETOOL_RUNTIME_PACKAGES=pdf-js. Or mount a node_modules directory that holds @llamaindex/liteparse@1.5.3, pdf-parse@2.4.5 and pdf-lib@1.17.1, and set NODETOOL_OPTIONAL_NODE_MODULES to it. The npx string names -p and the bin, because the CLI package has two bins (nodetool and nodetool-chat). The bring-your-own npm command is a plain npm install. It runs install scripts as the user’s npm configuration allows, which is the user’s choice. An npm library needs no restart, because applyRuntimeEnv set its root at startup (5.3). A binary needs no restart for the same reason. The Python bridge (D16). PythonStdioBridge.ensureConnected throws MissingRuntimePackageError with id python when no candidate exists (python-stdio-bridge.ts:148). The ensurePythonBridge path in server.ts passes it on. A workflow that names a node type from a PYTHON_NODE_PACKS namespace while the bridge is unavailable fails with the same error, not with an unknown node type. The server checks the bridge once at startup (server.ts:2075), so every Python fix on a server host needs a restart. Host managed for python restart desktop Install Python in Settings › Packages. none cli, npx, source before step 4 curl -fsSL https://raw.githubusercontent.com/nodetool-ai/nodetool/main/install.sh \| bash Restart nodetool serve. when a server failed cli after step 4 nodetool runtime add python (npx and source use their prefix) as above docker This server image has no Python worker. Set NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 and NODETOOL_PYTHON to a Python 3.11+ that has nodetool-core installed. Restart the container. bringYourOwn for Python on every host except docker: Or set NODETOOL_PYTHON to a Python 3.11+ that has nodetool-core installed. For a conda binary it is Or put ffmpeg on PATH (brew install ffmpeg, apt install ffmpeg). For a uv tool it is Or put manimgx on PATH (uv tool install manimgx). An example message on the cli host: The optional package @llamaindex/liteparse (PDF Libraries) is not installed. Install it, then run the command again: nodetool runtime add pdf-js or with npm: npm install --prefix "<dataDir>/optional-node" @llamaindex/liteparse@1.5.3 pdf-parse@2.4.5 pdf-lib@1.17.1 Structured forms (D12): CLI --json output carries the details as error: "missing_runtime_package". The CLI exits with 3, the code the pack spec uses for missing_pack. Callers check the discriminator, not the code alone. RunSandboxResult keeps missingRuntimePackage: RuntimePackageId and adds missingRuntime: MissingRuntimeDetails (packages/agents/src/js-sandbox.ts:2773). The kernel actor puts the details on the node error message, next to the id. 5.8 Desktop integration The Package Manager UI, the IPC surface and the progress rendering stay in Electron. Their implementation calls the installer package. getProcessEnv calls runtimeLocations for the desktop host and sets NODETOOL_HOST=desktop. The paths do not change. The desktop and the CLI share only the conda env. The installer refuses a nodetool-core change outside the range that the other host recorded in nodetool-env.json, and names the update of that host (D3). 5.9 Server image SERVER_OPTIONAL_PACKAGES stays curated. A test asserts that each entry belongs to a registry npm component (F20). A build argument, NODETOOL_RUNTIME_PACKAGES, adds registry ids to the image. The build installs them with --ignore-scripts and runs their declared rebuilds, as the installer does. The image sets NODETOOL_HOST=docker. 6. Decisions D1. One resolver, locations per host (revised). runtimeLocations computes every location from the host (5.1). Only the conda env is shared. New CLI envs go to <dataDir>/conda_env on every platform. Discovery of legacy envs stays. D2. No shared npm root (revised). The desktop keeps userData/optional-node. The cli, npx and source hosts use <dataDir>/optional-node. A shared root breaks when the desktop and the CLI run different releases (F11). It also needs a merge of populated trees (F13) and a migration that depends on the network (F19). TypeScript node packs stay under the pack spec, whose D2 does not change. D3. Pins install, ranges accept, per component (revised). add installs the running host’s exact pins. A component outside its range has status incompatible, and the error names add. This replaces the newest-compatible rule of the runtime package interface (F29). In the shared env, each host records its accepted nodetool-core range in nodetool-env.json. D4. Native npm packages use Node-API. The CLI runs whatever Node the user switches to. The installer’s load check in a child process detects an ABI failure as broken. @fugood/whisper.node is unverified (Q3). D5. Replaced by D10. D6. Install scripts run only for declared packages (revised). The installer installs with --ignore-scripts and rebuilds only the installScripts entries, with pinned version and integrity (pack spec 14.4). allowScripts is not a security boundary in npm 11.16 (F9). D7. Withdrawn. No files move (D1, D2). D8. runtime add covers every kind, from the CLI command layer only (revised). The servers, the job runner, chat and NodeTool agent tools never install (pack spec 13.1). There is no install capability. An external agent in the user’s shell runs the same command a person runs. D9. Every kind keeps a bring-your-own path, and the error names it. PATH for binaries, NODETOOL_PYTHON for Python, npm install --prefix for npm. D10. One lock per install target (revised). The lock file sits inside the target’s realpath: the npm root, the env, the uv tools root, the micromamba root. It follows pack spec 14.3: exclusive create, a heartbeat, staleness after 60 seconds without a heartbeat. A lock inside the shared Windows env protects every user (F14). A manual npm or uv command and older desktop releases do not take the lock. D11. One micromamba pin with hashes (revised). The installer package exports MICROMAMBA_VERSION and a SHA-256 per platform. A script generates the matching block in install.sh. A test compares them (F26). D12. The missing-runtime error is structured (revised). The details travel in --json output, tool results, node errors and the sandbox result. The existing id field stays (F28). D13. Withdrawn. install.sh stays a standalone Python installer with its options. Only its micromamba block changes (D11, F27). D14. The host identity is NODETOOL_HOST. The entry point sets it. The error text and the resolver read it. This replaces a phrasing registry (F41). D15. One startup step for every entry point. applyRuntimeEnv runs in the CLI entry, the websocket server entry and Electron’s getProcessEnv. It never replaces a value that is already set (F21). D16. A missing Python is a missing runtime package. The bridge and the node resolver throw MissingRuntimePackageError with id python (F38). 7. Migration Each step ships alone. The error names only commands that work on the host at that step (F43). Lock, resolver and host identity. Add the lock (shared with the pack spec), runtimeLocations, applyRuntimeEnv and NODETOOL_HOST in every entry point. The bridge adds the resolver env to its candidates and keeps its lists, which fixes F3. The CLI root gets "type": "module", and the CLI runs npm by the pack spec 14.1 rule. No files move. Error contract. Add MissingRuntimeDetails, missingRuntimeText, the bridge mapping (D16), the sandbox field, the CLI --json form and exit code 3. npm ids name runtime add. Python names install.sh on the CLI hosts. Conda and uv ids name only their bring-your-own command outside the desktop. Registry. Move the conda and uv specs from definitions.ts into the protocol registry, with per-component ranges. Electron’s definitions.ts becomes an adapter. Add the server subset test and the Docker build argument. Installer package. Move the install code and the packageManager.ts policy out of Electron. The desktop switches to it with no visible change. runtime add gains conda, uv and Python. The CLI error texts switch from install.sh to runtime add python. Docs. Update the pages in section 8. 8. Document changes Page Change cli-pack-system-design.md Runtime libraries that a pack declares install into the host’s npm root through runtime add, not into the pack install root. The lock implementation is shared. runtime-package-interface.md Replace “Electron-only for v1” with a link to this design. Replace the newest-compatible version rule with D3. installation.md Remove the macOS NODETOOL_PYTHON workaround after step 1. Use nodetool runtime add python after step 4. cli.md The runtime commands of section 5.6. docker-production-deploy.md The NODETOOL_RUNTIME_PACKAGES build argument. packages/AGENTS.md How to add a runtime package: registry entry, optional peer, Node-API rule, install scripts, error test. Link from electron/src/AGENTS.md. 9. Risks R1. Withdrawn. No migration moves files. R2. Split data directories on Linux. A Flatpak desktop or a per-app XDG_DATA_HOME gives the CLI a different settings.yaml, so the CLI does not see the desktop env. nodetool runtime list prints the resolved env path. This design does not support sharing across that boundary (F22). R3. Large installs from the CLI. Python with uv takes several hundred MB, torch takes gigabytes. add prints approxSizeMB before it starts and installs only on an explicit command. R4. Two Node versions on one CLI root. D4 and the load check report a broken status with the --reinstall command. R5. Windows shared env. The CLI uses an existing shared env when CONDA_ENV names it and takes the lock inside it. When the env is not writable, the error names an elevated shell. The CLI never creates an env there. R6. Scope. The change crosses Electron, config, protocol, runtime, cli, video-nodes, websocket and the Docker build, in five steps. R7. Old npm. The installer requires npm 10.9.0 or later (pack spec D15) and names the upgrade command. R8. Duplicate disk use. A machine with the desktop app and the CLI holds two copies of each npm library and uv tool it uses (D2). The conda env, the largest item, stays shared. R9. nodetool-core skew in the shared env. A desktop and a CLI of different releases can need incompatible nodetool-core versions. D3 refuses the change and names the update of the other host. 10. Open questions Q1. Decided. New macOS desktop installs create the env at <dataDir>/conda_env. Existing installs keep their recorded path. Q2. Decided. Section 5.3 gives the order. An active CONDA_PREFIX wins over the configured env only on the source host. Q3. Open. Does @fugood/whisper.node 1.1.3 use Node-API on every platform? webgpu and node-av are core dependencies now (pack spec D16), so no host installs them into its npm root. Q4. Moved to the pack spec. Pack precedence between a project and an install root is a pack question (F30). 11. Verification Test Proves Location runtimeLocations per host and platform, with and without CONDA_ENV, NODETOOL_CONDA_ENV and XDG_DATA_HOME D1, D14 packages/config/tests/ applyRuntimeEnv never replaces a set value and sets absent paths D15 packages/config/tests/ The exact error text per host and kind, with real paths D9, D12 packages/protocol/tests/ A clean-HOME run of each printed managed npm command against a local test registry, for cli, npx and source 5.7 scripts/__tests__/, on request, with the pack spec 19.3 matrix The never-install matrix: the installer refuses in a serve or mcp serve process, and no capability installs D8 packages/cli/src/commands/__tests__/runtime.test.ts, packages/agents/tests/ A missing runtime round trip: node error, sandbox result, CLI --json with exit code 3 D12 packages/kernel/tests/, packages/cli/src/commands/__tests__/ The bridge with no candidate, and a Python node type with the bridge unavailable, throw id python D16 packages/runtime/tests/, packages/websocket/tests/ Every id has a spec. Each SERVER_OPTIONAL_PACKAGES entry is a registry component. install.sh has the generated micromamba block D11, 5.9 packages/protocol/tests/, scripts/__tests__/ The installer with a stub npm and a stub micromamba: argv, --ignore-scripts, declared rebuilds only, env marker check D6, D10, F17 packages/runtime-install/tests/ Two processes start an install on one target at once, and one waits D10 packages/runtime-install/tests/ The npm resolution matrix of section 3.1 3.1 scripts/__tests__/, on request 12. Review resolutions Finding Resolution F9 3.1 corrected. D6 uses --ignore-scripts and declared rebuilds. F10 Packs stay under the pack spec. A runtime install never changes a pack tree, because the roots are separate (D2). F11 D2: no shared npm root. D3 and R9 cover the shared env. F12 5.3 keeps the legacy lists. No files move. F13 D2 and D7: no move, no merge. The lock ships in step 1. F14 D10: the lock sits inside the target. F15 5.5 moves the packageManager.ts policy. F16 D8 follows the pack spec. F17 5.5: micromamba create, ownership marker. F18 D1: new CLI envs are per user on every platform. F19 D7 withdrawn. Nothing migrates. F20 5.9: the list stays curated, with a subset test. F21 D15. F22 R2: no sharing across a Flatpak or XDG boundary. F23 5.4 per-component ranges. 5.5 status values. F24 5.4 provided kind, postInstall, bootstrap outside the registry. F25 5.5 uninstall rules. F26 D11. F27 D13 withdrawn. F28 5.7 keeps the id field and adds missingRuntime. F29 D3, R7, section 8. F30 Q4 moved to the pack spec. F31 to F34 Section 2.1 corrected. F35 F6 and F7 corrected. F36 3.1 corrected. F37 5.1 and 5.7 host tables. F38 D16. F39 5.6 status column. F40 Step 1 fixes the manifest and the npm invocation. 5.6 adds location and install command to list. F41 D14 replaces the registry. repair and update are cut. The cache directories are cut. The micromamba pin moves to the installer. path stays, because it exists and costs nothing. F42 Section 11. F43 Section 7 step 2. F44 5.7 uses pdf-js as the example.--- # Sandbox Package System — JS/WASM packages for the QuickJS sandbox Source: https://docs.nodetool.ai/sandbox-package-design.html Sandbox Package System — JS/WASM packages for the QuickJS sandbox Design for letting NodeTool packs ship code that runs inside the QuickJS sandbox: JS guest modules the Code node and CodeAct steps can import, and WASM modules bridged in from the host. Distribution rides the existing package manager — the nodetool manifest in a pack’s package.json, the npm install flow, and the registry index. Status: design, revised after two review rounds. M0 through M6 are implemented. A declared sandbox module is importable in Code nodes, CodeAct actions, the CLI, the server and the browser runner — M2 delivered module source to the browser over GET /api/sandbox-modules/*, ran the same loading/denial contract there, and removed the NODETOOL_SANDBOX_MODULES_V1 parity flag along with the browser refusal it covered. M3 adds the npm compiler: a config-only npm manifest entry is bundled, scanned, admission-probed, and cached by content. M4 adds scalar WASM modules behind generated facades. M5 adds trust-scoped disclosure: a pack’s SKILL.md reaches agents as a skill or a docs tool, and the package picker in the editor. M6 makes import the only way to reach a library: eight packs, host-facade modules for the libraries the guest cannot hold, and no data.* globals. M0 provides: protocol schemas, non-executing discovery, the catalog contract and its concrete host, catalog injection into server and CLI contexts, the sandbox-only host-loader guard, scripts-disabled installation with an install ledger, and the integrity-verified trust/rebuild flow with its Package Manager surface. The first review killed four assumptions (corrected under “Facts”); the second pinned the contracts that were still loose — CodeAct session consent, the untrusted-doc policy, exact scalar WASM signatures and instance ownership, memory-manifest semantics, generated facades, cycle-safe catalog placement, capability-free admission probes, exact-artifact installation, and the M1 parity flag. M-1 is approved to start; M0 onward implements this text. Problem Every library the sandbox offered was a host bridge on a global: papaparse, cheerio, exceljs, fflate and friends ran on the host, lazily imported inside their bridge in packages/agents/src/js-sandbox.ts, and the guest reached them through a data.* namespace it did not ask for. That kept the guest surface identical across dev, packaged Electron, and the in-browser runner — but it had a hard ceiling: Adding a library meant editing js-sandbox.ts, the sandbox manifest, and the drift tests, then shipping a NodeTool release. Third parties could not extend the sandbox at all. A pack that ships nodes had no way to ship helper code for Code nodes in its example workflows. Compute-heavy pure functions (parsers, codecs, geometry, hashing) are exactly what WASM is for, and there was no way to bring one. And the surface was two surfaces: some libraries were globals, some were imports, and which one a given library was depended on nothing a user could see. The goal: a pack installs through the normal package manager and its sandbox modules become importable in the guest — without weakening the sandbox’s security contract or breaking surface parity across the three runtimes. Facts the design must respect The engine is quickjs-ng (@sebastianwessel/quickjs ^3 over quickjs-emscripten-core). A fresh runtime/context per invocation; 64 MB memory limit, 512 KB stack, interrupt handler on a deadline; eval/Function deleted so the guest cannot self-generate code. quickjs-ng has no WebAssembly global. WASM cannot execute inside the guest. “WASM in the sandbox” therefore means: WASM instantiated on the host (Node and browsers both have WebAssembly), reached from the guest through a bridge — the same pattern as every existing library. wrapCode() cannot contain an import. User code is emitted as the body of an async IIFE inside the module (js-sandbox.ts:2194: export default await (async () => { ... })();), so an import statement in user code is a syntax error where it lands. Supporting imports requires an AST transformation that hoists static imports out of the IIFE — touching implicit returns, Code-node and CodeAct preludes, streaming rewrites, source locations, and validation. And because an ES module’s dependencies evaluate before the entry body, the timer-deletion hardening wrapCode emits would run after imported modules execute. Hardening must be applied before any loaded module evaluates, by the loader or an equivalent mechanism — not by the entry module. The wrapper’s virtual FS is not empty. createVirtualFileSystem in @sebastianwessel/quickjs unconditionally mounts Node-compat modules (buffer, fs, path, process, timers, url, …) into the guest’s node_modules, and its normalizer maps node:* specifiers onto them — await import("node:buffer") succeeds under the wrapper’s defaults. Passing nodeModules does not mean “nothing else resolves”. The allowlist must therefore be enforced by a custom module loader and normalizer at runtime; static validation alone cannot hold it, because a computed dynamic import (import("node:" + "buffer")) bypasses any declaration check. The pack system already carries manifests. Third-party packs are npm packages with a nodetool field in package.json, discovered by packages/node-sdk/src/pack-loader.ts, gated by a trust allowlist, installed by Electron into <userData>/optional-node, and listed in the registry index. Loading a pack with a register export means running it in-process with full server privileges, which is why untrusted packs are skipped. npm install executes lifecycle scripts unless disabled. The Electron installer must use npm install --ignore-scripts <spec> before discovery reads the installed manifest. “Installing a sandbox-only pack runs no pack code” depends on that flag. This is part of the trust model, not later hardening. Host-side value marshaling is settled. Typed arrays serialize guest→host natively; host→guest bytes travel as tagged base64 revived by a guest prelude. Async bridge errors return tagged objects, never rejected promises. New bridges must follow these rules. Packaged Electron flattens paths. Data files a package loads at runtime must be registered so bundle-backend.mjs stages them and verify-backend-bundle.mjs checks them. Design One import surface, three module kinds Every library the sandbox offers is an importable module. There is no second route and no library global. What differs between kinds is where the code runs, not how the guest reaches it. Kind Runs Capability model Risk JS guest module (kind: "js") inside QuickJS, in the same context as the node’s code everything the node’s code can reach: fetch, workspace, getSecret, asset bridges, injected globals; it can also mutate globals and prototypes for the rest of the invocation cannot escape QuickJS, but shares the node’s granted bridges — importing a module means trusting it with that node’s capabilities Host WASM module (kind: "wasm") on the host, behind a generated facade no imports: no I/O, no syscalls, no host references; memory bounded by binary validation host CPU and memory, bounded by worker budgets Host JS module (kind: "host") on the host, behind a generated facade a NodeTool-implemented function set, named by id. A pack declares the id; it never supplies the code host CPU and memory, bounded by the implementation’s own caps A guest JS module is the default answer. WASM is for compute a JS module cannot do at acceptable speed. A host JS module is for a library the guest cannot hold — one that needs Node builtins, a DOM, or a limit the guest could not enforce on itself. A pack can pair kinds: a WASM module plus a guest JS wrapper that gives it an ergonomic API. Capability globals are a different thing and stay as they are: fetch, workspace, getSecret, the asset bridges, format.*, image.*, audio.*, video.*, canvas.* and crypto.* are what the node granted this run, not libraries it chose. Trust model: explicit dependency consent The first draft claimed guest modules are capability-free and need no trust. That was wrong — a malicious dependency can exfiltrate through fetch, read what getSecret grants the node, act at module initialization, and poison prototypes for the invocation. The corrected statement: a guest module cannot escape QuickJS directly, but it shares the node’s granted bridges. v1 adopts explicit dependency consent: Importing a module means trusting it with that node’s sandbox capabilities. The Package Manager and the Code node UI say this in those words at install time and in the package picker. Sandbox-only packs (no register) still clear a far lower bar than register packs — no host execution, no lifecycle scripts (below), and the sandbox’s hard limits — so they install without the register-pack allowlist. But the UI presents them as “runs inside your workflows with the node’s capabilities”, never as “no trust needed”. Capability attenuation (packages in a separate QuickJS context with data-only arguments) is the stronger model and stays on the table as a later opt-in (isolation: true on a declaration); it is not v1 because it abandons normal in-context ESM semantics. Installation must not execute pack code, and classification must read the exact artifact. Registry metadata can describe a different artifact than what a tag resolves to at install time, and a package can change from sandbox-only to register between versions. The flow is therefore: Every initial install runs with --ignore-scripts. The installed manifest — the exact resolved artifact — is inspected. Sandbox-only: done. A register pack needing lifecycle scripts: request trust, then run scripts against the artifact already on disk (rebuild flow), or — if a reinstall is unavoidable — record the lockfile’s resolved URL and integrity from step 1 and verify the refetched artifact’s integrity is identical before scripts run. A version number is not integrity: a mutable private registry can serve a different artifact for the same version. Note that --ignore-scripts also suppresses dependency lifecycle scripts, not only the top-level pack’s — the rebuild flow must cover those too. The install UI shows which mode applied. The security-sensitive decision never rests on a registry summary. This lands in M0. SKILL.md is third-party prompt content, and quoting is risk reduction, not isolation. A model can follow malicious instructions inside quoted documentation; no delimiter changes that. The v1 policy: Trusted pack: its skill registers through the normal skill system. Untrusted pack: docs are visible in the UI; the agent sees the body only as untrusted tool output, and only after the package has been chosen for the session. The ambient one-line tier is derived from manifest fields under strict length and character limits — never arbitrary pack text — so a description cannot smuggle instructions into every prompt. CodeAct requires session consent, not installed-equals-authorized. Code nodes carry a persisted declaration a user saved; CodeAct parses model-generated imports, and auto-mounting anything installed would let the model grant an unrelated pack the action’s fetch, workspace, getSecret and tool capabilities. CodeAct therefore mounts only packs on the session’s package allowlist, sourced from agent/session configuration, user approval on first use, or the task/workflow’s own declarations — defaulting to trusted packs only. The prompt advertises only session-allowed packages, not the installed catalog. Package unit: manifest + SKILL.md A pack’s sandbox configuration is two declarative files: the nodetool.sandboxModules manifest in package.json and a SKILL.md at the package root. Neither is code. Manifest types are Zod-backed discriminated unions at the package boundary (per the repo’s untrusted-input rules), not bare TypeScript interfaces. // package.json of @acme/nodetool-geo { "name": "@acme/nodetool-geo", "version": "1.2.0", "nodetool": { "apiVersion": 1, "sandboxModules": [ { "name": ".", // root entry → specifier "@acme/nodetool-geo" "kind": "js", "file": "sandbox/geo.js" }, { "name": "extra", // subpath → "@acme/nodetool-geo/extra" "kind": "js", "file": "sandbox/extra.js" }, { "name": "simplify-wasm", "kind": "wasm", "file": "sandbox/simplify.wasm", "memoryPagesMax": 256, // binary's max must be ≤ this and ≤ host ceiling "exports": ["simplify"] // scalar-only in v1; see WASM section } ] } } Specifier rule (one rule, used everywhere): the import specifier is <packageName> when name is ".", else <packageName>/<name>. name is a single path segment (no separators, no .., no reserved forms), validated and normalized at discovery; duplicates are a discovery error. A single-module pack uses ".", which is what the migrated bridge packs do (import Papa from "@nodetool-ai/sandbox-csv"). Discovery-time rules (sandbox-pack-discovery.ts), enforced without executing pack code: file resolves inside the package directory (containment + symlink check). Size caps: 256 KB per authored JS source, 4 MB per WASM binary, 8 MB per pack; npm-bundled modules get 1 MB (below). JS modules parse with acorn. Allowed: ES module syntax and static import of sibling files of the same pack. Rejected: all ImportExpression nodes (dynamic import — in package and user code alike, v1), require, and imports of anything outside the pack. Helpers not meant for user code go in a pack-level internal list ("internal": ["sandbox/util.js"]): .js files, canonical id = the normalized pack-relative path, duplicates and cycles rejected at discovery. Internal files are mounted and importable by any public entry of the same pack (sharing is allowed), never valid as a node’s own import, and count toward the pack size caps and the graph digest. WASM binaries get a section-level parse; declared exports are checked against the export section, and the memory rule below is validated before compilation is ever attempted. SKILL.md is validated (frontmatter, 16 KB cap) — but an invalid or missing skill only disables agent discoverability with a warning. It never disables the module: a frontmatter typo must not break a workflow that imports working code. Guest import surface User code imports a declared module by its specifier: import { haversine } from "@acme/nodetool-geo"; const km = haversine(a, b); return { km }; Mechanics, in the order that matters: AST transform, not string wrapping. The entry is built by parsing user code, hoisting static ImportDeclarations to module top level, and wrapping the remainder in the async IIFE that gives return and top-level await their current meaning. Preludes, streaming rewrites, and source maps ride the same transform. This is proven by the M-1 spike before anything else is built. The loader is the authority. runInSandbox installs a custom module loader and normalizer (via the underlying setModuleLoader) that resolve only the run’s declared specifiers and their intra-pack siblings. Everything else — node:*, the wrapper’s compat modules, absolute paths, encoded traversals, computed specifiers — fails at resolve with an error naming the specifier. Static validation is a courtesy layer for early errors; the loader holds the boundary at runtime. Hardening precedes evaluation. Timer deletion and global hardening are applied before any loaded module evaluates — not by the entry module’s first statements. The M-1 spike proves the mechanism (loader hook, per-module preamble, or engine-level init). Adversarial tests are part of the contract: node:* in package and user code, computed dynamic imports, absolute and encoded paths, sibling escapes, and compat-module cache hits must all fail, on Node and in the browser runner. Guest modules evaluate under the same interrupt handler and memory limit as user code. No new budget knobs. Imports are the declaration A node declares no packages. Its body’s static imports name the specifiers the host resolves against the installed catalog, and the same rule holds for a JS script, an authoring run_code call, and the browser runner’s prefetch — one answer to “what may this body import”, read off the body. It was a packages property once: a list of SandboxModuleDeclaration (specifier plus a resolvedPackVersion and contentDigest stamped when the workflow was saved). Two lists that had to agree — the imports and the declaration — meant the picker and the editor could disagree, and every wrong answer was a run that failed on an import the body plainly made. The declaration type survives inside the catalog contract, where a caller builds it from what it read. Resolution always uses the installed version; a pack whose version or digest moved since the catalog was built is a validation warning (not a lock failure, not an auto-upgrade). contentDigest is a module-graph digest, not an entry-file hash: a canonical, sorted list of every transitive source with its normalized module id, generated facade source and generator version, compiler options and compiler version for npm modules, and WASM bytes where applicable. The delivery response, the bundle cache, and validation all use this same graph digest. CodeAct parses each step’s imports the same way, but checks them against the session package allowlist from the trust model rather than the whole installed catalog — an action is code a model just wrote, not a body a person saved. What the explicit declaration bought is still there without it: nodetool validate reads the imports and checks them offline (typo, missing pack, version drift) before anything runs, the loader mounts only what resolved, and a shared workflow names its dependencies in the one place a reader looks — the code. One catalog, injected everywhere The first draft left registry ownership implicit, which does not survive contact with the real wiring: bootstrapNodeRegistry() loads packs asynchronously, the CLI’s buildFullRegistry() is synchronous and loads no installed packs, validateGraph() sees only a narrow registry interface, and soft reload/uninstall differ from startup. The design therefore names one deep module with an injected interface: Placement respects the dependency order — ProcessingContext lives in @nodetool-ai/runtime, and node-sdk already depends on runtime, so runtime cannot import an interface defined in node-sdk. Data types (declarations, summaries, statuses) go in @nodetool-ai/protocol; the runtime-facing interface goes in @nodetool-ai/runtime; node-sdk provides the discovery adapter that implements it without owning it. The interface is split by consumer, so the general surface never hands out raw absolute paths or skill bodies. Delivery is asynchronous — entitlement checks may need database or remote state — and authorization and retrieval are one operation: the resolved AuthorizedSandboxModuleDelivery carries browser-safe content, media type, the graph digest, and dependency module ids, never a filesystem path, so the two checks cannot drift apart: interface SandboxModuleCatalog { summaries(): readonly SandboxModuleSummary[]; // UI + prompt tier resolveForExecution( declarations: readonly SandboxModuleDeclaration[] ): SandboxModuleResolution; // sources for the loader diagnostics(): readonly SandboxModuleStatus[]; // doctor / packs.list } authorizeDelivery is not part of the M0 contract. Delivery is the M2 subject, and an authorization method with no route behind it is an untested promise, not a boundary. M2 adds it as the asynchronous, entitlement-aware half of the browser route described under “Delivery and distribution” — one operation resolving authorization and content together, returning browser-safe bytes, media type, graph digest and dependency module ids, never a filesystem path. Until then nothing outside the process can ask the catalog for module source. One catalog instance is constructed where packs are discovered and injected into every consumer: kernel execution (via ProcessingContext), validation, CodeAct prompt generation, the websocket delivery route, and the CLI (which constructs its own from the same discovery code). No second global snapshot next to pack-snapshot.ts; catalog statuses are their own set, separate from node-pack loading statuses. Soft reload swaps the catalog’s contents; in-flight runs keep the resolution they started with. Config-only modules from npm packages A pack does not have to author sandbox code. A manifest entry can point at an npm dependency, and NodeTool produces the guest module from it: { "name": "@nodetool-ai/sandbox-yaml", "dependencies": { "js-yaml": "^4.1.0" }, "nodetool": { "apiVersion": 1, "sandboxModules": [{ "name": ".", "kind": "js", "npm": "js-yaml" }] } } Mechanics, at pack discovery, in a dedicated compiler module (not esbuild concerns spread through node-sdk): esbuild bundles the package: bundle: true, format: "esm", platform: "neutral", explicit conditions and mainFields (neutral alone does not pin which conditional export is chosen), no externals. An import of a Node builtin fails the bundle — that is the filter, surfaced as a named skip at install time. The output goes through the authored-module static checks plus a scope-aware forbidden-global scan (process, Buffer, require, eval, Function, timers, WebAssembly, DOM names). Hard references error; feature-detected references warn — and a warning is a heads-up, not a compatibility guarantee. Only the QuickJS probe below establishes that initialization actually works. Bundling proves resolution, not compatibility. Admission ends with a probe: the bundle is imported in the actual QuickJS engine — and the probe context is capability-free, because importing executes top-level code. No fetch, no workspace or secret bridges, no tools; a short deadline, normal memory limits, output/log caps. Discovery must never grant package code capabilities before a workflow imports it. The probe proves module initialization only — it cannot prove every export works — and runtime loading stays authoritative after admission. A module that fails the probe is skipped with the probe’s error. Cache keys are digests, not versions. pack/name@version is wrong — output changes under linked packs, transitive updates, lockfile changes, esbuild upgrades, and option changes while the version stays put. The cache key is concrete: a hash over esbuild’s metafile input list (every input file’s content hash), the resolution inputs that decided which files those were (the package.json manifests governing each input, and the places a nearer copy of the dependency could shadow it — recorded present-with-hash or absent), the esbuild version, and the normalized build options. Writes are atomic; every path component is sanitized. Bundled modules cap at 1 MB — and M-1 measures real candidates (zod, date-fns, cheerio) before that number is promised anywhere. What fits: pure-computation ESM (schema validation, functional utilities, date math, parsers). What cannot, regardless of tooling: Node builtins, DOM, network, timers, eval, WASM-shipping packages (no WebAssembly in the guest — those need the host-WASM path). Host WASM modules v1 is scalar-only, with exact signatures. The v1 call contract: arguments and returns are i32, f32, or f64 only; exactly one scalar return (or none — a void export resolves to undefined); argument count bounded (8). Binary validation rejects exports using i64 (maps to bigint, not number), v128, reference types, multi-value returns, or excess arity — an export outside the contract is named in the skip reason, not silently dropped — and verifies every named export is a function, not a memory, table, or global. Manifest export names must be valid, non-reserved JavaScript identifiers; a binary export that is not ("foo-bar") is mapped explicitly: { "wasm": "foo-bar", "as": "fooBar" }. Conversion is validated host-side before dispatch — never left to implicit WebAssembly coercion, so Node and browser behave identically: an i32 argument must be a finite integer in int32 range (no wrapping — out-of-range rejects), f32/f64 accept any JS number including NaN and infinities, and f32 rounds by WebAssembly’s normal rules. Byte-oriented calls wait for a future ABI, informed by the M-1 reference module from a documented toolchain. There is no interim workaround: guest JS cannot reach the host instance’s linear memory, so typed-array work is simply out of scope until that ABI exists. Memory: a host cannot cap a module-defined memory at instantiation — with an empty import object the binary’s own limits rule. The v1 rule is therefore validation, not override. The manifest declares memoryPagesMax; the binary’s own declared maximum must be at or below both the manifest request and the host hard ceiling (4096 pages / 256 MB), every memory must declare a maximum, shared memories are rejected, and a module that imports its memory makes the import object non-empty and is rejected — all checked in the binary before compilation. A manifest cannot lower a larger baked-in maximum except by rejecting the module. Instance ownership: instance per call, stateless. Each call instantiates fresh from the cached module inside the worker, runs, and discards the instance. Mutable globals and linear-memory contents do not persist between calls, and the docs say so — stateless semantics is the contract, not an accident. This is what makes pooling, concurrent calls, and timeout-terminate-replace coherent: a killed worker destroys nothing an invocation owns. Packs needing cross-call state keep it in guest JS and pass it in as scalars, or wait for the byte ABI. The import surface is a generated facade. For a WASM entry, import { simplify } from "@acme/nodetool-geo/simplify-wasm" resolves to an ESM facade the catalog generates: named async exports matching the manifest’s exports, calling into a per-run dispatcher. Authored JS in the same pack may import a sibling WASM entry by the same specifier rules; it resolves to the same facade. The security contract is enforceable behavior, not an unobservable handle — a facade needs some guest-visible binding to receive the host function, and other modules evaluate during dependency loading: The dispatcher serves only WASM modules declared for the run. Every call validates module identity, export allowlist, and argument count and scalar types before the worker runs. The static analyzer and the runtime loader both deny direct imports of the private bridge module. The dispatcher binding is removed before the user IIFE starts. A module that discovers the temporary binding anyway gains nothing beyond the run’s declared WASM surface — the dispatcher checks, not the hiding, are the boundary. M-1 proves the hiding mechanism before the design promises anything stronger. Execution and budgets: Compile once per process, cached (WebAssembly.Module is structured-cloneable, so it crosses to workers without recompiling). Calls run on workers (Node worker_threads, browser Web Worker) with a hard per-call timeout; a timed-out worker is terminated and replaced. Per-call timeout alone does not bound aggregate use, so the budgets are layered, with defaults fixed now rather than left open until M4: process-wide worker pool of 4; per-invocation call concurrency of 2; 256 calls per invocation; an aggregate WASM wall-clock budget of 30 s per invocation (matching the sandbox default timeout). A manifest may lower these, never raise them. Byte caps belong to the future byte ABI. How agents learn a pack: SKILL.md, progressive disclosure A sandbox pack may provide a SKILL.md — the compressed docs page for using the library inside the sandbox: the specifier, the main functions with one example each, and the gotchas that matter there (input caps, the 64 MB guest heap, no timers). A pack without one stays runnable; only agent discoverability suffers. Format and parser are the existing AgentSkill machinery (packages/agents/src/codeact/sandbox-package-docs.ts and parseSkillMarkdown in protocol), hoisted so node-sdk can call it. Disclosure is two-tier so prompt size stays flat: one line per specifier (specifier + a description derived from manifest fields under strict length/character limits) always — and for CodeAct, only specifiers on the session allowlist; the full body on demand when the agent reaches for the pack, under the trust rules above (trusted: normal skill registration; untrusted: untrusted tool output after the package is chosen for the session). The Code node’s package picker renders the same file for the human. A pack exposing several specifiers carries one skill with a section per module. Delivery and distribution Install: the existing Electron npm installer, with the --ignore-scripts split described under the trust model. Registry: index entries gain a sandboxModules: string[] summary for search; per-file hashes come with the ecosystem milestone. Browser runner: module sources are fetched by opaque module id — GET /api/sandbox-modules/:moduleId — never by path segments: scoped names contain /, and encoded slashes behave inconsistently across routers and proxies. The server resolves the id through the catalog (no route-to-filesystem translation), responses carry a content digest the client verifies, and the browser caches by digest. Whether private packs’ source may be delivered to a given browser client is an entitlement question the route must answer through the catalog, not assume from authentication alone. Packaged Electron: installed packs arrive through the optional-node root as today. Only if a bundled builtin ever ships sandbox modules does bundle-backend.mjs stage them (under _sandbox/<pack>/), verified by verify-backend-bundle.mjs. Validation and harnesses nodetool validate / validate_workflow: unknown specifier (error, names the pack), declared-but-unused (warning), version mismatch (warning), import in code missing from packages (error). nodetool node run / debug: resolve through the same catalog, so a failing import reproduces headlessly. The sandbox manifest’s generated packages section and its drift test pin the one-line disclosure tier. Node and browser run the same contract-test fixtures for loading, denial, hardening order, and WASM budgets. Alternatives considered Keep host bridges on globals as the only extension point. Safest, but every library becomes a NodeTool core change, and it leaves two surfaces where there should be one. Host execution remains the right answer for libraries the guest cannot hold — this design keeps it and changes only how the guest reaches it, from a data.* global to an import. Inject packages as globals instead of imports. Avoids the AST work, but invents a second module system and still needs the same declaration, validation, loader, and delivery machinery. Run WASM inside the guest. Impossible on quickjs-ng. Arbitrary npm imports at runtime. Rejected; the config-only form — declared, bundled and vetted at install time, admission-probed — is the answer, not a live resolver over node_modules. Trust-free guest modules. The first draft’s position; withdrawn. Guest modules share the node’s capabilities, so the model is explicit consent (v1) with capability attenuation as a later opt-in. Limits summary Bound Value Enforced JS module source (authored) 256 KB pack discovery JS module source (npm-bundled) 1 MB (validated in M-1) compiler + admission probe WASM binary 4 MB pack discovery Per-pack total 8 MB pack discovery SKILL.md 16 KB pack discovery (warning-grade) Guest eval of modules existing 64 MB / deadline / interrupt engine WASM memory binary-declared max ≤ memoryPagesMax ≤ host ceiling (4096 pages); no shared, no imported memory (v1) binary validation WASM signatures i32/f32/f64 only, ≤1 return, ≤8 args; no i64/v128/ref types/multi-value binary validation WASM per-call wall clock 5 s default, pack may lower worker timeout + replacement WASM aggregate worker pool size, per-invocation call count + concurrency + wall-clock budget host bridge WASM imports none binary validation WASM state none across calls (instance per call) execution model Dynamic import() rejected everywhere validation + loader Host module text input 5 MB the host implementation Host module byte input 10 MB the host implementation Zip inflation (sandbox-zip) 50 MB total the host implementation HTML matches (sandbox-html) 100 default, 1000 ceiling the host implementation Workbook write (sandbox-xlsx) 64 sheets, 250 000 cells the host implementation OCR words (sandbox-ocr) 20 000 per page the host implementation Host module identity id in NodeTool’s registry, declared by the pack that registry pins to it discovery + dispatcher Non-goals WASI, filesystem, network, host-function or memory imports for WASM. require, Node builtins, the wrapper’s compat modules, or undeclared npm packages in the guest. Typed-array WASM ABI in v1 (scalar-only until the reference-module spike pins a contract). Per-workflow version pinning (recorded versions warn on mismatch; resolution uses the installed version). Third-party host modules. kind: "host" names an id NodeTool implements; a pack can never supply the code behind one. Milestones Revised order: prove the risky mechanics first, then build outward. Each milestone lands green on its own. M-1 — Proof and threat model AST transform spike: static imports + IIFE body semantics (implicit return, top-level await, streaming, source locations) on real Code/ CodeAct corpora. Loader spike: custom loader/normalizer denying node:*, compat modules, computed imports, path escapes — on Node and browser. An explicit pass/fail criterion: the wrapper’s own prepareNodeCompatibility() bootstrap imports still work while the same modules are denied to guest code, including module-cache hits. If bootstrap and guest loading cannot be distinguished reliably, the answer is to bypass or patch the wrapper’s compatibility setup, not to ship a loader that cannot hold the line. Hardening-order spike: prove globals are hardened before any module evaluates. One scalar WASM and one byte-oriented WASM reference module from a documented toolchain (the byte one informs the future ABI, not v1). Measure bundle sizes of the real npm candidates against the 1 MB cap. Document exactly what capabilities imported code can reach; test npm lifecycle-script behavior with and without --ignore-scripts. M0 — Catalog and safe discovery Zod manifest schemas; the specifier + declaration model (versions included). SandboxModuleCatalog interface with injected ownership (kernel, validation, prompts, delivery, CLI); statuses separate from node-pack statuses. Sandbox-only installation with --ignore-scripts, the install-mode UI, and the direct-URL/unknown-manifest policy. Discovery validation: containment, symlinks, sizes, import-expression rejection, collisions; SKILL.md as warning-grade. M0 checkpoint — manifest and artifact discovery The first M0 slice is deliberately independent of pack execution. It is implemented by SandboxPackManifestSchema and SandboxModuleDeclarationSchema in packages/protocol/src/sandbox-package.ts, and by discoverSandboxPack() in packages/node-sdk/src/sandbox-pack-discovery.ts. It now provides: normalized root and one-segment specifiers, with duplicate declarations rejected at the manifest boundary. The sandbox schema covers only the sandbox fields; discovery extracts those fields from the larger existing nodetool manifest so register, nodes, and other pack metadata keep their current ownership; package-relative file checks with realpath containment and symlink rejection; authored-JS, WASM, per-pack, and SKILL.md size limits; static JS dependency discovery, rejection of dynamic imports, host imports, and undeclared helpers, internal-file cycle detection, and a complete source graph; section-level WASM checks for imports, memories, exports, and scalar signatures; and a stable graph digest for authored JS, internal JS, and WASM files. An npm entry joins that graph once a host injects its compiled artifact (M3); one nothing has compiled yet is recorded as pending-compile. @nodetool-ai/runtime now owns the read-only SandboxModuleCatalog contract, and node-sdk provides createSandboxModuleCatalog() over the discovery results. Its execution resolution returns browser-safe module graphs without absolute paths, reporting missing modules as errors and version or digest drift as warnings. discoverSandboxCatalog() (packages/node-sdk/src/sandbox-catalog-host.ts) is the concrete host: it scans the pack search paths, discovers every sandbox package without executing pack code, and builds one catalog. Nothing throws out of it — a pack that violates the static contract and a package name claimed by two roots both become catalog diagnostics, and a duplicated name is dropped rather than resolved by scan order, so shadowing never silently picks a winner. Injection: the server builds the catalog in bootstrapNodeRegistry() and again on soft reload, the CLI builds it in buildFullRegistry(), and each installs it as the process’s catalog default (setProcessSandboxModuleCatalog). ProcessingContext reads that default only when its constructor is given no sandboxModuleCatalog option; an explicit value — including null — always wins. The catalog stays an injected dependency, and the default exists so the twenty-odd context construction sites do not each thread it through. Runs already in flight keep the catalog instance they captured. packs.sandboxModules over tRPC returns the summaries and every diagnostic, kept separate from the node-pack snapshot packs.list reports: sandbox modules and nodes are discovered by different code and fail independently. The host pack loader also treats sandboxModules without an explicit register export as a sandbox-only manifest and skips it before resolving or importing its entry point. A hybrid pack with an explicit register still uses the existing host-pack trust path. Electron’s first-install path passes --ignore-scripts, reads the requested package from the installed node_modules directory, and classifies it: sandbox-only, register, hybrid, or unknown. The classification, the lockfile identity of the pack (version, resolved URL, integrity), and the identity of every package in its dependency closure are written to an install ledger at <userData>/optional-node/nodetool-packs.json. A sandbox-only pack is active immediately; register, hybrid, and unknown packs are recorded inactive with scripts still disabled. trustNodePack(name) is the approval flow. It re-classifies the pack from disk, refuses if the mode moved, and compares the whole recorded closure against the lockfile again — version, resolved URL and integrity, per package — before anything runs. Only then does it run npm rebuild <pack> <closure…> with scripts enabled, against the artifact already on disk. There is no refetch, so there is no window in which a different artifact could arrive; the closure is rebuilt because --ignore-scripts suppressed the dependencies’ scripts too. Success marks the row scripts: "ran", active: true. An unknown manifest can never be approved, and a sandbox-only pack is refused because it runs no host code. Trust here authorizes lifecycle scripts, which is a different question from whether the pack loader may import the pack: that stays the ~/.config/nodetool/packs.json allowlist the Settings → Packages toggle writes, and the approval message says so. A register pack therefore needs both before it registers nodes. The current SKILL.md check is also only a discovery warning. It verifies the size and minimal frontmatter shape; it does not register the file with the agent skill system. Full parsing and disclosure land in M5 (see the M5 checkpoint). Settings → Packages shows this rather than treating install as authorization: each pack installed by NodeTool carries its mode, an active/inactive chip, a sentence saying what that mode means, and — for an inactive register or hybrid pack — a Trust and rebuild action. An install that lands inactive reports as a warning with the mode named, not as a failure. What discovery still does not do: import a pack, compile npm modules, authorize browser delivery, or make a module available to QuickJS. Duplicate package names across roots and collisions after path normalization (for example, two spellings of the same package-relative file) stay failures; the catalog host preserves them as diagnostics instead of reintroducing order-dependent resolution. M0’s exit criteria are met for the sandbox-only path: such a package installs without running pack code, appears in packs.sandboxModules diagnostics, resolves through the same catalog validation and execution share, and stays unavailable to the guest until M1’s loader is enabled. The regression suites are packages/node-sdk/tests/sandbox-pack-discovery.test.ts, sandbox-module-catalog.test.ts, sandbox-catalog-host.test.ts, pack-loader.test.ts, packages/runtime/tests/context.test.ts for the injection default, and electron/src/__tests__/nodePackManager.test.ts for install classification and the integrity-bound rebuild. M1 — Guest JS end to end (feature-flagged) Authored modules through Code node, CodeAct, validation, CLI, and server execution together — import support is not claimed until the user-facing execution and validation paths agree. Prompt strings (“no module loader”) and their drift tests change here. Two things belong to M1 that earlier drafts deferred: The one-line catalog tier and the session allowlist. CodeAct cannot generate approved imports unless its prompt advertises the session-allowed specifiers, so the strict manifest-derived one-liner ships with import execution. M5 stays the documentation milestone (full SKILL.md retrieval, package picker, Package Manager presentation). The parity feature flag. Without it, a Code node using imports runs on the server and fails in the browser runner. While flagged, validation reports a rollout issue — “Sandbox package imports are unavailable in the browser runner until module delivery is enabled” — and the browser runner refuses such nodes with the same message. Nothing persistent is written: no node platform metadata is rewritten, so a workflow saved during M1 carries no stale server-only classification after M2. M1 checkpoint — imports end to end, behind the flag M1 is implemented. It shipped gated by NODETOOL_SANDBOX_MODULES_V1 (exact opt-in on "1"), which M2 removed; the flag paragraph below records what it covered. What shipped: Entry builder. buildEntryModule() (packages/agents/src/js-sandbox.ts) parses a snippet as a module, hoists its static ImportDeclarations above the async IIFE and blanks the ranges they vacate, so line numbers in a syntax error still point at the user’s code. Import-free code takes the old wrapCode path byte for byte. Normalizer-enforced loader. RunSandboxOptions.modules installs a module loader and normalizer on the context. The normalizer is the enforcement point: QuickJS serves a cached module without consulting the loader but normalizes every specifier first, which is what makes the denial hold for the wrapper’s own Node-compat bootstrap. Declared specifiers and their intra-pack siblings resolve; node:*, absolute and encoded paths, sibling escapes, another pack’s internals and every dynamic import() do not. Loaded modules carry the timer deletions on their first line, so hardening precedes their evaluation. Code node packages. CodeNode (packages/code-nodes/src/nodes/code-node.ts) declares what it may import and resolves it through context.sandboxModuleCatalog; nothing declared installs no loader at all. Validation. validateCodeNodeBody and validateGraph (packages/node-sdk/src/) read the declarations offline: an undeclared import and a dynamic resolution are errors, an unused declaration and version/digest drift are warnings, and a specifier no installed pack offers is an error naming the pack. nodetool validate, node run and debug reproduce all of it headlessly. CodeAct session consent. CodeActExecutorOptions.sandboxPackages and ChatCodeActSessionOptions.sandboxPackages are the session allowlist (default: none). Each action’s code is parsed for static imports; allowed specifiers are resolved and mounted, anything else is refused as the action’s observation, naming the specifier. The prompt carries one sanitized line per allowed specifier — never the installed catalog — and says plainly when nothing is importable (packages/agents/src/codeact/sandbox-packages.ts). The flag. NODETOOL_SANDBOX_MODULES_V1 gated all of the above while the browser runner could not fetch module sources. Off, the Code node refused a declaration before the guest started, CodeAct mounted nothing, and validation reported the declaration as an error; on, validation added a code_package_browser_parity warning because the browser runner refused such a graph outright. M2 removed the flag, the warning and the refusal. Nothing persistent was written while it existed — no node platform metadata changed — so no workflow saved during M1 needed migrating. The canonical sentence every prompt surface now states — replacing “there is no module loader” — is SANDBOX_MODULE_RULE in packages/agents/src/code-gen/sandbox-manifest.ts, pinned by the drift tests. Regression suites: packages/agents/tests/js-sandbox-modules.test.ts, codeact-sandbox-packages.test.ts, codeact-executor.test.ts, codeact-prompt-drift.test.ts, sandbox-manifest-drift.test.ts; packages/node-sdk/tests/code-node-validation.test.ts, graph-validation.test.ts; packages/code-nodes/tests/code-node-packages.test.ts; and web/src/lib/workflow/__tests__/browserWorkflowRunner.test.ts. M5 checkpoint — trust-scoped disclosure M5 is implemented: a pack’s SKILL.md is parsed where packs are discovered and reaches an agent only under the trust rule above. One parser. parseSkillDocument / skillSections (packages/protocol/src/skill-document.ts) is the agent skill parser, moved down the dependency order so discovery and the skill system read one format. Discovery attaches the parsed skill (name, description, body, ## sections) to SandboxPackDiscovery; a broken frontmatter is still only a skill-invalid warning and never withholds a module. Trust at the catalog. createSandboxModuleCatalog stamps each pack’s skill with trusted from isPackTrusted (packages/node-sdk/src/pack-loader.ts) — the allowlist, not allowUnlisted — and serves it through catalog.packSkill(packName). Summaries carry the sanitized, capped description and the module’s contentDigest; a body never rides in a summary. Disclosure to the agent. get_sandbox_package_docs (packages/agents/src/capabilities/packs.ts) refuses a specifier the session never allowed, returns a trusted pack’s body plainly, and wraps an untrusted pack’s body in <untrusted-package-docs> with the do-not-follow warning and escaped angle brackets. A trusted pack’s skill also registers as an ordinary AgentSkill, but only for a session whose allowlist names the pack. UI. A Code node names its packs by importing them, so there is no picker to state the consent sentence — Settings → Packages carries it, with the per-module one-liners and the SKILL.md view for sandbox-only and hybrid packs (web/src/components/packages/SandboxPackDisclosure.tsx). It reads packs.sandboxModules and the packs.sandboxPackageDocs procedure, which serves a body only when asked for that pack by name. Regression suites: packages/protocol/tests/skill-document.test.ts; packages/node-sdk/tests/sandbox-pack-skill.test.ts; packages/agents/tests/sandbox-package-docs.test.ts and codeact-prompt-drift.test.ts; web/src/components/packages/__tests__/SandboxPackDisclosure.test.tsx. M2 — Delivery parity (removes the flag) Browser delivery by opaque module id with digest verification; Electron staging for bundled builtins; the same loading/denial fixtures running on Node and browser. The M1 feature flag is removed here — parity restored is the exit criterion. M2 checkpoint — the browser runs what the server runs M2 is implemented; there is no flag. What shipped: The delivery half of the catalog. authorizeDelivery(moduleId) (packages/runtime/src/sandbox-module-catalog.ts, implemented in packages/node-sdk/src/sandbox-module-catalog.ts) resolves authorization and content in one call and answers with browser-safe source, its media type, the module-graph digest, a per-file contentSha256, the delivered file’s pack-relative id, and the opaque ids of everything it imports. Never a filesystem path. Public entry specifiers and internal graph-file ids (<pack>::<file>) are both deliverable, because the browser loader needs the whole graph. The route. GET /api/sandbox-modules/* (packages/websocket/src/routes/sandbox-modules.ts), a wildcard because ids contain /. It resolves only through the catalog: unknown id → 404, refusal → 403, otherwise the body plus X-Content-Digest, X-Content-Sha256, X-Sandbox-Module-Dependencies, X-Sandbox-File-Id and a digest ETag with immutable caching. The browser catalog. web/src/lib/workflow/sandboxModuleCatalog.ts fetches by module id, verifies each body against contentSha256 with crypto.subtle — a mismatch is never cached and never returned — walks the dependency header until the closure is complete, and turns the records into a SandboxModuleCatalog that answers synchronously. The digest versions the cache: a file reached from one root whose digest no longer matches is a leftover from a previous pack version and is re-fetched. Prefetch before the run. runBrowserGraphJob collects the specifiers the graph’s Code nodes import and fetches the closure before either execution path starts, because the catalog contract is synchronous and fetching is not. A module that cannot be had fails the job right there, naming the pack, instead of surfacing as a resolve error inside the guest. The verified records are plain data, so the Web Worker path receives them across postMessage and builds its catalog in the worker — one verification, on the side that owns the origin and the auth header. One contract, two runtimes. The loading and denial cases are data (@nodetool-ai/agents/sandbox-module-fixtures): declared imports, an intra-pack internal helper, node:* and compat-module denials, path escapes and encoded traversals, undeclared specifiers, another pack’s internals, computed and variable dynamic imports, and the module-level setTimeout/eval hardening. packages/agents/tests/js-sandbox-modules.test.ts drives them under vitest; packages/workflow-runner/e2e/tests/sandbox-modules.spec.ts drives the same array through a Code node in a real headless Chromium. The flag is gone. packages/config/src/sandbox-feature-flags.ts and every read of it are deleted, along with the code_package_disabled validation error, the code_package_browser_parity warning, and the browser runner’s property-aware refusal. A graph with packages is browser-eligible when its node types are. M3 — npm compilation The dedicated compiler module: content-addressed cache, explicit resolver conditions, scope-aware scan, QuickJS admission probe. M3 checkpoint — the compiler ships M3 is implemented in packages/sandbox-compiler (@nodetool-ai/sandbox-compiler), a workspace of its own because node-sdk must not depend on esbuild or a JavaScript engine. compileNpmModule({ packDir, npmName }) runs the pipeline in order: esbuild with bundle, format: "esm", platform: "neutral", pinned conditions (import, module, default) and mainFields (module, main), no externals and no minification; the scope-aware scan; the capability-free probe. Each way it can end short of admission is a named skip — npm-module-builtin-import, npm-module-unresolved, npm-module-too-large, npm-module-forbidden-global, npm-module-scan-rejected, npm-module-probe-failed. The scan (scan.ts) resolves every identifier against the real scope chain, so a shadowed process is not a hit. A free reference errors; a reference the module feature-detects — the argument of a typeof, or one in a branch a typeof on the same name guards — warns. The probe (probe.ts) instantiates quickjs-ng directly with allowFetch: false, allowFs: false, no env, no bridges, a 5 s deadline, a 64 MB heap and capped console output. Only the bundle resolves; every other specifier is denied by name. The cache (cache.ts) keys on a digest over every input file’s content hash from esbuild’s metafile, the resolution inputs that selected those files, the esbuild version, the compiler’s contract version, and the normalized options. Entries are written temp-file-plus-rename, keys are validated against ^[a-f0-9]{64}$ before they reach a path, and the cache root is getNodetoolCacheDir()/sandbox-modules. A pointer per (pack directory, dependency) lets a synchronous host read an entry back; it re-hashes every recorded input and every resolution input first, so content still decides. Inputs alone cannot: a manifest that re-points exports at another file, or a nearer install that shadows the copy which won, leaves every input untouched and the bundle wrong. Lockfiles are deliberately not hashed — a lockfile says what should be installed, so hashing one thrashes every pack’s cache on an unrelated dependency edit while saying nothing about the tree on disk. The install that follows moves an input or a manifest, which is what is watched. Discovery stays synchronous and engine-free. discoverSandboxPack takes an injected compiled lookup: an entry with an artifact joins the source graph as npm:<name>, one without becomes pending-compile naming nodetool packs compile. A compiled module’s digest is over the bundled source plus the compiler version and options digest, so recompiling under different conditions reports drift rather than silently running different code. Compilation runs where the host is already async: the server’s refreshSandboxCatalog (which imports the compiler lazily and degrades to a named compiler-unavailable warning), Electron’s install hook, and nodetool packs compile. The CLI’s synchronous buildFullRegistry only reads the cache. The measured candidate table and the cap decision are in m3-implementation-plan.md. Regression suites: packages/sandbox-compiler/tests/, packages/node-sdk/tests/sandbox-npm-discovery.test.ts, packages/websocket/tests/sandbox-catalog.test.ts, packages/cli/tests/packs-command.test.ts. M4 — WASM (scalar-only) Binary validation including the memory and signature rules, worker pool with global and per-invocation budgets, timeout-terminate-replace, instance-per-call execution, and the generated facades over a scoped per-run bridge. The typed-array ABI waits for the M-1 reference results. M5 — Agent and UI disclosure The one-line tier and session allowlist already shipped with M1; this milestone is the documentation layer, landing only after the untrusted-content trust handling is settled: on-demand SKILL.md retrieval, package-picker rendering, Package Manager consent language. M6 — Library packs, and the end of data.* One pack per library the sandbox offers, living in the monorepo under packages/sandbox-packs/ and consumed only via install. The first draft called this “an added import path, not a migration” and kept the data.* bridges beside it. That was wrong, and M6 was reworked: two surfaces for one question is a surface a user has to learn twice, and which one a library landed on depended on nothing they could see. data.* is removed. Every library is an import. M6 checkpoint — the uniform import surface Host-facade modules. The manifest gains a third kind: { "name": ".", "kind": "host", "host": "csv" } host is an id, never an implementation. The pack ships no code for it. The id resolves through SANDBOX_HOST_MODULES (packages/protocol/src/sandbox-host.ts) — NodeTool’s own table, which also pins the one package name allowed to declare each id. The specifier resolves to a generated ESM facade (generateSandboxHostFacade) with one async export per registry export plus a default namespace, calling a per-run dispatcher through a private bridge module (nodetool:host-bridge). It is the same mechanism M4 built for WASM, applied to host JavaScript: facade → private specifier resolved only for generated facades → dispatcher validating per call → binding deleted before the user IIFE starts. The trust rule, enforced twice. Discovery refuses a host entry whose id is not in the registry, and refuses one whose id belongs to another package — so @evil/x declaring {"host": "csv"} never becomes a module. The dispatcher repeats both checks when it is built, because a resolution can reach it from a catalog this process did not build (the browser fetches one over HTTP), and then validates the module key, the export name and the argument list on every call. A third-party pack cannot reach host execution through this path; adding a host module is a NodeTool change — a row in the protocol table and a loader in packages/agents/src/host-modules/registry.ts. Limits moved into the implementations. They are not sandbox limits any more; they are the library’s own, in the code that owns it: MAX_UNZIP_TOTAL_BYTES (50 MB) in host-modules/zip.ts, DEFAULT_SELECT_HTML_LIMIT/MAX_SELECT_HTML_LIMIT in host-modules/html.ts, and the shared MAX_HOST_INPUT_CHARS/MAX_HOST_INPUT_BYTES (5 MB / 10 MB) in host-modules/limits.ts. Each keeps its test. Marshaling follows Fact 7 unchanged: plain data in and out, typed arrays native guest→host, bytes host→guest as tagged base64 (the dispatcher applies toGuestBytesDeep at the boundary, so an implementation returns ordinary Uint8Arrays), errors as tagged objects rather than rejected promises. Browser parity holds because the implementations are pure-JS libraries and the “host” in the browser runner is the page, which can run them. The lazy-import pattern the bridges used is kept, so nothing enters an entry graph and bundling is unchanged. Delivery carries the generated facade as the body, verified against contentSha256 like any other module, plus an X-Sandbox-Host-Module header the browser catalog rebuilds the resolution from — the export list and the dispatcher’s allowlist still come from the protocol registry, so a delivery cannot widen what an id means. The shipped set: Pack Library Runs Why @nodetool-ai/sandbox-dates date-fns guest compiler admits it @nodetool-ai/sandbox-yaml js-yaml guest compiler admits it @nodetool-ai/sandbox-markdown marked guest compiler admits it @nodetool-ai/sandbox-csv papaparse host imports node:stream @nodetool-ai/sandbox-html cheerio + turndown host 25 Node builtins; turndown wants a DOM @nodetool-ai/sandbox-xml fast-xml-parser host reads a bare window @nodetool-ai/sandbox-xlsx exceljs host Node streams @nodetool-ai/sandbox-diff diff host schedules with setTimeout @nodetool-ai/sandbox-zip fflate host the inflation cap (below) @nodetool-ai/sandbox-ocr tesseract.js host WASM engine, workers, downloads its language data @nodetool-ai/sandbox-tfjs TensorFlow.js + model zoo host model weights outlive a run and outsize the guest heap Why zip moved off the guest. M3 admitted fflate, and M6 shipped it as a guest pack. That is now a host pack, and the reason is the cap: a zip bomb is a policy question, and a policy enforced inside the guest is enforced by code the guest can decline to call. The 64 MB guest heap does not replicate it either — fflate inflates into host memory first. There is no byte-transfer boundary below the guest to cap instead, because a guest module is the guest. So the library runs on the host, where the 50 MB ceiling sits between the archive and the guest with nothing around it. This is the one place where “the compiler admits it” was not the deciding question. Presence is never assumed: a missing pack fails validation with “Install <pack>” (sandbox-bridge-packs.ts, now eight rows), prompts advertise only installed and session-allowed packs, and the registry marks them recommended. Removing data.* therefore has a real cost — a workflow that parsed CSV now names the pack it parses with — and that cost is the point: what a node uses is declared on the node. The pack itself needs no install: every pack in packages/sandbox-packs/ ships with the product and is discovered from the shipped root (shippedPackSearchPaths() in pack-loader.ts), which sits last in the search order so an installed pack of the same name shadows it. A third-party pack is what the install flow above is for. Regression suites: packages/agents/tests/host-modules.test.ts (the moved library cases, the dispatcher’s refusals, the forged-manifest cases, and every limit), sandbox-bridge-packs.test.ts (presence), and packages/sandbox-compiler/tests/packs.test.ts (every shipped pack discovered, compiled where it compiles, resolved, and imported in the real loader — plus two adversarial manifests refused at discovery). Disposition of what has no pack is unchanged: format.*, image.*, canvas.*, crypto.* and every capability bridge stay host-side globals permanently, because they are capabilities rather than libraries. Release invariants Each invariant binds from the milestone that introduces its subject onward — WASM invariants from M4, browser parity from M2. Explicit per-node declarations; no ambient npm resolution. The loader, not static analysis, is the enforcement boundary. Fresh guest context per invocation; fresh WASM instance per call. Static size caps on everything a pack ships. Empty WASI/network/filesystem surface for WASM. One import surface: every library is an importable module, and a library safety limit lives in the implementation the guest cannot bypass. A host module’s implementation is NodeTool’s; a pack declares an id and nothing else. The same fixtures and contract tests on Node and browser (once browser delivery exists). Validation before execution.--- # Script Editor Source: https://docs.nodetool.ai/script-editor Write the words of a video, give each speaker a voice, and voice every line. A script is a document with sections, lines, a cast, and a set of recorded takes per line. It opens as a workspace tab and feeds the Storyboard Editor, the Video Editor, and subtitle files. Quick Access: Click + in the workspace tab bar and choose New script, or open a script from the Scripts panel. Where this fits The script owns words and voice. A linked storyboard owns pictures and motion. A timeline owns placement. Each one reads the others and does not rewrite them. Edit a line’s text or a speaker’s voice in the script, and the storyboard and timeline pick up the change. See AI Video Production for the full production path. Create a script Route How New menu + in the tab bar, then New script. Opens an empty script titled “Untitled script”. Scripts panel The New script button in the panel header creates and opens one. Guided flow Start Script from the guided starters. The flow runs Idea, Format, and Voices steps before the editor appears. See Guided Flows. From a storyboard In the storyboard’s script control, choose Extract script. It builds a script from the shots’ dialogue and narration, and the button becomes Open script. From a timeline In the timeline’s transcript panel, choose Extract as script. Each transcript paragraph becomes a line and each speaker label becomes a cast member without a voice. A recorded paragraph keeps its audio as the line’s first take. Agent Ask the assistant, or an agent calls create_script (see Agent tools). The guided flow’s Idea step accepts a typed brief, a pasted script, an uploaded file, or an imported subtitle file. An imported subtitle file keeps its cue timings. Pasting text over it drops them. A script still in its guided flow shows the flow in place of the editor. Finish or skip the last step to reach the editor. Write lines A script is a list of sections, and each section holds lines. The empty state offers Add first line. Use Add section below the last section for another. Write a line. Type in the line field (placeholder “Write a line…”). Press Enter to split a line at the cursor. Insert a line. Hover the gap between two lines and click the + rule. Reorder. Drag the handle (“Drag to reorder line”). Drop between lines, or into another section. Name a section. Edit its “Section title” field. Line menu. The More line actions menu sets a pause after the line (No pause after, 0.5s, 1s, 2s), Duplicate line, or Delete line. There are no line types. Dialogue and narration are the same kind of line, and the difference is the speaker. Add a speaker called Narrator for voiceover. Speakers and directions The speaker button in the gutter shows the line’s speaker. Click it to pick a cast member, choose No speaker, or choose Add speaker. The theater-mask button (Add direction / Remove direction) adds a second field, “Direction (e.g. whispering, tired)…”. A direction describes delivery. It is stored with the line and sent along when you voice it. The toolbar counts lines and words, and shows the total duration of the current takes. Cast voices The Cast tab in the right dock lists the speakers. Each one has: Field Purpose Speaker name The label shown in the gutter and in subtitles. Voice A text-to-speech model and voice. Pick from the providers you have configured in Models and Providers. Linked entity A character or other entity from the library. The entity’s name, kind, and first reference image show under the field. If a linked entity was deleted, the panel shows “Linked entity not found” so you can pick another or clear it. Add creates a speaker named “Speaker 1”, “Speaker 2”, and so on. A line can override its speaker’s voice. A line cannot be voiced until it has an effective voice (its own or its speaker’s). Voice lines and choose takes Every voiced line stores its audio as a take. A take records the text and voice it was made from. Line status Meaning Not voiced yet No current take. Voiced The current take matches the line’s text and voice. Stale The text or voice changed after the current take was made. The voice button reads Re-voice line. Voice line. The waveform button on a line voices it. It is disabled with the tooltip “Assign a voice to this speaker first” until a voice exists. Voice all. The toolbar button voices every unvoiced or stale line that has text and a voice, up to three at a time. It shows an estimate such as “Voice all · ~$0.12” when the provider catalog has a rate. Voicing is billed by the provider at its own rates. Retry. After a run, a banner reports “Voiced N lines.” or lists the lines that failed, with a retry action. Play. Play current take plays one line. Play through plays all current takes in order, and becomes Stop while playing. Takes. The Takes (N) button opens the take gallery. Each take shows its text, duration, and voice. Use Use this take to make it current, Play take, Favorite, Send take to a workflow, or Delete take. Re-voicing adds a take and keeps the old ones, so you can switch back. Link to a storyboard The header control links the script to a storyboard. Create storyboard derives one shot per line, linked to its line, with the words projected into each shot. The button then reads Open storyboard. A line covered by a shot shows the shot’s keyframe thumbnail in the gutter. Click it to open the storyboard on that shot. Until a still renders, the chip shows the shot number. A line with no shot on a linked board shows No shot in the gutter. Warnings about a broken link, such as a shot pointing at a deleted line, appear under the button. The link is stored on both documents, so it survives a reload. Re-voicing a line changes shot durations on the board. See Storyboard Editor for the board side, including extracting a script from a board. Send to a timeline Once at least one line is voiced, the toolbar offers a timeline action. Its label depends on the link: State Label No storyboard, no timeline yet Send to timeline Already sent Update timeline Linked to a storyboard Assemble video An unlinked script becomes a voiceover track with captions from the current takes. A linked script is cut with its storyboard: the board’s rendered shots carry the picture and each line’s take sits inside the shot that covers it. Shots without a render are left out and reported. The same actions appear in the command menu as Send to Timeline, Update Timeline, and Assemble Video. Open the result in the Video Editor. The script keeps a pointer to the timeline. Export subtitles Choose Export SRT from the toolbar’s More actions menu, or Export Subtitles (SRT) in the command menu. NodeTool downloads an .srt file named after the script title. Cues come from the current takes, so voice at least one line first. Assistant The Assistant tab in the right dock opens the Script Assistant, a chat that knows which script is open. Try “draft a 30-second intro for two hosts”, “add a line for Narrator”, or “voice every line and send it to a timeline”. It edits the open script live through the ui_script_* tools. Saving and conflicts Edits autosave. The toolbar shows Unsaved changes, Saving…, Saved, Save failed (changes are kept locally), or Reloaded newer version when the script was changed elsewhere. Undo and redo work for the open script. If an outside edit conflicts with a line you are still editing, a banner lets you accept or discard each change. Agent tools Agents can work on scripts without the editor. See Chat & Agents for how the agent uses them. Tool Purpose list_scripts, get_script Find and read scripts. create_script, edit_script Create a script and change its cast, sections, and lines. voice_script_lines Voice lines with the cast’s voices. assemble_script_timeline Build the timeline the toolbar button builds. derive_storyboard_from_script Create a linked storyboard, as Create storyboard does. delete_script Delete a script.--- # Script Editor — Concept Source: https://docs.nodetool.ai/script-editor-concept.html Script Editor — Concept Status: Concept / brainstorm · Owner: matti A script as a first-class resource: text plus the audio takes voiced from it, authored in an ElevenLabs-Studio-style editor, importable into a timeline sequence for video. This document scopes the idea against what already exists in the codebase. The pitch Today the only place to write and voice narration is inside a timeline: the Studio transcript layer (TimelineTranscriptStore) projects a document view over voiceover clips, and the clips are the source of truth. That is the right model for post-production — word-accurate ripple cuts on media that already exists — but wrong for pre-production. A writer wants to draft, restructure, assign speakers, and audition voices before any timeline exists, and wants the same script reusable across several videos (a 60s cut and a 15s teaser). The Script editor inverts the transcript’s ownership: text is the source of truth, audio is derived. A script lives on its own, line by line, each line voiced into one or more takes (audio assets with word timings). “Send to timeline” assembles the current takes into a sequence — the same move the storyboard already makes for images and video. What already exists (the substrate) The concept is mostly assembly of parts that are in main: Piece Where What it gives the script editor First-class-resource pattern packages/models/src/schema/storyboards.ts, StoryboardStore, StoryboardSurface Table shape (id, user_id, project_id, name, document, timeline_id), tRPC CRUD, workspace tab, autosave Assemble → timeline web/src/components/storyboard/assembleTimeline.ts Resource → sequence creation with linkage keys on clips Back-sync after regen web/src/stores/storyboard/timelineSync.ts Re-voiced line updates the assembled timeline clip (CAS update, never throws) Voicing pipeline TimelineTranscriptStore.generateBeat text → generate_media (TTS) → probe duration → transcribe_audio (word timings) → reflow Word-timing model captionWord / clipCaption in packages/protocol/src/api-schemas/timeline.ts Clip-local word timings that survive re-flow Take/version model clipVersion (same file), ClipVersionHistory, ShotTakesGallery Versioned generations with param snapshots, favorites, cost Document editor infra web/src/components/timeline/transcript/ (Lexical: WordNode, SceneBreakNode, slash commands) The text-editing surface, word-level selection, per-word playback highlight Providers packages/runtime/src/providers/elevenlabs-provider.ts + every other TTS-capable provider Voice generation; ElevenLabs also returns native timestamps What does not exist: the script document schema, the scripts table, the standalone editor surface, a cast/speaker→voice binding, and the script↔timeline linkage keys. Document model Zod schema in packages/protocol/src/api-schemas/ next to the timeline and storyboard schemas. Sketch: Script ├─ id, name ├─ cast: Speaker[] { id, name, color, │ voice: { provider, model, voice, settings? } } └─ sections: Section[] { id, title? } // scene / chapter breaks └─ lines: Line[] ├─ id, speakerId?, text ├─ direction? // freeform performance note ("whispering, │ // tired"), passed through to providers │ // that accept it — no structured tag schema ├─ pauseAfterMs? // authored silence between lines ├─ takes: Take[] │ { id, assetId, durationMs, words: CaptionWord[], │ textSnapshot, voiceSnapshot, createdAt, favorite?, costCredits? } └─ currentTakeId? Derived, not stored: Line status: draft (no takes), voiced (current take’s textSnapshot and voiceSnapshot match the line), stale (text or voice changed since the take — show a re-voice affordance, exactly like the timeline’s dependencyHash staleness). Script duration: sum of current-take durations plus pauses. Placeholder duration for unvoiced lines (the transcript layer’s PLACEHOLDER_BEAT_MS move). CaptionWord is reused as-is; timings stay take-local so re-ordering lines never rewrites word timings — the same invariant the timeline already relies on. Takes are the clipVersion idea relocated onto the script: textSnapshot + voiceSnapshot play the role of paramOverridesSnapshot, and the take gallery is the shot-takes gallery with an audio player instead of a thumbnail. Storage scripts table mirroring storyboards byte for byte in shape: document JSON column, timeline_id back-pointer once assembled, indexes on user/project/ updated. tRPC router with the same get/list/create/update(CAS)/delete surface, autosave hook copied from the timeline’s. Audio takes are ordinary assets — the script references them by assetId, so asset lifecycle, storage backends, and .nodetool bundle export (graph + referenced asset bytes) all work unchanged. The script itself is a DB resource, not an asset, matching workflows/storyboards/sequences. Editor surface A new workspace tab type (WorkspaceTabsStore already routes storyboard / timeline / sketch surfaces), laid out like ElevenLabs Studio: Document pane — the script as continuous prose, one paragraph per line, speaker chip in the gutter. Reuse the Lexical transcript editor: it already has word nodes, scene breaks, and slash commands. Enter splits a line, backspace at start merges, drag reorders. Cast panel — speakers with voice pickers (provider/model/voice) and a “preview voice” button. Assigning a speaker to a line inherits the voice; per-line overrides allowed. Line affordances — voice / re-voice button with per-line spinner (clipStatus pattern), take gallery popover, stale badge, per-line playback. Play-through — sequential playback of current takes with the word highlight the transcript editor already renders; unvoiced lines are skipped or beep-placeholded. This is a plain Audio-element chain, no compositor needed. Voice all — batch-generate every draft/stale line, bounded concurrency, per-speaker voices respected. Voicing pipeline Identical to generateBeat, minus the timeline coupling: line text (+ direction) → generate_media RPC (mode audio, speaker’s provider/model/voice) → asset id probe duration word timings onto the take: providers that declare a ttsTimestamps capability return them natively with the synthesis (ElevenLabs with-timestamps endpoints — more accurate, one call fewer); everything else falls back to the transcribe_audio RPC (best-effort, as today — a failed transcription still leaves a playable take) append take, set currentTakeId Timeline import (“Send to timeline”) The storyboard’s assemble move, for audio: Create (or update) a sequence: one voiceover clip per line on an audio track — optionally one track per speaker — laid end to end with pauseAfterMs gaps. Each clip carries the take’s asset, its CaptionWord[] as clip.caption, speaker, and two linkage keys: scriptId + scriptLineId (the storyboard’s storyboardBoardId/storyboardShotId pattern). Set script.timeline_id. The assembled sequence opens in the timeline editor, where the existing transcript layer takes over for word-level post-production — the imported clips are indistinguishable from beats voiced in-timeline. Re-voice after assembly: mirror syncShotClipToTimeline — when a linked line gets a new current take, patch the matching clip’s currentAssetId, duration, and caption via a CAS document update. Log-and-continue on failure, never block the take. Structural drift: adding/removing/reordering lines after assembly is the hard case. V1 answer: sync only per-line asset swaps; structural changes prompt “re-assemble” (which re-lays the voiceover track but preserves other tracks). Anything cleverer (three-way merge against the timeline’s own edits) is explicitly out of scope until the simple thing proves limiting. Reverse import: “Extract as script” from a timeline — project the transcript document (buildTranscriptDoc) into a new script resource. Cheap to build because the projection already exists, and it closes the loop for recorded/imported media: transcribe a recording in the timeline, extract the script, re-voice it with a cast. A storyboard is the second source the same action reads from: Extract script on a board projects every shot’s dialogue and narration into a linked script and keeps the shot→line ids (extractScriptFromScreenplay, script-storyboard-link/design.md §2.1). Timeline and board differ in what they carry back — the transcript brings word timings off real media, the board brings the shot the words belong to — so they stay two projections into one script shape. Relation to the in-timeline transcript Keep both, with distinct jobs: the script is pre-production (authoring, casting, auditioning — text owns audio), the transcript is post-production (ripple cuts on real media — clips own words). The linkage keys are what stop this from becoming two competing sources of truth: a clip either belongs to a script line (script wins for text/voice, timeline wins for placement/trims) or it doesn’t (transcript behaves exactly as today). Unifying them into one model was considered and rejected for now — the transcript’s clips-as-truth invariant is load-bearing for ripple editing and undo, and inverting it underneath the existing editor is a rewrite, not a feature. Automation Agent tools: ui_script_* mirroring the timeline set — get_state, add_line, set_line_text, set_speaker, set_speaker_voice, voice_line, voice_all, send_to_timeline. Same bridge pattern (timelineAgentBridge), same assistant panel. This is what makes “paste a topic, get a voiced script” a chat interaction. Graph nodes: ScriptRef in the protocol type system, plus a small node family — LoadScript, VoiceScript (batch TTS over a cast), ScriptToTimeline. Enables headless pipelines: LLM writes script → voice → assemble → render. Exports: SRT/VTT straight from take word timings; audio-only mixdown (concatenate takes + pauses) as an asset; scripts included in .nodetool bundles with their take assets. Open questions Dialogue-mode models. ElevenLabs v3 renders multi-speaker dialogue in one call with better prosody than line-by-line synthesis, but produces one asset spanning many lines — it breaks the line↔take 1:1. Likely modeled as a section-level “dialogue take” whose word timings are split back onto lines. Phase 4 at the earliest. Take retention. Takes are cheap to keep (assets already exist); surface costCredits per take like clip versions do, revisit pruning only if it hurts. Collaboration. Same last-write-wins CAS as timeline and storyboard; nothing script-specific. Phasing Author + voice (implemented) — protocol schema (api-schemas/scripts.ts), scripts table + tRPC (scripts router), store autosave (ScriptStore, useScriptServerSync), workspace surface (ScriptSurface, script tab type), cast panel, per-line voicing (scriptVoicing, reusing generate_media / transcribe_audio), take gallery, play-through. The document pane is a plain per-line editor rather than the Lexical transcript editor — that reuse is deferred to a later pass. To timeline (implemented) — assemble the current takes into a voiceover sequence with scriptId/scriptLineId linkage keys. The mapping is a pure function in @nodetool-ai/timeline (buildScriptTimeline) with three callers: the editor button (“Send to timeline”, useAssembleScriptTimeline), the headless assemble_script_timeline tool, and the ScriptToTimeline node. Plus per-line back-sync on re-voice / take switch (stores/script/timelineSync), re-assemble in place on structural drift (preserving foreign tracks), and extract-as-script from a timeline transcript (extractScript, useExtractScript, the transcript panel’s “Extract as script”). A script that links a storyboard assembles through a different builder — phase 5. Automation (implemented) — ui_script_* agent tools (scriptAgentBridge, useScriptAgentBridge, builtin/script: get_state, add_speaker, set_speaker_voice, add_line, set_line_text, set_speaker, voice_line, voice_all, send_to_timeline) + an in-surface assistant panel (ScriptAgentPanel, a Cast/Assistant dock toggle), ScriptRef in the protocol type system with a script data type and property editor, and a graph node family (ConstantScript, LoadScript, VoiceScript — batch TTS over a cast, ScriptToTimeline) reading/writing scripts through new ProcessingContext script methods. Depth — SRT/VTT export (implemented): a shared, framework-agnostic subtitle codec (@nodetool-ai/timeline assembleSubtitleCues / cuesToSrt / cuesToVtt) that lays each line’s current take end to end with the authored pauses and renders line- or word-granularity cues from the take word timings, surfaced as a graph node (ScriptToSubtitles), an agent tool (ui_script_export_subtitles), and an “Export SRT” button on the script surface (exportScriptSubtitles). Still open: dialogue-mode rendering, provider-native timestamps, audio-only mixdown, and .nodetool bundle support. Storyboard link — one script and one storyboard describing the same video, keyed to each other, per script-storyboard-link/prd.md. Status by sub-phase: Link schema + extract (implemented) — script_id on the screenplay, script_line_ids / script_text_snapshot / duration_source on a shot, the storyboard_id back-pointer on the script row, the pure projection (extractScriptFromScreenplay, validateScriptLink in @nodetool-ai/protocol script-link.ts), the extract_script_from_storyboard tool and ui_storyboard_extract_script, Extract script / Open script / Open storyboard in both headers, and the deletion downgrade in both directions. Derive board from script (implemented) — deriveShotScaffold pins linkage, order and projected text; the director pass fills in shot content and is refused and retried when it drops or reassigns script_line_ids. Exposed as derive_storyboard_from_script, ui_script_derive_storyboard, and Create storyboard. Audio-led shot timing (in progress) — linkedShotDurationMs is in @nodetool-ai/timeline; the editor and render tools do not read it yet and there is no duration_source toggle. Joint assemble (in progress) — buildLinkedTimeline is in @nodetool-ai/timeline linked.ts with tests, including fixtures that pin unlinked buildStoryboardTimeline / buildScriptTimeline output unchanged. The assemble buttons and headless tools still call the unlinked builders. Drift, badges, Studio (not started as UI) — shotDialogueDrifted and orphanedLineIds exist and get_storyboard / get_script report link state, drift and orphans; no badges, no re-project action, no linked Studio flow. Phase 1 is deliberately shippable alone: a script you can write, cast, voice, and listen to is already the ElevenLabs-Studio use case, before any timeline integration.--- # Script ↔ Storyboard Link — Technical Design Source: https://docs.nodetool.ai/script-storyboard-link/design.html Script ↔ Storyboard Link — Technical Design Status: all four phases shipped, with two deviations noted below (§4, §2.4). Companion: prd.md, tasks.md. The design links the two documents with keys and derives everything else. No document is merged, no ownership moves: the script keeps words and voice, the storyboard keeps visuals and motion, the timeline keeps placement. Every mapping is a pure function in a shared package, consumed identically by the web editors, the server-side agent tools, and the evals — the pattern buildStoryboardTimeline and buildScriptTimeline already follow. 1. Data model 1.1 Storyboard document (the link lives here) The storyboard references the script, not the other way around, because the board is the consumer of the words: it projects line text into shots and reads take durations for timing. Additions to packages/protocol/src/api-schemas/storyboards.ts and the mirrored interfaces in packages/protocol/src/creative.ts: Screenplay └─ script_id?: string | null // the linked script resource Shot ├─ script_line_ids?: string[] // ordered lines this shot covers ├─ script_text_snapshot?: string // linked line texts as last projected, │ // joined "\n" — drift comparison only └─ duration_source?: "audio" | "manual" // "audio": derived from takes (default // when linked); "manual": user override, // audio never touches duration_seconds Tool-surface aliases (SHOT_KEY_ALIASES / SCREENPLAY_KEY_ALIASES in api-schemas/storyboards.ts): scriptId → script_id, scriptLineIds → script_line_ids, scriptTextSnapshot → script_text_snapshot, durationSource → duration_source. All fields optional; the schemas are passthrough, so old documents load unchanged and new fields travel through old clients. 1.2 Script resource (back-pointer for navigation) scripts table and scriptResponse / patchScriptInput (packages/protocol/src/api-schemas/scripts.ts, packages/models/src/schema/scripts.ts): add nullable storyboard_id, exactly like the existing timeline_id back-pointer. Set when a board is derived from or linked to the script; cleared when the board is deleted (same log-and-continue posture as timeline back-sync). The script document itself gains no shot references — line↔shot membership is owned by the board and read by inverting script_line_ids. 1.3 Invariants (validated, not assumed) Checked by a new pure validateScriptLink(screenplay, scriptDoc): A line id appears in at most one shot’s script_line_ids (error). Every referenced line id exists in the script (error, names the shot). script_id set but script missing/deleted (warning — board still works, link affordances disable). A shot with script_line_ids on a board without script_id (error). Runs wherever storyboard documents are normalized and saved (the normalizeStoryboard* path every ui_storyboard_* write already goes through), and in the headless tools before acting. 2. Pure mappings (shared packages) 2.1 Extract: screenplay → script document New module packages/protocol/src/script-link.ts (protocol because it maps between two protocol document shapes and must stay dependency-free, like entitiesForShot): extractScriptFromScreenplay( screenplay: Screenplay, entities: Entity[] ): { document: ScriptDocumentSchema; lineIdsByShotId: Record<string, string[]> } Deterministic, no model call: One section per screenplay (or per scene grouping if shots carry scene slugs later — V1: single section titled from the screenplay). Per shot, in index order: dialogue becomes a line whose speakerId is the character entity matched by the existing entitiesForShot rules; narration becomes a line with the narrator speaker. Cast: one narrator speaker plus one speaker per character entity that contributed a line; Entity.voice_id seeds voice.voice when the entity carries one (provider/model left for the cast panel or Studio’s curated lineup to fill). Returns the shot→lines map so the caller can stamp script_line_ids and script_text_snapshot onto the shots in the same save. 2.2 Derive: script → shot scaffold Same module: deriveShotScaffold( script: { id: string; cast: Speaker[]; sections: ScriptSection[] }, options?: { maxLinesPerShot?: number } // default 1; sections never split ): ShotScaffold[] // { script_line_ids, dialogue|narration, index } The scaffold pins linkage, order, and the projected text deterministically. Shot content — action, camera, motion, slug, entities — is the Director agent’s job: the director prompt gains a variant that receives the scaffold plus line texts and must return shots that keep script_line_ids untouched (the normalizer enforces it: a director response that drops or reassigns linkage keys is rejected and retried, the same posture the storyboard normalizer takes on missing action). Headless fallback when no model is configured: scaffold shots with action copied from line text, status planned. 2.3 Timing: takes → shot durations packages/timeline/src/script-link.ts (timeline package because it needs currentTake from script.ts): linkedShotDurationMs(shot: Shot, linesById: Map<string, ScriptLine>): number | null Sum of linked lines’ current-take durations plus pauseAfterMs, or null when any linked line is unvoiced (caller falls back to duration_seconds / DEFAULT_SHOT_MS). Applied only when duration_source !== "manual". The storyboard editor recomputes on script changes through the existing server-sync refresh; nothing is stored — duration_seconds is only written on assemble and render so prompts see the real target length. 2.4 Joint assemble: linked pair → one timeline packages/timeline/src/linked.ts: buildLinkedTimeline(input: { boardId: string; shots: Shot[]; musicPrompt?: string | null; script: { scriptId: string; cast: Speaker[]; sections: ScriptSection[] }; }): AssembledTimeline & { skippedLineIds: string[] } Video track: as buildStoryboardTimeline — assemblable shots end to end — except each shot’s duration is linkedShotDurationMs(...) ?? shotDurationMs(...). Voiceover track: one clip per voiced line, starting at shotStartMs + offset where offset is the sum of preceding linked takes and pauses within the same shot. Clip payload identical to buildScriptTimeline’s (caption words, speaker, voice, scriptId + scriptLineId) plus storyboardBoardId/storyboardShotId, so both existing back-sync paths (stores/storyboard/timelineSync, stores/script/timelineSync) patch the same clips with no changes. Music: as today. The whole-cut narration draft clip is not emitted — the script supersedes it. Lines whose shot is skipped (unrendered) are reported in skippedLineIds, not silently dropped. buildStoryboardTimeline and buildScriptTimeline stay untouched; linked assembly is a third function, so unlinked behavior is provably unchanged (regression tests assert identical output on existing fixtures). Deviation found in implementation. The claim above that both back-sync paths patch a joint cut “with no changes” held for stores/script/timelineSync and failed for stores/storyboard/timelineSync: it matched on the shot keys alone, which the voiceover clips also carry, so re-rendering a shot handed its video asset to a line’s audio clip. The fix restricts that match to video clips. A second gap stands open: a re-voiced line shifts later voiceover clips but never the shot clips, so picture and audio drift apart until someone re-assembles. 2.5 Drift Derived helpers next to needsVoicing (same “derived, never stored” rule): shotDialogueDrifted(shot, linesById): boolean // joined texts ≠ snapshot orphanedLineIds(screenplay, scriptDoc): string[] // lines with no shot, on a linked board “Re-project” recomputes script_text_snapshot and the shot’s dialogue/narration projection in one document update; it never touches rendered assets — re-rendering stays a separate, explicit action. 3. Server surface 3.1 tRPC scripts router: patchScriptInput accepts storyboardId (nullable), same CAS update as timelineId. storyboards router: no new procedures — the link is document content. New procedures are not needed for derive/extract in the web app: both run client-side over the stores (they are document→document mappings) and save through the existing CAS updates. Headless runs use the agent tools below. 3.2 Agent tools (headless, packages/agents/src/tools/) Extend the storyboard/script tool files (storyboard-render-tools.ts re-exports, capabilities/storyboards.ts, capabilities/scripts.ts): extract_script_from_storyboard {storyboard_id, name?} — runs extractScriptFromScreenplay, creates the script row, stamps script_id/script_line_ids/snapshots onto the board, sets the script’s storyboard_id. Errors if the board already links a script (idempotence: pass relink: true to re-project instead). derive_storyboard_from_script {script_id, provider?, model?, name?} — scaffold + director pass (provider/model resolved like the render tools: from the call or the board’s selection, unset is an error naming find_model); creates the board linked and back-pointed. With no provider: deterministic scaffold only, shots planned. assemble_storyboard_timeline (existing) — when the board carries script_id, load the script and call buildLinkedTimeline; otherwise unchanged. One tool, no new name: “assemble this board” always does the right thing. get_storyboard / get_script responses gain the link fields and derived drift/orphan summaries so an agent can decide what to fix without a second round trip. 3.3 ui_* tools (web, web/src/lib/tools/builtin/) ui_storyboard_extract_script, ui_storyboard_relink_script, ui_storyboard_set_duration_source in builtin/storyboard.ts. ui_script_derive_storyboard in builtin/script.ts. ui_storyboard_assemble_timeline and ui_script_send_to_timeline gain the linked path (both call buildLinkedTimeline when linked; the script-side button becomes “Assemble video” when a board is linked). The headless eval bridges (packages/agents/src/evals/surfaces/storyboard.ts, script.ts, creative-pipeline.ts) mirror the new tools, keeping the tool-loop suites honest. 4. Web UX Storyboard shot inspector: a “Script” section listing linked lines with speaker chip, voice status (draft/stale/voiced — needsVoicing), play button on the current take, and the drift badge with Re-project. Voicing a line from here calls the script store’s existing scriptVoicing path. Storyboard header: Extract script (unlinked) / Open script (linked). Assemble button label unchanged; behavior switches on the link. Script editor gutter: per line, the linked shot’s keyframe thumbnail chip (click → open board, select shot). Orphan badge when linked board has no shot for the line. Script header: Create storyboard (unlinked) / Open storyboard. Studio (web/src/studio/): home groups documents sharing a link into one project card (script + board + timeline); the prompt-first flow runs derive right after the director drafts, so the user lands linked. Curated models stamp as today. Deviation found in implementation. The shipped flow runs the other direction — prompt becomes a board brief, the director drafts, then extract_script_from_storyboard — because no script-drafting path is callable from the web client: scripts are authored by hand or through the chat assistant, and the one-shot drafting lives agent-side with no route. Same outcome (one prompt, one pass, a linked pair) with no new backend surface. Script-first needs either a nodetool.script.* node the client can run like the Director node, or a route over the agent capability. Deletion: deleting a linked script downgrades the board to unlinked (link fields cleared, projected text kept — it is ordinary shot text); deleting a board clears the script’s storyboard_id. Both are log-and-continue, never blocking the delete. 5. What deliberately does not change buildStoryboardTimeline, buildScriptTimeline, both timelineSync modules, the transcript layer, and all timeline editor code. Unlinked storyboards keep the narration/music draft-clip behavior. The nodetool.script.* and storyboard node families (a LinkedAssemble node can come later; the agent tools cover headless first). 6. Testing and harnesses Unit (Vitest, packages/protocol + packages/timeline): extract, scaffold, duration, joint assemble, drift, link validation — including regression fixtures proving unlinked assembly output is byte-identical. Tool-loop evals: extend storyboard-tools and script-tools cases for the new tools; add a creative-pipeline case script → derive → render (stub) → joint assemble → validate_timeline. Web (Jest): inspector line panel, gutter chips, assemble switch, deletion downgrade. nodetool timeline validate already checks the assembled output; the joint assembler’s tests feed it the built document. Harness registry (packages/cli/src/harness/registry.ts): the linked flow is part of the storyboard and script surfaces; extend their harness entries’ selfchecks with the new deterministic tests so harness gate picks them up on diffs touching these paths. 7. Risks Director drops linkage keys. Mitigated by normalizer enforcement + retry, and the deterministic scaffold as the floor. Drift snapshot bloat. script_text_snapshot is bounded by line text length; no take/audio data is ever copied into the board. Two documents, one save race. Extract/derive write two resources; both writes are CAS and the link is only stamped after the created row exists. A failed second write leaves an unlinked-but-valid document and a toast, never a half-link that fails validation.--- # Script ↔ Storyboard Link — PRD Source: https://docs.nodetool.ai/script-storyboard-link/prd.html Script ↔ Storyboard Link — PRD Status: all four phases shipped · Owner: matti Companion documents: design.md (technical design), tasks.md (implementation checklist). Problem NodeTool has two pre-production documents that both feed the timeline, and no connection between them: The script owns words and voice: lines, cast, takes, word timings (packages/protocol/src/api-schemas/scripts.ts). The storyboard owns visuals and motion: shots, stills, clips, style (packages/protocol/src/api-schemas/storyboards.ts). A shot already carries dialogue and narration text fields (packages/protocol/src/creative.ts), but that text is dead weight: storyboard assembly turns the whole screenplay’s narration into a single draft text-to-audio clip with no cast, no takes, and no word timings (packages/timeline/src/storyboard.ts), while the script editor has the full voicing machinery and no way to receive that text. A user who wants both picture and voice must maintain the words twice and assemble two timelines. The entry order also varies by video. Sometimes the script exists first (a narrated explainer, an ad with approved copy); sometimes the visuals come first (a director brief broken into shots); sometimes neither (a one-line prompt in Studio). Today only one order works per document type. Goals One video project can have a script and a storyboard that agree on the words, with each document edited in its own editor. Either document can exist first. The other is derived from it in one action, with the link established automatically. One “Assemble” produces one timeline: shot clips on the video track, voiced takes on the voiceover track, aligned shot by shot. Voiced audio drives shot timing. A shot linked to voiced lines gets its duration from the takes, not from a guessed duration_seconds. Drift is visible. When line text changes after a shot was rendered, or a linked shot is deleted, the affected document shows it. The whole flow works headlessly through agent tools, so “here is my script, make it a video” is one chat conversation. Non-goals No merged document. Script and storyboard stay separate resources with separate editors. The same reasoning that kept the script separate from the in-timeline transcript applies (see script-editor-concept.md § Relation to the in-timeline transcript): one document per source of truth. No automatic three-way merge. Structural drift after assembly (lines or shots added, removed, reordered) prompts a re-derive or re-assemble; it is not silently merged. No change to unlinked behavior. A storyboard without a linked script assembles exactly as today, narration draft clip included. A script without a board keeps “Send to timeline” unchanged. No lip-sync or in-shot audio placement. Alignment is shot-granular: a take starts when its shot starts. Users and entry orders Entry order User First action Script first Writer with approved copy; narrated explainers, ads “Create storyboard” in the script editor Storyboard first Visual thinker; director-agent output with dialogue “Extract script” in the storyboard editor Prompt first Studio beginner Studio prompt flow creates both, already linked User stories Script-first. I wrote and voiced a 20-line script. I press Create storyboard. A board opens with one shot per beat, each shot linked to its lines, shot durations matching my voiced takes. I render stills and clips, press Assemble, and get one timeline where every take starts with its shot. Storyboard-first. The director agent gave me a 12-shot board with dialogue on each shot. I press Extract script. A script opens with one line per shot dialogue, speakers seeded from the board’s character entities (their voice_id becomes the cast voice). I voice all lines, return to the board — shot durations now follow the takes — and assemble. Prompt-first. In Studio I type an idea. The director drafts the screenplay and the script together, linked. I land on a project card that shows board, script, and (later) timeline as one project. Drift. I rewrite line 7 after its shot was rendered. The shot shows a “dialogue drifted” badge; one click re-projects the text (and offers a re-render). I delete a shot; its lines show an “unlinked” badge. Headless. In chat: “Turn script X into a video.” The agent derives a board, renders stills and clips, voices the remaining lines, assembles one timeline, and validates it — no editor open. Requirements Functional R1 — Link. A storyboard can reference one script. Each shot can reference an ordered list of that script’s line ids. A line belongs to at most one shot; the link is validated. R2 — Derive board from script. From the script editor and from chat: create a linked storyboard whose shots cover the script’s lines in order. Uses the director agent for shot content; linkage assignment is deterministic. R3 — Extract script from board. From the storyboard editor and from chat: create a linked script from shot dialogue/narration text, cast seeded from character entities (voice_id). Deterministic; no model call. R4 — Audio-led timing. A shot linked to lines with voiced current takes derives its duration from take durations plus authored pauses. A per-shot manual override wins and is marked as such. R5 — Joint assemble. Assembling a linked board produces one timeline: shots on the video track, one clip per voiced line on the voiceover track starting at its shot’s start, music as today. The single narration draft clip is not created when a script link exists. Clips carry both linkage key families, so both existing back-sync paths keep working. R6 — Drift visibility. Line-text drift against the shot’s projected snapshot, orphaned lines, and dangling line references are derived (never stored as status) and shown in both editors. R7 — Navigation. “Open script” / “Open storyboard” from either editor. Studio home groups linked documents into one project card. R8 — Agent surface. Headless tools for derive, extract, and linked assemble; ui_* equivalents in both editors; the creative-pipeline eval covers script→board→timeline end to end. Non-functional Derive/extract/assemble mappings are pure functions in shared packages, used identically by the editors, the server tools, and the evals — the established pattern (buildStoryboardTimeline, buildScriptTimeline). All new document fields are optional; existing storyboards and scripts load unchanged. Schemas stay passthrough where they are today. Extract and derive-scaffold are deterministic and covered by unit tests; only shot content (action, camera, motion) comes from a model. Success measures A script-first video needs zero manual re-typing of words into shots, and zero manual shot-duration entry for voiced shots. The prompt-first Studio flow lands the user on a linked pair in one step. The creative-pipeline eval passes a case that goes script → derived board → rendered → joint assemble → validate_timeline green. Unlinked flows produce byte-identical assembly output to today (regression tests on buildStoryboardTimeline / buildScriptTimeline). Rollout Four phases, each shippable alone — see tasks.md: Link schema + extract-script-from-storyboard (unblocks board-first with real voicing; cheapest). Derive-storyboard-from-script + audio-led timing. Joint assemble + navigation. Drift badges, Studio project cards, eval case, polish. Open questions Multi-speaker in-shot dialogue. A shot whose lines belong to several speakers assembles fine (clips carry speaker), but the ElevenLabs dialogue-mode question from the script concept doc applies here too. Deferred with it. Narration vs. dialogue tracks. V1 lays all takes on one voiceover track. One-track-per-speaker is a script-side option that already exists conceptually; revisit after use. Re-derive semantics. When the script gains a section after the board was derived, V1 offers “derive again” which appends/updates shots for unlinked lines and never deletes rendered shots. Anything smarter waits for evidence the simple rule hurts.--- # Script ↔ Storyboard Link — Tasks Source: https://docs.nodetool.ai/script-storyboard-link/tasks.html Script ↔ Storyboard Link — Tasks Companion: prd.md, design.md. Each phase ships alone. After any code change: npm run typecheck && npm run lint && npm run test && npm run test:packages (scope package runs with nodetool affected). Phase 1 — Link schema + extract script from storyboard Protocol: link fields. Add script_id to Screenplay and script_line_ids, script_text_snapshot, duration_source to Shot in packages/protocol/src/creative.ts; mirror in the zod shapes and key aliases in packages/protocol/src/api-schemas/storyboards.ts. Tests: normalization round-trips camelCase aliases; old documents parse unchanged. Protocol: script-link.ts. extractScriptFromScreenplay + validateScriptLink per design §2.1/§1.3, with unit tests covering speaker matching via entitiesForShot, voice_id seeding, the shot→lines map, duplicate/dangling line references. Script back-pointer. Nullable storyboard_id on the scripts table (packages/models/src/schema/scripts.ts + migration), scriptResponse/patchScriptInput in packages/protocol/src/api-schemas/scripts.ts, CAS patch in the scripts tRPC router. Headless tool: extract_script_from_storyboard. In packages/agents/src/capabilities/storyboards.ts (creates script row, stamps link, sets back-pointer, relink re-projects). Tool-loop eval case in packages/agents/src/evals/surfaces/storyboard.ts. Web: extract + navigation. ui_storyboard_extract_script in web/src/lib/tools/builtin/storyboard.ts; Extract script / Open script in the storyboard header; Open storyboard in the script header. Deletion downgrade both directions (design §4). Link validation on save. Wire validateScriptLink into the storyboard normalize/save path; surface issues in both editors. Phase 2 — Derive board from script + audio-led timing Protocol: deriveShotScaffold. In script-link.ts per design §2.2, unit-tested (order, one-line-per-shot default, narration vs. dialogue projection). Director variant. Scaffold-constrained director pass; normalizer rejects responses that drop or reassign script_line_ids (retry, then fall back to scaffold-only shots). Lives with the existing director prompt path used by the storyboard agent. Headless tool: derive_storyboard_from_script. In packages/agents/src/capabilities/scripts.ts; no-provider mode emits the deterministic scaffold. Eval case in packages/agents/src/evals/surfaces/script.ts. Timing. linkedShotDurationMs in packages/timeline/src/script-link.ts (design §2.3); storyboard editor and render tools read it when duration_source !== "manual"; ui_storyboard_set_duration_source toggle + inspector control. Web: derive. ui_script_derive_storyboard in web/src/lib/tools/builtin/script.ts; Create storyboard button in the script editor. Phase 3 — Joint assemble buildLinkedTimeline. packages/timeline/src/linked.ts per design §2.4. Unit tests: shot-aligned take starts, in-shot offsets with pauses, both linkage key families on voiceover clips, narration draft clip suppressed, skippedLineIds/skippedShotIds; regression fixtures proving unlinked buildStoryboardTimeline / buildScriptTimeline output is unchanged. Feed the built document to the timeline validator in tests. Assemble switch. assemble_storyboard_timeline (headless) and ui_storyboard_assemble_timeline / ui_script_send_to_timeline (web) call buildLinkedTimeline when linked; script-side button relabels to Assemble video. Re-assemble in place preserves foreign tracks (reuse the script re-assemble pattern). Back-sync verification. Jest tests that a re-voiced line and a revised shot both patch the same jointly-assembled sequence through the existing timelineSync modules, unmodified. Creative-pipeline eval. Case: script → derive → render (stubbed assets) → joint assemble → validate_timeline green, in packages/agents/src/evals/surfaces/creative-pipeline.ts. Phase 4 — Drift, Studio, polish Drift helpers. shotDialogueDrifted, orphanedLineIds (design §2.5) with tests; Re-project action updating snapshot + projected text in one CAS save. Badges. Drift badge + Re-project in the shot inspector; orphan badge in the script gutter; linked-line panel (speaker chip, voice status, play, voice-from-board) in the shot inspector; keyframe thumbnail chips in the script gutter. Studio. Prompt-first flow derives script + board linked in one pass; home groups linked documents into one project card (web/src/studio/). Tool/summary surface. get_storyboard / get_script report link state, drift, and orphans (design §3.2). Harness registry. Extend the storyboard/script surface entries in packages/cli/src/harness/registry.ts with the new deterministic selfchecks; nodetool harness audit stays clean. Docs. Update docs/script-editor-concept.md (phase table) and docs/agentic-video-product.md (reuse table, home cards) to reflect the link; move this feature’s status lines from Proposed to shipped per phase.--- # SDK Lifecycle v1 Draft Source: https://docs.nodetool.ai/sdk/lifecycle-v1-draft.html SDK Lifecycle v1 Draft Implementation status Capabilities and preflight are implemented as feature-flagged production HTTP and correlated WebSocket operations. The remaining job lifecycle operations in this draft are planned. The server contains transport-independent, schema-validating builders for: capability responses; static preflight using the existing graph/per-node validator; workflow etag and graph-derived input validation; cost summaries using the existing workflow cost estimator; deterministic credential, runtime, provider, model, and asset requirement discovery; availability preflight through injected read-only probes, with conservative runnable decisions and redacted probe failures; execution preflight composition for worker, target, capacity, and queue readiness, again through injected read-only probes; one transport-neutral orchestrator and an adapter over NodeTool’s existing owner/public/collaborator workflow authorization and graph-derived interface service; a principal-bound NodeTool service that reuses per-user credential, registered-provider, and owner-scoped asset checks; locally authoritative Node/TypeScript, hydrated Python bridge, and allow-listed FFmpeg/FFprobe runtime probes; asset requirement discovery from both structured asset_id values and canonical asset:// or /api/storage/ references nested in inputs; a read-only view of the exact authenticated live runner’s real admission counters and execution readiness composition that reports likely queueing without reserving work; explicit local, attached-worker, and live-runner execution targets without attaching, provisioning, or falling back to a different target. GET /api/sdk/v1/capabilities, POST /api/sdk/v1/preflight, and POST /api/sdk/v1/assets/temporary are available by default. A deployment can disable them with NODETOOL_DISABLE_SDK_LIFECYCLE_V1=1. They share the same typed service boundary and authenticated principal. They are not duplicated as WebSocket RPC commands. This document is the design note for possible future acknowledged submission, snapshot, subscription, and lifecycle-cancellation operations. Those roadmap items are intentionally absent from the public v1 declarations and generated client schemas until implemented. The capability profile temporary_asset_upload = available advertises the multipart temporary-input route. It writes directly to configured temporary storage and returns a runtime-resolvable URI. It deliberately creates no Asset row, thumbnail, asset-list entry, or durable provenance record. The temporary store’s retention policy owns cleanup. SDK execution defaults generated-asset persistence to temporary; callers explicitly request auto when normal asset autosave is required. Unannotated non-SDK run_job behavior remains auto. Identifiers and time request_id correlates one WebSocket command with exactly one rpc_response. It is client-generated, non-empty, and not a durable job key. client_request_id correlates application logs and submission attempts. It is echoed by the submission acknowledgement. idempotency_key identifies one logical submission within the authenticated user/workspace scope and advertised retention period. job_id, workflow_id, workspace_id, and asset_id are opaque strings. Clients must not derive routing, ownership, or time from their format. Timestamps are RFC 3339 strings with an explicit offset. The server emits UTC. Clients compare instants, not timestamp text. Event sequence is a positive, monotonically increasing integer scoped to one job. Sequence zero means no event has yet been persisted. Retrying a submission with the same idempotency key and equivalent normalized request returns the original job. Reusing the key with a different workflow, revision, workspace, or inputs is a conflict. The server advertises the retention window before idempotent submission becomes available. Public job states The public states are: State Terminal Meaning accepted No Persisted and admitted, before queue or execution queued No Waiting for capacity; queue_position is present running No Actively executing suspended No Paused on a resumable workflow condition recovering No Server is reconciling execution after interruption cancel_requested No Cancellation accepted but execution not terminal completed Yes Authoritative result manifest is persisted failed Yes Safe public error is persisted cancelled Yes Execution is durably cancelled Internal scheduled/pending states map to accepted. Internal paused maps to suspended. Internal error aliases map to failed. These mappings do not change the existing database vocabulary. Terminal states are immutable. A job cannot be observed as completed until its final outputs, asset references, cost, and provenance manifest are persisted. The runtime schema enforces a non-null result for completed and forbids a terminal result on non-terminal snapshots. Preflight and submission POST /api/sdk/v1/preflight is side-effect free. It does not create a job, reserve a provider request, download a model without explicit approval, or start paid work. It reports: graph, pin, and type issues; provider, credential, model, node-pack, runtime, asset, worker, and approval requirements; whether each requirement blocks execution; estimated, partial, exact, or unknown cost confidence. submit_job requires a successful-enough authoritative preflight, a workflow interface version, workflow etag, client_request_id, and idempotency_key. The server persists the accepted/queued job and idempotency record before returning acknowledgement. Submission acknowledgement is separate from the event stream. Snapshots, events, and reconnect get_job_snapshot is authoritative. A snapshot contains current state, timestamps, last persisted sequence, preflight summary, and terminal result or error when applicable. subscribe_job(job_id, after_sequence) first establishes an authoritative snapshot, then replays retained events after the requested sequence and continues with live events. Every event carries job ID, workflow ID, workspace ID, timestamp, and sequence. Clients must: discard duplicate sequences; process known events in sequence order; safely advance past unknown event types; request a fresh snapshot if replay is expired or a sequence gap cannot be filled; treat the snapshot as authoritative when provisional events disagree. Reconnect to queued, running, suspended, recovering, or cancel-requested jobs returns their current snapshot and replay range. Reconnect to a terminal job returns the persisted terminal snapshot. Missing, inaccessible, and cross-workspace jobs share the same not-found response. Expired jobs use a stable expiry code without revealing another tenant’s resources. Cancellation Cancellation is idempotent: queued work can become cancelled immediately; active work first becomes cancel_requested; a provider may finish before cancellation takes effect; cancellation after a terminal state returns the existing terminal snapshot. Clients must not treat a cancellation acknowledgement as terminal unless the returned snapshot is cancelled. Results and assets The result manifest contains JSON-safe output values plus an explicit list of every durable asset referenced by those outputs. Each asset reference carries an opaque ID, canonical URI, MIME type, optional size and SHA-256 checksum, and an optional expiring download URL. Large image, audio, video, document, and model data travels by asset reference, not inline WebSocket payload. outputs may contain scalar or structured JSON; the assets bindings identify the output name and nested path at which each durable asset is materialized. Provenance contains normalized/redacted inputs, workflow revision, providers, models, seed, and reconciled cost. It never contains credentials, authorization headers, provider secrets, or unbounded inline media. Ownership All preflight, submission, snapshot, subscription, cancellation, result, and asset operations use the same authenticated user/workspace authorization. Caller-supplied identity headers are not trusted. A job and every result asset inherit the accepted submission’s ownership scope.--- # SDK Non-Regression Baseline - 2026-07-24 Source: https://docs.nodetool.ai/sdk/non-regression-baseline-2026-07-24.html SDK Non-Regression Baseline - 2026-07-24 Historical snapshot: the later default-on rollout replaced the positive lifecycle/workflow-interface enable flags with the emergency kill switches documented in protocol-v1-scope.md. This record supports the Phase 0 gate in the SDK server-contract roadmap. It separates contract-related results from pre-existing Windows, sandbox, and dependency failures so later changes can be compared against the same baseline. Passing checks Public protocol suite: 625 tests passed, including discovery, lifecycle, generated-artifact, framing, and baseline-fixture coverage. SDK-affected WebSocket suites: 116 tests passed. C# base/VL unit suite: 102 tests passed after adding nullable Python capability-metadata coverage. Headless VL document suite: 2 tests passed. Electron TypeScript check passed. CLI TypeScript check passed. CLI source smoke executed the base-node DSL fixture in pretty and JSON modes; both returned output value 99 with exit code zero. Live backend health returned NodeTool 0.7.0-rc.32. Live Vite web root returned HTTP 200. Live web workspace loaded 81 workflows, connected to the local worker, emitted primitive outputs, and cancelled the test run cleanly. The saved primitive fixture did not terminate until its required select and list inputs were provided. Live C# TestConsole discovery fetched 2,527 hybrid node types, 81 workflows, and assets over MessagePack WebSocket. Live C# TestConsole execution completed SDK Test - Primitives with all seven graph-derived inputs/outputs after using the authoritative pin names. Unrouted capability/static/availability/execution preflight, authorization source, orchestration, route-policy, workflow-interface, and legacy read-only RPC coverage passes 77 focused tests; the related existing graph-validation and cost-estimation suites pass 21 tests. The default-off lifecycle flag, live capability snapshot adapter, validated response builder, and standalone redacting HTTP handler pass 18 focused tests. The production HTTP dispatcher still returns 404 for the reserved capabilities path. The standalone preflight HTTP adapter passes 10 transport-boundary tests for default-off behavior, validation, authenticated-principal injection, authorization-safe service errors, and redaction. Its reserved path also remains absent from the production dispatcher. Provider-scoped model availability passes 6 focused tests covering recorded provider/type context, registered and configured state, exact model presence in an injected cache/local inventory, bounded timeout, and conservative unknown results. It performs no implicit remote provider list. Cached model inventory composition passes 4 focused tests covering Python/TypeScript-shaped records, ID/repository aliases, explicit provider association, bridge readiness, and redacted cache failures. The combined feature-flag, disabled-current-route, REST/tRPC/WebSocket discovery, authorization, and existing execution/admission regression set passes 264 tests across 22 files. The WebSocket package TypeScript check passes after replacing the Sharp-namespace annotation with the constructor’s inferred return type. Baseline failures outside this change The full Node SDK run passed 602 tests, skipped 2, and failed 8: seven tests compare POSIX slash suffixes against Windows paths; one metadata-cache test depends on filesystem timestamp/cache behavior. The full WebSocket run passed 1,460 tests and failed 31: Windows path and simulated POSIX-home expectations; sandboxed home/file access returning FORBIDDEN; restricted writes below the real roaming asset directory; macOS font/config fixtures on Windows; model discovery startup timing; one SDK inventory test timing out only in the full parallel run (it passes in the focused suite). The full CLI run passed 371 tests and failed 8 before the Windows DSL import fix: two output assertions expected POSIX separators; three source-level DSL imports failed because a Windows drive path was not converted to a file:// URL; three dist-level integration checks used the stale pre-fix CLI build. The source-level path and output issues are fixed. Rebuilding the CLI dist is currently blocked by the repository’s pre-existing sharp callable-type errors in runtime/image-node workspaces. Direct source smoke coverage passes. The first live C# discovery pass found Python metadata with supports_dynamic_outputs: null. The portable SDK now maps null optional capability flags to the same false default as omitted fields, and the live discovery pass succeeds. The Web TypeScript check did not complete within the 180-second local timeout. Phase 2 read-only readiness verification (2026-07-25) The SDK/lifecycle WebSocket regression selection passes 289 tests across 26 files, including default-off route policy, package provenance, cached model inventory, active download state, persisted-worker readiness, authorization, capacity, and current workflow execution. The Hugging Face package passes its complete suite: 187 tests across 9 files. Node SDK package-provenance focused coverage passes 68 tests across registry, Python metadata, and third-party pack loading. The complete Node SDK suite passes 604 tests with 2 skipped and retains 8 existing Windows-sensitive failures: six forward-slash path assertions, one environment path-list assertion, and one metadata-cache timing assertion. None exercises package provenance; the focused provenance tests pass. TypeScript checks pass for node-sdk, huggingface, and websocket. git diff --check passes, and lifecycle capabilities/preflight handlers remain absent from the production dispatcher. A live read-only check against the restarted development server returned HTTP 200 with the current node metadata payload, while /api/sdk/v1/capabilities still returned HTTP 404 as required before route activation. Interactive checks still required Do not mark the Phase 0 client gate complete until these are rerun against the current working tree: execute a terminal small workflow through the Electron shell (the browser client and C# TestConsole paths now pass); with SDK discovery enabled, open the VL help patch and execute primitives plus one asset/media workflow; restart with all newly introduced SDK flags disabled and verify a current Electron workflow plus current asset behavior. The new VL discovery surface is expected to be unavailable in this mode. This includes NODETOOL_ENABLE_SDK_LIFECYCLE_V1=0. Automated workflow, node, asset, and execution/admission coverage with the flags explicitly disabled passes.--- # SDK v1 Contract Baseline - 2026-08-22 (Phase 0) Source: https://docs.nodetool.ai/sdk/phase-0-baseline-2026-08-22.html SDK v1 Contract Baseline - 2026-08-22 (Phase 0) This record freezes the SDK v1 server contract for Phase 0 of sdk-trpc-consolidation.md. Everything below is pinned by fixtures and tests; the convergence phases must keep these checks green without regenerating the fixtures. Commit and commands Server commit: 8fb47bba9f59f214485680bbe0fa5aa42d5dc3f6. Captures were made in this working tree while the Phase 1 protocol-registry commits (322dbc0..8fb47bb) landed in packages/protocol; all 23 checks were re-run green at this commit against the rebuilt protocol dist. Golden fixtures: packages/protocol/fixtures/sdk-v1/ (11 HTTP files, 8 WebSocket files, README with the format and regeneration rules). Reproduce: cd packages/websocket npx vitest run tests/sdk-v1-http-goldens.test.ts tests/sdk-v1-ws-goldens.test.ts \ tests/sdk-v1-route-inventory.test.ts tests/sdk-v1-trpc-caller-inventory.test.ts # Regenerate fixtures (then rerun without the flag and review the diff): NODETOOL_UPDATE_SDK_V1_GOLDENS=1 npx vitest run tests/sdk-v1-http-goldens.test.ts tests/sdk-v1-ws-goldens.test.ts The four files run 23 tests. The route plugins are booted exactly as server.ts boots them (same plugins, same raw-Buffer body parser); services are injected constants and the clock is frozen so bodies are byte-stable. Post-baseline note: This document intentionally records the Phase 0 topology. Phase 3 later moved all 11 HTTP registrations into routes/sdk-v1.ts, removed SDK ownership from handleApiRequest, and made the four discovery WebSocket commands call the shared SDK boundary directly. The fixtures and their bytes remain unchanged. HTTP inventory (11 implemented operations) All Fastify mounts are registered in server.ts:1337-1339 through three plugins. handleApiRequest (http-api.ts:2517) is a second dispatcher used by the standalone Node HTTP server path (createHttpApiServer / handleNodeHttpRequest); it covers only 6 of the 11, which is the recorded drift risk. sdk-v1-route-inventory.test.ts pins both sets. Operation Route Owner (Fastify mount) In 2nd dispatcher Auth Feature flag Node type inventory GET /api/sdk/v1/node-types routes/nodes.ts:39 yes discovery workflow-interface Capabilities GET /api/sdk/v1/capabilities routes/nodes.ts:45 yes discovery lifecycle Model catalog GET /api/sdk/v1/models routes/nodes.ts:51 no discovery none List model downloads GET /api/sdk/v1/model-downloads routes/nodes.ts:57 no authenticated none Start model download POST /api/sdk/v1/model-downloads routes/nodes.ts:63 no authenticated none Cancel model download POST /api/sdk/v1/model-downloads/cancel routes/nodes.ts:69 no authenticated none Preflight POST /api/sdk/v1/preflight routes/nodes.ts:75 yes authenticated lifecycle Workflow summaries GET /api/sdk/v1/workflows routes/workflows.ts:111 yes discovery workflow-interface Workflow interfaces POST /api/sdk/v1/workflow-interfaces routes/workflows.ts:117 yes discovery workflow-interface One workflow interface GET /api/workflows/:id/interface routes/workflows.ts:183 yes discovery workflow-interface Temporary asset upload POST /api/sdk/v1/assets/temporary routes/assets.ts:220 no authenticated lifecycle Handlers: handleSdkNodeTypeInventory (http-api.ts:1929), handleSdkCapabilities (:215), handleSdkModelCatalog (:263), the three model-download wrappers (:286-332), handleSdkPreflight (:235), handleSdkWorkflowSummaries (:1994), handleWorkflowInterface (:2053), handleWorkflowInterfaces (:2092), handleSdkV1TemporaryAssetUpload (sdk/sdk-temporary-asset-upload-http-handler.ts:49). Auth policy The server onRequest hook (server.ts:897) lets a request through without credentials when isSdkV1DiscoveryRequest (sdk/sdk-route-policy.ts) matches it and SDK auth is not required. The discovery allowlist is exactly: GET capabilities, models, node-types, workflows, /api/workflows/{id}/interface, and POST workflow-interfaces. Preflight, model downloads, and temporary upload are never in the allowlist; the server hook authenticates them and answers 401 {"error":"Unauthorized"} itself when it cannot. Inside the handlers, getUserId falls back to the x-user-id header and then "1"; only preflight carries its own 401 (AUTHENTICATION_REQUIRED, captured in the goldens) when no principal is resolved. bridge() strips a client-sent x-user-id and forwards only the server-authenticated identity. The second dispatcher trusts x-user-id directly; its deployment (loopback Node server) provides the boundary. Feature flags (sdk/sdk-feature-flags.ts) NODETOOL_REQUIRE_SDK_AUTH_V1 - opt-in, active only when exactly "1"; removes the discovery exemption. NODETOOL_DISABLE_SDK_WORKFLOW_INTERFACE_V1 - kill switch; the feature is on unless the value is exactly "1". Guards node-types, workflow summaries, workflow-interfaces, and the single interface route. NODETOOL_DISABLE_SDK_LIFECYCLE_V1 - same semantics. Guards capabilities, preflight, and temporary upload. Models and model downloads check no flag. Feature-disabled bodies are 503 with the SDK error shape: SDK_NODE_TYPE_INVENTORY_DISABLED, SDK_WORKFLOW_INTERFACE_DISABLED, or SDK_LIFECYCLE_DISABLED. Content types, statuses, error shapes Every SDK response body is content-type: application/json (no charset). Fastify’s own responses (its 404 for a wrong method on a single-method path) use application/json; charset=utf-8. SDK error shape: {code, message, retryable, detail} with detail duplicating message for older clients. Statuses captured per operation: 200 success (202 for download start), 400 bad input, 401 (preflight only, in-handler), 404 not-found (WORKFLOW_NOT_FOUND, MODEL_DOWNLOAD_NOT_FOUND), 413 (UPLOAD_TOO_LARGE), 415 (preflight non-JSON content type), 500 redacted (INTERNAL_ERROR, retryable: true), 503 feature-disabled. 405 inconsistency (kept as-is): the handlers written before the SDK error shape answer a wrong method with the legacy body {"detail":"Method not allowed"} - node-types, workflow summaries, workflow-interfaces, and the single interface route. The newer standalone handlers (capabilities, preflight, models, model downloads, temporary upload) answer the SDK shape with code: "METHOD_NOT_ALLOWED". On the Fastify server a 405 is reachable only where a path has another method bound; otherwise the wrong method is Fastify’s 404 ({"message":"Route POST:/api/sdk/v1/models not found", ...}). The goldens capture all three behaviors. Order of checks differs per handler and is pinned by the goldens: the single interface route validates version=1 before its feature flag (400 wins over 503), workflow-interfaces validates the body first, and preflight checks flag, then content type, then body, then principal. Multipart temporary upload Handler: sdk-temporary-asset-upload-http-handler.ts. Field name file is required; a non-multipart body is 400 INVALID_REQUEST. Size limit: getMaxUploadBytes() - default 1 GiB, override NODETOOL_MAX_UPLOAD_BYTES; exceeding it is 413 UPLOAD_TOO_LARGE with the limit in the message (goldens pin the limit at 64 bytes through the handler’s injection point). The Fastify server additionally caps any request body at 100 MB (bodyLimit, server.ts:702). Media type defaults to application/octet-stream; an empty filename falls back to <id>.<ext> derived from the media type. Temporary ID shape: key temp/sdk-inputs/<id>.<ext>; the returned uri is the storage URI, except a file:// URI is rewritten to /api/storage/temp/sdk-inputs/<id>.<ext>. Response: {version: 1, uri, name, content_type, size, expires_at: null}. Cleanup semantics: no Asset row and no thumbnail are created; retention belongs to the configured temporary store (getTempAdapter()), and expires_at is always null today. WebSocket inventory Six request/response commands are implemented, all through UnifiedWebSocketRunner.handleCommand (unified-websocket-runner.ts:8666), answering a binary MessagePack rpc_response frame (packWebSocketMessage, useRecords: false, encodeUndefinedAsNil: true): Command Dispatch list_workflow_summaries in-process tRPC workflows.sdkSummaries (:9011) get_workflow_interface in-process tRPC workflows.interface (:9019) get_workflow_interfaces in-process tRPC workflows.interfaces (:9027) get_node_type_inventory in-process tRPC nodes.sdkTypeInventory (:9059) get_capabilities handleSdkV1LifecycleRpc (:9067, no tRPC hop) preflight_workflow handleSdkV1LifecycleRpc (:9067, no tRPC hop) No SDK v1 server events are implemented; sdkV1LifecycleRpcCommand also names submit_job, get_job_snapshot, subscribe_job, and cancel_job as planned commands the handler does not serve. Two error-envelope shapes exist and both are pinned as msgpack goldens: Lifecycle commands: error: {code, message, retryable} (for example SDK_LIFECYCLE_DISABLED). tRPC-backed commands (runRpc, :8591): error: {code, message, retryable, apiCode, trpcCode} - the disabled workflow-interface capture is {code: "SERVICE_UNAVAILABLE", message: "SDK discovery is disabled", retryable: false, apiCode: "SERVICE_UNAVAILABLE", trpcCode: "INTERNAL_SERVER_ERROR"}. tRPC procedures and callers All four are protectedProcedure queries validating with the schema objects exported by @nodetool-ai/protocol (identity pinned by sdk-v1-trpc-caller-inventory.test.ts): Procedure Declaration In-repo callers workflows.sdkSummaries trpc/routers/workflows.ts:472 unified-websocket-runner.ts:9014 workflows.interface trpc/routers/workflows.ts:546 unified-websocket-runner.ts:9022 workflows.interfaces trpc/routers/workflows.ts:561 unified-websocket-runner.ts:9030 nodes.sdkTypeInventory trpc/routers/nodes.ts:126 unified-websocket-runner.ts:9062 Also name-referencing (not calling): trpc/sandbox-coverage.ts:505,871 classifies nodes.sdkTypeInventory and workflows.sdkSummaries in the sandbox capability table; a rename must update that table too. The web app’s tRPC client, the CLI, and the agents package call none of the four. Out-of-repository consumer audit Evidence commands (run at this commit from the repo root; all returned nothing): grep -rn "sdkSummaries" web/src electron/src mobile/src grep -rn "sdkTypeInventory" web/src electron/src mobile/src grep -rn "trpc.workflows.interface" web/src electron/src mobile/src grep -rn "workflow-interfaces" web/src electron/src mobile/src grep -rn "/api/sdk/v1" web/src electron/src mobile/src grep -rn "list_workflow_summaries\|get_node_type_inventory\|get_workflow_interface" web/src electron/src mobile/src Positive control: grep -rn "trpc.workflows.list" web/src returns hits, so an empty result is a real absence, not a broken probe. This confirms the plan’s statement that the web app does not call the SDK routes. The known external consumer remains the C#/VL SDK in nodetool-sdk, which uses the public HTTP and WebSocket surface, not tRPC. Drift checks verified Each new check was made to fail once by doctoring one input, then restored: Check Doctored input Observed failure sdk-v1-http-goldens one byte in http-get-workflows.json success body ("SDK Golden One" -> "SDK Golden 0ne") AssertionError: workflow_summaries/success: expected { …(2) } to deeply equal { next: null, workflows: [ …(2) ] } sdk-v1-ws-goldens first hex char of messagepack_response_hex in ws-get-workflow-interface.json (d -> f) AssertionError: expected 'de0004a474797065ac7270635f…' to be 'fe0004a474797065ac7270635f…' sdk-v1-route-inventory removed GET /api/sdk/v1/capabilities from the expected route list AssertionError: expected [ …(11) ] to deeply equal [ …(10) ] sdk-v1-trpc-caller-inventory call-site pattern changed to caller.nodes.sdkTypeInventoryX( AssertionError: expected +0 to be 1 Determinism notes Date is frozen at 2026-08-22T12:00:00.000Z; Workflow.beforeSave bumps a same-instant updated_at by 1 ms, so the summaries revision is …:00.001Z and the interface etag (a hash of the row) is stable. unavailable_packs in the node-type inventory reflects the in-repo built-in pack catalog (7 packs disabled by default at this commit) with NODETOOL_PACKS_CONFIG pointed at a nonexistent file; a catalog change legitimately regenerates that fixture. The multipart boundary is the one random request element; it is not recorded (the fixture stores a multipart descriptor instead). Test-run record New Phase 0 files: 4 files, 23 tests, all passing (sdk-v1-http-goldens 11, sdk-v1-ws-goldens 8, sdk-v1-route-inventory 2, sdk-v1-trpc-caller-inventory 2). npm run lint --workspace=packages/websocket (tsc --noEmit) passes. The package tsconfig excludes tests/, so the new test files were also typechecked directly (strict, same base config) with no errors. Full npm run test --workspace=packages/websocket: 218 files, 2,343 tests, all passing (263 s, Linux sandbox, Node 22). No pre-existing failures in this environment; the Windows-sensitive failures listed in non-regression-baseline-2026-07-24.md do not reproduce here.--- # SDK v1 Phase 8 release candidate Source: https://docs.nodetool.ai/sdk/phase-8-release-candidate-2026-08-23.html SDK v1 Phase 8 release candidate This is the review record for the first publishable HTTP plus execution- WebSocket SDK v1 contract. It is a candidate until a NodeTool release tag and release asset exist. Producer identity NodeTool candidate commit: bdf19bb57d33986fc323a8a51ac7a7d3d11c0a26 Protocol version: 1 Bundle format: 1 Contract digest: b9297a9cddb7778fcadbd7ef288a82761f64afe75369f54fd3ad3bcdccab9554 Deterministic tar SHA-256: 50a1bfe3ebbb9a85329406b911074ec917f3e5ea54cc72679824018888b8bb8c Minimum matching SDK server version: 0.7.0-rc.36 Bundle contents: 23 files Two independent bundle generations produced the same contract digest and tar SHA-256. The matching nodetool-sdk candidate pins the contract digest above. Semantic operation change from the previous candidate Added execution declarations: execution.cancel_job execution.chunk execution.end_input_stream execution.execution_target execution.job_resumed execution.job_update execution.node_progress execution.node_update execution.output_update execution.protocol_rejection execution.reconnect_job execution.run_job execution.stream_input execution.update_node_properties Removed compatibility and discovery-WebSocket declarations: cancelJob getJobSnapshot lifecycleRpc.cancel_job lifecycleRpc.get_capabilities lifecycleRpc.get_job_snapshot lifecycleRpc.preflight_workflow lifecycleRpc.submit_job lifecycleRpc.subscribe_job receiveJobEvent sdkRpc.get_node_type_inventory sdkRpc.get_workflow_interface sdkRpc.get_workflow_interfaces sdkRpc.list_workflow_summaries submitJob Changed HTTP declarations: cancelModelDownload getNodeTypeInventory getWorkflowInterface getWorkflowInterfaces listModelDownloads listModels listWorkflowSummaries preflightWorkflow startModelDownload uploadTemporaryAsset The public boundary is now HTTP for discovery and control, and MessagePack WebSocket for execution and live events. No SDK-specific tRPC procedure or discovery WebSocket command remains. Release procedure Merge the NodeTool candidate and publish a NodeTool release. Attach the deterministic contract bundle and this semantic operation list. Update nodetool-sdk from the released bundle so its lock records the release tag rather than source_kind: candidate_commit. Publish the matching C# and VL packages only after their clean-package and live-server gates pass.--- # Phase 8A caller and execution-wire audit Source: https://docs.nodetool.ai/sdk/phase-8a-caller-and-execution-audit-2026-08-23.html Phase 8A caller and execution-wire audit Date: 2026-08-23 NodeTool base: origin/main at 7729d6691a SDK audit checkout: nodetool-sdk at 2c57c93 This audit replaces the Phase 0 compatibility inventory. It defines what the first public SDK contract keeps and what Phase 8A removes. SDK-only tRPC procedures The four procedures exist only as wrappers around SdkV1ImplementationBoundary: Procedure Definition Product callers nodes.sdkTypeInventory packages/websocket/src/trpc/routers/nodes.ts none workflows.sdkSummaries packages/websocket/src/trpc/routers/workflows.ts none workflows.interface packages/websocket/src/trpc/routers/workflows.ts none workflows.interfaces packages/websocket/src/trpc/routers/workflows.ts none The search covered packages/, web/src/, electron/src/, and mobile/src/, excluding tests, generated schemas, fixtures, and build output. The remaining references are the definitions, the shared SDK handler map, tests, and the sandbox coverage inventory. No TypeScript product caller uses these procedures. Phase 8A can delete all four procedures, sdk-v1-trpc-error.ts, their sandbox coverage entries, and their wrapper-only tests after the HTTP boundary remains covered. Discovery and lifecycle WebSocket commands The unified runner still dispatches these compatibility commands: list_workflow_summaries get_workflow_interface get_workflow_interfaces get_node_type_inventory get_capabilities preflight_workflow No NodeTool web, mobile, or Electron caller uses them. The C# execution client still implements the four discovery commands as an optional compatibility path. Capabilities and preflight have HTTP clients. Phase 8C removes the C# command path after it pins the replacement NodeTool bundle. Phase 8A removes the server dispatchers after the execution contract below is declared and frozen. C# execution commands sent to /ws NodeToolExecutionClient sends these MessagePack command envelopes: Command Payload type Purpose run_job RunJobRequest Start a workflow, inline graph, or single-node graph. cancel_job CancelJobData Cancel one active job. reconnect_job ReconnectJobData Reattach and request retained frames or a persisted terminal snapshot. stream_input StreamInputData Send one value to a live input. end_input_stream EndInputStreamData Close one live input. update_node_properties UpdateNodePropertiesData Change one active node executor. The client does not use submit_job, get_job_snapshot, or subscribe_job. Those planned lifecycle commands must not appear in the public v1 manifest. Server frames used by C# execution The SDK routes these frames into an ExecutionSession: Frame SDK behavior sdk_execution_target Records the runner identity used for reconnect diagnostics. job_update Tracks queued, running, failed, cancelled, suspended, and completed states. node_update Raises node status and result events. node_progress Updates session progress. output_update Updates named outputs and append/replace streams. chunk Delivers standalone streamed content. reconnect_job can first produce job_resumed, followed by retained copies of the same execution frames. If no retained runner exists, it produces a persisted job_update. The SDK currently ignores the job_resumed header but depends on the replay or persisted update that follows it, so the header remains part of the declared wire. A completed SDK run is not terminal until job_update.status is completed and job_update.result.outputs exists. The SDK requests this with require_terminal_result: true. Malformed frames and command failures use the untyped protocol-rejection envelope with error equal to invalid_frame, invalid_message, or invalid_command. The SDK converts an uncorrelated rejection into a failed execution session. This error envelope is part of the execution contract until Phase 8A replaces it with the release error shape. Recognized but unused C# message types The C# MessagePack decoder recognizes preview_update, progress_update, connection_status, and typed error messages. NodeToolExecutionClient does not route them into an execution session, and the current unified runner does not publish preview_update or progress_update on the SDK execution path. They are not part of the first public SDK execution contract. Phase 8A deletion boundary Keep: the six execution commands; job correlation, cancellation, reconnect, replay, live input, and node updates; sdk_execution_target, job_resumed, the six consumed execution/error frame shapes above, and the authoritative terminal result; the HTTP SDK service boundary and multipart temporary upload. Remove after the replacement execution declarations and goldens pass: the four SDK-only tRPC procedures; the six discovery/lifecycle WebSocket commands; planned lifecycle job declarations; discovery RPC fixtures, handlers, tRPC error mapping, and compatibility-only inventories.--- # SDK Protocol v1 Compatibility Rules Source: https://docs.nodetool.ai/sdk/protocol-v1-compatibility.html SDK Protocol v1 Compatibility Rules Rules for changing the public SDK v1 contract, the layout of the released contract bundle, and the semantic diff that classifies contract changes. The plan behind this file is docs/sdk/sdk-trpc-consolidation.md §5.5. Change rules Response additions are tolerated. Generated response schemas allow additional fields at every object level. Consumers must ignore response fields, event types, and enum values they do not understand. Adding an optional response field, a new operation, or a new planned operation is an additive change and needs no version decision. Request changes need an explicit decision. Generated request schemas reject unknown fields, so a new request field, a changed request schema, or a new required body on an existing operation is not additive. It requires a protocol version bump or an advertised capability that the client checks before sending the new shape. Error codes are stable. A declared error status (HTTP) or error code (WebSocket) may be added to an operation, but not removed or repurposed within protocol v1. Descriptions may be reworded; the status/code set a client can rely on only grows. Operation identity is fixed. An operation ID keeps its method and path (HTTP) or channel, command, and direction (WebSocket) for the life of protocol v1. A planned operation may become implemented; the reverse is a regression that the diff flags. Bundle layout npm run build:sdk-contract-bundle --workspace=packages/protocol stages the bundle in packages/protocol/dist/sdk-contract-bundle/ and writes the archive packages/protocol/dist/nodetool-sdk-v1-contract-<protocol_version>.tar: sdk-v1.bundle.json bundle index: digests, commit, release schema/sdk-v1.openapi.json full HTTP contract (implemented + planned) schema/sdk-v1.openapi.implemented.json implemented-only client-generation input schema/sdk-v1.asyncapi.json full WebSocket contract schema/sdk-v1.asyncapi.implemented.json implemented-only WebSocket profile schema/sdk-v1.discovery.schema.json discovery request/response components schema/sdk-v1.lifecycle.schema.json lifecycle request/response components schema/sdk-v1.execution.schema.json execution command/event components schema/sdk-v1.operations.json operation manifest with status per operation schema/sdk-v1.manifest.json artifact manifest with per-artifact digests fixtures/… JSON goldens and MessagePack hex captures docs/protocol-v1-compatibility.md this file The archive is a plain USTAR tar with fixed metadata (mtime 0, uid/gid 0, mode 0644, sorted entries) under one top-level directory named after the archive. The same inputs and the same --commit/--release values produce byte-identical output; the producing commit and release live only in sdk-v1.bundle.json. Digest scheme and verification sdk-v1.bundle.json lists every bundled file with its SHA-256 digest and size, plus one bundle digest. To verify a bundle: For each entry in files, recompute the SHA-256 of that file’s bytes and compare it to the recorded sha256. Format one line per file as <sha256> <path>\n (two spaces, LF), sort the lines bytewise, and concatenate them. The SHA-256 of that concatenation must equal bundle_digest. The index itself is not part of files — it carries the digest, so it cannot be covered by it. Consumers pin bundle_digest, the NodeTool release, and protocol_version. Semantic diff categories tsx packages/protocol/scripts/diff-sdk-contract.ts <old> <new> [--json] compares two operation manifests (or two bundle/schema directories, which also compares JSON-schema $defs and the implemented OpenAPI route inventory) and classifies every change: additive — new operation (implemented or planned), planned → implemented, added error declaration, new schema definition, new response property. risky — changed auth or feature policy, changed error description, implemented → planned, changed response content type or success status, changed response schema reference, removed or changed non-request schema definition. breaking — removed operation, changed method/path or channel/command/direction, changed transport, removed declared error status/code, changed request content type, parameters, body requirements, or request schema on an existing operation, removed required response property, route removed from the implemented OpenAPI profile. Exit codes: 0 for no or additive-only changes, 2 when the diff contains a risky change and no breaking one, 3 when it contains a breaking change.--- # Public SDK Protocol v1 Scope Source: https://docs.nodetool.ai/sdk/protocol-v1-scope.html Public SDK Protocol v1 Scope This document defines the first public NodeTool SDK transport boundary. The declaration registry and generated artifacts are authoritative for individual schemas and status codes. Transport split SDK discovery and control use JSON HTTP under /api/sdk/v1. SDK execution and live events use MessagePack WebSocket frames on /ws. C# clients do not use tRPC; tRPC remains an internal product API for TypeScript clients. The HTTP profile contains: workflow summaries and workflow interfaces; node-type inventory; capabilities and workflow preflight; model catalog and model-download control; temporary asset upload. The single-workflow route is GET /api/sdk/v1/workflows/{id}/interface?version=1. There is no legacy /api/workflows/{id}/interface SDK alias. The execution profile contains these client commands: run_job; cancel_job; reconnect_job; stream_input; end_input_stream; update_node_properties. It also declares target selection, replay, live updates, terminal results, and protocol errors consumed by the SDK. Discovery, capabilities, preflight, and model operations are not duplicated as WebSocket RPC commands. HTTP behavior One declaration-driven Fastify plugin owns every SDK v1 route and calls the typed SDK service boundary directly. Multipart temporary upload is the only specialized adapter. Every SDK HTTP failure has exactly this JSON shape: { "code": "STABLE_MACHINE_CODE", "message": "Safe human-readable message", "retryable": false } Responses use application/json. The public error contract has no detail, apiCode, trpcCode, stack trace, or provider/backend detail. Unexpected failures are logged server-side and return INTERNAL_ERROR with Internal server error. Request schemas reject unknown fields. Response schemas are additive: clients must ignore response fields they do not understand. WebSocket behavior Binary frames contain one MessagePack object. Execution commands keep their existing job correlation, cancellation, reconnect/replay, streaming input, streaming output, live progress, and terminal delivery behavior. JSON-text mode may remain available for NodeTool diagnostics, but it is not the C# SDK’s public execution encoding. The byte-exact execution fixture is packages/protocol/fixtures/sdk-v1/execution-wire.json. HTTP goldens are under packages/protocol/fixtures/sdk-v1/http-*.json. Generated contract artifacts The Zod declarations generate: sdk-v1.openapi.json and sdk-v1.openapi.implemented.json; sdk-v1.asyncapi.json and sdk-v1.asyncapi.implemented.json; discovery, lifecycle, and execution JSON Schema bundles; the operation manifest and artifact manifest. Use npm run generate:sdk-protocol --workspace=packages/protocol to regenerate them and npm run check:sdk-protocol --workspace=packages/protocol to detect stale artifacts. Release bundles include the artifacts, fixtures, compatibility policy, and deterministic digests. Authentication and feature policy The normal Fastify authentication hook owns authenticated identity. The SDK route layer never trusts a caller-supplied identity forwarded through a second HTTP adapter. Missing and inaccessible workflows both map to WORKFLOW_NOT_FOUND. The emergency kill switches remain: NODETOOL_DISABLE_SDK_WORKFLOW_INTERFACE_V1 for workflow discovery and node-type inventory; NODETOOL_DISABLE_SDK_LIFECYCLE_V1 for capabilities, preflight, and temporary upload; NODETOOL_REQUIRE_SDK_AUTH_V1 for deployments that require SDK discovery authentication in otherwise trusted-local mode. These policies do not change non-SDK web, Electron, mobile, CLI, agent, asset, or execution behavior. Not public SDK contracts The following remain internal: tRPC procedures and batching details; Python bridge messages and process lifecycle; database schemas; editor and ReactFlow state; provider-specific request objects; runner implementation details and legacy full-workflow editor responses. The SDK uses NodeTool’s existing workflow, validation, runner, job, model, and asset services; it does not introduce a second execution engine.--- # SDK v1 contract convergence: canonical cross-repository plan Source: https://docs.nodetool.ai/sdk/sdk-trpc-consolidation.html SDK v1 contract convergence: canonical cross-repository plan Status: Canonical plan Last updated: 2026-08-23 Repositories: nodetool, nodetool-sdk Public protocol: SDK v1 HTTP and WebSocket contracts This is the only active plan for SDK contract convergence. It combines the server consolidation work with the C# SDK drift-prevention and usability work. All task status, design changes, and acceptance evidence belong in this file. Execution status (2026-08-23): NodeTool producer Phases 0 through 3 and the Phase 5 producer gates are complete on branch claude/sdk-trpc-consolidation-ut79fs through commit 686a0d710e. Phase 0 evidence: docs/sdk/phase-0-baseline-2026-08-22.md. The byte-frozen goldens remain unchanged. In nodetool-sdk, Phase 4, the Phase 5 consumer gates, and the non-preset Phase 7 facade work are complete on the matching branch through commit 2c57c93 (facade commit 765ebe2). Phase 6 benchmarks, benchmark-dependent presets, the Phase 8 pre-release reset, and releases remain open. Pre-release decision (2026-08-23): There are no external SDK users. Phase 8 therefore replaces the compatibility-preserving adapter set with the intended first public contract. The Phase 0 baseline remains evidence of the old shape, but it is not the release target. Phase 8 may remove unused SDK-specific tRPC procedures, SDK discovery RPCs, old URLs, mutable C# configuration, and public low-level escape hatches. It must preserve product functions through the session facade. It creates one reviewed replacement baseline, then freezes it. 1. Outcome Simplify the SDK implementation before its first public release: Use HTTP for SDK discovery and control operations. Use MessagePack WebSocket only for execution and live events. Implement each SDK operation through one transport-neutral server path. Declare each public HTTP or WebSocket operation once and generate route and contract artifacts from the shared operation registry. Keep tRPC for NodeTool’s TypeScript applications, not as an SDK transport. Publish a deterministic contract bundle with each NodeTool release. Make nodetool-sdk pin and mechanically consume that bundle. Detect server/SDK drift before either repository releases. Make the C# connection/session facade the public entry point. Keep each useful operation, but make transport clients implementation details. This reduces drift between NodeTool and the C# SDK. Moving the C# client to raw tRPC would not solve drift by itself. It would replace a stable, language-neutral contract with tRPC URL encoding, envelopes, and internal procedure names. 2. Decisions 2.1 Public transport decisions /api/sdk/v1 is the public discovery and control contract for non-TypeScript SDKs. Every SDK HTTP operation moves under this prefix. The public SDK WebSocket contract contains execution and live events only. tRPC remains the internal API for the web, mobile, and Electron TypeScript clients. Its dotted procedure names are not part of the public SDK contract. SDK-specific tRPC procedures with no TypeScript caller are removed before the first release. Public C# types do not expose tRPC request or response envelopes. Phase 8 defines one clean pre-release contract. After its replacement baseline lands, paths, methods, status codes, JSON names, content types, and MessagePack execution envelopes are frozen again. 2.2 Implementation decisions A protocol operation registry owns stable operation IDs, transport metadata, schemas, authentication policy, feature policy, and errors. Its HTTP declarations include methods and paths; its WebSocket declarations include commands/events and directions. The release registry contains implemented public operations only. Roadmap operations stay in design notes. A typed server handler map, keyed by request/response operation IDs, owns the binding to server code. Event publishers have a separate completeness check. The protocol package does not contain tRPC procedure-name strings. A transport-neutral SDK service facade coordinates existing domain services. It is not a replacement for those services and must not become a large class containing all business logic. SDK HTTP routes call the shared service. TypeScript product tRPC procedures call domain services directly. SDK discovery does not have tRPC or WebSocket adapters. Temporary asset upload remains a multipart Fastify adapter over the shared temporary-storage service. It is not exposed as JSON or base64 tRPC input. 2.3 Pre-release compatibility boundary Preserve these independent controls and their current error behavior: NODETOOL_REQUIRE_SDK_AUTH_V1; NODETOOL_DISABLE_SDK_WORKFLOW_INTERFACE_V1; NODETOOL_DISABLE_SDK_LIFECYCLE_V1. Do not replace the two feature switches with one switch. Discovery and lifecycle must remain independently controllable. Remove the four SDK-specific tRPC procedures after repeating the in-repository caller audit. Do not remove general product tRPC procedures. Keep separate procedures when response shapes differ. Do not add a fields=full|summary mode to workflows.list. Do not remove /ws/download or change the web model-download flow as part of core convergence. That is separate product work with separate measurements. Phase 8 may replace obsolete adapter tests and goldens in one baseline-reset commit. Keep service and behavior coverage for every retained operation. After that commit, fixture changes again require an explicit protocol change. 3. Repository ownership and phase map Repository labels used throughout this plan: [nodetool] means work only in M:/P/NODETOOL/____REPOS____/nodetool. [nodetool-sdk] means work only in M:/P/NODETOOL/____REPOS____/nodetool-sdk. [both] means coordinated changes or verification in both repositories. A joint phase should still use separate, independently releasable pull requests unless a shared CI job requires both checkouts. Phase Owner Purpose Main output 0 [nodetool] Freeze and measure Current contract inventory and goldens 1 [nodetool] Declare and publish Route declaration and deterministic bundle 2 [nodetool] Unify implementation Service facade and stable service errors 3 [nodetool] Converge adapters One route plugin; thin REST, WS, and tRPC adapters 4 [nodetool-sdk] Consume mechanically Pinned bundle and generated/verified C# wire layer 5 [both] Prevent drift Cross-repository conformance and compatibility CI 6 [both] Measure options Reproducible execution and upload benchmarks 7 [nodetool-sdk] Simplify C# use Primary session facade and measured presets 8 [both] Reset before release Clean HTTP/WS boundary and lean C# public API 9 [both] Release Publish the clean producer and matching consumer Phases 0 through 7 record the compatibility-preserving implementation. Phase 8 uses that evidence to remove duplicate surfaces before release. It does not wait for Phase 6. Phase 6 must still finish before a later change names or recommends a low-latency preset. Phase 9 is the first public release gate. 4. Confirmed current state The following inventory was measured on 2026-08-22. Phase 0 must refresh file locations and counts if the implementation changes before work starts. 4.1 Public HTTP surface in nodetool Operation Public route Current policy Node type inventory GET /api/sdk/v1/node-types Discovery; workflow-interface flag applies Capabilities GET /api/sdk/v1/capabilities Discovery; lifecycle flag applies Model catalog GET /api/sdk/v1/models Discovery List model downloads GET /api/sdk/v1/model-downloads Authenticated Start model download POST /api/sdk/v1/model-downloads Authenticated Cancel model download POST /api/sdk/v1/model-downloads/cancel Authenticated Preflight workflow POST /api/sdk/v1/preflight Authenticated; lifecycle flag applies Workflow summaries GET /api/sdk/v1/workflows Discovery; workflow-interface flag applies Workflow interfaces POST /api/sdk/v1/workflow-interfaces Discovery; workflow-interface flag applies One workflow interface GET /api/workflows/:id/interface?version=1 Discovery; workflow-interface flag applies Temporary asset upload POST /api/sdk/v1/assets/temporary Authenticated multipart; lifecycle flag applies The ten /api/sdk/v1 operations use nine paths because model-downloads supports both GET and POST. The single-workflow interface is a public SDK v1 operation even though its compatible URL is outside the SDK prefix. “Discovery” describes the route’s default SDK policy. Normal server authentication or NODETOOL_REQUIRE_SDK_AUTH_V1 can still require authentication for discovery routes. At investigation time, the Fastify route modules mounted handlers through bridge() into WHATWG Request/Response handlers while http-api.ts also dispatched a subset. Phase 3 replaces those duplicate registrations with one declaration-driven Fastify plugin and removes the SDK entries from the second dispatcher. 4.2 WebSocket and tRPC overlap in nodetool handleSdkV1LifecycleRpc handles get_capabilities and preflight_workflow. At investigation time, the unified WebSocket runner called these tRPC procedures in process: workflows.sdkSummaries; workflows.interface; workflows.interfaces; nodes.sdkTypeInventory. Phase 3 removes that internal hop but retains the procedures as thin wrappers for external compatibility. Any rename or removal still requires a consumer audit. Current tRPC uses plain JSON without a transformer. This makes binary temporary upload a poor tRPC operation. tRPC is mounted at /trpc; protected procedures read the authenticated user from the request context. The SDK policy is separate and must not be bypassed by a new direct caller. 4.3 Existing contract material packages/protocol already contains generated SDK v1 OpenAPI, AsyncAPI, JSON schemas, a manifest, and a baseline fixture. Generation starts from a hand-maintained path table. The route registrations and that path table can still diverge. The known focused parity baseline at the time of investigation was 3 test files and 28 passing tests after rebuilding stale workspace dependencies. It covers REST, tRPC, and SDK WebSocket parity for the focused operations. Phase 0 must record the exact commands and commits used for the new baseline. 4.4 Current consumer boundary The C#/VL SDK in nodetool-sdk is the in-development out-of-tree consumer. It is the release candidate for the public SDK, but there are no external SDK users yet. The TypeScript package uses tRPC and /ws; it does not depend on the public C# HTTP surface in the same way. The NodeTool web application does not currently call the public SDK routes. The server and C# SDK duplicate some endpoint, DTO, and error knowledge, so they can drift even when each repository’s local tests pass. 4.5 Existing execution controls The WebSocket run_job.execution_options contract already carries the main performance and durability controls: persistence mode; event detail; asset persistence. The current C# WorkflowExecutionOptions defaults are job persistence, full events, and temporary asset persistence. Other NodeTool clients can use server defaults with different asset behavior. The consolidation must keep every individual option and both sets of current defaults. A named preset is only a C# usability layer over these controls. 5. Target architecture flowchart LR D["SDK v1 operation registry"] --> A["OpenAPI, AsyncAPI, JSON Schema, manifest"] D --> R["Typed SDK HTTP handler map"] D --> E["Execution dispatcher and publisher checks"] R --> S["Transport-neutral SDK service facade"] S --> X["Existing domain services"] H["SDK HTTP adapter"] --> R W["MessagePack execution adapter"] --> Q["Execution runtime"] Q --> X T["General product tRPC"] --> X A --> B["Deterministic released contract bundle"] B --> C["Pinned C# wire layer"] C --> P["Session-first handwritten C# API"] 5.1 Protocol operation registry in [nodetool] Add declarations such as packages/protocol/src/api-schemas/sdk-v1-http-operations.ts and sdk-v1-websocket-operations.ts, backed by shared operation metadata types. Each implemented public operation declares: a stable operation ID; transport and direction; an HTTP method and path, or a WebSocket command/event identity and channel; applicable path, query, header, body, or message schemas; response schema and content type when applicable; authentication and feature policy identifiers; declared public error codes and statuses; a binary or streaming marker where JSON generation does not apply. The declarations must not import server packages or name tRPC procedures. A representative shape is: type SdkV1OperationCommon = { id: SdkV1OperationId; auth: "discovery" | "authenticated"; feature: "workflow-interface" | "lifecycle" | null; errors: readonly SdkV1ErrorDeclaration[]; }; type SdkV1OperationDeclaration = SdkV1OperationCommon & ( | { transport: "http"; method: "GET" | "POST"; path: string; request: SdkV1HttpRequestSchemas; response: SdkV1HttpResponseDeclaration; } | { transport: "websocket"; direction: "client-command" | "server-event"; channel: string; command?: string; message: SdkV1MessageDeclaration; } ); Generate OpenAPI paths from HTTP declarations and AsyncAPI channels/messages from WebSocket declarations. The generated artifacts and client input contain only operations that exist in the release. A proposed operation cannot enter the registry until its server implementation and contract tests land together. 5.2 Typed handler map and service facade in [nodetool] Define a server handler type from each HTTP request/response declaration’s input and output schema. Require an exhaustive handler map for all declared HTTP operation IDs. Check execution command dispatchers and server-event publishers separately against the WebSocket declarations. This gives compile-time or CI failures for missing implementations and avoids runtime lookup by a dotted tRPC string. The handler map calls a transport-neutral facade such as SdkV1Service. The facade may be a small collection of domain-focused services rather than one class. It should coordinate: workflow discovery and interface lookup; node type inventory; runtime capabilities; workflow preflight; model catalog and download lifecycle; temporary asset storage. Existing domain services remain the source of business behavior. The facade owns common input validation, authorization requirements, feature policy, correlation metadata, and stable service errors where those concerns are shared. 5.3 Stable transport-neutral errors in [nodetool] Define an SdkV1ServiceError with at least: stable SDK error code; safe public message; logical status/category; retryability when applicable; internal cause and logging context that are never serialized directly. Map it at each edge: SDK HTTP maps it to exactly { "code": string, "message": string, "retryable": boolean }, the declared HTTP status, and application/json; it does not serialize detail, an internal cause, or an RPC envelope; execution WebSocket errors remain part of the execution runner protocol; general product tRPC errors remain part of the internal TypeScript API. After the Phase 8 reset, golden tests prove the release error bodies exactly. Content type, method handling, and validation order use one SDK HTTP policy. 5.4 Adapter rules in [nodetool] HTTP adapters may parse HTTP, enforce the declared route policy, call the typed handler, and serialize the declared response. They contain no domain logic. The SDK WebSocket adapter contains execution commands and live events only. It does not serve discovery or control reads and does not call tRPC. General product tRPC procedures remain typed TypeScript API adapters. They are not aliases for SDK HTTP operations. Shared behavior lives below both adapters in a domain service when both product and SDK calls need it. Temporary multipart upload stays a dedicated Fastify adapter. Do not register an assets.uploadTemporary tRPC procedure with Uint8Array, JSON, or base64 input. The adapter calls the shared temporary-storage service directly. 5.5 Deterministic contract bundle in [nodetool] Build one versioned archive that contains: OpenAPI for implemented HTTP operations; AsyncAPI for public SDK execution WebSocket operations; JSON schemas for requests, responses, and stable errors; the implemented operation manifest; JSON golden success and error fixtures; exact execution MessagePack golden frames and their semantic JSON form; per-file SHA-256 digests and one bundle digest; protocol compatibility rules and the producing NodeTool commit/release. Generation must be deterministic: stable ordering, line endings, archive metadata, and digests. Publish the same bytes as a NodeTool release asset and inside @nodetool-ai/protocol, or publish one location plus a verified pointer. If both locations are used, their bundle digests must match. 5.6 C# consumption boundary in [nodetool-sdk] Pin the NodeTool release, protocol version, and bundle digest in contracts/sdk-v1/. Normal builds and tests must work offline. Generate or mechanically verify only the internal wire layer: endpoint paths and methods; HTTP request and response DTOs; stable error DTOs; JSON property and enum serialization metadata. Keep public domain records and SdkApiException handwritten. Map wire DTOs to public types in one place. Keep transport clients internal behind the session and session-owned domain services. Keep the MessagePack execution runtime handwritten, but verify its envelopes and bytes against AsyncAPI and goldens. Evaluate available OpenAPI generators before selecting one. Require deterministic, reviewable output, correct nullable annotations, snake-case wire names, additive response-field tolerance, and compatibility with the SDK’s supported .NET targets. If none meet these requirements, add a focused repository generator from the published schemas. 6. Execution plan Phase 0 [nodetool] - Freeze and measure the current contract Purpose: make hidden behavior explicit before moving code. Record the exact server commit and commands for the baseline. Inventory every implemented and planned SDK HTTP operation. Inventory every WebSocket command and event declared in the intermediate SDK v1 registry. Phase 8A separately inventories and declares the actual execution wire used by C#. Inventory each SDK-related tRPC procedure and all in-repository callers. Audit known out-of-repository tRPC consumers before planning removal. Record route ownership, auth policy, feature policy, request limits, content types, statuses, headers, error shapes, and retry semantics. Capture JSON success and error goldens for every HTTP operation. Capture exact MessagePack fixtures for public WebSocket requests, responses, events, and errors. Capture multipart upload size-limit, filename, media-type, temporary-ID, and cleanup behavior. Add route and WebSocket inventory tests around the current declarations. Prove each new drift check fails by changing one fixture or declaration in a temporary local test change, then restore it. Done in commit 7398c2d: 20 fixtures under packages/protocol/fixtures/sdk-v1/, five test files under packages/websocket/tests/ (sdk-v1-*.test.ts), and docs/sdk/phase-0-baseline-2026-08-22.md with the drift-check proof table. Exit criteria: Current public behavior is reproducible without reading handler code. Active tRPC callers are correctly identified. Goldens include feature-disabled and authentication failures. No runtime behavior has changed. Phase 1 [nodetool] - Declare and publish the contract Purpose: make one machine-readable declaration drive public artifacts and route completeness. Add HTTP and WebSocket operation declarations to packages/protocol. Give every operation a stable ID and explicit implementation status. Include the compatible non-prefixed workflow-interface route. Mark temporary upload as multipart/binary. Generate OpenAPI route entries from HTTP declarations. Generate or validate AsyncAPI command, event, and direction inventory from WebSocket declarations. Generate implemented-only and full-manifest profiles. Fail when operation IDs or method/path pairs are duplicated. Fail when an implemented route lacks schemas, declared errors, or policy. Fail when a registered route and implemented OpenAPI inventory differ. Add deterministic bundle generation and digest verification. Add a semantic contract diff that labels additive, risky, and breaking changes. Document v1 rules: response additions are tolerated; request changes need an explicit version or capability decision when inputs are strict. Done in commits 322dbc0 (registry: sdk-v1-operations.ts, sdk-v1-http-operations.ts, sdk-v1-websocket-operations.ts; declaration-driven generation with byte-identical artifacts; implemented-only profiles; sdk-v1.operations.json) and 8486b9e (build:sdk-contract-bundle, diff:sdk-contract, docs/sdk/protocol-v1-compatibility.md, release-CI bundle attachment). Exit criteria: Repeated generation produces byte-identical artifacts. Planned job operations are visible in the full manifest but excluded from the normal C# generation input. Changing an operation declaration changes the generated artifacts or fails CI. Phase 2 [nodetool] - Add the transport-neutral service boundary Purpose: remove duplicated behavior before changing route registration. Define SdkV1ServiceError and transport mappings. Define the typed handler map from implemented request/response operation IDs and an event-publisher completeness check. Migrate one read-only operation as a proof of the boundary. Prove its REST, WebSocket or tRPC forms remain byte/semantically equal as applicable. Migrate workflow summaries and interfaces. Migrate node inventory, capabilities, and preflight. Migrate model catalog and model-download lifecycle. Move temporary-storage behavior behind a shared service while keeping multipart parsing at the HTTP edge. Keep existing domain services focused; split the facade by concern if it starts accumulating business logic. Keep focused service tests and add facade contract tests. Exit criteria: Each implemented request/response operation has one service/handler path, and each implemented event has one declared publisher path. Adapters no longer duplicate validation, feature checks, or error decisions. Existing flags remain independent and retain existing public errors. Phase 3 [nodetool] - Converge REST, WebSocket, and tRPC adapters Purpose: make registrations and transport wrappers unable to drift from the declared contract. This completed phase records the compatibility-preserving intermediate state. Phase 8 removes the duplicate adapters before release. Phase 3A - Prove a shadow facade Add a temporary opt-in or test-only shadow mount for every implemented HTTP declaration. Use /api/sdk-next/v1 where possible and an explicit test-only alias for the compatible non-prefixed workflow-interface route. Compare old and shadow routes for bodies, headers, content types, status codes, authentication, feature flags, and errors. Include reverse-proxy subpaths and URL/query encoding in parity tests. Do not expose planned operations from the shadow mount. Phase 3B - Cut over HTTP registration Add one SDK v1 Fastify route plugin driven by the HTTP declarations. Keep the compatible public path for the single-workflow interface. Keep multipart upload on its binary adapter path. Remove duplicate http-api.ts dispatch entries only after parity passes and no supported caller depends on that internal dispatcher. Remove the shadow mount after the real mounts use the same registration path. Phase 3C - Converge SDK WebSocket calls Make lifecycle commands call the shared handler/service boundary. Replace the unified WebSocket runner’s in-process tRPC hop with typed handler/service calls where this does not alter scheduling or envelopes. Preserve caller identity and scopes, local-trust behavior, runner registration, sdk_execution_target, correlation IDs, cancellation, reconnect, event order, and MessagePack bytes. Phase 3D - Retain thin tRPC adapters Make the four current SDK-related procedures call the same boundary. Apply normal tRPC auth plus the declared SDK policy to direct calls. Keep summary and full-result procedures separate. Do not add a JSON temporary-upload procedure. Keep existing names until the consumer audit and compatibility policy permit deprecation. Exit criteria: One plugin visibly owns all public SDK HTTP registrations. HTTP route inventory equals implemented OpenAPI inventory. SDK WebSocket command/event inventory equals implemented AsyncAPI inventory. REST, WebSocket, and retained tRPC adapters contain transport logic only. All Phase 0 goldens still pass. Phase 4 [nodetool-sdk] - Pin and consume the contract Purpose: replace handwritten server-contract duplication with a mechanical, reviewable update. This completed phase records the compatibility-preserving consumer. Phase 8C reuses its generator and tests while replacing the public API baseline. Add contracts/sdk-v1/ with the pinned manifest and bundle, or a lock file plus a verified local cache. Record NodeTool release, protocol version, bundle digest, and minimum supported server version. Add an explicit contract-update command that fetches or copies a named release, verifies digests, regenerates the wire layer, runs conformance tests, and prints a semantic diff. Keep normal build and test commands offline. Run the generator evaluation described in Section 5.6. Generate or mechanically verify internal HTTP wire DTOs, errors, endpoints, methods, and serialization metadata. Map internal wire types to the existing public types in one module. Add a freshness check that fails after manual generated-file edits. Verify handwritten MessagePack code against the pinned fixtures. Preserve INodetoolClient, IWorkflowDiscoveryClient, model services, execution clients, session APIs, injected HttpClient ownership, cancellation, retries, token refresh, request IDs, and proxy subpaths. Done in nodetool-sdk commits e5e297e (pinned bundle and HTTP/MessagePack golden conformance) and e0b4fd7 (focused wire generator, mappings, runtime validation, update command, freshness checks, and conformance CI). Exit criteria: A contract update produces a deterministic C# wire-layer diff. Existing public C# consumers compile unchanged. Every pinned JSON and MessagePack fixture is consumed successfully or fails with the documented safe behavior. Phase 5 [both] - Add cross-repository conformance CI Purpose: detect incompatible drift before release, not after SDK users report it. The completed checks describe the intermediate contract. Phase 8D replaces their fixture inventory and keeps the same drift-detection mechanism. [nodetool] producer gates Validate every golden JSON fixture against generated schemas. Validate exact MessagePack fixtures. Verify route and command inventories against OpenAPI and AsyncAPI. Run REST, WebSocket, and retained tRPC parity tests. Fail on removed required response fields, repurposed fields, stricter v1 inputs, or changed retry semantics without an explicit protocol decision. Build and publish the candidate contract bundle in release CI. [nodetool-sdk] consumer gates Deserialize every success and stable error fixture. Decode every MessagePack fixture and compare its public meaning. Re-encode supported C# requests and validate them against request schemas. Prove unknown additional response fields are ignored. Prove invalid enum values and missing required fields fail safely. Run against every supported .NET target and the VL adapter/package build. [both] compatibility matrix and update flow Test SDK main against the minimum supported NodeTool release, latest release, and NodeTool main/candidate contract. Test a released SDK against NodeTool main before a contract-affecting NodeTool release. Treat documented feature-disabled responses as supported outcomes. Publish the tested server/SDK matrix with SDK releases. Open an automated, review-required SDK contract update when NodeTool publishes a new digest. Include manifest, generated changes, semantic diff, test results, and any proposed minimum-version change. Never auto-merge a contract update. Exit criteria: Drift in routes, schemas, errors, or MessagePack frames fails before release. Contract updates are reproducible and reviewable. Supported server/SDK combinations are explicit. Phase 6 [both] - Benchmark execution options and binary transfer Purpose: base SDK convenience presets and transport choices on measurements. [nodetool] benchmark support Expose or record server timing for database, asset, thumbnail, and queue work without changing public response semantics. Keep multipart temporary upload as the binary baseline. Record server commit, configuration, hardware, workflow IDs, and raw results. [nodetool-sdk] benchmark runner Cover a primitive no-media workflow, repeated image input/output, one generated image, streaming text, streaming audio, and a large temporary upload. Compare job/session persistence, full/outputs/terminal events, and automatic/temporary asset persistence. Measure connection and capability negotiation time, upload time, submission-to-first-output, total time, event count and bytes, database and asset time, allocations, peak memory, reconnect, and terminal result. Prove multipart upload is faster or more memory-safe than a JSON/base64 alternative for representative media. Verify that terminal-only delivery intentionally withholds provisional outputs and is not presented as interactive streaming. Exit criteria: Results are reproducible from stored configuration and raw data. Any proposed low-latency preset has a measured benefit and documented durability/delivery costs. No transport change is justified only by assumption. Phase 7 [nodetool-sdk] - Simplify the public C# experience Purpose: make the common path small without hiding advanced controls. This completed phase records the intermediate public facade. Phase 8C keeps its useful operations but removes compatibility-only low-level exposure. Make NodeToolConnectionSession the primary documented entry point. Let the session coordinate discovery, capabilities, preflight, model catalog/downloads, execution, and asset transfer. Preserve design-time HTTP fallback when WebSocket is unavailable. Keep low-level HTTP and WebSocket clients public for advanced use, testing, and host adapters. Keep every WorkflowExecutionOptions value independently settable. Add named presets only if Phase 6 supports them. Candidate names: Default: job persistence, full events, and temporary assets, matching current C# defaults; LowLatencyInteractive: session persistence, output-only events, and temporary assets; Durable: job persistence, full events, and automatic asset-library persistence. Implement presets as immutable factories/values that callers can copy and override. Negotiate non-default values through capabilities before submission. Return a clear compatibility error for an unsupported requested option; never silently change semantics. Document history, reconnect, event delivery, temporary retention, and asset-library visibility for each preset. Exit criteria: Ordinary SDK use needs one connection/session abstraction. Existing code and low-level control remain supported. Presets preserve server defaults and expose their tradeoffs. Phase 8 [both] - Define the first public contract Purpose: remove compatibility code before it becomes a released API. Work is sequential: NodeTool defines the contract, then the SDK pins and consumes it. Phase 8A [nodetool] - Remove duplicate SDK transports Repeat the Phase 0 caller audit against the current branch. Record every in-repository caller of the four SDK-specific tRPC procedures and six SDK discovery/lifecycle WebSocket commands. Inventory the actual C# execution wire before deleting any WebSocket declaration: run_job, cancel_job, reconnect_job, stream_input, end_input_stream, update_node_properties, and every response, live event, replay, terminal result, and error that the SDK consumes. Add that execution inventory to the SDK v1 WebSocket registry, AsyncAPI, dispatcher/publisher completeness checks, and byte-exact MessagePack goldens. The six discovery/lifecycle RPC declarations may be removed only after the replacement execution contract is complete. Remove nodes.sdkTypeInventory, workflows.sdkSummaries, workflows.interface, and workflows.interfaces after the audit confirms that no TypeScript product caller uses them. Remove sdk-v1-trpc-error.ts, related sandbox inventory entries, and tRPC-only parity tests after equivalent HTTP service coverage exists. Remove the six SDK discovery/lifecycle WebSocket commands. Keep the WebSocket transport, execution commands, job correlation, cancellation, reconnect, replay, streaming outputs, and live events. Remove unimplemented SDK lifecycle job declarations from the public v1 manifest. Keep roadmap items in design notes, not client input schemas. Move the single workflow-interface operation from /api/workflows/{id}/interface to /api/sdk/v1/workflows/{id}/interface. Give every SDK HTTP failure exactly { "code": string, "message": string, "retryable": boolean }, one declared status mapping, and application/json. Remove the duplicate detail field, legacy method fallbacks, Fastify 404 emulation, legacy RPC error fields, and validation-order exceptions. Make the declaration-driven Fastify plugin call the typed boundary directly. Delete Fetch/Fastify bridges and focused HTTP adapter files that have no remaining caller. Require the SDK implementation boundary at server construction. Remove the fallback WeakMap boundary after all tests inject it explicitly. Keep multipart temporary upload as the only specialized SDK HTTP adapter. It must still call the same typed service and error mapper. Phase 8B [nodetool] - Keep tRPC focused and enforce its contract Keep tRPC for the web, mobile, Electron, and other TypeScript product callers. Do not expose it as the C# SDK wire protocol. Set the server maxBatchSize to the same value as the clients’ httpBatchLink.maxItems limit. Keep POST method override for batched queries and add a test that rejects an oversized batch. Add Zod output schemas to the remaining 33 of 218 tRPC procedures found by the 2026-08-23 AST inventory: 3 in jobs, 14 in models, 2 in triggers, and 14 in worker. Add an inventory test that prevents new unvalidated procedures. Put the web React and vanilla clients behind one shared link factory so authentication, maxItems, POST override, logging, and base URL handling cannot drift. Keep mobile and Electron limits in the same tested policy. Keep plain JSON serialization. Add a transformer only if a product route has a measured need for Date, Map, Set, or another non-JSON value. Keep execution streaming on the existing MessagePack WebSocket. Do not add a second tRPC subscription socket or SSE path for the SDK. Use createCallerFactory only at a transport boundary or in integration tests. Shared server behavior belongs in domain services, not calls from one tRPC procedure to another. Log expected client/authentication errors at an appropriate level and keep unexpected server/output-validation errors at error level with the request ID. Phase 8C [nodetool-sdk] - Make the session the only normal entry point Pin the replacement NodeTool bundle and regenerate internal wire metadata. The update must report every removed command and changed path. Remove SDK MessagePack discovery fixtures, DTO mappings, conformance tests, request correlation, retry paths, and dispatch code. Keep all execution/event MessagePack coverage. Remove IWorkflowDiscoveryClient from INodeToolExecutionClient. Discovery, capabilities, models, downloads, preflight, and asset control use HTTP. Execution and live events use WebSocket. Remove UseWebSocketDiscovery from VL settings, pins, factories, tests, and documentation. Migrate VL discovery to session-owned HTTP services. Inventory every public low-level client method. For each unique useful function, expose a session or domain-service method before hiding the raw client. Do not preserve duplicate transport-specific forms. Add missing session operations for graph execution, node execution, connection status, cancellation, and any internal VL/TestConsole need found by the inventory. Make NodeToolConnectionManager, raw HTTP/WebSocket clients, and session client getters internal. Remove CurrentClient, GetApiClientAsync(), GetConnectedClientAsync(), and NodeToolSessionServices.Api from the public surface after internal callers migrate. Replace the default NodetoolClient constructor, mutable Configure(), NodetoolOptions, and duplicate endpoint/token option paths with one immutable connection profile. Replace source-compatibility default interface methods and duplicate execution overloads with one canonical method per operation. Keep independent WorkflowExecutionOptions values. Update the C#, VL, TestConsole, README, examples, and package API tests to show only the session-first path. Phase 8D [both] - Replace and freeze the baseline Commit the protocol reset separately from mechanical generated-file changes. Include the semantic operation diff and the old/new fixture inventory in the commit or review notes. Replace Phase 0 HTTP and MessagePack goldens once. Delete obsolete discovery RPC fixtures, add the complete execution-wire fixtures, and do not teach tests to accept both contracts. Generate a new deterministic NodeTool bundle, pin its exact digest in the SDK, and prove that a second generation is byte-identical. Run NodeTool HTTP/service/auth/feature-policy tests and the full protocol and websocket package tests. Run C# contract, session, execution, asset, model, VL, package, and lint gates. Prove there is no WebSocket connection during discovery. Run web, mobile, and Electron tRPC tests affected by removed procedures or shared link configuration. Mark the replacement fixtures as the release baseline. From this point, fixture changes require an explicit protocol review again. Verification note (2026-08-23): the SDK Release package gate passed with 222 core tests, 139 VL unit tests, and 5 VL document tests (1 live-server test skipped). The full NodeTool WebSocket run passed 2,315 tests and retained four known Windows/environment baseline failures in local-file access, executable bit detection, and workspace storage setup; no SDK/tRPC-changed test failed. Exit criteria: SDK discovery and control use HTTP only. SDK execution and live events use MessagePack WebSocket only. No SDK-specific tRPC procedure remains. Every execution command and consumed event is declared and byte-frozen. Every retained product function is reachable through the C# session or a session-owned domain service. General TypeScript tRPC callers retain end-to-end types, authentication, batching, and runtime input/output validation. The NodeTool bundle and pinned SDK bundle have the same digest. Phase 9 [both] - Publish the first SDK contract Purpose: release the clean producer before the matching consumer without claiming compatibility with an unreleased baseline. Release A [nodetool] Publish the Phase 8 HTTP/execution-WebSocket contract bundle. Attach the deterministic digest and semantic operation list. Keep the candidate bundle available until the SDK release finishes. Release B [nodetool-sdk] Pin the exact Release A bundle rather than a candidate commit. Publish the session-first C# and VL packages. Document the minimum NodeTool release as Release A. Release verification [both] Install both published artifacts into a clean environment. Run discovery without WebSocket, then run one workflow with streaming, cancellation, reconnect, temporary input, and materialized output. Verify authenticated and reverse-proxy-subpath deployments. Publish a one-row compatibility matrix for the first release. Add older rows only after a second supported release exists. Preparation note (2026-08-23): the locally packed Nodetool.SDK 0.1.6 was restored into a clean temporary .NET 8 project. The project compiled and ran using only the package API, confirmed that raw transport types are not exported, and confirmed that creating session-owned HTTP services does not open a WebSocket. Published-artifact and live-server checks remain open. Exit criteria: The published SDK pins the published NodeTool contract digest. The clean-environment smoke test passes without repository checkouts. Release notes describe the supported HTTP and WebSocket boundary without migration text for unreleased compatibility code. 7. Optional product follow-ups, outside core acceptance These suggestions may use the shared service boundary later, but they must not delay or broaden the core convergence work: [nodetool] let the web app adopt the unified model catalog/download service; [nodetool] let the CLI use temporary multipart upload; [nodetool] expose preflight in more TypeScript product flows; [nodetool] evaluate retirement of /ws/download only in a separate design with real-time behavior, server load, compatibility, and benchmark evidence. These follow-ups may add adapters. They must not create a second business-logic implementation or change SDK v1 behavior without the normal protocol process. 8. Verification matrix 8.1 [nodetool] focused gates packages/protocol generation, schema, digest, and freshness tests; SDK HTTP declaration and Fastify registration inventory tests; SDK execution-WebSocket declaration and publisher/dispatcher inventory tests; HTTP success/error golden tests for every operation; MessagePack execution request, event, reconnect, cancellation, and terminal result tests; workflow-interface HTTP and service parity tests; an inventory test that rejects SDK-specific tRPC procedures and SDK discovery WebSocket commands; tRPC server batch-limit and an output-schema inventory test that reports zero missing procedures; authentication and feature-policy matrix tests; model catalog and model-download lifecycle tests; multipart temporary-upload limits and cleanup tests; reverse-proxy subpath and URL/query encoding tests; affected TypeScript builds and lint. 8.2 [nodetool-sdk] focused gates contract digest and generated-code freshness checks; all HTTP route success/error fixtures; all execution MessagePack request, response, event, and error fixtures; unknown response-field tolerance and strict required-field tests; session ownership, cancellation, retry, request-ID, token refresh, and redaction tests; discovery, interface, capabilities, preflight, models, downloads, and assets; execution negotiation, streaming, cancellation, reconnect, and terminal result tests; all supported .NET targets, NuGet API compatibility, and VL package build. 8.3 [both] release gates live local-server SDK smoke test; authenticated-server smoke test; reverse-proxy subpath smoke test; published producer and matching consumer compatibility row; NodeTool web, Electron, CLI, C# SDK, and VL non-regression checks relevant to the changed services; published artifact digest equals the digest pinned by the SDK update. 9. Pull request sequence Keep each change reviewable, reversible, and usable without an unreleased counterpart. [nodetool] PR 1: Phase 0 inventory, goldens, and drift checks. [nodetool] PR 2: operation registry, implemented/planned profiles, and generated OpenAPI and AsyncAPI inventories. [nodetool] PR 3: deterministic contract bundle and semantic diff. [nodetool] PR 4: service error plus one read-only operation as proof. [nodetool] PR 5: remaining discovery, capability, and preflight operations. [nodetool] PR 6: models, downloads, temporary storage, and one route plugin. [nodetool] PR 7: direct shared-boundary WebSocket calls and temporary compatibility-preserving tRPC wrappers. [nodetool-sdk] PR 1: pin bundle and add conformance tests without production-code changes. [nodetool-sdk] PR 2: generated or mechanically verified internal wire layer behind the existing public API. [both] CI changes: compatibility matrix and review-required contract update automation. [both] benchmark changes: server timing evidence and SDK runner. [nodetool-sdk] PR 3: primary facade documentation and measured presets. [nodetool] PR 8: declare and freeze the actual execution wire, then perform the Phase 8A transport deletion, normalize SDK HTTP routes, and produce the reviewed candidate baseline. [nodetool] PR 9: Phase 8B tRPC batch enforcement, output schemas, and shared client-link policy. [nodetool-sdk] PR 4: pin the replacement bundle, remove WebSocket discovery, migrate VL, and reduce the public surface. [both] release changes: publish NodeTool, pin its released digest, publish C#/VL, and run the clean-environment smoke test. 10. Risks and controls Risk Control The facade becomes a large class Keep domain services separate; the facade coordinates contract operations and shared policy only. The route table becomes coupled to tRPC Store stable operation IDs, never dotted procedure-name strings. A general tRPC call bypasses authentication Keep one protected procedure and test direct and HTTP callers. Generated C# code changes the public API Keep it internal and map through the existing handwritten API. Planned jobs re-enter the public v1 schema Keep roadmap operations outside the implemented manifest. SDK HTTP errors diverge between routes Use one service error and one HTTP error mapper. A flag consolidation removes operational control Preserve both feature flags and the auth override independently. A deleted procedure has a hidden TypeScript caller Repeat the caller audit, remove the procedure in the same PR, and run web/mobile/Electron tests. A raw C# client has a unique useful operation Add the session/domain-service operation before hiding the raw client. Execution frames are missing from the SDK registry Inventory the C# wire first; block adapter deletion and release until declaration and golden inventories match. A client batch limit is bypassed Enforce the same maximum on the tRPC server. New response fields break C# Test additive-field tolerance against every fixture. Strict request changes break protocol v1 Require a protocol version or capability decision. Multipart becomes JSON/base64 Keep a binary-route architecture test and benchmark guard. CI depends on the network Pin verified artifacts locally; fetch only during explicit updates. Contract generation creates noisy diffs Require deterministic output and a semantic review summary. A convenience preset loses durability Preserve defaults, require explicit selection, negotiate support, and document behavior. Web download migration changes real-time behavior Keep it outside core acceptance and require separate measurements. 11. Definition of done One NodeTool service/handler path implements each request/response operation, and one declared publisher path emits each server event. Protocol declarations match the implemented HTTP registration, WebSocket dispatcher, and event-publisher inventories. Every execution command and SDK-consumed response, event, replay, terminal result, and error is declared and has a byte-exact MessagePack golden. SDK HTTP is the only discovery/control transport, and SDK MessagePack WebSocket is the only execution/live-event transport. No SDK-specific tRPC procedure or discovery WebSocket command remains. All SDK HTTP operations use the /api/sdk/v1 prefix and the exact { "code", "message", "retryable" } error shape without detail. Multipart upload, MessagePack execution, streaming, cancellation, and reconnect behavior remain intact. NodeTool publishes one deterministic contract bundle per release. nodetool-sdk pins and verifies that bundle offline. C# wire changes are generated or mechanically verified. The public C# API is session-first and contains no raw transport client getter or mutable connection configuration. Cross-repository CI detects incompatible drift before release. The tested NodeTool/SDK version matrix is published. Every useful low-level function remains available through the session or a session-owned domain service. Any named performance preset is measured, explicit, and overrideable. Every general product tRPC procedure has runtime output validation, and the server enforces the client batch limit. The replacement contract fixtures are frozen after the reviewed reset. Optional product follow-ups are tracked separately from this plan’s core acceptance. 12. Resolved decisions and open measurement Resolved 2026-08-23: There are no supported external consumers of the four SDK-related tRPC procedures. Remove them in Phase 8 after repeating the in-repository audit. Resolved for release: The NodeTool release asset is the authoritative contract bundle. If @nodetool-ai/protocol also carries the staged bundle, it must contain identical bytes and the same digest. Resolved for release: No planned operation appears in the public v1 registry, manifest, schemas, AsyncAPI, or client input. Roadmap operations remain in design notes until implemented. Resolved in Phase 4: Use the focused deterministic C# wire generator and verifier. Do not generate the handwritten public domain API. Open measurement in Phase 6: What measured thresholds justify the name LowLatencyInteractive? Resolved for the first release: Move the single-workflow interface under /api/sdk/v1; there is no compatibility alias. Replacing /ws/download remains a separate product plan. Until the measurement is resolved, keep individual execution options and do not publish a named preset. In all cases, pin generated artifacts, keep tRPC details out of the public C# contract, and do not retain an unreleased duplicate transport only because it exists in the Phase 0 baseline.--- # Security Hardening Source: https://docs.nodetool.ai/security-hardening.html Use this page as a checklist before exposing NodeTool beyond your laptop. It covers security practices for each deployment stage and links to canonical guides for configuration, authentication, and proxy settings. Universal Checklist These apply to every deployment, regardless of environment: Network Security Require TLS at the proxy or ingress layer. Use real certificates (e.g., Let’s Encrypt) and redirect HTTP to HTTPS. Restrict Docker access: Run the proxy on a dedicated Docker network and avoid exposing the Docker socket beyond the host. Firewall rules: Only expose the ports you need (typically 443 for HTTPS). Block direct access to internal service ports. Authentication ⚠️ NODETOOL_TRUST_LOCAL_NETWORKS bypasses login entirely. Every source IP in that list is trusted as admin user "1" with no password — full access to data, secrets, and API keys. The Docker Compose file trusts the Docker bridge (172.16.0.0/12) and Docker Desktop’s VM gateway subnet (192.168.65.0/24) so a single-user local install works out of the box. Never set it to 0.0.0.0/0 on a public IP, and never widen it as a substitute for real auth — enable Supabase mode instead. See Authentication → Local mode in Docker. Enforce auth on any network-accessible deployment. The server enforces authentication only when it is in Supabase mode — that is, when both SUPABASE_URL and SUPABASE_KEY are set. Without them it runs in Local mode and trusts loopback connections (and any NODETOOL_TRUST_LOCAL_NETWORKS you configure). See Authentication. Lock down localhost trust. Behind a reverse proxy or in a container, set NODETOOL_TRUST_LOCALHOST=false (it already defaults off when auth is enforced) and list only your real proxies in NODETOOL_TRUSTED_PROXIES, so a proxy connecting from loopback cannot silently bypass auth. Keep NODETOOL_TRUST_LOCAL_NETWORKS tight. Trust the smallest range that works (the Docker bridge, not 0.0.0.0/0), and firewall the published port. It is ignored in Supabase mode. Rotate keys regularly: set calendar reminders to rotate Supabase service-role keys. Know what the server ignores. SERVER_AUTH_TOKEN and ~/.config/nodetool/deployment.yaml are read only by the @nodetool-ai/deploy helpers. The running server does not check them, so they do not protect the API. See Authentication. Use access tokens for agents. A signed-in person can mint a revocable ntk_ token for an MCP client or script. It works in every auth mode, so there is no need to widen NODETOOL_TRUST_LOCAL_NETWORKS to reach /mcp. Keep /mcp off unless needed. With NODETOOL_ENV=production the /mcp mount is not registered unless NODETOOL_ENABLE_MCP=1. It carries the full agent toolbelt, so enable it only when clients authenticate. See MCP production. Keep rate limits on. The per-IP HTTP limiter defaults to 1000 requests per 60 seconds and exempts loopback. Behind a proxy set NODETOOL_TRUSTED_PROXIES and NODETOOL_RATE_LIMIT_TRUST_PROXY so clients are keyed by their real address. See Authentication. Secrets Management Keep secrets out of Git: Load provider API keys and tokens from environment variables or a secrets manager. Never commit .env files with secrets. Add .env to .gitignore. Provide the encryption master key (SECRETS_MASTER_KEY) on every server so they share one key, and store all other deployment secrets in your platform’s secrets manager. Rotating the master key also invalidates every delegated (messaging bridge) token and deployed-app session token, which are signed with keys derived from it. Set NODETOOL_INTEGRATION_TOKEN to a random value of 16 or more characters only if you use messaging integrations. Shorter values are ignored and the /api/integrations routes are not registered. Asset Storage Asset objects live at <userId>/<assetId>.<ext>. The owner is the leading path segment, so the tenant boundary is enforceable by the storage backend itself, not just by the API: Run nodetool storage migrate-keys when upgrading a deployment that predates this layout. The local file backend falls back to the old flat key on a miss, but Supabase and S3 resolve through signed URLs and need the objects actually moved. --dry-run reports what would move. Keep the bucket private. Reads go through short-lived signed URLs minted per request; a public bucket bypasses that entirely. Add a storage policy keyed on the prefix on Supabase (an RLS policy on storage.objects matching (storage.foldername(name))[1] = auth.uid()::text) or S3 (a bucket policy on the key prefix). The API already enforces this, but a policy makes it hold even if a signed URL leaks or a handler regresses. Never ship SUPABASE_KEY to a client. The service-role key stays server-side; browsers upload through the one-shot, key-scoped targets that assets.createUpload mints, which authorise a write to exactly one object. Local File Access The file browser (tRPC files.list) and the preview stream (GET /api/files/local) read the server’s own filesystem. Both are off when NODETOOL_ENV=production, and outside production both resolve paths against an allowlist of roots — the user’s home directory unless NODETOOL_LOCAL_FILE_ROOTS overrides it (platform-delimited, like PATH). Sensitive entries under home (.ssh, .aws, .gnupg, .kube, .docker, .config/gcloud, .netrc, .pgpass, .npmrc) are refused inside a root, and symlinks that leave the roots are refused too. Keep the roots narrow. Widen NODETOOL_LOCAL_FILE_ROOTS only to the directories that actually hold media you preview — an external volume, a scratch dir — never to /. The single entry * lifts the containment check (the denylist still applies); it belongs to the desktop app, which runs the server as one user’s own process on loopback and lets them drag a file onto the canvas from anywhere. Electron sets it, and a launching environment that configured its own roots keeps them. Set NODETOOL_ENV=production on any deployment more than one person can reach, which disables both surfaces outright. Resource Limits Set container resource limits: Use mem_limit and cpus to prevent runaway workloads. Use read-only mounts where possible to limit filesystem write access. See Docker Resource Management for configuration details. Development (Local) Local development has the lowest risk but still deserves basic hygiene: Action Details Bind to localhost only Use 127.0.0.1 for all services; avoid publishing container ports to the LAN Use temporary tokens Mint throwaway ntk_ access tokens for demos and revoke them when finished Don’t expose to the internet Never use ngrok or similar tunneling without authentication in place Staging Staging environments should mirror production security with some allowances for debugging: Action Details Gate access Use VPN or IP allowlists; do not rely on obscurity Separate credentials Use distinct Supabase projects and tokens from production Rotate on personnel changes Rotate service-role keys when team members leave Enable end-to-end TLS Including internal hops if traversing untrusted networks Regular backups Back up workspace volumes and databases on a schedule; restrict who can read backups Audit logging Enable request logging at the proxy layer to track access patterns Production Production deployments require the strictest security posture: Authentication & Authorization Enable Supabase mode (set both SUPABASE_URL and SUPABASE_KEY) for any multi-user or network-accessible deployment so every request is authenticated Keep NODETOOL_TRUST_LOCALHOST off and restrict NODETOOL_TRUSTED_PROXIES to your real proxy addresses Unset NODETOOL_TRUST_LOCAL_NETWORKS (or leave it ignored under Supabase mode) — never carry a 0.0.0.0/0 trust rule into production Use dedicated service accounts for each deployment; avoid shared credentials Distribute bearer tokens and keys through your secrets manager, not through files in the image or repository Public app links are a production-only surface (NODETOOL_ENV=production). An owner can deploy a published mini app to /a/<token>, where anyone with the link runs it without logging in — on the owner’s provider keys, against the app’s spend budget. Set a budget on every deployed app (Governance › Spend budget), and treat the link as the credential it is: withdrawing it is the only way to invalidate it, and outstanding run sessions expire after one hour rather than immediately Infrastructure Set per-container resource caps to prevent runaway workloads on multi-tenant hosts Use separate networks for the proxy, API server, and worker containers Pin container image versions (avoid latest tags in production) Monitoring & Alerting Centralize logging – Forward container logs to your logging platform (ELK, CloudWatch, etc.) Monitor for anomalies: Auth failures, 429 rate limits, container restarts, unusual traffic spikes Set up alerts for failed authentication attempts and service health degradation Track resource usage to detect potential abuse or misconfiguration Image Management Rebuild regularly to pick up base image security patches Track CVEs in your base images using tools like trivy or grype Prune unused images from registries and hosts to reduce attack surface Sign images if your workflow supports it, to ensure integrity Data Protection Encrypt data at rest – Use encrypted volumes for workspace and database storage Encrypt data in transit – TLS everywhere, including internal service communication Minimize data retention – Clear temporary assets on a schedule. The temp bucket is set with TEMP_BUCKET, separate from ASSET_BUCKET, so it can have its own retention policy Backup strategy – Regular automated backups with tested restore procedures Quick Security Audit Run through this checklist before any deployment goes live: TLS enabled with valid certificates Supabase mode enabled (SUPABASE_URL + SUPABASE_KEY set) for any network-accessible deployment NODETOOL_TRUST_LOCAL_NETWORKS unset or tightly scoped — no 0.0.0.0/0 on a network-accessible deployment NODETOOL_TRUST_LOCALHOST off and NODETOOL_TRUSTED_PROXIES scoped to real proxies All API keys loaded from env vars or secrets manager (not hardcoded) Container resource limits configured Docker socket not exposed to containers Firewall rules restrict access to necessary ports only Logging enabled and forwarded to monitoring platform Backup schedule configured and tested Service tokens rotated within the last 90 days Base images updated within the last 30 days Related Authentication – Auth provider configuration Configuration Guide – Environment variables and settings Deployment Guide – Deployment overview and workflows Self-Hosted Deployment – Self-hosted setup details Docker Resource Management – Container resource limits Storage – Storage backend configuration--- # Self-Hosted Deployment Guide Source: https://docs.nodetool.ai/self-hosted-deployment.html This guide covers deploying NodeTool on your own infrastructure. Overview Self-hosted deployment runs NodeTool in a Docker container — on localhost or on a remote host reached over SSH. Docker is the only supported deployment type (SUPPORTED_TYPES = ["docker"]). Two paths: Docker Compose — a single docker-compose.yml you run yourself. The fastest way to stand up one server. See below. nodetool deploy CLI — a managed flow driven by deployment.yaml that also handles remote hosts over SSH, image transfer, and workflow sync. See Deployment Configuration. Docker Compose (reference) The repository ships a reference docker-compose.yml for running one server on a host you control. cp .env.example .env # fill in the provider keys you use docker compose up -d # open http://localhost:17777 The server binds to 0.0.0.0:7777 inside the container and is published on the host as ${NODETOOL_PORT:-17777}. All persistent state — SQLite database, assets, vector store, model cache, and the generated secret key — lives under /workspace, backed by the named nodetool-data volume, so it survives restarts and image upgrades. Common overrides (set in .env or the shell): Variable Default Purpose NODETOOL_IMAGE ghcr.io/nodetool-ai/nodetool:${NODETOOL_VERSION} Full image reference. Set it to run a locally built image (for example nodetool:dev) NODETOOL_VERSION latest Image tag to pull (pin a release in production) NODETOOL_PORT 17777 Host port mapped to the container’s 7777 NODETOOL_TRUST_LOCAL_NETWORKS 172.16.0.0/12,192.168.65.0/24 ⚠️ Source CIDRs trusted as user 1 without a login (Local mode). Docker’s Linux bridge plus Docker Desktop’s VM gateway subnet by default — never 0.0.0.0/0 on a public IP. Ignored in Supabase mode SECRETS_MASTER_KEY generated on first start 32-byte base64 key encrypting stored secrets. If unset, the image entrypoint generates one and stores it in /workspace/.secrets_master_key (override the path with SECRETS_MASTER_KEY_FILE). The line is commented out in the compose file, so uncomment it to pass your own (openssl rand -base64 32) OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, FAL_API_KEY, HF_TOKEN unset Model provider keys The compose file also sets NODETOOL_ENV=production and NODETOOL_NODE_PROFILE=full. Production defaults to the curated cloud node catalog with no local model runtimes, and full restores the whole catalog, including the llama.cpp, vLLM, and Ollama providers. The database is SQLite at /workspace/nodetool.sqlite3 (DB_PATH). The entrypoint refuses to start when neither DB_PATH nor DATABASE_URL is set. The compose file has commented-out blocks you can enable: Ollama: set OLLAMA_API_URL (http://host.docker.internal:11434 for Ollama on the host, with the extra_hosts entry uncommented). llama.cpp and vLLM services: uncomment the service and set LLAMA_CPP_URL or VLLM_BASE_URL. vLLM needs an NVIDIA GPU on Linux. PostgreSQL: uncomment the postgres service, remove DB_PATH, and set DATABASE_URL. With DATABASE_URL set, the entrypoint runs db-migrate.mjs on every start unless NODETOOL_MIGRATE_ON_BOOT=0. Upgrade in place: docker compose pull docker compose up -d To store data in a host directory instead of the named volume, replace the nodetool-data:/workspace mount with a bind mount (e.g. ./nodetool-data:/workspace) and make sure the host directory is writable by the container’s node user. Authentication / login screen Auth is configured entirely on the backend. The web UI fetches its auth mode and public Supabase credentials at runtime from GET /api/config (a public, non-secret endpoint), so the same frontend build works with or without login — no rebuild, and it works even when the frontend is served from a different origin. Login off (default). With SUPABASE_URL/SUPABASE_KEY unset the server runs in Local mode: it trusts requests by source IP (loopback, plus any NODETOOL_TRUST_LOCAL_NETWORKS you set) and runs them as a single user; other requests are rejected, and the UI shows no login screen. In Docker the bundled Compose file trusts the Docker bridge (172.16.0.0/12) and Docker Desktop’s VM gateway subnet (192.168.65.0/24) so a local install works out of the box — see the warning below. Login on. Set these three on the server to switch to Supabase mode — the server requires a valid Supabase JWT on every request and the UI shows the login screen: SUPABASE_URL=https://your-project.supabase.co SUPABASE_KEY=your-service-role-key # server-only, never sent to the browser SUPABASE_ANON_KEY=your-anon-key # public key the login screen uses Optionally set AUTH_REDIRECT_URL when serving behind a domain/proxy (it must be in the Supabase project’s redirect allow list). GET /api/config returns authMode, supabaseUrl, supabaseAnonKey, authRedirectUrl, and version — never the service-role key. See Authentication and Supabase Deployment. 🔒 Do not expose Local mode to the internet Local mode has no login. NODETOOL_TRUST_LOCAL_NETWORKS trusts a set of source IPs as admin user "1" with no password — full access to your data, secrets, and API keys. Anyone who can reach the published port from a trusted range gets in. Safe on a laptop or a private LAN/VPN behind a firewall. The default 172.16.0.0/12,192.168.65.0/24 trusts only Docker’s bridge and Docker Desktop’s VM gateway, not the wider internet. Never change it to 0.0.0.0/0 on a public IP. Putting NodeTool on a public address or sharing it with untrusted users? Enable Supabase mode (above) so every request needs a real login, and terminate TLS in front of the server or give it a certificate (Serving the Web UI and TLS). Serving the web UI from a different origin (e.g. a CDN)? Point the frontend at the backend with the build-time VITE_API_URL, and add that origin to the server’s NODETOOL_ALLOWED_ORIGINS so the cross-origin GET /api/config and API calls are permitted. Everything else still comes from /api/config. Deployment Configuration Deployments are configured via deployment.yaml. Docker Deployment deployments: my-server: type: docker enabled: true host: 192.168.1.10 ssh: user: ubuntu key_path: ~/.ssh/id_rsa container: name: nodetool-server port: 8000 gpu: "0" paths: workspace: /data/nodetool hf_cache: /data/hf-cache image: name: ghcr.io/nodetool-ai/nodetool tag: latest For a local host, set host: localhost and omit the ssh block. Apply Flow Directory Creation: Creates workspace (with data, assets, temp, proxy, and acme subdirectories) and hf_cache on the host. Image Check: Verifies the configured image exists on the host. deploy apply does not pull. A local host without the image fails with an error, so pull or build it first. Image Transfer: For a remote host that lacks the image, pipes docker save from your local daemon into docker load over SSH. An image the remote already holds is never replaced. Container Management: Stops and removes the existing container, and any other nodetool-* container publishing the same host port, then runs a new one named nodetool-<container.name>. Docker is used when installed, otherwise Podman. Set NODETOOL_CONTAINER_RUNTIME=docker or podman to choose. Health Check: Polls http://127.0.0.1:<host port>/health on the host, 10 attempts 2 seconds apart, and fails the apply if the server never answers. container.port is the host port. The container serves on 7777, and a container.port of 7777 is published as 8000. The container is started with --restart unless-stopped and a Docker health check on /health. End-to-End: Local Docker Deployment This walkthrough matches a common local setup flow: Pull the Docker image. Add a docker deployment interactively. Review the generated deployment. Apply deployment and validate health. Sync workflows. Run a synced workflow on the deployed instance. 0. Pull the Image First docker pull ghcr.io/nodetool-ai/nodetool:latest 1. Add Local Docker Deployment nodetool deploy add local --type docker --type docker is required. The command then prompts for the rest: Docker host (IP or hostname): localhost SSH user and SSH key path: asked only for a remote host (defaults root and ~/.ssh/id_rsa) Docker image name: ghcr.io/nodetool-ai/nodetool Image tag: latest Container name: nodetool-<deployment name> Container port: 8000 The command does not prompt for GPU or paths. It sets paths.workspace to ~/.nodetool-workspace and paths.hf_cache to the Hugging Face cache it finds (HF_HUB_CACHE, then $HF_HOME/hub, then ~/.cache/huggingface/hub). Edit container.gpu and paths with nodetool deploy edit. Path meanings: Workspace: mounted at /workspace. Holds the database, assets, and temporary runtime data. HF cache: mounted at /hf-cache, read-only. Holds downloaded Hugging Face models. 2. Review Deployment Config nodetool deploy show local This dumps the deployment entry as YAML. It also lists enabled, state, server_auth_token, and container.environment (with a generated SECRETS_MASTER_KEY). You should see something like: local: type: docker host: localhost image: name: ghcr.io/nodetool-ai/nodetool tag: latest container: name: nodetool-local port: 8000 paths: workspace: <your workspace path> hf_cache: <your HF cache path> The server endpoint is at http://localhost:8000. The container EXPOSEs 7777; when container.port is 7777 the deployer maps it to host port 8000 (any other container.port is used as-is). You can also inspect the raw config: cat ~/.config/nodetool/deployment.yaml To change a field deploy add did not prompt for, open the file in $EDITOR: nodetool deploy edit deploy edit takes no deployment name — it opens the whole deployment.yaml. Without one on disk it stops and tells you to run nodetool deploy init first. 3. Apply Deployment nodetool deploy apply local Apply prints each step, then the result as JSON. A successful run shows: directories created Image already present. app container started Health endpoint OK: http://127.0.0.1:8000/health "status": "success" in the final JSON If apply fails (for example because the image is missing or the health check times out), fix the cause and run it again: nodetool deploy apply local Then confirm runtime state: {% raw %} nodetool deploy status local docker ps --filter name=nodetool --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" curl http://127.0.0.1:8000/health {% endraw %} 4. Sync Workflows to the Deployment List local workflows first: nodetool workflows list Sync one workflow by ID to the deployed instance: nodetool deploy workflows sync local <workflow_id> Sync pushes the workflow definition (name, description, access, graph, and settings) to PUT /api/workflows/<id> on the deployment. It does not upload assets or download models. Add assets and models on the target yourself. All deploy workflows subcommands (sync, list, delete, run) need an admin bearer token, resolved like the API users commands below. delete prompts first unless you pass --force. Verify remote workflows: nodetool deploy workflows list local 5. Test Workflow Execution on Deployed Instance Run a synced workflow remotely: nodetool deploy workflows run local <workflow_id> Pass parameters with a repeatable -p key=value: nodetool deploy workflows run local <workflow_id> -p prompt="hello" If a run fails, inspect logs: nodetool deploy logs local --tail 200 API Users A deployment that serves more than one person needs API users. Four nodetool deploy subcommands manage them over the running server’s admin API, so the deployment must already be applied and reachable. The caller must be an admin: user 1, an id listed in ADMIN_USER_IDS, or an API user created with --role admin. Records live in the file named by USERS_FILE, which the image sets to /workspace/users.json so they survive a redeploy. Every one of them needs an admin bearer token. It comes from --token, else from NODETOOL_ADMIN_TOKEN, else from an interactive prompt — with no TTY and neither set, the command exits with Admin token required. Provide --token or set NODETOOL_ADMIN_TOKEN. The deployment’s server_auth_token (in deployment.yaml) is that admin token. apply passes it to the container as SERVER_AUTH_TOKEN, and a Local-mode server accepts it as Authorization: Bearer <token> for user 1. A Supabase-mode server ignores it. A Local-mode server accepts each token users-add prints as Authorization: Bearer <token> for that user’s own id, so each API user has separate workflows and assets. API users exist only in Local mode: a Supabase-mode server refuses users-add and users-reset-token, because the token would never authenticate there. Supabase users sign in, or mint an access token in the app. users-add <deployment> <username> — create a user and print its token. --role <admin|user> defaults to user; any other value is refused. The token is printed once and never again. users-list <deployment> — username, user id, role, a 16-character preview of the stored token hash, and creation time. --json prints the raw records. users-remove <deployment> <username> — delete a user. Prompts first; --force skips the prompt. users-reset-token <deployment> <username> — issue a new token and invalidate the old one. Like users-add, the new token is shown once. Examples: export NODETOOL_ADMIN_TOKEN=<admin token> nodetool deploy users-add local alice --role admin nodetool deploy users-list local nodetool deploy users-list local --json nodetool deploy users-reset-token local alice nodetool deploy users-remove local alice --force MCP over HTTP and Python nodes Two surfaces are off when NODETOOL_ENV=production — which the published image, fly.toml, and docker-compose.yml all set. Each has an opt-in flag. /mcp — NODETOOL_ENABLE_MCP=1 The streamable-HTTP MCP mount carries the full agent toolbelt, so it registers in production only with the flag: environment: NODETOOL_ENABLE_MCP: "1" Without it the route is not registered, /mcp answers 404, and the boot log names the flag. The mount is not a second door. It sits behind the same onRequest auth hook as /api, and binds the user that hook resolved: a request it cannot authenticate is refused at initialize with 401, never given an anonymous session. The session id in the mcp-session-id header is scoped to its owner — a second user presenting the same id gets 404 Session not found, the same answer as an id that never existed. How an agent authenticates: A token minted in the app — the short path, and the one to reach for first. Settings → MCP → Connect an agent remotely hands back a ready-to-paste claude mcp add command carrying the URL and a bearer token. The token is revocable one at a time, stored only as a hash, and works in every auth mode. Supabase mode (SUPABASE_URL + SUPABASE_KEY): the agent sends a Supabase access token as Authorization: Bearer <jwt>, like any web client. Fine for a script that already signs in; the JWT expires within the hour, so it is a poor fit for a config file. Local mode behind a VPN: set NODETOOL_TRUST_LOCAL_NETWORKS to the VPN’s CIDR. Requests from that range are trusted as user 1 with no token. Scope it to the VPN — anything reaching the server from a trusted range gets the whole toolbelt, and there is nothing to revoke afterwards. A bot or bridge: mint a delegated token through the integration routes (NODETOOL_INTEGRATION_TOKEN) and send it as the bearer token. The session binds the linked account rather than a shared identity. Full walkthrough, including how to check a setup and what each error answer means: MCP on a production server. Not covered by the flag: the tRPC mcpConfig router stays disabled in production. It edits MCP client config files on the server’s filesystem, which has no meaning on a shared host. Python nodes — NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 The Python bridge refuses to connect in production. The flag lifts that: environment: NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION: "1" NODETOOL_PYTHON: "/opt/venv/bin/python" The published image ships no Python worker. ghcr.io/nodetool-ai/nodetool carries the TypeScript server only, so the flag by itself changes nothing except the error you get. To run Python nodes, derive an image that installs nodetool-core and set NODETOOL_PYTHON to that interpreter: FROM ghcr.io/nodetool-ai/nodetool:latest # The image runs as `node`, and /opt/venv belongs to root. USER root RUN /opt/venv/bin/pip install --no-cache-dir nodetool-core USER node ENV NODETOOL_PYTHON=/opt/venv/bin/python \ NODETOOL_ALLOW_PYTHON_BRIDGE_IN_PRODUCTION=1 With the flag set and no interpreter found, the boot log says Python not found — Python nodes will not be available. With the flag unset it names the flag instead, so the log tells you which of the two you are missing. Manual Troubleshooting Container Logs nodetool deploy logs local --tail 200 For a remote host you can also read the container logs directly: ssh user@host "docker logs nodetool-<container.name>"--- # Programmatic SEO Playbook Source: https://docs.nodetool.ai/seo-programmatic Companion to SEO Strategy. That document is the audit and roadmap; this one is the build plan for programmatic pages. Source: a teardown of photoai.com (9,466 indexed URLs, ~78% generated from “user activity”, the rest templated per keyword). Written 2026-07-02, revised the same day to add the seeded-showcase engine, model pages, and the ranked list of further dimensions. File paths and counts below drift — re-verify before building. The thesis, adapted Photo AI treats pages as a product output, not a content cost: adding a keyword or generating an image both create a fully-optimized, schema-rich, interlinked page automatically. The thing to copy is the machine, not the pages. Three facts shape NodeTool’s version: Photo AI’s “UGC” pages did not wait for users. The founder seeded /photos/* with his own generations; user volume came later. NodeTool can do the same: batch-generate real outputs from its own workflows and publish each run as a page. The outputs are genuine (real model, real prompt, real workflow) — what’s seeded is who pressed the button. The line not to cross is fabricated attribution: no invented user names, no synthetic Review/aggregateRating markup. Google’s spam policies name fake review markup specifically; the generations themselves are fine. Search Console already shows what thin programmatic pages earn here. The 375 node-reference URLs on docs.nodetool.ai took 7.8k impressions for 12 clicks (SEO Strategy §0.4) because the queries were “official docs” lookups for third-party libraries. Page count without intent match is a proven zero on this domain. Every engine below has an index gate for this reason. NodeTool’s unique assets are workflows and multi-provider model access. A workflow has structure an image doesn’t: a rendered graph, a node list, typed inputs/outputs, a category. And the node packages expose the models people actually search for — Seedance, Veo, Kling, Sora, Hailuo, Wan, Seedream, FLUX, GPT-Image — each through several providers (Seedance alone via fal, kie, replicate, and atlascloud nodes). Same prompt, two models, side by side, with real outputs: content no single-model vendor can publish and no competitor bothers to. What survives the adaptation intact: fixed template per page type, output-first page layout, the dense internal-link mesh, per-page-type JSON-LD, one derived sitemap, and the principle that adding one row of data ships one complete page. Engine 1 — Template pages (/templates/*) 37 example workflows ship in packages/base-nodes/nodetool/examples/nodetool-base/*.json, each with name, description, tags, and the full graph. Three have marketing pages (/use-cases/*); 34 have none. Every one becomes a page at /templates/<slug>. Page anatomy, in render order: Sample output first. The generated video/image/audio leads the page (see Engine 2 for supply); the graph explains it below. Photo AI’s photo pages lead with the photo for the same reason: the output is what the searcher wants to judge. Slug and H1 from the workflow name (Movie Posters → /templates/movie-posters, H1 “Movie Poster Generator — free AI workflow template”). Meta description from the workflow’s description, extended with node count and category. Rendered graph. The static flow components in marketing/src/components/flow/ (NodeCanvas, FlowNode, FlowEdge, proven at marketing/src/app/flow-preview/page.tsx) render the actual graph server-side from the JSON’s ui_properties. No sibling page and no competitor has this DOM. “Nodes in this workflow”: one line per distinct node type — what it does, which provider runs it — with links into the model pages (Engine 3). How to run it: open in NodeTool Cloud CTA plus the download path. Related templates: 8–9 links by shared tags/category — Photo AI’s mechanism for circulating equity and crawl depth through the cluster. JSON-LD: HowTo + ItemList via the existing JsonLd.tsx. Build mechanics: a build-time script reads the example JSONs into a generated templateEntries.ts (same shape-discipline as use-cases/useCaseEntries.ts), one dynamic route app/templates/[slug]/page.tsx with generateStaticParams, and a /templates hub page in the footer. New example workflow in the repo → new page on deploy. That is the machine. Index gate: non-boilerplate description, graph renders, and a sample output. Below the bar, the page builds with noindex until the gap is filled. Engine 2 — Seeded showcase (/showcase/*): the UGC engine, without waiting for users The direct clone of /photos/* (7,428 pages, the biggest lever in the teardown), seeded the way Photo AI seeded it: generate the content ourselves. Every batch run of a workflow becomes a page. The pipeline. The CLI already does the hard part: nodetool workflows run <file> --params and nodetool generate <provider> <model> <prompt> produce outputs headlessly, and the debug harness captures per-run metadata. A seeding script iterates templates × models × a curated prompt list, writes each output plus a manifest (prompt, model, provider, workflow slug, duration, params) to the asset store, and emits one showcaseEntries row per run. Marginal cost per page is inference, not writing. Page anatomy (mirrors Photo AI’s /photos/* mechanics precisely): URL slug is the prompt, truncated and hyphenated, with a short id appended: /showcase/neon-jellyfish-drifting-through-a-flooded-subway-station-a7f3k2. The output is the hero — full-width video or image, playing on load. Title and meta description are the prompt text; H1 is the prompt plus the model name. Below the output: model chip (→ its Engine 3 page), workflow chip (→ its Engine 1 page), provider, generation params. This block is what makes the page more than a bare asset. ~9 related showcase links (shared model, shared workflow, shared tag) — the equity mesh. “Remix this” CTA: opens the workflow in NodeTool Cloud with the prompt pre-filled. JSON-LD: ImageObject/VideoObject + ItemList, matching Photo AI’s photo-page stack. Volume math: 37 templates × 3–4 models × 5–10 prompts is 500–1,500 pages from one weekend of compute, before any real user publishes anything. Start smaller: ~200 pages across the video and image templates, watch indexation for 4–6 weeks, then scale. Provenance rules (the adaptation of “fake UGC” that doesn’t blow up): outputs must be real generations from the named model and workflow — never stock, never misattributed to another model. Pages carry no user identity and no review markup; they are a showcase, not testimonials. Dedupe near-identical prompts before publishing (same model + cosine-similar prompt → one page, others canonical to it). Engine 2b — real publish loop, later. Once Cloud has a Publish action, user-published workflows and runs enter the same /showcase/* and /w/* templates behind the same index gate (description ≥ 140 chars, ≥ 5 nodes or a real output, no near-duplicate already indexed, noindex from birth until the gate passes). The seeded corpus is what makes the section already rank by the time real UGC arrives. Engine 3 — Model pages (/models/*) People search the model, not the tool: “seedance”, “veo 3”, “kling ai”, “sora 2”, “gpt image”, “seedream”, “wan 2.5”, “flux”. Every one of these is runnable in NodeTool today through multiple providers (verified in packages/*-nodes/src: Seedance via fal/kie/replicate/atlascloud, Veo via fal/kie/replicate/together, and so on). The honest angle writes itself: one canvas runs them all, through whichever provider is cheapest, on your own API keys. /models/<slug> (“Seedance”, “Veo 3”, “Kling 2.6”, …): Hero: 2–3 real outputs from Engine 2 (the showcase supplies every model page’s proof). What the model does, in NodeTool’s words — capability, resolution/duration limits, what it’s best at. This slice is hand-written per model; it’s the intent-match that node-docs pages lacked. Provider table: which node packages expose it, BYOK pricing note per provider. This is the differentiator no single-provider page has, and it’s generated from the package manifests. A template that uses the model (→ Engine 1), embedded graph and all. Prompt examples: 5–10 showcase items for this model (→ Engine 2), each linked. FAQ: “is Seedance free”, “Seedance vs Kling, which is better” (→ Engine 3b), “can I run it locally”. FAQPage markup. Title: “Seedance in NodeTool — run it in a visual AI workflow (2026)”. Engine 3b — model-vs-model comparisons (/models/veo-3-vs-kling-2-6 or similar). The highest-upside new dimension found in this revision. “veo vs kling”, “sora vs veo 3”, “seedance vs hailuo” are real queries with commercial intent, the SERPs are Reddit threads and listicles, and NodeTool can do the one thing they can’t: run the identical prompt through both models and embed both outputs side by side, generated by the same seeding pipeline as Engine 2. Feature table (resolution, duration, audio, price per clip via each provider), 3–4 same-prompt pairs, an honest “pick X when…” verdict, FAQ, and a CTA into the workflow that produced the comparison — which the reader can run with their own prompt. That last link is a page mechanic no static listicle can copy. Pairs are data rows too: ~10 video models → ship the ~15 pairs people actually search, not all 45. Freshness dimension: model releases are page events. When fal/kie add a new model (they ship within days of release), a NodeTool model page + a same-day showcase batch can rank on “how to try " while the query spikes and before the listicles consolidate. Keep a standing runbook: new model in a node package → entry in `modelEntries.ts` → seed 10 showcase runs → page live. Refresh version numbers in titles on each release (the Photo AI year-in-title trick, applied to model versions). Engine 4 — Keyword landing pages One master template, a swappable hero + proof slice, a shared body. /use-cases/* and useCaseEntries.ts are the half-built version at 3 pages. Keyword axes, each a data array feeding the same template: Task keywords from shipped templates: “AI product video generator”, “AI movie poster maker”, “chat with PDF open source”. 34 unshipped templates ≈ 34 pages. Task hubs (“image to video”, “lip sync AI”, “AI video upscaler”, “text to music”): one hub per capability listing the models that do it (→ Engine 3), the templates that wire it (→ Engine 1), and showcase output (→ Engine 2). These sit one level above model pages and catch the searcher who hasn’t picked a model yet. Persona keywords: four exist (/creatives, /agents, /developers, /marketing); add local-first/privacy and researcher/knowledge-worker (SEO Strategy §3 names both as uncovered). Trend/aesthetic keywords (Photo AI’s /old-money): weakest fit; skip until the axes above are exhausted. Do not copy the 15,000-word / 59-question body: Photo AI carries that on domain authority NodeTool doesn’t have. Target ~2,500 words with the keyword-specific slice ≥ 30%. Title formula: front-load the keyword, add the concrete differentiator, short suffix — “AI Product Video Generator — Free, Open Source, Your API Keys | NodeTool”. Numbers where honest; skip the emoji. Engine 5 — Product comparison and alternatives mesh Six /vs/* pages shipped 2026-07-01/02 with the honest-concession pattern (SEO Strategy §4.1). Remaining work, per the teardown: /alternatives/<competitor> twins — “comfyui alternative” and “nodetool vs comfyui” are different queries with different SERPs. One template: the competitor’s real limitation, a 4–6 tool list (NodeTool first, honestly framed), feature table, FAQ, CTA. Year in the title, computed at build: “NodeTool vs Langflow: Which Is Better? (2026)”. In-content cross-link block (“Compare NodeTool with other tools”) on every page, beyond the existing footer column — this is what makes the cluster rank as a unit. Wide net (~20 competitors × 2 page types): the shipped six plus Flora, Krea, Freepik Spaces, Griptape Nodes, Reflet.ai (creative canvases); Lindy, Gumloop, Activepieces, Zapier AI, Make AI (automation); LM Studio, Jan, Open WebUI (local-first). Factory-ize the hand-built ~250-line pages before scaling past ten. JSON-LD: BreadcrumbList + FAQPage (Photo AI deliberately uses a different stack here than on landing pages). Engine 6 — FAQ, ideas, and glossary hubs /faq/*: seed from questions with evidence — the GSC export’s conversational queries (“which is free”, “are any of these open source?”), docs/troubleshooting.md, provider-setup friction, and the FAQ blocks already written for /vs/*. Each question is a standalone page plus a row in the shared FAQ module that Engines 3–5 render, so every answer exists twice: standalone for question SERPs, embedded with FAQPage markup for snippets and AI-answer surfaces (§0.8 shows AI assistants are a live acquisition channel). /ideas/*: category roundups generated from Engine 1 metadata (“12 AI video workflow ideas”), each linking every template in its category. New template pages get their first internal links here on day one. Glossary: docs/glossary.md already exists; per-term pages (“what is image-to-video”, “what is a node-based editor”) are near-free definitional surface that AI assistants quote. Other dimensions considered, ranked Evaluated for this revision; the first two graduated into engines above, the rest are queued: Model-vs-model comparisons → Engine 3b. Highest upside: real query volume, weak SERPs, and a same-prompt side-by-side only NodeTool can generate. Task hubs → Engine 4 axis 2. “Ollama frontend” positioning page. “ollama gui”, “ollama frontend”, “ollama web ui” carry real volume; NodeTool genuinely is one (Ollama provider in packages/runtime). One hand-built page (/ollama), same honest-table pattern as /vs/*, linking the local-first persona page. Quick win, do alongside Engine 5. Prompt-library pages (“seedance prompts”, “veo 3 prompt examples”): curated collections of Engine 2 showcase items per model. Ship as /models/<slug>/prompts once a model has 20+ showcase items — a view over existing data, not a new engine. New-model release notes (“Wan 2.6 is out — how to try it”): blog posts tied to the Engine 3 freshness runbook. Needs the blog to exist (SEO Strategy §4.4). Integration pages (/integrations/supabase, Google Sheets, the Chrome extension): integration-nodes exists, but query volume for “ + AI workflow" is unproven — collect GSC evidence first. Free interactive mini-tools (workflow JSON viewer, prompt enhancer): strong link magnets, real engineering cost. Revisit after the engines ship. The technical layer (all engines) Page type JSON-LD Related-links block Title pattern /templates/* HowTo + ItemList 8–9 related templates <Name> — free AI workflow template /showcase/*, /w/* ImageObject/VideoObject + ItemList ~9 related showcase items the prompt, verbatim /models/* SoftwareApplication + FAQPage sibling models + its prompt/showcase items <Model> in NodeTool — run it in a visual AI workflow (<year>) /models/*-vs-* BreadcrumbList + FAQPage all sibling model comparisons <A> vs <B>: same prompt, side by side (<year>) Landing (tasks, personas) Product + FAQPage templates grid + sibling landings <Keyword> — Free, Open Source \| NodeTool /vs/*, /alternatives/* BreadcrumbList + FAQPage all sibling comparisons NodeTool vs <X>: Which Is Better? (<year>) /faq/* QAPage related FAQs + one money page the question, verbatim /ideas/* ItemList every template in category <N> <category> workflow ideas (<year>) Sitemap becomes derived, not maintained. marketing/src/app/sitemap.ts is a hand-ordered array of 21 entries; at 1,000+ pages that breaks. Each engine’s data module exports its entries; sitemap.ts concatenates them. Standing rules stay: priority tiers, and no route ships without a sitemap entry, an opengraph-image.tsx, and a row in marketing/tests/e2e/smoke.spec.ts (which should walk the same data modules). Internal-link floor. Photo AI runs ~400 internal links per page; NodeTool needs the invariant, not the number: every page reachable from a hub, every page linking onward to 8+ siblings, no orphans (the /vs/* pages spent their first day orphaned — SEO Strategy §1). Canonicals self-referencing everywhere; near-duplicate showcase pages canonical to the first-published copy. llms.txt regenerates from the same data modules, hubs first. Build sequence Revised: the showcase pipeline moves early because Engines 1, 3, and 4 all lead with its outputs. The factory + Engine 1 (template pages, ~35). Data modules, graph-render route, /templates hub, derived-sitemap refactor. Everything else reuses this plumbing. Engine 2 seeding pipeline + first ~200 showcase pages. CLI batch runner, asset store, showcaseEntries generation, the /showcase/* route. Gate scaling on indexation data. Engine 3 model pages for the searched models already in the node packages (Seedance, Veo, Kling, Sora, Hailuo, Wan, Seedream, FLUX, GPT-Image, Imagen, ~15 pages) + the top ~15 model-vs-model pairs (Engine 3b), heroes supplied by step 2. Engine 5 completion (~30 pages: /alternatives/*, year helper, mesh block, wide net) plus the /ollama page. Engine 4 task keywords and task hubs (~40–60 pages), then the two missing personas. Engine 6 FAQ/ideas/glossary (~40 pages) + the schema table everywhere. Engine 2b real publish loop once Cloud has the product surface. Steps 1–6 take nodetool.ai from 21 indexed pages to ~600–800, every one output-first and from a fixed template, before any real user publishes anything. Execution: NodeTool workflows run the machine The work splits cleanly into code that ships once and content that regenerates forever. The second half is exactly what NodeTool is for, so the SEO machine’s production layer is NodeTool workflows run headlessly — which is also a marketing story in itself (the site’s content is made by the product it sells, and the seeder workflows ship as /templates/* pages like any other). Layer 1 — the factory (TypeScript, one-time, ~3 PRs). Not workflows; this is Next.js build plumbing in marketing/: Data modules + /templates/[slug] route with the JSON→flow-components graph renderer, the /templates hub, derived sitemap, and the smoke-test walker. Everything later reuses this. /models/[slug] and /models/[a]-vs-[b] routes reading modelEntries.ts. /showcase/[slug] route reading the generated showcase manifest. Layer 2 — content production (NodeTool workflows in marketing/seo/workflows/, DSL .ts files, run via nodetool workflows run <file> --params). Four workflows: Showcase Seeder — inputs: template slug, model list, prompt count. An agent or ListGenerator writes prompt variants in the template’s domain, each fans out through TextToImage/ImageToVideo per model, outputs land in marketing/seo/out/<batch>/ with one manifest row per run (prompt, model, provider, params, file). The same pattern as the shipped Image to Video Animation and Movie Posters examples, parameterized. Model Duel — one prompt, two model parameters, identical generation nodes; emits the paired outputs plus an agent-drafted observation block for the /models/a-vs-b page. Copy Writer — the shipped SEO Content Engine template (strategist agent → briefs → full Markdown per brief), retargeted: given a model id and its provider manifest, draft the hand-written slice of its model page (capability blurb, FAQ answers, title/meta) as Markdown for review. Drafts only — a human edits before merge. Model Watcher — a plain CI script, not a workflow (reliability beats dogfooding here): weekly generate <provider> --list-models --json diff against modelEntries.ts; a new model opens a PR with the stub entry and kicks a Seeder batch for it. Layer 3 — glue (GitHub Actions). One seo-seed.yml with cron + manual dispatch: runs the seeder/duel workflows with provider keys from repo secrets and an explicit per-batch spend cap in --params, then an ingest script that converts manifests into showcaseEntries.generated.ts, moves assets to the CDN path, and opens a PR. The PR is the quality gate: a human eyeballs every batch’s outputs before they go live — this is where the playbook’s index gate is enforced, not in code review of the pipeline. Merge deploys the site. What deliberately stays out of workflows: page rendering, sitemap, canonicals — all derived at Next.js build time from the checked-in data. The workflows produce data and assets; the build consumes them. That boundary keeps a bad generation batch revertable with git revert. Guardrails The node-docs lesson is the null hypothesis. Any engine can reproduce §0.4 (thousands of impressions, no clicks) if pages chase queries whose intent NodeTool can’t satisfy. The gate for every new page type: name the query, say why that searcher wants a workflow tool, and check the unique slice answers it. Model pages pass because “run X cheaper via any provider” is what a model searcher wants next; node-reference pages failed because “pdfplumber docs” searchers didn’t. Seeded content stays honest. Real generations, correctly attributed to model and workflow; no invented users, no review markup, no passing off one model’s output as another’s. The showcase label is load-bearing — it’s what distinguishes this from the spam Google actually targets. Watch site-level dilution, not just per-page failure. If a new engine’s pages sit indexed-but-not-ranking after a quarter, prune or noindex the tail rather than adding more. Photo AI’s thin tail is carried by unique images; NodeTool’s showcase pages have the same anchor, but template/model pages need their text to match intent too. Cap seeding spend. The showcase pipeline’s cost is inference; set a per-batch budget and prefer models NodeTool gets via cheap providers for the long tail, reserving expensive models (Veo, Sora) for pages that target their queries. Measure per engine. Segment GSC by URL prefix (/templates/, /showcase/, /models/, /vs/, /alternatives/, /faq/, /ideas/) plus conversions to /studio and /pricing. An engine with zero non-branded impressions after 8 weeks gets its keyword axis re-examined before its page count grows. Numbers in this file rot. 37 templates, 24 node packages, 21 sitemap entries were true on 2026-07-02. The factory design makes the counts irrelevant; the audit claims age worse.--- # SEO Strategy Source: https://docs.nodetool.ai/seo-strategy Research basis: an audit of marketing/ (the nodetool.ai Next.js site), docs/, the 61 example workflows in packages/base-nodes/nodetool/examples/, and the node packages in packages/, plus external research on competitor positioning and search behavior (sources linked inline). Written 2026-07-01, updated 2026-07-02 — re-audit the live site before acting on the numbers below if this file is more than a quarter old. Status: the movie-trailer sitemap fix shipped 2026-07-01. /vs/langflow and /vs/n8n (§4.1 items 1 and 3) and the footer “Compare” column (§1, orphaned-pages finding) shipped 2026-07-02. /marketing (§4.0), /vs/flowise, and /vs/dify (§4.1 items 2 and 5) shipped 2026-07-02. §0 (the Search Console audit) added 2026-07-02 — it re-prioritizes §7 with real query data. The programmatic build plan (template pages, the seeded showcase engine, model and model-vs-model pages, keyword landing matrix, comparison/alternatives mesh at scale, and FAQ/ideas hubs) lives in SEO_PROGRAMMATIC.md — it extends §4 from hand-built pages to data-driven page factories, adapted from a photoai.com teardown. Shipped from §0 on 2026-07-02: 79 redirect stubs under docs/redirects/ (§0.5, plus a fix to the three pre-existing stubs, whose trailing-slash redirect_to 404s on GitHub Pages); providers.md and installation.md rewrites (§0.6); comparisons.md cross-links to /vs/* (§0.6); /agents and /cloud retitles (§0.7); llms.txt on nodetool.ai and JSON-LD on /agents, /creatives, /marketing (§0.8). The Weavy → Figma Weave capture plan (§0.9) shipped 2026-07-16. §0.10 is the second Search Console audit (2026-08-10) — read it before §0, which it supersedes on every number. It re-prioritizes §7 again: the programmatic build of §4/SEO_PROGRAMMATIC.md has shipped (≈400 indexed www URLs, up from 13), so the work is no longer “add page types”. It is de-duplicating the families that now compete with each other and giving the one large non-branded cluster a page of its own. 0.10 Second Search Console audit (2026-08-10) Export: 1,000-row page report (1,543 clicks / 74.1k impressions) and 1,000-row query report (776 clicks / 19.2k impressions), trailing window ending 2026-08-10. Both reports are truncated at 1,000 rows by GSC, so totals are floors, and the query report’s tail is all zero-click. Against the 2026-07-02 baseline (939 clicks / 48.2k impressions): +64% clicks, +54% impressions. Twelve findings, each with the action it forces. Growth is real; the branded ceiling has not moved. Branded queries take 579 of 776 visible query clicks (75%) on 2,281 impressions. Non-branded: 197 clicks / 16.9k impressions / 1.16% CTR — the same 1.2% as in §0, on 1.8× the impressions. The programmatic build bought surface, and the surface earns impressions. It does not yet earn clicks, because almost none of it ranks above position 7. Read every CTR number below with finding 8 in mind. The “node AI” head cluster tripled and is still stuck on the lower half of page one. 120 query variants, 11,123 impressions — 66% of all non-branded impressions — at weighted position 7.5 and 1.18% CTR. “ai node” alone: 4,527 impressions, 4 clicks, position 6.4. Every variant lands on /, which absorbs 32.9k impressions and 80% of all site clicks. The signal inside the cluster is that specificity converts: “ai node editor” (310 impressions, position 5.4) runs 3.87% CTR, 3× the cluster average, and “node ai tool” (130 impressions, position 1.4) runs 23.8%. Action: this is the largest single lever on the site. Build one entity page for the cluster — /node-based-ai (or promote /studio) whose title, H1 and opening paragraph are the query, not a homepage that has to serve every intent at once. Link it from the homepage hero and from every /alternatives/* page. Moving “ai node” from 6.4 to 3 is worth roughly 450 clicks a quarter at category CTR — more than the entire non-branded total today. /vs/* and /alternatives/* are one dataset behind two templates, and they eat each other. Both routes call generateStaticParams over the same competitors array (marketing/src/data/competitorEntries.ts, 13 entries), so every competitor ships two URLs from two ~220-line templates that differ in copy but not in intent. /alternatives/*: 4,817 impressions / 68 clicks. /vs/*: 1,117 / 9. /alternatives wins 11 of 12 head-to-head pairs — comfyui 821 vs 115, lm-studio 1,456 vs 132, figma-weave 674 vs 46, flowise 210 vs 74, langflow 216 vs 48 — losing only n8n (87 vs 232) and jan (57 vs 110). §0.9 warned about cannibalization between the weavy and figma-weave pages; the export shows it is the general case, not a special one. Action: make /alternatives/<slug> canonical, 301 /vs/<slug> → it, and delete one template. Keep the “X vs NodeTool” query surface as H2 sections inside the alternatives page. Move /vs/n8n and /vs/jan’s copy angle across before redirecting them, since those two out-rank their twins. The same duplication runs through /apps/*, /templates/* and /use-cases/*. 20 slugs exist under both /apps and /templates (415 vs 384 impressions on the overlap). Movie trailer has four URLs: /apps/movie-trailer-generator (47 impressions, position 38.9), /templates/movie-trailer-generator, /use-cases/movie-trailer (182, 21.4), and docs.nodetool.ai/use-cases/movie-trailer.html — which is why the best of them ranks 21st for “ai movie trailer generator” (32 impressions, position 46.9) while converting 4.95% when it does show. Product video, music video visualizer, image-to-video animation, SEO content engine, color-boost video, and research paper summarizer are each split the same way. Action: one intent, one URL. Split the families by what they actually are — /templates/* is the workflow file, /apps/* is the runnable mini app — and where a pair is one thing, redirect the weaker URL. Do this before adding more of either. Corrected 2026-08-10 after reading marketing/next.config.mjs: the /apps ↔ /templates half of this finding was already fixed in code. The examples-revamp retired those /apps/<slug> pages and redirects each one to its absorbing app or template, and every slug named above is in that map. The export was measuring URLs that had already stopped resolving — a trailing-window artifact, not a live duplicate. Read it as a caution about this whole audit: GSC reports lag the deploy, so check the route table before acting on a duplication finding. What survives the correction is the movie-trailer split, which is real and still live: /use-cases/movie-trailer (182 impressions, position 21.4), /templates/movie-trailer-generator, and the docs.nodetool.ai/use-cases/movie-trailer.html mirror all resolve today and all target one intent. Three query sets have tool intent, page-one impressions, and no page that matches. Billing block generator. “billing block generator” 140 impressions at position 7.7, plus “movie poster billing block generator” (12), “movie billing block generator” (5), “billing block creator/maker” (7), “credit block generator” (2). ~170 impressions of buy-intent currently absorbed by /use-cases/movie-poster. A poster billing block is a text-layout job NodeTool can do end to end, and the query has no strong incumbent. Ship a page for it. Brand asset generator. “brand asset generator” 126 impressions at position 6.1, 0 clicks, plus “brand assets generator” (15) and “brand asset tool” (7). /apps/brand-asset-generator already exists (193 impressions, position 8.4, 1.04%). This is a title/H1 problem, not a missing-page problem — match the query exactly. KIE — corrected 2026-08-10 after reading the page. 652 impressions across 20 queries at position 7.5 for 2 clicks: “kie ai” (609), plus kie api / models / suno / kling / omni / kie_api_key. /providers/kie takes 838 impressions at 0.24% CTR. This finding first called for rewriting the page “as a landing page, not a spec dump” — wrong, inferred from the CTR without opening it. /providers/[slug] already ships a differentiated title, a tagline, three blurb paragraphs, strengths, an FAQ, and a generated model catalog. The CTR has a simpler cause: “kie ai” is navigational. Searchers want kie.ai, and a third-party page at position 7 will convert badly no matter how good it is. The same reading applies to the rest of the family (/providers/replicate 198 impressions / 0 clicks, /providers/fal 146 / 0, whole family 1,470 at 0.20%). Action: do not invest in the provider pages for brand-navigational queries — that CTR is the expected outcome, not a defect. The winnable slice is the capability tail those brands carry (“kie suno”, “kie kling”, “kie api”, “kie models”, “kling 3.0 local model”, “together ai image generation pricing flux”), which asks what a model can do and what it costs. That question belongs on the model pages (finding 9’s per-model cost text), not on a provider hub. “Mini apps” is an emerging category term and the hub does not own it. 11 queries — “mini apps ai”, “ai mini apps”, “mini app ai”, “ai generator mini apps” and misspellings — 85 impressions at position 7.7, 0 clicks. The /apps hub took 115 impressions at position 7.6. Action: cheap fix. Give /apps the category title, an intro that defines the term, and links to all 47 app pages. Docs are 28% of impressions and 5% of clicks, and part of the index is machine noise. docs.nodetool.ai: 80 clicks / 20.9k impressions (0.38% CTR). The 529 node-reference URLs took 7,867 impressions for 22 clicks — §0’s finding 4, unchanged and larger. New in this export: 32 internal planning documents are indexed (plans/*, *-prd, *-design, *-implementation-plan, RELIABILITY_*, KERNEL_*, sdk/non-regression-baseline-*), 306 impressions between them — including this file, at docs.nodetool.ai/seo-strategy (20 impressions, position 13.4). Action: noindex the planning docs and the two SEO docs. Leave the node reference indexed as a support asset and stop treating it as an opportunity. Third-party documentation queries are a permanent ~620-impression tax on sitewide CTR. 119 queries — pdfplumber extract_text, stable diffusion img2img/controlnet/inpainting “official docs”, notion api, aws s3, apify, plus site:/ext:/after: operators — took 619 impressions and zero clicks. They will never click a NodeTool page. Action: exclude them before reading any CTR figure, and never optimize a page for them. AI-assistant traffic is now measurable, and it arrives shaped like conversation. 64 queries (278 impressions, weighted position 8.5, several at 1–3) are LLM-mediated: two full persona prompts (a CTO and a CPO brief, both ending “how can i chain multiple llms like claude and gpt in one visual workspace?”), evaluation commands (“evaluate the generative ai company figma weave / weavy on local and self-hosted ai generation for privacy-conscious work”, position 2.0), and bare follow-up fragments — “which is free”, “is it safe and private?”, “how much do these cost?”, “open source?”, “does this already exist?”, “what other similar tools”. §0.8’s bet paid off; the answer-shaped pages are what rank here. Action: two named gaps. The phrase “chain multiple LLMs like Claude and GPT in one visual workspace” appears in three separate prompts and has no page — build it. And “how much do these cost” is per-model demand answered by a single /pricing page (173 impressions); put plain crawlable per-model cost text on the model pages. Non-English demand exists and nothing serves it. “node ia” (70 impressions, position 8.5), “ia node” (19), “nodes ia” (2), “노드ai”, “图像生成”, “quels sont les meilleurs espaces de travail créatifs ia” (7), “alternativas a lm studio”, “hay alguna alternativa gratuita”, “para imagenes”, plus Portuguese, Russian, Turkish and Vietnamese follow-up fragments — ~130 impressions, no localized page anywhere. Action: not a translation project on this evidence. Pull the GSC country/language report first. If it holds up, the cheap test is Spanish and French copies of / and /alternatives/comfyui with hreflang, nothing more. Page-two rescue list — ranked, not missing. Pages with ≥50 impressions at position 11–30, by impressions: /alternatives/figma-weave (674, 11.3), /alternatives/weavy (667, 12.3), /tasks/lip-sync (426, 11.0 — “lipsync model”, “best lipsync model”, “open source lip sync”), /templates hub (270, 15.1), /vs/n8n (232, 13.9), /use-cases/movie-trailer (182, 21.4), /tasks/image-to-video (135, 15.0), /use-cases/product-video (124, 10.3, 0 clicks), /apps/audio-to-image (94, 11.3 — 5.32% CTR, the best-converting page on the site). Action: internal links and content depth on these nine beats nine new pages. This is the highest ratio of return to effort in the whole audit after finding 2. /agents proves the sitelink theory: 1,054 impressions at position 4.8, zero clicks. §0’s finding 7 guessed that mid-position segment pages were absorbing branded sitelink impressions; a full window at position 4.8 with no clicks confirms it. /cloud (1,463 impressions, position 5.3, 2 clicks) and /creatives (1,776, 9.1, 10) behave the same. The §0.7 retitles did not fix it because the title was never the problem. Action: stop reading these impressions as demand — they are branded overflow, and they flatter the sitewide numbers. Judge these pages on the non-branded queries they earn, or accept them as conversion pages and not entry pages. What this changes about the plan §4 and SEO_PROGRAMMATIC.md described building page factories. Those shipped. The next quarter is consolidation, not expansion: Merge, don’t add (findings 3, 4): 26 competitor URLs → 13; 20 duplicated app/template slugs resolved; the four movie-trailer URLs → one. One new page that matters (finding 2), plus three small ones with named demand behind them (findings 5, 6, 9). Rewrite nine ranked pages rather than ship nine new ones (finding 11). Shrink the index (findings 7, 8): noindex on 32 planning docs; stop counting node-reference and third-party-docs impressions as an audience. 0.9 Weavy → Figma Weave capture plan (2026-07-16) Figma acquired Weavy in October 2025 and renamed it Figma Weave (weave.figma.com, separate billing and its own AI credits). The integration is deepening — Weave Workflows on Figma Community (April 2026), Weave tools inside Figma Design (Config 2026) — so search demand is splitting into two query sets, and the second one is growing: “weavy alternative” — the established term. Every ranking page now leads with the rebrand (Wireflow’s “Top 8 Weavy Alternatives (Weavy Is Now Figma Weave)”, Chase Jarvis’s “Best Weavy Alternatives for Creative Pros”, designtools.fyi). A /vs/weavy page that never mentions Figma reads stale to searchers and to the AI assistants in §0.8. “figma weave alternative” — the new term. Wireflow, Astorie, and Krea already run dedicated pages for it; NodeTool had zero surface. What shipped 2026-07-16: /vs/weavy and /alternatives/weavy refreshed — title, hero, bullets, and a new FAQ row (“What happened to Weavy?”) acknowledge the acquisition while keeping the BYOK/open-source argument. The slug and query target stay “weavy”. /vs/figma-weave and /alternatives/figma-weave added (competitorEntries.ts, isNew) — distinct copy angle from the weavy pages: ecosystem lock-in and post-acquisition ownership (“build on a canvas nobody can acquire”) rather than a re-tread of the credits argument, with an honest “when is Figma Weave the better pick?” FAQ and a free-tier concession row. Both pages are one product, so watch GSC for cannibalization between the weavy and figma-weave pages; if one set stops earning impressions, fold it into the other with a redirect. Keeping both mirrors how Wireflow serves the two query sets today. FAQ page /faq/open-source-figma-weave-alternative — targets the question form AI assistants ask (§0.8); JSON-LD comes with the FAQ template. Metadata: “Figma Weave alternative” keyword on / and /creatives; homepage JSON-LD and the homepage comparison card renamed; docs/comparisons.md updated; llms.txt regenerated. Distribution follow-ups (the §6 play, with named targets — these pages rank today and take listings): request inclusion in Wireflow’s “Top 8 Weavy Alternatives”, Chase Jarvis’s roundup, and designtools.fyi’s Figma Weave page. The migration audience — Weavy users re-evaluating after the acquisition — is the highest-intent segment this site can address, and “your workflows are files you own” is the argument acquisitions make for us. Watch queries (GSC, weekly, alongside the §0.2 set): “weavy alternative”, “figma weave alternative”, “figma weave pricing”, “figma weave vs”, “open source figma weave”. 0. What Search Console actually says (audit 2026-07-02) Everything below §0 was written from a content audit. This section is the first pass with real Google Search Console data (export pulled 2026-07-02, trailing window): 939 clicks / 48.2k impressions across 474 pages; 504 clicks / 11.1k impressions across the visible query set; 933 clicks / 40.1k impressions by country. Eight findings, each with the action it forces: Traffic is almost entirely branded. “nodetool”-variant queries deliver 389 of 504 visible query clicks (77%) at position ~1.7 and 23.6% CTR. Non-branded queries: 115 clicks on 9.4k impressions — 1.23% CTR. The doc’s premise (the site lacks non-branded surface) is confirmed, not just plausible. Exactly one non-branded cluster ranks, and it’s stuck at position 7–9. “ai node” (3,171 impressions, pos 7.4), “node ai” (1,093, pos 8.7), “node based ai” (797, pos 8.7), “ai node editor” (197, pos 5.8) and ~20 variants — all landing on the homepage. At position 7–9 CTR is ~1%; the same impressions at top-3 would be 10–25×. This cluster is also intent-ambiguous (Node.js AI? ComfyUI nodes?), which is why the better-intent comparison and use-case queries in §3–§4 matter more: the export shows zero impressions for “comfyui alternative”, “weavy alternative”, “langflow”, or “n8n alternative” — the /vs/* pages shipped 07-01/07-02 and have no data yet. Watch them weekly. The US is the weak market, and it’s the biggest. 17.3k of 40.1k impressions are US, but average US position is 12.5 vs 6.4–7.3 in every other top-10 country (US CTR 0.59% vs 3–5% elsewhere). That gap is authority/competition, not on-page tags — it upgrades §6 (distribution: directories, awesome-lists, Show HN, roundup outreach) from “ongoing” to a priority. Docs earn 39% of impressions and 0.4% CTR. docs.nodetool.ai: 75 clicks / 18.9k impressions. The 375 node-reference URLs in the export took 7.8k impressions for 12 clicks — the queries are “official docs” lookups for third-party libraries (pdfplumber, stable diffusion img2img, mlx_whisper) that will never click a NodeTool page. Do not invest in ranking node pages further; fix their meta (§4.3) and move on. The node-docs restructure orphaned 234 of 460 docs URLs in the export (~6.9k impressions) with no redirects. nodes/comfy/*, nodes/mlx/*, nodes/huggingface/image_to_image/*, nodes/lib/pdfplumber/*, and runpod-deployment all 404 now. The mechanism to fix it already exists (_layouts/redirect.html + redirect_to front matter, used by desktop-app.md, api.md, packs.md). Batch-generate redirect stubs where a successor page exists (lib/pdfplumber/* → lib/pdf/*, huggingface/image_to_image/* → huggingface/imagetoimage, runpod-deployment → deployment); let genuinely removed packs 404. Three docs pages carry real demand and rank badly. providers.html is the second-highest impression URL on either domain (3,181 impressions, position 20, 0.09% CTR); installation.html (1,904, pos 15.1); comparisons.html (716, pos 21 — it overlaps the new /vs/* pages and should link to them prominently). These three are the only docs pages worth active optimization: restructure providers.html with per-provider H2s and a capability table; expand installation.html with per-OS sections and troubleshooting. Marketing pages at position 5–9 get near-zero clicks — /cloud (716 impressions, pos 4.9, 0 clicks), /agents (664, pos 5.1, 0), /developers (1,570, pos 6.1, 0.51%), /creatives (1,034, pos 9.1, 0.48%), /studio (942, pos 6.8, 0.32%). Most of these impressions are likely sitelinks under branded queries, where the homepage absorbs the click — so title rewrites alone won’t fix CTR. The durable fix is giving each page a distinct non-branded query target so it earns its own impressions: /agents → “AI agent workflow builder” (its current title, “Agents for creative work”, targets nothing anyone searches), /cloud → “browser AI workflow builder / no-install”, /developers already targets “AI workflow SDK” reasonably. AI assistants are a real, visible acquisition channel. Dozens of queries in the export are LLM-generated: persona-length prompts (“shortlist node-based ai editors that let design teams…”), site:/after: operators, and conversational follow-ups (“which is free”, “are any of these open source?”, “you sure?”). NodeTool is being evaluated inside AI-assisted research sessions. Actions: add llms.txt to nodetool.ai (docs already has one); keep license/pricing/free-tier facts as plain crawlable text on /pricing and every /vs/* page; extend the existing JSON-LD (JsonLd.tsx, currently on a handful of pages) to all indexed routes. One stray demand signal: “real time agent api” (132 impressions, pos 36) plus the now-deleted nodes/openai/agents/realtimeagent.html (192 impressions) — there is search demand for a realtime agent guide; a proper docs page would inherit it. 1. Where nodetool.ai stands today Stale as of 2026-08-10 — this section describes the site before the programmatic build. The site now ships ≈400 indexed www URLs across /alternatives, /vs, /templates (146 slugs), /apps (47), /providers (20), /tasks (8), /solutions (8), /models, /faq, /ideas, /blog, and /showcase. See §0.10 for what that surface actually earns. Kept below for the gap analysis, which still reads correctly as history. The site (marketing/src/app/) has 13 indexed pages: /, /studio, /cloud, /pricing, /agents, /creatives, /developers, two comparison pages (/vs/comfyui, /vs/weavy), two use-case pages (/use-cases/product-video, /use-cases/movie-poster), and three legal pages. Five gaps stand out: /use-cases/movie-trailer exists but is missing from sitemap.ts (marketing/src/app/sitemap.ts). It has a page folder and an entry in useCaseEntries.ts but never reaches search engines. Fixed 2026-07-01. The /vs/* comparison pages are orphaned. Nothing on the site links to them — they’re reachable only through sitemap.xml. Pages with zero internal links get crawled less, rank worse, and pass no authority. Fixed 2026-07-02 with a “Compare” column in the shared footer (marketing/src/components/SiteFooter.tsx), which gives every comparison page a sitewide internal link. Keep this rule for every future page type: nothing ships without at least one internal link from an existing page. No blog. Every competitor identified in research below (Dify, n8n, Flowise, Langflow, Lindy, Gumloop, Vellum, StackAI, Wireflow) runs a blog and ranks on “X vs Y” and “best tools for Z” queries, which is exactly the traffic NodeTool needs and currently cedes to them. 3 use-case pages for 61 shipped example workflows. The useCaseEntries.ts pattern (a single data array plus a matching page folder) is already built to scale — it’s just underused by a factor of 20. No /marketing segment page. Fixed 2026-07-02. /marketing ships, featuring the Product Video Generator, in the nav, footer, and sitemap alongside Creatives/Agents/Developers. See §4.0. The existing meta setup is solid: layout.tsx already ships title/description/OG/Twitter tags and a keyword list (agent-first creative workspace, BYOK AI canvas, ComfyUI alternative, Weavy alternative, node-based AI canvas). robots.txt and sitemap.ts both exist. The problem is page count and content depth, not technical hygiene. 2. Competitive landscape Three groups of tools currently absorb the search intent NodeTool could capture: LLM/agent workflow builders — Dify, Flowise, Langflow, n8n. StackAI’s comparison names Dify “the best starting point for most teams in 2026” on debugging and knowledge-base features; Langflow is positioned for teams that need custom Python nodes and LangGraph; Flowise for the fastest path to a RAG chatbot; n8n where orchestration (retries, branches, schedules) is the hard part (Runchat comparison). None of these natively render image, video, or audio — they’re text/LLM-first, which is NodeTool’s opening. Node-based creative canvases — ComfyUI (already covered at /vs/comfyui), Weavy (covered at /vs/weavy and, post-rebrand, /vs/figma-weave — see §0.9; acquisition first flagged in Chase Jarvis’s review), plus Flora, Freepik Spaces, Krea, Layer AI, Griptape Nodes, and Reflet.ai — all covered in a single “2026: The Year of the Node-Based Editor” roundup and multiple “Weavy alternatives” posts (Wireflow). Most are browser-only, credit-metered SaaS with no local execution and no BYOK — the same gap NodeTool already argues against Weavy, reusable against this whole category. Local-first AI tooling — Ollama, LM Studio, Jan, Open WebUI cover local chat; none combine local inference with cloud generation or a visual workflow canvas (overchat.ai roundup, needaitool.com roundup). Net position: NodeTool is the only tool showing up across all three searches — LLM/agent builders don’t do media, creative canvases don’t do local models or agents, local-first tools don’t do workflows. That’s the argument every comparison page and use-case page should make concrete, not assert abstractly. 3. Audience segments and what they search Segment What they search Where they currently land instead Creatives (filmmakers, designers) “text to video pipeline”, “ComfyUI alternative no code”, “AI movie poster generator” Runway, Krea, Melies, Weavy alternatives posts Marketing teams (campaigns, growth, social) “product video AI tool”, “AI content calendar generator”, “brand asset generator AI”, “AI ad video generator” Lindy, Gumloop, Runway, Weavy alternatives posts Developers / ML engineers “open source AI agent framework”, “RAG pipeline no code”, “TypeScript AI workflow SDK”, “custom LLM node builder” Langflow, Flowise, LlamaIndex RAGArch Automation / ops teams “n8n alternative for AI agents”, “no-code AI workflow local”, “email triage AI automation” n8n, Activepieces, Gumloop Privacy-conscious / local-first users “run AI models locally desktop app”, “local LLM privacy no cloud”, “self-hosted AI agent” Jan, LM Studio, Ollama-based roundups Researchers / knowledge workers “chat with PDF tool”, “summarize research papers AI”, “document RAG open source” Kapa.ai, Prisme.ai The site already has landing pages for creatives, developers, and agents (automation). It has none targeting marketing teams specifically, even though the homepage copy names them by title (see §4.0) — nor the local-first/privacy segment (this is argued piecemeal on /studio and in comparisons, never as its own page), nor the researcher/knowledge-worker segment (despite shipping 5 document-processing example workflows: Chat with Docs, Index PDFs, Summarize Paper, Fetch Papers, Meeting Transcript Summarizer). 4. Page-type playbooks 4.0 Segment landing page — /marketing Shipped 2026-07-02. /creatives, /agents, /developers, and now /marketing are each a persona-framed rewrite of the same product, built to rank on that persona’s job-title searches and to give paid/social traffic a landing page that speaks their language instead of the generic homepage. /marketing (marketing/src/app/marketing/page.tsx) follows the /creatives pattern — headline, “why marketing teams choose NodeTool” section, a lead use-case feature, community/CTA — and is in sitemap.ts at priority 0.8/monthly, the main nav, and the footer “Solutions” column: Lead use case: Product Video Generator (use-cases/product-video, already category: "Marketing" in useCaseEntries.ts), featured with its own section and real assets. Supporting use cases still to build (§4.2 backlog): Social Media Calendar Filler, Brand Asset Generator, Cold Outreach Co-Pilot, YouTube Thumbnail Pipeline. /marketing lists these as “coming soon” teaser cards (no dead links — they get real hrefs once useCaseEntries.ts gains matching entries and page folders with real generated assets). Target keywords: “AI product video generator”, “AI content calendar tool”, “brand asset generator AI”, “AI ad video tool open source” (§3) — in the page’s metadata/keywords. Differentiation angle: same BYOK/no-markup argument as /creatives, reframed around output volume and cost-per-asset at campaign scale rather than single-artifact craft. 4.1 Comparison pages (/vs/*) Pattern already proven at marketing/src/app/vs/comfyui/page.tsx and vs/weavy/page.tsx (feature-table format, ~250 lines each, hand-built per page). Expand from 2 to a target list based on what people are already comparing: /vs/langflow — pull in the “no media generation” gap Shipped 2026-07-02. Angle used: both cover agents/RAG; only NodeTool generates media natively (Langflow is MIT, has a macOS/Windows desktop app — the honest table concedes both). /vs/flowise — same gap, plus BYOK vs. hosted credits Shipped 2026-07-02. Angle used: Flowise is the fastest path to a LangChain RAG chatbot; NodeTool covers the same agent/RAG ground and adds native image, video, and music generation plus editing tools on the same canvas. /vs/n8n — orchestration-only vs. orchestration + native generation Shipped 2026-07-02. Angle used: “workflows that create, not just connect”, plus AGPL-3.0 (open source) vs. Sustainable Use License (fair-code, commercially restricted) — a real differentiator n8n’s own docs confirm. The FAQ answers “when should I pick n8n instead?” honestly (400+ connectors, schedules/retries), which is what makes the rest of the page credible. /vs/flora (or a combined “AI canvas alternatives” page covering Flora, Freepik Spaces, Krea) /vs/dify — RAG/agent platform, no creative media Shipped 2026-07-02. Angle used: Dify is a strong text-first LLM app platform (prompt orchestration, knowledge bases, agent debugging); NodeTool starts from the same agent/RAG ground and adds native media generation and editing tools. License claim about Dify’s modified Apache 2.0 terms is hedged (“check Dify’s own license file for current terms”) rather than asserted precisely, since Dify’s commercial-use conditions change between releases. /vs/lmstudio or /vs/jan — local chat only vs. local + cloud workflow canvas Each page keeps the existing format: one comparison table, one differentiation section, one CTA. Do not invent a new template per competitor — the value here is coverage, not novelty. 4.2 Use-case pages (/use-cases/*) The useCaseEntries.ts array is the scaling mechanism: add an entry, build the matching page folder. 61 example workflows already exist in packages/base-nodes/nodetool/examples/nodetool-base/ with titles, descriptions, and (per the internal audit) natural category groupings. Prioritize by search-intent match, not by what’s easiest to build: Ship first (clear existing search terms, per §3): Research Paper Summarizer, Chat with Docs (→ “chat with PDF”), Meeting Transcript Summarizer, Music Video Visualizer Ship second: Hacker News Agent / YouTube Research Agent (agent-mode examples double as /agents landing-page proof) Marketing segment (feature on /marketing, see §4.0): Cold Outreach Co-Pilot, Social Media Calendar Filler, Brand Asset Generator, YouTube Thumbnail Pipeline Fix immediately: add /use-cases/movie-trailer to sitemap.ts (see §1) At 61 candidates, treat this as a backlog, not a one-time push — 3-5 new use-case pages per month sustains a steady stream of long-tail landing pages without a content team. 4.3 Node / provider pages docs/nodes/<provider>/ and docs/developer/providers/<provider>.md already exist for FAL, KIE, Gemini, ElevenLabs, Together, Replicate, Ollama, HuggingFace, Topaz, AtlasCloud, and more (20+ providers). These are developer docs, not landing pages, but each one is a near-zero-cost target for provider-name search traffic (“FAL nodes for NodeTool”, “use Kling in NodeTool”) once they carry proper titles/descriptions and get linked from /developers. No new pages needed here — just meta tags on what already exists (see §5). 4.4 Blog (net new) No blog exists. This is the biggest structural gap relative to every competitor in §2. Recommended scope, not a general-purpose blog: Comparison/roundup posts that NodeTool can legitimately win a mention in: “best ComfyUI alternatives 2026”, “open source Weavy alternatives”, “local-first AI tools 2026” — these are the exact post titles already ranking for competitors (§2 sources). Writing NodeTool’s own version, plus getting listed in the external ones (see §6), covers both sides. Cookbook walkthroughs built directly from docs/cookbook/core-concepts.md and the 15+ recipes in docs/cookbook.md — these already exist as docs; a blog post is a rewrite with a narrative frame and screenshots, not new research. Technical posts for the developer segment: actor-model workflow execution, the Python bridge (PythonStdioBridge), the planning agent system (packages/agents/) — these map to real architecture (docs/architecture.md, docs/AGENTS.md) and target “how do AI agent frameworks work” / “TypeScript LLM orchestration” searches from engineers evaluating tools. 5. Technical SEO checklist Add /use-cases/movie-trailer to marketing/src/app/sitemap.ts (shipped 2026-07-01) Internal-link the /vs/* pages — footer “Compare” column in SiteFooter.tsx links all four comparison pages sitewide (shipped 2026-07-02); every future page needs at least one internal link before it ships Build /marketing (§4.0) and add it to sitemap.ts at priority 0.8 / monthly — same tier as /agents, /creatives, /developers — plus main nav (shipped 2026-07-02) Confirm every new page under /vs/, /use-cases/, /marketing, and any future /blog/ gets an opengraph-image.tsx (all six vs/* pages and /marketing do — keep the pattern) Add priority/changeFrequency entries to sitemap.ts for every new page as it ships — don’t let the list drift out of sync with app/ folders again (/marketing, /vs/flowise, /vs/dify added 2026-07-02) Add every new indexed route to marketing/tests/e2e/smoke.spec.ts — it guards title, single-h1, and render status per route (/marketing, /vs/flowise, /vs/dify added 2026-07-02) Verify docs/nodes/<provider>/index.md pages carry unique titles and descriptions (currently generated docs; check for duplicate/boilerplate meta across providers) Internal linking: /agents, /creatives, /developers should each link to the use-case pages relevant to their segment (per §3’s mapping) — /marketing now does this for its lead use case, but the use-case showcase otherwise only appears on the homepage and /creatives 6. Distribution (off-site) Content only ranks if something points at it: “Alternatives to X” roundups (Wireflow, Lindy, Vellum, Gumloop, StackAI, and similar sites identified in §2 already publish and rank these lists) — get NodeTool listed via outreach when a new comparison or use-case page ships. This is lower effort than link-building from scratch and targets people already close to a decision. Directories already listing NodeTool: futuretools.io, theaidb.com, SourceForge (confirmed via search in this audit). Keep listings current with each release; check for others in the same category (There’s An AI For That, AI Tools Directory sites). Adjacent projects to note for competitive tracking, not for outreach: alvinreal/awesome-opensource-ai and light-and-ray/awesome-alternative-uis-for-comfyui on GitHub — both are community-curated lists where a PR adding NodeTool is a legitimate, low-effort listing (not unsolicited outreach). GitHub topics and README SEO: confirm the repo carries relevant GitHub topics (ai-workflow, comfyui-alternative, llm-agents, node-based, local-first) — these surface in GitHub’s own search and get scraped into “awesome-*” lists. Reddit / Hacker News: no existing discussion threads found in this audit (searched directly). This is upside, not a gap to fix — a Show HN or r/LocalLLaMA / r/comfyui post timed to a specific release (e.g., a new local-model integration) reaches an audience already primed for exactly this positioning. 7. Prioritized roadmap Current plan — re-prioritized 2026-08-10 against §0.10. The 2026-07-02 list follows it, kept for the shipped record. Immediate: noindex the indexed planning docs and both SEO docs (§0.10 finding 7); give /apps the “AI mini apps” H1 (finding 6). Shipped 2026-08-10. The noindex set is a defaults block in docs/_config.yml (36 paths, noindex: true + sitemap: false) plus the robots meta in _layouts/default.html; add new plans/PRDs/designs there in the PR that adds them. /apps H1 is now “AI mini apps anyone can use” — the <title> already said it. Consolidation: permanently redirect /vs/<slug> → /alternatives/<slug> and delete the second template (finding 3). Shipped 2026-08-10. The wildcard redirect lives in marketing/next.config.mjs; src/app/vs/ is gone; the head-to-head copy (at-a-glance cards and explainer, with competitorBulletTone honored) now renders on the alternatives page, so the “NodeTool vs X” queries keep on-page support and the n8n/jan angles came across with it. Five fields that only described the retired page (vsTitle, vsDescription, vsOgTitle, vsOgDescription, heroHeading) were deleted from the type and all 13 records; ComparisonMesh’s basePath prop went with them. Note the status code: Next’s permanent: true emits 308, not 301 — equivalent for search engines, and what routes-manifest.json will show if you verify it. Also shipped: the movie-trailer split from finding 4’s correction. All six docs/use-cases/*.md mirrors now carry a canonical_url pointing at their marketing equivalent, so the cross-domain duplicate consolidates without deleting docs content the use-case index still links. The /apps ↔ /templates half needed no work — see the correction under finding 4. The one big page: the /node-based-ai entity page for the 11.1k-impression cluster (finding 2). Shipped 2026-08-10. marketing/src/app/node-based-ai/, priority 0.9, linked from the homepage hero (on the words “node-based”), the footer Product column, and every /alternatives/* page through ComparisonMesh. Sections target specific variants rather than restating the head term, and one section disambiguates the three intents the cluster mixes (visual graph / Node.js / ComfyUI). Watch the cluster’s weighted position — that is the metric this page exists to move. Rescue before build (next month): content depth and internal links on the nine page-two pages in finding 11, starting with /alternatives/figma-weave, /alternatives/weavy and /tasks/lip-sync. The provider-page rewrite that used to sit here is struck — see finding 5’s correction: those pages are already built, and their CTR is a navigational-query artifact. New pages with named demand only (next month): billing block generator (finding 5); a “chain multiple LLMs in one visual workspace” page (finding 9); per-model cost text on the model pages (finding 9). Measure before translating (next quarter): pull the GSC country/language report against finding 10 before committing to any localized page. Unchanged and still ongoing: distribution (§6) — the US authority gap in §0.3 is the reason ranked pages sit at 7–12 rather than 1–3, and no on-page work in items 1–5 substitutes for it. Roadmap as of 2026-07-02 (shipped record) Re-prioritized 2026-07-02 against the Search Console audit in §0. Immediate (hours): fix the movie-trailer sitemap omission (§1, §5) done 2026-07-01; internal-link the orphaned /vs/* pages (§1, §5) done 2026-07-02; batch-generate redirect stubs for the moved node-docs URLs (§0.5) done 2026-07-02 (79 stubs); link comparisons.html to the /vs/* pages (§0.6) done 2026-07-02; add llms.txt to nodetool.ai (§0.8) done 2026-07-02. Phase 1 (this month): rework providers.html and installation.html — the two highest-demand docs pages, both ranking on page 2 (§0.6) done 2026-07-02; give /agents and /cloud distinct non-branded query targets (§0.7) done 2026-07-02; build /marketing (§4.0) done 2026-07-02 — its four supporting use-case pages (Cold Outreach Co-Pilot, Social Media Calendar Filler, Brand Asset Generator, YouTube Thumbnail Pipeline) still need real generated assets before they can ship; ship the remaining “ship first” use-case pages (§4.2); add /vs/langflow and /vs/n8n done 2026-07-02, add /vs/flowise and /vs/dify done 2026-07-02 — next comparison target is /vs/flora (or a combined AI-canvas-alternatives page, §4.1 item 4). Phase 2 (next month): distribution (§6) — upgraded from “ongoing” because the US position gap (§0.3) is an authority problem: submit to the two GitHub awesome-lists, refresh directory listings, and time a Show HN / r/LocalLLaMA post to a release; stand up the blog with 3 posts — one comparison/roundup post, one cookbook walkthrough, one technical post (§4.4). Ongoing: 3-5 use-case pages/month from the remaining 61-workflow backlog; one comparison page per new credible competitor; internal-link every new page from its matching segment landing page (§5); check the /vs/* pages’ query impressions weekly until they register (§0.2). 8. Measurement Current baseline (GSC export, trailing window ending 2026-08-10): 1,543 clicks / 74.1k impressions across the 1,000-URL page report; non-branded 197 clicks / 16.9k impressions / 1.16% CTR; docs 80 clicks / 20.9k impressions / 0.38% CTR; homepage 1,242 clicks (80% of the site) on 32.9k impressions. The four numbers to move, with the finding each one tests: Metric Now Tests “ai node” cluster weighted position 7.5 the entity page (§0.10 finding 2) Non-branded CTR, third-party-docs queries excluded 1.16% consolidation (findings 3, 4) /alternatives/* clicks 68 the redirect merge — this should absorb /vs’s 9 and grow Share of clicks not on / 20% whether any page other than the homepage can win a query Read non-branded CTR only with the 619 third-party-documentation impressions excluded (finding 8), and do not count /agents, /cloud or /creatives impressions as demand (finding 12). Re-audit when this section is more than a quarter old; the last two audits were six weeks apart and the category moved in both. Baseline as of 2026-07-02 Baseline (GSC export, trailing window ending 2026-07-02): 939 clicks / 48.2k impressions total; non-branded 115 clicks / 9.4k impressions / 1.23% CTR; US position 12.5; docs 75 clicks / 18.9k impressions. The numbers to move: non-branded clicks (proves §4’s content plan), US average position (proves §6’s distribution plan), and impressions on /vs/* and /use-cases/* (currently zero — proves the pages are entering the index at all). Track per page: organic impressions/clicks (Search Console), and which comparison/use-case pages convert to /studio or /pricing visits (the two highest-priority pages in the existing sitemap). Re-run the competitive scan in §2 quarterly — this is a fast-moving category and new entrants (Flora, Reflet.ai, both surfaced only in this audit’s research) can appear within months.--- # SerpAPI: every search engine as one discovered capability Source: https://docs.nodetool.ai/serpapi-integration.html SerpAPI: every search engine as one discovered capability SerpAPI is one HTTP endpoint. GET /search.json?engine=… answers for Google and its verticals (news, images, scholar, maps, jobs, flights, hotels, trends, lens, shopping, local), Bing, Baidu, DuckDuckGo, Yandex, Naver, Brave, YouTube, Amazon, eBay, Walmart, Home Depot, Yelp, TripAdvisor, the app stores, and about a hundred more — each with its own parameters. NodeTool exposes that endpoint as it is, rather than as a shelf of wrappers. There is no GoogleScholarSearch node and no walmart_search tool. There is serpapi_search, plus the two discovery calls an agent needs to find an engine it has not seen and read that engine’s contract before spending a search on it. Nothing here hard-codes the engine list. list_serpapi_engines and get_serpapi_engine_schema read SerpAPI’s own engine table — the same table that drives its playground — so an engine SerpAPI ships tomorrow is callable tomorrow, with no diff in this repo. Setting it up Get a key at https://serpapi.com/manage-api-key. Settings → Secrets → SERPAPI_API_KEY. This is the same secret the web_search capability’s SerpAPI backend reads, so an install that already has search configured needs nothing further. What an agent gets Five capabilities in the serpapi namespace. They are agent tools and sandbox imports at the same time — one spec behind both — so nothing can drift between what a model is told and what guest code can call. Capability Does Class list_serpapi_engines Every engine SerpAPI ships, and what each requires read get_serpapi_engine_schema One engine’s parameters, types, and allowed values read serpapi_search Run one search on any engine read get_serpapi_account Plan and searches left this month read get_serpapi_locations Canonical values for the location parameter read All five are read. A search spends a plan credit but changes nothing on the other side — the class web_search already sits in, and this is the same call with the engine chosen by the caller instead of by the install. The order that matters list → schema → search. It is in the tool descriptions because skipping it is expensive in a way that does not look like an error: SerpAPI ignores a parameter it does not recognize. A call that says query where the engine wants q does not fail; it succeeds, bills a search, and returns results for an empty query. So serpapi_search checks the parameter bag against the engine’s catalogued contract first, and an unknown key, a missing required one, or a value outside an enumerated set comes back as a message with no HTTP call behind it. From sandbox code import { get_serpapi_engine_schema, serpapi_search } from "@nodetool-ai/sandbox-nodetool/serpapi"; // Read the contract before constructing parameters for an unfamiliar engine. const schema = await get_serpapi_engine_schema({ engine: "google_scholar" }); const result = await serpapi_search({ engine: "google_scholar", params: { q: "attention is all you need", as_ylo: 2020 }, fields: ["organic_results"], max_items: 10 }); A response is a whole results page — google alone answers with organic results, ads, a knowledge graph, related questions, and inline images — so serpapi_search trims it. fields picks the top-level keys to return and max_items caps each result array. Nothing is dropped silently: available_keys lists everything the engine returned, omitted names the keys left out, and truncated counts the entries cut from each array. serpapi_search or web_search? web_search is one query against whichever SERP provider this install configured (SERP_PROVIDER: SerpAPI, DataForSEO, Brave, Apify, OpenAI, or Gemini), normalized to title/url/snippet — the two model-backed providers answer in prose and their citations are what gets normalized. Reach for it for a plain web question — it is provider-agnostic and its results are already in the shape an agent wants. serpapi_search is the layer under that, for the questions a plain web search cannot ask: a citation count, a maps listing’s hours, a product’s price history, a job posting’s apply link, a flight itinerary. It needs a SerpAPI key specifically, and it returns the engine’s own fields rather than a normalized subset. Where the key lives In the host, and nowhere else. Guest code calls serpapi_search; it never sees SERPAPI_API_KEY. The whole HTTP path is packages/agents/src/serpapi/client.ts, which takes the key as a private constructor field. api_key and output are host-owned. A caller that sets either is refused rather than obeyed, so the credential slot cannot be overwritten and the response cannot be switched to HTML this layer cannot parse. Errors are redacted before they are constructed. SerpAPI echoes the request URL in some failure bodies, and the key rides on the query string, so an error that reaches a model, a tool result, or a log line has the key replaced. How discovery works, and how it fails SerpAPI publishes no schema endpoint. Its playground page, however, ships every engine and every parameter — name, label, description, required flag, type, and allowed values — as one JSON blob in a data-react-props attribute. packages/agents/src/serpapi/catalog.ts reads that, flattens the documentation markup to text, and caches the result process-wide for six hours (one fetch is ~3 MB; contracts change on SerpAPI’s release schedule, not per call). That is a page, not a documented API, so the parse is defensive and the failure is named: a page whose markup changed comes back as catalog_unavailable with “the catalog parser needs updating”, not as an empty engine list that reads like SerpAPI ships nothing. packages/agents/tests/serpapi-catalog.test.ts runs the parser against a real playground response, trimmed to three engines and otherwise byte-for-byte as SerpAPI served it, so a shape change fails a test rather than a user’s run. Two parameters SerpAPI’s own table marks required are dropped from every engine’s caller-facing contract: api_key and engine. The first is the host’s, the second is the selector that chose the contract. Leaving them in would make every checked call demand the one field a caller must never set. Code What Where Capability specs (descriptions, schemas, classes) packages/agents/src/capabilities/serpapi.specs.ts Capability implementations packages/agents/src/capabilities/serpapi.ts Belt tools (chat runner, MCP bridge) packages/agents/src/tools/external-capability-tools.ts HTTP client — the only holder of the key packages/agents/src/serpapi/client.ts Engine catalog and its parser packages/agents/src/serpapi/catalog.ts Response trimming and parameter checking packages/agents/src/serpapi/normalize.ts Classified failures packages/agents/src/serpapi/errors.ts web_search’s SerpAPI backend (shares the client) packages/agents/src/tools/serp-providers/serpapi-provider.ts--- # Sketch Editor Source: https://docs.nodetool.ai/sketch-editor.html Draw, paint, mask, and generate AI imagery on a layered canvas — without leaving your workflow. Quick Access: Click the + button in the workspace tab bar and choose New sketch for a blank canvas, or open a sketch from the left Sketches panel. Overview The Sketch Editor is a layered raster editor with a built-in AI generation pipeline. It combines a familiar painting toolset (brush, eraser, fill, shapes, transform, selection) with the ability to generate image content directly onto a layer — bound either to a model or to one of your own workflows. Features: Layer stack with blend modes, per-layer opacity, lock, and visibility Painting tools: brush, pencil, eraser, fill, gradient, blur, clone stamp, color adjust Selection tools: rectangle, ellipse, lasso, polygon lasso, and magic wand, plus an AI Segment tool (SAM) that selects objects from point or box prompts Shape tools: line, rectangle, ellipse, arrow Crop and free transform (scale, rotate, skew, perspective, deform) Pen-pressure support, stroke stabilization, and drawing symmetry AI layers — generate a layer from a prompt or bind it to a workflow; regenerate when inputs change Undo/redo for the last 30 history steps Exports a flattened image, a mask, and per-layer outputs back into your workflow Where this fits The Sketch Editor is one of NodeTool’s editing surfaces, and it sits between manual editing and workflow automation. It opens an image asset, lets you paint or generate layers by hand or bind a layer to an image workflow, then renders the result back into an asset. That asset is the same material every other surface reads — drop it back on the canvas, drag it onto a timeline, or expose the workflow behind it as a Mini-App. Every surface shares one asset store and the same model/provider system. See Key Concepts → How everything fits together for the full loop. Opening the Editor The Sketch Editor runs in three places, all backed by the same document so your work stays in sync. From the new tab button Click the + button in the workspace tab bar (top of the editor). Choose New sketch. NodeTool creates a blank white image asset and opens it in a new tab in edit mode. To edit an existing image, open the asset in a tab and switch it to edit mode. The editor fills the tab. Use Save to image to render the composite back into the asset, then Done to return the tab to view mode. From the Sketches panel Open the Sketches panel in the left sidebar to browse documents you’ve created, grouped by date. Click one to open it, or use the New sketch button to start a blank canvas. As a standalone page Every sketch has its own URL at /sketch/<documentId>. Opening a sketch in its own workspace tab gives you the full-screen editor for focused work. Changes autosave as you go. The Workspace Region What it does Toolbar (left) Pick a tool; switch foreground/background colors Tool options (top) Settings for the active tool — size, opacity, hardness, shape type, transform handles Canvas (center) Draw, paint, and composite; zoom and pan Layers panel (right) Add, reorder, blend, lock, hide, and group layers; mark a mask layer Inspector (right) Per-layer details — including AI generation controls for generated layers Press Tab to hide the panels for a clean, full-canvas view. Tools Select tools from the toolbar or by keyboard shortcut. Move — V Reposition the active layer or selection contents. Brush — B Paint freehand with the foreground color. Supports round, soft, airbrush, and spray brush styles, plus pen-pressure sensitivity. A stabilizer option smooths shaky strokes. Size: [ decreases, ] increases Hardness: { decreases, } increases Opacity: press a digit 0–9 for a preset (5 = 50%, 0 = 100%) Pencil — P Paint hard-edged, aliased strokes. Eraser — E Erase pixels on the active layer. Shares size, hardness, and opacity controls with the brush. Fill — G Flood-fill contiguous regions with the foreground color, using an adjustable tolerance. Gradient — T Drag to draw a gradient between the foreground and background colors. Blur — Q Paint to soften pixels under the brush. Clone Stamp — S Alt-click to set a source point, then paint to copy detail from there — useful for retouching and removing blemishes. Color Adjust — J Paint hue, saturation, and brightness adjustments onto the layer. Eyedropper — I Click the canvas to sample a color into the foreground swatch. Crop — C Reframe the canvas. Drag the crop box, then press Enter to commit or Esc to cancel. Transform — F (or Ctrl/⌘ + T) Free-transform the active layer or selection with a handle box. The Mode toggle in the tool options picks how corner drags behave: Mode Behavior Move/Scale Scale and rotate the box. Ctrl/⌘ on a side handle shears; Ctrl/⌘ + Alt + Shift promotes the drag to Perspective. Perspective Four-point projective transform — drag a corner and the opposite edge mirrors via axis projection. Deform Free four-point quad — only the corner you drag moves, the other three stay put. Shift locks the drag to one axis. Perspective and Deform produce a four-point quad, so a layer carrying one is transformed on its own: a multi-layer union applies a single affine delta and cannot carry a quad. Enter commits, Esc cancels, . resets the box to identity. Ctrl/⌘ + Shift + T repeats the last transform. Selection Tools Tool Shortcut Behavior Select (rectangle, ellipse, lasso, polygon lasso) M Pick the mode in the tool options, then drag or click points to select Magic wand W Click to select a region by color. Tolerance, contiguous, and sample-all-layers options sit in the tool options Segment none AI object selection (SAM). Click to add a positive point, Alt-click for a negative point, or drag a box Once a selection is active, paint and edit operations are constrained to it. Ctrl/⌘ + D deselects. Shape Tools — U Draw vector-style shapes onto a layer. Shape Shortcut Line L Rectangle R Ellipse O Arrow A Hold Shift while dragging to constrain proportions (squares and circles) or to snap lines and arrows to 30° steps. Hold Alt to draw rectangles and ellipses from the center. Layers The editor is layer-based. Stack layers, reorder them, toggle visibility, lock them, group them, and composite them with blend modes. Blend modes — choose how a layer combines with those beneath it (Normal, Multiply, Screen, Overlay, Darken, Lighten, Color Dodge, Color Burn, Hard Light, Soft Light, Difference, Exclusion, and Add). In the Layers panel, ↑/↓ step through blend modes. Opacity — set per-layer transparency. Lock — protect a layer from edits. Mask layer — designate a layer as the document’s mask. It’s exported as a separate mask output for inpainting and compositing nodes. Layer order — Ctrl/⌘ + ] moves the active layer forward and Ctrl/⌘ + [ sends it back. Layer via copy / cut — Ctrl/⌘ + J copies the current selection to a new layer; Ctrl/⌘ + Shift + J cuts it to a new layer. Clear layer — Delete or Backspace clears the active layer (or selection). Fill layer — Ctrl/⌘ + Backspace fills with the background color; Alt + Backspace fills with the foreground color. AI Generation What sets the Sketch Editor apart is that a layer can be generated, not just painted. Select a layer and open the Inspector to bind it to a generator: Prompt-based generation — pick a provider and model, write a prompt, and generate text-to-image or image-to-image results directly onto the layer. Image-to-image takes a source layer plus edit strength and inference steps. The Inspector shows an estimated cost before you generate. Workflow-bound generation — click Generate Layer in the Inspector toolbar and pick one of your workflows that has an image output node (ImageOutput, MaskOutput, or Output). If it has several, choose which output node feeds the layer. The layer becomes a live surface for any pipeline you’ve built. Each generation is tracked as a version on the layer, so you can compare results and roll back. NodeTool also tracks each layer’s inputs: if you change a prompt, parameter, or an upstream layer the generation depends on, the layer is flagged stale so you know it’s worth regenerating. Re-generate Stale in the Inspector toolbar runs all stale layers in dependency order and skips locked layers. Generation runs as a job over the WebSocket connection, with live progress shown on the layer. The editor also has an assistant chat panel. It receives the active layers and can generate images or edit layers from a description. This makes the editor a natural place to sketch a rough composition, mask a region, and let a model fill it in — then keep painting on top. Colors The editor maintains a foreground and background color. X swaps the foreground and background colors. D resets them to the defaults (black / white). Ctrl/⌘ + I inverts the colors of the active layer. Symmetry & Pen Pressure Symmetry — mirror your strokes in horizontal, vertical, dual, radial, or mandala modes. Radial and mandala modes take 2 to 12 rays (6 by default). Pen pressure — when using a pressure-sensitive device, choose whether pressure affects size, opacity, or both, and tune the light-press scale and curve in the pen-pressure settings. Stroke assist — the brush, pencil, and eraser settings offer a rolling-average stabilizer and a drag algorithm for cleaner linework. History & Undo The editor keeps the last 30 history steps. Older steps are merged into the baseline. Action Shortcut Undo Ctrl/⌘ + Z Redo Ctrl/⌘ + Shift + Z (or Ctrl/⌘ + Y) History persists while the editor is open. Zoom & Navigation Action Shortcut Zoom in + / = or scroll up Zoom out - or scroll down Reset zoom (fit) Ctrl/⌘ + 0 Zoom to 100% Ctrl/⌘ + 1 Pan Space + drag, or middle-click drag Toggle panels Tab Saving & Exporting Autosave — sketches save automatically as you work. Save to image — render the flattened composite back into the backing image asset, ready to use anywhere an image is. Workflow nodes — to pull a sketch into a graph, use the Render Sketch and Sketch Layers nodes, which flatten the document or expose its individual layers. Because the document is shared across the tab, the Sketches panel, and the standalone page, your edits stay in sync everywhere it’s open. Keyboard Shortcuts Tools Key Tool V Move B Brush P Pencil E Eraser G Fill T Gradient Q Blur S Clone stamp J Color adjust I Eyedropper M Select (rectangle, ellipse, lasso, polygon lasso) W Magic wand select C Crop F Transform U Shape L Line R Rectangle O Ellipse A Arrow Edit & Selection Shortcut Action Ctrl/⌘ + Z Undo Ctrl/⌘ + Shift + Z (or Ctrl/⌘ + Y) Redo Ctrl/⌘ + C / X / V Copy / cut / paste Ctrl/⌘ + Shift + V Paste masked Ctrl/⌘ + A Select all Ctrl/⌘ + D Deselect Ctrl/⌘ + Shift + D Reselect Ctrl/⌘ + Shift + I Invert selection Ctrl/⌘ + T Free transform Ctrl/⌘ + Shift + T Repeat last transform Ctrl/⌘ + Shift + Alt + T Repeat last transform on a copy Ctrl/⌘ + ] / [ Move layer forward / back Ctrl/⌘ + J Layer via copy Ctrl/⌘ + Shift + J Layer via cut Ctrl/⌘ + I Invert colors Delete / Backspace Clear layer/selection Ctrl/⌘ + Backspace Fill with background color Alt + Backspace Fill with foreground color ↑ ↓ ← → Nudge selection/layer by 1px Enter Commit transform / crop Esc Cancel / deselect Colors & Brush Shortcut Action X Swap foreground/background colors D Reset colors to default [ / ] Decrease / increase brush size { / } Decrease / increase hardness 0–9 Set tool opacity preset (0 = 100%) Navigation Shortcut Action + / = Zoom in - Zoom out Ctrl/⌘ + 0 Reset zoom (fit) Ctrl/⌘ + 1 Zoom to 100% Tab Toggle panels Space + drag Pan canvas Common Workflows Sketch-and-generate Open a blank canvas from + New → New image. Block in a rough composition with the brush on one layer. Add a new layer, select the region you want to generate, and bind the layer to a model or workflow in the Inspector. Write a prompt and Generate. Keep painting on top, then choose Save to image and Done — the flattened result flows downstream. Masking for inpaint Open or paste an image into a sketch. Add a layer, paint over the area to change, and mark it as the mask layer. Wire the node’s image and mask outputs into an inpaint node. Quick retouch Open the image in the editor. Select the Clone Stamp (S), Alt-click clean pixels, and paint over the blemish. Adjust the layer opacity to blend, then save. Related Nodes The sketch type is also available as workflow nodes: Create Sketch — create a sketch document from an input image, placed on the base layer, ready to edit. Render Sketch — flatten layers, applying blend modes and opacity, to a single image. Outputs image and, when a mask layer is set, mask. Sketch Layers — expose each visible layer as a separate image (layers) with its name (names). Related Features Asset Management — organize and reuse generated images Workflow Editor — main editor documentation Game Editor — use finished images as sprites and backgrounds in a 2D game CLI: nodetool sketch — validate a sketch, replay a scripted edit session headlessly, and manage saved versions Last updated: June 2026--- # Skills Source: https://docs.nodetool.ai/skills.html A skill is a saved set of instructions for one kind of work. It has a name, a one-line description of when it applies, and a Markdown body. The agent reads the name and description on every turn. It reads the body only when it needs it, or when you name the skill with / in chat. Quick Access: Click + in the workspace tab bar and choose New skill, or open the Skills panel from the left panel’s More menu. In chat, type / to pick a skill. Two kinds of skill   User skill System skill Who wrote it You, or the agent for you NodeTool ships it Where it lives Your database A SKILL.md file in the install Editable Yes No. It opens read-only Shown in the panel as A skill with a sparkle icon A skill with a lock icon and the label Built in Available to the agent Yes Yes, on every install System skills cannot be renamed, edited, or deleted from the app or by the agent. Their names are reserved: you cannot create a skill with the name of a system skill. A user skill that already had such a name before the system skill shipped keeps working and wins over the system one. Create and edit a skill Click + in the workspace tab bar and choose New skill. A new skill opens in a tab with a generated name such as skill-m3x9k2, a placeholder description, and a short starter body. The + button at the top of the Skills panel does the same. Set the Name. Use lowercase letters, digits, and hyphens, up to 64 characters. The name cannot contain anthropic or claude. Set the Description. Write it as when to use the skill, for example “Use when writing release notes from a changelog.” It is the only part the agent always sees. It can be up to 1024 characters and cannot contain markup such as <tag>. Write the instructions in the Markdown editor below the two fields. The editor saves on its own about three-quarters of a second after you stop typing. Save saves immediately, and a dot beside it shows unsaved changes. The tab title follows the skill name. In the Skills panel, right-click a skill for Rename and Delete, and for Duplicate, which creates a copy named <name>-copy. Built-in skills open as a read-only view and have no rename or delete action. You can also ask the agent to do this in chat, for example “Save these steps as a skill called weekly-report.” See How the agent finds and loads skills below. What makes a good skill Write the description as a trigger. “Use when the user asks for a changelog summary” lets the agent decide from one line whether to load the skill. Keep one skill to one kind of work. Several narrow skills load faster and conflict less than one large one. Put the steps, defaults, and checks in the body. The agent follows the body as instructions for the turn. The SKILL.md format System skills are files. Each lives in its own directory, packages/system-skills/<name>/SKILL.md, and the directory name is the skill name. A SKILL.md has front matter between two --- lines, then the body. --- name: release-notes description: "Use when writing release notes from a changelog or a list of merged changes." featured: true --- # Release notes 1. Group the changes by user-visible area. 2. Lead each group with the change people will notice. Front matter fields Field Required Meaning name Yes The skill name. Same rules as above, and it must equal the directory name or the skill is skipped. description Yes One line saying when the skill applies. Up to 1024 characters, no <tag> markup. featured No true lists a system skill in the agent’s always-on catalog. Any other value, or no field, leaves it out. The parser reads key: value lines, not full YAML. Keep each value on one line. A value wrapped in a matching pair of single or double quotes has the quotes removed. Other front matter keys are ignored. A SKILL.md is skipped, without an error, when any of these hold: The first line is not ---, or the closing --- line is missing. name or description is missing, invalid, or empty. The body after the front matter is empty. name differs from the directory name. The body is everything after the closing ---. It is plain Markdown. load_skill returns it in one piece, so a skill is a single SKILL.md and nothing beside it is read. A user skill has the same three parts, but you edit them in the app as the Name and Description fields and the Markdown editor, so you never write front matter by hand. Where NodeTool looks for system skills On startup the agent loads system skills from the first of these that exists: The directory in NODETOOL_SYSTEM_SKILLS_DIR, if you set it. See Configuration. A _skills directory beside the bundled server, which is how the desktop app and the Docker image ship them. packages/system-skills in a source checkout. How the agent finds and loads skills Every chat turn carries a catalog of skills: the name and description of each of your own skills, plus the system skills marked featured: true. The remaining system skills are counted but not listed. That keeps the prompt short. The agent finds them by searching. The agent reaches skills through five tools. Tool What it does list_skills Lists skills with name, description, and whether each is a system skill. An optional query matches word by word against name and description, and returns the best matches first. load_skill Returns one skill’s full instructions by name. The agent calls it before acting on a skill the catalog only summarizes. create_skill Creates a user skill from name, description, and content. All three are required. update_skill Changes a user skill’s description, content, or name (through new_name). Only the fields passed change. delete_skill Deletes a user skill permanently. The agent should ask first unless you asked for the deletion. Both list_skills and load_skill read system and user skills. The three authoring tools refuse a system skill’s name. Names are matched in lowercase and a leading slash is ignored, so /Release-Notes and release-notes are the same skill. The / slash command in chat Type / at the start of a word in the chat composer to open a skill list. The list includes your skills and the built-in ones, and it filters by name and description as you type. Pick one and the composer inserts /<skill-name> into your message. When a message names a skill this way, the agent receives that skill’s full instructions with the turn and follows them, with no load_skill round trip. A / inside a word, such as the path src/utils, does not count. The same picker appears in the prompt box on New project, where the chosen skill becomes the starter for the project. See Global Chat and User Interface. Shipped skills The list below comes from the directories in packages/system-skills. Each skill is available to the agent on every install, and you can open any of them read-only from the Skills panel. Type / and the name to use one in chat. Video production Skill Use it for storyboard-core Storyboards, entity casting, rendering, and timeline assembly, including the tool contracts the other video skills quote. product-commercial A polished product film or brand commercial. ugc-video Phone-shot creator videos, UGC ads, testimonials, and vlogs. script-video A video whose timing follows written voiceover, such as an explainer or narrated b-roll. short-film A narrative scene, short film, or trailer with characters, dialogue, and score. video-clone Analyzing a reference video and recreating its shot structure with a new product or cast. video-workflow A reusable video workflow for batches or repeated inputs. launch-kit A campaign with shared product and cast entities, stills, and several video cuts. launch-commercial A launch commercial built from a product page URL, delivered as an editable project with its real cost. motion-ad A short sound-off motion-graphics ad from a product’s real screenshots and photos. commercial-beat-sheet A precisely timed commercial beat sheet stored as a storyboard. explainer-storyboard A 30 to 120 second explainer storyboard from a confirmed brief. music-video-treatment A beat-synced music-video treatment and storyboard from track metadata. trailer-template A trailer or teaser structured as eight audio-first beats. video-audio-continuity Keeping sound continuous across a multi-scene piece cut from generated video. nodetool-video-post Repairing or regenerating footage already on a timeline: cutouts, upscaling, outpainting, lip sync, and more. Motion and timeline craft Skill Use it for motion-graphics The entry point for building and reviewing a timeline piece. timeline-edit-ops Reference for editing an existing timeline with edit_timeline ops. motion-direction Setting one motion language for a piece and auditing a timeline against it. motion-principles Timing, easing, stagger, anticipation, and follow-through. motion-curves Animation curves written by hand or baked from a JavaScript body. frame-composition Grids, focal placement, safe areas, depth, and camera moves. beat-sync-editing Cutting to music and shaping pacing. color-motion Colour, gradients, grades, and 3D LUTs. caption-titles Titles, lower-thirds, captions, and end cards. logo-reveal Animating a brand mark for an intro sting or end card. motion-background Ambient backdrops and quiet loops. Image, 3D, and games Skill Use it for nodetool-sketch Building and editing sketches: layers, blend modes, placed images, version history. nodetool-3d-scene Building, editing, validating, and rendering 3D models. native-game Building or revising a playable game in NodeTool’s built-in engine and preparing a web build. game-direction Setting game feel, level pacing, and art direction before building. Workflows, apps, and code Skill Use it for nodetool-workflow-builder Creating and editing workflow graphs, connections, and properties. nodetool-app-builder Building or repairing a mini app. nodetool-js-scripting JavaScript for the QuickJS sandbox: Code node bodies and saved JS scripts. nodetool-custom-node-developer New TypeScript node types and node packages. nodetool-rag-indexing Document ingestion, vector indexing, and retrieval pipelines. nodetool-browser-agent Browser automation agents for navigation, extraction, and forms. nodetool-troubleshooter Diagnosing a failing run on any surface, including stuck media generations. nodetool-skill-author Writing or revising a skill, yours or a shipped one. Model prompting guides Each guide covers how one model line reads a prompt: the structure it expects, the controls that matter, and common failures. NodeTool points the agent to the matching guide when it picks a model. Skill Model line nano-banana-pro-prompting Google Nano Banana Pro (image) gpt-image-2-prompting OpenAI GPT Image 2 (image) flux-2-klein-prompting Black Forest Labs FLUX.2 [klein] (image) seedream-prompting ByteDance Seedream 4 and 5 (image) qwen-image-prompting Alibaba Qwen-Image (image, typography) veo-3-prompting Google Veo 3 (video) seedance-2-prompting ByteDance Seedance 2 (video) kling-video-prompting Kuaishou Kling 2.5-turbo and later (video) hailuo-prompting MiniMax Hailuo (video) minimax-h3-prompting MiniMax H3 (video) wan-2-6-prompting Alibaba Wan 2.6 (video) elevenlabs-audio-prompting ElevenLabs speech, dialogue, sound effects, and music stable-audio-prompting Stability Stable Audio (music and sound effects) API and integration Skill Use it for nodetool-api-reference Integrating over REST, tRPC, WebSocket, MCP, or the OpenAI-compatible chat API. nodetool-model-provider-config Providers, credentials, local models, and model selection. nodetool-chat-cli Chat CLI sessions, provider selection, and Global Chat features. The api-* skills document the nodetool object model that the agent calls from code actions. The agent loads one before its first call into that area. Skill Covers api-workflows nodetool.workflows, jobs, nodes api-models nodetool.models api-media nodetool.media, generations api-assets nodetool.assets, documents api-web nodetool.web, email api-memory nodetool.memory, shared, threads api-agents nodetool.agents api-settings nodetool.settings, secrets api-collections nodetool.collections api-apps nodetool.apps api-timelines nodetool.timelines api-sketches nodetool.sketches api-scripts nodetool.scripts api-storyboards nodetool.storyboards api-games nodetool.games Operations Skill Use it for nodetool-deployment Deploying and operating servers and workers with Docker, SSH, Runpod, or cloud configuration. Related pages Global Chat for the composer and slash commands. Editor Panels for the More menu that holds Skills. User Interface for starting a project from a skill. Configuration for NODETOOL_SYSTEM_SKILLS_DIR.--- # Storage Guide Source: https://docs.nodetool.ai/storage.html NodeTool stores user assets, workflow artifacts, and temporary files through pluggable backends defined in @nodetool-ai/storage (packages/storage/src/). The active backend is selected per execution by the runtime context. Asset Storage Backends Backend Module When it is used Notes In-memory @nodetool-ai/storage / src/memory-storage-adapter.ts Tests Keeps data in process-local dictionaries. Not selectable through configuration. Local filesystem @nodetool-ai/storage / src/file-storage-adapter.ts Default (NODETOOL_STORAGE_BACKEND unset or file) Stores assets under getDefaultAssetsPath() (@nodetool-ai/config / src/paths.ts), which defaults to $XDG_DATA_HOME/nodetool/assets, falling back to ~/.local/share/nodetool/assets (%APPDATA%\nodetool\assets on Windows). Override with ASSET_FOLDER or STORAGE_PATH. URLs are served via the API (/api/storage/*). Supabase Storage @nodetool-ai/storage / src/supabase-storage-adapter.ts NODETOOL_STORAGE_BACKEND=supabase Uses a Supabase bucket for asset storage. Amazon S3 / S3-compatible @nodetool-ai/storage / src/s3-storage-adapter.ts NODETOOL_STORAGE_BACKEND=s3 Optional custom endpoint for MinIO/Wasabi. The backend is chosen once by NODETOOL_STORAGE_BACKEND (loadAssetStorageConfig() / loadTempStorageConfig() in @nodetool-ai/config / src/storage-config.ts); assets and temp files always use the same backend, with different buckets. Required Environment Variables Variable Description NODETOOL_STORAGE_BACKEND file, s3, or supabase. Defaults to file. ASSET_BUCKET Bucket name for permanent assets (S3 and Supabase). Required for both. TEMP_BUCKET Bucket name for ephemeral workflow outputs (S3 and Supabase). Required for both. SUPABASE_URL, SUPABASE_KEY Project URL and API key. Required when the backend is supabase. NODETOOL_MAX_UPLOAD_BYTES Largest single upload any backend accepts, in bytes. Defaults to 1 GiB (2 GiB for local file reads and uploads). See src/storage-limits.ts. NODETOOL_EXTERNAL_ASSET_THRESHOLD_BYTES Local file backend only. See External assets. FONT_PATH, VECTORSTORE_DB_PATH Additional paths for specific nodes (defined in @nodetool-ai/config / src/setting-catalog.ts). For S3-compatible services, set: S3_ENDPOINT (or S3_ENDPOINT_URL): custom endpoint, optional. When set, requests use path-style addressing. S3_REGION: defaults to us-east-1 when building signed URLs. Credentials come from the standard AWS chain (@nodetool-ai/storage / src/s3/credentials.ts): AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN, or a profile in ~/.aws/credentials selected with AWS_PROFILE. Metadata-service chains (ECS, EC2 IMDS, EKS web identity) are not built in — pass a custom credentialProvider to S3Client for those. Temporary Storage Temp storage returns a location for scratch files, mirroring asset storage. It uses the same backend as assets (getTempAdapter() in @nodetool-ai/websocket / src/lib/storage.ts), with TEMP_BUCKET in place of ASSET_BUCKET so temp files can carry different retention and access policies. On the file backend both share the local assets directory. Supabase Storage With NODETOOL_STORAGE_BACKEND=supabase, NodeTool uses Supabase for asset and temp storage. Adapter: SupabaseStorageAdapter (@nodetool-ai/storage / src/supabase-storage-adapter.ts) Selection: createStorageAdapter() (@nodetool-ai/storage / src/factory.ts) from the config the backend env var picks URLs: createAssetUrlBuilder() (@nodetool-ai/storage / src/url-builder.ts) returns a Supabase signed URL valid for SIGNED_URL_TTL (7 days, from @nodetool-ai/config), so private buckets work as-is. S3 gets a pre-signed GET URL with the same lifetime. Minimum configuration: NODETOOL_STORAGE_BACKEND=supabase SUPABASE_URL=https://your-project.supabase.co SUPABASE_KEY=your-service-role-key ASSET_BUCKET=assets TEMP_BUCKET=assets-temp Recommendations: Create the buckets (assets, assets-temp) in the Supabase dashboard. Scope the service role key to server-side environments only. Do not expose it to browsers. Use the temporary storage for intermediate files that do not need long-term retention. External assets On a local desktop or dev server with the file backend, an import at or above NODETOOL_EXTERNAL_ASSET_THRESHOLD_BYTES (default 1 GiB) is not copied. The asset row stores external_path and the file stays where it is (externalAssetsAvailable() in @nodetool-ai/websocket / src/lib/external-assets.ts). The server records the file’s size and modification time. If the file is missing or either value changed, the asset reads as offline until you relink it. NODETOOL_ENV=production and the S3 and Supabase backends never create external assets. Database History Retention Run history, workflow versions, and prediction payloads live in the database, not in asset storage. A per-user retention policy trims them (@nodetool-ai/websocket / src/storage-retention.ts, defaults in @nodetool-ai/models / src/storage-maintenance.ts). The policy is edited under Settings in the storage history section (StorageHistorySettings). Setting Default Autosaves kept per workflow 10 Autosave retention 7 days Manual version retention 90 days Finished job retention 30 days Run event retention 30 days Prediction request payload retention 400 days Automatic cleanup Off locally. On when NODETOOL_STORAGE_AUTO_CLEANUP=1. With automatic cleanup on, the sweep runs at most once every 24 hours. Node and Workflow Caches NodeTool caches node outputs and resolved asset URIs for repeated executions: @nodetool-ai/storage / src/memory-node-cache.ts – in-process dictionary (MemoryNodeCache). @nodetool-ai/storage / src/memory-uri-cache.ts – caches resolved asset URIs for quick lookup. ProcessingContext builds node cache keys with generateNodeCacheKey() (@nodetool-ai/runtime / src/context.ts). Accessing Storage in Workflows Workflows interact with storage through ProcessingContext (@nodetool-ai/runtime / src/context.ts): resolveAssetBytes(assetId) – loads the raw bytes for an asset. assetsToStorageUrl(value) – rewrites asset references in a value to storage URLs. downloadFile(url) – fetches file contents from a URL. assetsToWorkspaceFiles(value) – writes asset values to the assets/ folder of the run workspace. uploadAssetsToTemp(value) – uploads asset values to temporary storage and returns temp URLs. Deployment Considerations For self-hosted deployments, mount persistent volumes for /workspace (workspace files) and the asset storage directory. In Docker-based execution (@nodetool-ai/deploy / src/docker-run.ts), the workspace path is mounted into containers, so ensure the host directory exists and has the correct permissions. When using S3, grant read/write access to the buckets named by ASSET_BUCKET and TEMP_BUCKET. Troubleshooting Asset bytes fail to load in a run – the context falls back to GET <NODETOOL_API_URL>/api/storage/<id>.<ext> when it cannot read an asset directly. NODETOOL_API_URL defaults to http://localhost:7777, so set it when the API runs elsewhere. Upload rejected as too large – raise NODETOOL_MAX_UPLOAD_BYTES. S3 authentication errors – verify credentials and endpoint configuration; run nodetool settings show for the resolved environment and nodetool secrets list for the stored keys. Local file permissions – ensure the configured asset folder is writable by the user running the service (especially in Docker). Docker jobs cannot access assets – mount the asset directory into the server container and ensure the assets path (getDefaultAssetsPath(), overridable via ASSET_FOLDER/STORAGE_PATH) points to the mounted path. Related Documentation Configuration Guide – environment variable hierarchy and secret management. Deployment Guide – configuring storage via deployment.yaml. Docker Resource Management – limiting resource usage for storage-heavy jobs.--- # Storyboard Editor Source: https://docs.nodetool.ai/storyboard-editor Plan a film as a board of shots. Each shot has a prompt, a camera setup, and the characters and locations it shows. Render a still, then a clip, compare takes, and send the shots you keep to a timeline. Quick access: Click the + button in the workspace tab bar, choose New storyboard…, then pick Blank storyboard or an example. You can also start from the Storyboard guided flow or ask the agent to create one. Overview The Storyboard opens as a workspace tab. The board is on the left. The assistant panel opens beside it on request. A queue overlay appears over the board while renders run. Area What it holds Header Board title, genre chip, shot summary, card size slider, Slideshow, and the board tools Board settings Screenplay fields, direction settings, and the Direct button Banners A stale-version notice and a Retry N failed button, shown only when they apply Shot grid Scene headers with shot cards under them Footer Open script or Extract script, and Create timeline The board is the working copy of a film. Stills, clips, and takes stay in your asset library, so closing the tab loses nothing. Create a storyboard Way in Steps New menu + in the tab bar, then New storyboard…, then Blank storyboard Examples Same menu, then pick an example. Each one lists its shot count and says “stills included” Command palette New storyboard Guided flow Choose Storyboard on the new-project screen. The flow reviews the screenplay, entities, and look before it opens the board. See Guided Flows Agent Ask the creative agent or the board’s assistant to create one A blank board is named “Untitled storyboard”. The left panel also has a New storyboard button. Direct a screenplay Open Board settings to write the brief. Press Direct and the screenplay model turns it into scenes and shots. A board with shots shows Re-direct instead, which asks for confirmation because it replaces every shot. Generated stills and clips stay in your asset library. Field Use Title Shown on the board and the tab. Placeholder: “Untitled film” Brief The film in one or two sentences Style Palette, light, lens, texture Entities Characters, locations, styles, and props that apply to the whole board Screenplay model The language model that writes the screenplay Aspect ratio 16:9, 9:16, 1:1, 4:3, or 21:9 Shots How many shots Direct writes You can skip directing. Add shot in the header adds one shot at a time, and the + that appears between two cards inserts a shot after the first. The board Shots sit under scene headers labeled Scene 1, Scene 2, and so on, with the scene’s slugline beside the label. Each card shows the shot’s still or clip, its caption, and a status pill. Status pill Meaning planned No still or clip yet still A still exists still · clip queued A still exists and its clip is queued or running rendering still / rendering clip A render is running covered Another shot’s generation already produced this shot’s picture failed The last render failed stale Added to any of the above when the shot’s inputs changed after the render Card size: The Size slider sets the minimum card width. It is hidden on phones, where one column fills the width. Reorder: Drag a card onto another. Dropping across a scene header moves the shot to that scene. Hover actions: A card shows Edit, Iterate, a regenerate button, and an upload button for your own still. Double-click the media to view it fullscreen. Shot actions menu: Render still…, Render clip…, Download, Send to workflow…, Duplicate shot, Delete shot. Undo and redo: The undo buttons in the header cover board edits. The Undo tooltip reads “Undo (⌘Z)”. Genre: The chip beside the title shows the genre. Set genre appears when none is set. Select a card to open the shot inspector under the grid. It repeats Edit, Iterate, Regenerate, and Delete, and its Appears in row links to the timeline cut and the script lines that use the shot. Edit a shot Edit opens the shot editor over the board. Form fields are on the left, the viewer and tools are on the right. Previous shot and Next shot step through the board. Save stays disabled until you change something, and closing with unsaved edits asks “Discard changes?”. Field Notes Description “What the shot shows” Dialogue “What is said in shot”. Edit it in place, or jump to the linked script In this shot The entities that appear. Pick a character, location, style, or prop to keep it consistent. See Entities Size, Focal length, Perspective, Movement, Equipment Camera setup, chosen from lists Duration (seconds) The length of the clip. With a linked script, the chip reads From voice takes or Set by hand. Click it to switch between the two. The shot card shows the duration next to the shot number Shot title Optional, placeholder “Untitled shot” Render mode Keyframe (still first, then animate it), Direct (clip straight from the prompt), or Reference Notes Text the render should not read as direction Composition mode, graphic direction None, Overlay, Graphics first, or Hybrid for on-screen text and graphics Title, render mode, notes, and graphics sit under Advanced. Prompts shows the still prompt and clip prompt the render will send, so you can check them before you spend anything. The Regenerate button renders a new still. The More shot actions menu holds Render clip (or Re-render clip), Move earlier in scene, and Move later in scene. A clip in keyframe mode needs a still first. Render stills and clips Render one shot or the whole board. Action Where What happens Render stills Header Opens a dialog to pick the Still model, then renders every shot without a still Render clips Header Opens a dialog with one model picker per kind of clip in the batch Render still… / Render clip… Shot actions menu Same dialog for one shot Regenerate Card, inspector, editor Re-renders the still with the model the shot used last Iterate Card, inspector Asks for a note such as “make it darker, add rain” and re-renders the current clip with it The shot remembers the model you picked, so later renders skip the dialog choice. If an entity is attached and the chosen still model cannot take reference images, a warning appears under the model picker. The header’s render buttons show a price in the dialog and on the confirm button, for example “Render stills · ~$0.40”. Text under the picker explains any request that could not be priced, and Select a priced model to see the estimate. appears when no priced model is selected. After you choose stills, the dialog also shows what rendering the clips for the same shots would add. For how provider charges work, see Costs and Credits. The queue overlay lists running renders with progress. You can expand or collapse it. Stop tracking hides local progress only, and the provider may still finish the request and charge for it. Takes Every render adds a take. A new still becomes the current still, and the earlier stills stay as takes. A new clip does not replace your current clip until you choose. Open the shot editor. The takes list has a View takes button that expands the stills and clips. Pick a Preview chip to play a clip take without changing the current one. Choose Set as current clip. For stills, click the still thumbnail to use it as the current still. The current take is marked Take N · Current. Use the remove buttons to delete a take you do not want. The current still is what Regenerate edits and what the timeline uses. If you have rendered a clip but not chosen a take, Create timeline asks you to set a clip take as current. Change a still In the shot editor, the Change still tab beside Takes edits the current still. Each result is saved as a new take and becomes the current still. Tab What it does Edit Describe a change in What to change, for example “Make her scarf deep red”. Choose an Edit model and press Generate edit Upscale Pick an Upscale model and a scale of 2× or 4× Reframe Pick a New aspect ratio and a Reframe model. Optionally describe what fills the new space Adjust Brightness, Contrast, and Saturation sliders, then Save as new take. Reset clears them. It runs in your browser and calls no provider To change only part of the image, press Paint area on the Edit tab. The Paint the area to change dialog has Brush and Eraser tools, a Brush size slider, and Clear. Press Use this area to keep the mask, and the toggle Only the painted area changes controls whether the mask applies. With cast on the shot, Keep (names) on model sends their reference images with the edit. Some models cannot paint over an area and some only can. The panel tells you which when the chosen model does not match. Style Press the board actions button (More board actions) and choose Change style…. Pick an Art style and it applies to every shot. Rendered stills and clips are marked stale and nothing re-renders until you ask. When versions are stale, a banner reads for example “Inputs changed (style). 3 stills and 1 clip are stale.” with Regenerate stale stills and Regenerate stale clips buttons. A confirmation shows the estimated cost. Each shot renders a new take with the model it last used, and the current take stays selected until you choose the new one. The same menu offers Download ZIP and Send to workflow…, which passes the board’s stills and clips to a workflow. Retry failed shots When a render fails, Retry N failed appears under the header. It retries the latest failed request for each shot and kind. The queue overlay has the same button. Slideshow and preview Slideshow opens a fullscreen player. It shows each shot’s clip or still with its caption. Use the left and right arrow keys or Previous shot and Next shot. Autoplay advances for you, and Edit shot jumps to the editor. Unrendered shots read “Not rendered yet”. Preview plays the board’s shots in sequence inside the board. It is disabled until a shot has a still or clip. Link a script Dialogue and narration can live in a script. Press Extract script under the grid to project the board’s lines into a new script. The button then reads Open script. A linked board keeps shot timing tied to the voiced takes, and warnings appear if a script line is claimed twice or no longer exists. The shot inspector links to the script lines for each shot. Assemble a timeline Create timeline builds a timeline from the current stills and clips. It needs at least one shot with a still or a current clip, and skips shots without either. With a linked script, each shot runs as long as the takes it covers and each voiced line gets a voiceover clip. Once a timeline exists, the button reads Rebuild linked timeline…. The confirmation says how many storyboard-owned clips it replaces, including trims you made on them. Tracks and clips you added yourself stay. You can also run Assemble Timeline from the command palette. Assistant The Assistant button opens the Storyboard Assistant beside the board, and Hide Assistant or Show Assistant in the command palette toggles it. It sees the board and the selected shot. Try “break this brief into six shots”, “add a close-up after shot 3”, or “assemble the shots into a timeline”. On phones, Board and Assistant are separate tabs. The agent tools for storyboards are list_storyboards, create_storyboard, get_storyboard, render_storyboard_stills, render_storyboard_clips, revise_storyboard_clip, assemble_storyboard_timeline, edit_storyboard, extract_script_from_storyboard, and delete_storyboard. Related pages AI Video Production covers the guided Storyboard flow and how takes fit into a full production. Entities covers the characters, locations, and styles shots reuse. Templates Gallery lists starting points.--- # Supabase Deployment Integration Source: https://docs.nodetool.ai/supabase-deployment.html Supabase provides both authentication and object storage for deployed NodeTool instances. This guide covers how to configure Supabase as your auth and storage backend. What Supabase Provides Feature What It Does Authentication User sign-up, login, and JWT-based session management. The server checks each token against your project’s /auth/v1/user endpoint Object Storage Asset storage in Supabase Storage buckets, with public or signed URLs NodeTool talks to Supabase with the service role key, which bypasses Row Level Security. RLS policies do not restrict what the NodeTool server can read or write. Prerequisites A Supabase project at supabase.com Your project’s URL, service role key, and anon key from the Supabase dashboard A NodeTool deployment target (self-hosted Docker server) Setup 1. Create Storage Buckets In your Supabase dashboard, go to Storage and create the following buckets: Bucket Purpose Visibility assets Permanent workflow assets (images, documents, audio) Private or Public assets-temp Temporary files during workflow execution Private Public vs. Private buckets: Public buckets generate direct URLs that anyone can access – suitable for assets shared externally Private buckets require signed URLs or authenticated access – better for sensitive content 2. Configure Environment Variables Add these variables to your deployment target’s container.environment section: container: environment: # Supabase connection (presence of both enables Supabase auth) SUPABASE_URL: https://your-project.supabase.co SUPABASE_KEY: your-service-role-key # server-only # Public key the web login screen uses. Served by GET /api/config. SUPABASE_ANON_KEY: your-anon-key # Select Supabase as the STORAGE backend (default is "file"). # Setting only SUPABASE_URL/KEY enables Supabase AUTH but NOT storage. NODETOOL_STORAGE_BACKEND: supabase # Storage buckets ASSET_BUCKET: assets TEMP_BUCKET: assets-temp Without SUPABASE_ANON_KEY the server logs a warning, the web app has no public key to use, and every login fails with 401. Add AUTH_REDIRECT_URL when you serve NodeTool behind a domain, and allow-list it in the Supabase project. Or set them as environment variables directly: export SUPABASE_URL=https://your-project.supabase.co export SUPABASE_KEY=your-service-role-key export SUPABASE_ANON_KEY=your-anon-key export NODETOOL_STORAGE_BACKEND=supabase export ASSET_BUCKET=assets export TEMP_BUCKET=assets-temp 3. Deploy Apply the configuration to your deployment target: nodetool deploy apply <target-name> apply recreates the container with the new container.environment. For Docker Compose, put the variables in .env, which the bundled docker-compose.yml forwards through env_file, and run docker compose up -d. See Self-Hosted Deployment. Authentication The server enables Supabase auth automatically when both SUPABASE_URL and SUPABASE_KEY are present. There is no AUTH_PROVIDER switch read by the server entrypoint — Supabase-vs-local auth is selected purely by the presence of those two variables. When set, NodeTool uses Supabase JWTs for all API authentication: Users authenticate through Supabase (email/password, OAuth, magic link, etc.) API requests require a valid JWT in the Authorization header: Authorization: Bearer <supabase_jwt> NodeTool validates tokens against your Supabase project automatically When SUPABASE_URL/SUPABASE_KEY are not set, the server falls back to a local auth provider. Storage Behavior Backend Selection NodeTool selects the storage backend from NODETOOL_STORAGE_BACKEND (file | s3 | supabase, default file). Any other value fails at startup: file (default) – local filesystem under ASSET_FOLDER, then STORAGE_PATH, then the assets folder in the NodeTool data directory. s3 – requires ASSET_BUCKET and TEMP_BUCKET. Optional S3_REGION and S3_ENDPOINT (or S3_ENDPOINT_URL). supabase – requires SUPABASE_URL, SUPABASE_KEY, ASSET_BUCKET, and TEMP_BUCKET. A missing variable throws an error that names it. Storage is not auto-selected from the presence of SUPABASE_URL/SUPABASE_KEY: those enable Supabase auth, but you must set NODETOOL_STORAGE_BACKEND=supabase to route storage through Supabase. The asset bucket is ASSET_BUCKET and the temp bucket is TEMP_BUCKET. Asset URLs Public buckets generate direct Supabase Storage URLs Private buckets generate signed URLs that expire after 7 days (SIGNED_URL_TTL, the Supabase maximum) For a controlled proxy layer, configure your reverse proxy to mediate access Verification After deploying with Supabase, verify the integration: Check logs – Confirm the server started without a storage or auth error: nodetool deploy logs <target-name> Test asset storage – Run a workflow that writes assets and verify the resulting URLs point to your Supabase storage Test authentication – Call an API endpoint with a Supabase JWT: curl -H "Authorization: Bearer <supabase_jwt>" \ https://your-deployment.example.com/api/workflows Check Supabase dashboard – Verify assets appear in your storage buckets Security Considerations Never expose your service role key in client-side code – it has full admin access Use Row Level Security (RLS) policies if multiple users share the same Supabase project Rotate your service role key periodically and update deployment configs Consider using separate Supabase projects for staging and production See Security Hardening for the full production checklist Related Deployment Guide – Overview of all deployment options Authentication – Detailed authentication configuration Storage – Storage backend options and configuration Configuration – All environment variables and settings Security Hardening – Production security checklist--- # Telegram Bot Source: https://docs.nodetool.ai/telegram-bot The Telegram bridge lets you message your NodeTool agent from a phone. It is a separate process started with nodetool telegram serve. It turns each private-chat message into an agent turn on a running NodeTool server and streams the answer back into the chat. The bridge holds no conversation state and no user credentials. The agent loop, tools, permissions, threads and cost tracking stay on the server, and each turn runs as the NodeTool account linked to the Telegram user. This guide covers setup from end to end. For the command reference see CLI. For the design see telegram-bot-design.md. Requirements A NodeTool server in an authenticated mode, such as Supabase auth. In local single-user mode every request is user 1, so a delegated token would isolate nothing. The server refuses to issue one and the bot answers that the server is single-user. See Authentication. A Telegram account. A machine that can reach both the NodeTool server and the Telegram Bot API. 1. Create the bot with BotFather In Telegram, open a chat with @BotFather. Send /newbot and answer the prompts for a name and a username. Copy the token BotFather returns. It looks like 123456:AA.... Note the bot’s username without the @. 2. Configure the server Set these on the NodeTool server and restart it: Variable Purpose NODETOOL_INTEGRATION_TOKEN Service token that lets the bridge call the linking routes. At least 16 characters. When it is unset or shorter, the /api/integrations/* routes do not exist TELEGRAM_BOT_USERNAME The bot’s username without the @. Optional. With it, the settings page shows an Open Telegram and press Start link. Without it, you type the link code into the bot yourself NODETOOL_PUBLIC_URL Optional. The base URL that link confirmation pages use. Set it when the bridge reaches the server at an address your browser cannot, such as http://nodetool:7777 inside a compose network See Configuration for the full variable table. 3. Set the token and start the bridge Set these on the bridge process, which can run on the same machine as the server or on another one: Variable Required Purpose TELEGRAM_BOT_TOKEN Yes The BotFather token NODETOOL_INTEGRATION_TOKEN Yes The same value the server has NODETOOL_API_URL No Server address. Default http://127.0.0.1:7777 export TELEGRAM_BOT_TOKEN=123456:AA... export NODETOOL_INTEGRATION_TOKEN=the-same-value-the-server-has # Publish the command list so Telegram autocompletes it. Run it once per deploy. nodetool telegram register-commands # Poll Telegram and serve turns nodetool telegram serve Both subcommands check the configuration first and print every field that failed. serve prints Telegram bridge polling against <server URL> when it starts, and SIGINT or SIGTERM stops it. The bridge uses long polling with getUpdates and nothing else. Webhook mode is not implemented. Setting TELEGRAM_WEBHOOK_URL makes serve refuse to start. Only one process can poll a bot token. If a second one starts, the polling loop ends with a message that another getUpdates consumer holds the token. Stop the other instance and start again. 4. Link your NodeTool account Each Telegram account must be linked to a NodeTool account before the bot runs turns for it. There are two ways, and both write the same link. From NodeTool Open Settings and go to Integrations, then Connected Accounts. Click Connect Telegram. Click Open Telegram and press Start, then press Start in the bot chat. If the server has no TELEGRAM_BOT_USERNAME, send the bot /start <code> with the code shown on the page. The page shows “Connected as Telegram account” followed by your Telegram id. From Telegram Send the bot /link. Open the URL it replies with while signed in to NodeTool. The URL points to /integrations/link?code=.... The page names the Telegram account. Check it, then click Confirm. Return to the chat and say hello. A link code is valid for 10 minutes and works once. An expired or used code shows an error, and /link issues a new one. The server decides which NodeTool account is linked from your signed-in session, never from the code alone. Allowlist By default any Telegram user who finds the bot can start linking. To restrict it, list Telegram user ids in allowUsers in a telegram-bot.json file for the bridge: { "allowUsers": ["123456789"], "editThrottleMs": 1500, "maxQueuedTurns": 3 } Pass the file with nodetool telegram serve --config ./telegram-bot.json. Without --config, the bridge reads telegram-bot.json from its working directory if one exists. A user who is not on a non-empty list gets “This bot is restricted to a configured list of Telegram accounts, and yours is not on it.” for every message and command. The allowlist is a file setting, not an environment variable. The other two keys control the minimum gap between edits of one streamed reply in milliseconds, and how many messages queue behind a running turn. What the bot can do Send a text message and the bot runs it as an agent turn on your account with your tools, assets and budget. The bridge sends turns in agent mode with permission mode auto. The reply streams into the chat by editing messages. A status line shows what the agent is doing, and an inline stop button cancels the turn. Assets the agent produces arrive as photos or documents. Command What it does /start Shows a welcome and your link state. With a code, /start <code> finishes linking /link Starts linking and replies with a confirmation URL /unlink Unlinks this Telegram account. Needs a linked account /new Starts a fresh conversation thread. Needs a linked account /stop Cancels the running turn. Needs a linked account /status Reports server health, link state, current thread, whether a turn is running, and queue depth Commands never reach the model. Anything else starting with / gets “I do not know that command.” Limits to know: The bot works in private chats only. In a group it answers once that it works in private chat and then ignores the group. It handles text only. Photos, documents and voice notes get a reply saying so. A private chat is one conversation. While a turn runs, further messages queue up to maxQueuedTurns (default 3). Beyond that the bot asks you to resend when it is done. Uploads from the bot are bounded by Telegram’s 50 MB limit. Unlink Use /unlink in the chat, or open Settings, Integrations, Connected Accounts, click Disconnect and confirm. The bot stops answering that Telegram account.--- # Telegram Bot Integration — Technical Design Source: https://docs.nodetool.ai/telegram-bot-design.html Telegram Bot Integration — Technical Design Author: Matti Georgi Status: Draft — for review Last updated: 2026-08-18 Implementation plan: telegram-bot-implementation-plan.md Reference: georgi/claude-pipe — a Telegram/Discord/CLI bridge onto a coding-agent SDK, whose channel-adapter and progress-editing patterns this design reuses. Its Telegram channel is raw fetch against the Bot API with zero dependencies, which this design follows. 1. Summary A Telegram bot that gives Telegram users access to NodeTool’s unified agent loop, multi-tenant from the start: each Telegram user links their own NodeTool account once (/link), and from then on every message to the bot is a turn in that user’s own chat threads — their tools, their assets, their secrets, their budget. Streaming text, visible tool activity, and generated files come back into the chat. Telegram is the first platform because it is the cheapest one that proves the whole design. Bots are DM-first by nature — users message a bot directly, no server/guild to join, no install mode — so the conversation-ownership question that makes group chat hard (whose account runs, whose budget pays, whose data the agent may touch) does not arise: v1 handles private chats only, where the answer is always the one linked human in the chat. There is no gateway WebSocket, no privileged intents, no platform verification program: the Bot API is plain HTTPS, consumable by long polling or a webhook. Groups are deferred (§13, phase 4), and Discord follows as a second adapter on the same core (§13, phase 5). The bot is a bridge process, not a second agent runtime. It ships as packages/telegram (@nodetool-ai/telegram), depends on @nodetool-ai/sdk (and nothing platform-specific — the Bot API is called with fetch), and talks to a running NodeTool server over the existing /ws chat protocol. It owns exactly two translations: Telegram updates → chat_message commands, and ProcessingMessage frames → Telegram messages. The agent loop, tools, permissions, thread persistence, per-user cost tracking, and credit gating are the server’s, unchanged. The one server-side addition this design requires is the identity layer in §5: an external-identity table and two routes for linking and delegated tokens — deliberately provider-generic, so the later Discord adapter reuses them untouched. 2. Design goals D1. Zero new agent surface. The bot reaches the same WebSocketClientSession chat path the web UI uses (chat_message / resume_chat over /ws). No forked loop, no second toolbelt assembly, no drift. D2. Tenant isolation is the server’s, not the bot’s. Every turn runs on a token scoped to the sender’s NodeTool user, so thread history, assets, memories, secrets, and the credit gate isolate per user by the server’s existing rules. The bot never enforces isolation itself — it only presents the right identity. D3. The bot holds no user credentials. One service token identifies the bot to the server; per-user access is a short-lived delegated token the server mints per connection from its own Telegram↔user mapping. A compromised bot host leaks the service token (revocable, mints nothing without the server) — not a store of user tokens. D4. One conversation, one thread, both sides. A private chat maps to NodeTool threads owned by the linked user, so conversations are resumable from the web UI, the CLI, and Telegram alike, and survive bot restarts — the bot keeps no conversation state. D5. Small dependency cone. The bot process needs no database, no secret store, no native modules, and no Telegram SDK — @nodetool-ai/sdk + ws + fetch. It can run on a different machine than the server. D6. Fail visible, recover silent. A dropped WebSocket resumes the in-flight turn via resume_chat {thread_id, last_seq}; a dead server produces one status message in the chat, not silence. D7. Headlessly drivable. The frame renderer, update routing, and link flow are pure modules exercised by a fake-Telegram/fake-socket harness (docs/HARNESS_FIRST.md); the Bot API calls are the only part a test cannot own, and even those fake cleanly because the adapter’s HTTP surface is eight methods. D8. Platform-agnostic core. Only the adapter knows Telegram. Renderer, router, identity, and chunking are written against neutral types so the Discord adapter (phase 5) is a second implementation of one interface, not a fork — the lesson taken directly from claude-pipe’s Channel contract. Non-goals for v1: group chats and channels (phase 4); Discord (phase 5); voice/video calls; inline mode (@bot query in other chats); Telegram Stars/payments; per-group configuration. 3. Architecture Telegram Bot API (HTTPS) NodeTool server getUpdates / webhook │ │ │ ▼ │ ┌──────────────────┐ InboundTurn ┌──────────────────────┐ │ TelegramAdapter │ ────────────────▶ │ TurnRouter │ │ (raw Bot API, │ │ per-user ChatSocket │ │ private chats, │ ◀──────────────── │ + delegated token │ │ commands) │ RenderPlan └──────────┬───────────┘ └──────────────────┘ │ ▲ │ chat_message / resume_chat │ sendMessage / editMessageText / │ │ sendDocument / sendChatAction ┌─────────▼───────────┐ └───────────────────────────────│ @nodetool-ai/sdk │ │ ChatSocket (/ws, │ │ msgpack, ?token=) │ └─────────────────────┘ Identity (server-side, §5): bot ── service token ──▶ POST /api/integrations/telegram/token ──▶ delegated user token user ── /link ──▶ one-time URL ──▶ signed-in web session ──▶ external_identities row (or t.me/<bot>?start=<code> deep link, started from the NodeTool settings page) One Node.js bridge process, four modules: TelegramAdapter (src/telegram-adapter.ts) — owns the Bot API. Consumes updates (long polling by default, webhook mode for hosted deployments), accepts private-chat messages and commands, rejects everything else, downloads attachments via getFile, and executes RenderPlans (sendMessage, editMessageText, sendDocument/sendPhoto, sendChatAction). All Telegram knowledge — update shapes, parse mode, retry on 429 with retry_after — lives here. IdentityClient (src/identity-client.ts) — resolves a Telegram user id to a delegated NodeTool token via the server’s integration routes (§5), with an in-memory cache keyed by token expiry. Also drives the link/unlink flows, including the /start <code> deep-link payload. TurnRouter (src/turn-router.ts) — owns one ChatSocket per Telegram user with an in-flight turn, authenticated with that user’s delegated token. Serializes turns per conversation (a second message while a turn runs is queued, mirroring the server’s own chatTurnRegistry per-thread lock), forwards frames to the renderer, and handles reconnect/resume — including re-minting an expired delegated token before reconnecting. FrameRenderer (src/frame-renderer.ts) — pure function domain: folds ProcessingMessage frames (chunk, tool_call_update, message, output_update, error, job_update) into a RenderPlan (create/edit/finalize/attach operations), applying Telegram’s 4096-char limit and an edit-rate throttle. No I/O, no Telegram types — the adapter maps plan operations onto Bot API calls. This is claude-pipe’s bus/adapter/loop split with the agent loop replaced by a socket, plus an identity module claude-pipe never needed (it is single-user by construction). Update ingestion: polling and webhook getUpdates long polling is the default: zero inbound network requirements, right for self-hosted deployments. It is single-consumer per bot token, so the bridge is one process in this mode; a PID guard (as in claude-pipe) prevents a second instance from silently stealing updates. For the hosted deployment, webhook mode (setWebhook onto an HTTPS route the bridge serves) removes that constraint and scales horizontally behind a load balancer — the bridge is stateless (D4), so any instance can serve any update, with per-user turn serialization living in the server’s chatTurnRegistry rather than bridge memory. nodetool telegram serve --webhook <url> switches modes; the webhook secret token (X-Telegram-Bot-Api-Secret-Token) is verified on every delivery. 4. Conversation model Activation The bot answers exactly two things: Private-chat messages from linked users — every message is a turn. Commands — /start, /link, and /status work for anyone; the rest require a linked account. A message from an unlinked user gets one reply: what the bot is, and a /link prompt. Group messages are ignored in v1 — the bot additionally sets its Bot API group-privacy mode on and declines group membership gracefully (replies once with “I work in private chat for now” and leaves configuration of anything group-shaped to phase 4). An optional allowUsers list further restricts who may even link, for closed deployments. Thread identity A private chat is a permanent channel, not one conversation, so the chat maps to a sequence of NodeTool threads: the derived id is telegram-<chatId>-<uid8>-<n>, where n starts at 1 and /new increments it. <uid8> is a short hash of the NodeTool user id: thread ids are globally unique across users, so a purely chat-derived id would let one tenant occupy an id another tenant’s derivation produces (a chat can be re-linked to a different account, and chat ids are not secret) — the hash makes cross-tenant id collision structurally impossible rather than merely unlikely. The current n is recoverable without bot-side state: on the first turn after a restart, the bot lists the user’s threads via tRPC (threads.list, filtered by the id prefix) and resumes the highest n. The server creates thread rows lazily from client-supplied ids (ensureThreadExists in packages/websocket/src/session/chat-turn.ts), always under the authenticated user — so the thread belongs to the linked account and appears in that user’s web-UI thread list. Because a private chat has one human and the server-side thread has the full history, there is no context seeding — claude-pipe’s channel-history block exists to import a shared channel’s conversation, which v1 does not have. Thread titles: after the first turn, the bot calls trpc.threads.update (as the user, on the delegated token) to set a title from the first message, so the conversation is findable in the web UI. 5. Identity: linking and delegated tokens This is the one part of the design that adds server surface. It lives in packages/websocket (routes) and packages/models (table), and is provider-generic by construction — provider is a column, not a route family per platform — so the phase-5 Discord adapter adds a string, not a schema. Data model external_identities ( id TEXT PRIMARY KEY, provider TEXT NOT NULL, -- "telegram" (later: "discord") external_id TEXT NOT NULL, -- Telegram user id user_id TEXT NOT NULL, -- NodeTool user linked_at TEXT NOT NULL, UNIQUE(provider, external_id) ) One NodeTool user may link several Telegram accounts; one Telegram account maps to exactly one NodeTool user. Link flow Two directions, one table row either way: Bot-initiated (/link): The bot calls POST /api/integrations/telegram/link/start {external_id} with its service token (NODETOOL_INTEGRATION_TOKEN, a dedicated static token the server recognizes as the messaging integration — not a user token). The server stores a one-time link code (10-minute TTL) and returns a URL. The bot sends the user the URL. The user opens it, signs in to NodeTool normally (Supabase mode: their real account; local mode: user “1”), and confirms “Link Telegram account <name>?” on a minimal confirmation page. Confirming writes the external_identities row. The pending code is single-use; an unconfirmed code expires. Web-initiated (deep link): the NodeTool settings page (“Connect Telegram”) mints the same one-time code for the signed-in user and renders https://t.me/<bot>?start=<code>. Pressing Start in Telegram delivers /start <code> to the bot, which completes the link via POST /api/integrations/telegram/link/complete {external_id, code} (service token). This direction has less friction — no mid-chat URL to trust — and the confirmation is implicit in who minted the code while signed in. /unlink deletes the row (bot-initiated, service token, scoped to that external_id), and the settings page offers unlink to the signed-in user. The user’s browser session does the authentication in both directions, so the bot never sees a password, an OAuth code, or a long-lived user credential. Delegated tokens Per connection, the bot exchanges identity for access: POST /api/integrations/telegram/token {external_id} with the service token returns {token, expires_at, user_id} — a short-lived (1 h) token that authenticates as the linked user on /ws, /trpc, and asset URLs. Unlinked external_id → 404, and the bot renders the link prompt. Implementation slots into the existing AuthProvider seam: delegated tokens are signed server-side (HMAC over user_id + expiry with the master key from @nodetool-ai/security) and verified by a DelegatedTokenProvider chained before the configured provider, so no token table and no cleanup job. Revocation is coarse but sufficient: unlinking removes the mapping, so no new tokens mint, and outstanding ones die within the hour; rotating the master key kills them instantly. The integration routes are enabled only when NODETOOL_INTEGRATION_TOKEN is set; a server without it exposes none of this surface. 6. Turn lifecycle Inbound, per private-chat message: IdentityClient.resolve(telegramUserId) → delegated token (cached until near expiry) or the link prompt. Build content: message text plus the attachment section (§8). A voice note becomes text via the server’s own transcribe_audio command in phase 2; until then it gets a “voice notes arrive in phase 2” reply. TurnRouter.submit(user, conversation, content). If a turn is already running for this conversation the message queues; queue depth is capped (default 3), overflow gets a “still working on the previous message” reply. The router ensures a connected ChatSocket for this user and sends: socket.send({ threadId: conv.currentThreadId, // telegram-<chatId>-<uid8>-<n> text: content, provider: prefs.provider, // per-user override or server default model: prefs.model, agentMode: true, permissionMode: "auto" // §9 }); Outbound, per frame (all folding in FrameRenderer, all I/O in TelegramAdapter): Frame Telegram effect first chunk create the stream message (sendMessage); also sendChatAction("typing") until then, refreshed every 5 s (typing indicators expire) later chunk append to buffer; editMessageText at most every 1500 ms; past 3800 chars, finalize the current message and start a new one (headroom under the 4096 limit for formatting entities — the same constant claude-pipe converged on) tool_call_update / tool_call one status line message (🔧 web_search — "quickjs sandbox"), edited in place per tool; replaced by the stream message when text starts; ✅/❌ on completion task_update, planning_update, node_update folded into the status line (latest wins), never separate messages message (final assistant) finalize: edit the stream message to the final content’s tail chunk, send remaining chunks. Only an assistant message with no execution_event_type and no tool_calls counts: the compaction echo (role: "user"), the mid-turn tool-call echoes and tool results are ignored, or their text would be delivered as the answer and the real reply dropped output_update / saved assets fetch via the delegated token, send as sendPhoto/sendDocument (≤ 50 MB; larger files become a link to the server’s asset URL: 📎 too large for Telegram: <url>) error replace status/stream with ⚠️ <message>; the turn ends job_update (workflow-target runs) status line only Formatting: the agent emits markdown; Telegram’s MarkdownV2 parse mode is escape-hostile (every ., -, ( in prose must be escaped), so the renderer converts the safe subset — bold, italic, inline code, code blocks, links — to Telegram HTML parse mode instead, which only needs <, >, & escaped. A message the conversion cannot make valid falls back to plain text; a turn never fails on formatting. Rate discipline: edits are throttled per conversation (1500 ms) and coalesced. The Bot API answers 429 with retry_after; the adapter honors it and drops superseded intermediate edits rather than replaying them (only the latest buffered content matters). Interrupt and disconnect /stop — or tapping the inline ⏹ Stop button the bot attaches to the status message — sends stop(threadId); the server emits generation_stopped, rendered as ⏹ stopped. In a private chat the sender is necessarily the owner, so no permission check is needed. (Inline keyboards are a first-class Bot API primitive; the stop button costs nothing and previews the phase-3 approval buttons.) On socket drop mid-turn: re-mint the delegated token if expired, reconnect with backoff (the SDK’s ChatSocket already does this), then resume_chat {thread_id, last_seq} replays missed frames into the same renderer state. Only after resume also fails does the chat get one ⚠️ lost connection to NodeTool message. 7. Commands Registered once via setMyCommands (so they autocomplete in the Telegram UI), handled entirely in the bot — they never reach the LLM: Command Effect /start [code] welcome + link state; with a deep-link payload, completes web-initiated linking (§5) /link / /unlink account linking (§5) /new rotate this chat onto a fresh NodeTool thread /stop cancel the in-flight turn /model [id] show or set this user’s model override (stored server-side in the user’s settings via tRPC, so it survives bot restarts) /workflow <id> [params] run a turn with workflow_target: "workflow" against one of the user’s own saved workflows — the server’s workflow-chatbot path enforces ownership /status bot → server connectivity, link state, provider/model in effect, queue depth 8. Attachments Inbound: a Telegram attachment is a file_id; the bot resolves it with getFile and downloads from the Bot API file endpoint (bot-side download capped at 20 MB by the platform). Photos are passed as image content parts on the chat message (the server’s resolveContentForProvider inlines them for the provider). Documents are uploaded to the user’s NodeTool assets on the delegated token and described in the content ([Attached file: report.pdf, 1.2 MB, asset://…]) so the agent reaches them with its normal asset tools — unlike Discord’s public CDN URLs, Telegram file URLs embed the bot token, so they must never be handed to the agent as fetchable links. Voice notes: phase 2, via the server’s transcribe_audio. Outbound: asset references on the final message and output_update frames are fetched from the server on the delegated token and sent as photos/documents (≤ 50 MB upload). Over the limit, the bot posts the asset’s server URL instead — useful only where users can reach the server, so it is stated plainly rather than pretended around. 9. Security and access control Tenant isolation is delegated-token deep, not bot deep (D2): thread listing, asset fetches, workflow runs, collections, secrets, the credit gate, and cost attribution all execute server-side as the linked user. The bot contains no cross-tenant code path to get wrong. Credentials. The bot’s env carries two secrets: the Telegram bot token and the NodeTool service token. Per-user tokens are short-lived, in-memory only (D3). Provider API keys stay in the server’s secret store, per user. The bot token additionally leaks through file URLs (§8), which is why inbound files are re-homed as assets instead of passed as links. Server exposure. Multi-tenant means the server runs in Supabase (or another enforcing) auth mode; the integration routes refuse to start-link when the server is in local single-user mode with localhost trust, because “link any Telegram user to user 1” is not linking. Local mode remains supported for the personal-deployment case, where every chat is user “1” and /link replies “this server is single-user; you’re already in”. Permission mode. Turns run with permission_mode: "auto": there is no interactive approver on this surface, and the server’s default mode would park every write/execute tool on a tool_approval_request forever. In the multi-tenant frame this is materially safer than a shared-account design — the tools act on the sender’s own data under the sender’s own budget — but it is still the most consequential setting and sits at the top of the config with a comment. Phase 3 maps approval requests onto inline-keyboard buttons and makes the mode configurable. Prompt injection. Private-chat-only v1 has no channel-history or co-participant vectors: every byte the agent sees was typed or attached by the account owner it acts as. The residual risks are the ones NodeTool already owns server-side (fetched web content, attachment contents). Abuse. Linking is the admission gate (optionally allowlisted); per-user rate limiting beyond the turn queue is the server’s credit gate doing its normal job. The link URL is the one phishing-shaped artifact, so the confirmation page names the Telegram account being linked, the code is single-use with a 10-minute TTL — and the web-initiated deep link avoids mid-chat URLs entirely, so it is the flow the docs lead with. Update authenticity. Long polling is authenticated by the bot token; webhook mode verifies X-Telegram-Bot-Api-Secret-Token on every delivery and rejects everything else. 10. Global deployment One official bot can serve the hosted NodeTool cloud globally: the bridge is multi-tenant and stateless, so “global” is one bridge deployment pointed at api.nodetool.ai in webhook mode (horizontally scalable, §3), one @NodeTool bot username, and nothing else. Telegram imposes no guild caps, verification program, or privileged-intent review — the platform frictions that complicate the equivalent Discord rollout do not exist here. Self-hosted servers each register their own bot with @BotFather (a two-minute step) and run the bridge next to their server in polling mode. A central relay — the official bot fronting third-party servers — is rejected for the same reason as a bot-side token store (§12): it would make NodeTool-operated infrastructure a credential holder and traffic hub for servers it does not run. The code is identical in both cases — same package, different TELEGRAM_BOT_TOKEN/NODETOOL_API_URL. 11. Configuration Bridge process, env-first: TELEGRAM_BOT_TOKEN required (from @BotFather) NODETOOL_API_URL default http://127.0.0.1:7777 NODETOOL_INTEGRATION_TOKEN required (the bot's service token; also set on the server) TELEGRAM_WEBHOOK_URL optional; set = webhook mode, unset = long polling TELEGRAM_WEBHOOK_SECRET required in webhook mode // telegram-bot.json (optional) { "allowUsers": [], // Telegram user ids allowed to link; empty = anyone "editThrottleMs": 1500, "maxQueuedTurns": 3 } Server: NODETOOL_INTEGRATION_TOKEN enables the /api/integrations/telegram/* routes (§5). 12. Package layout and entry points packages/telegram/ package.json # @nodetool-ai/telegram — deps: @nodetool-ai/sdk, ws tsconfig.json # references: ../sdk src/ index.ts # startTelegramBot(config), stopTelegramBot — the programmatic surface config.ts # env + json loading, zod-validated at the boundary telegram-adapter.ts # Bot API client + update routing (polling and webhook) identity-client.ts turn-router.ts frame-renderer.ts # pure: frames → RenderPlan chunk.ts # 3800-char splitter (port of claude-pipe text-chunk) markdown-html.ts # pure: agent markdown → Telegram HTML parse mode register-commands.ts # setMyCommands tests/ # vitest: renderer folding, chunker, md→HTML, routing, # link flow, resume replay Server-side additions land in their owning packages: external_identities in packages/models, the integration routes and DelegatedTokenProvider wiring in packages/websocket / packages/auth. Two ways to run the bridge: npx @nodetool-ai/telegram (a bin in the package) for standalone deployment next to any server. nodetool telegram serve / nodetool telegram register-commands in the CLI — thin wrappers over startTelegramBot, consistent with how the CLI fronts the other harnesses. Command registration is explicitly separate from serving, as in claude-pipe: it is a deploy step, not a boot step. The dependency-cone rule (D5) is enforced structurally: packages/telegram references only packages/sdk in its tsconfig, so an import of @nodetool-ai/agents or @nodetool-ai/models fails the build. Shared core: identity-client, turn-router, frame-renderer, and chunk contain nothing Telegram-specific. They stay in packages/telegram for v1 — extracting a shared package for one consumer is premature — and move to a packages/messaging-bridge core when the Discord adapter lands (phase 5), which is the moment two consumers exist to keep it honest. 13. Alternatives considered In-process agent loop (import processChat + createChatCodeActSession, the way packages/cli/src/stdin.ts does). Rejected: the bot would own the DB, secret store, toolbelt assembly, and native modules, duplicate the server’s credit gate and turn registry, and — fatally for multi-tenancy — would have to re-implement per-user isolation the server already enforces. OpenAI-compatible endpoint (POST /v1/chat/completions). Rejected: a stateless provider passthrough — no threads, no agent toolbelt, no per-user anything. Bot-side token store (bot keeps long-lived per-user tokens after linking). Rejected for D3: it turns the bridge into a credential vault, needs encrypted storage and revocation plumbing, and breaks the stateless-bridge property. The service-token + delegated-token exchange keeps all credentials and the identity mapping on the server. A Telegram SDK (grammY, telegraf). Rejected: claude-pipe’s adapter demonstrates the needed surface is ~8 Bot API methods over fetch; an SDK adds a dependency tree and its own update-loop abstractions for no coverage this design uses. Revisit if inline mode or payments ever land. Group chats in v1 (group belongs to whoever linked it; co-participants’ messages run on that account, or fragment into per-sender threads). Deferred: both ownership rules are defensible and both are surprising — one bills the linker for other people’s prompts, the other splits one visible conversation into invisible parallel contexts. Telegram’s group-privacy mode adds a third wrinkle (the bot only sees mentions/replies unless privacy is disabled). Private-chat-only sidesteps all of it until the identity layer is proven; phase 4 decides deliberately. MarkdownV2 parse mode. Rejected for HTML: MarkdownV2 requires escaping most ASCII punctuation in prose, which is a per-message correctness tax; HTML needs three entities escaped and fails visibly in tests, not in production. Discord first. Deferred, not rejected — the earlier revision of this document designed the Discord adapter in full. Telegram ships first because it needs no gateway process, no privileged-intent approval, no verification program, and no user-install configuration, and because private-chat-first matches the v1 ownership model natively. The identity layer, router, and renderer are built platform-agnostic (D8) so the Discord adapter is phase 5, not a rewrite; its platform specifics (thread-per-conversation UX, 2000-char limit, gateway sharding, MessageContent intent) are recorded in this document’s git history. Finishing messaging.telegram.TelegramBotTrigger instead. The existing workflow node long-polls for messages inside a graph run. A workflow trigger answers a different question — “start this workflow when a Telegram message arrives” — and per triggers-design.md, that belongs in trigger_registrations + the dispatcher, not in a graph node holding a poll loop open. A later telegram_message trigger kind could share the bridge’s update stream. 14. Phasing Identity + bridge core — external_identities, link/token routes, DelegatedTokenProvider; package, config, private-chat routing, router, renderer, chunker, markdown→HTML; streaming edits + tool status line + stop button; final message + error rendering; reconnect/resume with token re-mint; register-commands + /start|link|unlink|new|stop|status; long-polling mode. Vitest suites for renderer/chunker/md-html/router/link-flow against a scripted fake socket and fake adapter; a telegram-bridge entry in the harness registry whose selfcheck runs those suites keylessly. Files, voice, workflows, preferences — inbound photos as image parts, documents re-homed as assets, voice notes via the server’s transcribe_audio, outbound asset sending, /workflow, /model persisted server-side, thread-title sync via tRPC, webhook mode. Approvals — render tool_approval_request as inline-keyboard buttons (allow / allow for chat / deny), answer with tool_approval_response, make permission_mode configurable. Retires the auto-mode caveat in §9 for operators who want it. Group support — mention/reply activation under group-privacy mode and an explicit conversation-ownership rule, on top of the proven identity layer. This is where the deferred ownership decision from §13 is made. Discord adapter — extract the shared core to packages/messaging-bridge, add packages/discord (gateway client, thread-per-conversation UX, 2000-char chunking, user-install app); provider: "discord" reuses the identity layer as-is. Trigger sharing — expose the update stream to a telegram_message trigger kind (separate design, per triggers-design.md). Phase 1 is the deliverable that makes everything after it iterable: from there, a change to the renderer, the formatter, or the link flow is a unit test, not a live Telegram session.--- # Telegram Bot Integration — Implementation Plan Source: https://docs.nodetool.ai/telegram-bot-implementation-plan.html Telegram Bot Integration — Implementation Plan Design: telegram-bot-design.md Status: Draft — for review Last updated: 2026-08-18 Six milestones. M1–M3 track the design’s phases 1–3 and produce a bot worth shipping; M4–M6 are the deliberate follow-ons (groups, Discord, triggers). Each milestone lists its tasks with owning files and a done-when that is checkable, and closes with exit criteria — the milestone is not done until every one holds. Task ids (T1.4) are for PR references; one task is roughly one reviewable PR-sized change, and tasks inside a milestone are ordered by dependency. Verification baseline for every task, per AGENTS.md: npm run typecheck, npm run lint, npm run test, npm run test:packages (or the packages nodetool affected names), and any new check must be proven able to fail before it counts. M1 — Identity layer (server) The provider-generic linking + delegated-token surface. Everything else depends on it, and it is testable with no Telegram account. # Task Files Done when T1.1 external_identities table + model. Drizzle schema {id, provider, external_id, user_id, linked_at}, unique (provider, external_id); ExternalIdentity DBModel with findByExternal(provider, externalId), listForUser(userId), link(...), unlink(provider, externalId); migration. packages/models/src/schema/external-identities.ts, packages/models/src/external-identity.ts Vitest CRUD suite passes; linking the same (provider, external_id) twice replaces, not duplicates; migration applies on a fresh DB. T1.2 Delegated tokens. HMAC-signed token (user_id, expires_at, master key from @nodetool-ai/security); DelegatedTokenProvider extends AuthProvider chained before the configured provider in the server’s auth hook; 1 h TTL. packages/auth/src/providers/delegated-token-provider.ts, wiring in packages/websocket/src/server.ts A minted token authenticates on /trpc and /ws as the mapped user; an expired or tampered token falls through to the next provider and is rejected; unit tests cover both, including the failure being provoked (flip one byte, shift the clock). T1.3 Integration routes. POST /api/integrations/:provider/link/start, /link/complete, /token, DELETE /link; guarded by NODETOOL_INTEGRATION_TOKEN (constant-time compare); routes absent when the env var is unset; link codes single-use, 10-minute TTL; /token refuses in local single-user trust mode per design §9. packages/websocket/src/routes/integrations.ts, zod bodies in http-body-schemas.ts Route tests: happy path mints a working token; wrong service token → 401; unknown external_id → 404; reused code → 410; unset env var → 404 on every route. T1.4 Link confirmation page + settings entry. Minimal signed-in confirmation page (“Link Telegram account <name>?”) for bot-initiated links; Connect Telegram card on Settings → Integrations that mints a code and renders the t.me/<bot>?start=<code> deep link; unlink from the same card. web/src/components/settings/ (ui_primitives only), one trpc router addition Playwright-free Jest tests for the card states (unlinked / code pending / linked); the confirmation page names the external account; design tokens and primitives pass npm run lint. Exit criteria A scripted test links a fake telegram:12345 to a user, exchanges the service token for a delegated token, opens a ChatSocket with it, runs one chat_message turn on a fake provider, and reads the thread back over tRPC as that user only — a second user’s token cannot see it. docs/configuration.md documents NODETOOL_INTEGRATION_TOKEN. M2 — Bridge core (packages/telegram) The bot process, private-chat-only, long polling, streaming UX. Pure modules first, adapter last, so every task lands with its tests. # Task Files Done when T2.1 Package scaffold. Workspace packages/telegram (@nodetool-ai/telegram), deps @nodetool-ai/sdk + ws only, tsconfig referencing ../sdk, bin, vitest config; registered in root workspaces. packages/telegram/* npm run build:packages builds it in order; npm run check passes; importing @nodetool-ai/agents from the package fails the build (proven once, then reverted). T2.2 Config. Env + optional telegram-bot.json, zod-validated at the boundary (TELEGRAM_BOT_TOKEN, NODETOOL_API_URL, NODETOOL_INTEGRATION_TOKEN, webhook vars; allowUsers, editThrottleMs, maxQueuedTurns). src/config.ts Invalid config fails fast with the field named; defaults match design §11. T2.3 Chunker + markdown→HTML. 3800-char splitter that never breaks inside a code fence or entity; agent-markdown → Telegram HTML for the safe subset, plain-text fallback on anything unconvertible. src/chunk.ts, src/markdown-html.ts Property-style vitest suites: no emitted chunk exceeds 4096 after HTML escaping; round-trip samples with nested fences, links, </& in prose; fallback path exercised. T2.4 FrameRenderer. Pure fold of ProcessingMessage frames → RenderPlan ops per design §6: stream-message lifecycle, status line (latest tool wins), 1500 ms coalescing, rollover, finalize, error, output_update attachments. src/frame-renderer.ts Scripted frame sequences (chunk storm, tool-then-text, error mid-stream, resume replay with duplicate seqs) produce the expected op lists; throttle verified with injected clock — no Date.now() in the pure core. T2.5 IdentityClient. Resolve telegram_user_id → delegated token via M1 routes, cache to expiry minus slack, link/complete/unlink calls, /start <code> payload handling. src/identity-client.ts Fake-server tests: cache hit/miss, expiry re-mint, 404 → link-prompt signal, complete with used code surfaces the 410 message. T2.6 TurnRouter. Per-user ChatSocket ownership, per-conversation turn serialization + capped queue (default 3), thread-id derivation telegram-<chatId>-<uid8>-<n> with highest-n recovery via threads.list, reconnect → re-mint → resume_chat {last_seq}. src/turn-router.ts Fake-socket tests: interleaved users don’t block each other; queue overflow answers busy; kill-socket-mid-turn resumes without duplicated render ops; restart recovery picks the highest existing n. T2.7 TelegramAdapter. Raw Bot API client (getUpdates long polling, PID guard; sendMessage/editMessageText/sendChatAction/answerCallbackQuery), private-chat + command routing, unlinked-user prompt, group messages declined once, 429 retry_after honored, superseded edits dropped. src/telegram-adapter.ts Adapter tests against a recorded fake Bot API: routing matrix (linked/unlinked × private/group/command), 429 backoff, edit coalescing; no real network in tests. T2.8 Commands + stop. /start, /link, /unlink, /new, /stop, /status; setMyCommands registration as a separate entry point; inline ⏹ Stop button on the status message wired to stop(threadId). src/register-commands.ts, command handling in adapter Each command’s handler covered; register-commands is idempotent; stop during a fake turn renders ⏹ stopped. T2.9 Entry points. startTelegramBot/stopTelegramBot; bin for npx @nodetool-ai/telegram; CLI nodetool telegram serve + nodetool telegram register-commands wrapping them. src/index.ts, packages/cli/src/commands/telegram.ts nodetool telegram serve against a fake token fails with the Bot API’s error, not a stack trace; CLI help lists both commands. T2.10 Harness registry entry. telegram-bridge harness whose keyless selfcheck runs the T2.3–T2.6 suites; surface mapping for packages/telegram/**. packages/cli/src/harness/registry.ts nodetool harness audit shows the surface covered; harness gate on a diff touching packages/telegram runs the selfcheck; the check proven able to fail (break one renderer expectation, watch it go red, restore). Exit criteria End-to-end against a real Telegram test bot and a local server in Supabase-mode test config: link via deep link, send “hello”, watch streamed edits and the tool status line, /new, /stop mid-turn, kill the bridge mid-turn and see the resumed turn complete. Recorded once as a manual checklist in the PR description (this is the one loop CI cannot own). npm run check green; harness gate --base main on the branch runs the telegram-bridge selfcheck. Milestone M2 = shippable v1 for self-hosted single-user servers (link short-circuits per design §9). M3 — Files, voice, workflows, preferences, webhook Everything that turns the text loop into the full product surface. # Task Files Done when T3.1 Inbound photos → image content parts (getFile download, size cap respected). adapter A photo message reaches the provider as an image part in the fake-server test; >20 MB answers the platform’s limit plainly. T3.2 Inbound documents → user assets on the delegated token, described in content as asset:// (never Bot-API file URLs — they embed the bot token, design §8). adapter, identity-client Asset appears under the linked user; the content string contains no api.telegram.org URL (asserted in test). T3.3 Voice notes → server transcribe_audio → text turn, with the transcript quoted in the reply for confirmation. adapter, turn-router Fake-transcription test; unsupported codec answers plainly. T3.4 Outbound assets: sendPhoto/sendDocument ≤ 50 MB, else the server URL line. adapter, renderer Renderer emits attach ops with size decision; adapter test covers both branches. T3.5 /workflow <id> [params] → workflow_target: "workflow" turns; ownership enforced server-side; param parsing errors answered before spending a turn. commands Running another user’s workflow id fails with the server’s error relayed; params round-trip in the fake-server test. T3.6 /model persisted server-side per user (settings via tRPC), read back on every turn. commands, turn-router Preference survives a bridge restart in test. T3.7 Thread-title sync after first turn. turn-router Title visible via threads.get in test; failures are logged, never fatal. T3.8 Webhook mode: HTTPS route, X-Telegram-Bot-Api-Secret-Token verification, setWebhook/deleteWebhook on start/stop, --webhook flag; polling remains default. adapter, src/index.ts Webhook delivery with wrong secret → 403 (test); mode switch documented; both modes share the routing matrix suite. Exit criteria The M2 manual checklist re-run extended with: send a photo, send a PDF, send a voice note, /workflow, /model, restart bridge, all green. Milestone M3 = shippable multi-tenant v1 for the hosted server (webhook mode). M4 — Approvals # Task Files Done when T4.1 Render tool_approval_request as inline keyboard (allow / allow for chat / deny); answer with tool_approval_response; decision timeout renders as denied. renderer, adapter Scripted approval round-trip in the fake-socket suite; unanswered request times out to deny. T4.2 permission_mode configurable (global default + per-user override via /mode), auto remains default; design §9 caveat updated. config, commands Mode default runs a write tool only after the button tap in test. Exit: a default-mode turn is usable end to end from Telegram; the design doc’s §9 caveat paragraph is rewritten to point here. M5 — Groups Scope decided at milestone start per design §13: pick the ownership rule (linker-pays vs per-sender threads) with a short ADR appended to the design doc before code. Tasks (mention/reply activation under group-privacy mode, per-group config, context seeding) are enumerated then — writing them now would pretend a decision that has not been made. Exit: ADR merged; group activation matrix covered by tests; the M2 checklist gains a group section. M6 — Discord adapter + shared core # Task Done when T6.1 Extract identity-client, turn-router, frame-renderer, chunk to packages/messaging-bridge; packages/telegram consumes it with zero behavior change. The T2 suites move with the code and stay green; packages/telegram’s diff is imports only. T6.2 packages/discord per the Discord revision of the design (git history): gateway client, thread-per-conversation UX, 2000-char chunking, user-install app; provider: "discord" on the untouched identity routes. Discord’s own adapter suite + the shared-core suites green; the M2-style manual checklist run against a Discord test bot. Exit: two adapters on one core; the shared package exists because two consumers do. Milestone summary Milestone Delivers Depends on M1 Identity: linking + delegated tokens, settings UI — M2 v1 bot: private chat, streaming, commands, harness M1 M3 Hosted v1: files, voice, workflows, prefs, webhook M2 M4 Interactive approvals M2 M5 Groups (ADR first) M3 M6 Discord on shared core M3 M4 and M5/M6 are independent of each other; M4 can land in parallel with M3.--- # Temp Storage Garbage Collection Source: https://docs.nodetool.ai/temp-storage-gc-design.html Temp Storage Garbage Collection Problem The runtime writes workflow media outputs to the temp/ storage prefix and never deletes them. ProcessingContext writes temp/<uuid>.<ext> for every temp_url output (packages/runtime/src/context.ts, the mode === "temp_url" branch). Workflow jobs and chat turns select this mode for every non-text session (packages/websocket/src/session/job-execution.ts, packages/websocket/src/session/chat-turn.ts). The SDK temporary-asset service writes temp/sdk-inputs/<id>.<ext> and returns expires_at: null (packages/websocket/src/sdk/sdk-temporary-asset-service.ts). No code lists or deletes keys under temp/. The retention sweep (cleanupStorage in packages/models/src/storage-maintenance.ts) deletes database rows only. On the file backend, temp/ shares the assets directory (~/.local/share/nodetool/assets/temp). One developer machine measured 3.9 GB in 20,561 files. On S3 and Supabase, TEMP_BUCKET has no documented lifecycle rule (docs/storage.md). Durable references to temp keys A temp URL is meant to be short-lived, but some durable rows store it. A measurement on one local SQLite database found: Table Rows Distinct temp refs Refs with an asset_id storyboards.document 3 20 19 nodetool_jobs.graph 16 13 1 nodetool_jobs.metadata_json 1 not measured not measured nodetool_workflows.graph 0 0 0 nodetool_assets (all columns) 0 0 0 No asset row points at temp/. An asset owns a separate file (<assetId>.<ext>, or <userId>/<assetId>.<ext> in the new layout). A temp ref that carries an asset_id can therefore switch to the asset without data loss. Storyboards leak temp URLs because packages/storyboard/src/io/render-shots.ts copies asset.uri next to asset_id into KeyframeVersion and ClipVersion. The asset_id is the durable locator, so the uri is redundant and expires. Decisions D1. Sweep by age through the storage adapter Add sweepTempStorage to packages/storage. It takes a StorageAdapter, a cutoff time, and a pin predicate. It uses the existing list("temp/"), stat, and delete methods, so it works on the file, S3, Supabase, and in-memory backends without backend branches. interface TempSweepOptions { olderThanMs: number; // age threshold, measured on StorageEntry.modifiedAt now: number; // injected for tests isPinned: (key: string) => boolean; dryRun: boolean; maxDeletes?: number; // bound one pass } interface TempSweepResult { scanned: number; deleted: number; pinned: number; bytesFreed: number; errors: number; } Rules: Consider only keys that start with temp/. Refuse any other prefix. Delete an entry only when now - modifiedAt >= olderThanMs and it is not pinned. Continue after a failed delete. Count it in errors. Delete with bounded concurrency. Log one summary line per pass. D2. One global retention window, not a per-user setting The temp/ prefix has no owner segment. Keys from all users share it, so a per-user policy cannot decide a key’s lifetime. Use one server-wide value: NODETOOL_TEMP_RETENTION_HOURS, default 168 (7 days), minimum 24. The minimum protects running jobs and open editor sessions, which fetch a temp URL minutes after the write. Register the variable in packages/websocket/src/settings-registry.ts and document it in docs/storage.md. D3. Pin temp keys that durable rows still reference Before a pass, build a set of temp keys from durable rows across all users: storyboards.document nodetool_workflows.graph and nodetool_workflow_versions.graph nodetool_jobs.graph and metadata_json for jobs that are not terminal Terminal jobs do not pin. The job retention window (terminalJobRetentionDays) already bounds their lifetime, and a finished run’s preview is scratch output. Extract keys with one pattern, temp/(sdk-inputs/)?[0-9a-f-]{32,36}\.[a-z0-9]+, from the raw column text. Put the scan in packages/models next to storage-maintenance.ts, because it reads model tables. Scan in pages. Do not load all documents into memory at once. The pin set is a safety net during migration. D4 removes the cause. D4. Store the asset locator, not the temp URL Change render-shots.ts to write uri: \asset://${assetId}` on keyframe and clip versions. Check other writers that copy a run output uri next to an asset_id` into a durable document, and fix them the same way. Add a one-time data migration: in storyboards.document, replace a /api/storage/temp/... uri with asset://<asset_id> when the same object carries an asset_id and that asset row exists. Leave refs without an asset_id unchanged. D3 keeps them pinned. D5. Run the sweep with the existing maintenance timer Call the sweep from the 6-hour runHistoryCleanup timer in packages/websocket/src/server.ts, through a new function in packages/websocket/src/storage-retention.ts. The temp adapter comes from getTempAdapter() (packages/websocket/src/lib/storage.ts). Run one pass at startup, like the history cleanup. Log a failure and continue. A sweep error must not stop the server. D6. Give the sweep a headless surface Add a CLI command, for example nodetool storage temp-gc [--dry-run] [--older-than <hours>], that prints the TempSweepResult. Register a selfcheck in packages/cli/src/harness/registry.ts per Harness-First Engineering. Follow the existing CLI command layout in packages/cli/src/. D7. Make the SDK expiry contract true Set expires_at in the SDK temporary-asset upload response to the upload time plus the retention window. Update the protocol schema if it requires null. Cloud backends The adapter sweep (D1) works on S3 and Supabase. Verify that list in s3-storage-adapter.ts and supabase-storage-adapter.ts follows continuation tokens. A single unpaginated call returns at most 1,000 keys, and the sweep then misses everything after that. Also add a bucket lifecycle rule on TEMP_BUCKET that expires objects after 30 days, as a backstop for a stopped sweep. Document the rule in docs/storage.md and docs/fly-production-deploy.md. Tests Test Location Asserts Sweep age cutoff packages/storage/tests/ Deletes old keys, keeps new keys, with injected now. Sweep prefix guard packages/storage/tests/ Never deletes a key outside temp/, even when it is old. Sweep pins and dry run packages/storage/tests/ Pinned keys survive. Dry run deletes nothing and reports the same counts. Sweep pagination packages/storage/tests/ More than 1,000 entries are all scanned (adapter stub). Pin scan packages/models/tests/ Finds keys in a storyboard, a workflow graph, and a running job. Ignores a terminal job. Storyboard migration packages/models/tests/ Rewrites a ref with asset_id to asset://. Keeps a ref without one. Render writer packages/storyboard/tests/ New keyframe and clip versions carry asset://<id>. Prove each new test fails against the unfixed code before you keep it. Risks R1. A client holds a temp URL longer than the window, for example a tab left open for a week. The preview breaks. The job result is still in history only as a dead link. The 7-day default makes this rare. R2. A durable writer that D3 does not scan stores a temp URL. The sweep then deletes a file the row still needs. Mitigation: search for writers during D4, and run the first production pass with --dry-run, then compare the delete list against a full-text search of the database. R3. A pin scan over large JSON columns is slow on PostgreSQL. Scan in pages, and filter with LIKE '%temp/%' before the regex. Open questions Q1. Is 7 days the right default window, or is 24 hours enough for the hosted service? Q2. Should refs without an asset_id (1 storyboard ref and 12 job refs on the measured machine) stay pinned forever, or expire after a longer window? Implementation Prompt Give this section to the implementing agent. It builds on the decisions above. Tasks Do the tasks in this order. Each task must leave the tree green. Read AGENTS.md, packages/AGENTS.md, and the storage, models, websocket, and cli package overlays that apply. Reproduce the leak. Write a failing test in packages/storyboard/tests/ that renders a keyframe through render-shots.ts and asserts the stored uri is asset://<assetId>. Then fix the writer (D4). Search for other writers that copy a run output uri next to an asset_id into a durable document, and fix them the same way. Add sweepTempStorage to packages/storage (D1). Use only the StorageAdapter interface. Refuse keys outside temp/. Inject now. Write the storage tests from the Tests table first. Verify that list in the S3 and Supabase adapters follows continuation tokens. If it does not, write a failing test, then fix the pagination. Add the pin scan to packages/models (D3). Scan across all users, page through rows, and pin only non-terminal jobs. Add its tests. Add the one-time storyboard migration (D4) through the existing migration mechanism in packages/models. Rewrite a temp uri to asset://<asset_id> only when the asset row exists. Add NODETOOL_TEMP_RETENTION_HOURS (default 168, minimum 24) to packages/websocket/src/settings-registry.ts (D2). Wire one sweep pass into the startup call and the 6-hour runHistoryCleanup timer in packages/websocket/src/server.ts, through packages/websocket/src/storage-retention.ts (D5). A sweep failure logs a warning and never stops the server. Add the CLI command with --dry-run and --older-than <hours>, and register its selfcheck in packages/cli/src/harness/registry.ts (D6). Set expires_at in the SDK temporary-asset upload response (D7). Update docs/storage.md with the retention variable, the sweep, and the recommended TEMP_BUCKET lifecycle rule. Update docs/fly-production-deploy.md if it lists bucket configuration. Constraints Deliver only this scope. Do not refactor the storage adapters or the retention settings UI. Do not add a per-user temp retention setting. The temp/ prefix has no owner segment (D2). Do not branch on the storage backend inside the sweep. Prove each new test fails against the unfixed code, then restore the fix. Use git checkout <base> -- <files> for the red proof, not git stash. Do not run the sweep against a real user data directory. Use temp directories and InMemoryStorageAdapter in tests. Do not add a co-author line to commits. Verification Run all four checks from the repository root. All must pass: npm run test:affected npm run typecheck npm run lint npm run dev:nodetool -- harness gate --base origin/main Then run the new CLI command with --dry-run against a scratch storage directory that holds old files, new files, and one file a test storyboard references. Report the printed counts. Report In the final response, state: The files changed, grouped by task. The red-proof evidence for each new test. The output of the four checks. The dry-run output. Any other durable writer of temp URLs you found, and the fix you applied. Your answers or evidence for open questions Q1 and Q2.--- # Templates Gallery Source: https://docs.nodetool.ai/templates-gallery.html The Examples page is a library of shipped example workflows, apps, storyboards, timelines, sketches, 3D models, and games that you can copy into your own project. Open it from the Examples item in the More panel, or from Browse examples on the new-project surface. It opens as a workspace tab, not as a URL route. Tabs Tab What a card does Apps Add to your apps installs the app and the workflows it runs Workflows Click a card to copy the workflow into your project Storyboards Adds an editable copy of the board Timelines Open editable timeline adds a copy Sketches Open sketch creates your own copy and opens it in the sketch editor 3D models Open model saves the model as an asset in your project and opens it in the 3D editor Games Play and edit adds a copy Sketches are 1200 × 900 with five vector layers each, and a category row filters them (All, NodeTool, Ads, Movies, Art studies). The 3D models tab groups low-poly glTF models into packs: Adventure, Dungeon, Platformer, Seaside, Sci-fi, and Sci-fi outpost. Neither tab needs a model or API key. See Sketch Editor for editing sketches. The rest of this page covers the Workflows tab, which is the template gallery. Opening a Template Click a template card. NodeTool creates a private copy named after the template, tagged example, in your current project, and opens it in a workspace tab. The original is never modified, so you can edit and save the copy freely. Templates ship with NodeTool and are loaded from the workflows.examples tRPC query. They are grouped by tag into the categories below. Category Tags that match Image image, design Video video, youtube Audio audio Multimodal multimodal Agents agent, agents, ai, claude, huggingface Data & Web data, web, search, serp, google, news, reddit, amazon, trends, analysis, research, rag Browse walkthroughs of individual templates on the [Workflow Examples]({{ ‘/workflows/’ relative_url }}) page. Filtering and Searching Category pills filter the list. All shows everything. Search matches the template name, description, and tags. Press / to focus the search box. Templates tagged getting-started or start sort first in the unfiltered list. Everything else sorts by name. If nothing matches, the page offers Clear search or Show all templates. Anatomy of a Template Card Each card shows: Thumbnail, or a category-colored placeholder when the template has none. Title and description. Inputs, each marked required, optional, or requirement unknown. Provider/model, the models the graph selects, or the providers it uses. Execution, whether it runs locally, on provider-hosted models, or both. Required setup and Runtime, when the graph needs them. Estimated cost, computed from published node prices. A graph with unpriced nodes shows “at least” or “unknown” with the count of unpriced nodes. Category label. Model fields in shipped templates are empty. When you open one, NodeTool fills each from your own default model for that type. If a model is missing, the Recommended Models dialog offers to install it. Adding Your Own Template Shipped templates are JSON files in packages/base-nodes/nodetool/examples/nodetool-base/. Their listing metadata (name, description, tags) lives in packages/base-nodes/nodetool/package_metadata/nodetool-base.json. A new template needs: A graph that does real work and is not a re-skin of an existing example. Empty model fields, a description under 80 characters, and a thumbnail at packages/base-nodes/nodetool/assets/nodetool-base/<name>.jpg. A passing npm run dev:nodetool -- validate "<file>.json". The full checklist, including thumbnail generation and the marketing catalog step, is in packages/base-nodes/nodetool/examples/nodetool-base/README_EXAMPLES.md. To turn a saved workflow into code, use nodetool workflows export-dsl <id>. Related Docs [Workflow Examples]({{ ‘/workflows/’ relative_url }}) — in-depth walkthroughs of individual templates [Cookbook]({{ ‘/cookbook’ relative_url }}) — reusable workflow patterns [Getting Started]({{ ‘/getting-started’ relative_url }}) — runs a template as your first workflow--- # Text and SVG Documents Source: https://docs.nodetool.ai/text-and-svg-documents The New menu in the workspace tab bar creates blank documents. Two of them are plain files you edit directly: Text (Markdown, JSON, YAML, CSV, TSV, or plain text) and SVG. Both are stored as project assets, open in a tab, and save back to the same asset. Quick Access: Click + New in the tab bar. Under Blank documents, choose New text file… or New SVG. What the New menu offers The menu has a Guided flows section and a Blank documents section. Blank documents are created in the project you have open, and each one opens in a new tab. Menu item Creates More New workflow An empty workflow, opened in edit mode Workflow Editor New chat A chat thread Chat New text file… A text file from a template. Opens a submenu Text files below New sketch A blank 1024 by 1024 white image, opened in the sketch editor Sketch Editor New SVG A blank SVG drawing SVG documents below New timeline An empty timeline named “Untitled timeline” Video Editor New storyboard… A blank storyboard, or an example storyboard with stills included. Opens a submenu   New app An empty mini app named “Untitled app” App Builder New script A script document named “Untitled script”   New JS script A JavaScript script document named “Untitled JS script” JavaScript Sandbox New skill A skill with a generated name and a starter body Skills New 3D model A model containing a single box 3D Editor Items marked with an ellipsis open a second list. Use Back at the top of that list to return. If creating a document fails, a notification names what could not be created and why. Games are not in this menu. See Game Editor for how to start one. Text files New text file… opens a list of six templates. Choosing one creates an asset with the starter content shown and opens it in Edit mode. Template File name Starts with Markdown (.md) Untitled.md # Untitled JSON (.json) Untitled.json {} YAML (.yaml) Untitled.yaml --- CSV (.csv) Untitled.csv A single column named Column 1 TSV (.tsv) Untitled.tsv A single column named Column 1 Plain text (.txt) Untitled.txt Empty The file extension decides how the document is shown and highlighted. View and Edit A text tab has a View and Edit toggle at the right of the tab bar. Mode Markdown CSV and TSV Everything else View Rendered Markdown A read-only table. The first 2000 rows are shown Read-only code with syntax highlighting Edit Code editor An editable table Code editor The view header shows the file name, the detected language, and a copy button that copies the whole text. The code editor Edit mode uses the Monaco editor, the same editor as VS Code, with a minimap and word wrap on by default. The language comes from the file extension, for example .json as JSON, .py as Python, .ts as TypeScript, .sql as SQL, .html as HTML, and .sh as shell. Files with no known extension but a text/ content type are treated as plain text. The toolbar above the editor has: Undo and Redo Toggle Word Wrap Save, with a dot beside it while there are unsaved changes Press Ctrl+S (Cmd+S on macOS) inside the editor to save. The buttons for Find & Replace and Format as Code Block appear in the toolbar but are not wired up in this editor. The table editor CSV and TSV files open in a table instead of text. You edit cells directly, and the file is re-serialized with the right delimiter (comma for .csv, tab for .tsv) when you save. Save sits in the table toolbar. SVG documents New SVG creates Untitled.svg containing an empty 512 by 512 drawing: <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512" width="512" height="512"> </svg> An SVG is both a picture and a text file, so it has its own tab type. View and Edit Mode What you see View The drawing, centered on a checkerboard so transparent areas read as transparent Edit The SVG markup in a code editor on the left and a live preview on the right The preview renders the markup as you type, before you save. If the markup has no <svg> element, the preview shows “No <svg> element to render”. The markup is XML, so the editor highlights it as XML. There is no drawing canvas. You write or paste the markup and watch the preview. Saving The toolbar shows the file name, a dot while there are unsaved changes, and a Save button. Ctrl+S (Cmd+S on macOS) also saves. Save is disabled until you change something. A notification confirms the save or shows the error. Saving writes the markup back to the asset with its original content type, image/svg+xml by default. What the preview removes The preview shows sanitized markup. It drops <script> elements, <foreignObject>, and the onload, onerror, and onclick attributes, and keeps the drawing elements: shapes, paths, text, gradients, filters, masks, and symbols. Your saved file is not changed by this, so the preview can look different from the raw file if you rely on scripts or embedded HTML. Related Workspaces for tabs, projects, and the View / Edit toggle Asset Management for the assets these documents are stored as Sketch Editor for raster painting--- # Timeline Code Capture Source: https://docs.nodetool.ai/timeline-code-capture.html Timeline Code Capture A timeline built with @nodetool-ai/sandbox-timeline stores the code that builds it in document.source. This page defines what that code is and how it is produced. The rule The stored code is a retained program, not the script the author ran. The retained program holds only what rebuilds the document: the scene callbacks, as source text the declarations those callbacks use, as source text, when they are pure the values those callbacks read from anything else, as literals A research call, a generation, a fetch or a model call runs once, in the author’s run. Its result enters the retained program as a literal. A rebake runs the retained program with no capabilities, so it cannot repeat a side effect. Capture buildEntryModule (packages/agents/src/js-sandbox-worker/interpreter.ts) passes guest code that imports @nodetool-ai/sandbox-timeline through captureTimelineScenes (packages/agents/src/timeline-capture.ts). Every guest run passes through this function, so every agent path gets the same capture. The transform finds each <receiver>.scene(name, seconds, fn, extra?) call and appends one argument: the capture record. The record adds no line breaks, so stack lines stay correct. It holds: Field Content src The source text of fn, or the name of the function it refers to decls The retained declarations that fn needs, with source text and source order imports The pack imports that fn and the declarations use snap A thunk for each value to snapshot, called when the scene is built errors Diagnostics for values that cannot be kept Classification The transform resolves each free variable of fn with eslint-scope, then follows the free variables of each retained declaration in the same way. Variable Result A standard global (Math, JSON, Number, …) Kept as is An import from a @nodetool-ai/sandbox-* pack Kept as an import The video(...) binding Printed from its recorded options A const or function whose initializer has no await, is not mutated, and uses only kept variables Kept as source text Any other variable: let, a loop variable, a parameter, a result of await Snapshot of its value when the scene is built nodetool, tools or another capability global Error A snapshot value that is a function, a class instance or a cycle Error A declaration that uses a snapshot is kept, but it is printed inside the scene’s own block, after the snapshot. The printed program v.save() prints the retained program from the capture records: import { video, rad } from "@nodetool-ai/sandbox-timeline"; const v = video({"width":1920,"height":1080,"fps":30,"palette":{}}); const PAD = 120; // kept declaration function title(s, text) { /* kept helper */ } const __scene0 = (() => { const headline = "Rain returns to the valley"; // snapshot return v.scene("intro", 3, (s) => { title(s, headline); }); })(); v.series([__scene0, v.transition("fade", 0.5, {}), __scene1]); v.__restore({ /* tracks, clips, markers and fields added after series */ }); return await v.save(nodetool.timelines, { name: "…", ops: [] }); Everything that v.adjust, v.audio, v.music, v.midi, v.beats and v.document add after v.series is stored as data in v.__restore. The embed gate v.save() passes the printed program to nodetool.timelines.code.set with require_match: true. The host runs the program in the hermetic bake. It attaches the program only when the bake gives exactly the saved document. In all other cases it writes nothing and returns a warning, which v.save() returns in code.warnings. The usual causes are Math.random() or Date.now() in a scene callback, and a scene clip that changes after v.series. el.react() needs an audio decode that the bake cannot do, so a timeline that uses it is not embedded. Not in this phase Keys and overrides. Clip ids come from call order in each scene. A key option, s.each(items, {key}, fn) and per-instance overrides come next. With them, a hand edit becomes an override instead of a conflict. Expression editing. Expressions are not stored as ASTs. A rebake runs the scene callback again, so a changed constant still moves every value that depends on it.--- # Custom timeline animations Source: https://docs.nodetool.ai/timeline-custom-animations.html Custom timeline animations Motion written as JavaScript instead of picked from the preset catalog. For the wider visual authoring surface — style tracks, text animators, camera, layout, repeaters, effects, and transitions — see Timeline motion design. The body runs once, host-side, and returns keyframes. Those keyframes are stored on the clip and compiled exactly like a preset’s, so nothing evaluates JavaScript at render time. Why baking rather than per-frame evaluation Five surfaces sample animations: the WebGPU preview (PreviewCompositor.tsx), the web export renderer (TimelineRenderer.ts), the text rasterizer (rasterClipFrames.ts), and the headless compositor (packages/video-nodes/src/nodes/timeline/compositeRender.ts). They share one pure sampler in packages/timeline, and there is no JS sandbox in the browser — QuickJS runs server-side only (web/src/components/jsScript/runJsScript.ts). Evaluating a body per layer per frame would mean a second engine in the web bundle, an async hop inside a loop that already has no headroom (docs/timeline-editor-performance-audit.md, tier 2), and two implementations to keep bit-identical. Baking gives up one thing — a script cannot react to playback state — and buys identical output on every surface for free. Sampling f(t) densely is what makes that equivalent: a body emits its function at N points and the sampler interpolates, exact to the sampling resolution. The script contract A body is a Code-node body. It reads inputs and returns its result through output(): const samples = []; for (let i = 0; i <= inputs.sampleCount; i++) { const t = i / inputs.sampleCount; samples.push({ t, opacity: t, offsetY: (1 - t) * inputs.canvasHeight * 0.1, }); } await output("samples", samples); Inputs Field Meaning role "in", "out", "emphasis", or "loop" durationMs The animation’s own window length clipDurationMs The clip it sits on canvasWidth, canvasHeight Resolve a normalized distance to px, as a preset does params The animation’s params, untouched staggerCount Stagger units the clip splits into (a text clip’s word count), else 0 sampleCount Suggested density for a body sampling a continuous function Outputs Return exactly one of: samples — one bag per point in time, {t, opacity, offsetY, …}. A property must be set on every sample or none; a hole would make the sampler invent motion the body never wrote. curves — per-property keyframes, {property, keyframes: [{t, value, easing?}]}, for a body that authored them directly. A body driving wipeProgress must also output("mask", {direction, softness}). Direction and softness never animate, and defaulting them would render a wipe nobody described. Animatable properties ANIMATED_PROPERTIES in packages/timeline/src/animation/types.ts is the list; ANIMATED_PROPERTY_DOCS in animation/custom.ts pairs each channel with its fold, its identity value and its range, and is what list_animation_presets prints. Transform: offsetX, offsetY, scale, scaleX, scaleY, rotation, rotationX, rotationY, positionX, positionY, anchorX, anchorY. Compositing: opacity, wipeProgress. Grade and blur: blur, brightness, saturation, contrast, hue, temperature, tint. Shape stroke: trimStart, trimEnd. A channel folds one of four ways when several animations drive it: add, multiply, min (wipeProgress — more hidden wins), or replace (the absolute channels: position, anchor, trim). Two differences from a preset No time reversal for "out". A preset authors forward motion and the compiler reverses it; a body is handed its role and writes the motion it wants. Segments default to linear. A preset’s role easing on top of a densely sampled f(t) would distort values the body already shaped. An explicit animation.easing, or a per-keyframe easing, still wins. Storage The animation carries preset: "custom" and a custom payload (clipAnimation in packages/protocol/src/api-schemas/timeline.ts): { "id": "anim-1", "role": "in", "preset": "custom", "durationMs": 600, "custom": { "scriptId": "js-script-row-id", // or "code": "…" — provenance, never run at render time "bakedAt": "2026-09-01T12:00:00Z", "curves": [{ "property": "opacity", "keyframes": [{ "t": 0, "value": 0 }, { "t": 1, "value": 1 }] }] } } Limits, enforced at bake and again at compile: 16 curves per animation, 4096 keyframes per curve, one curve per property. Source-anchored curves custom.timeBase picks the clock a curve’s keyframes are placed on: "clip" (the default) normalizes t over the animation window, the way a preset does. "source" places every keyframe at an absolute sourceMs instead — a time in the clip’s own media — and the sampler evaluates the curve at the clip’s current source time (clipSourceMsAt, timeRemap.ts) rather than at the window’s t. That is the same time the compositor seeks the clip’s video to, so a speed change, an in-point trim, or a timeRemap moves the animation exactly the way it moves the footage. bakedFrom names what produced the curve when a hand edit did not — an audio loudness bake, a motion tracker — as provenance to re-bake from (kind, and optionally clipId, assetId, settings), never something rendered. Because the curve is placed in the media, a trim or a split re-slices it instead of letting the window stretch it (animation/sourceCurves.ts): keyframes outside the retained source window are dropped, and an interpolated keyframe is added at each new edge, so the motion at a timeline instant the edit did not touch is unchanged. That is exact for a linear segment — the compiler’s default for a custom animation — and an approximation when a cut lands inside an eased segment. A clip-based curve carries no sourceMs and is left alone: stretching with the window is what its t means. Baking POST /api/timelines/animations/bake, with code or script_id (a js_scripts row), the role, the timings, and the canvas: curl -sX POST localhost:7777/api/timelines/animations/bake \ -H 'content-type: application/json' \ -d '{"code":"await output(\"samples\",[{t:0,opacity:0},{t:1,opacity:1}]);", "role":"in","duration_ms":500,"clip_duration_ms":3000, "canvas":{"width":1920,"height":1080}}' The response carries curves (and mask), the body’s logs, and error when the body failed — a body that throws is a result to show its author, not a 500. The run is hermetic: no toolbelt, no secrets, no network, capped at 10s. A curve generator is a function of time, and reach would let the same animation bake differently depending on where it ran. The agent path An agent writes a custom animation the way it writes any other: one animate_clip op through edit_timeline (or ui_timeline_animate_clip in an open editor), with preset: "custom" and exactly one of curves and code. { "op": "animate_clip", "target": "Title", "animations": [{ "role": "in", "preset": "custom", "durationMs": 800, "curves": [{ "property": "offsetY", "keyframes": [{ "t": 0, "value": 120 }, { "t": 1, "value": 0 }] }] }] } curves are the keyframes themselves — t runs 0..1 over the animation’s window, and a curve that stops short of either end is extended by holding its end value. code is the body above: the host bakes it once through bakeCustomAnimation — the same hermetic run the endpoint uses — and stores code beside the curves as provenance. Both paths end at normalizeCustomCurves, so what is stored is what will render. Four rules the op enforces, each with the message that says how to fix it: Both curves and code, or neither, is refused. One says what the motion is; the other says what computes it. A property outside ANIMATED_PROPERTIES is refused, and the message lists the ones a curve may drive. A wipeProgress curve needs mask ({direction, softness}) — the same refusal the compiler and the validator raise. A machine-produced curve goes through set_baked_animation, not this op. code needs a host that can bake it. edit_timeline wires one; a surface with none says so and points at curves rather than storing an unbaked body. Without durationMs the animation spans the clip, so hand-written curves need no arithmetic to line up with a cut. list_animation_presets reports the custom contract and every animatable property with its fold, identity and range, so the channels can be read off the tool rather than out of this document. A curve something measured A bake is re-run — change the sensitivity, bake again — so it cannot use animate_clip, whose mode is “append everything” or “replace everything”. set_baked_animation writes one source-anchored curve and carries its provenance: { "op": "set_baked_animation", "target": "Shot 3", "animation": { "property": "scale", "keyframes": [{ "sourceMs": 0, "value": 1 }, { "sourceMs": 1500, "value": 1.12 }], "timeBase": "source", "bakedFrom": { "kind": "audio", "clipId": "clip_music", "assetId": "asset_x" } } } bakedFrom.kind plus the driven property is the identity a re-bake matches: a second write of the same kind on the same property replaces that animation in place, keeping its id, and replace: false appends beside it instead. An animation with no bakedFrom — anything a person keyframed — never matches and is never overwritten. role defaults to emphasis over the whole clip, and bakedAt is not stamped: bakedFrom is this path’s provenance, and a wall-clock field would make two hosts running the same bake write two different documents. bake_audio_animation is the producer that exists today. It measures an audio clip’s own stretch of its file, turns loudness (mode: "envelope") or onsets (mode: "beats") into keyframes, maps each one from audio-source time through timeline time into the target clip’s source time, and applies the op. It refuses a time-remapped clip on either end, because a remap makes “which source millisecond plays here” a curve over the clip’s window with no inverse to map a beat through. Checks nodetool timeline validate and the validate_timeline tool report custom_animation_invalid for curves the compiler would skip (an unknown property, a keyframe with no finite value, a wipeProgress curve with no mask) Curves that are absent or empty are reported by that same code — an animation that drives nothing. Inline curves need no script or code behind them: under the agent path they are the source, not a bake of one. Code Piece Where Contract, normalization, limits (pure) packages/timeline/src/animation/custom.ts Compiler path packages/timeline/src/animation/compile.ts Wire schema packages/protocol/src/api-schemas/timeline.ts Bake (the one place the body runs) packages/agents/src/custom-animation-bake.ts HTTP surface packages/websocket/src/routes/timeline-animations.ts Shared edit engine packages/timeline/src/ops/apply.ts Server host adapter packages/agents/src/capabilities/timeline-bridge.ts Baked-curve write and its replace rule (pure) packages/timeline/src/animation/bakedAnimation.ts Audio bake: the three clocks and the two curve shapes packages/agents/src/capabilities/timeline-audio-bake.ts Validation packages/execution/src/timeline-debug/validate.ts Not built yet No editor UI. Outside the agent path above, a custom animation is authored through the bake endpoint and written onto the clip by whatever holds the document — the CLI, or a client calling the route directly.--- # Timeline Editor Performance Audit Source: https://docs.nodetool.ai/timeline-editor-performance-audit.html Timeline Editor Performance Audit Audited 2026-07-02 at commit 6c3004e. Scope: web/src/components/timeline/, web/src/stores/timeline/, web/src/hooks/timeline/, and the render/GPU layers they drive (~28k lines). Line numbers refer to that commit. Verdict The core architecture is right, and the most expensive problem a timeline editor can have — React re-rendering at 60 fps during playback — is already solved. TimelinePlaybackStore pushes time through a transient non-React channel (setTimeMs/subscribeTime, TimelinePlaybackStore.ts:55-118), the playhead/timecode/karaoke highlight all consume it imperatively, and the compositor gates React updates behind a scene signature. Steady playback causes zero React renders. The real costs are elsewhere, in three clusters: Gestures write to the reactive document store on every pointermove. Clip drag/trim, the transform gizmo, ~40 inspector/FX sliders, EQ and compressor curves, scrubbing, and zoom all publish store updates at pointer-event rate (60–240 Hz). Identity-preserving reducers keep most re-renders in check, but every publish still runs every subscriber’s selector, and several wide subscriptions re-render the whole editor shell per tick. Sliders and the gizmo also push one undo entry per tick, so a two-second drag wipes the entire 100-entry undo history. Per-frame playback work that scales with project size. The rAF tick re-derives the full scene model (sort + Map build + per-word caption resolution) every frame just to detect “nothing changed”, and the GPU effects chain re-processes static layers 60×/s while any video plays. Unbounded scaling. No virtualization (every clip on every track is always mounted, with thumbnail/waveform fetches), play-gesture audio decoding of the entire timeline, and several unbounded or full-frame-sized caches. Fixes are listed per finding; a priority table is at the end. The single highest-leverage theme: the repo already contains the right tools — useTimelineHistoryBatch, the transient time channel, stable generatingClipIds membership arrays — several hot paths just don’t use them. Tier 1 — interactive gestures (highest user-visible impact) 1.1 Inspector/FX sliders and curves: per-tick store writes, no history batching NodeSlider is a MUI Slider whose onChange fires per pointermove. Every call site pipes it straight into the zundo-wrapped TimelineStore: Inspector/InspectorPrimitives.tsx:641-643 (InspectorSliderRow — used by all ~30 ClipAdjustments rows: opacity 167-169, color 441-444, blur 533, anchor 338/348, radius 357-359) Tracks/TrackEffectsPanel.tsx:250 (ParamRow → updateTrackEffect, TimelineStore.ts:896-907) Tracks/TrackEffectsPanel.tsx:520-540 and 985-1020 (EQ / compressor SVG curve drags patch per pointermove; wheel handlers at 560-575, 1039-1054 patch per wheel tick) TimelineStore.ts:1498-1507 (setClipPrompt via DirectGenClipPanel.tsx:146-151 — one undo entry per keystroke) Arrow-key nudge (TracksRegion.tsx:504-518) — one entry per key repeat Each tick: full tracks.map/clips.map allocation, zundo partializedEqual scan over all clips/tracks (TimelineStore.ts:489-497), one undo entry (limit is 100 — a single knob drag evicts all prior history), and a publish to every doc-store subscriber (compositor, tracks region, transcript panel, autosave, attachUiPruning). Clip drag/trim and track resize already solve this with useTimelineHistoryBatch (Clip.tsx:496, TrackHeader.tsx:323); nothing in the inspector or FX panel uses it. Fix: in the two shared choke points (InspectorSliderRow, ParamRow), render the drag from local state and commit on onChangeCommitted, or keep live writes for preview but wrap the gesture in useTimelineHistoryBatch.begin()/mark()/end() and rAF-throttle the writes. Same treatment for the EQ/compressor pointerdown/up handlers. Pause the temporal store during focused prompt typing; treat held key-repeat as one gesture. 1.2 Transform gizmo: per-pointermove patchClip, no batching, forced layout PreviewCompositor.tsx:928: onChange={(id, next) => patchClip(id, { transform: next })} Every gizmo move (60–240 Hz) allocates a new clips array (TimelineStore.ts:1278-1282), pushes an undo entry, and re-renders every clips subscriber — compositor scene memo, video-pool layout effect, tracks region, inspector. The pointer handler also calls getBoundingClientRect() per move (TransformGizmoOverlay.tsx:330-333). Fix: wire useTimelineHistoryBatch into the gizmo gesture, or drive the live transform from a ref straight into the compositor and commit one patchClip on pointerup. Cache the SVG rect at pointerdown (a ResizeObserver already invalidates on resize). 1.3 Editor-root subscriptions re-render the whole shell per drag tick Three subscriptions sit at TimelineEditorBody — the root above TopBar, preview, inspector, tracks, and transcript: useTimelineExport (useTimelineExport.ts:57-66, hosted at TimelineEditor.tsx:352-359) reactively selects tracks + clips that are only read inside the click-time exportVideo callback. clips gets a new identity per drag/trim/gizmo/slider tick → whole-shell re-render per tick. During export, onProgress: setProgress (line 119) re-renders the shell per encoded frame. Fix: read store.getState() inside exportVideo; throttle progress to ~4 Hz or move it to a leaf component. useTimelineGenerationSubscriptions (useGenerateClip.ts:318-354, hosted at TimelineEditor.tsx:335) selects the whole clipJobs record. updateJobProgress (TimelineGenerationStore.ts:347-365) replaces the map per WebSocket progress message, so the shell re-renders and the subscription-reconcile effect re-runs per message for the entire life of every generation. The store maintains progress-stable generatingClipIds membership arrays for exactly this purpose (TimelineGenerationStore.ts:51-63) — this hook bypasses them. Fix: key the effect on generatingClipIds (or a useShallow projection of id/status/workflow) and read job details via getState() inside the effect. msPerPx (TimelineEditor.tsx:338) — every zoom tick re-renders the shell; only BottomStatusBar needs it. The tracks-height resize drag (TimelineEditor.tsx:397-405) sets React state per mousemove with the same fanout. Fix: push the zoom wiring into a wrapper around BottomStatusBar; apply resize height via ref during the drag, commit on mouseup. Also stabilize TopBar props — onOpenSettings={() => ...} and activitySlot={<ActivityIndicator />} (TimelineEditor.tsx:495-498) defeat its memo every render. 1.4 O(N²) selector churn during drags Every doc-store publish runs every subscriber’s selector. During a drag that is per-pointermove: Every mounted Clip selects s.clips.find((c) => c.id === clipId) (Clip.tsx:396-398) — N clips × O(N) = O(N²) per tick. Same pattern in TimelineInspector.tsx:110, GeneratedClipPanel.tsx:88, ClipVersionHistory.tsx:199, ClipActions.tsx:41, useGenerateClip.ts:375. At 300–500 clips this is tens of millions of comparisons/second mid-drag. Every TrackLane re-filters all clips per lane (TrackLane.tsx:113-122); the custom equality then discards the arrays. attachUiPruning (TimelineInstance.tsx:80-116) builds new Set(state.clips.map(...)) per tick whenever a selection exists — and dragging a selected clip is the common case. TracksRegion.contentEndMs rescans all clips per tick (TracksRegion.tsx:204-223; the cache key is clips identity). Fix: maintain a Map<string, TimelineClip> keyed on clips array identity (module-level WeakMap<TimelineClip[], Map> used inside selectors, or a clipsById field rebuilt in the same set). Turns every id lookup O(1). Defer attachUiPruning to a microtask/idle callback — pruning only matters after removals. 1.5 Transcript projections rebuilt per drag tick, ×3 consumers clips identity changes per drag tick; three consumers rebuild the full transcript projection from it, each O(words · log): ScriptLane.tsx:126-136 — buildTranscriptDoc(clips) per tick, then reconciles a <span> per word TranscriptPanel.tsx:41-42 — same rebuild plus a filler-count reduce TranscriptEditor.tsx SyncPlugin (206, 233-240) — transcriptSignature (80-87) runs buildTranscriptDoc and builds one giant joined string over every word, per clips change, just to detect external edits With the script feature on, a drag pays this 3–5× per pointermove. Dragging a B-roll clip with no captions pays it too. Fix: share one projection via a module-level WeakMap<TimelineClip[], TranscriptDoc>, and gate on transcript-relevant change — select the caption-bearing clip subset with an equality that compares member identities, or bump a transcriptRev counter from caption-touching mutations and key the SyncPlugin check on it. 1.6 Scrub fanout Ruler scrub and playhead drag write reactive currentTimeMs per pointermove (TimeRuler.tsx:457-465, Playhead.tsx:200-214). Repainting the preview per event is intended; the incidental subscribers are not: TimeRuler.tsx:266 subscribes to currentTimeMs solely for aria-valuenow (482) — re-renders the ruler per tick. Set it imperatively via subscribeTime, as Playhead already does. ScriptLane re-renders every phrase chip and word span per tick (ScriptLane.tsx:130, 186-199). Subscribe transiently and toggle classes. While playing, each scrub tick bumps seekNonce, whose effect runs graph.stopAll() + a full async handlePlay() — asset resolution, audio re-scheduling, clock restart — per pixel (PreviewArea.tsx:337-345). Debounce the restart (~100 ms trailing). The paused one-shot composite runs twice per scrub tick — eagerly with a stale frame and again on seeked (PreviewCompositor.tsx:811-828). Skip the eager pass when all video layers are seeking. 1.7 Zoom: per-wheel-event store publish, full re-layout, listener churn TracksRegion.tsx:375-419: every wheel/pinch event calls setZoom — one store publish per event (trackpads deliver 60–120+ Hz), re-rendering every clip (new leftPx/widthPx), lane, ruler, and scrollbar; the scroll correction (444-465) then triggers a second full region render via setScrollLeftPx. The effect re-attaches the wheel listener on every msPerPx change (dep at 419), so events landing before the next render compute from a stale scale (zoom skips). getBoundingClientRect() runs per event (389). Fix: rAF-coalesce setZoom with the latest wheel state in a ref; read msPerPx from a ref inside a stable listener; cache the container rect. For pinch smoothness, apply a temporary transform: scaleX() during the gesture and commit one re-layout at the end. Tier 2 — per-frame playback costs 2.1 Scene model fully recomputed every rAF tick The playback loop (PreviewCompositor.tsx:855-887) calls sceneSignature(liveMs) per frame, which runs computeActiveLayers (sceneModel.ts:226-312): [...tracks].sort, a Map rebuild over all clips, per-track filter().sort(), a ResolvedCaption object per word of every captioned active clip, plus signature string concatenation — usually to conclude nothing changed. On a 6-track/200-clip timeline that is thousands of allocations per second of GC churn inside the playback loop. The scene is also computed twice per actual bump (signature + the useMemo at 434-509). Fix: compute a nextChangeMs horizon with each scene (min of active clip ends, upcoming starts, caption word boundaries) and skip signature work while liveMs < nextChangeMs — steady-state playback becomes one float compare per frame. Or cache sortedTracks/clipsByTrackId in a WeakMap keyed on the arrays. 2.2 GPU effects re-process static layers every frame gpu/compositor.ts:230-244 runs the full compute chain (chroma key → grade → 2× blur → sharpen → vignette) unconditionally per render(). The rAF loop marks frames dirty whenever any video plays, so a static image with a heavy blur under a playing video is re-blurred 60×/s. Fix: cache the processed output per layer keyed on (source upload key, effects array identities — they are reference-stable until edited); return the cached texture on match. Videos self-invalidate via currentTime. 2.3 WebGPU micro-churn per frame setLayers re-sorts a copied array and rebuilds a prune Set every tick while video plays (gpu/compositor.ts:129-143) — skip when layer ids and order are unchanged. renderBlendPass creates a fresh GPUBindGroup + three createView() calls per layer per frame (packages/gpu/src/compositor/compositor.ts:340-353, plus blit 367-370). Views are immutable per texture; cache them and key bind groups on (source texture, ping-pong read texture). 2.4 Playhead writes style.left and churns attributes per frame Playhead.tsx:152-166: left invalidates layout each frame where transform: translateX() would stay on the compositor; aria-valuenow and the pill textContent mutate 60×/s. Throttle both to ~10 Hz; switch to transform. 2.5 Karaoke plugins re-query the word DOM per time tick TranscriptEditor.tsx — ActiveWordPlugin (330-355) and ScriptCaretPlugin (382-430) each run querySelectorAll(".transcript-word") + dataset string parsing + a linear scan per transient time tick (×2, 60 Hz), and the caret plugin interleaves style writes with offsetLeft/offsetWidth reads — forced reflow per frame. Both re-run their full setup on every clips identity change (deps at 354/430). Fix: build one shared sorted {el, startMs, endMs, offsets} cache per reseed (offsets refreshed on resize), binary-search the active word per tick, and reuse the cache in SelectionHighlightPlugin (445-455) and arrow-key stepping (621-637). Tier 3 — scaling limits (large projects) 3.1 No virtualization TrackLane.tsx:476-478 mounts every clip on every track regardless of the visible window. Each clip carries ~8–12 DOM nodes, three asset-URL effects (Clip.tsx:963-1015), a 24-frame thumbnail request per video URL (useClipThumbnails.ts:12), and a fetch+decode for audio waveforms — for clips hours off-screen. Every per-tick cost in Tier 1 scales with this total. Fix: window horizontally from scrollLeftPx/msPerPx (both already in the UI store — the visibility predicate is trivial from startMs/durationMs) with one viewport of overscan; window tracks vertically; gate thumbnail/peaks requests on visibility. 3.2 Play gesture decodes every future audio clip before the clock starts PreviewArea.tsx:261-304 filters all remaining audio clips and scheduleClips Promise.alls a fetch + decodeAudioData for each (AudioGraph.ts:406-414) before clock.start. Play latency grows with timeline length; peak memory is the whole timeline’s decoded PCM; and with more than BUFFER_CACHE_MAX = 16 distinct assets (AudioGraph.ts:43) the LRU evicts and re-decodes on every play/seek gesture. Seek-while-playing repeats everything. Fix: schedule only clips starting within a lookahead window (~30 s, matching PRELOAD_LOOKAHEAD_MS), top up as the playhead advances, and start the clock once the currently audible clips are ready. 3.3 Caption raster cache: full-frame bitmaps, up to ~530 MB captionRender.ts:19, 138-166: each caption state rasterizes at full sequence resolution (a lower-third strip inside a 1920×1080 bitmap); MAX_CACHE_ENTRIES = 64 × 8 MB ≈ 530 MB worst case (4K: ~2 GB). A new OffscreenCanvas is allocated per miss, and on the GPU side each new bitmap destroys and re-creates a full-frame texture per word change (gpu/compositor.ts:155-160) instead of re-uploading into the same-sized one. Fix: rasterize at the caption bounding box and position via the layer transform; reuse one OffscreenCanvas; re-upload into the existing texture when dimensions match; cap the cache by bytes. 3.4 Unbounded caches and export memory clipThumbnails.ts:31, 197-215: module-level Map of 24 base64 JPEGs per video URL, never evicted; failed entries permanently poisoned. Use blob URLs + LRU; allow retry. OffscreenVideoPool.ts:29, 92-113: export accumulates one <video> per video clip, never released until dispose — 50-clip exports risk hitting browser media-element and hardware-decoder caps. Release entries behind the export playhead. TimelineRenderer.ts:179-182, 303: BufferTarget holds the whole MP4 in memory. Fine today; a streaming target bounds it. ResultsStore.ts:704-722 (setProgress): growing string concat + whole record spread per WebSocket chunk message. Accumulate chunks in an array; coalesce progress writes to ~10 Hz. 3.5 Export frame loop: serialized seeks, main-thread TimelineRenderer.ts:248-266 awaits videoPool.seek() sequentially per layer; overlapping videos pay seek round-trips back-to-back. Issue seeks with Promise.all. Longer term, move the loop to a Worker with OffscreenCanvas. Tier 4 — smaller wins Filmstrip cells hash multi-KB data URLs through emotion per render (Clip.tsx:337-345, 1090): css({ backgroundImage: url(<base64>) }) per cell per render — emotion hashing is O(string length) and every distinct URL inserts a permanent CSSOM rule. Runs per pointermove during trims. Use style={{ backgroundImage }} exactly as the image path at 1060 does. TracksRegion subscribes to scrollLeftPx and selectedClipIds (TracksRegion.tsx:227, 231): whole-region re-render per pan frame and per rubber-band selection change; selection is only used in event handlers — read getState() there; let the memoized TimelineScrollbar subscribe to scroll itself. Every Clip subscribes to the entire ErrorStore.errors record (Clip.tsx:457-475): any error anywhere re-renders every clip and re-scans all error keys per clip. Narrow to a per-run derived lookup. Inspector sections stay mounted when folded (CollapsibleSection.tsx:118-120, MUI Collapse without unmountOnExit): ~30 control rows render per inspector re-render regardless of fold state. Add unmountOnExit/lazy children. ClipAdjustments passes fresh lambdas to every memoized row (ClipAdjustments.tsx:167-169, 441-444 and the IIFE blocks at 252, 396, 501, 572): dragging one slider re-renders all rows. Give rows (clipId, field) props or useCallback per field. DirectGenClipPanel subscribes to the whole clips array (DirectGenClipPanel.tsx:78, 104-117) to build a dropdown list; recomputed per doc change even outside image-to-image mode. Select {id,name} pairs with useShallow, gated on mode. ClipListPopover subscribes to clips while closed (ActivityIndicator.tsx:83, 205-239): during generation, every clip patch re-renders two closed popovers. Render only when open. Right-edge drags churn totalWidthPx (TracksRegion.tsx:204-223, 320-323, 735): dragging the last clip re-layouts the scroll area per move. Quantize the width (e.g. 256-px steps) or freeze during gestures. contentEndMs/boundary work in PreviewArea (PreviewArea.tsx:201-207, 374-385): re-render + O(n log n) boundary sort per drag tick; clips is otherwise only used in click handlers. Select primitives; read getState() in handlers. Cold-pool video preload never fires mid-clip (PreviewCompositor.tsx: 553-683): the effect’s time dep advances only on scene bumps, so the next clip isn’t preloaded until its boundary; upcomingVideoClips picks clips in array order, not soonest-first; and activation reloads the element from scratch, discarding the warm decoder. Re-evaluate on a coarse timer, sort by startMs, and swap the cold element into the hot slot. Autosave can flush mid-gesture (useTimelineAutosave.ts:211-216): holding a drag still for >750 ms fires a wasted full-document PATCH (which serializes every caption word). Gate the flush on “no history batch open”. Related: useTimelineSave.ts:33-38 omits transcript from the manual-save PATCH while autosave includes it — a correctness risk if the server replaces the whole document. Rubber-band and gizmo read getBoundingClientRect per pointermove (TrackLane.tsx:394, TransformGizmoOverlay.tsx:330-333): cache at pointerdown. TimelineListPanel: inline onCancelRename defeats item memo (:572); sort comparator allocates two Date objects per comparison (:374-376). useCallback + precompute getTime(). instanceStoreHook.ts:18-20 footgun: the wrapper silently drops a second (equality) argument — useTimelineStore(sel, shallow) compiles and ignores shallow. Forward it. el.src = "" teardown (PreviewCompositor.tsx:307-308) resolves to the document URL and can fire a spurious request; use removeAttribute("src"); load() as OffscreenVideoPool.dispose does. AudioContext instantiated on first play even with zero audio clips (PreviewArea.tsx:257-258). Gate on activeAudioClips.length > 0. What is already done well (do not “fix”) Transient playhead channel + PlaybackClock: zero React work per frame; Playhead, control-bar timecode, and karaoke highlighting are imperative. Store split (document / UI / playback, per-instance via TimelineInstance) with a memoized provider bundle — disjoint subscriber sets, no context churn. Identity-preserving reducers everywhere (patchClip/patchById/moveClip return same state on no-ops; untouched clips keep identity) — memoized siblings don’t re-render during another clip’s drag. Undo: zundo partialize stores references (structural sharing, no deep clones); useTimelineHistoryBatch collapses drag/trim/resize gestures. Autosave: transient store.subscribe, O(1) reference dirty-check, 750 ms debounce, single-flight — zero React renders. Clip.tsx gesture internals: snap candidates snapshotted per gesture, elementsFromPoint rAF-coalesced, waveform redraw rAF-coalesced with whole-pixel gating; TimeRuler draws a viewport-sized canvas from a latest-inputs ref. Compositor: scene-signature gating, dirty-flag composite skip for static scenes, stable clip→slot binding, LRU image cache, asset-URL cache with failed states, caption bitmap WeakMap signature memo. GPU: pipelines/samplers cached (no per-frame shader compilation anywhere), uniform-buffer ring, effects intermediate-texture pooling, seek-guard upload keying. AudioGraph: in-place param updates when the chain shape matches, ref-equality short-circuits, click-free ramps, in-flight decode dedupe. TimelineGenerationStore deliberately keeps generatingClipIds stable across progress ticks; TrackEffectsPanel mounts lazily; AddClipMenu mounts on demand with enabled-gated queries. No JSON.parse(JSON.stringify), no polling loops, TanStack Query keys stable with enabled guards throughout. Priority order # Fix Files Effort Payoff 1 Commit-on-release / history-batch for InspectorSliderRow + ParamRow + EQ/compressor + gizmo InspectorPrimitives, TrackEffectsPanel, TransformGizmoOverlay S–M Ends undo-history destruction; removes the densest store-write path 2 useTimelineExport → getState() reads; throttle export progress useTimelineExport S Stops whole-shell re-render per drag tick and per encoded frame 3 useTimelineGenerationSubscriptions → key on generatingClipIds useGenerateClip S Stops whole-shell re-render per progress message during generations 4 clipsById map (WeakMap-keyed) for all id selectors TimelineStore + call sites S O(N²)→O(N) selector work per drag tick 5 Shared WeakMap transcript projection + transcript-relevant gating ScriptLane, TranscriptPanel, TranscriptEditor M Removes O(words) rebuild ×3 per drag tick 6 Scene nextChangeMs horizon in the rAF tick PreviewCompositor, sceneModel M Removes per-frame scene recompute + GC churn during playback 7 rAF-coalesced zoom with ref-read scale TracksRegion S One React commit per displayed frame while zooming 8 Scrub hygiene: ruler aria via subscribeTime, debounced seekNonce audio restart, ScriptLane transient highlight TimeRuler, PreviewArea, ScriptLane S–M Cheap scrubbing, no audio-graph rebuild per pixel 9 Windowed audio scheduling PreviewArea, AudioGraph M Play latency stops scaling with timeline length 10 Effects output cache for static layers gpu/compositor, effectsProcessor M Removes 60 fps GPU work on static effected layers 11 Caption strip-sized rasterization + texture reuse captionRender, gpu/compositor M ~10–20× caption memory/upload reduction 12 Clip virtualization + visibility-gated media fetches TrackLane, TracksRegion, clipThumbnails L Caps all per-tick costs at viewport size; biggest ceiling 13 Tier-4 batch (filmstrip inline style, selector narrowing, memo fixes, cache bounds) various S each Broad small wins Items 1–4 are small, independent, and remove the worst interactive behavior; they are the right first PR. Item 12 is the largest single ceiling-raiser for big projects but should land after 4 so the windowing predicate can reuse the id map.--- # Timeline motion design Source: https://docs.nodetool.ai/timeline-motion-design.html Timeline motion design This guide describes the timeline document fields available to composition authors. These are document examples. The editor does not expose controls for every field. The protocol schema and renderer are the source of truth when a field is absent here. Easing and animation Animation easing accepts named curves, cubic-bezier(x1,y1,x2,y2), and spring(stiffness,damping,mass). Named curves include linear, easeIn, easeOut, easeInOut, easeOutBack, easeOutElastic, easeOutBounce, easeInExpo, easeOutExpo, easeInOutExpo, easeInQuint, easeOutQuint, easeInOutQuint, easeInCirc, easeOutCirc, easeInOutCirc, and hold. Unknown easing strings resolve to linear. Easing may be set on an animation or on an individual keyframe segment. { "role": "in", "preset": "slide", "durationMs": 700, "easing": "cubic-bezier(0.16,1,0.3,1)", "params": { "direction": "up", "distance": 0.12 } } spring(...) is an easing curve for an authored animation segment, not a physical simulation that responds to changing forces. The animation engine’s numeric curves drive transform, opacity, blur, grade, wipe, shape trim, and 3D model camera channels. For arbitrary visual properties use styleTracks: { "role": "in", "preset": "fade", "durationMs": 600, "styleTracks": [ { "target": "text.color", "keyframes": [ { "t": 0, "value": "#8bd5ff" }, { "t": 1, "value": "#ffffff", "easing": "easeOutExpo" } ] }, { "target": "effect.glow-1.intensity", "keyframes": [ { "t": 0, "value": 0 }, { "t": 1, "value": 1.2 } ] }, { "target": "effect.bg-field.colorB", "keyframes": [ { "t": 0, "value": "#1e1b4b" }, { "t": 1, "value": "#0f3b3a" } ] } ] } Targets include supported numeric shape and text style fields, including font size, weight, and spacing, shape/text colors, gradient fill angles and stop values, effect.<effect-id>.<numeric-field>, supported effect color fields, mask bounds and feather, compatible shape.d / mask.d SVG paths, clip.borderRadius, mask.radiusPx, and per-glyph color, blur, or tracking. Color keyframes accept the CSS colors supported by the shared parser, including named colors, hex, RGB, and HSL. A clip must have the matching style, mask, or effect. Path morph endpoints must have the same SVG command and point count structure. Glyph tracks require a text stagger. Links are the supported cross-property mechanism. Arbitrary expressions are not evaluated at render time. Effect color targets are generator.colorA, generator.colorB, and color on dropShadow, chromaKey, or the color-using stylize modes (lightRays, lensFlare, innerShadow, innerGlow, edgeHighlight, and lightLeakOverlay), addressed through each effect’s ID. The targeted color field must already be present. String fields such as an effect’s mode and the currently unused glow.color cannot be animated by a style track. Text content can count or decode over the animation window: { "role": "in", "preset": "fade", "durationMs": 900, "textAnimator": { "kind": "ticker", "from": 0, "to": 2847, "groupSeparator": "," } } Ticker also accepts decimals, padTo, prefix, and suffix. groupSeparator inserts the given 1–4 character string between groups of three integer digits, after zero padding. Without it, existing tickers remain ungrouped. A deterministic scramble can be authored with {"kind":"scramble","charset":"ABC123","seed":17}. Both operate on a text clip’s content during the animation. Per-unit transform and opacity animation supports character, word, or wrapped-line stagger, with order start, end, or center: "stagger": { "unit": "word", "offsetMs": 90, "from": "center" } Unknown stagger units fall back to block animation. Existing text path support sets textStyle.path to an SVG path in normalized canvas coordinates. The text baseline follows that path. It is not a new per-character motion-path animator. The followPath animation preset moves a clip along its authored path parameters. An animation can start on a beat by storing beat on the animation. index is one-based. scope: "clip" counts from the clip start. scope: "sequence" counts from the sequence tempo’s offsetMs. offsetMs shifts the selected beat, and delayMs remains an additional animation delay. Beat spacing uses the sequence BPM (or the default tempo if none is set). { "role": "emphasis", "preset": "pop", "durationMs": 240, "beat": { "index": 3, "scope": "clip", "offsetMs": -20 } } To stagger selected clips in a visual order without moving their media, call ui_timeline_stagger_animations with: { "clip_ids": ["logo", "wordmark", "tagline"], "offset_ms": 120 } List clip IDs, not names. Every selected clip must already have an animation. The first listed clip keeps its animation delays. Each later clip adds its index multiplied by non-negative offset_ms to each animation’s delayMs. Property links Links let a clip read a transform or opacity channel from another clip, with optional clock offset, looping, scale, and offset. A deterministic wiggle is also available. Links read the source’s authored animation and do not chain through other links, so cyclic dependencies are not evaluated. { "animationLinks": [ { "target": "positionX", "sourceClipId": "camera-card", "source": "positionX", "timeOffsetMs": 0, "scale": 0.35 }, { "target": "rotation", "kind": "wiggle", "amplitude": 0.025, "frequencyHz": 1.4, "seed": 8 } ] } Channels are positionX, positionY, scale, rotation, and opacity. Links are per clip and support up to 32 entries in the document schema. Space, layout, and timing camera2d is optional on the sequence document, persists through timeline load, autosave, update, and document versioning, and may be set to null to clear it. Layers with transform.depthPx participate in camera perspective. The camera position and depth can be keyframed on absolute timeline times. focusDepthPx and aperturePx set a shallow depth-of-field blur. { "camera2d": { "position": { "x": 0, "y": 0 }, "depthPx": 0, "focalLengthPx": 1400, "focusDepthPx": 0, "aperturePx": 18, "keyframes": [ { "timeMs": 0, "position": { "x": 0, "y": 0 }, "depthPx": 0 }, { "timeMs": 1200, "position": { "x": 90, "y": -20 }, "depthPx": 120 } ] } } Camera keyframes interpolate linearly. rotationX, rotationY, and perspective on a clip transform provide flat-layer tilt. Layout supports row or stack containers listing child clip IDs, and relative placement against another clip’s box. fitText sizes the relative clip to text plus padding. Boxes are resolved from current text or shape content at render time. { "layout": { "kind": "row", "children": ["logo", "wordmark"], "gapPx": 24 } } Repeater copies advance by position and time, with an optional hue and brightness step. count includes the original clip. The renderer caps copies at 128. A repeater copy is a regular rendered clip with its own generated ID. "repeater": { "count": 6, "positionStep": { "x": 180, "y": 0 }, "timeStepMs": 67, "colorStep": { "hueDegrees": 8, "brightness": -0.02 } } Motion blur may be configured per clip with samplesPerFrame (1–32) and shutterAngle (0–360 degrees). Preview and export evaluate each clip at its own shutter sample times, so a clip can blur independently of other layers. A value of 1 disables blur for that clip. Higher values set the minimum sample count. Blurred clips share the scene’s highest requested count, with evenly weighted samples across each clip’s own shutter window. temporalEcho adds up to 32 delayed copies. steppedTime quantizes that clip’s sampling clock. Effects and transitions The effect schema includes color grade, blur, glow, drop shadow, vignette, sharpen, chroma key, curves, levels, lift/gamma/gain, grain, pixelate, posterize, directional blur, lens distortion, LUTs, stylize modes, and generators. Stylize modes include RGB split, angular radial blur, zoom blur, displacement, turbulence, glitch, halftone, optional dither, light rays, lens flare, inner shadow/glow, and edge highlight. Generator modes include noise, fractal, conic and mesh gradients, animated gradient fields, particles, light leak, and grid pattern. A compact example: "effects": [ { "id": "grain-1", "type": "grain", "enabled": true, "amount": 0.12, "size": 1.1, "seed": 4 }, { "id": "bg-field", "type": "generator", "enabled": true, "mode": "gradientField", "scale": 6, "time": 1.2, "animate": true, "seed": 9, "colorA": "#10233c", "colorB": "#ec6b45" }, { "id": "grade-lut", "type": "lut", "enabled": true, "intensity": 0.8, "cube": "LUT_3D_SIZE 2\n0 0 0\n1 0 0\n0 1 0\n1 1 0\n0 0 1\n1 0 1\n0 1 1\n1 1 1" } ] An effect must use the fields in its effect type. For example, blur uses radius, glow uses radius and intensity, and generator accepts mode, scale, time, seed, and color fields. .cube LUTs accept 3D tables. 1D tables are rejected. Store the file contents in cube through the document or effect tools. The editor does not yet provide a LUT file picker. The Canvas compositor implements the effect chain through CPU pixel passes. Hosts must supply scratch surfaces. Missing surfaces and unknown effect types are reported as rendering limitations. Compositing blend functions operate on the stored encoded-sRGB channel values. The blend calculation does not linearize to light before combining layers. The compositor retains premultiplied alpha for the composite step. For dark gradients, an optional stylize effect with mode: "dither", amount: 1, and a fixed seed adds reproducible noise around the stored 8-bit values. This reduces visible banding but does not recover precision already lost in the source or guarantee that a video encoder preserves it. Transitions live on the incoming clip as transitionIn, with durationMs. On a group clip, the transition applies to one precomposed surface containing its children. Keep children parented to the group on their own tracks. An overlapping previous sibling on the same track and under the same parent is the outgoing partner; without one, the group enters over the existing frame. Supported types are crossfade, dipToColor, wipe, push, slide, zoom, whip, zoomBlur, glitch, gradientWipe, iris, and lightLeak. Type-specific fields include color for dipToColor and lightLeak, direction for wipe, push, slide, whip, and gradient wipe, and softness for wipe, gradient wipe, and iris. gradientWipe thresholds a spatial field selected by map: "linear" | "radial" | "noise". scale sets noise frequency and seed fixes its pattern. lightLeak overlays a localized color field rather than covering the frame with a solid. Its scale sets falloff and seed shifts the field. Easing uses the same grammar as animation easing. "transitionIn": { "type": "gradientWipe", "durationMs": 420, "direction": "left", "map": "noise", "scale": 8, "seed": 7, "softness": 0.12, "easing": "easeInOutExpo" } Failure modes These authoring mistakes render without an error. The frame is wrong, and the render does not say why. nodetool timeline validate reports the ones with a code. Symptom Cause Fix Code Every letter or word of a staggered text moves at once. The compiler fits durationMs plus every unit’s delay into the time left in the clip. When the span does not fit, it shrinks the offset, down to 0. An animation as long as its clip, or an "out" animation that ends mid-clip, leaves no room. Use role "in" with delayMs + durationMs + (units − 1) × offsetMs inside the clip. End the curves and style tracks at rest. stagger_compressed A generator or stylize effect with animate: true stays still. animate stamps the frame time into the effect’s time. Some modes never read it, such as conicGradient, which turns only through angle. Animate the field the mode reads, such as effect.<id>.angle, with a style track. effect_animate_ignored A stylize gradientWipe changes shape over time. That mode reads time as its map selector. animate stamps seconds into it, so the map goes from linear to radial to noise. Set time and leave animate off. effect_animate_ignored A color effect turns the picture solid white or black. brightness is an offset from -1 to 1 where 0 is unchanged; contrast and saturation use 1 as unchanged. Reusing the “1 is neutral” convention for brightness sends the value that blows the picture out. Leave brightness near 0; use contrast/saturation near 1 for a subtle grade. effect_washes_out A wiggle or a follow link removes the clip’s own motion or placement on that channel. A link sets the channel’s value. It does not add to the clip’s animations or transform, and a wiggle centres on the channel’s identity. Put the link on a child clip and the other motion on its parent group. animation_link_overrides Repeater copies show on top of the next clip on the same track. Each copy starts index × timeStepMs later and keeps the original duration, so the copies run past the original’s end. Keep (count − 1) × timeStepMs inside the gap, or parent the repeater to a group whose window ends at the cut. clips_overlap A spinning layer that is tilted in 3D wobbles instead of turning flat. The clip’s own rotation turns the picture after its own rotationX tilt, in the screen plane. Put rotationX and perspective on a parent group. Put the spin on the child.   A generated green-screen subject keeps a dark smudge under it. chromaKey matches by RGB distance. A floor shadow is darker than the key colour and survives the tolerance. Add a rect mask that ends above the floor.   A lens flare, light rays or a light leak does not reach the background. The lighting stylize modes (lightRays, lensFlare, innerGlow, edgeHighlight, lightLeakOverlay) add light only where the layer has alpha. Put the effect on an adjustment clip, which treats the composite below it.   Composition examples The repository includes stored composition examples for title-slam, word-cards, window-frame, number-ticker, lower-third, and logo-sting. See the example compositions. They are starting documents, not a guarantee that every editor surface exposes each authoring field as a control.--- # Timeline Preview Decoding: What Other Web Editors Do Source: https://docs.nodetool.ai/timeline-preview-decoding-research.html Timeline Preview Decoding: What Other Web Editors Do Question How do other web video editors and compositors get video frames for preview playback? Specifically, do they drive preview from <video> elements or from a WebCodecs/WASM decode pipeline, and what clock, prefetch, seek, and fallback designs do they use? The NodeTool timeline preview plays <video> elements, syncs them to an AudioContext clock, and composites them in WebGPU (web/src/components/timeline/preview/PreviewCompositor.tsx). Export decodes with mediabunny and WebCodecs (web/src/components/timeline/render/SequentialVideoSource.ts) and falls back to seeked <video> elements for unsupported codecs and retiming (render/OffscreenVideoPool.ts). The decision is whether preview moves to a pull-model WebCodecs frame source. Conclusion Every engine with published source that targets editing and was updated in 2025–2026 decodes preview frames with WebCodecs through mediabunny or a mediabunny-like iterator: Remotion @remotion/media, Diffusion Studio Core v4, and WebAV. The engines that still play <video> elements in preview are animation tools first (Motion Canvas, Revideo, Etro) or a project with the same preview/export split NodeTool has today (Omniclip). The large commercial editors that published anything (CapCut, Kapwing, Clipchamp) describe WebCodecs or WASM decoding, but none of them documents its preview loop. The evidence supports the move. No engine examined solves <video> drift better than NodeTool’s current rate-correction loop. The engines that have frame-exact preview get it by not using <video> for preview at all. Two design choices differ between the WebCodecs engines, and NodeTool must pick one for each: Clock owner. Diffusion Studio uses AudioContext.currentTime as the master clock. Remotion and WebAV use a wall-clock frame counter and schedule audio against it. Late frame policy. Remotion stops the clock and enters a buffering state until the frame is ready. Diffusion Studio and the mediabunny player example draw the best frame they have and keep going. WebAV skips the render tick. Comparison Engine Preview frame source Master clock Late frame Seek / backward Export decoder Fallback Remotion @remotion/media <Video> mediabunny CanvasSink iterator, drawn to <canvas> performance.now() frame counter. Audio scheduled against an AudioContext anchor Pauses the Player (“buffering”) until the frame decodes Sequential advance waits. Any jump or backward step restarts the iterator Same mediabunny path <OffthreadVideo> (FFmpeg) for unsupported codecs Diffusion Studio Core v4 mediabunny CanvasSink plus a frame cache. getBestFrameFor(t) AudioContext.currentTime minus an offset Draws the best cached frame, never blocks seekTo restarts the iterator and records the next key packet Same engine, WebCodecs encode Not documented WebAV (MP4Clip.tick) mp4box demux, VideoDecoder, queue of up to 10 decoded frames 30 fps worker timer, corrected to performance.now() Skips the tick while a preview frame is pending Backward or more than 3 s forward resets the decoder Same tick path Retries in software decode after a hardware decode error mediabunny media player example CanvasSink.canvases(t) iterator, one frame of lookahead AudioContext.currentTime Draws a late frame at once, then pulls the next New iterator per seek n/a n/a Omniclip <video> elements requestAnimationFrame timecode n/a Sets currentTime, awaits seeked WebCodecs VideoDecoder in a worker n/a Motion Canvas Playing <video> element Scene frame clock n/a Re-seeks when drift exceeds 0.2 s Seek per frame on <video> n/a Revideo Playing <video> element Scene frame clock n/a Re-seeks when drift exceeds 1 s web (WebCodecs, MP4 only), ffmpeg, or slow (seeked <video>) ffmpeg for WebM Etro <video>/<audio> elements performance.now() n/a Sets currentTime Same elements, recorded n/a CapCut web WebCodecs decode drawn to the editing canvas (C++ engine via Emscripten) Not published Not published Not published Not published Earlier WASM SIMD decoders Kapwing Moved frame decoding from <video> to WebCodecs in workers, mp4box demux Not published Not published Decodes forward from the preceding keyframe Not published Not published Clipchamp Not published. The 2021 talk covers export only Not published Not published Not published FFmpeg WASM with WebCodecs codec stubs Software or server encode ByteDance NLE (W3C 2021) WASM decode to WebGL textures, 480p limit Web Audio mentioned, not detailed Not published Not published Not published n/a Canva, Descript, Veed, and Adobe have no primary source about their preview pipeline. A Canva job listing mentions a “native video engine” for web, but no decoder detail. Evidence Remotion The docs comparison table says @remotion/media <Video> uses “Mediabunny and WebCodecs” and is frame-accurate in preview and render. The older <Html5Video> is “not guaranteed” frame-accurate. <OffthreadVideo> plays a browser <video> in preview and extracts exact frames with FFmpeg only at render time (video tags). <Video> “extracts the exact frame from the video using Mediabunny” and draws it to a <canvas>. premountFor mounts a clip early so it can “buffer before it becomes visible”. Reverse playback is not supported. If the codec is unsupported, it falls back to <OffthreadVideo> (Video docs). Source, packages/media/src/video-iterator-manager.ts (commit 102a548): a sequential forward step uses pendingFrameBehavior: 'wait', and any other time change uses 'restart-iterator'. When the frame is pending, it calls delayPlaybackHandleIfNotPremounting(), which puts the Player into a buffering state. Within 1 s of a loop end, it prewarms a second iterator. Source, packages/player/src/use-playback.ts: the frame comes from performance.now() - startedTime. The frame does not advance while isBuffering() is true. set-global-time-anchor.ts maps timeline time to audioContext.currentTime and re-anchors only past a tolerance, “to avoid audio glitches from frame-quantized re-anchoring”. Diffusion Studio Core v4 The README describes “interactive playback for editing and a high fidelity rendering mode”, built on Canvas 2D and WebCodecs (repository). The repository no longer contains source. The findings below come from the published @diffusionstudio/core@4.0.3 bundle. package.json depends on mediabunny ^1.25.0. dist/clips/video/decoder.d.ts declares a VideoDecoder class with canvasSink, packetSink, frameCache, nextKeyPacket, seekTo(), and getBestFrameFor(). In core.es.js, playbackTime is audioCtx.currentTime - hardwareOffset + playbackOffset. The clip render() calls getBestFrameFor(playbackTime - delay) and returns without drawing if no frame exists. A clip primes its decoder with seekTo(range[0]) before it becomes active. WebAV Source, packages/av-cliper/src/clips/mp4-clip.ts (commit 42dd6fd): the frame finder resets the decoder when time <= lastTime or when the jump is more than 3 s. Otherwise it pops queued frames until it reaches the one that covers time, and it starts more decoding when fewer than 10 frames are queued. On a decode error with no output yet, it logs “Downgrade to software decode” and resets. Source, packages/av-canvas/src/av-canvas.ts: a 30 fps worker timer drives render, corrected against performance.now(). It skips the tick while #waitingPreviewFrame is set, “to avoid flicker from seeking back and forth between clips” (translated from the comment). Each tick collects PCM from the clips and schedules it on the AudioContext. mediabunny VideoSampleSink.getSample(t) “returns the last sample with a timestamp less than or equal to the search timestamp”. CanvasSink with poolSize reuses canvases round-robin, which “keeps the amount of allocated VRAM constant” (media sinks). Source, examples/media-player/media-player.ts (commit 9d36fc5): the clock is audioContext.currentTime - audioContextStartTime + playbackTimeAtStart. Each requestAnimationFrame draws nextFrame once its timestamp is at or before the clock, then pulls frames until one is in the future. Audio buffers are scheduled ahead with node.start() and throttled to 1 s of lead. <video>-based engines Motion Canvas packages/2d/src/lib/components/Video.ts (commit 7b91435): during playback it plays the element and sets currentTime only when Math.abs(video.currentTime - time) > 0.2. When paused or rendering, it seeks and awaits seeked. Revideo packages/2d/src/lib/components/Video.ts (commit b5de67a): playback uses the playing element and re-seeks when the drift is more than 1 s. Rendering selects webcodecSeekedVideo, ffmpegSeekedVideo, or the seeked element. Omniclip s/context/controllers/compositor/ (commit cbe581a): preview creates <video> elements and sets currentTime on seek. Export uses a VideoDecoder worker (video-export/parts/decode_worker.ts). Commercial editors CapCut: “decoding a 4K image to the editing canvas on a high-performance computer takes tens of milliseconds” without hardware decode. WebCodecs gave hardware decode, and “CapCut now supports multiple simultaneous 4K streams” (web.dev case study). Kapwing: “The bottleneck to achieving smooth frame drawing is decoding frames, which until recently we did using an HTML5 video player.” The performance “wasn’t reliable”. “Recently we have moved over to WebCodecs, which can be used in web workers” (web.dev case study). Clipchamp: the 2021 talk covers the export pipeline (decoder, compositor, encoder) with FFmpeg WASM and WebCodecs encoder stubs. It does not describe preview (W3C talk). ByteDance: “For each video track, we first use WebAssembly to decode the video frame”, composited in WebGL, limited to 480p because of CPU cost (W3C talk). Platform limits of <video> “The video element does not guarantee frame-accurate seeking.” The requestVideoFrameCallback API is best-effort and can be one vsync late (web.dev rVFC). The W3C issue on frame-accurate seeking has stayed open since 2018. In that thread, a Remotion maintainer says requestVideoFrameCallback “doesn’t lead to perfect results always” (w3c/media-and-entertainment#4). Implications for NodeTool (inference) These points are inferences from the evidence above, not observations. Use mediabunny for preview, as for export. Remotion and Diffusion Studio build preview on the same CanvasSink/VideoSampleSink iterators that SequentialVideoSource already uses. Use one frame source for both paths. Keep the AudioContext as master. Diffusion Studio and the mediabunny example do this, and NodeTool already does. Remotion’s frame-counter clock suits a frame-stepped composition, not continuous audio playback. Pick a late frame policy per mode. During playback, draw the best available frame (Diffusion Studio). When paused or scrubbing, wait for the exact frame. Remotion’s stop-the-clock buffering is right for a final preview but makes editing feel stalled. Prewarm the next clip. Remotion (premountFor, loop prewarm) and Diffusion Studio (decoder priming) both open the decoder before the cut. Restart the iterator on any non-sequential move. Every WebCodecs engine does this. Reverse playback has no published solution: Remotion does not support it, and the others restart per step. Keep <video> as a fallback only. Remotion falls back to FFmpeg for unsupported codecs, WebAV falls back to software decode, and NodeTool export falls back to seeked elements. The current drift loop can stay as that fallback path. Uncertainty and gaps CapCut, Kapwing, Clipchamp, Canva, Descript, Veed, and Adobe do not publish their preview clock or prefetch design. The table marks those cells as not published. The Clipchamp and ByteDance talks are from 2021. Their current architecture can differ. The Diffusion Studio findings come from a minified bundle, not readable source. No source measures the cut latency or memory use of a WebCodecs preview with many concurrent clips. NodeTool must measure that itself.--- # Tips and Tricks Source: https://docs.nodetool.ai/tips-and-tricks.html Shortcuts, hidden features, and workflow efficiency tips. Essential Shortcuts Shortcut Action Space Open node menu Ctrl/⌘ + Enter Run workflow Esc Stop workflow Ctrl/⌘ + S Save F Fit view Ctrl/⌘ + Z Undo Ctrl/⌘ + K Command menu Ctrl/⌘ + / Show all keyboard shortcuts Quick Node Actions Adding Nodes Faster Search by typing: Press Space, then just type what you need (“agent”, “image”, “whisper”) Smart connect: Drag a connection to empty space → get compatible node suggestions Duplicate quickly: Ctrl/⌘ + D copies selected nodes horizontally Model Selection Recommended Models: Click the “Recommended Models” button on AI nodes to see compatible options Quick switch: Use the Model dropdown to change models without rewiring Node Organization Drag from header: Move nodes by grabbing the header bar (top of node) Group related nodes: Select multiple, press Ctrl/⌘ + G Align selection: Press A to align, Shift + A to align with even spacing Collapse a node: Press C Disable a node: Press B Select connected nodes: Shift + C (all), Shift + I (inputs), Shift + O (outputs) Duplicate vertically: Ctrl/⌘ + Shift + D Canvas Navigation Action How Pan around Left-drag on empty canvas (Windows/Linux default) or right/middle-drag (macOS default). Change it in Settings → General → Canvas → Left-Click Drag Zoom Scroll wheel (Windows/Linux), pinch or Ctrl/⌘ + scroll (macOS, where plain scroll pans) Fit to screen Press F Focus on selection Select nodes, then press F Reset zoom Ctrl/⌘ + 0 Zoom in / out Ctrl/⌘ + = / Ctrl/⌘ + - Snap to grid Enable in the desktop View menu, the command menu, or Settings → General → Canvas Connections Making Connections Type matching: Colors show compatible connections Quick connect: Drop on empty space for auto-suggestions Multi-output: One output can connect to multiple inputs Connection Tips Preview intermediate data: Add Preview nodes between connections Disconnect: Right-click connection or drag it away Re-route: Delete and recreate, or drag to a new target Workflow Management Organization Save often: Ctrl/⌘ + S – autosave is on by default, and each manual save also creates a version (Settings → General → Autosave) Use descriptive names: Rename workflows and nodes for clarity Examples: Open the Examples page from the sidebar menu to start from a working graph History & Undo Action Shortcut Undo Ctrl/⌘ + Z Redo Ctrl/⌘ + Shift + Z Saved versions Created on each manual save Layout Recovery Auto layout: Click the Auto Layout button (or use the command menu) to tidy up Debugging Workflows Finding Problems Add Preview nodes between steps to see data at each stage Check node errors – red borders or icons indicate issues Verify connections – ensure types match Test incrementally – run partial workflows first Disable nodes – select and press B, or right-click → Disable Node, to exclude suspicious nodes and their connections from the run Disabling Nodes for Debugging Isolate issues: Disable a failing branch to run a smaller graph Keep unfinished work: Leave nodes on the canvas without executing them Skip slow steps: Temporarily disable heavy processing during testing Common Fixes Problem Solution A node reports a missing model Use the model picker’s Recommended Models list or Model Manager to install it Wrong output Check input data and node settings Workflow won’t run Look for disconnected required inputs Slow execution Try cloud providers for heavy tasks Node causing errors Disable its branch and run the remaining graph Power User Features Command Menu Press Ctrl/⌘ + K to open the command menu – the fastest way to: Open any workflow by name Switch views and panels Access settings Search for anything Multi-Tab Workflow Multiple workflows: Open in tabs, switch with Ctrl/⌘ + 1-9 Reference between: Copy nodes from one workflow to another Reorder: Drag tabs to change their order Keyboard Navigation Shortcut Action 1 / 2 Toggle Workflows / Assets panel i Toggle Inspector (right panel) w Toggle Workflow Settings panel o Toggle Operator panel Ctrl/⌘ + F Search nodes on canvas by label Arrow keys Nudge selected nodes Ctrl/⌘ + Alt + N / P Move focus to next / previous node AI Model Tips Choosing Models Local for privacy: Use local models for sensitive data Cloud for speed: API models are faster Mix both: Local preprocessing → cloud generation → local post-processing Performance Smaller models first: Test with fast models, upgrade for quality Quantized models: Smaller files, similar quality (Q4, Q8) Streaming nodes: See progress during execution Asset Management Working with Files Drag and drop: Drop files directly onto the canvas Asset panel: Press 2 to open (or click the Assets icon) Preview: Click any asset to preview it Organizing Create folders: Keep projects organized Name clearly: Descriptive names save time later Clean up: Delete unused assets to save space Collaboration Tips Sharing Workflows Export workflow: Use the workflow export feature Mini-Apps: Share as simplified interfaces Screenshots: Document your workflows visually Working with Teams Consistent naming: Agree on conventions Document: Add comment nodes to explain complex sections Template library: Build shared templates Troubleshooting Quick Fixes Issue Try This Node menu won’t open Refresh page, check for modals Connection won’t attach Check type compatibility Workflow stuck Press Esc to stop, check error messages Getting Help Node info: Select a node and press Ctrl/⌘ + I Discord: Ask the community GitHub Issues: Report bugs Cheat Sheet Most Used Shortcuts Windows/Linux Mac Action Space Space Node menu Ctrl + Enter ⌘ + Enter Run workflow Ctrl + S ⌘ + S Save Ctrl + Z ⌘ + Z Undo Ctrl + D ⌘ + D Duplicate F F Fit to screen A A Align nodes Ctrl + K ⌘ + K Command menu i i Toggle Inspector Next Steps Workflow Editor – Full editor documentation Sketch Editor – Image editing guide Game Editor – Build, play, and export 2D and 3D games Cookbook – Workflow patterns Keyboard Shortcuts – Complete list--- # Retiring the Tool class: every capability is a pack module behind one action tool Source: https://docs.nodetool.ai/tool-class-retirement-design.html Retiring the Tool class: every capability is a pack module behind one action tool Two decisions are fixed. First, every agent capability becomes a nodetool API module exposed as an ordinary QuickJS pack, and the Tool class goes away. Second, the MCP server exposes exactly two tools, execute_code and view_image, and everything else registered in mcp-server.ts — the native tools, the frontend renderer bridge, and all MCP App code — is deleted. This document designs how, and works out what breaks. The problem Every agent step already acts by writing JavaScript in the QuickJS sandbox. PR #4826 (commit ccb1643) finished that consolidation: run_code, js, and script mode are gone, execute_code is the only code path, and the MCP server assembles its surface from the same createChatCodeActSession the chat runner uses (packages/websocket/src/mcp-agent-tools.ts:464). Two things predate CodeAct and were never reconciled with it. Capabilities are still Tool classes. packages/agents/src/tools/base-tool.ts:31 defines an abstract class whose reason to exist — toProviderTool() at base-tool.ts:120, rendering one JSON-schema tool per capability for a provider loop — no longer matches how capabilities are reached. A model reaches list_workflows as nodetool.workflows.list() inside a sandbox action; the class’s schema is rendered into a prompt catalog at most, and for the ~91 wrapped tools not even that (nodetoolApiCoveredToolNames, packages/agents/src/codeact/nodetool-api.ts:141, filters them out of the catalog). Measured on this tree: extends Tool appears 180 times across 77 files, 68 of them in packages/agents (tests included); outside agents the subclasses are RunNodeTool and two UI bridge tools in websocket, the Code-node tool factory in code-nodes, SandboxTool in sandbox-tools, the CLI’s ExecuteCodeTool shim (packages/cli/src/chat-codeact.ts:35), and two browser-agent factories in automation-nodes. 104 files import the Tool symbol; 27 of those import it purely as a type or for Tool[] plumbing without subclassing — executors, the permission gate, the eval runners, node hydration in llm-nodes, the MCP bridge. The MCP server still hand-builds a product surface. createMcpServer (packages/websocket/src/mcp-server.ts:364) natively registers: the fifteen uiToolSchemas entries routed to a connected editor over renderer transports (mcp-server.ts:384-447), list_renderers (:449), run_workflow (:468), get_asset (:543), get_node_info (:593), get_job (:637), get_collection (:705), query_collection (:745), and seven registerAppTool registrations (list_assets, list_workflows, get_workflow, list_jobs, list_nodes, search_nodes, list_collections — call sites at mcp-server.ts:983, :1054, :1146, :1188, :1239, :1276, :1338) that attach interactive HTML views from packages/websocket/src/mcp-apps/ — 7 files, 1,487 lines. Nothing outside mcp-apps/ and mcp-server.ts imports shell.ts or calls registerAppTool (verified by grep), so Decision 2’s deletion is clean at the import graph level. What it deletes in capability is worked out below. The blocker: host modules have no per-run context The pack mechanism that must absorb the capabilities was built for libraries, and libraries are pure. loadSandboxHostModule caches one implementation per process (packages/agents/src/host-modules/registry.ts:60-75), and the dispatcher’s entry point is call(moduleKey, exportName, args) (packages/agents/src/host-modules/dispatcher.ts:50, validation ladder at :101-133) — the only inputs an implementation ever sees are the guest’s arguments. That contract is exactly right for papaparse and fflate. It cannot serve a single NodeTool capability, because every one of the 180 needs state that exists only per run: Needs Where it comes from today ProcessingContext (user, secrets, workspace, storage, asset interfaces) passed into Tool.process(context, params) (base-tool.ts:58) by the executor or router The permission gate and its approval round trip GatedTool wraps every belt tool per turn (packages/agents/src/tools/tool-permissions.ts:274, applied at packages/websocket/src/unified-websocket-runner.ts:5368) The chat tool router (browser round trip for ui_* tools) createChatCodeActSession’s caller-supplied executeTool (packages/agents/src/codeact/chat-codeact.ts:82) provider + model + parent belt + event forwarding RunSubtaskTool’s constructor options (packages/agents/src/tools/run-subtask-tool.ts:52, constructed per turn at unified-websocket-runner.ts:5403) Today the per-run state arrives by closure at construction time: the runner builds a fresh belt every turn (unified-websocket-runner.ts:5296-5310), handing constructors whatever they need — a registry, a providers map, a runSingleNode closure, loaders for tRPC-only rows (mcp-agent-tools.ts:340-343). A process-level pack registry cannot close over any of that. The design’s center is therefore a second dispatcher that can. What is verifiably there (and two corrections) The nodetool object model is already most of the migration. Evaluating NODETOOL_API_PRELUDE_FULL (nodetool-api.ts:803) yields 21 top-level members — 18 namespaces plus the callables capabilities(), batch(), graph() — carrying 97 methods over 91 distinct belt tools (measured by executing the prelude against a recording tools proxy; the namespace map at nodetool-api.ts:20 lists the same 91). Each method is a thin wrapper: __need(name) resolves the belt tool or throws naming it (nodetool-api.ts:161-169), and buildNodetoolApiPromptSection (nodetool-api.ts:1086) documents only namespaces the belt can serve. So the migration is not “write 180 modules”; it is “move 91 wrappers off tools.* name resolution onto real pack exports, and give their implementations a run context”. Tools constructed outside the zero-arg catalogs. Eleven wrapped tools are absent from getAgentToolbelt() + getAllMcpTools({}) and come from their owning subsystems: vector_index, vector_batch_index, vector_text_search, vector_hybrid_search — constructors take a VectorCollection (packages/agents/src/tools/vector-tools.ts:63, :117, :190, :645). list_provider_models — constructor takes the configured-providers map (packages/agents/src/tools/model-tools.ts:20). run_subtask — provider, model, parent belt, forward callback (run-subtask-tool.ts:52; builtin-tools.ts:15-19 documents the exclusion). run_node — a runSingleNode closure (packages/websocket/src/agent/run-node-tool.ts:18, constructed at unified-websocket-runner.ts:5306). validate_timeline, validate_sketch — row loaders, because those surfaces are tRPC-only (mcp-agent-tools.ts:143-171, :340-343). list_collections, query_collection — one correction to the brief: these are zero-arg (packages/agents/src/tools/collection-tools.ts:17, :49). They sit outside the catalogs because they import @nodetool-ai/vectorstore at call time (collection-tools.ts:31-33) and the runner adds them explicitly (unified-websocket-runner.ts:5304-5305). The conclusion stands — availability depends on what the host supplies — but for these two the dependency is a package, not a constructor argument. Belt tools with no nodetool.* method. Measured against the two catalogs, exactly ten: read_file, write_file, edit_file, list_directory, glob, grep (deliberate — the sandbox’s own workspace.* API is in-process and costs no tool call), todo_write, view_image (deliberate — pixels, see below), and the two stragglers asset_list (beside the wrapped list_assets) and export_workflow_digraph. The graph editor is a second prelude. The eight workflow-document tools (packages/node-sdk/src/workflow-document-tools.ts:8) reach the guest as openWorkflow() (packages/agents/src/codeact/graph-model.ts:49): synchronous mutators against a local mirror, replayed through the same ui_* contract on commit(). Correction on inline thumbnails. The brief said get_asset, get_job and run_workflow return image blocks. get_asset and get_job do (mcp-server.ts:566-574, :680-686), and so do the App-registered list_assets, list_workflows and get_workflow (:1029-1035, :1121-1127, :1166-1172). The native run_workflow does not — its handler returns JSON text only (mcp-server.ts:517-524). The execute_code description. Registered as providerTool.description + "\n\n" + systemPromptSection (mcp-agent-tools.ts:491-494). Measured by building the session over the two catalogs (86 tools after dedup, 11 direct): 18,048 characters. The real mount adds Google Workspace, the media tools, and the two validators, so ~19k is right. Design What replaces Tool Three pieces: a spec (the schema-shaped identity), an implementation that takes a run context as its first argument, and a per-run object that carries everything the table above lists. The class dissolves into data plus a function. // packages/agents/src/capabilities/types.ts (new) export interface CapabilitySpec { readonly name: string; // wire name, unchanged: "list_workflows" readonly description: string; readonly inputSchema: JsonSchema; // rendered for prompts, nodetool.searchTools(), MCP readonly category: PermissionCategory; // REQUIRED — no default-to-external fallback userMessage?(args: Record<string, unknown>): string; } export type CapabilityImpl = ( run: CapabilityRun, args: Record<string, unknown> ) => Promise<unknown>; export interface CapabilityRun { readonly context: ProcessingContext; /** The one gate. decide → (ask ↔ UI) → monitor → run; owns the session allow-set. */ readonly gate: CapabilityGate; /** Browser round trip for ui_* capabilities; absent on headless runs. */ readonly client?: ClientToolRouter; /** provider, model, parentBelt(), forwardMessage — what run_subtask/run_search need. */ readonly subAgent?: SubAgentRuntime; // The injected singletons getAllMcpTools takes today (mcp-tools.ts:2168-2210): readonly nodeRegistry?: NodeRegistry; readonly providers?: Record<string, BaseProvider>; readonly examples?: ExampleWorkflowCatalog; readonly exportDsl?: WorkflowDslExporter; readonly loaders?: { timeline?: TimelineLoader; sketch?: SketchLoader }; /** The single choke point: lookup → gate → impl. Every surface calls this. */ invoke(name: string, args: Record<string, unknown>): Promise<unknown>; } The registry entry is (run) => impl in effect, but flattened: implementations are registered once per process as (run, args) => result, and the run threads through every call. This moves the per-run closure from construction time (today’s fresh belt per turn) to call time, which is the property a process-level pack registry needs. The registry itself mirrors the host-module loader table — one lazy loader per module, nothing in an entry graph: // packages/agents/src/capabilities/registry.ts (new) — same shape as // host-modules/registry.ts:24-49, one dynamic import per module. const MODULES: Record<string, () => Promise<CapabilityModule>> = { workflows: () => import("./workflows.js").then((m) => m.module), models: () => import("./models.js").then((m) => m.module), // ... }; The dispatcher change: a second dispatcher, not a widened one createSandboxHostDispatcher stays exactly as it is. Its contract — pure functions over plain data, one implementation per process — is a security property, not a limitation: a library pack that could see a run context would hold the action’s capabilities, which is the scenario the consent design exists to prevent (packages/agents/src/codeact/sandbox-packages.ts:1-13). Widening it to optionally carry a CapabilityRun would make every third-party pack one registry bug away from the platform surface. Instead, a createCapabilityDispatcher(run, modules) is built per invocation, in the same place the tool bridge is built today (buildToolBridge, packages/agents/src/codeact/tool-api.ts:99, and the chat session’s callTool, chat-codeact.ts:219). It runs the same validation ladder as the host-module dispatcher (declared module, listed export, array args — dispatcher.ts:101-133) and then, where the library dispatcher loads an implementation, it calls run.invoke. Both dispatchers share generateSandboxHostFacade’s facade shape (packages/protocol/src/sandbox-host.ts:175) and the private-bridge mechanics (sandbox-host.ts:140-165); only the far side differs. Where the permission gate lives Today the gate is GatedTool, a wrapper class whose process() runs decide → session-allow → approval → monitor before the inner tool (tool-permissions.ts:274-409), applied by gateTools at tool-permissions.ts:412. With no Tool.process left to wrap, the gate moves into CapabilityRun.invoke — the one place lookup already happens. The dispatcher does not gate on its own; it delegates to invoke, and so does every other entry path (a direct MCP registration, a core provider tool, run_subtask’s child loop). That keeps the current invariant — a direct call and an in-sandbox call run the same code (runBridgedTool, mcp-agent-tools.ts:412) — and strengthens it: today the invariant holds because both paths call the same Tool instance; after, it holds because there is only one call site. What this costs: The decide/ask/block matrix (decidePermission, tool-permissions.ts:220) and the monitor consult move verbatim; the read-class fast path (tool-permissions.ts:319) must be preserved in invoke, with a test. Classification changes ownership. TOOL_PERMISSION_CATEGORIES (tool-permissions.ts:41-209) dissolves into each CapabilitySpec.category, which is required. The current default — anything unlisted is external (tool-permissions.ts:212-214) — is a good fail-closed rule for a string-keyed map and a bad one for a typed registry: a required field plus a registry drift test (the analog of sandboxHostModuleDrift, host-modules/registry.ts:86) is stricter than a silent conservative default, because a misclassification becomes a reviewable diff instead of a runtime surprise. Approval prompts suspend the sandbox clock. The runner already owns this (createSandboxClock wired around requestApproval, unified-websocket-runner.ts:5356-5367); the suspension moves inside the gate so every host gets it, not just the chat runner. What shipped since. The gate is still a wrapper class, but it decides nothing itself: GatedCapabilityTool (capabilities/gate-tools.ts) builds a one-call CapabilityRun over a capability view of the inner tool and lets invoke run the ladder, so the single call site described above exists and a Tool is one door into it. Classification has not moved yet: a tool is classed by name through permissionCategoryFor, a capability by its spec’s category. What did move is where the gate comes from. A host publishes one PermissionGateOptions on the context under PERMISSION_GATE_CONTEXT_KEY, and a loop the host never constructed (an AgentNode reached through run_node, a JS script) reads it with gateFromContext instead of building an ungated run of its own; absence resolves to a headless gate that runs auto and denies what the ladder escalates, so a host that sets nothing fails closed. The per-host table is in packages/agents/AGENTS.md § Where the permission gate is set. ui_* capabilities: in the pack surface, outside its implementation ui_* tools are schemas, not host functions — the chat runner routes them over the ToolBridge to a browser (chat-codeact.ts:1-18), and the MCP mount routes the eight document tools to a live renderer when one is connected (mcp-server.ts:794-807). They stay that way. Their pack exports are one-line implementations: impl: (run, args) => { if (!run.client) throw new Error("ui_sketch_stroke needs a connected editor"); return run.client.execute("ui_sketch_stroke", args); } The guest-side graph object model (openWorkflow, graph-model.ts) is untouched by this: it is already plain guest JS over tools.ui_*, and becomes plain guest JS over the pack’s ui exports. The alternative — leaving ui_* outside the pack as a residual tools.* namespace — would keep two calling conventions alive forever and is rejected. Import or global? Import. nodetool.* is a global today, which contradicts the rule that killed data.*: libraries are imports, never globals (docs/sandbox-package-design.md §M6, packages/sandbox-packs/README.md:8). The counter-argument is real: nodetool is a capability surface, not a library, and the consent mechanism exists to keep a model from handing third-party pack code the action’s capabilities (sandbox-packages.ts:1-13) — a concern that does not apply to the platform’s own surface, whose availability the host already decides by composing the belt. Decide it anyway: the platform becomes import { workflows } from "@nodetool-ai/sandbox-nodetool", for three reasons. One module system. Two conventions (imports for libraries, a global for the platform) is a permanent tax on the prompt, the static analyzer (code-analysis.ts already validates import specifiers), and every future surface. Availability becomes visible where code is checked. Today a missing capability surfaces when __need throws mid-action (nodetool-api.ts:161-169). An undeclared import is refused before the action runs, with the module named (mountActionModules, sandbox-packages.ts:75). The session story unifies. What a session can do = the modules mounted for it. With one carve-out: the platform pack does not go through the model-facing consent allowlist. Consent gates third-party code; the platform modules are mounted by the host, per session, for exactly the namespaces its CapabilityRun can serve — the same decision the host makes today when it assembles the belt. A session whose host mounts nothing imports nothing, and the import fails naming the module. During migration the nodetool global stays as a generated shim over the imports for at least one release, then dies. Pack layout: one pack, one module per namespace @nodetool-ai/sandbox-nodetool, with submodule specifiers per namespace: @nodetool-ai/sandbox-nodetool/workflows, /models, /media, and so on, following the boundaries already drawn by NODETOOL_API_NAMESPACE_TOOLS (nodetool-api.ts:20). One pack, because availability is decided per session and documented per namespace — several packs would multiply manifests without adding a boundary anyone checks. Module-per-namespace, because: the prompt tier already documents per namespace and drops namespaces the run cannot serve (buildNodetoolApiPromptSection, nodetool-api.ts:1086-1118) — mounting per namespace makes that real instead of advisory; lazy loading holds per namespace: the registry’s loader table means a run that never touches storyboards never loads storyboard-render-tools’ dependency cone; the eleven subsystem-dependent capabilities map onto mount decisions: a host that has no vector store simply does not mount /collections, and the failure mode is an import error naming the module instead of a __need throw naming a tool. Like the host-module table pins each id to one pack (SANDBOX_HOST_MODULES, sandbox-host.ts:47; foreign claims refused at dispatcher.ts:87-91), the capability module table is first-party only: a third-party pack cannot declare a capability module, for the same reason it cannot declare a host module. batch() and the graph DSL core are guest-side helpers with no host call of their own (nodetool-api.ts:273-310, graph-dsl-core); they ship as guest code inside the pack, not as dispatched exports. Lazy loading and the two bundlers Nothing new enters an entry graph. The capability registry copies the host-module pattern exactly: lazy loader per module (host-modules/registry.ts:24-49), each implementation importing its heavy dependencies inside itself (collection-tools.ts:31-33 is already the model). The packaged Electron backend needs no PACKAGE_RUNTIME_ASSETS entry — capability modules are code, and esbuild inlines dynamic imports into server.mjs the way it already inlines the host-module implementations (host-modules/registry.ts:8-12). The browser runner is unaffected: capability dispatch exists only where a CapabilityRun exists (server, CLI, tests); the generated facades are text served like any sandbox module, and a browser session’s tool calls already execute server-side. What stays schema-shaped, and where it lives Three kinds survive, and none of them needs the class. The first two are the product surface’s own: execute_code is already constants, not a Tool: EXECUTE_CODE_TOOL_NAME and EXECUTE_CODE_INPUT_SCHEMA (packages/agents/src/codeact/codeact-executor.ts:110, :195). The CLI’s ExecuteCodeTool wrapper (cli/src/chat-codeact.ts:35) and the MCP registration consume the session’s providerTool record (chat-codeact.ts:447-457); both become plain records. view_image becomes a CapabilitySpec + impl with category: "read" (today packages/agents/src/tools/view-image-tool.ts:96). Hosts register it as a direct provider tool because pixels cannot ride the JSON observation envelope — the runner strips image payloads from sandbox results and hands pixels only through the direct path (chat-codeact.ts:277-281, unified-websocket-runner.ts:5738-5762, mcp-agent-tools.ts:456-459). That rule is unchanged. The third kind is loop-protocol tools, and they are exempt from this migration by kind rather than by schedule. A loop-protocol tool is the structured-output or feedback channel of a provider-driven loop. Three properties identify one, and all three must hold: Its state is loop-instance state read back by the code that constructed it. GraphPlanner builds a SubmitGraphTool per attempt (graph-planner.ts:420), aborts the provider loop from inside the tool’s own execute on acceptance (:486), then reads _graph / lastCode / lastErrors off the instance (submit-graph-tool.ts:57). PlanBuilder plus its three tools is the same shape (task-planner.ts:423). It is never gated. GraphPlanner passes {} as ProcessingContext (graph-planner.ts:463); there is no context, no gate, and invoke could not run it. It is never belt-assembled and never sandbox-reachable, so capabilities-coverage.test.ts does not and should not see it. Named instances: submit_graph, submit_code, add_task / remove_task / finish_plan, create_task, finish_step, the per-edge control_* tools, get_run_state / read_node_output, and the eval fakes. The supervisor pair reads least like the others, so state why it belongs. createSupervisorTools builds a GetRunStateTool and a ReadNodeOutputTool per decision, bound to that decision’s Escalation and the run’s RunStateReader (supervisor/supervisor-agent.ts:150-158), and hands them to a StepExecutor that never calls gateTools. All three properties hold: the state is the loop instance’s, read back by the code that constructed it; nothing gates them, because the scoping is the control — read_node_output refuses a read outside the failing invocation’s causal lineage, which no permission category expresses; and no belt assembles them, so capabilities-coverage.test.ts never sees them. Registering them would put one escalation on the run, which is the shape this design rejects. Schema-per-tool is the mechanism here, not an artifact of the old design, because the provider owns the loop — the comment at graph-planner.ts:441 records that this is what makes the flow work on backends running their own agent loop (the Claude Agent SDK). Nor is “let the planner act via sandbox code” a unification: SubmitGraphTool already evaluates the submitted DSL program in QuickJS itself (submit-graph-tool.ts:82). Routing it through a CodeAct session would add a sandbox hop and forfeit SDK-loop compatibility. Their end form is the same as execute_code’s — plain {name, description, inputSchema, execute} records, which is already the shape GraphPlanner flattens them into at :503. That conversion is cosmetic and buys nothing on its own. CapabilitySpec also feeds everything else that consumes schemas today: prompt signature rendering (toolSignature, tool-api.ts:351), in-sandbox nodetool.searchTools() (chat-codeact.ts:299-328), and the chat direct-tool set (CORE_TOOL_NAMES members offered as plain provider tools, tool-api.ts:204, :223). The core set is a chat concern and is not touched by Decision 2 — see the MCP section. What Decision 2 deletes, item by item Decision 2 leaves the /mcp mount with execute_code and view_image. Note this is narrower than today’s post-#4826 surface, which also registers the ~11 core tools as direct MCP tools (mcp-agent-tools.ts:516-525): those registrations go too. An MCP caller that wants to read a file in the session workspace writes a one-line action. That is an acceptable cost on MCP — callers are agents with their own file tools — and it is the decision’s stated shape. Inline thumbnails: partially regress, say so. The native get_asset and get_job put real image blocks in the tool response (mcp-server.ts:566-574, :680-686), and the App-registered list tools attach thumbnails (:1021-1035, :1113-1127). Those channels die with the native registrations. view_image covers the load-bearing case — inspect one image by asset id, region and detail included — because it stays a direct tool whose result carries image content. What it does not cover is browsing: a gallery of twenty thumbnails in one response has no equivalent, since a sandbox action can only return handles. This is a real regression for Claude Desktop-style hosts and should be listed in the release notes, not smoothed over. If it turns out to matter, the recovery path is an MCP resource serving thumbnails by asset id, not a third tool. The frontend renderer bridge: route the survivors through the belt. Removing the native uiToolSchemas loop deletes two different things. The eight workflow-document tools survive unchanged for scoped sessions — they are on the bridged belt and already route to a live renderer through executeFrontendDocumentTool (mcp-server.ts:794-807, mcp-agent-tools.ts:421-424). The seven editor-steering tools (ui_open_workflow, ui_run_workflow, ui_switch_tab, ui_copy, ui_paste, ui_search_nodes, ui_search_models — packages/protocol/src/toolSchemas.ts:185) exist only as native registrations and would vanish. Keep the capability: they become exports of the pack’s ui module, implemented as run.client.execute(...), so a scoped MCP caller steers a connected editor from inside an action. What is dropped outright is list_renderers as a tool; it becomes a ui.renderers() export backed by the same transport map (mcp-server.ts:101-127). The multi-renderer renderer_id parameter survives as an argument on those exports. MCP Apps: dropped outright. The seven modules render interactive HTML galleries in App-aware hosts; shell.ts is their shared chrome and nothing else imports it. A user of an App-aware host loses the asset gallery, workflow gallery, graph viewer, jobs dashboard, node catalog, and collections browser as inline surfaces. The data all remains reachable through actions; the views are gone, with no planned replacement. If NodeTool wants inline views again, the right vehicle is whatever succeeds MCP Apps upstream, not a parallel maintenance of 1,487 lines behind two tools. Unscoped sessions: the concept goes away. Today a session without agentToolsScope gets no bridged tools (mcp-agent-tools.ts:385-391) and lives on the native registrations plus the renderer bridge kept for backwards compatibility (mcp-server.ts:385-390). Under Decision 2 that session would hold zero tools, which is a server that answers and cannot act — worse than an error. Make the scope required: stdio-local and local-dev-http already bind one (mcp-server.ts:63-75), and an HTTP mount that cannot bind an authenticated user refuses the session at initialize with a message naming the fix. The backwards-compatibility renderer path dies with the bridge. Discovery: the description is the contract, plus one cheap channel. A client that lists two tools learns what NodeTool does from the execute_code description — measured at 18,048 characters over the two-catalog belt (~19k on the full mount), carrying the action contract, the direct-tool list, the namespace docs, and the sandbox summary (mcp-agent-tools.ts:486-494). That is sufficient for a model and poor for everything else (token cost on every list, no structure for tooling). Two additions, both cheap: keep capabilities() as a guest call reporting the mounted namespaces (nodetool-api.ts:256-265 already does this), and publish the same catalog as an MCP resource (nodetool://capabilities, JSON: modules, exports, one-line descriptions, categories). Resources cost nothing at list time and give non-model clients something to render. Do not shorten the description below what the action contract needs; it is the only channel every client is guaranteed to show the model. Worked example: nodetool.workflows Today. list_workflows is a class (packages/agents/src/tools/mcp-tools.ts:396), constructed per belt with an injected example catalog (mcp-tools.ts:2214), wrapped in GatedTool per turn (unified-websocket-runner.ts:5368), name-resolved by the guest prelude, and documented by a hand-maintained namespace doc (nodetool-api.ts:823): // packages/agents/src/tools/mcp-tools.ts:396 (abridged) export class ListWorkflowsTool extends Tool { readonly name = "list_workflows"; readonly description = "List workflows (id, name, description, tags only — ...)"; readonly jsonSchema = { type: "object", properties: { workflow_type: {...}, ... } }; constructor(private readonly examples?: ExampleWorkflowCatalog) { super(); } async process(context: ProcessingContext, params: Record<string, unknown>) { // ... Workflow.paginate(userIdOf(context), ...) / this.examples.list(...) } userMessage(params) { return `Listing ${params["workflow_type"] ?? "user"} workflows`; } } // guest side, nodetool-api.ts:357 — name resolution against the belt workflows: { list: (opts) => __need("list_workflows")(__merge(opts)), ... } After. One module in the capability registry; the guest imports a generated facade whose dispatcher is the per-run invoke: // packages/agents/src/capabilities/workflows.ts (new) export const module: CapabilityModule = { module: "workflows", exports: [ { spec: { name: "list_workflows", description: "List workflows (id, name, description, tags only — ...)", inputSchema: { type: "object", properties: { workflow_type: {...}, ... } }, category: "read", userMessage: (a) => `Listing ${a["workflow_type"] ?? "user"} workflows` }, impl: async (run, args) => { const { Workflow } = await import("@nodetool-ai/models"); if (args["workflow_type"] === "example") { return run.examples ? lightList(await run.examples.list(...)) : NO_EXAMPLES; } const [workflows, next] = await Workflow.paginate(run.context.userId, {...}); return lightList({ workflows: workflows.map(workflowRecord), next }); } }, // get_workflow, create_workflow (category: "write"), // run_workflow (category: "execute", impl reads run.nodeRegistry), ... ] }; // guest side — the model writes this in an action import { workflows } from "@nodetool-ai/sandbox-nodetool/workflows"; const { workflows: mine } = await workflows.list({ limit: 20 }); const report = await workflows.run(mine[0].id, { prompt: "hi" }); The facade behind that import is generated the way host-module facades are (sandbox-host.ts:175-190): one async export per spec, each forwarding (moduleKey, exportName, args) — but to the capability dispatcher, which calls run.invoke("list_workflows", args): gate first (read-class fast path), then the registered impl with the run threaded in. The example catalog that was a constructor argument is now run.examples, built by the same host code that builds it today (packages/websocket/src/mcp-tool-deps.ts). Migration Mapping Pack module Wraps today (source) Run needs beyond context+gate /workflows 9 tools in mcp-tools.ts (list/get/create/run/debug/validate/escalation/example/start) nodeRegistry, examples, workflowEnvironment /nodes local-{list,search,get-node-info} + run_node nodeRegistry, single-node runner /models find-model-tool.ts, list-models-tool.ts, model-tools.ts providers /media media-tools.ts, creative-critique-tools.ts providers /assets, /jobs mcp-tools.ts asset/job tools, asset-library-tools.ts, view-image-tool.ts storage interfaces on context /web search-tools.ts, browser-tools.ts, http-tools.ts — /documents pdf-tools.ts — /memory, /email, /style memory-tools.ts, email-tools.ts, critique-style pair — /collections collection-tools.ts, vector-tools.ts vector provider / collection /apps build_app, debug_app (mcp-tools.ts) nodeRegistry /timelines, /sketches, /scripts, /storyboards version/edit/voice/render tool files loaders /agents run-subtask-tool.ts, run-search-tool.ts subAgent /ui ui_* schemas (toolSchemas.ts:185) + graph model prelude client /files (or none) the ten unwrapped: file set stays workspace.*; todo_write stays a direct tool; fold asset_list into /assets, export_workflow_digraph into /workflows — The table maps one inventory: what getBuiltinTools() and getAllMcpTools({}) assemble. The planner- and executor-constructed tools are not in it and are not migration debt — they are exempt by kind, for the reasons in What stays schema-shaped. PR order Each step merges and reverts independently. PR 1 — the MCP reduction. Ships first; needs none of the pack work. The reduced surface is assembled from machinery that already exists: registerAgentMcpTools already builds the session and registers execute_code + view_image (mcp-agent-tools.ts:375-534). The PR deletes: packages/websocket/src/mcp-apps/ (7 files), every native registration and register*App helper in mcp-server.ts (everything from the uiToolSchemas loop at :384 through registerCollectionsApp at :1337, plus the App imports), the renderer-bridge fallback for unscoped sessions, and the direct core-set registration loop in mcp-agent-tools.ts:516-525 (keep view_image). It adds: belt entries for the seven editor-steering ui_* schemas routed through the frontend executor, the nodetool://capabilities resource, and the scope-required initialize error. Tests: rewrite packages/websocket/tests/mcp-server.test.ts and mcp-server-coverage.test.ts to pin the two-tool surface; delete mcp-apps-coverage.test.ts. Someone can start this Monday. PR 2 — capability types, registry, adapters. capabilities/types.ts, registry.ts, toolFromCapability (a Tool subclass wrapping a spec+impl so existing belts consume capabilities unchanged) and capabilityFromTool (the reverse, for the long tail during migration). The gate-parity test lands here (see Verification). No caller changes. PR 3 — pilot namespace: workflows. Port the nine workflow tools from mcp-tools.ts to capabilities/workflows.ts; getAllMcpTools returns them via toolFromCapability, so every consumer — runner, MCP, CLI, evals — is untouched. Tests that must stay green: packages/agents/tests/nodetool-api.test.ts, tool-permissions*.test.ts, chat-codeact.test.ts, and the websocket MCP suites. PRs 4–9 — remaining namespaces, grouped as in the mapping table, including the subsystem-constructed eleven (their constructor arguments become CapabilityRun fields, built where the constructors are called today: the runner, mcp-agent-tools, mcp-tool-deps). PR 10 — hosts build CapabilityRun; the gate moves into invoke. The runner, MCP mount, CLI, and executors construct runs instead of belts; gateTools becomes a shim over invoke and GatedTool is deleted. This is the riskiest PR; it is also the first one where a capability cannot be reached ungated, because there is no other call path left. PR 11 — the guest pack. Generated facades per module, the capability dispatcher, prompt sections switch to import form, the nodetool global becomes a shim over the imports. codeact-prompt-drift.test.ts and the codeact eval cases are the regression net for the prompt change. PR 12 — deletion. The nodetool global shim, base-tool.ts, the tool registry and llm-nodes hydration shims, capabilityFromTool. The capability-shaped extends Tool count reaches zero; the loop-protocol tools stay, and base-tool.ts stays with them. What breaks, and what each costs llm-nodes node hydration. agent-tool-hydration.ts holds a registry of zero-arg Tool constructors (STATIC_TOOL_CLASSES, packages/llm-nodes/src/nodes/agent-tool-hydration.ts:45-61) plus load-time class registration from sandbox.ts, and hydrates name-stubs into instances for runAgentLoop and the AgentNode. The concept dissolves: a name-stub becomes a capability name resolved against the registry, and “hydration” becomes binding a name to run.invoke. Cost: mechanical rewrites of agent-tool-hydration.ts, agent-loop.ts, agents.ts (normalizeTools), plus a capability-module equivalent for the load-time browser_* registration pattern. The stub contract (“not executable until hydrated”) goes away entirely, which deletes a documented footgun. Eval surfaces. The offline suites instrument fake Tool subclasses — evals/planner-tools.ts, evals/subtask-cases.ts (shared instances that record SUBTASK_DEPTH_KEY), evals/codeact-api-core.ts / codeact-api-surfaces.ts (fakes named like real belt tools so the object-model prelude lights up). They keep working through PR 11 via capabilityFromTool, then convert to fake capability modules in PR 12. The tool-loop suites are unaffected — they already drive schemas against headless bridges, not Tool instances. Cost: one mechanical pass over packages/agents/src/evals/, with codeact-api-coverage.test.ts guarding that no namespace loses its last case in the shuffle. getAllMcpTools dependency injection. The registry / examples / exportDsl options (mcp-tools.ts:2168-2210) become CapabilityRun fields. The three builders in packages/websocket/src/mcp-tool-deps.ts survive unchanged; what changes is the consumer: instead of spreading them into a tool-constructor call, the host puts them on the run. getAllMcpTools itself shrinks to a compatibility shim in PR 3 and is deleted in PR 12. Cost: every getAllMcpTools call site (runner, MCP bridge, CLI) touches once. Verification Existing suites that cover this seam, per step: packages/agents/tests/tool-permissions.test.ts (+ -hardening, -monitor) pin the gate matrix and monitor behavior; chat-codeact.test.ts and codeact-executor.test.ts pin the bridge, the observation envelope, and the image strip; nodetool-api.test.ts and nodetool-api-*.test.ts drive every namespace against a real sandbox with scripted providers; codeact-prompt-drift.test.ts pins the prompt surface; packages/websocket/tests/mcp-server*.test.ts pin the MCP surface; the offline codeact eval cases exercise the full object model end to end. Each migration PR’s definition of done is: these pass unmodified, except where the PR explicitly rewrites a pinned surface (PR 1 rewrites the MCP suites; PR 11 rewrites the prompt-drift snapshots). The new test that fails if a capability silently loses its gate. Two parts, landing in PR 2: Classification drift. A registry walk asserting every registered export carries a category, plus a checked-in snapshot of name → category. Moving create_workflow from write to read becomes a one-line diff a reviewer sees, instead of a behavior change nobody does. This replaces the protection the external default gives today, and unlike the default it also catches the wrong-direction change. Gate parity through the real path. For one canary capability per category, drive an execute_code action through the real prelude and dispatcher with a scripted approver in default mode, and assert: read ran without a prompt, write and execute round-tripped an approval, a denied call returned the permission_denied payload without running the impl, and plan mode blocked. Run the same four assertions through a direct invoke call — if the two transcripts ever differ, the one-implementation invariant broke. Today’s equivalent coverage lives in tool-permissions.test.ts but only at the GatedTool unit level; the parity test covers the seam this migration moves. For the MCP reduction, one integration test pins the surface: initialize a scoped session, list tools, assert exactly execute_code and view_image, and assert an unscoped initialize is refused. What this buys, and what it costs Bought: One shape for every capability. The 180-subclass inventory, the wrapper prelude, the hand-maintained namespace map, and the classification map collapse into one registry where spec, implementation, category, and documentation cannot drift apart — the drift tests this repo keeps writing (sandboxHostModuleDrift, codeact-api-coverage, prompt-drift) become structural. The gate becomes unbypassable by construction. Today gating is a wrapper applied where someone remembered gateTools; Agent applies it (agent.ts:427), the runner applies it, but any new host that assembles a raw belt gets ungated tools. After PR 10 there is no ungated call path to forget. The MCP surface stops being a second product. 120 registrations became 34 in #4826; this takes it to 2, deletes 1,487 lines of App code, and ends the drift class where a native handler reimplements a run path and rots (the run_workflow handler’s own comment records exactly that happening, mcp-server.ts:482-485). Session capability becomes declarative and inspectable: mounted modules, not belt composition spread across a 5,000-line runner method. Costs, stated plainly: MCP hosts lose thumbnails-in-lists and every MCP App view, and light MCP clients lose one-call reads (get_asset as a plain tool call becomes a two-line action). Nothing replaces the gallery experience. The migration crosses every executor, both bridges, the MCP mount, the CLI, node hydration, and the eval fakes. The adapter strategy keeps each PR small, but PR 10 (the gate move) is a genuine risk concentration and needs the parity test in place two PRs before it lands. Two dispatchers and two module kinds (library, capability) is more machinery than one. The alternative — one dispatcher with an optional run — was rejected for a security reason, not a taste one, but the cost is real: the facade generator, the loader table, and the drift tests exist twice. One part of the current design should survive on the merits: the chat runner’s direct-tool split (splitCoreTools, core tools as plain provider calls) and the view_image pixel path are not Tool-class artifacts, they are provider-interface facts, and nothing here changes them. And one honest reservation: if MCP App-style inline views turn out to matter to desktop users, Decision 2 removed them with no successor, and the right response will be a new surface — not quietly re-growing native tool registrations behind the two-tool contract. Status (2026-08-10) PRs 1–11 landed on this branch, in this order: agents: add capability types, registry, adapters, and gated invoke (PR 2 of tool-class retirement) websocket: reduce the MCP surface toward the two-tool contract (PR 1 of tool-class retirement, in progress) agents: begin porting the workflows namespace onto the capability registry (PR 3 of tool-class retirement, in progress) agents: workflows namespace runs on the capability registry (PR 3 of tool-class retirement) agents: models, media and style namespaces run on the capability registry (PR 4 of tool-class retirement) agents: collections and nodes namespaces run on the capability registry (PR 7 of tool-class retirement) agents: jobs, assets and apps namespaces run on the capability registry (PR 5 of tool-class retirement) agents: web, documents, memory and email namespaces run on the capability registry (PR 6 of tool-class retirement) agents: files, agents and google namespaces run on the capability registry (PR 9 of tool-class retirement) agents: timelines, sketches, scripts and storyboards namespaces run on the capability registry (PR 8 of tool-class retirement) agents+websocket: the permission gate moves into CapabilityRun.invoke (PR 10 of tool-class retirement) agents+protocol+websocket: the platform is importable in the sandbox (PR 11 of tool-class retirement) agents: break the tool-permissions/capabilities import cycle that deadlocked the bundled backend agents+websocket: the ui and graph-planner capabilities leave mcp-tools (PR 13 of tool-class retirement) PR 13 ported the last two capability-shaped Tool classes in mcp-tools.ts. WorkflowDocumentTool became the ui module — one capability per document-tool name, the registry on the run instead of in a constructor — and survives as a thin CapabilityTool subclass that keeps the Zod schema, so the class path still validates exactly once. PlanWorkflowGraphTool was deleted outright: its spec and implementation are plan_workflow_graph in the workflows module, and its constructor options became CapabilityRun.graphPlanner (provider, model, forwardMessage, signal), built where the constructor stood, in unified-websocket-runner.ts. The wrapper reads a new CapabilitySpec.needsToolCallId, which is how the planner’s events keep nesting under the caller’s card. The capability is registry-visible but not on the belt: only a host that can build a graphPlanner run can serve it. The esbuild async-cycle lesson gateTools was written beside the classification map it reads, in tools/tool-permissions.ts, and importing capabilities/adapters.ts from there closed a cycle: tool-permissions → adapters → tool-permissions. Node’s ESM breaks a synchronous cycle by handing back a partly-initialized namespace, so vitest, tsc and the dev server all stayed green. The packaged backend does not run ESM — scripts/bundle-backend.mjs gives esbuild one server.mjs whose modules are __esm wrappers, and a wrapper that awaits its own cycle never settles. init_tool_permissions awaited init_adapters, which awaited init_tool_permissions, and the process hung on a top-level await before serving /health. The fix was direction, not ordering: gateTools moved to capabilities/gate-tools.ts, whose header states the rule it now enforces — capabilities import tool-permissions, never the reverse. Two lessons for the rest of this migration: an import cycle across the tools/↔capabilities/ seam is a bundle-only failure, and npm run backend:smoke is the only check in the repo that sees it. What PR 12 did not delete, and why The PR-12 sketch above describes an end state this tree cannot reach yet. PR 12 as executed deletes what is provably dead, pins the coverage the migration won, and records the rest as named debt: The deprecated thin CapabilityTool subclasses stay. getBuiltinTools and getAgentToolbelt are synchronous and every caller assembles a belt synchronously, while capability modules load through import(). A belt cannot be built from the registry alone until either the belt assembly turns async or the registry gains eager specs. That is its own PR, with its own callers to touch. (Resolved — see Eager specs below: the registry gained eager specs and all ninety-two are gone.) The nodetool global stays. The design gives the generated shim over the imports at least one release before it dies; PR 11 shipped the import form, and this is that release. capabilityFromTool stays. It is load-bearing in gateTools: the shim turns each Tool into a capability view so invoke can run the one ladder over it. It goes when belts stop being Tool[], not before. base-tool.ts stays. SubAgentTool, SandboxTool (packages/sandbox-tools), the Code-node tool factory (packages/code-nodes), and the two browser-agent factories (packages/automation-nodes) still subclass Tool, as do the websocket UI bridges and the CLI’s execute_code shim. Measured extends Tool Before (recorded above): 180 occurrences across 77 files. After PR 13 landed on top of the fifteen deletions below: 32 across 24 files (source only — packages/**, no dist/, no test directories), plus 92 extends CapabilityTool across 31 files — the deprecated subclasses that carry no implementation of their own. Outside packages/agents the remaining Tool subclasses are the ones listed above, and the fake tools in the chat and code-nodes test suites. packages/agents/tests/capabilities-coverage.test.ts is what keeps the count from growing back: everything getBuiltinTools() and getAllMcpTools({}) assemble must resolve through findCapability. Its exception list is empty — PR 13 unpinned the eight workflow-document ui_* schemas and the deletions below retired the nine AGENT_TOOLBELT_EXCLUDED names, so every belt name is a capability. Two counters, not one (2026-08-11) The single number above understated the work by counting one inventory and implying it was the whole set. Split it: Capability-shaped subclasses — anything getBuiltinTools() or getAllMcpTools({}) assembles. Target: zero. Defended by capabilities-coverage.test.ts plus its pinned exceptions (seventeen when this was written, eight after the deletions below). Loop-protocol subclasses — planner- and executor-constructed, enumerated by name in What stays schema-shaped. Allowed. No target. Neither the mapping table nor the coverage test ever saw the second group, because a planner constructs its tools inside its own loop and never assembles a belt. That is correct behavior, but it meant roughly twenty subclasses in packages/agents/src/tools/ were tracked nowhere. Two decisions taken while writing this down: The pre-submit_graph incremental family is deleted. add_node, add_edge, remove_node, remove_edge, finish_graph and create_plan were the tool-call-per- node flow the one-shot DSL replaced. No planner constructed any of them; the only references left were the src/index.ts re-exports and their own tests. The two live helpers in finish-graph-tool.ts moved to tools/graph-validation-registry.ts, which is what submit_graph and the Code-node refiner actually import. The run-scoped memory tools are a real gap, and stay open. (Closed 2026-08-12 — see The shared module below.) list_shared / read_shared / share_result (renamed from memory_list / memory_read / memory_write) are not loop-protocol tools: they are prompt-documented, permission-classified, and the model calls them from inside the sandbox as tools.read_shared(...) — the CodeAct executor’s own prompt instructs exactly that (codeact-executor.ts:884). “Executor-internal” in capabilities/memory.ts:10 means mounted by the executor rather than the host, not not model-facing. They belong in a shared capability module — not folded into memory, which is thread memory and a different lifetime — with the executors still mounting them, since that is mount policy. The port waits on the sync-belt problem PR 12 named: executors attach these in constructors, synchronously, while capability modules load through import(). Doing it before the async-belt PR would fork a third pattern beside the sixteen ported namespaces and the deprecated CapabilityTool subclasses. Fifteen deleted (2026-08-11) Two commits took extends Tool from 71 across 35 files to 56 across 28 (source only — packages/**, no dist/, no tests). The dead cluster, six classes, deleted outright: workspace_read / workspace_write / workspace_list (the sandbox’s in-process workspace.* API is the live path), ltm_recall / ltm_remember (Agent calls LongTermMemory directly — the feature stays, the wrappers were mounted nowhere), and ToolSearchTool (constructed only by its own test; discovery runs through the __searchTools bridge). The per-user LTM registry moved into long-term-memory.ts; searchTools and the two formatters stayed where they were. The nine AGENT_TOOLBELT_EXCLUDED names, all nine gone from the belt — but not in one way, because the pinned reason was wrong for five of them. “A routed capability already covers each” held for the media four (image_generation, openai_image_generation, google_image_generation, openai_text_to_speech), which generate_image / generate_speech cover and which were deleted. It did not hold for openai_web_search, google_grounded_search and the three dataforseo_*: capabilities/web.ts constructed those very classes as the openai, gemini and dataforseo backends of web_search. They were not duplicates of the routed capability, they were the inside of it, and deleting them would have deleted three search backends. They became plain async functions the capability calls — one wire name fewer, one implementation unchanged. Read a pinned exception’s reason as a claim to check, not a finding. Both counters move: the capability-shaped counter loses nine and AGENT_TOOLBELT_EXCLUDED is gone with them, so getAgentToolbelt() now returns what getBuiltinTools() returns and capabilities-coverage.test.ts pins only the eight workflow-document ui_* schemas. The loop-protocol counter loses the six dead ones. Eager specs, and the ninety-two that went with them (2026-08-12) The sync-belt problem PR 12 named is solved, and the way out was the third option: not an async belt, and not a spec on every class, but a spec table the registry can read synchronously. Each capability module gained a data-only sibling — workflows.specs.ts, media.specs.ts, one per namespace — holding the wire name, description, JSON schema, category and message template, and nothing else. A spec file imports types.ts, zod, and the schema constants it needs; it imports no implementation, so importing all twenty-two costs one object graph and no import(). registry.ts imports them eagerly beside its lazy loader table and exposes capabilitySpec(name) / listCapabilitySpecs(). The module file imports its own specs back and attaches each to an implementation, so there is one spec object behind both halves — and eagerSpecDrift compares them by identity, not by field, because a module that copied its spec would pass a field check and still be two things to keep in step. toolFromLazyCapability(spec, run) and toolForCapabilityName(name, run) (capabilities/lazy-tool.ts) are the wrapper. Tool.process() is already async, so only the spec has to be there when a belt is assembled: the name, the description and the schema are what a provider list and a permission prompt read. The implementation arrives on the first invoke, from the one module that owns it — loadCapabilityImpl resolves the owning module from the eager table and loads only that one, where findCapability would have loaded all twenty-two. Gating stays single-pass: like CapabilityTool before it, the wrapper calls the implementation directly, because a belt is gated from the outside by gateTools, which runs the one ladder in invoke.ts. CapabilitySpec grew one optional field, zodSchema, for the handful of capabilities whose identity is a Zod schema — view_image, list_images and the eight ui_* document tools. The wrapper returns it from Tool.schema, so a malformed call still comes back as invalid_tool_arguments from Tool.execute instead of reaching the implementation, which is what the classes did. getBuiltinTools() is now a list of names, getAllMcpTools() a list of names plus the run each group needs, and getGoogleWorkspaceTools() a walk over googleSpecs. The belt is unchanged name for name. All ninety-two deprecated extends CapabilityTool subclasses are gone, and with them sixteen files that held nothing else. What each file still owned that was not a class stayed: htmlToText, requestSignal, serpApiConfigured, splitTextRecursive, getThreadTodos/clearThreadTodos, formatThreadMemoriesForPrompt, VecCollection, and the five *_TOOL_NAMES lists. Three constructor call sites outside packages/agents moved to the name form: mcp-agent-tools.ts (the two validators, the seven media capabilities, find_model/list_models), unified-websocket-runner.ts (the two collection capabilities), and graph-planner.ts (the three discovery capabilities plus find_model). runBridgedTool’s instanceof FindModelTool became a name check, which is what it meant. One capability could not be built from the spec table, and stayed as it was: createSearchTool (tools/serp-tool-factory.ts) binds a resolved SERP provider into the implementation, not into the run, so it builds its tool with toolFromCapability(webSearch.spec, webSearchImpl(provider), …). Measured after: 33 extends Tool across 25 files and zero extends CapabilityTool (source only — packages/**, no dist/, no test directories). The one new extends Tool is the lazy wrapper itself. The capability-shaped counter is now zero on both halves: nothing getBuiltinTools() or getAllMcpTools({}) assembles is a class any more. base-tool.ts stays for the loop-protocol tools and the five subclasses outside this package. npm run backend:smoke is green, which is the only check that would have seen a new cycle across the tools/ ↔ capabilities/ seam. The eager spec table does not pull an implementation into the entry graph, so laziness is stronger than it was: before this, mcp-tools.ts imported ten implementation modules statically to build its belt. One thing a deletion could have broken quietly: an AgentNode saves its tools as bare name stubs and resolveBuiltinAgentTool hydrates them by wire name, so a workflow naming a retired tool would have hydrated to nothing and been uncallable without an error. RETIRED_TOOL_NAMES (packages/llm-nodes/src/nodes/agent-tool-hydration.ts) maps all nine onto their replacements. The shared module (2026-08-12) The gap the previous section left open is closed. list_shared / read_shared / share_result are a capability module, shared, built on the eager-specs pattern: shared.specs.ts holds the three wire names, descriptions and schemas unchanged, shared.ts attaches each to an implementation reading run.context.memory, and both halves appear in MODULES, DECLARED_CAPABILITY_MODULES and the spec table, so eagerSpecDrift and the category snapshot cover them like every other namespace. Not folded into memory. That module is the durable store — a database row that outlives the run — and these are the run’s own AgentMemory, discarded with it. The naming now shows the lifetimes: nodetool.shared run-scoped beside nodetool.memory thread-scoped. Mount policy did not move. getMemoryTools() keeps its signature and its callers, and now assembles the belt from the specs with toolFromLazyCapability; every step executor still pushes it onto its own toolset, and the host still mounts nothing. What eager specs changed is that a synchronous constructor can build that belt from the registry — the sync-belt problem that kept this port waiting. The three joined the object model as nodetool.shared.list/read/publish and therefore nodetoolApiCoveredToolNames, so they leave the raw tool catalog the way every wrapped tool does. The CodeAct executor’s upstream-context line teaches await nodetool.shared.read([...]) instead of await tools.read_shared({keys: [...]}), and a shared-handoff eval case keeps the namespace covered. The survey, the sandbox packs, and what the count is made of (2026-08-12) Eager specs left thirty-five extends Tool subclasses. A survey classified every one of them, which is the first time the second counter has an itemized answer rather than a total: Kind Count Where it goes Loop-protocol 12 Exempt by kind — the enumeration above, plus the eval and harness fakes Supervisor 2 Exempt by kind; tracked nowhere until now, hence the paragraph above Infrastructure 3 CapabilityTool, LazyCapabilityTool, GatedCapabilityTool — the wrappers that serve capabilities Sandbox packs 2 Ported by this commit Run-scoped memory 3 list_shared / read_shared / share_result — ported by the shared module above Genuinely dynamic / host bridge 13 Pending an interface conversion, not a port The last row is the honest remainder: the websocket UI bridges and list_renderers, RunNodeTool, SandboxTool, SubAgentTool, the two browser-agent factories, the Code-node tool factory, and the CLI’s execute_code shim. None is a spec plus a function. Each is either a factory whose identity is built at run time from something no spec table can hold (a node’s dynamic slots, a pack’s manifest), or a bridge whose implementation is a transport. They move when the interface they sit on moves — a Tool[] belt becoming a capability list — not by being rewritten as capabilities first. What this commit ported. SandboxPackageDocsTool and SandboxPackageListTool become the packs module — the namespace nodetool.packs.* already wrapped them under. Their three constructor arguments (allowlist, catalog, whether the host mounts platform modules) are per-session state, not per-run: the allowlist is computed where the session is built. So it rides in the implementation’s closure and reaches a belt through toolFromCapability, the shape createSearchTool uses, instead of becoming three fields on CapabilityRun for two capabilities. The registry serves the specs — wire names, Zod schemas, read classification, the drift walk — behind an implementation that fails closed naming nodetool.packs.list(), which is what the dispatcher’s contract asks of a capability whose dependency the run does not carry. CapabilityTool gained the zodSchema accessor toolFromLazyCapability already had, so a malformed call still comes back as invalid_tool_arguments. The two SubAgentTool belt sites — unified-websocket-runner.ts and the CLI’s stdin.ts — now name run_subtask / run_search over a run carrying that turn’s subAgent. The classes still run, one per call, so the depth gate, the read-only filter, and buildChildToolset’s self-stitching are untouched; both hosts still snapshot the parent belt before the unshift, which is what makes that stitching necessary and correct. The move surfaced a real gap: neither spec carried needsToolCallId, which SubAgentTool declares, so a belt built from the specs would have dropped parent_tool_call_id and un-nested every child card in the chat UI. Both specs declare it now. Re-measured (source only — packages/**, no dist/, no test directories): 31 extends Tool across 23 files, down from 33 across 25. Three of the thirty-one are the wrappers, and the loop-protocol group is the bulk of the rest. capabilityFromTool must live until belts stop being Tool[]. PR 12 recorded it as load-bearing in gateTools; the survey says why more precisely. The two gateTools call sites — agent.ts:426 and unified-websocket-runner.ts:5428 — gate heterogeneous belts: registry-backed tools next to host-constructed ones (RunNodeTool, the UI bridges, a Code-node’s dynamic tool). capabilityFromTool is what gives a non-registry tool a category at all, by reading the classification map. Delete it and those tools reach invoke with no category, which is exactly the fail-open the required CapabilitySpec.category exists to prevent. The runner uses it a third time on purpose, at :5589, to hand run_node to a CapabilityRun as a host-supplied capability. It goes when a belt is a capability list, and not before.--- # Workflow Triggers — Technical Design Source: https://docs.nodetool.ai/triggers-design.html Workflow Triggers — Technical Design Author: Matti Georgi Status: Draft — for review Last updated: 2026-07-26 PRD: triggers-prd.md Implementation plan: triggers-implementation-plan.md 1. Summary A trigger is a row in trigger_registrations. One loop claims due rows and dispatches them into the existing job queue. Nothing about triggering lives in the graph. The design rests on one reframe. “Wake up” is doing double duty for two problems: Start a run because something happened (a clock tick, an HTTP POST, a file change). Resume a run parked mid-flight (a human approval, an external callback). The shipped code answers both by keeping a process alive inside the graph, then adds suspension, a durable inbox, and a watchdog to survive the fact that processes die. Problem 1 does not need any of that. It needs a start. Once starting moves out of the graph, problem 2 shrinks enough to defer entirely (PRD §11). 2. Design goals D1. State lives in the database, not in a process. Restart safety is then a property of the storage engine, not of a watchdog. D2. One dispatch path. The clock, the HTTP route, the UI, and the desktop adapter all insert rows; none of them call the runner. D3. The graph stays a pure function of its inputs. No node knows it was triggered. D4. Scheduling logic that is really business logic belongs in the workflow, not the scheduler. D5. Reuse: the job queue, the secret store, the cost ledger, the budget model, and share tokens all exist. Add no parallel versions. 3. Architecture clock tick ─┐ HTTP POST ──┤ desktop ───┼──▶ trigger_registrations ──▶ claim ──▶ job queue ──▶ WorkflowRunner UI / API ───┤ (one row per trigger) │ │ a run ──────┘ │ │ └──── settle ◀───────────┘ (state, next_fire_at, attempts, cost) One loop drives the middle. Everything on the left inserts or updates a row. Everything on the right is unchanged NodeTool. 4. Data model trigger_registrations exists (packages/models/src/schema/trigger-registrations.ts) with id, user_id, workflow_id, node_id, kind, config_json, enabled, cursor, last_fired_at, last_error. It is unused, so it can be reshaped freely. Two changes to what is there: Drop node_id and the unique(workflow_id, node_id) index. There are no trigger nodes, and a workflow needs more than one trigger. Widen cursor TEXT to state JSON. Use case 20 needs a latch ({alerted_at, month}), not a cursor string. Additions: -- scheduling next_fire_at TEXT, -- the loop's only index; NULL = event-driven only every_s INTEGER, -- fixed interval cron_expr TEXT, -- 5-field cron, mutually exclusive with every_s cron_tz TEXT, -- IANA zone, e.g. "Europe/Berlin" -- lifecycle status TEXT NOT NULL DEFAULT 'pending', -- pending | claimed — dispatch state only. -- "Disabled" is not a status: it is enabled=0 -- plus disabled_reason, so a registration can be -- re-enabled without losing its dispatch state. disabled_reason TEXT, -- failures | expired | max_runs | budget | user attempts INTEGER NOT NULL DEFAULT 0, consecutive_failures INTEGER NOT NULL DEFAULT 0, expires_at TEXT, max_runs INTEGER, -- counts successful runs (see §5) run_count INTEGER NOT NULL DEFAULT 0, -- concurrency serialize_key TEXT, -- rows sharing a key run one at a time -- safety max_attempts INTEGER NOT NULL DEFAULT 3, max_usd REAL, budget_period TEXT DEFAULT 'month', -- multi-tenancy and ops secret_scope TEXT, tags JSON, scope TEXT NOT NULL DEFAULT 'cloud', -- cloud | desktop -- event auth auth_mode TEXT NOT NULL DEFAULT 'token', -- token | hmac_sha256 | none auth_secret TEXT, -- reference into the secret store, not the -- secret itself; the row stays safe to list auth_header TEXT -- signature header name for hmac_sha256 Indexes: (status, next_fire_at) for the claim, (workflow_id), (kind, enabled), (serialize_key, status). A second table holds inbound events, so a burst of webhooks does not contend on the registration row: CREATE TABLE trigger_events ( id TEXT PRIMARY KEY, registration_id TEXT NOT NULL, payload_json JSON, dedupe_key TEXT, serialize_key TEXT, -- copied from the registration at insert; a config -- payload path can override it per event, which is -- how per-conversation ordering (use case 5) works status TEXT NOT NULL DEFAULT 'pending', attempts INTEGER NOT NULL DEFAULT 0, next_fire_at TEXT NOT NULL, run_id TEXT, created_at TEXT NOT NULL ); CREATE UNIQUE INDEX idx_trigger_events_dedupe ON trigger_events(registration_id, dedupe_key); CREATE INDEX idx_trigger_events_due ON trigger_events(status, next_fire_at); Events carry their own serialize_key (denormalized at insert) so the claim can enforce ordering across both tables with one predicate: a key is busy if any row in either table holds it in claimed. The existing trigger_inputs table is close to this but is keyed by (run_id, node_id), which is the parked-run addressing scheme from the abandoned design. Drop it with the rest. 5. The dispatch loop // one setInterval(1000) in packages/websocket async function tick() { const due = await claim(); // atomic; see below for (const item of due) void dispatch(item); } async function claim() { // UPDATE ... WHERE status='pending' AND next_fire_at <= now // AND (serialize_key IS NULL OR serialize_key NOT IN ( // SELECT serialize_key FROM ... WHERE status='claimed')) // SET status='claimed' LIMIT n RETURNING * } async function dispatch(item) { const reg = item.registration; if (overBudget(reg)) return disable(reg, 'budget'); // check BEFORE spending, not only at settle if (reg.expires_at && nowIso() > reg.expires_at) return disable(reg, 'expired'); const params = { ...reg.config_json, ...item.payload, // event triggers only state: reg.state, last_fired_at: reg.last_fired_at, now: nowIso(), is_first_run: reg.run_count === 0, }; const job = await jobQueue.enqueue(reg.workflow_id, params, { userId: reg.user_id, secretScope: reg.secret_scope, registrationId: reg.id, }); await settle(item, await job.completion); } settle is where the semantics live. For a scheduled item the retry and next-fire bookkeeping lands on the registration row; for an event item it lands on the event row (the registration only accumulates consecutive_failures, last_error, run_count, and state). Both end by returning the claimed row to pending — a settle that forgets to release the claim is a stuck trigger: async function settle(item, result) { // 1. State: write back whenever the run produced one, success or not. if (result.outputs.state !== undefined) reg.state = result.outputs.state; // 2. Next fire: the run wins, then the schedule. const next = result.outputs.next_fire_at ?? nextScheduled(reg); if (result.ok) { reg.consecutive_failures = 0; reg.run_count++; // counts successes; max_runs and is_first_run key off it reg.last_fired_at = item.claimed_at; // so last_fired_at = last SUCCESSFUL fire; a failed // occurrence re-covers its window on the next run, // which is the at-least-once contract reg.next_fire_at = next; } else { reg.consecutive_failures++; reg.last_error = result.error; reg.attempts++; reg.next_fire_at = reg.attempts < reg.max_attempts ? nowPlus(backoff(reg.attempts)) // 2^attempts seconds, capped at 300s : next; // give up on this occurrence if (reg.attempts >= reg.max_attempts) reg.attempts = 0; } reg.status = 'pending'; // release the claim on BOTH paths if (reg.consecutive_failures >= 5) disable(reg, 'failures'); if (reg.expires_at && next > reg.expires_at) disable(reg, 'expired'); if (reg.max_runs && reg.run_count >= reg.max_runs) disable(reg, 'max_runs'); if (overBudget(reg)) disable(reg, 'budget'); } is_first_run is run_count === 0, so it stays true until the first success. A first run that fails and retries sees is_first_run again, which is what a backfill-seeding run wants. 5.1 Why a 1s poll and not long timers setTimeout clamps any delay above 2³¹−1 ms (~24.85 days) to 1 ms. The kernel documents this at trigger.ts:98 and re-arms in chunks to work around it; WaitNode reintroduces it. A next_fire_at <= now comparison against a 1s tick cannot have the bug, at the cost of up to one second of scheduled latency. Event triggers do not pay that cost (§5.3). 5.2 Claiming is the only lock UPDATE ... WHERE status='pending' is atomic in SQLite and in Postgres (schema-pg mirrors every table). Whoever’s update lands first owns the row. This is why run_leases is unnecessary: the claim is the lease, and it lasts exactly as long as the dispatch. One portability note: UPDATE ... LIMIT ... RETURNING needs an optional SQLite build flag. The portable claim is SELECT the candidate ids, then UPDATE ... WHERE id IN (...) AND status='pending' RETURNING *, keeping only the rows the update actually flipped. Same atomicity, standard builds. A crashed process leaves rows in claimed. A reaper in the same tick returns rows claimed longer than a threshold to pending, which is at-least-once and consistent with the retry semantics. 5.3 Eager dispatch A chat reply cannot wait a second. Inserting a due event calls the same claim-then-dispatch path inline. The poller becomes the safety net for anything an insert-time crash missed. Two callers, one code path, so there is still one place where a run starts. 6. State The state blob is passed in as a run param and read back from an output of the same name. No new node types: a workflow with an Input named state and an Output named state is a stateful trigger, and one without is stateless. It stays runnable by hand: npm run dev:nodetool -- workflows run <id> --params '{"state":{"since":"2026-07-01"}}' 6.1 Write back on failure too Run status controls retry scheduling. The state output controls position. Conflating them is a bug: an RSS run that posts items 1–6 and fails on 7 must not reprocess 1–6, and it will if state only persists on success. The workflow is responsible for advancing state incrementally if it wants partial-progress semantics. A workflow that emits state only at the end gets all-or-nothing, which is a reasonable default. 6.2 What does not belong in it State is what the trigger needs to decide what to fetch next. Forty content hashes qualify at about 3KB. A corpus of every URL ever seen does not, and belongs in a vectorstore collection or a SQLite node. Enforced with a 64KB cap that fails the settle with a clear error, and {v: 1, ...} versioning so the shape can migrate. 6.3 Most pollers need no explicit state last_fired_at and now are passed to every run. “Everything since the last run” is the common case and needs nothing on the row. 7. Scheduling Three sources, in precedence order: The run. A next_fire_at output overrides everything for the next fire. cron_expr + cron_tz. Calendar semantics, DST-correct. every_s cannot express “8am local” (use case 1). every_s. next = anchor + ceil((now − anchor) / every_s) * every_s, which skips missed slots after downtime rather than firing a burst. Self-scheduling (D4) is the load-bearing one. It collapses several features that would otherwise each need their own mechanism: Want Return Poll a render job with backoff now + min(2^n, 300s) Remind a client on Thursday now + 2 days Check the day before each renewal the computed date Nothing happened, back off now + 1h This client is hot, check often now + 30s Cron and interval become defaults for the first fire. After that the workflow schedules itself, in the graph, where the user can see and edit the logic. 8. Concurrency Rows sharing a serialize_key run one at a time, enforced in the claim. Poll registrations default serialize_key to their own id, which is the no-overlap rule (use case 11, use case 16) as a special case rather than a bespoke mechanism. Event triggers leave it null for concurrency, or set it per conversation for ordering. Global limits come from the job queue: MAX_CONCURRENT_JOBS (default 4) and MAX_CONCURRENT_RUNS_PER_WORKFLOW (default 4), already in packages/websocket/src/settings-registry.ts. A 50-comment webhook burst inserts 50 rows and drains at the queue’s rate. No new limiter. 9. Events and auth One route: POST /api/triggers/:token. The token identifies the registration. Auth is per registration: token — the URL token is the credential. Fine for internal and desktop adapters. hmac_sha256 — verify auth_header against HMAC-SHA256 of the raw body with auth_secret, compared in constant time. Required for GitHub, Stripe, and most providers. The raw body must be captured before JSON parsing. none — explicit opt-in, warned in the UI. The route inserts a row and returns 202 {event_id, run_id?} — run_id is present when eager dispatch (§5.3) enqueued the run inline, absent when the queue was saturated and the poller will pick the event up. A duplicate dedupe_key returns 200 with the original event’s ids and inserts nothing. A disabled or expired registration returns 410 and inserts nothing, so a forgotten webhook endpoint cannot accumulate a backlog that fires on re-enable. Body cap 1MB. dedupe_key comes from a configurable header (X-GitHub-Delivery) or a JSON body path (Stripe’s id — its Stripe-Signature header is a signature, not a delivery id) so provider retries do not duplicate runs. Regenerating a registration’s token invalidates the old URL; that is the revocation story for a leaked token-mode URL. Client-facing approval links (use case 13) reuse nodetool_workflow_shares as the token source instead of minting a second capability system. 9.1 Trigger payloads are untrusted Anyone who can email the agency, comment on a PR, or submit a form can put text in front of an agent that has tools and a wallet. This is the one requirement that is mostly docs and defaults rather than code: Templates wrap payloads in a delimited untrusted-content block. Triggered runs disable destructive tools unless explicitly enabled on the registration. The editor warns when a triggered workflow gives an agent write or shell tools. 10. Multi-tenancy nodetool_secrets is uniquely keyed (user_id, key), so one human has exactly one GA_TOKEN. Eighteen clients need eighteen. secret_scope on the registration makes the resolver try <scope>/<key> then fall back to <key>. One string on the row, one prefix in the lookup, no new store, and unscoped workflows keep working unchanged. tags covers bulk lifecycle (a churned client’s eight triggers disabled in one action). Stamping registration_id on the cost ledger gives per-client spend from records that are already written. Then use case 17 removes the need for the platform to do any of this itself: registrations are ordinary data a workflow can create, update, and delete. An onboarding workflow registers a weekly-report trigger per client with that client’s config, scope, and tags. Offboarding deletes by tag. Eighteen clients or two hundred, no fan-out feature. 11. Adapters Filesystem watching needs a live process, and it belongs in the host rather than the graph. The Electron main process reads trigger_registrations WHERE kind='file' AND enabled=1, watches, debounces, waits for writes to settle, and POSTs to the trigger route. It restarts with the app and holds no state the database does not have. scope = 'desktop' marks registrations the cloud cannot serve, so a user who builds a folder-watch workflow and then signs in on the web is told why it is idle instead of wondering. IMAP polling is scope = 'cloud' and is an ordinary polling workflow, not an adapter. 12. What gets deleted Path Lines packages/kernel/src/trigger.ts, trigger-manager.ts, trigger-wakeup.ts 784 packages/kernel/tests/trigger*.test.ts 1143 packages/automation-nodes/src/nodes/triggers.ts (all but Wait) ~660 Workflow.hasTriggerNodes() and its tests ~40 trigger_inputs, run_inbox_messages, run_leases, workers schemas and models ~200 Wait stays, with the setTimeout clamp fixed and a documented ceiling. run_node_state and run_leases were deleted on 2026-08-26 along with the whole suspend/resume path — see PRD §11. The kernel symbols are exported from @nodetool-ai/kernel, so they are deprecated in one release and deleted in the next. Net: roughly 2800 lines removed, roughly 900 added. 13. Alternatives considered Wire up the existing durable-execution substrate. run_inbox_messages, run_leases, and run_node_state described a durable actor runtime with claim leases and per-node inbox persistence. It solves resume-from-mid-flight, which this design explicitly does not (PRD §N1). Rejected: polling and webhooks retry from the top correctly, and approval flows split into two workflows. run_inbox_messages was kept and is now the trigger delivery inbox; run_leases and run_node_state were deleted on 2026-08-26. Keep trigger nodes, fix them. Rejected. Every fix (cancellation, restart, port binding, persistence) is a step toward reimplementing a scheduler inside the runtime, and it leaves the graph impure and untestable by hand. Cron in the process, jobs in memory. Rejected. The watchdog in trigger-manager.ts exists only because in-memory schedules die with the process. Moving the schedule to a row deletes the watchdog, the singleton, the start/stop race, and the abort bookkeeping. A separate wakeups table for both starts and resumes. Attractive for Phase 2 symmetry, but it splits scheduling state across two tables in Phase 1 for no gain. Phase 2 can add kind='resume' rows to trigger_events addressed at (run_id, node_id). 14. Open questions Reaper threshold for stuck claimed rows. A 25-minute video render (use case 16) must not be reaped as stuck. Proposal: derive from the job’s own timeout rather than a fixed constant. Does next_fire_at from a run need a floor? A workflow returning now loops as fast as the queue allows. Proposal: clamp to 1s and log. Budget accounting granularity. application_budgets reserves an estimate before the run and reconciles after. Reuse that ledger keyed on registration_id, or add a simpler per-period sum over the cost records? Should state write-back require an explicit opt-in on the registration, so a workflow that happens to have an output named state does not silently become stateful? Proposal: keep it automatic — outputs are user-named, so an output called state is already deliberate — but display the detected state contract in the Triggers tab when the registration is created, so accidental statefulness is visible rather than silent.--- # Implementation Plan: Workflow Triggers Source: https://docs.nodetool.ai/triggers-implementation-plan.html Implementation Plan: Workflow Triggers Author: Matti Georgi Status: Draft — for review Last updated: 2026-07-26 PRD: triggers-prd.md Design doc: triggers-design.md This plan turns the design into ordered, reviewable PRs with file-level targets and exit criteria. Estimates assume one engineer full-time. Workstreams marked ∥ can run in parallel with a second engineer. 0. Ground rules Every PR passes npm run check (typecheck, lint, test) and follows repo conventions: strict TS, no any, ui_primitives only, design tokens, Zustand selectors, tests in __tests__/. Schema changes are additive, mirrored in packages/models/src/schema/ and packages/models/src/schema-pg/, registered in packages/models/src/migrations/versions.ts, and asserted in migrations.test.ts via tableExists. New backend code lands in packages/triggers (a new workspace package) plus surgical changes to packages/websocket, packages/models, and web/. No changes to packages/kernel or packages/runtime except the deletions in PR 9. Feature flag through the existing settings registry (packages/websocket/src/settings-registry.ts → registerSetting): triggers_enabled, default off until PR 8. The dispatcher runs in the websocket process. It starts only when the flag is on. No PR leaves the tree with a trigger that can fire without the safety rules from PR 3. PRs 1–3 land together or not at all. Phase A — the mechanism PR 1 — Schema and model ∥ Files packages/models/src/schema/trigger-registrations.ts — drop node_id and its unique index, widen cursor to state, add the columns from design §4 (including disabled_reason) packages/models/src/schema/trigger-events.ts — new; carries its own serialize_key, denormalized from the registration at insert packages/models/src/schema-pg/trigger-registrations.ts, schema-pg/trigger-events.ts — mirrors packages/models/src/trigger-registration.ts — rewrite the model; add findDue, claim, settle, disable packages/models/src/trigger-event.ts — new packages/models/src/migrations/versions.ts — one migration reshaping trigger_registrations, one creating trigger_events, one dropping trigger_inputs packages/models/tests/trigger-registration.test.ts, migrations.test.ts Exit criteria claim() is atomic: a test firing 20 concurrent claims against 5 due rows dispatches each row exactly once. serialize_key excludes a second row while the first is claimed. The reaper returns rows claimed past the threshold to pending. Migration up and down both pass on SQLite and Postgres. Estimate: 3 days PR 2 — Scheduler package Files packages/triggers/ — new workspace (package.json, tsconfig.json, src/index.ts) src/schedule.ts — nextScheduled(reg): cron with IANA timezone, interval with slot skipping, self-scheduled override with a 1s floor src/dispatcher.ts — the tick, claim → dispatch → settle src/settle.ts — the state, retry, and disable rules from design §5 packages/triggers/tests/ — schedule table tests including DST transitions in both directions, settle-semantics tests Dependencies: a cron parser. Prefer cron-parser (no native deps, timezone support). Add to packages/triggers/package.json explicitly, per the automation-nodes lesson about hoisting. Exit criteria DST: a 0 8 * * * / Europe/Berlin registration fires once at local 8am across both transitions, verified with a fixed clock. Interval after 3 hours of downtime fires once, not 12 times. A run returning next_fire_at overrides the schedule; returning now clamps to 1s and logs. State writes back on a failed run that emitted one; does not write when the output is absent. Settle returns the claimed row to pending on success and failure — a claim leaked by the failure path is a permanently stuck trigger. 5 consecutive failures disables. expires_at, max_runs, and max_usd each disable with a distinct reason. Estimate: 4 days PR 3 — Wire the dispatcher to the job queue Files packages/websocket/src/triggers/service.ts — start and stop the tick with the server lifecycle, behind triggers_enabled packages/websocket/src/unified-websocket-runner.ts — accept secretScope and registrationId on enqueue; thread them onto the run packages/runtime/src/secrets.ts (resolver) — try <scope>/<key> then <key> packages/websocket/src/settings-registry.ts — registerSetting({ key: "triggers_enabled", ... }) cost ledger write path — stamp registration_id Exit criteria An interval registration runs a real workflow end to end and its state round-trips. secret_scope resolution falls back correctly and is covered by a test with both a scoped and an unscoped key. Dispatch respects MAX_CONCURRENT_JOBS; a burst of 50 events drains without exceeding it. Flag off means no timer is created. Estimate: 3 days Phase B — the surfaces PR 4 — HTTP event route Files packages/websocket/src/routes/triggers.ts — POST /api/triggers/:token packages/websocket/src/server.ts — register the route, add the path to the unauthenticated allowlist alongside /api/kie/webhook raw-body capture before JSON parsing, for HMAC packages/websocket/tests/routes/triggers.test.ts Exit criteria HMAC-SHA256 verification against a real GitHub sample payload, compared in constant time; a tampered body is rejected 401. Body over 1MB is rejected 413. Duplicate dedupe_key returns 200 with the original event’s ids and starts no second run. A disabled or expired registration returns 410 and inserts nothing. Response is 202 {event_id, run_id?} in under 50ms, with dispatch happening eagerly rather than on the next tick. Estimate: 3 days PR 5 — tRPC CRUD and the Triggers node ∥ Files packages/websocket/src/trpc/routers/triggers.ts — list, create, update, delete, fire now, by tag packages/automation-nodes/src/nodes/trigger-admin.ts — a node that creates, updates, and deletes registrations (use case 17) packages/cli/src/commands/triggers.ts — nodetool triggers list|create|fire|disable tests for each Exit criteria A workflow can register a trigger for another workflow, and the created registration fires. nodetool triggers list --tag client-7 filters. Deleting a workflow cascades to its registrations. Estimate: 3 days PR 6 — Triggers tab ∥ Files web/src/components/workflows/TriggersPanel.tsx and children web/src/hooks/useTriggers.ts — TanStack Query, hierarchical keys web/src/components/workflows/TriggerCreateDialog.tsx — schedule or webhook, the first-run backfill question, the secret display Registered in the workflow’s tab set Exit criteria ui_primitives only, design tokens only, no raw MUI, verified by lint. Enabling a polling trigger asks the backfill question and seeds state. Last error, next fire, and disabled-with-reason are all visible without opening anything. Webhook URL and secret are copyable; the secret is shown once. RTL tests use getByRole and userEvent. Estimate: 4 days PR 7 — Desktop file adapter ∥ Files electron/src/triggers/fileWatcher.ts — read desktop registrations, watch, debounce, wait for write settle, POST electron/src/main.ts — start and stop with the app web/ — mark scope: 'desktop' registrations as unavailable in the browser with the reason Exit criteria A file written in three chunks produces one event. A watch path that disappears disables the registration with a clear error rather than throwing in a loop. Restarting the app resumes every enabled desktop registration. Estimate: 3 days Phase C — safety, rollout, deletion PR 8 — Guardrails and enable Files packages/websocket/src/triggers/budget.ts — per-period spend check against the cost ledger keyed on registration_id, reusing the application_budgets reserve-then-reconcile shape workflow save path — validate the state input/output contract when a registration exists; warn on break (O6) agent tool policy — destructive tools off by default in triggered runs; editor warning when a triggered workflow gives an agent write or shell tools (O8) templates — wrap trigger payloads in a delimited untrusted-content block triggers_enabled default flips to on Exit criteria A registration over max_usd disables with a budget reason and does not start the run. Renaming the state output on a workflow with a live registration produces a save-time warning. A triggered agent run cannot call a destructive tool unless the registration opts in. Estimate: 4 days PR 9 — Delete the dead layer Files removed packages/kernel/src/trigger.ts, trigger-manager.ts, trigger-wakeup.ts and their exports from src/index.ts packages/kernel/tests/trigger.test.ts, trigger-manager.test.ts, trigger-wakeup.test.ts ManualTrigger, IntervalTrigger, WebhookTrigger, FileWatchTrigger from packages/automation-nodes/src/nodes/triggers.ts Workflow.hasTriggerNodes() and its tests trigger_inputs, run_inbox_messages, run_leases, workers schemas, models, and migrations Files changed WaitNode — re-arm in chunks below 2³¹−1 ms, mirroring the fix at packages/kernel/src/trigger.ts:98, and document the ceiling packages/dsl/src/generated/nodetool.triggers.ts — regenerate packages/automation-nodes/README.md, AGENTS.md — drop the trigger section, point at the new docs CLAUDE.md — add the triggers CLI and the packages/triggers package to the architecture list Migration for existing users: any workflow containing a removed node type fails validation with a message pointing at the Triggers tab. Nothing auto-migrates, because none of these nodes work today and no user can have a functioning trigger workflow. Deprecation: the kernel symbols are exported from @nodetool-ai/kernel. Mark them @deprecated in the release before this PR, delete in the release after. Exit criteria npm run check green. rg "hasTriggerNodes|TriggerWorkflowManager|TriggerWakeupService" returns nothing outside changelogs. Net line count of the whole feature is negative. Estimate: 2 days PR 10 — Docs and templates ∥ Files docs/triggers.md — user guide: scheduling, webhooks, state, self-scheduling, safety docs/cookbook/ — three worked examples: the 18-client weekly report, the RSS repurposer with per-item cursor advance, the client-approval two-workflow split Template workflows shipped with app_doc, generated by the existing scripts/generate-template-apps.mjs Exit criteria Every template that posts to a third party documents its action-level dedupe key. The RSS example demonstrates partial-progress state, since that is the semantic most likely to be misread. Estimate: 3 days Sequencing PR1 ─┬─ PR2 ── PR3 ─┬─ PR4 ─┬─ PR8 ── PR9 │ ├─ PR5 ─┤ │ ├─ PR6 ─┤ └──────────────┴─ PR7 ─┴─ PR10 PRs 1–3 are the critical path and land as one reviewable stack. Phase B parallelizes across two engineers. PR 8 gates the flag; PR 9 cannot land before it. Total: ~32 engineer-days, or about 3 weeks with two engineers on Phase B. Rollout checklist triggers_enabled off in production while PRs 1–7 land. Dogfood: the repo’s own weekly examples-freshness check runs as a cron registration for two weeks. Deploy-survival test: enable a 1-minute interval trigger, deploy, confirm exactly one fire per minute across the restart with no duplicate dedupe_key. Burst test: 200 webhook posts in 10 seconds, confirm the queue drains and no run is lost or duplicated. Poison test: a registration whose workflow always throws disables itself after 5 failures with the error visible in the UI. Budget test: a registration with max_usd = 1 stops after the run that crosses it. Flip triggers_enabled on by default. Deprecate the kernel trigger exports. Next release: PR 9. Risks to the plan Risk Handling Cron plus timezone plus DST is where scheduler bugs live PR 2 exits on a fixed-clock table test covering both transitions; no hand-rolled cron The eager-dispatch path bypasses a rule the tick applies Both callers go through the same claim → dispatch → settle; a test asserts parity Reaper reclaims a live 25-minute render Open question 1 in the design; resolve before PR 3 Deleting kernel exports breaks an external consumer One-release deprecation, PR 9 is last Phase 2 pressure arrives early (“but I need approval mid-run”) Use case 6’s two-workflow split is the documented answer; revisit only when a case it cannot serve appears--- # PRD: Workflow Triggers Source: https://docs.nodetool.ai/triggers-prd.html PRD: Workflow Triggers Author: Matti Georgi Status: Draft — for review Last updated: 2026-07-26 Design doc: triggers-design.md Implementation plan: triggers-implementation-plan.md 1. Summary A NodeTool workflow runs when a human presses Run. Triggers make it run on its own: at 8am every weekday, when a webhook fires, when a file lands in a folder, when a client approves a draft. Five trigger nodes ship in the palette today (nodetool.triggers.*). None of them work as an automation, and one of them cannot work at all. Behind them sits a kernel trigger runtime and six database tables that were ported from the Python codebase and never connected to anything. This PRD proposes replacing all of it with one idea: a trigger is a row in a table, not a node in a graph. The graph goes back to being a pure function of its inputs. A scheduler outside the graph decides when to call it and with what. The scope is one table, one dispatch loop, one HTTP route, and a Triggers tab on the workflow. Estimated cost is roughly 3 engineer-weeks to a shippable Phase 1, with a net deletion of about 1900 lines. 2. Problem 2.1 What ships today does not work Node State ManualTrigger Cannot receive events. Its inbox has no registered upstream, so run() sees end-of-stream on the first read and exits emitting nothing. pushInputValue() writes to a node’s downstream inboxes, not its own, so the mechanism named in its own doc comment cannot reach it. WebhookTrigger Binds http.createServer inside the workflow process. Two concurrent runs collide on the port, and on Fly only the app port is routed, so it is unreachable from outside. No body size cap, no signature verification, always answers 200 with no way to return a result. IntervalTrigger Works, but loops forever with no cancellation. interval_seconds has no minimum, so 0 produces a 1ms hot loop. FileWatchTrigger Works locally. Leaks one map entry per touched path for the life of the watcher. Cannot run on Fly, which has no filesystem to watch. Wait Any wait over ~24.8 days returns instantly. setTimeout clamps delays above 2³¹−1 ms, a pitfall the kernel documents at packages/kernel/src/trigger.ts:98 and works around, and that WaitNode.process() then reintroduces. None of the three generator nodes observe runner.cancel(). Cancelling a webhook run leaves the port bound and the generator parked until the process exits. There is no product surface. Nothing in web/src references trigger nodes. A user who drops one on the canvas gets a run that never finishes. 2.2 What was built and never connected packages/kernel/src/trigger.ts, trigger-manager.ts, trigger-wakeup.ts (784 lines, 1143 lines of tests, 63 test cases). Zero non-test callers. Workflow.hasTriggerNodes() (packages/models/src/workflow.ts:86). Zero callers. Six tables with models and migrations, no callers: trigger_registrations, trigger_inputs, run_inbox_messages, run_leases, run_node_state, workers. resumeJob() (packages/websocket/src/unified-websocket-runner.ts:2978) delegates to reconnectJob(). Resume-from-suspension does not exist. The code is careful and well tested. It guards races that cannot happen, because nothing calls it. The tests report green on a feature that is not there. 2.3 Why this matters now Every workflow in NodeTool is a thing a person opens. Triggers are what turn it into a thing that runs while they sleep. For the target user that difference is most of the product’s value: the report writes itself on Monday morning, the brief becomes a proposal before anyone reads the email, the render starts overnight. 3. Users Solo freelancers (designers, editors, copywriters, developers, social managers). One person, 3–15 clients, no ops budget. They need automation that survives a laptop reboot and does not require a server. Small agencies (2–30 people). The same workflow run for 18 clients, each with its own credentials, brand config, and invoice line. They need per-client isolation, bulk lifecycle, and cost attribution. Both share one hard constraint: a broken automation that silently spends money is worse than no automation. Operational safety is a feature requirement, not a polish item. 4. Use cases Twenty-one scenarios were run against the design. These are the ones that shaped it. 4.1 Scheduled work # Scenario What it requires 1 Every weekday 8am, summarize unread email, post to Slack cron with timezone, not a fixed interval 4 Poll an RSS feed every 15 min, process only new items a cursor; partial-failure progress 12 Weekly report for 18 clients from one workflow per-registration config, credentials, cost attribution 15 Daily campaign posts until Nov 30 an end date 16 Monday 60-second brand video, 25 min, ~$8 per run no overlap, spend ceiling, low retry count 18 Watch 40 competitor pricing pages, alert on real changes small state on the row, corpora elsewhere 20 Alert when a retainer client exceeds their monthly hours state as a latch, not a cursor, so the alert fires once 4.2 Event-driven work # Scenario What it requires 2 GitHub PR comment starts a workflow HMAC signature verification, burst tolerance 3 New PDF in ~/Downloads gets OCR’d and filed a desktop adapter; the trigger cannot run in the cloud 5 Chat message gets a workflow reply sub-second dispatch latency 19 New YouTube upload becomes 5 shorts and 3 posts action-level dedupe so a retry does not double-post 21 Client emails a brief, workflow drafts a proposal IMAP polling; the payload is untrusted input to an agent 4.3 Work that waits # Scenario What it requires 6 Draft an email, wait for approval, then send splits into two workflows plus a stored draft; no parked run needed 13 Client picks one of 4 ad variants via a link a share-token trigger, a reminder, an expiry 14 Poll a render farm until the 4K export finishes the run decides when it next fires 4.4 Operations # Scenario What it requires 7 Re-run last night’s failed job manual fire, visible last error 8 A broken workflow fires every minute for a day consecutive-failure auto-disable, backoff 9 Someone renamed an output while the trigger was live validate the state contract on save 10 First tick of a new RSS trigger ask whether to backfill, seed initial state 11 Per-sender ordering; a poll that must not overlap itself one serialization key covers both 17 New client signs, provision everything including their triggers a run can create registrations 5. Goals G1. A workflow can be started by a schedule or an external event, with no trigger node in the graph. G2. A triggered workflow is an ordinary workflow. It runs from the editor, from the CLI, and from a trigger with the same inputs and the same result. G3. Schedules survive process restart, deploy, and crash without a watchdog. G4. A trigger cannot silently burn money: bounded retries, a spend ceiling, and auto-disable on repeated failure. G5. One workflow serves many clients, each with its own config, credentials, state, and cost line. G6. Net deletion. The feature ships smaller than the dead code it replaces. 6. Non-goals N1. Resuming a run that died mid-flight. A failed run retries from the top. This is correct and cheap for polling and webhooks. It is wrong for expensive multi-step runs with non-idempotent side effects in the middle, and that is the case that would justify the durable-execution substrate already sitting in the schema. Not now. N2. Synchronous webhooks. The endpoint answers 202 with a run id. Returning workflow output in the HTTP response is a separate feature. N3. A distributed scheduler. One process claims and dispatches. The claim is safe across machines, but there is no work-stealing, no partitioning, and no leader election. N4. Sub-second scheduled precision. Dispatch is on a 1s tick. Event triggers dispatch eagerly and do not pay it. 7. Requirements 7.1 Functional F1. A registration binds a workflow to a schedule or an event source, with a JSON config, and can be enabled or disabled. F2. Schedules are expressed as a fixed interval or a cron expression with an IANA timezone. F3. A run may return a next_fire_at that overrides its schedule for the following fire. F4. A registration carries a JSON state blob. It is passed to the run as a param and written back from the run’s state output whenever the run produces one, regardless of run status. F5. Every run receives last_fired_at, now, and is_first_run as params. last_fired_at is the last successful fire, so a failed occurrence’s window is re-covered; is_first_run stays true until the first success. F6. POST /api/triggers/:token inserts an event and returns 202 with the event id, plus the run id when the run was dispatched inline. A duplicate delivery returns 200 with the original ids; a disabled or expired registration returns 410. Auth is per registration: shared token, HMAC-SHA256 over the raw body, or none. F7. Rows sharing a serialize_key run one at a time. Poll registrations default to their own id. F8. A registration may set expires_at and max_runs. F9. Secrets resolve <secret_scope>/<key> before falling back to <key>. F10. Registrations carry tags for bulk enable, disable, and filtering. F11. A workflow can create, update, and delete registrations through a node and the REST API. F12. Desktop-scoped registrations (filesystem) are driven by an adapter in the Electron main process and are marked as unavailable in the browser. 7.2 Operational O1. After N consecutive failures (default 5) a registration disables itself and retains the last error. O2. Retries back off exponentially, capped by max_attempts (default 3). O3. A registration may set max_usd per period. The check runs before dispatch as well as after settle, and exceeding it disables the registration rather than skipping a run silently. O4. Cost records carry registration_id so spend groups by trigger and by tag. O5. Dispatch goes through the existing job queue and inherits MAX_CONCURRENT_JOBS and MAX_CONCURRENT_RUNS_PER_WORKFLOW. O6. Saving a workflow that a registration depends on validates the state input/output contract and warns when it breaks. O7. Deleting a workflow deletes its registrations. O8. Trigger payloads are marked untrusted. Templates that feed one into an agent wrap it in a delimited untrusted-content block, and the editor warns when a triggered workflow gives an agent destructive tools. 8. UX Triggers tab on the workflow. A list of registrations: kind, schedule or endpoint, last fired, last error, next fire, enabled toggle, and a Run now button. Adding one asks for the schedule or copies the webhook URL and secret. First-run question. Enabling a polling trigger asks whether to process existing items or only new ones. The answer seeds the initial state. Failure surfacing. A disabled-on-failure registration shows in the tab and in the workflow list. The error text is the run’s error, not a generic string. Cost. nodetool costs gains --by-trigger and --tag. 9. Success metrics A workflow can be scheduled and running within 60 seconds of opening the Triggers tab, with no code. Zero registrations in the wild that have failed more than 5 times consecutively and are still enabled. Scheduled fires survive a deploy with no missed or duplicated runs, verified by a dedupe-key assertion in the rollout check. Net line count of the change is negative. 10. Risks Risk Mitigation At-least-once delivery double-posts to a client’s social account Action-level dedupe keyed on the source item id, documented in every template that posts A cron trigger with an agent is a standing prompt-injection target O8: untrusted-content framing, destructive tools off by default in triggered runs, editor warning Users put business data in state and it becomes an unqueryable second database 64KB cap with a clear error, and docs pointing at collections and SQLite nodes Backfill on first enable generates hundreds of paid runs The first-run question, plus max_usd Deleting the dead kernel layer removes something a downstream consumer imports The symbols are exported from @nodetool-ai/kernel; deprecate in one release, delete in the next 11. Phasing Phase 1 (this PRD). Registrations, the dispatch loop, cron and interval schedules, the webhook route, self-scheduling, state, serialization, the safety rules, the Triggers tab, and the desktop file adapter. Deletion of the dead layer. Phase 2 — dropped (2026-08-26). Parked runs (a node that suspends a run until an event addresses it) needed a resume path that was never built: only the suspending node’s state was ever persisted, so a resume would have re-run every upstream node. The suspend half, run_leases, and run_node_state are deleted. Use case 6’s two-workflow split is the answer for approval flows. Reviving this means designing output persistence and a replay contract first, not restoring the tables.--- # Triggers Source: https://docs.nodetool.ai/triggers.html A trigger starts a workflow run when something happens, so you do not have to press Run. You add a trigger node to the graph, save the workflow, and arm the trigger. From then on the NodeTool server starts one run for every event. Quick Access: Add a trigger node from the node menu, save the workflow, then click the bolt button in the floating toolbar and turn on Workflow active. What a trigger is Each trigger node in a workflow becomes a trigger registration when you save. The registration is what the server watches. It records whether the trigger is armed, when it last fired, the last error, and how many runs it has started. Saving never arms a trigger. A new registration starts off, and an existing one keeps the state you last set. Arming is always something you do on purpose. When an event arrives, the server starts a separate run of the workflow and hands the event to the trigger node, which passes it to the nodes connected to its outputs. Each event is one run. Triggered runs have no editor open, so they run headlessly in the server and appear in your run history like any other job. A triggered run also receives three parameters. An Input node with a matching name picks them up, and a workflow without such nodes ignores them. Parameter Value last_fired_at When this trigger last fired, or null now The time the run started, as an ISO 8601 string is_first_run True until one run has completed successfully Trigger nodes The nodes live in the nodetool.triggers namespace. Interval, file watch, webhook, and manual triggers create registrations. The Wait node is not a trigger and is listed here because it sits in the same group. Interval Trigger Fires on a fixed interval. It is the schedule trigger. Setting Default What it does Interval Seconds 60 Seconds between runs Initial Delay Seconds 0 Delay before the first run Emit On Start On When on, the first run is due as soon as the trigger is armed after the initial delay. When off, the first run waits one full interval Max Events 0 Applies only when the node runs inside the editor. The scheduler ignores it Include Drift Compensation On Applies only when the node runs inside the editor. The scheduler ignores it Outputs: tick, elapsed_seconds, interval_seconds, timestamp, source, event_type. The server schedules each run from the previous one, so a slow or stopped server shifts the next run instead of catching up on missed ones. The scheduler checks for due triggers every 5 seconds, so timing is accurate to a few seconds, not to the millisecond. An interval has no fixed clock times. To run at 09:00 every day you need a different approach, such as an external scheduler calling a webhook. File Watch Trigger Fires when a file or folder changes. Setting Default What it does Path . File or folder to watch. A relative path resolves against the server’s working directory, so use an absolute path Recursive Off Also watch subfolders Patterns * File names to include, such as *.csv. Matching uses the file name only. * matches any run of characters and ? matches one character Ignore Patterns none File names to skip. These are checked before Patterns Events created, modified, deleted, moved Which event types start a run Debounce Seconds 0.5 Ignores a repeat event for the same path within this window Max Events 0 Applies only when the node runs inside the editor Outputs: event, path, dest_path, is_directory, timestamp. The server watches the path on its own machine, so the folder must exist there. If the path does not exist, the trigger stays quiet and the popover shows Watch path does not exist. The server’s file watcher reports created, modified, and deleted. It does not report moved events, and dest_path is empty. The server does not see changes made while it is not running. Webhook Trigger Fires when an HTTP request arrives at the trigger’s URL. The node has no settings. Saving the workflow creates the URL token and a shared secret, and the popover shows both. They stay the same across later saves. Outputs: body, headers, query, method, path, timestamp, source, event_type. See Calling a webhook. Manual Trigger Fires only when you click Fire now in the trigger popover, or when something calls the triggers.fire procedure. Use it to start a workflow by hand from the popover, or as a stand-in while you build another trigger. Setting Default What it does Name manual_trigger Emitted on the source output Max Events 0 Applies only while the node listens in a running workflow Timeout Seconds empty Applies only while the node listens in a running workflow Outputs: data, timestamp, source, event_type. Wait Pauses a workflow for a fixed delay and passes its input through unchanged. It is a normal node, not a trigger, so it has no popover entry and starts no runs. Setting Default What it does Timeout Seconds 0 Seconds to wait. 0 means no wait Input empty Value to pass through Outputs: data, resumed_at, waited_seconds. Add a trigger and arm it Open the workflow and add a trigger node from the node menu. Set its properties and connect its outputs to the rest of the graph. Save the workflow. A trigger registration exists only after a save, and a changed setting takes effect only after the next save. Click the bolt button in the floating toolbar. It appears only when the graph contains at least one trigger node. Turn on Workflow active to arm every trigger, or turn on Enabled for one trigger. Until you save, the switch is off and the popover says to save the workflow first. Removing a trigger node and saving deletes its registration, including a webhook’s token and secret. Adding the node again creates new ones. The toolbar button reads Triggers: active, partly active, or inactive to screen readers, with the extra note last run failed or disabled automatically when that applies. In the popover, the header badge shows the same state: Badge Meaning Active Every trigger in the workflow is armed Partly active Some are armed. Turning on Workflow active arms the rest Inactive None are armed Not registered The workflow has no saved registrations yet Unavailable The status could not be loaded. Check the connection to the server The bolt badge A bolt appears next to a workflow’s name in the workflow list when that workflow has an armed trigger or a trigger that stopped by itself. Hover it for the explanation. Bolt Tooltip Meaning Green Trigger armed or <n> triggers armed Listening, and the last run did not fail Red Trigger armed, last run failed: ... Armed, but the most recent run ended in an error Red Disabled after 5 consecutive failures., or an expiry or run-limit message The server turned the trigger off A workflow with no armed or stopped triggers shows no bolt. Automatic disarm The server disarms a trigger by itself in three cases. Failures. After 5 failed runs in a row, the trigger turns off. A successful run resets the count. Expiry. A registration with an expiry time turns off once that time passes. Run limit. A registration with a maximum number of runs turns off after it reaches that number of successful runs. Neither the expiry nor the run limit can be set from the editor. Failures are the case you will meet. A trigger that was disarmed this way shows Stopped in the popover with the reason and the line Turn it back on to retry. Turning it on clears the failure count and the reason, so the trigger starts fresh. If the cause is not fixed, it stops again after 5 more failures. The trigger popover Click the bolt button in the toolbar to open it. It lists one block for each trigger node, labeled Webhook, Schedule, File watch, or Manual. Item What it shows Status badge Active, Inactive, or Stopped (disarmed by the server) Enabled Arms or disarms this one trigger Schedule line For an interval trigger, Runs every 5m — next in 4m. due now means the next run is waiting for the scheduler’s next check Last fired The local date and time of the last event, or Never Stopped reason Why the server disarmed the trigger Last error The message from the most recent failure. A later successful run clears it Webhook URL and secret Webhook triggers only. Each has a copy button, and the secret is hidden until you click the eye button Fire now Starts one run immediately. Disabled until the trigger is armed An error in the toggle or in Fire now appears as a notification, for example Could not fire trigger: Trigger is not active. Fire now sends an empty event payload. It is the quickest way to test the rest of the graph. To test a webhook with real data, send a request to its URL. Calling a webhook Copy the Webhook URL and the secret from the popover. The URL has the form /api/webhooks/{token} on your server. Send the secret in the x-webhook-secret header. curl -X POST "http://localhost:7777/api/webhooks/YOUR_TOKEN" \ -H "x-webhook-secret: YOUR_SECRET" \ -H "Content-Type: application/json" \ -d '{"order_id": 41, "status": "paid"}' { "status": "accepted", "input_id": "webhook:YOUR_TOKEN:9f2c…", "duplicate": false } The trigger node outputs the request as body, headers, query, and method. A body that is not JSON arrives as a plain string. The secret is removed from headers before anything is stored. The route has no session. Anyone with both the URL and the secret can start runs, so treat the secret like a password and send requests over HTTPS when the server is reachable from the internet. The route enforces these limits: Limit Value Body size 1 MiB, larger bodies get 413 Rate 120 requests per minute for the route, over which 429 is returned Retries Send x-webhook-id to make a retry count once. Without it, an identical body in the same minute counts once The 404, 401, and 410 responses, and the idempotency rules, are described in Triggering a Workflow by Webhook. Where triggers run Triggers run inside the NodeTool server process, so the server must be running for any of them to fire. Nothing fires while the server is stopped. Desktop. The server runs on your computer, so file watch triggers see your own folders. The webhook URL points at that local server, so an outside service can reach it only if you expose the port. Self-hosted. The server watches folders on the machine it runs on, which inside a container is the container’s filesystem. Mount the folder into the container. See Self-Hosted Deployment. On restart, a webhook or file event that the server stored but had not yet run starts its run. A missed interval tick does not repeat. The trigger fires once when the server comes back, then continues from there. To turn trigger handling off for a process, set NODETOOL_DISABLE_TRIGGERS=1. The process then starts no dispatcher, scheduler, or file watcher, and it does not serve the webhook route, so webhook requests return 404. Use it when a second server shares the same database and should not run triggers too. See Configuration. You can test a triggered workflow from the command line without arming anything. nodetool debug and nodetool run accept --trigger-event to deliver one event to a trigger node. See CLI. The mobile app has a Triggers screen for monitoring. See Mobile App. Troubleshooting The bolt button is missing. The graph has no interval, file watch, webhook, or manual trigger node. The Wait node does not count. The switch is off and says to save first. The workflow has no registrations. Save it. Nothing fires. Check that the trigger shows Active, not Inactive. Check that the server is running and that NODETOOL_DISABLE_TRIGGERS is not set on it. Save the workflow after changing a property. The server uses the last saved settings. An interval trigger fires right after I arm it. A schedule is measured from when the registration was created or last fired. If that moment is more than one interval ago, the next run is already due and starts at the scheduler’s next check. Turn off Emit On Start and set Initial Delay Seconds to shape the first run. A file trigger stays silent. Read the last error. Watch path does not exist means the path is missing on the server’s machine. Also check that the file name matches Patterns and not Ignore Patterns, that Recursive is on for files in subfolders, and that the event type is in Events. Rapid repeat changes to one file count once within the debounce window. A webhook call returns an error. 404 means the token is wrong, or triggers are disabled on that server. 401 means the secret is missing or wrong. 410 means the trigger is not armed. 413 and 429 are the limits above. A webhook call returns "duplicate": true. The request matched an earlier one, so no new run started. Change the body or the x-webhook-id. The trigger shows Stopped. Read Last error, fix the cause in the workflow, save, and turn the trigger back on. The bolt is red but the trigger is armed. The latest run failed. The trigger keeps running and stops after 5 failures in a row. I clicked Fire now and nothing happened. A run started, but the graph may have failed. Check the run history and the Last error line.--- # Troubleshooting Guide Source: https://docs.nodetool.ai/troubleshooting.html Find your problem by the message on screen or by what is going wrong. For installation problems (GPU drivers, CUDA, platform issues), see [Installation Troubleshooting]({{ ‘/installation’ relative_url }}#if-installing-goes-wrong). Jump to: Quick checks Errors by message Runs and nodes Models and providers Search and documents Deployment Performance Debugging techniques Getting help Quick checks Start here when a run fails and you do not know why. The failing node. A failed node shows a red error box with the message, a View logs button, and a Report button. An animated ring around a node means it is still running. Provider keys. Open Settings → Models & Providers and confirm the key for the provider your node uses. A provider without a key is marked “API key required” in the model menu. The Logs panel. Press l in the editor to open it. It lists log lines for the open workflow, newest first, and you can filter by Info, Warn, and Error. The Trace panel (Ctrl/⌘ + Shift + T) shows per-node timing. The model. For a local model, open Model Manager (logo menu, or Tools → Model Manager in the desktop app) and confirm the model is installed. The local server. For Ollama or another local engine, confirm it is running. NodeTool reports “Could not reach …” when it is not. Preview nodes. Add a Preview node after each major step to see where the data stops. Connections and file paths. Check that required inputs are connected, handle colors match, and referenced files exist and are readable. Tip: A failed node’s Report button collects the error, node settings, and logs for a bug report. See Report a Bug from inside NodeTool. Errors by message Search this page (Ctrl/Cmd+F) for the text you see. Text in <angle brackets> varies with your workflow. What you see What it means Fix   Select a model An Agent or other LLM node has no model chosen. Pick a model in the Inspector, or set a default in Settings → Default Models. See [Workflow Debugging]({{ ‘/workflow-debugging’ relative_url }}#understanding-error-messages). <PROVIDER>_API_KEY is not configured (for example COHERE_API_KEY is not configured) The node’s provider has no key. The name in the message is the key to set. Add the key in Settings → Models & Providers. See [Models & Providers]({{ ‘/models-and-providers’ relative_url }}). OPENAI_API_KEY is required The OpenAI provider has no key. Add it in Settings → Models & Providers. See [Configuration]({{ ‘/configuration’ relative_url }}). Missing required secret: <key> A node or Code body reads a secret that is not stored. Add the secret in Settings. See [Configuration]({{ ‘/configuration’ relative_url }}). Authentication failed: Invalid API key or token The provider returned 401 and rejected the key. Replace the key in Settings → Models & Providers.   Authentication failed: <provider> rejected the configured credentials. Check the API key in Settings → Models & Providers. The stored credential was refused. For sign-in providers the message says to sign in again. Update the key or sign in again in Settings → Models & Providers.   Rate limited: Too many requests or insufficient provider quota. Check your provider plan and try again later. The provider returned 429. You sent too many requests or used up your quota. Wait and retry, or check your plan limits with the provider.   Account billing or credit limit reached for <provider>. Check your <provider> plan or billing. The provider returned 402. Your account is out of credit. Top up the account with the provider. See [Costs & Credits]({{ ‘/costs-and-credits’ relative_url }}). Account quota exhausted for <provider>/<model>. Check your <provider> quota or billing and try again later. A Gemini quota error. The account has no quota left for this model. Check quota and billing with the provider, or choose another model.   Model <model> was not found or is unavailable on <provider>. Choose another model or check account access. The provider returned 404 for the model id, or your account cannot use it. Pick another model, or check that your account has access. See [Models & Providers]({{ ‘/models-and-providers’ relative_url }}). Property "<path>" selects model "<model>", which provider "<provider>" does not offer. Workflow validation found a model id the provider does not list. Choose a model from the node’s model menu.   Property "<path>" selects provider "<provider>", which is not registered. Workflow validation found a provider that is not available on this install. Choose a registered provider from the node’s model menu.   Could not reach <provider>/<model>. Check your internet connection, proxy settings, and — for local providers — that the server is running. A network failure before the provider answered. Check your connection and proxy, and start the local server if the provider is local.   Could not load Ollama models The model menu cannot reach Ollama. Ollama is a separate program that NodeTool does not start. Install it from ollama.com and start it, or fix the OLLAMA_API_URL setting if it runs elsewhere. See [Models & Providers]({{ ‘/models-and-providers’ relative_url }}). The <provider>/<model> request is too large for the model context window. Shorten the conversation or remove some attachments and try again. The prompt is longer than the model’s context window. Shorten the input, remove attachments, or use a model with a larger window.   Missing required workflow input "<name>" A workflow input has no value. Give the input node a value, or pass the input when you run the workflow.   Cycle detected in graph; a cycle may only close on the "next" or "condition" input of a Loop node. The graph loops back on itself. Remove the connection that closes the cycle, or wire it into a [Loop node]({{ ‘/nodes/nodetool/control/loop’ relative_url }})’s next or condition input. Cannot connect these types The editor refused a connection between incompatible handle types. Add a conversion node between the two, such as nodetool.code.Code. See Type mismatch.   Cannot create a cyclic connection The new connection would close a cycle. Connect to a different input, or restructure the graph.   Python node "<type>" cannot execute: Python worker is not connected. The node needs the Python bridge and the worker is not running. Check that Python is installed and the worker started. See [Installation]({{ ‘/installation’ relative_url }}). Collection '<name>' not found A search or index node names a collection that does not exist. Run the indexing workflow first, and check the name matches. See [Collections]({{ ‘/collections’ relative_url }}). Model is booting, taking minutes. A hosted model is starting up. This is a status, not a failure. Wait for the node to finish starting.   WebSocket connection lost. The download may still be running on the server. Click Reconnect to restore progress updates. The Model Manager lost its live connection during a download. Click Reconnect. If progress does not return, cancel and restart the download.   Failed to save workflow: Server unreachable The editor cannot reach the NodeTool server. Check that the server is running, then save again.   Runs and nodes My workflow is stuck or very slow Signs: the progress bar does not move, there is no output after several minutes, CPU or GPU use is low, or the app freezes. First look at the node. An animated ring means it is running, an error box means it failed, and “Model is booting, taking minutes.” means a hosted model is starting up. Then check CPU, RAM, GPU, and disk use in Task Manager (Windows) or Activity Monitor (macOS). Add Preview nodes after each major step to find where execution stalls. Model not downloaded. Open Model Manager and install the model. Install models before running workflows that use them. Large file processing. Process in batches or split the data, for example split large PDFs before indexing. Not enough memory. The workflow crashes or hangs at specific nodes. Close other applications, reduce batch size, use a smaller or quantized model (Q4 instead of Q8), or increase swap or the page file. Network timeout (cloud models). Check your connection and API keys, or switch to a local model to work offline. An endless loop. The workflow never completes and keeps processing. NodeTool blocks circular connections, but conditional logic can still loop forever. Review the workflow logic. A connection won’t snap, or an input is empty Signs: the connection snaps back and no edge is created, “Cannot connect these types” appears, “Cannot create a cyclic connection” appears for a cycle, or the workflow will not run because a required input is empty. Handle colors show each slot’s type, so match them. Click a node to read its input and output types, and use Preview nodes to check what a node actually produces. Wrong data type. For example, List[String] connected to a node that expects String. Add a conversion node, such as a nodetool.code.Code node that returns one item from the list. Null or empty output. The previous node returned null and the next expects data. Add validation or a default value. Media type mismatch. A node outputs an image and the next expects a different media type. Add a conversion node. Any type to any other type: use nodetool.code.Code and return the shape the next node wants. Preview shows empty or null output Signs: the Preview panel is blank, shows null or undefined, and no error appears. Work backwards from the Preview to find where data is lost. Place a Preview after each node, and check warnings even when a node looks green. A node failed silently. Check the Logs panel for hidden errors. For example, an API call returned 404 without showing an error. A filter removed everything. A FilterCode or FilterEqual node removed all items. Review its conditions. A timing issue. NodeTool runs upstream nodes first, but custom nodes can break this. Confirm upstream nodes finish before downstream ones start. A path or file does not exist. Verify the path, prefer absolute paths, and check permissions, spelling, and case (Linux and macOS are case-sensitive). Models and providers The LLM gives poor results Signs: irrelevant answers, hallucinations, repeated text, or incomplete responses. Check the prompt in Template or Agent nodes, the model selection, and the temperature. Add few-shot examples, and use Preview nodes to inspect intermediate steps. Vague prompt. Be specific. “Summarize this” is weaker than “Summarize this document in 3 bullet points, focusing on action items”. Wrong model for the task. Creative writing: larger models (13B and up). Factual Q&A: chat or instruct fine-tunes. Code: code-specialized models. Fast iteration: smaller models (7B). Temperature. 0.0-0.3: deterministic and factual (Q&A, extraction). 0.5-0.7: balanced (general chat). 0.8-1.2: creative (stories, brainstorming). Context window exceeded. Responses are cut off or earlier context is forgotten. Shorten the input, retrieve relevant snippets with RAG instead of passing full documents, or use a model with a larger window. Missing retrieval context (RAG). Confirm the vector database is populated, increase the number of retrieved documents, check hybrid search parameters, and confirm embeddings are generated. See the next section. A model download fails or stalls Signs: the progress bar stops, “Connection timed out” or “HTTP error” appears, the model looks partially downloaded, or “Model not found” appears after downloading. Check free disk space first (models run 4-20 GB each, df -h on macOS and Linux). Then check that you can reach huggingface.co, and look at the Models panel for download status. Not enough disk space. Models are stored in ~/.cache/huggingface/ by default, and you need at least 2x the model size free. Set HF_HOME to a larger drive to move the cache. Network interruption. Retry from Model Manager. Behind a proxy, set HTTP_PROXY and HTTPS_PROXY. Models of 12 GB and up can take 30+ minutes on slow connections. Hugging Face rate limiting. Wait a few minutes and retry. For frequent downloads, add a Hugging Face token in Settings → Models & Providers. Create one at huggingface.co/settings/tokens. Corrupted download. The model appears downloaded but fails to load. Delete it in the Models panel and download again, or remove its folder from ~/.cache/huggingface/hub/. Search and documents RAG returns irrelevant documents Signs: retrieved documents do not match the query, relevance scores are low, or answers miss the question. Check that documents are indexed in SQLite-vec (or your configured backend), use Preview nodes to see what is being embedded, run the search node alone with a known query, and review chunk size. Documents not indexed. Run an indexing workflow first. The collection count should match the number of chunks. Poor chunking. Retrieved text is too short, too long, or cut mid-sentence. Split on a semantic boundary with nodetool.code.Code, use chunks of about 500-1000 tokens, and add 50-100 tokens of overlap. Embedding model mismatch. Use the same embedding model to index and to query. If you indexed with text-embedding-ada-002, query with it too. Query too vague. Make the query more specific, or have an LLM expand or rephrase it before the search. top_k too low. Try 5-10 first. More documents improve recall but slow inference. Deployment A deployment fails or the service won’t start Signs: nodetool deploy apply fails, the container exits immediately, the health check fails, or the service returns 503. Start with nodetool deploy logs <name> and nodetool deploy status <name>. Review deployment.yaml for typos, confirm the workflow runs in the desktop app first, and confirm the target has enough CPU, RAM, and GPU and that API keys and tokens are set. Invalid deployment.yaml. Run nodetool deploy plan <name> or nodetool deploy apply <name> --dry-run to see what apply would do. Required fields are host, image.name, container.name, and container.port. The only deployment type is docker. Missing environment variables. Add them under container.environment: container: name: nodetool port: 7777 environment: HF_HOME: "/workspace/hf-cache" NodeTool writes a generated SECRETS_MASTER_KEY there, and a server_auth_token, when it loads a deployment.yaml that lacks them. Volume mount errors (self-hosted). Container logs show “Permission denied” or “No such file”. Verify host paths exist, check that the user running Docker has access, and use absolute paths. Docker not running (self-hosted). Start the daemon with sudo systemctl start docker. Port conflict. “Port already in use”. Change container.port in deployment.yaml, or find the conflicting container with docker ps and stop it with docker stop <id>. Image not found. Check image.name, image.tag (default latest), and image.registry (default docker.io) in deployment.yaml, and that the image exists in that registry. Verify with docker images | grep nodetool on the host. SSH access fails (remote hosts). deploy apply cannot reach the host. Check the ssh block (user, key_path, port) in deployment.yaml and test ssh <user>@<host> by hand. Performance Speeding up a slow workflow Diagnosis: Profile with Preview nodes to identify bottlenecks Optimization strategies: Use local models for fast tasks Example: Local Whisper for transcription instead of API Benefit: No network latency Batch processing Process multiple items together Use Chunk to group stream items into batches and Collect to gather a stream into a list Parallel execution Split workflow into independent branches NodeTool automatically parallelizes non-dependent nodes Cache results Save intermediate outputs to files Reuse across workflow runs Use SaveText, SaveImage, etc. Right-size models Don’t use GPT-4 when GPT-3.5 suffices Use quantized models (Q4) for faster inference Balance quality vs speed Optimize chunking Smaller chunks = faster processing but may lose context Larger chunks = slower but better context Typical sweet spot: 500-1000 tokens Reducing memory use Diagnosis: Monitor system resources during workflow execution Fixes: Use smaller models – Switch to quantized versions (Q4 instead of FP16) Process in batches – Don’t load all data into memory at once Clear caches – Restart NodeTool periodically to clear accumulated memory Close Preview panels – Preview nodes keep data in memory for display Limit parallel execution – Reduce concurrent node execution Debugging Techniques Using Preview Nodes Effectively Strategy: Add Preview nodes after every major transformation Example workflow: Input → Preview(1) → Transform → Preview(2) → LLM → Preview(3) → Output Benefits: See exact data at each step Identify where data is corrupted/lost Verify transformations are correct Understand intermediate results Enable Verbose Logging Desktop app: Open Tools → Log Viewer to read the backend logs Open Tools → Performance Monitor to watch resource use To raise the log level, start the app with NODETOOL_LOG_LEVEL=debug set CLI/Server: nodetool serve only accepts --host and --port. Set the log level via the NODETOOL_LOG_LEVEL (or LOG_LEVEL) environment variable: NODETOOL_LOG_LEVEL=debug nodetool serve Isolate Problem Nodes Strategy: Test nodes individually Create new workflow with just the problem node Provide known good inputs Verify outputs If works in isolation, issue is with upstream data Use Reproducible Inputs Strategy: Save test inputs as files Create test-inputs/ folder Save sample images, text, audio Use nodetool.input.FilePathInput nodes for consistent testing Share test cases with teammates Getting Help Quick Self-Help Checklist Before reaching out, try these steps — they resolve most issues: Read the error message — it often contains the solution or a clear hint Run the Quick checks above Search this page — use Ctrl/Cmd+F to find your error message in Errors by message Try a simpler workflow — isolate the problem by testing with fewer nodes Restart NodeTool — clears cached state and frees memory **Check [Workflow Debugging]({{ ‘/workflow-debugging’ relative_url }})** — step-by-step debugging guide Where to Get Help Channel Best For   Discord Real-time help, community tips, sharing workflows   GitHub Issues Bug reports, feature requests, reproducible problems   **[Documentation]({{ ‘/’ relative_url }})** Guides, API reference, node documentation Report a Bug from inside NodeTool Every error in NodeTool carries a Report button. It opens a form that collects the technical detail for you, so you only have to describe what happened. You reach it from: A failed node — the Report button on the red error box A failed job — the Report button in the Jobs panel An error notification — the Report button in the notification list A crashed screen or panel — the “Report a bug” button on the error screen Anywhere — the command menu (Cmd/Ctrl+K) → Report a Bug The form has three steps: Describe it. What went wrong, the steps to reproduce, what you expected. Check what you send. The form lists every file it will attach — the error and stack trace, the failing node’s settings, the workflow graph, the run logs, the browser console. Each row has a View button that shows the exact content, and a checkbox to leave it out. You can add screenshots here too; a picture of the broken screen is the most useful thing you can attach. Send or save it. What happens depends on where you run NodeTool: Web app on a hosted server, such as nodetool.ai: Send report stores your description and the checked files on that server, where its operators read them. You need no GitHub account. Desktop app or local server: Save report bundle writes one zip to your downloads. Open GitHub issue then opens a pre-filled issue. Drag the zip into it and submit. NodeTool uploads nothing, and the zip never leaves your machine if you close the dialog. Only the rows you leave checked are sent or saved. API keys, tokens and embedded media are stripped before anything is written to the file. How to Ask Effectively The Report a Bug form gathers your version, OS, error text, node settings and workflow for you. When you ask elsewhere — Discord, an issue you write by hand — include these: NodeTool version — from Settings → About Operating system — macOS/Windows/Linux + version What you’re trying to do — describe the goal, not just the error Full error message — copy the complete text, not just “it doesn’t work” Steps to reproduce — exact sequence that causes the issue Screenshots — especially of error messages or unexpected behavior Workflow file — export the workflow JSON if possible Good example: I’m running NodeTool 1.5.0 on macOS 14.1. When I run the Chat with Docs workflow (attached JSON), I get “Collection not found: docs”. I’ve already run the Index PDFs workflow and can see the collection in the SQLite-vec store. Screenshots attached. Poor example: Chat with docs doesn’t work, help! Related pages [Workflow Debugging]({{ ‘/workflow-debugging’ relative_url }}): step-by-step debugging with Preview nodes and logs [Editor Panels]({{ ‘/editor-panels’ relative_url }}): the Logs, Queue, and Trace panels [Installation Troubleshooting]({{ ‘/installation’ relative_url }}#if-installing-goes-wrong): hardware, CUDA, and installation issues [Models & Providers]({{ ‘/models-and-providers’ relative_url }}): provider keys and model choice [Model Manager]({{ ‘/models-manager’ relative_url }}): install and remove local models [Configuration]({{ ‘/configuration’ relative_url }}): settings, secrets, and environment variables [Deployment Guide]({{ ‘/deployment’ relative_url }}) and [Self-Hosted Deployment]({{ ‘/self-hosted-deployment’ relative_url }}): deployment-specific issues--- # Tutorials Source: https://docs.nodetool.ai/tutorials.html Each video is under a minute and replays a real session in the NodeTool editor. You see what to ask for, what the assistant does, and where the result lands. The same videos play in the app: open the logo menu and choose Tutorials. The first group shows how to make something by asking. The second shows how to steer the assistant when the first answer is not right. The last group shows how to repeat a result with a workflow, one node at a time. New to NodeTool? Start with the Quick Start. The Glossary explains the terms used below. Make it by asking Edit a sketch by asking Say what you want changed. The Sketch Assistant uses the real layer tools: it adds a layer, sets the blend mode, and sets the opacity while you watch the layer panel. Try it: open a sketch and ask for a soft vignette. The new layer appears in the layer panel, and you can change it by hand afterwards. Read more: Sketch Editor, Chat & Agents. Write and voice a script Go from a blank page to voiced audio in one request. The Script Assistant casts the speakers first, writes their lines, then records a take for each line. Try it: ask for a short two-speaker script, then have the assistant voice it. Each line gets a take you can replace or record again. Read more: Script Editor, Creative Agent. Board a shot list Describe the video. The Storyboard Assistant writes the shots, including camera direction, before it generates any image. Approve the board, and the stills render shot by shot. Try it: describe a short piece, read the shot list, and change a shot while it is still free to change. Read more: Storyboard Editor, Creative Agent. Build a mini app Describe an app in one sentence: an input, a button, and an answer. The App Assistant adds the operation, declares a setting that persists between sessions, and places each widget against something the app declares. Try it: open a mini app from the Examples page, fill in its input, and run it. Read more: Mini Apps, App Builder, Building Mini Apps. Write a JS script Say what goes in and what should come out. The assistant declares the ports first, writes the body, and saves a test case that runs in the sandbox. The script runs sandboxed with no file system access, and the saved case catches the next edit that breaks it. Read more: JS Scripts, JavaScript Sandbox. Ask the chat agent Ask Chat about a recent discovery. The agent runs a web search where you can see it, then streams its answer. Try it: start a new chat and ask about something that happened this week. The tool call appears before the answer. Read more: Chat, Chat & Agents. Cut a scene together The video editor trims the opening clip, moves the next shot to meet its edge, and adds a caption on its own track. Then it plays the finished cut. Try it: place two clips on a timeline, trim the cut between them, add a caption, and preview the result. Read more: Video Editor, AI Timeline Editing. Steer the assistant Correct it without starting over A warm wash comes back too strong. Ask for less in the next message, and the assistant edits the layer that is already there. The layer count stays the same and nothing is generated again. The chat shows which tool call the correction ran, and the layer panel lets you finish the edit yourself. Read more: Sketch Editor, Chat & Agents. It asks before it spends A brief that leaves out the format gets a question back instead of a guess. The board stays empty and nothing renders until you answer. Your answer sets the format and the number of shots. Read more: Storyboard Editor, Creative Agent. A test catches it Name the case you do not trust. The assistant saves it as a test and runs it. The test fails and shows why. The assistant repairs the body and runs the same cases again, and they pass. The case stays saved, so the next edit that breaks it fails there first. Read more: JS Scripts, JavaScript Sandbox. Repeat it with a workflow A workflow saves the steps behind a result so you can run them again with new inputs. These videos build small workflows on the canvas. To build one yourself from an empty canvas, follow Your First Workflow. Build your first workflow A prompt goes into an LLM node that expands it, and the expanded prompt goes into an image model. Watch the nodes run and the image appear on the canvas. You see how nodes pass data through their handles, how a running node shows progress, and where the generated image appears. Read more: Your First Workflow, Workflow Editor. Connect and run Follow a text value through three connected nodes: a text input, a Code node that uppercases it, and a Preview that shows HELLO NODETOOL. You see how outputs connect to inputs, how nodes report completion, and where a result appears. Read more: Workflow Editor, Key Concepts. Generate a list One topic goes into a List Generator node, and the items stream out one by one. The list can then feed every node after it, which is how batch generation starts. Read more: Workflow Editor, Cookbook. Ask the AI Type a question, send it to an Agent node, and watch the answer stream in before it lands in a Preview. Read more: Workflow Editor, Models & Providers. Combine two inputs Two text inputs flow into one Prompt node, which fills a template and produces one sentence. You see how several inputs connect to one node and how {% raw %}{{ placeholders }}{% endraw %} build a prompt from reusable parts. Read more: Workflow Editor, Key Concepts. Summarize a document A long passage goes into a Summarizer node, which writes the key points as it goes. Read more: Summarize RSS, Meeting Transcript Summarizer. Describe an image A picture goes into an Image Input, which connects to an Agent node. The model looks at the picture and describes it in words you can use in the next node. Read more: Image to Audio Story, Models & Providers. Ready to make something? Read the Quick Start, then browse the [Examples]({{ ‘/workflows/’ | relative_url }}).--- # UGC Strategy Source: https://docs.nodetool.ai/ugc-strategy Research basis: external reporting on closed creative platforms, plus first-party material from PixVerse, OpenArt, Freepik/Magnific, and Krea where linked below. The plan also audits NodeTool’s own distribution surfaces — the example workflow gallery, the example mini apps and their generated /apps/* landing pages, the demo/ Remotion cast harness, and SEO_STRATEGY.md. Written 2026-08-18. Numbers about competitors decay fast; re-verify before quoting them externally. The one-line thesis: A closed creative platform replaced much of its performance marketing with paid creators posting template-made videos. NodeTool cannot copy the mechanism (we have no credit currency and no engagement budget), but it can copy the structure: a high-cadence template engine, a distinctive shareable artifact, and a creator program that pays in things an open-source BYOK tool actually has. Constraint: zero cash budget. Nothing below assumes ad spend, paid creators, or cash bounties. Every incentive in this plan is one we can grant at no marginal cost: attribution, featuring, early access, and shipping a creator’s work inside the product. The spend is team time — mostly the weekly template and its demo clip. 1. What the closed-platform model did External reporting and first-party material describe these mechanics: Presets, not prompts. Cinematic templates (dolly-ins, crash-zooms, FPV drone shots) made “click-to-video” the product. New presets shipped frequently, and underperformers were cycled out using engagement data. The preset is the unit of virality: a viewer sees an effect, wants that exact effect, and the preset name is the search query that converts. Creators as the paid channel. The platform paid creators according to real-time engagement. Mid-tier and micro creators got polished templates to showcase, while the company gained distribution outside paid ads. A partner tier above the mercenary tier. The creator partnership program pays in plan + monthly credits, early model access, and featuring — audience size explicitly does not matter, only that AI content is central to the creator’s work. Persistent characters. Soul ID gave creators consistent AI personas they could run as accounts — turning one-off outputs into ongoing series, which is what platforms reward. Move upmarket on the same asset. The company repackaged the same preset engine for social media marketers, who became its durable revenue base. Where it broke. Paying on raw engagement with no content review bought shock-value content. Forbes documented payment offered for deliberately offensive videos, plus creator payment disputes and an X account suspension for inauthentic behavior. The lesson is not “don’t pay creators”; it is that the approval gate must score content quality, not engagement alone, and that paying creators late destroys the channel. 2. The rest of the field PixVerse runs the classic affiliate motion: 30% recurring for 12 months, 90-day cookie, 2-tier payouts, and — notably — distributes playbooks from top affiliates on which hooks and formats convert. The playbook is the enablement. OpenArt splits the two tiers the way the closed platform does: a Creative Partner Program paying in tools, early access, and collaboration, plus a separate affiliate program for commission-motivated referrers. Freepik/Magnific competes on model aggregation (Kling, Veo, Runway, Seedance, PixVerse, Wan, LTX in one subscription) — the closest positioning overlap with NodeTool, but credit-metered. Krea’s real-time brush-driven canvas is inherently screen-recordable: the product demo is the content. No preset engine needed when watching the tool is the hook. Pattern across all of them: the shareable unit is small and named (a preset, an effect, a real-time session), creators get enablement (templates + format playbooks) rather than just a link, and the partner tier pays in access and featuring, not only cash. 3. What transfers to NodeTool — and what does not Closed-platform mechanism NodeTool equivalent Status Preset (click-to-video) Example workflow / example mini app, installable in one click Exists — 61 example workflows, curated app bundles, /apps/* landing pages Preset engine (10/day, cull losers) Trend-reactive template cadence with kill criteria Missing — examples ship with releases, not with trends Paid-per-engagement creator pool No equivalent: no credit currency, no engagement budget Do not copy (it is also the part that blew up) Credits as creator compensation Featuring, attribution, early access, roadmap voice — no cash Adapt Affiliate on subscriptions Only viable where a paid surface exists (cloud); dormant until then Defer Soul ID persistent personas Workflows that hold a character/identity constant across outputs Template category, not a feature build The output video as the artifact The output plus the canvas — the node graph is visually distinctive and no competitor can show one Our edge Two structural differences do the work: BYOK cuts both ways. We cannot hand out credits, but every credit-based tool meters credits — and creator comment sections fill with “how many credits does this cost?” NodeTool’s answer (“your keys, provider prices, no markup”) is a hook in itself, and it means a creator’s audience can reproduce the result without a paywall between them and the first output. The canvas is the format. A closed-platform video looks like every other AI video. A screen recording that ends by zooming out from the output to the node graph that made it is instantly attributable to NodeTool — the watermark is the product. The demo/ Remotion harness already replays recorded graph-UI casts into polished video; pointing it at user-submitted casts turns “share your workflow” into “we render your workflow into a post-ready clip.” 4. The strategy 4.1 Template engine (the preset analog) Ship one trend-reactive template per week as a floor: an example workflow + mini app bundle built around whatever effect is currently moving on TikTok/Reels/Shorts, named after the effect the way closed platforms name presets (“the search query is the name”). Each template ships with: the installable bundle, a landing page, a 15–30s canvas-reveal clip rendered through the demo/ harness, and a caption block creators can paste. Kill criteria, stated up front: a template that produces no measurable installs or social pickup in 30 days is retired from the featured row. A daily cull is the discipline to copy even at weekly cadence. 4.2 The signature format: output → canvas reveal Standardize one recognizable clip shape and use it everywhere (our channels and creator enablement kits): hook with the finished output for 3–5 seconds, cut or zoom out to the canvas, scrub one or two parameter changes live, end on the template name + “free, open source, your own keys.” This combines Krea’s insight that the tool on screen is the content with the closed-platform insight that the named effect is the search query. 4.3 Creator program, adapted Two tiers, mirroring the OpenArt/closed-platform split, paid entirely in non-cash currency: Partners (apply, curated, size-blind): early access to releases, featured placement on the site and example gallery with attribution and a link, a direct channel to the team, and their templates shipped as named examples (“the X workflow by creator”). For an audience of makers, having your workflow ship inside the product — with a permanent link on its /apps/* page — is compensation a closed platform cannot offer at any price. Template wanted-list (open, contribution-based): a public list of templates we want (trend effects, model showcases, persistent-character series), run like open-source issues. An accepted submission — judged on a published quality bar + nodetool validate passing + a rendered demo clip, never on engagement metrics — ships as a named example with attribution and gets featured across our channels. This is the OSS contribution loop applied to templates: the same motivation that produces pull requests produces presets, and it costs review time, not money. The quality-gated acceptance rule directly corrects the closed-platform failure mode: engagement-metered rewards selected for shock content. We reward acceptance, not reach. 4.4 Persistent characters as a template category Soul ID’s lesson: series beat one-offs. Ship a “consistent character” template family (reference image + identity-preserving workflow across image and video nodes) and pitch it to creators as “run an AI account from a canvas you own” — with the BYOK line as the differentiator against subscription-locked persona tools. 4.5 What we explicitly do not do No paid creators, no cash bounties, no ad spend. There is no budget, and the closed-platform trajectory shows the paid version is also the dangerous one: engagement-metered payouts selected for shock content, and late payments burned the channel. No affiliate program yet. Recurring-commission affiliates need a metered subscription to commission; revisit only when a paid cloud surface has stable pricing. No inauthentic amplification. One suspended X account costs more reach than a botnet buys. 5. Measurement Per template: installs (example install endpoint already exists — POST /api/applications/examples/:slug/install), landing-page sessions (/apps/* is already in Search Console per SEO_STRATEGY §0.10), and tracked social pickups. Per program: accepted wanted-list submissions, partner-attributed templates shipped, and share of new-user sessions that start from a template versus a blank canvas. North star: weekly count of publicly posted “made with NodeTool” clips we did not make ourselves. 6. First 90 days Weeks 1–2: pick the first four trend templates; wire the demo-cast → clip pipeline into a repeatable checklist; write the creator enablement kit (format spec from §4.2, caption block, asset pack). Weeks 3–6: ship weekly templates; open partner applications with the size-blind criteria; post the first wanted-list (5 templates, published quality bar, attribution promise). Weeks 7–12: first partner-authored template ships as a named example; publish the PixVerse-style “what converted” playbook to partners from our own template metrics; review kill list; decide whether cadence moves from weekly toward 2–3/week.--- # ui_primitives overhaul — implementation brief Source: https://docs.nodetool.ai/ui-primitives-overhaul.html ui_primitives overhaul — implementation brief Audit finding: web/src/components/ui_primitives/ is a naming layer over MUI, not a design system. The import policy is enforced (never import raw MUI) but the visual decisions were never made, so screens built 100% from primitives still look amateur by default. This brief lists the verified defects with file:line evidence, then the fixes in priority order. Ground rules for the implementing agent: All work in web/. After changes: cd web && npm run typecheck && npm run lint && npm test. Keep the 4px grid (spacing.ts), design tokens (docs/DESIGN.md), and the primitives-first policy (web/src/components/ui_primitives/STRATEGY.md). Fix defaults in the primitives, not by adding props at call sites. The measure of success: a bare <Panel><FlexColumn><TextInput/><SelectField/></FlexColumn></Panel> with no styling props should look like a professional form. Reference implementation of the target form look: the storyboard header in web/src/components/storyboard/StoryboardBoard.tsx (local Field wrapper, labels above controls, uniform 36px control heights, bordered panel). That file hand-fixes everything this brief systematizes; once the primitives are fixed, migrate it to the shared versions and delete the local workarounds. Defect inventory (verified) 1. No control-level tokens tokens.ts exports FONT_WEIGHT, FONT_SIZE_*, TYPOGRAPHY, MOTION, Z_INDEX, BORDER_RADIUS — but no control height, control padding, or field radius. Every primitive invents its own: Component Value Source EditorButton 24 / 28 raw px editor_ui/EditorButton.tsx:55 Editor selects 28px / 32px theme.editor.heightNode/heightInspector, themes/components/editorControls.ts:104,124 TagInput 48px ui_primitives/TagInput.tsx:51 SearchInput radius 8px ui_primitives/SearchInput.tsx:46 Editor control radius 6px themes/ThemeNodetool.tsx:76 MuiButton radius theme.rounded.buttonLarge themes/ThemeNodetool.tsx:241 Consequences: a compact EditorButton (24px) next to an editor select (28px) misaligns by 4px; next to an inspector select by 8px. Four independent radius sources. A second, parallel control scale lives on theme.editor that EditorButton does not read. Off-grid values inside the primitives whose own docs ban them (spacing.ts:23 forbids 0.25/0.75/1.25 steps): Slider.tsx:45 — 13px padding NodeSlider.tsx:56 — 3px margin Panel.tsx:39 — comfortable: 2.5 units; Panel.tsx:151 divides padding to 1.25 units (5px) 2. Unrelated defaults across primitives TextInput.tsx:52 → variant="outlined", size="medium" SelectField.tsx:98 → size="medium", variant="standard" A no-props TextInput above a no-props SelectField renders a boxed field above an underline-only field — the default composition is broken. Surfaces: Panel.tsx:88-90 → background.default, bordered=false. Dark background.default is #08090A (themes/paletteDark.ts:268) — same as the page. A default Panel is invisible. Surface.tsx:64-68 → background.paper, bordered=false, elevation=0, explicit boxShadow: "none". Different background and radius source (theme.rounded[...]) than Panel (theme.shape.borderRadius). size semantics diverge: TextInput maps compact→MUI small; SelectField’s standard variant barely responds to size; EditorButton.tsx:65 does height: size ? undefined : height — passing size silently discards the density height. Flex containers: FlexRow.tsx:55, FlexColumn.tsx:53, Stack.tsx:61 all default gap/spacing to 0, so naive composition glues controls together. 3. Four label treatments, none shared MUI floating label — TextInput, hand-correcting MUI’s transform (TextInput.tsx:96 translate(14px, 17px)), forced to 15px, opacity: 0.6. MUI InputLabel — SelectField.tsx:123-129, 15px, no opacity, no transform fix. Renders differently from TextInput’s at rest. Label.tsx:79-90 — Typography component="label", 13px, weight 500, text.secondary, baked marginBottom: 2px. FormField.tsx:85-95 — reimplements the same thing (does NOT use Label), duplicating the required-asterisk logic (FormField.tsx:98-108 vs Label.tsx:16). The theme sets a fifth treatment on MuiFormLabel (ThemeNodetool.tsx:249-259): capitalize, 13px, padding: 0 0 8px 0 — which TextInput/SelectField then override back to 15px, defeating the theme’s label token exactly where labels matter. The floating-label choice is also a bug class: the shrunk label renders above the input box, and Panel.tsx:136 is overflow: hidden, so a TextInput at the top of a Panel gets its label clipped. (This shipped: the storyboard title. Worked around in StoryboardBoard.tsx with overflow: visible.) 4. No usable form composition primitive FormField exists but per STRATEGY.md:47 is used in 1 file, with 14+ places doing label+input+helper by hand. FormField double-labels: it renders its own label while TextInput/SelectField render theirs. SelectField has hideLabel (SelectField.tsx:48); TextInput has no escape hatch. No FormRow, FormSection, FieldGroup, or grid primitive. Only flex helpers. Spacing ownership is incoherent: SelectField.tsx:117 comments “spacing is the parent’s job” while Label.tsx:88, FormField.tsx:120, AutocompleteTagInput.tsx:110, EmptyState, ProgressBar, SectionHeader, and Panel bake their own margins. Stacking three fields yields four different gaps. TextInput additionally inherits MUI’s default helper-text margin (3px 14px 0) which nothing overrides — off-grid and different from SelectField’s mt: 0.5 (SelectField.tsx:158). 5. The type scale is fake Text.tsx:14 advertises 8 sizes; ThemeNodetool.tsx:40-46 collapses them to ~4 real values: fontSizeBigger = fontSizeBig = 18px; fontSizeSmaller = fontSizeTiny = fontSizeTinyer = 11px. <Text size="tiny"> and <Text size="smaller"> are indistinguishable. 6. Zero-opinion passthroughs masquerading as primitives Box.tsx — 2-line re-export. muiReexports.ts:108-137 re-exports Tabs, ToggleButton, Accordion, LinearProgress, Modal etc. with a comment claiming “ThemeNodetool already styles them” — but the theme has no MuiTabs, MuiToggleButton, MuiAccordion, or MuiLinearProgress overrides (only MuiButton, MuiFormLabel, MuiFormControl, MuiPopover, MuiModal, MuiToolbar, MuiDialogTitle, MuiTooltip, MuiDivider — ThemeNodetool.tsx:234-324). Those re-exports render stock MUI. Contrast: MuiTooltip (ThemeNodetool.tsx:301-312) gets backdrop blur, a divider border, and a layered shadow — more surface craft than any panel, card, or input receives. Inputs have no default field background: only editor-scoped controls get Paper.overlay (editorControls.ts:23, gated on a marker class). SearchInput.tsx:48 gives itself action.hover — so search boxes read as fields and plain text inputs don’t. Fixes, in priority order Do them as separate commits/PRs in this order. 1–3 are mechanical and improve every existing screen without call-site changes. Fix 1: control tokens Add to tokens.ts: export const CONTROL = { height: { compact: 28, normal: 36, comfortable: 44 }, // px paddingX: { compact: 8, normal: 12 }, // px, on-grid radius: BORDER_RADIUS.sm, // one field radius } as const; Wire it into: EditorButton (replace raw 24/28), SelectField, TextInput, SearchInput (replace 8px radius), TagInput (replace 48px), and theme.editor.heightNode/heightInspector/controlRadius (derive from CONTROL so the two scales can’t drift). Fix the off-grid values in Slider.tsx:45, NodeSlider.tsx:56, Panel.tsx:39/151 while touching them. Acceptance: a compact EditorButton, a small SelectField, a small TextInput, and the ModelSelectButton in one FlexRow all share one height, one radius, one horizontal padding. Fix 2: one label treatment Standardize on label above the control: 13px (fontSizeSmall), weight 500, text.secondary, no text-transform, 4px gap to the control. Remove the floating-label path: TextInput no longer forwards label to MUI; it renders the standard label above (via the shared Label). Delete the transform-correction sx (TextInput.tsx:88-101). SelectField same: drop InputLabel, render the shared label above. FormField uses Label instead of its own Typography block; delete the duplicated asterisk logic. Remove textTransform: "capitalize" and the 8px label padding from MuiFormLabel in ThemeNodetool.tsx:249-259 (labels are no longer MUI-managed). This kills the clipped-label bug class; the overflow: visible workaround in StoryboardBoard.tsx styles can then go. Acceptance: TextInput, SelectField, and a labelled ModelSelectButton render pixel-identical labels; no label ever renders outside the control’s box. Fix 3: coherent defaults SelectField: default variant="outlined" (matches TextInput). Audit the ~few call sites that relied on standard. Inputs get a default field background (Paper.overlay in dark; verify light) so fields read as fields outside the editor scope, matching SearchInput. Panel: default background="paper" and bordered — a default Panel must be visible on the page background. Align its radius source with Surface (pick one: theme.rounded or shape.borderRadius). FlexRow/FlexColumn/Stack: keep gap=0 default (too many call sites assume it) — instead fix via Fix 4’s form primitives which own their gaps. EditorButton: passing size must not discard density height — make size map onto the CONTROL.height scale or remove the prop. Acceptance: the bare no-props composition in the ground rules renders as a visible bordered panel containing two identically-styled fields. Fix 4: real form primitives Rework FormField: it owns the label (children render label-less), the 4px label gap, and the helper/error line with a fixed on-grid margin. Ensure TextInput/SelectField compose without double labels (either via context flag or by convention: never pass label to a control inside FormField). Add FormSection (optional uppercase group label — see .group-label in StoryboardBoard.tsx — plus a FlexColumn gap={3} body) and FormGrid (n-column CSS grid with a stack breakpoint, like .header-grid there). Migrate StoryboardBoard.tsx to these, deleting its local Field and grid CSS. Migrate 2–3 of the 14+ hand-rolled label+input sites listed in STRATEGY.md as proof. Fix 5: honest type scale Either give the collapsed sizes real values (a modular scale: e.g. 11/12/13/15/18/22) or remove the aliased size names from Text and fix call sites. Don’t leave 8 names for 4 values. Fix 6: passthrough honesty For each raw re-export in muiReexports.ts that the theme does not style (Tabs, ToggleButton, Accordion, LinearProgress at minimum): either add the theme override or move it out of the “themed” comment block so the file stops claiming coverage it doesn’t have. Low priority; do last. Verification cd web && npm run typecheck && npm run lint && npm test Visual: storyboard header (/workspace storyboard tab), node inspector, settings dialogs — before/after screenshots for each fix. Grep gates after Fix 1: no raw minHeight: "2[48]px" in primitives; no borderRadius: "8px"/"6px" literals in ui_primitives/ or editor_ui/; no 13px/3px/5px paddings in primitives.--- # Upgrading Source: https://docs.nodetool.ai/upgrading Updating NodeTool replaces the program and leaves your data directory in place. On start, the server applies any database migrations the new version needs. Back up before you update, because a migration cannot be undone by installing the old version. Back up first Stop NodeTool, then copy the data. The desktop app and a local server keep these in the data directory. What Where Database nodetool.sqlite3, plus nodetool.sqlite3-wal and nodetool.sqlite3-shm if they exist Assets assets/ Vector store vectorstore.db App settings settings.yaml in ~/.config/nodetool/ on macOS and Linux, or in %APPDATA%\nodetool\ on Windows Vaults (desktop) vaults/<id>/ The data directory is ~/.local/share/nodetool/ on macOS and Linux (or $XDG_DATA_HOME/nodetool/) and %APPDATA%\nodetool\ on Windows. Vaults sit under ~/.local/share/nodetool/vaults/ on macOS and Linux and %LOCALAPPDATA%\nodetool\vaults\ on Windows. See Where your data lives for the full table. Stored provider keys are in the database, encrypted with a master key. The master key is in your operating system keychain unless SECRETS_MASTER_KEY is set. A database restored on another machine decrypts only where the master key has the same value, so keep a copy of SECRETS_MASTER_KEY if you set one. See Secret Storage and Master Key. Export the workflows you cannot afford to lose as a second copy. The API serves a .nodetool bundle at GET /api/workflows/{id}/export-bundle. See API Reference. Desktop app Turn on automatic updates Automatic Updates is off by default. With it off, the app does not check for updates at all. Open Settings. In General → Workspace, find Updates. Turn on Automatic Updates. Choose an Update Channel. Auto-update runs only in installed builds, not in a development checkout. The app reads releases from the nodetool-ai/nodetool GitHub repository. It checks when it starts. Update channels Channel Follows Stable Full releases Nightly Prerelease nightly builds, published daily A build whose version ends in -nightly.<date>.<n> starts on Nightly. Any other build starts on Stable. Your own choice in Settings overrides that default. The updater allows downgrades, so moving from Nightly to Stable can install an older build. What you see When a newer version exists, the app downloads it in the background. A card at the top right of the main window reads “Version X is available. Downloading in the background…” with a View Release Notes link. When the download finishes the card reads “Version X has been downloaded and will be installed on restart.” with a Restart to Update button. A system notification appears for both events. See Update Notification. If the app logs that app-update.yml is missing, you have an older installation. Reinstall from GitHub releases to enable auto-updates. Update by hand Download the current installer from nodetool.ai or from GitHub releases, and install it over the existing app. Your data directory stays. See Installation. Node packs and runtimes Node packs and runtime packages update separately, from Tools → Package Manager. See Node Packs. Command line If you installed the nodetool command with npm, install it again: npm install -g @nodetool-ai/cli Self-hosted Docker The reference docker-compose.yml pulls ghcr.io/nodetool-ai/nodetool:${NODETOOL_VERSION:-latest}. All state lives under /workspace on the nodetool-data volume, so it survives an image upgrade. Choose a tag Tag Meaning latest The newest build of main 1.2.3, 1.2, 1 A release, published when a v1.2.3 git tag is pushed main-<shortsha> A specific commit on main In production, pin a release with NODETOOL_VERSION in .env so an upgrade happens when you choose it. Set NODETOOL_IMAGE to run an image you built yourself. Back up the volume Stop the server, then archive the volume. With the compose project name nodetool, the volume is nodetool_nodetool-data: docker compose down docker run --rm -v nodetool_nodetool-data:/workspace -v "$PWD":/backup \ alpine tar czf /backup/nodetool-data.tgz -C /workspace . The archive includes /workspace/.secrets_master_key if the entrypoint generated your master key. Keep that file with the backup. If you use PostgreSQL, take the dump with your own PostgreSQL tools. Upgrade docker compose pull docker compose up -d Check that the server is healthy at http://localhost:17777/health (or your NODETOOL_PORT). See Self-Hosted Deployment. Roll back Set NODETOOL_VERSION to the previous tag and run docker compose up -d. If the newer version applied a migration, restore the volume backup first, because the older image may not understand the migrated database. This is advice from how migrations work and not a tested guarantee. Database migrations on start Database When migrations run SQLite (desktop app, local server, and the default Docker setup) Every time the server starts. It applies pending migrations before it opens the database and logs the versions it applied PostgreSQL or Supabase in Docker The image entrypoint runs db-migrate.mjs on every start when DATABASE_URL is set. Set NODETOOL_MIGRATE_ON_BOOT=0 if a release step migrates instead For PostgreSQL or Supabase you can also run migrations yourself: nodetool db status --direct-url "$DIRECT_URL" nodetool db migrate --direct-url "$DIRECT_URL" --dry-run nodetool db migrate --direct-url "$DIRECT_URL" nodetool db rollback --steps <n> rolls PostgreSQL or Supabase migrations back. See CLI Reference. For a rolling release across replicas, migrations run once before either replica changes, so they must work with the previous image. See Docker Production Deploy. Related pages Changelog for what changed in recent releases Desktop App for settings and data locations Self-Hosted Deployment for Docker setup Configuration for storage and secrets variables--- # URL egress inventory Source: https://docs.nodetool.ai/url-egress-inventory.html URL egress inventory Where NodeTool’s server can be told to open an outbound request, who tells it, and what decides whether it may. A URL that reaches this process is rarely typed by the person who owns the machine. It rides in on a media ref inside a workflow, in a provider’s result body, on a chat client’s attachment, or straight out of a model that read someone else’s web page a moment ago. A bare fetch(uri) on any of those is the server making a request on a stranger’s behalf from inside the perimeter, which is how http://169.254.169.254/latest/meta-data/ gets read. The machine-readable list is packages/runtime/tests/url-egress-inventory.ts; the audit that keeps it honest is url-egress-audit.test.ts beside it. Run both with npm run test --workspace=packages/runtime -- url-egress. The policy Default: https, to a host that is not a loopback, link-local, RFC1918, CGNAT or benchmarking address — and every redirect hop re-checked. safeFetch in packages/runtime/src/providers/safe-url.ts is that policy as a function, and it is the only thing that counts as protection. isSafePublicHttpsUrl and friends are predicates: a predicate can refuse an initial URL, but a public host that answers 302 Location: http://127.0.0.1:6379/ walks past one. Media refs go through fetchExternalMedia (packages/runtime/src/external-media-fetch.ts), which is safeFetch plus one documented opt-out (below). Audio loading in packages/audio-nodes/src/lib/audio-wav.ts delegates to loadMediaRefBytes in packages/runtime/src/media-ref-bytes.ts, whose HTTP fallback calls fetchExternalMedia and checks every redirect hop. The sandbox fetch bridge runs its own policy on purpose: plain http is allowed, and a host — never guest code — may waive the address check for a node that replaces a trusted lib.http node. It reads the same address table. One address table isBlockedIpLiteral in providers/safe-url.ts decides which literals are internal. It covers IPv4 (loopback, 0.0.0.0/8, RFC1918, 169.254/16, CGNAT, 198.18/15), the alternate spellings the WHATWG parser normalizes for us (decimal, hex, octal, short form), IPv6 loopback/unspecified/ULA/link-local, and the IPv6 forms that carry an IPv4 inside them — IPv4-mapped, IPv4-compatible, 6to4, NAT64. Four tables existed before: this one, the sandbox’s in packages/agents/src/network-guard.ts (which now imports it), a private copy inside unified-websocket-runner.ts (deleted — it allowed http and screened only the first URL), and isSafeHttpUrl in together-nodes / atlascloud-nodes, which survive as local predicates in front of a protected fetch. The audit fails if a private copy comes back. DNS rebinding Decision: not defended in-process; the deployment’s egress control is the boundary. A name that passes the address check can resolve to a private address a millisecond later, and this process does not choose the address fetch connects to — pinning one means resolving ourselves and dialing an IP with a Host header, which breaks TLS verification and virtual hosting for every provider CDN we download from. One exception, and it earns the round trip: when the URL is handed to a separate process that opens its own sockets, assertResolvedHostAllowed (packages/agents/src/network-guard.ts) resolves the name and refuses a blocked answer first. yt_dlp is the surface that uses it. Every inventory entry records which of the two it is, in dnsRebinding. An entry that claims resolve-and-check must call the resolver. The self-hosted opt-out NODETOOL_ALLOW_PRIVATE_MEDIA_FETCH=1 turns the guard off for media-ref fetches — and for nothing else. It exists for an install that genuinely serves media off its own LAN (http://nas.local/clip.mp4). It is read per call, so it takes effect without a restart, and it is off by default. See configuration.md. Guarded surfaces Everything here fetches a URL somebody else chose, through the protected fetch. Surface File URL comes from media ref → bytes (Python bridge + TS nodes) packages/runtime/src/media-ref-bytes.ts workflow AudioRef → shared media resolver packages/audio-nodes/src/lib/audio-wav.ts workflow Model3DRef / ImageRef → bytes packages/video-nodes/src/nodes/model3d/utils.ts workflow Model bytes → bytes (shared resolution) packages/nodes-utils/src/model-bytes.ts workflow chat source images packages/websocket/src/websocket-client-session.ts chat client workflow output auto-save packages/websocket/src/session/asset-autosave.ts workflow asset ref → bytes on export packages/websocket/src/lib/asset-export.ts chat client save_asset / view_image packages/agents/src/capabilities/assets.ts model yt_dlp download packages/agents/src/capabilities/media.ts model Apify run artifacts → assets packages/agents/src/apify/assets.ts provider response isolate_subject mask video → asset packages/agents/src/capabilities/timeline-isolate-subject.ts provider response Google Fonts family resolution (timeline) packages/timeline/src/fonts/google-fonts-fetch.ts workflow Reve reference image packages/reve-nodes/src/reve-base.ts workflow HF pipeline media input packages/huggingface-nodes/src/huggingface-base.ts workflow FAL media upload packages/fal-nodes/src/fal-base.ts workflow Replicate files upload packages/replicate-nodes/src/replicate-base.ts workflow OpenAI media input packages/llm-nodes/src/nodes/openai.ts workflow Together image result downloads packages/runtime/src/providers/together-provider.ts provider response Together asset resolution packages/together-nodes/src/together-base.ts workflow AtlasCloud media pass-through packages/atlascloud-nodes/src/atlascloud-factory.ts workflow AtlasCloud prediction downloads packages/runtime/src/providers/atlascloud-transport.ts provider response Higgsfield uploads and prediction downloads packages/runtime/src/providers/higgsfield-transport.ts provider response useapi.net Google Flow and Dreamina result downloads packages/runtime/src/providers/useapi-provider.ts provider response vgenv MiniMax H3 result downloads packages/runtime/src/providers/vgenv-provider.ts provider response KIE result downloads packages/kie-nodes/src/kie-base.ts provider response Topaz result downloads packages/topaz-nodes/src/topaz-base.ts provider response MiniMax audio / file downloads packages/minimax-nodes/src/minimax-base.ts provider response MiniMax image download packages/minimax-nodes/src/nodes/text-to-image.ts provider response Gemini node video download packages/llm-nodes/src/nodes/gemini.ts provider response provider result downloads packages/runtime/src/providers/{fal,replicate,kie,topaz,meshy,rodin,minimax,evolink,gemini,anthropic}-provider.ts provider response durable FAL output recovery packages/websocket/src/generation-recovery.ts provider response MCP OAuth Client ID Metadata Document fetch packages/websocket/src/oauth/cimd.ts model/client (an MCP client’s self-hosted client_id URL) external MCP server (HTTP transport) packages/websocket/src/external-mcp.ts operator (a user’s own MCP server URL; guarded under the cloud profile, loopback allowed on a local install) custom OpenAI-compatible provider packages/runtime/src/providers/custom-openai-provider.ts, checked on save in packages/websocket/src/custom-providers.ts operator (a user’s own base URL; guarded under the cloud profile, loopback allowed on a local install) Azure OpenAI provider packages/runtime/src/providers/azure-openai-provider.ts operator (a user’s own Azure resource endpoint; guarded under the cloud profile, private endpoints allowed on a local install) Amazon Bedrock chat media packages/runtime/src/providers/bedrock-provider.ts chat client (https URLs on image and document parts; API calls use the fixed regional AWS host) The provider row is eleven files, each downloading a URL a provider’s response named — plus one reading a URL the caller’s own message named (video-frame-fallback.ts, which fetches a video content part so ffmpeg can sample stills from it for a model that cannot read video): packages/runtime/src/providers/fal-provider.ts packages/runtime/src/providers/replicate-provider.ts packages/runtime/src/providers/kie-provider.ts packages/runtime/src/providers/topaz-provider.ts packages/runtime/src/providers/meshy-provider.ts packages/runtime/src/providers/rodin-provider.ts packages/runtime/src/providers/minimax-provider.ts packages/runtime/src/providers/evolink-provider.ts packages/runtime/src/providers/gemini-provider.ts packages/runtime/src/providers/anthropic-provider.ts packages/runtime/src/providers/video-frame-fallback.ts Screening code itself: packages/runtime/src/providers/safe-url.ts, packages/runtime/src/external-media-fetch.ts, packages/agents/src/network-guard.ts, packages/agents/src/apify/policy.ts. The guest bridge is packages/agents/src/js-sandbox.ts. Exemptions Dreamina browser requests. packages/runtime/src/providers/dreamina-provider.ts runs API and upload requests inside the user’s logged-in Dreamina browser tab. Cookies stay in that tab. Server-side result downloads use fetchExternalMedia, which checks each redirect hop and sends no Dreamina credentials. ChatGPT browser requests. packages/runtime/src/providers/chatgpt-provider.ts runs every chatgpt.com request inside the user’s logged-in ChatGPT tab, including generated-image downloads. Cookies and the session token stay in that tab. An input image given as a URL is fetched server-side with fetchExternalMedia, which checks each redirect hop and sends no credentials. Gemini web browser requests. packages/runtime/src/providers/gemini-web-provider.ts runs every gemini.google.com request inside the user’s logged-in Gemini tab. The browser itself loads generated media with the user’s Google cookies, through ExtensionTab.download. It loads only URLs that the Gemini response named: a https://lh3.googleusercontent.com/gg-dl/ image or a https://contribution.usercontent.google.com/download video. A chat input image given as a URL is fetched server-side with fetchExternalMedia, which checks each redirect hop and sends no credentials. Fixed provider hosts. The URL is a constant in this repo, at most with a path or query interpolated — api.elevenlabs.io, fal.run, generativelanguage.googleapis.com, the OAuth token endpoints, the codegen schema fetchers. Screening them would refuse nothing and would break an install whose configured endpoint is deliberately internal. They carry the provider’s own credential, so the rule that matters for them is the opposite one: a URL that is not fixed must not inherit that credential. Each is listed in the data module with its auth scope. FAL webhook verification fetches only https://rest.fal.ai/.well-known/jwks.json. It sends no credentials, aborts after five seconds, coalesces concurrent refreshes, and caps key-cache freshness at 24 hours. Deliberately private hosts. Reaching an internal address is the feature: packages/whisper-cpp/src/whisper-server-provider.ts — the operator’s WHISPER_CPP_SERVER_URL, normally localhost or the LAN. This local-only provider uses native fetch, following the llama.cpp server provider’s policy. Requests carry audio and transcription options without provider credentials. packages/integration-nodes/src/nodes/stable-diffusion-cpp.ts — the graph author selects a running sd-server, normally localhost or the LAN. Requests send no credentials and reject redirects. Job polling and cancellation stay on the configured server, ignoring response-provided URLs. The cloud profile excludes lib.stable_diffusion_cpp.GenerateImage by default. packages/runtime/src/comfy-executor.ts — a ComfyUI server, normally localhost or the LAN, so screening private addresses would refuse the ordinary case. The address is the node’s endpoint property, so the graph author picks it, and lib.comfy.RunWorkflow / lib.comfy.RunWorkflowOnWorker are allowlisted on the cloud profile. packages/integration-nodes/src/nodes/comfy-sdk.ts — the Comfy API v2 transport. @comfyorg/sdk opens the socket. For lib.comfy.RunWorkflowOnCloud it opens against the constant https://cloud.comfy.org, unless COMFY_BASE_URL in the server’s environment names another deployment, which is the documented way to point it at a local comfy-api-proxy on 127.0.0.1:8189. The same transport carries the v2 paths of the other two nodes, where the base URL is the operator’s: the endpoint property of lib.comfy.RunWorkflow when its api is v2, and the HTTP origin derived from the worker_url of lib.comfy.RunWorkflowOnWorker. Those are normally localhost, the LAN, or a rented worker, the same case as comfy-executor.ts above. The user’s COMFY_API_KEY or the worker’s bearer token rides as an Authorization header, and the SDK attaches it only to requests on the client’s own origin: a job’s urls.self / cancel / events link resolves against that origin, and a signed output URL on another origin is fetched without the key. packages/agents/src/capabilities/web.ts — BROWSER_URL, the operator’s screenshot service. Every model-named URL that file fetches goes through safeFetch; this one is the operator’s own. With no BROWSER_URL, take_screenshot renders the page in a local headless Chrome instead (packages/agents/src/capabilities/local-browser.ts), which is egress no fetch wrapper can screen — the browser resolves and follows on its own. The URL is model-named, so the capability applies the policy before launching: non-http(s) schemes are always refused, and on an auth-enforced deployment (isAuthEnforced) the URL must pass isSafePublicHttpsUrl. A local install allows loopback and private addresses, because screenshotting the app on localhost is the reason to take one there. packages/cli/src/nodetool-main.ts — runs on the operator’s machine against their own API URL, loopback by default. packages/websocket/src/mcp-server.ts — the nodetool mcp serve forwarder reaches the local server’s /mcp mount and refuses any host other than 127.0.0.1 or localhost. packages/deploy/src/admin-client.ts — the server being deployed. Browser-side. The socket belongs to the viewer’s browser, not the server: packages/game-renderer/src/standalone-player.ts (fixed relative game document and bundled ./assets/ paths), packages/game-renderer/src/standalone-player3d.ts (fixed relative game.json and manifest.json, then digest-named ./assets/ paths validated against the closed export manifest), packages/game-renderer/src/audio/player.ts and packages/game-renderer/src/browser-fonts.ts (asset URLs resolved by the web app, or bundled ./assets/ paths in a standalone export), packages/image-nodes/src/nodes/image-io.ts (behind !IS_NODE), packages/core-nodes/src/nodes/fake-media.ts, packages/browser/src/capture.ts (inside page.evaluate). The standalone 3D export applies connect-src 'self' CSP. The browser follows redirects subject to that same-origin policy and its own network restrictions. These fetches open no server-side socket. Adding a surface When a change lets a new URL reach fetch, answer these in the PR and add the row to url-egress-inventory.ts — the audit fails without it: Input source. Who picks the URL: a model, a workflow, a chat client, a provider’s response, or a constant in this repo? Anything but the last is caller data. Egress target. Does this process open the socket, or does a browser, a subprocess, or the provider? Only the first is our SSRF. Authorization. What credential rides along, and would it leak on a cross-origin redirect? safeFetch strips Authorization, Cookie, Proxy-Authorization and X-Goog-Api-Key when the origin changes. Redirects. Who checks each hop? If the answer is “the runtime follows them”, the URL had better be fixed. CSP. If the fetched bytes end up rendered in the web app, check web/src/__tests__/contentSecurityPolicy.test.ts — a new media origin needs a connect-src/img-src entry, and stored media renders through ResponsiveImage / VideoPlayer / AudioPlayback, never a raw locator. DNS rebinding. deployment-egress unless the URL leaves this process for another one, in which case resolve and check.--- # Use Cases Source: https://docs.nodetool.ai/use-cases.html NodeTool is an agent-first creative workspace. You describe the result, the agent builds it as a project, and you finish it in an editor. This page groups what people make by the kind of work. Each section names where to start: a shipped skill you type after / in chat, a guided flow on New project, or an app from the [Examples]({{ ‘/workflows/’ | relative_url }}) page. Every model runs on your own provider keys, on hosted credits, or locally. You can switch any model for another one without rebuilding the project. See [Costs & Credits]({{ ‘/costs-and-credits’ | relative_url }}). Product marketing and e-commerce Launch films, product spots, and listing images built from your real product photos and pages. You want Start with A launch film from a product page /launch-commercial with the page URL A polished product film /product-commercial, or the Storyboard flow A full image set from one packshot The SKU Factory app, or the packshot recipe Lifestyle mockups, then video The Product Launch Kit app A campaign with one cast across stills and cuts /launch-kit, or the Directed Campaign Kit app Your brand filled in from your website The Brand & Social app Add the product as an [entity]({{ ‘/entities’ | relative_url }}) once, and every shot uses its reference image. Performance ads Many ad variants, each editable, so you can test angles and refresh them. You want Start with A 15-second sound-off ad from screenshots /motion-ad, or one of the sixteen ad-format apps A phone-shot creator testimonial /ugc-video, or the UGC Product Video app Copy and hooks compared before rendering The Viral Ad Engine or Ad Maker app Your version of an ad you saw /video-clone with the reference video The same ad in 9:16, 1:1, and 16:9 The Three Ratios workflow The ad library shows each ad format with a beat sheet and a sample cut. Film, trailers, and narrative Stories boarded shot by shot, with stills approved before any video renders. You want Start with A short film, scene, or trailer The Storyboard flow, or /short-film A trailer from a story pitch The Trailer Room app A music video treatment /music-video-treatment One brief in, a cut film out, unattended The Direct a Short Film workflow Characters and locations are [entities]({{ ‘/entities’ | relative_url }}), so they look the same in every shot. The [Creative Agent]({{ ‘/creative-agent’ | relative_url }}) page covers redoing a single shot by note. Creators, podcasts, and explainers You want Start with   Show notes, a newsletter, and social posts from an episode The Podcast Production Desk app   A narrated explainer The Script flow, or /script-video   A vlog or creator video /ugc-video   Captions and titles on a finished cut The [timeline]({{ ‘/video-editor’ relative_url }}), or /caption-titles Localization You want Start with A clip dubbed into another language The Dubbing Desk app, or the dubbing recipe An explainer voiced and captioned per language The Localized Explainer workflow You review and edit the translation before it is voiced. Design and key art You want Start with   One image with variations to choose from The Image flow, which opens the pick as a [sketch]({{ ‘/sketch-editor’ relative_url }}) Concept art directions from one brief The Concept Studio app   One change to an image, such as lighting or pose The Vary Image app   Photo grading and batch retouching The Photo Studio app   A set blocked out in 3D as a shot reference The [3D editor]({{ ‘/3d-editor’ relative_url }}), or /nodetool-3d-scene Games Describe a world, a player, and a goal. The agent builds a playable 2D or 3D game in the [game editor]({{ ‘/game-editor’ | relative_url }}), where you play it, change it, and export a web player. Start with the Game flow or /native-game, or open one of the five shipped games. Team tools and automation You want Start with   A simple screen a coworker runs without seeing the graph A [mini app]({{ ‘/mini-apps’ relative_url }}), or /nodetool-app-builder Answers from your own documents [Collections]({{ ‘/collections’ relative_url }}), or /nodetool-rag-indexing Work in Drive, Gmail, Docs, Sheets, and Calendar [Google Workspace]({{ ‘/google-workspace’ relative_url }}) A run on a schedule, a file drop, or a webhook [Triggers]({{ ‘/triggers’ relative_url }}) Your agent in Telegram [Telegram Bot]({{ ‘/telegram-bot’ relative_url }}) A procedure the agent repeats the same way Your own [skill]({{ ‘/skills’ relative_url }}) Developers NodeTool runs inside other tools as well as on its own. Claude Code and other agents drive it through the [MCP server]({{ ‘/mcp-server’ | relative_url }}). Your code calls workflows over the [API]({{ ‘/api-reference’ | relative_url }}) or runs them with the [CLI]({{ ‘/cli’ | relative_url }}). New node types are written in TypeScript, as covered in the [Developer Guide]({{ ‘/developer/’ | relative_url }}). Private and local AI Run language, image, and speech models on your own machine with Ollama, llama.cpp, or MLX on Apple Silicon, so your files stay on it. Self-host the server with Docker for a team. See [Local vs Cloud]({{ ‘/models’ | relative_url }}) and [Self-Hosted]({{ ‘/self-hosted-deployment’ | relative_url }}). Showcase walkthroughs These pages take one showcase workflow apart node by node. Each one starts from a few inputs and ends with a finished result you can re-run with your own brief. Film Movie Trailer Generator Type one logline and the canvas builds a cinematic teaser: a Director node storyboards it into shots, each shot is rendered as key art, animated, and cut into a finished trailer. LoglineStoryboardShotsKey artTrailer Documentary Documentary Teaser Generator Describe the film in a sentence and the storyboard boards it shot by shot: a card per beat, a still on every card, animated clips, and a cut teaser on the timeline. PremiseStoryboardStillsClipsTimeline Marketing Product Video Generator Turn a campaign brief and a single product photo into a cinematic 16:9 product video. Your inputs feed a prompt, an agent directs the shot, and an image-to-video model animates the photo. BriefPromptAgentImage-to-Video Advertising Ad Creative Factory One product photo and one offer become a batch of ready-to-test vertical video ads. A strategist agent plans a persona × angle test matrix, and every cell becomes a spoken hook, a staged scene, an animated clip, and a voiceover. OfferMatrixHooksScenesAds Design Movie Poster Generator From a title, genre, and visual style, the canvas writes a creative strategy and renders a batch of cinematic poster concepts, one image per concept. BriefStrategyConceptsKey art Creators Podcast Repurposing Studio Drop in one episode and ship the whole content pack: titles and show notes, a newsletter edition, five social posts, and quote cards rendered as square images. Transcribed once, written four ways. EpisodeTranscriptNotesNewsletterCards Content SEO Content Engine One topic in, a keyword-targeted article batch out. A strategist agent plans the cluster, and every brief becomes a full article with title, meta description, keywords, and body, plus an editorial hero image. TopicClusterBriefsArticlesHeroes The Movie Trailer Generator, Product Video Generator, Movie Posters, Podcast Repurposing Studio, and SEO Content Engine ship as workflow templates. The Documentary Teaser Generator and Ad Creative Factory do not, so build them from the node lists on their pages. Related pages CookbookWhere to start for each kind of project, and when to turn it into a workflow. ExamplesEvery shipped example: apps, recipes, workflows, storyboards, timelines, sketches, 3D models, and games. Shipped SkillsThe procedures the agent follows for each kind of project. Quick StartInstall NodeTool and make a first project.--- # Ad Creative Factory Source: https://docs.nodetool.ai/use-cases/ad-creative-factory.html Use case · Advertising Turn one product photo and one offer into a batch of ready-to-test vertical video ads. A strategist agent plans the test, and every variant comes out staged, animated, and voiced — what an agency bills per video, this graph produces by the batch at provider cost. Since ad platforms took over targeting, creative volume is the lever that’s left: accounts win by testing many angles and refreshing them before they fatigue. That makes ad variants a recurring line item — bought monthly, per video. This workflow moves that line item onto your canvas. How it works This page describes a graph design. NodeTool does not ship it as a template, so there is nothing to open under Examples. Build it on the canvas from the nodes named below, or start from one of the shipped ad workflows listed under Next steps. Four inputs, one fan-out. The strategist plans once; everything after the list generator runs once per variant. graph LR brief["Product & offer (String)"] audience["Audience (String)"] count["Variant count (Integer)"] photo["Product photo (Image)"] strategist["Strategist (Agent)"] hooks["Hooks (ListGenerator)"] scene["Stage scene (ImageToImage)"] clip["Animate (ImageToVideo)"] vo["Voiceover (TextToSpeech)"] mix["Mix (AddAudio)"] brief --> strategist audience --> strategist count --> strategist photo --> strategist strategist --> hooks hooks --> scene photo --> scene scene --> clip hooks --> vo clip --> mix vo --> mix Feed in the campaign. The product and offer, the audience, a variant count, and one clean product photo — the same shot anchors every variant. (e.g. “Aurora Trail smart fitness watch · 20% off launch week · weekend hikers · 6 variants”) Plan the test. A strategist agent studies the photo and brief, then writes a test matrix: buyer personas × message angles (pain point, bold claim, social proof, offer urgency), and picks the strongest pairings to test. Write the hooks. A list generator turns each matrix cell into one spoken hook — 18 words or fewer, plain language, said in the first three seconds. Stage every scene. For each hook, an image-to-image model places the product from your photo into that hook’s world: native UGC framing, natural light, 9:16, upper third kept clear for caption text. Animate and voice. An image-to-video model adds a subtle handheld push-in; a text-to-speech model reads the hook. Mix and ship. The voiceover lands on the clip, and the finished ads collect in a preview grid — one ready-to-upload vertical ad per matrix cell. One photo in, a test cell out Every variant isolates one persona × angle pairing, so when one ad wins in the ad account you know why it won — and the next run can breed from it. Paste last week’s winning hooks into the audience or offer input, raise the variant count, and the strategist plans the next generation around them. Make it yours Swap any model. Language, image, video, and voice are each one node. Trade the video model up for hero variants, down for cheap broad tests. Retune the strategist. The persona × angle matrix is a prompt you can read and edit — push it toward founder-story ads, comparison ads, or a single angle across ten personas. Change the format. Aspect ratio, duration, and resolution live on the image and video nodes. Go 1:1 for feed placements or 16:9 for YouTube in one click. Localize the batch. Point the voiceover at another voice or language and re-run — the same matrix ships in every market. Models in this workflow Pick one model per role when you build it. Called with your own keys, so the bill comes from the provider. Role Node Works well with Test-matrix strategist Agent Any strong language model Hook writer List Generator Any language model Scene staging Image To Image An image-editing model that keeps the product intact (Nano Banana, FLUX) Ad animation Image To Video Veo, Kling, Seedance Voiceover Text To Speech OpenAI TTS, ElevenLabs See Models & Providers to set up keys. Next steps Shipped ad workflows (Examples in the sidebar, Workflows tab): Ad Loop from a Product Photo animates one photo into a looping hero shot with Kling 2.6 on Kie, and Per-SKU Ad Factory turns one approved storyboard into one ad per product. Product Video Generator — one cinematic hero video instead of a test batch Movie Trailer Generator — the same fan-out pattern, pointed at film All use cases--- # Documentary Teaser Generator Source: https://docs.nodetool.ai/use-cases/documentary-teaser.html Use case · Documentary Describe the film in a sentence and the storyboard boards it shot by shot — a card per beat, a still on every card, then animated clips and a cut on the timeline. The teaser below is one run: DEEP — Life Below the Light, a dive from the surface at dusk to a bioluminescent whale on the abyssal plain, six shots and 26 seconds. How it works The storyboard editor runs the whole thing. Each step leaves something you can read and change before the next one spends anything. This is an editor workflow, not a workflow template, and NodeTool does not ship the DEEP board. The shipped boards are under Examples in the sidebar, Storyboards tab. graph LR brief["Premise · style · shot count"] board["Direct → shot cards"] stills["Still per card"] clips["Animate approved cards"] timeline["Timeline → export"] brief --> board board --> stills stills --> clips clips --> timeline Write the premise. In the Screenplay section, fill in Title, Brief, and Style. Under Direction, pick the screenplay model, the aspect ratio, and the number of shots (3, 4, 5, 6, 8, 10, or 12). (e.g. “A dive from the surface to the abyssal plain · IMAX documentary look · 6 shots”) Direct the board. Press Direct and the cards fill in — one beat each, with action, framing, and movement, all under one style bible. Review the stills. Render stills opens a dialog where you pick the still model and see the estimated cost. Re-render a single card from its edit panel until it looks right; the rest of the board stays put. Animate and cut. Render clips animates every shot that has a still and no clip yet, with a video model you pick per clip type. Then Create timeline (or Rebuild linked timeline on later passes) sends the clips to the timeline in shot order, ready to trim, narrate, score, and export. Six shots, one teaser The style bible under every card: IMAX 70mm look, volumetric light shafts, a deep teal-and-black palette with cyan and magenta bioluminescent accents, subtle film grain, 16:9. Aerial wide at dusk, a lone research vessel on a glass-calm ocean The descent, headlights cutting through ink-black water First encounter, a field of jellyfish pulsing cyan Creature reveal, a colossal squid's eye in the floodlights The vent city, black smokers swarming with white shrimp Finale, a freediver under a bioluminescent whale Make it yours Nothing here is locked. Change the look, change the model, change one shot. Swap the video model. Veo, Seedance, Kling, Runway. Render that shot again; the board, the stills, and the other clips stay as they are. Restyle the series. The visual style you typed is the style bible behind every card. Change one line and the next pass boards it that way. Fix one shot, not the reel. Revise a single clip and it swaps back into its card. The other five never re-roll. Keep the subjects consistent. Save the vessel, the submersible, and the creatures as named entities. A shot picks up an entity when its text names it, or when you attach it on the shot, and the entity’s description goes into that shot’s prompt. Models in this use case Three jobs, one model each, every one a dropdown. They are called with your own keys, so the bill comes from the provider and you can switch any of them for a better model the day it ships. Job Role Pick from Director Writes the shot list, action, and camera notes Gemini, Anthropic, OpenAI Stills Renders each card’s still GPT Image, Flux, Nano Banana Motion Animates each still into a clip Veo, Seedance, Kling See Models & Providers to set up keys. Next steps Movie Trailer Generator — the same board, cut as a chase teaser All use cases--- # Movie Poster Generator Source: https://docs.nodetool.ai/use-cases/movie-poster.html Use case · Design From a title, genre, and audience, the canvas writes a creative strategy and renders a batch of cinematic poster concepts — title, tagline, billing block and all. One run gives you a spread of directions, not a single guess. How it works Three text inputs and a strategy-then-render pipeline turn a one-line brief into a batch of theatrical posters. The shipped template is named Movie Posters (Examples in the sidebar, Workflows tab). {% mermaid %} graph LR title[“Title (String)”] genre[“Genre (String)”] style[“Visual Style (String)”] strategy[“Strategy (Prompt)”] strategist[“Art director (StructuredOutputGenerator)”] plots[“Concepts (ListGenerator)”] poster[“Poster prompt”] render[“Render (TextToImage)”] title –> strategy genre –> strategy style –> strategy strategy –> strategist strategist –> plots plots –> poster title –> poster genre –> poster poster –> render {% endmermaid %} Set the brief. Three text inputs hold the film’s title, its genre, and the visual style you want. That’s the entire creative input. (e.g. “Singularity · Sci-Fi Thriller · Neo-noir, high-contrast, cinematic”) Write the strategy. A Prompt node frames the brief and a Structured Output Generator returns it as fields: positioning, audience insight, a core visual concept, and a design direction covering palette and typography. Spin up concepts. A list generator turns the strategy into a batch of distinct plot angles, so one run gives you a spread of directions. Render the key art. Each concept is templated into a full theatrical poster prompt, then an image model renders it — title, tagline, billing block and all. Concepts from one run "The end of time isn't the end." "The end of limits. The beginning of everything." "The future is not ours to control." "The end of man is only the beginning." "The future doesn't evolve. It accelerates." Make it yours Swap the image model. GPT Image-2, Flux, Nano Banana, Ideogram. Change one node and the whole batch re-renders in a new look. Shift the tone. Change the genre or visual style and the strategy, palette, and posters all follow — no manual re-briefing. Edit the poster recipe. The poster Prompt node owns the composition, typography, and billing-block layout. Tune it once, every render obeys. Batch any title. Drop in a new title and run it again. The workflow is the reusable part; the posters are just this run’s output. Models in this workflow Called with your own keys. The bill comes from the provider. The template ships with these defaults and each is one dropdown. The gallery images above were rendered with other models, so your output will differ. Model Role Provider GPT-5 mini Writes the strategy and the plot concepts OpenAI FLUX.1 Schnell Renders the poster key art (3:4, 2K) fal.ai See [Models & Providers]({{ ‘/models-and-providers’ relative_url }}) to set up keys. Next steps [Movie Posters workflow]({{ ‘/workflows/movie-posters’ relative_url }}) — the walkthrough for the Movie Posters template [Movie Trailer Generator]({{ ‘/use-cases/movie-trailer’ relative_url }}) — from one logline to a cut teaser [All use cases]({{ ‘/use-cases’ relative_url }}) </content> </invoke>--- # Movie Trailer Generator Source: https://docs.nodetool.ai/use-cases/movie-trailer.html Use case · Film Type one logline and the canvas builds a cinematic teaser — a storyboard, key art for every beat, then animated and cut into a finished trailer. No editor, no studio, one canvas you can re-run for any story. How it works A handful of nodes do the work. One line becomes a storyboard, the storyboard becomes shots, the shots become a trailer. {% mermaid %} graph LR logline[“Logline (String)”] vstyle[“Visual style (String)”] count[“Shot count (Integer)”] director[“Director → screenplay”] shots[“Screenplay Shots”] keyart[“Key art (TextToImage)”] i2v[“Animate (ImageToVideo)”] concat[“Concat → Trailer”] logline –> director vstyle –> director count –> director director –> shots shots –> keyart keyart –> i2v i2v –> concat {% endmermaid %} Start with one line. Type the logline. Two more inputs set the visual style and the shot count (1 to 20, default 6). That’s the entire brief. The shipped template is named Movie Trailer Generator (Examples in the sidebar, Workflows tab). (e.g. “A getaway driver outruns a collapsing bridge · gritty daylight · 6 shots”) Direct the storyboard. The Director node writes the screenplay: one shot per beat, each with framing, lens, angle, and movement, under a single style bible. It is the same node the Storyboard editor runs behind its Direct button. Render the key art. Screenplay Shots turns each shot into an image prompt — action, camera, style bible — and a text-to-image model renders it as a cinematic 16:9 frame. Animate and cut. An image-to-video model animates every frame, then a Concat node stitches them into one finished trailer. Six shots, one trailer Every beat is rendered as its own cinematic frame, then animated and cut together. Here are all six frames from a single run, straight off the canvas. Blown supercharger spits fire down the straight A raider hauls the war-rig in by the chain Tires tear through the canyon floor A lone rider guns it through the ruins Last repairs before the run The getaway car breaks loose across the flats Make it yours Nothing here is locked. Swap models, change the tone, or point it at a different story. Swap the video model. Veo, Seedance, Kling, Runway. Change one node and the storyboard, shots, and key art stay exactly the same. Redirect the storyboard. Raise the shot count, change the aspect ratio, or point the Director at a different model. The rest of the graph is untouched. Restyle every shot. The visual-style input becomes the screenplay’s style bible, and the style bible lands in every shot prompt. Change one line and the whole trailer shifts mood. Re-run any story. Drop in a new logline and run it again. The workflow is the reusable part, not this trailer. Models in this workflow Called with your own keys. The bill comes from the provider, and you can switch any of them for a better model the day it ships. Model Role Provider Gemini 3.1 Pro Preview Directs the storyboard Gemini GPT Image-2 Renders each shot’s key art kie Veo 3.1 Preview Animates the frames into video Gemini See [Models & Providers]({{ ‘/models-and-providers’ relative_url }}) to set up keys. Next steps [Product Video Generator]({{ ‘/use-cases/product-video’ relative_url }}) — one product photo to a cinematic clip [All use cases]({{ ‘/use-cases’ relative_url }}) </content> </invoke>--- # Podcast Repurposing Studio Source: https://docs.nodetool.ai/use-cases/podcast-repurposing-studio.html Use case · Creators Drop in one episode and ship the whole content pack: titles and show notes, a newsletter edition, five social posts, and quote cards rendered as square images. This is the work podcasters rent per-seat repurposing tools for, or skip entirely — here it’s one canvas that already knows your show’s voice. An episode’s value doesn’t stop at the RSS feed: the episode page, the email list, and the social feeds each need their own version of it, every week. That recurring rewrite is exactly what this graph automates — transcribe once, fan out to every format. How it works One transcription, four writer branches. Everything downstream of the transcript runs in parallel. {% mermaid %} graph LR audio[“Episode audio (Audio)”] show[“Show context (String)”] count[“Quote count (Integer)”] asr[“Transcribe (ASR)”] notes[“Show notes (Agent)”] news[“Newsletter (Agent)”] posts[“Social posts (ListGenerator)”] quotes[“Quotes (ListGenerator)”] cards[“Quote cards (TextToImage)”] audio –> asr asr –> notes asr –> news asr –> posts asr –> quotes show –> notes show –> news show –> posts count –> quotes quotes –> cards {% endmermaid %} Feed in the episode. The recording, a line of show context (name, host, voice, call to action), and how many quote cards you want (2 to 8, default 4). The shipped template is named Podcast Repurposing Studio (Examples in the sidebar, Workflows tab). Transcribe once. A speech-recognition model turns the episode into a transcript that every branch reads from. Write the episode page. An agent drafts three title options, a summary, chapter-style show notes, a resource list, and the closing CTA. Draft the newsletter. A second agent writes the email edition in the host’s voice — subject-line options and a body that teases instead of summarizes. Generate the posts. A list generator writes five standalone social posts in mixed formats: a bold take, a question, a mini-story, a stat, a lesson. Render the quote cards. Another list generator pulls the most quotable verbatim lines, and each one becomes a square typographic card via a text-to-image model. One recording, a week of content Everything reads from the same transcript, so the pack is consistent: the newsletter teases what the show notes explain, and the quote cards say it in the host’s own words. Run it as the last step of every edit session and publishing day becomes an upload, not a writing shift. Make it yours Tune each voice separately. Show notes, newsletter, and posts each have their own prompt and system prompt — make the newsletter personal and the posts punchy without compromise. Swap the transcription model. Whisper variants, local or hosted — the branch prompts don’t change. Restyle the cards. The quote-card prompt controls typography, palette, and format. Match your cover art, or go 9:16 for stories. Add branches. A YouTube description, a LinkedIn essay, a blog post — each is one more Prompt → Agent pair reading the same transcript. Models in this workflow Called with your own keys. The bill comes from the provider. The template ships with the defaults below and each is one dropdown. Role Node Shipped default Also works with Transcription Automatic Speech Recognition Whisper Large v3 on fal.ai Other Whisper variants, hosted or local Show notes, newsletter Agent GPT-5 mini Any strong language model Posts, quotes List Generator GPT-5 mini Any language model Quote cards Text To Image FLUX.1 Schnell (1:1, 1K) A model with reliable text rendering (Nano Banana, GPT Image) See [Models & Providers]({{ ‘/models-and-providers’ relative_url }}) to set up keys. Next steps Open the template: Examples in the sidebar, Workflows tab, Podcast Repurposing Studio [Ad Creative Factory]({{ ‘/use-cases/ad-creative-factory’ relative_url }}) — the same fan-out, pointed at paid social [Transcribe Audio]({{ ‘/workflows/transcribe-audio’ relative_url }}) — the transcription building block on its own [All use cases]({{ ‘/use-cases’ relative_url }})--- # Product Video Generator Source: https://docs.nodetool.ai/use-cases/product-video.html Use case · Marketing Turn a campaign brief and a single product photo into a cinematic 16:9 product video. Your inputs feed a prompt, an agent directs the shot, and an image-to-video model renders a ready-to-post clip. How it works Four inputs and four nodes: a brief, an audience, a feature list, and the hero photo go in; a finished clip comes out. The shipped template is named Product Video Generator (Examples in the sidebar, Workflows tab). {% mermaid %} graph LR brief[“Brief (String)”] audience[“Audience (String)”] features[“Features (String)”] photo[“Product photo (Image)”] prompt[“Prompt”] agent[“Director (Agent)”] i2v[“Render (ImageToVideo)”] brief –> prompt audience –> prompt features –> prompt prompt –> agent photo –> agent agent –> i2v photo –> i2v {% endmermaid %} Feed in the campaign. Three text inputs hold the brief, the target audience, and the key features. A product photo goes in alongside them as the hero shot. (e.g. “Aurora Trail smart fitness watch · active millennials who hike · GPS, heart-rate, adaptive coaching, water resistance”) Assemble the prompt. A Prompt node templates those inputs into a precise, rule-based request: one camera move, one subject action, lighting, color anchors, and hard constraints (no on-screen text, logos, or watermarks). Direct the shot. An agent turns the request into a concrete, cinematic prompt with framing, lens, and motion cues — the part most people get wrong by hand. Render the video. An image-to-video model animates the product photo from the agent’s prompt into a finished, ready-to-post clip. One photo in, a clip out The whole pipeline runs from a single product render. Same graph, any product. Input · product photo Output · cinematic clip Make it yours Swap the video model. Veo, Seedance, Kling, Runway. Change one node, the rest of the graph stays exactly the same. Retune the agent. Rewrite the system prompt for a luxury, playful, or hyper-technical tone without touching the pipeline. Change the format. Aspect ratio, resolution, and duration all live on the video node. Go vertical for social in one click. Reuse for any product. Drop in a new photo and brief, run it again. The workflow is the reusable part, not the output. Models in this workflow Called with your own keys. The bill comes from the provider, and Veo bills per second of video. The template ships with these defaults and each is one dropdown. Model Role Provider GPT-5 mini Writes the video prompt (the agent also sees the photo) OpenAI Veo 3.1 Preview Renders an 8-second 720p 16:9 clip from the product photo Gemini See [Models & Providers]({{ ‘/models-and-providers’ relative_url }}) to set up keys. Next steps [Movie Trailer Generator]({{ ‘/use-cases/movie-trailer’ relative_url }}) — one logline to a cut teaser [Color Boost Video]({{ ‘/workflows/color-boost-video’ relative_url }}) — polish footage on the canvas [All use cases]({{ ‘/use-cases’ relative_url }}) </content> </invoke>--- # SEO Content Engine Source: https://docs.nodetool.ai/use-cases/seo-content-engine.html Use case · Content One topic in, a keyword-targeted article batch out. A strategist agent plans the cluster, and every article arrives written, structured for search, and paired with an editorial hero image — the deliverable content agencies bill per article, produced by the batch at provider cost. Ranking rarely comes from one article. Search engines reward clusters — several pieces that own a topic together, each answering a different intent. Producing a cluster is what makes content expensive; this graph makes it one run. How it works The strategist plans once; everything after the list generator runs once per article. {% mermaid %} graph LR business[“Business & site (String)”] seeds[“Audience & seeds (String)”] count[“Article count (Integer)”] strategist[“Strategist (Agent)”] briefs[“Briefs (ListGenerator)”] writer[“Writer (Agent)”] hero[“Hero image (TextToImage)”] business –> strategist seeds –> strategist count –> strategist strategist –> briefs briefs –> writer briefs –> hero {% endmermaid %} Feed in the campaign. Who’s publishing and which pages the articles support, who searches and the seed topics, and how many articles to produce (2 to 10, default 4). (e.g. “ultralight hiking shop · beginner hikers · trail runners vs boots · 4 articles”) Plan the cluster. A strategist agent picks the topic cluster, assigns one primary keyword per article with its search intent, and divides coverage so the articles don’t compete with each other. Write the briefs. A list generator turns the plan into one brief per article: title, keyword, intent, H2 sections, and what the reader must be able to do afterward. Write the articles. For each brief, a writer agent returns typed fields: title, meta_description, keywords, and a Markdown body of 900 to 1,200 words with an answer-first opening, the brief’s section structure, a three-question FAQ, and one unforced CTA. The template previews each field separately, so nothing is merged into one blob to paste into a CMS. Render the heroes. In parallel, each brief becomes a 16:9 editorial hero image with room left for a headline overlay. One run, one cluster Because every article comes from the same plan, the batch behaves like a cluster instead of a pile: keywords don’t cannibalize, internal logic is consistent, and the voice section of the plan keeps ten articles sounding like one publication. Point the seed inputs at the next cluster and run it again. Make it yours Retune the writer. The article rules — length, structure, point of view, FAQ — are a prompt, not a setting. Make it terse and technical or warm and story-led. Raise the stakes per article. Swap the writer’s model up for money pages, down for supporting pieces. Change the imagery. The hero prompt controls the visual language — photography, flat illustration, product-in-context. Chain the next step. Add a Save Text node per article, or a translation branch to ship each cluster in every market’s language. Models in this workflow Called with your own keys. The bill comes from the provider. The template ships with the defaults below and each is one dropdown. Role Node Shipped default Also works with Cluster strategist Agent GPT-5 mini Any strong language model Brief writer List Generator GPT-5 mini Any language model Article writer Agent GPT-5 mini A long-output model you trust with prose Hero images Text To Image FLUX.1 Schnell (16:9, 1K) FLUX, Imagen, Nano Banana See [Models & Providers]({{ ‘/models-and-providers’ relative_url }}) to set up keys. Next steps Open the template: Examples in the sidebar, Workflows tab, SEO Content Engine [Ad Creative Factory]({{ ‘/use-cases/ad-creative-factory’ relative_url }}) — paid acquisition from the same brief-style inputs [Podcast Repurposing Studio]({{ ‘/use-cases/podcast-repurposing-studio’ relative_url }}) — the content pack for what you’ve already recorded [All use cases]({{ ‘/use-cases’ relative_url }})--- # NodeTool User Interface Source: https://docs.nodetool.ai/user-interface.html A tour of the interface. Same views on desktop and in the browser. New here? Start with Getting Started, then come back. Where everything lives View What it is Docs Workspace /workspace Where the app opens: your tabs, or the new-project surface when you have none Getting Started Projects — rail button Documents grouped by the job they belong to, with their own agent Projects Workflow Editor — workspace tab The node canvas, with panels on every edge Workflow Editor · Panels Chain Editor /chain/:workflowId? Linear card pipeline instead of a graph Chain Editor Chat — Chats panel Threads open as workspace tabs; the agent edits what you have open Chat Mini-Apps — Documents panel A form over one or more workflows Mini Apps Assets /assets Every file your workflows touch Assets · Sketch Editor Video Editor /timeline/:sequenceId Multi-track timeline; clips can be live workflow outputs Video Editor Collections /collections Indexed documents for RAG Collections · Indexing Examples — workspace tab Ready-to-run workflows, apps, storyboards, timelines, and games Templates Gallery Models /models and Model Manager tab Find, install, and manage local and cloud models Models Manager Settings — workspace tab API keys, folders, secrets, remote Configuration · Providers The logo at the top of the left rail opens the app menu: Settings, Help, and Downloads. Settings opens as a workspace tab. The other app pages are in the More panel on the left rail: Tutorials, Examples, Costs, Model Manager, Package Manager (development builds only), Assets, Collections, Workspaces, and Memory. Each opens as a workspace tab. More also holds Favorite Nodes, Recent Nodes, Workflow Settings, Skills, and Workspace files. The left rail’s direct icons are Documents, Chats, Library, and Nodes. Two extras: Mobile gives you a touch-optimized Chat, Mini-Apps, and Graph Editor, and the desktop app adds an install wizard, a system tray, and frameless mini-app windows. Projects A project is a name over the documents that belong to one job, such as a board, a script, a cut, or the key art. The Project selector at the left of the tab bar switches the active project, and new documents land in it. See Projects for the project list, starting a project, filing and copying documents, and archiving or deleting. Workflow Canvas An infinite canvas. Pan by dragging empty canvas with the left mouse button (the default on Windows and Linux) or, on macOS, by two-finger scroll or right or middle-button drag. Scroll to zoom, and press F when you have lost the graph off-screen. The Left-Click Drag setting under Settings → General → Canvas switches between panning and box selection. Add a node: press Space or double-click empty canvas. The node menu opens. Type what you want (“generate image”), or browse the categories on the left. Connect two nodes: drag from an output circle on the right of one node to an input circle on the left of another. Tip: drop a connection on empty space and NodeTool offers only the nodes that accept that type. Select a node and the Inspector on the right shows its properties, input and output slots, and help. Chat The agent, one panel over from whatever you have open. Describe a workflow and it builds one; ask for a change and it edits the open document — graph, sketch, timeline, storyboard, script, or mini app. It also runs workflows, takes image and audio and document attachments, and keeps each thread’s history separate. On desktop, the tray icon opens a standalone chat window. Threads sync with the main app. Mini-Apps A form or dashboard over your workflows, with the graph hidden — the version you hand to someone who has never heard of NodeTool. Build the workflows in the editor. Choose + New → New app in the tab bar. To scaffold one from a graph, use the from-workflow button in the left panel’s Apps list. Lay out input widgets, a run button, and display widgets in the app’s tab. Flip the tab to Run, then publish. The workflows stay separate resources. Linked workflows on the app tab opens one in its own tab. On desktop, right-click the tray icon to launch any app in its own window. Assets Images (PNG, JPG, GIF, WebP), audio (MP3, WAV, M4A), video (MP4, MOV, WebM), and documents (PDF, TXT, Markdown). Drag files into the panel to upload, drag one onto the canvas to use it, click to preview. Audio previews render a WaveSurfer waveform you can scrub, in the explorer and in node results. Panels and layout Open documents share one tab bar: workflows, sketches, timelines, storyboards, games, apps, chats, and app pages such as Settings. Drag a tab to reorder it. The left panel, Inspector, and bottom panel each open from their own edge and resize by dragging their inner border. Open or collapsed state and size are remembered between sessions. With nothing open, the workspace is a chat composer and a few sample prompts. Command Menu Ctrl+K / ⌘+K on any view, then start typing. It switches between your open tabs and projects, opens app pages (Projects, Assets, Model Manager, Settings, Studio, and the rest), creates new documents, opens your workflows by name, and offers Keyboard Shortcuts and Report a Bug. The view on screen adds its own group after your open tabs: View Commands Workflow Run Entire Workflow, Save, Auto Layout, import and export as JSON or bundle, edit and align, view and zoom, panel toggles Sketch Every sketch action and tool, with its shortcut Timeline Every timeline shortcut for the active keyboard preset Game The game editor’s commands for the current dimension and play state 3D model Undo, Redo, Save, transform tools, selection, and view commands Storyboard Undo, Redo, Assemble Timeline, Show or Hide Assistant Script Undo, Redo, Send to Timeline, Export Subtitles JS script Undo, Redo, Run Script, Run Tests Text file Save, Undo, Redo, word wrap Chat New Chat, Stop Generating The shortcut also opens the menu while the cursor is in a code editor. Keyboard Shortcuts The six worth memorizing: Shortcut Action Space Open node menu Ctrl/⌘ + Enter Run workflow Ctrl/⌘ + S Save Ctrl/⌘ + Z Undo F Fit view Esc Stop workflow Press Ctrl/⌘+/ for the full list inside the app. Global Shortcut Action Ctrl/⌘+K Command Menu Ctrl/⌘+, Settings Ctrl/⌘+Shift+Z Redo Ctrl/⌘+1…9 Switch to tab 1 to 9 Ctrl+PageUp / Ctrl+PageDown Previous or next tab Ctrl/⌘+T New workflow (desktop app) Ctrl/⌘+W Close tab (desktop app) Editor Shortcut Action Ctrl/⌘+F Find in workflow Ctrl/⌘+D Duplicate Ctrl/⌘+G Group B Disable or enable selected nodes A Align nodes I Toggle Inspector L Toggle Logs panel Delete / Backspace Delete selection See the Workflow Editor for the complete editor list. Chat Shortcut Action Enter Send message Shift+Enter New line Esc Stop generation Next Steps Workflow Editor – The canvas in depth Editor Panels – Left, right, bottom, and floating panels Tips & Tricks – Shortcuts the docs bury Cookbook – Workflow patterns--- # Vector Storage Source: https://docs.nodetool.ai/vector-storage.html NodeTool stores embeddings through a pluggable provider abstraction in @nodetool-ai/vectorstore. The same VectorProvider / VectorCollection interface backs the local SQLite-vec store, Supabase/pgvector, and a Pinecone stub, so every caller (core nodes, the tRPC collections router, the file-upload REST handler, and agent tools) uses the same interface for any backend. Choosing a Backend Backend When to use Setup sqlite-vec (default) Local-first, single-machine deployments. Embedded, zero-config. None — first use creates vectorstore.db in the NodeTool data dir. supabase Hosted multi-user deployments backed by a Supabase / Postgres+pgvector project. Install packages/vectorstore/sql/supabase-migration.sql once on the project, then set SUPABASE_URL + SUPABASE_KEY. pinecone Stub. Every collection method throws PineconeProvider is not yet implemented. Selecting it requires PINECONE_API_KEY. PINECONE_ENVIRONMENT and PINECONE_CONTROLLER_HOST are optional. Selection happens in createVectorProviderFromEnv() (packages/vectorstore/src/provider-factory.ts), which is called by getDefaultVectorProvider() on first use. An unknown value throws a ProviderConfigError listing the three valid kinds. Configuration The active backend is chosen by NODETOOL_VECTOR_PROVIDER. Defaults to sqlite-vec if unset. NODETOOL_VECTOR_PROVIDER=sqlite-vec (default) Env var Purpose VECTORSTORE_DB_PATH Override the SQLite database file. Defaults to vectorstore.db in the NodeTool data dir (getDefaultVectorstoreDbPath() in @nodetool-ai/config). The provider re-uses the process-wide getDefaultStore() so any code that touches the lower-level SqliteVecStore API and any code that goes through getDefaultVectorProvider() share one connection. NODETOOL_VECTOR_PROVIDER=supabase Env var Purpose SUPABASE_URL Supabase project URL (e.g. https://xyz.supabase.co). SUPABASE_KEY or SUPABASE_SERVICE_ROLE_KEY API key. Service-role is recommended server-side; anon-key requires RLS policies that grant the role access to the registry/records tables. NODETOOL_VECTOR_SCHEMA Optional Postgres schema; defaults to public. When both key variables are set, SUPABASE_SERVICE_ROLE_KEY wins. Constructing SupabaseProvider directly also accepts registryTable, recordsTable, and matchRpc to rename the two tables and the RPC. SUPABASE_URL and SUPABASE_KEY are the same variables the Supabase storage backend uses, so one Supabase project can back both file storage and vector data. One-time SQL install The provider does no DDL itself — it expects the schema to exist already. Run packages/vectorstore/sql/supabase-migration.sql once via the Supabase SQL editor or supabase migration. It creates: nodetool_vec_collections — registry table (name PK, metadata jsonb, dimension int, metric text). nodetool_vec_records — shared records table (collection FK, id, document, embedding vector, uri, metadata jsonb). The FK has ON UPDATE CASCADE so renaming a collection follows automatically — no record rewrite. nodetool_vec_match — RPC for similarity search. Required because PostgREST cannot use pgvector operators (<=>, <->, <#>) directly. Once the dimension is fixed for a collection, you can add an ivfflat index for faster search — the migration file documents the exact statement. Filter Support Matrix VectorFilter is a MongoDB-style predicate language. Adapters translate the subset they can support and must throw UnsupportedFilterError for anything else, rather than silently dropping conditions. Operator sqlite-vec supabase {field: scalar} (equality) — yes (via jsonb @>) {field: { $eq: v }} — yes {field: { $ne | $gt | $gte | $lt | $lte: v }} — — {field: { $in: [...] }} — — {$and: [...]} — yes (flattens into the metadata predicate) {$or: [...]} — — {$document: { $contains: "text" }} yes yes (case-insensitive ILIKE) {$document: { $or: [{$contains}, ...] }} yes — If you need a richer filter on Supabase, extend the SQL function and the splitFilter() translator in packages/vectorstore/src/supabase-provider.ts together — the JS-side translator is the contract. Programmatic API import { getDefaultVectorProvider, type VectorCollection } from "@nodetool-ai/vectorstore"; const provider = getDefaultVectorProvider(); // Create / get a collection. const docs = await provider.getOrCreateCollection({ name: "support_articles", dimension: 1536, metric: "cosine", metadata: { embedding_model: "text-embedding-3-small" } }); // Upsert records — embeddings can be pre-computed or generated by the // collection's embedding function. await docs.upsert([ { id: "kb-101", document: "How to reset your password", uri: "https://…" } ]); // Vector + metadata + document filtering. const matches = await docs.query({ text: "I forgot my password", topK: 5, filter: { $document: { $contains: "password" } } }); For tests or migrations, swap the active provider with setDefaultVectorProvider(...) (and resetDefaultVectorProvider() to release it). When constructing SupabaseProvider directly, pass a pre-built client to mock the network layer: import { SupabaseProvider } from "@nodetool-ai/vectorstore"; const provider = new SupabaseProvider({ client: fakePostgrestClient // any object satisfying PostgrestClientApi }); Where it’s used Workflow nodes: every node under vector.* in @nodetool-ai/core-nodes (packages/core-nodes/src/nodes/vector.ts) goes through the default provider. Collections REST/tRPC API: packages/websocket/src/trpc/routers/collections.ts and packages/websocket/src/collection-api.ts. Agent tools: list_collections, query_collection, create_collection, delete_collection, vector_index, vector_text_search, vector_hybrid_search, vector_recursive_split_and_index, vector_markdown_split_and_index, and vector_batch_index in the collections capability module (packages/agents/src/capabilities/collections.specs.ts, collections.ts). The collection each tool works on rides on the run (CapabilityRun.vectorCollection) rather than being a constructor argument. Adding a New Backend Implement VectorProvider and VectorCollection (packages/vectorstore/src/provider.ts) in a new <backend>-provider.ts. Translate VectorFilter to the backend’s query language. Throw UnsupportedFilterError for predicates you can’t express — never silently drop. Wire it into createVectorProviderFromEnv() in provider-factory.ts under a new kind. Add a tests file under tests/ that injects a mock client (no live backend). Re-export the provider class and its options interface from src/index.ts. The existing SqliteVecProvider and SupabaseProvider are good references — keep adapter state mutable so modify({ name, metadata }) is observable on the same handle, and surface uri on every read path (it’s part of the VectorRecord / VectorMatch contract).--- # Video Editor Source: https://docs.nodetool.ai/video-editor.html Cut a sequence on a multi-track timeline, bind any workflow to a clip, and generate the footage in place. NodeTool’s Video Editor is a non-linear editor where clips can be imported media or live workflow outputs that regenerate when you change their parameters. Quick Access: Open a timeline at /timeline/:sequenceId, or add a timeline tab from the workspace. Create a new sequence with + then New timeline in the workspace tab bar, or from the timeline list panel. Overview The Video Editor (timeline editor) is a non-linear, multi-track surface for assembling video, audio, and image clips. What sets it apart from a conventional NLE is that clips can be bound to NodeTool workflows — a clip can be the output of a text-to-image, image-to-video, or text-to-speech pipeline, and it stays editable: change a parameter and regenerate just that clip. Features: Multi-track timeline — video, audio, overlay, subtitle, and MIDI tracks Imported clips (drag media from the Asset Explorer) and AI-generated clips side by side Authored text, shape, 3D model, group, and adjustment-layer clips with preset-based motion Clip editing — move, trim, split, duplicate, delete, ripple and roll edits, three drop modes, with snap-to-playhead and snap-to-clip alignment Real-time preview compositing with a GPU compositor (WebGPU, Canvas2D fallback) and WebAudio mixing Frame-accurate transport — play/pause, stop, frame-step, J/K/L shuttle, in/out range, markers, skip to clip boundaries, timecode readout Generated clips bound to any workflow, with parameters exposed in the Inspector Staleness tracking — edit a bound workflow and the clip is flagged for regeneration Version history per clip — keep the last N successful generations and restore them Export — render the whole timeline to MP4, WebM, or a PNG sequence at the sequence resolution, in the browser or server-side Full undo/redo and debounced autosave Where this fits The Video Editor is the surface where outputs become finished, time-based media. It pulls assets in as clips, binds workflows to generated clips so footage regenerates when parameters change, and exports the sequence back to a video asset — the same material a sketch, another workflow, or a Mini-App can pick up. Every surface shares one asset store and the same model/provider system. See Key Concepts → How everything fits together for the full loop. Opening the Video Editor There are a few ways in: Direct route — navigate to /timeline/:sequenceId. Workspace tab — open a timeline as a tab alongside your workflows in /workspace. New sequence — click + in the workspace tab bar and choose New timeline, or use the timeline list panel. Each timeline (whether a standalone page or a workspace tab) runs in its own isolated set of stores, so multiple timelines can be open at once without interfering. Anatomy of the Editor Region What it does Tab-row actions Save, Export video, and an overflow menu with Project settings, Adapt format, Save as Asset, Export project (.zip), Export for Premiere Pro, and Export media (.zip). A code button appears when the timeline has embedded authoring code Preview The composited video with transport controls, timecode, a preview quality select (Auto, Full, Half, Quarter), and FPS readout Tracks The multi-track timeline — a canvas-rendered ruler, the playhead, and the clips Side panel Tabs for Inspector, Instruments, Source, Assistant, and History, plus Script and Code when the timeline has them The transcript panel on the left turns a script into beats: each line becomes a voiced, captioned clip on the timeline. Tracks & Clips The timeline holds multiple tracks. Add one with the add-track button and choose Video, Audio, Overlay, Subtitle, or MIDI. Video and overlay tracks composite top-down. Audio and MIDI tracks mix together. The toolbar above the tracks toggles Select and Cut tools, Ripple (trims and deletes close the gap they leave), the drop mode (Overwrite, Insert, or Overlap), Snap, Bars (a bars-and-beats ruler instead of timecode, with a grid division select), and Linked. Clip operations: Add — drag media from the Asset Explorer onto a track, or add a generated clip from the add menu. Images and video fit video and overlay tracks, audio fits audio tracks, and 3D models fit video and overlay tracks. A drop on an incompatible track is refused. Select — click a clip; Shift/Ctrl-click to multi-select. Move / trim — drag the body to move, drag the edges to trim the in/out points. Split — position the playhead and press S to cut a clip in two. Duplicate / delete — Ctrl/⌘ + D to duplicate, Delete to remove. Snapping — clips snap to the playhead and to neighbouring clip boundaries for clean alignment. Hold Alt while dragging to turn snapping off. Insert on drop — hold Ctrl/⌘ when releasing a dragged clip to push later clips right. Ripple delete — Shift + Delete removes a clip and closes the gap in the NodeTool layout. Roll edit — Ctrl/⌘-drag a clip edge to move the cut with its neighbour. Markers and range — M adds a marker, I and O mark the in and out points of a loop range. Transitions — Ctrl/⌘ + T cross-fades into the selected clips. Other transition types are dipToColor, wipe, push, slide, zoom, whip, zoomBlur, glitch, gradientWipe, iris, and lightLeak, set from the clip’s Transition section. Fades — drag the grip at a clip’s top corner for an audio fade. Alt + T fades the selection in and out. Linked clips that share a link (for example a video and its extracted audio) move and trim together. Preview & Playback The preview composites every visible, unmuted track in real time. Transport — play/pause, stop, step one frame forward/back, and skip to the previous/next clip boundary. Timecode — frame-accurate HH:MM:SS:FF readout, with the sequence FPS shown alongside. Scrubbing — drag the playhead across the ruler to scrub. Zoom — Ctrl/⌘ + scroll zooms the timeline anchored at the cursor. Shift + scroll pans. The timeline view control holds horizontal and vertical zoom sliders. Audio — per-clip gain and fades, per-track mute/solo and volume, mixed down to a master output. Audio clips render a waveform in the clip body. Video and audio are kept in sync by driving playback against the audio clock. Generated Clips A clip doesn’t have to be a file — it can be the output of a workflow. Generation modes: Text-to-video — on a video track, generate a clip from a prompt and a video model. Text-to-image / image-to-image — on an overlay track, generate a still from a prompt, optionally from a source clip. Image-to-video — animate an image clip from its Inspector. See Image to Video. Text-to-speech / text-to-music — on an audio track, synthesize speech or music. Workflow — bind any NodeTool workflow with an output node to the clip. The add menu opens prompt-first: type a prompt, pick a model, and press Enter to drop a clip at the playhead. Workflow-bound clips are under the Templates and All workflows tabs. The menu also adds a 3D model clip and an adjustment layer. When you bind a workflow, NodeTool clones it into a clip-private variant and exposes the workflow’s Input* nodes as editable parameters in the Inspector — so you can tweak the prompt, seed, or any input and regenerate without leaving the timeline. Staleness & versions: Each clip tracks a content hash of the bound workflow plus its parameter overrides. Edit the workflow (or its inputs) and the clip is flagged stale. A clip moves through states: draft → queued → generating → generated, plus stale, failed, locked, and missing. Successful generations are kept as versions — restore, favourite, or delete previous results per clip. Generation runs on the standard WorkflowRunner; status streams back over the WebSocket connection keyed by job, so the timeline only listens to the jobs it cares about. Round-tripping to the node editor: from a generated clip you can Open in Node Editor to edit the bound workflow on the full canvas, then return — the clip auto-marks stale so you can regenerate with the new logic. Inspector The Inspector swaps based on what’s selected: Imported clip — timing, render (transform, opacity, blend mode), and media controls, plus sections for time remap, keyframes, effects, mask and matte, tracking, Smart Reframe, and AI Edit where they apply. Generated clip — a vertical node stack of the bound workflow’s nodes, each with its parameters (reusing NodeTool’s property fields) and a per-node status indicator, with a Generate or Regenerate button. The clip-actions row has Duplicate, Regenerate as new clip, Lock, Replace Output, and Open in Node Editor. The History section lists versions. Text or shape clip — content and appearance controls for text, fill, stroke, geometry, and corner radius. Every visual clip includes an Animate section. Add an entrance, exit, emphasis, or loop preset, then adjust its timing, easing, and preset parameters. Image to Video Select an image clip and open Image to Video in the Inspector. Describe the motion, pick a model that supports image_to_video, and press Generate video. The video clip lands on the video track directly above the image, over the same span, and a new track is inserted when that span is taken. It keeps the image clip’s duration when the model can render it, and otherwise takes the shortest duration the model offers that covers the clip. The aspect ratio and resolution are the model’s nearest match to the image’s own size. The new clip is an image-to-video clip with the image as its source, so it regenerates like any other generated clip. Effects Clips and tracks carry an effects chain that runs in the same compositor as the live preview: Clip effects — color, blur, glow, drop shadow, vignette, sharpen, chroma key, curves, levels, lift/gamma/gain, grain, pixelate, posterize, directional blur, lens distortion, stylize, generator, and LUT. Track effects — video tracks take color correction, blur, sharpen, vignette, and chroma key. Audio tracks take gain, 3-band EQ, filter (lowpass, highpass, bandpass), and compressor. Clip transforms — opacity, blend mode (normal, multiply, screen, overlay, darken, lighten, color dodge, color burn, hard light, soft light, difference, exclusion, add), speed, time remap, and fade-in/out. The Canvas2D fallback skips chroma key, vignette, and sharpen. Animations Motion presets add animation without a keyframe editor. Select a visual clip, open Animate in the Inspector, choose a role and preset, then adjust duration or period, delay, easing, and preset parameters. In / Out — fade, slide, pop, spin, wipe, blur, and colorFade at a clip boundary. On text clips, typewriter is an In preset. Wipe reveals the clip behind a directional mask with an adjustable feathered edge; blur racks from soft to sharp; colorFade blooms grayscale into full color. Emphasis — pulse, flash, shake, bounce, squash, or followPath while the clip is active. Loop — kenBurns, float, breathe, rotate, hueShift, orbit, or followPath. Word stagger — on text clips, any animation can run once per word, each word offset in time from the previous (offset and on/off in the inspector; stagger on ui_timeline_animate_clip for the agent). Words pop, slide, or fade in sequence — motion typography without keyframes. See plans/motion-typography.md. Track markers — shaded edge wedges show entrance and exit windows; a loop icon marks emphasis or repeating motion. Select a marker to open its controls. Agent authoring — timeline tools can list presets, apply or clear motion, and inspect fixed-time frames before export. Preview and export sample the same animation resolver, so transforms and opacity match at each frame. Export Export video in the tab row opens the Export timeline dialog. Pick a format: Format Notes MP4 (H.264) Plays everywhere. No transparency. WebM (VP9) VP9 video with Opus audio. Can keep an alpha channel with Export with transparency. PNG sequence (.zip) One PNG per frame plus a manifest.json with the rate and size. No audio. Rendering is frame-by-frame at the sequence resolution through the same compositor used for preview, so the export matches what you see. Audio is mixed offline and muxed with the encoded video. A progress dialog reports the phase (preparing, audio, video, finalizing) and the export can be cancelled. Save as Asset renders the timeline and stores the result in an asset folder you pick. Export project (.zip) bundles the timeline project. Export for Premiere Pro downloads a zip with a Final Cut Pro 7 XML sequence, the media its clips use, and export-report.txt. In Premiere Pro, choose File > Import and pick the XML. Video, image, and audio clips keep their tracks, positions, in and out points, constant speed, opacity, volume, and enabled state. A video’s own sound lands on a linked audio track. Text, shapes, captions, effects, animations, transitions, and fades are not carried, and the report lists every clip affected. If Premiere asks to link media, locate one file in the media folder and it relinks the rest. Export media (.zip) downloads the original file of every video, image, and audio clip under media/, untrimmed, with clips.csv listing each clip’s track, timeline start and end, source in-point, and file. Import (the upload button beside New timeline in the timeline list) takes a NodeTool project zip or a Premiere Pro sequence. For Premiere, choose File > Export > Final Cut Pro XML, zip the XML together with its media files, and import the zip. NodeTool matches media by file name and creates a new timeline with the same tracks, clip positions, in points, constant speed, opacity, volume, enabled state, and markers. Transitions become overlapping cuts. Nested sequences, titles, generators, and effects are left out, and the import says what it skipped. The render_timeline agent tool and the nodetool.timeline.RenderTimeline node render the same sequence server-side. They run this exporter in headless Chrome or Chromium, so the server render matches the export. Without a browser they fall back to a Node compositor and say so in the job log. render_timeline returns a job id: poll it with get_job and stop it with cancel_job. Change the sequence width, height, frame rate, and tempo in Project settings. AI timeline editing Generated clips can hold multiple takes, source-time media tracks, derived reframes, and nondestructive generative edits. Use them to compare a result, follow a subject, make another aspect-ratio cut, or repair a shot without changing the original edit. See AI Timeline Editing. Keyboard Shortcuts Shortcut Action Space Play / pause V / C Select tool / cut tool S Split clip at playhead Ctrl/⌘ + Shift + K Cut all tracks at playhead Delete / Backspace Delete selected clip(s) Shift + Delete Ripple delete Ctrl/⌘ + C / X / V Copy / cut / paste clips Ctrl/⌘ + D Duplicate selected clip(s) Ctrl/⌘ + A Select all clips ← / → Nudge selection one frame Alt + ← / Alt + → Step the playhead one frame J / K / L Shuttle back / stop / forward I / O Mark in / mark out M Add marker N Toggle snapping Ctrl/⌘ + Z Undo Ctrl/⌘ + Shift + Z Redo Ctrl/⌘ + scroll Zoom timeline (anchored at cursor) ? Open the full shortcut sheet These are the NodeTool layout. The shortcut sheet (?) also offers Premiere Pro and Final Cut Pro layouts, stored in the timelineKeyboardPreset setting, and lists every binding for the layout you pick. Common Workflows Assemble a rough cut from your assets Open a new sequence and drag clips from the Asset Explorer onto the video track. Trim and reorder clips; use S to split and Delete to remove. Add an audio track for music or narration and adjust per-clip volume and fades. Scrub the preview to check timing, then Export to MP4. Generate a shot in place Add a generated clip on a video track and pick a video model (or bind a custom workflow). In the Inspector, set the prompt and inputs exposed from the workflow. Generate — the clip streams through queued → generating → generated. Not quite right? Tweak a parameter and Regenerate, or use Regenerate as new clip to keep the current render and compare takes. Iterate on a bound workflow Select a generated clip and Open in Node Editor. Edit the workflow on the full canvas and save. Return to the timeline — the clip is flagged stale. Select the stale clip and choose Regenerate. Related Features Workflow Editor — build the workflows you bind to generated clips Asset Management — organize the media you import as clips AI Timeline Editing — takes, tracking, reframing, generative edits, and interchange Sketch Editor — edit stills before or after they land on the timeline User Interface — tour of the main NodeTool views Next Steps Bind one of your existing workflows to a clip and generate a shot. Build a multi-track sequence mixing imported footage with generated clips. Explore the Cookbook for workflow patterns you can drop onto the timeline. Last updated: July 2026--- # Wan2GP Source: https://docs.nodetool.ai/wan2gp.html The Wan2GP pack adds three video nodes that call a Wan2GP server over MCP. The model runs in Wan2GP’s own process and Python environment, never inside NodeTool, so the pack itself installs no PyTorch. Node What it does wan2gp.text_to_video.TextToVideo Generates a clip from a prompt wan2gp.image_to_video.ImageToVideo Animates a still image wan2gp.generate.Generate Runs any Wan2GP model with a settings dict on top of that model’s defaults 1. Run a Wan2GP server Wan2GP needs its own conda environment and its own PyTorch build. Do not install it into the NodeTool environment. Follow Wan2GP’s installation guide. It gives one PyTorch build for RTX 20xx to RTX 50xx cards and an older one for GTX 10xx cards. Start it with its MCP server on streamable HTTP: python wgp.py --mcp --mcp-transport streamable-http --mcp-host 127.0.0.1 --mcp-port 7866 The MCP endpoint is then http://127.0.0.1:7866/mcp. Bind to 0.0.0.0 only on a trusted network or behind an authenticated reverse proxy. 2. Install the pack Desktop app: open Tools → Package Manager → Python packs and install Wan2GP. Python is set up first if it is missing. Without the desktop app: run pip install nodetool-wan2gp in the Python environment NodeTool uses. See Python nodes without the desktop app. 3. Point the nodes at the server Each node resolves the server URL when it runs, in this order: The node’s server_url field, when it is not blank. WAN2GP_MCP_URL, set in Settings → Integrations or in the environment. http://127.0.0.1:7866/mcp. A server on the same machine at the default port needs no setting. The Python worker receives WAN2GP_MCP_URL when it starts, so restart NodeTool after changing it. A shell export does not reach a desktop app opened from the Finder or the Start menu. Use the setting or the node’s server_url there. Combined GPU worker The pack repository also builds Dockerfile.combined, a GPU worker image that runs the NodeTool Python worker with a pinned Wan2GP runtime in one container. It does not start the NodeTool server. Attach a NodeTool server to it as a remote worker: export NODETOOL_WORKER_URL=ws://<gpu-host>:7777 export NODETOOL_WORKER_TOKEN=<the token the container was started with> nodetool serve Without NODETOOL_WORKER_URL, the server starts a local worker and never contacts the container. Build and run instructions are in the nodetool-wan2gp README. See Worker deployment for remote workers in general. Licensing The pack is AGPL-3.0-or-later. Wan2GP has its own license, which limits commercial use, and each model has a separate license. The combined image is for free, non-monetized use only. Read the pack’s NOTICE before you redistribute or sell access to it. Related Supported Models: local engines and hardware Node Packs: installing and updating packs Configuration: WAN2GP_MCP_URL and the worker variables--- # WebSocket API Source: https://docs.nodetool.ai/websocket-api.html This document describes the WebSocket API used to run workflows, stream results, and receive real-time updates from the NodeTool backend. Overview NodeTool exposes a single WebSocket endpoint for workflow execution and live updates. Clients connect once and multiplex commands and responses for many concurrent workflows over that connection. Endpoint: ws(s)://<host>/ws Auth: Send Authorization: Bearer <token> on the handshake (preferred) or append ?api_key=<token> to the URL. Tokens are optional in local/none auth modes. The connection’s identity comes from the handshake. A run_job frame needs no token of its own. Protocol: Binary (MessagePack) or Text (JSON) frames. The server decodes either kind of inbound frame. Outbound frames are binary until the client sends set_mode. See examples/workflow_runner/js/workflow-runner.js for a complete client implementation used by the bundled runner UI. Encoding The protocol supports two encoding modes on the same connection: Mode Frame type When to use Binary (default) Binary WebSocket frame MessagePack-encoded. Preferred for production use — compact format and native binary data (images, audio) without Base64 overhead. Text Text WebSocket frame JSON-encoded. Useful for debugging, curl/websocat testing, and lightweight clients that don’t need binary payloads. Binary frames are decoded with MessagePack and text frames as JSON, whichever the client sends. The format of the server’s replies does not follow the client’s frames. A new connection receives binary frames, and a client that wants JSON text frames sends set_mode with "mode": "text" first. Every server message is a map/object. Most carry a type field, but the acknowledgements described under Command replies do not. Limits on one connection: Limit Default Setting Inbound frame size 256 MiB NODETOOL_WS_MAX_MESSAGE_BYTES. An oversized frame is answered with an invalid_frame error and the connection stays open Inbound message rate 200 messages per 1000 ms NODETOOL_WS_RATE_LIMIT_MAX, NODETOOL_WS_RATE_LIMIT_WINDOW_MS, NODETOOL_WS_RATE_LIMIT_DISABLED. The socket closes with code 1008 Undelivered inbound frames 2000 NODETOOL_WS_MAX_QUEUED_FRAMES. The socket closes with code 1008 Protocol ping interval 20000 ms NODETOOL_WS_PING_INTERVAL_MS Silence before the peer is dropped 70000 ms NODETOOL_WS_IDLE_TIMEOUT_MS Unsent outbound bytes before sends wait 8388608 NODETOOL_WS_MAX_BUFFERED_BYTES, NODETOOL_WS_DRAIN_TIMEOUT_MS (30000 ms) NODETOOL_WS_HEALTH_DISABLED turns off the ping and idle checks. Connection Lifecycle Connect — open a WebSocket to ws://<host>/ws (or wss:// for TLS). For binary mode set binaryType = "arraybuffer". Ready — the onopen event fires; the client can now send commands. Heartbeat — the server sends {"type": "ping", "ts": <seconds>} every 25 seconds. A client may send {"type": "ping"} at any time and receives {"type": "pong", "ts": <seconds>}. A client pong is accepted and ignored. Reconnect — if the connection drops, retry. The web app backs off exponentially from 1 s to a 30 s ceiling with no attempt cap. The bundled workflow runner example retries every 5 s. Close — call socket.close() or let the server close the connection. Live Editor Renderer Bridge The web editor can execute ui_* tools for an MCP client through this same /ws connection. It does not open a second agent socket and it does not create a chat thread. The bridge uses these connection-level messages: Direction Type Purpose Server → editor renderer_registered Assign an ID to this editor connection. Editor → server client_tools_manifest Advertise the frontend tools available in this editor. Server → editor renderer_tool_call Ask the editor to execute one advertised tool. Editor → server renderer_tool_result Return the result or a structured error. The server keeps a user-scoped registry of ready editors. An MCP ui_* call can include renderer_id to select one editor. If it omits the ID, the server uses that user’s most recently active editor. The list_renderers CodeAct belt tool returns the available IDs. A disconnected editor is removed immediately. The normal MessagePack or JSON encoding rules apply. These frames do not use the { command, data } envelope. { "type": "renderer_tool_call", "renderer_id": "<renderer-id>", "tool_call_id": "<call-id>", "name": "ui_get_graph", "args": {} } { "type": "renderer_tool_result", "renderer_id": "<renderer-id>", "tool_call_id": "<call-id>", "ok": true, "result": { "nodes": [], "edges": [] }, "elapsed_ms": 12 } Multi-Instance Deployments A run’s replay buffer and its cancel/stream hooks live in the one server process executing it, so on a deployment with more than one instance both reconnect_job and cancel_job have to reach that process. Two environment variables drive this, and with neither set the whole mechanism is inert: NODETOOL_INSTANCE_ID — this instance’s identity. FLY_MACHINE_ID — the fallback, set by Fly on every machine. It is also the value fly-replay: instance=<id> addresses. The instance executing a run stamps its id on the job row (runner_instance). Two things follow. Resuming lands on the owner. A reconnecting client appends ?resume_job=<job_id> to the handshake URL. If that job is non-terminal and owned by another instance, the server answers the upgrade with fly-replay: instance=<owner> instead of accepting it, and Fly’s proxy re-issues the whole handshake there. A request the proxy already replayed (fly-replay-src present) is never replayed again, so this cannot ping-pong. The hint names one job — with runs in flight on several instances the rest reconnect wherever they land and fall back to reconnect_job’s persisted-row answer: the right status, without the replayed frames. The client retires the hint after two consecutive failed connects, so the third attempt goes out bare. Without that, a deploy that retires the owning machine while its row still reads running would have every reconnect replayed at a machine that no longer exists — and since the browser shares one socket across chat and every other consumer, all of them would stay dark. A successful connect resets the count. Cancel travels through the row. cancel_job for a run this process does not hold, whose row names a different instance, marks the row cancelled with a conditional update — only while it is still non-terminal, so it cannot overwrite the owner’s own outcome. Every instance re-reads its own running runs on a timer (NODETOOL_JOB_CANCEL_POLL_MS, default 15000, 0 disables) and cancels any whose row now reads cancelled: one indexed query per tick, bounded by that instance’s concurrency. The row is the only transport, so a cross-instance cancel takes up to a poll interval to land — the trade for having exactly one signal, the durable one. A cancel on the machine that does hold the run does not go through any of this; it reaches the session’s hooks directly and is immediate. A row with no runner_instance (an HTTP, trigger, or MCP run — nothing holds a session for those anywhere) is left alone and still answers “Job not found or already completed”. Draining A restart is what a detachable turn does not survive: the process goes away with the turn’s unwritten transcript rows and its unflushed spans. So a machine is drained before it is restarted. SIGUSR2 starts the drain, with no deadline of its own. From then on: /health answers 503 with status: "draining", so the proxy stops routing new clients here. The payload always carries turns and jobs — the counts of chat turns and workflow runs this process is still executing — whether it is draining or not, so a poller can read them either way. /ready is unchanged. A new /ws handshake is refused with 503; the client’s reconnect backoff carries it to a machine that is staying. A connection with nothing in flight is closed with 1012 (service restart). One driving a turn or a run is closed when that settles. A turn whose client has disconnected is aborted with the reason shutdown, at the start of the drain or when the client leaves during it. No client can reach it any more to answer an approval or press Stop, so it would otherwise hold the drain open. chat_message and run_job answer an error frame, and the socket closes under the rule above. The refusal comes before the user message is persisted, so the client’s retry on the other machine is the only copy. scripts/fly-rolling-deploy.sh is the caller: it signals one machine, polls its /health until turns and jobs are both 0, replaces it, waits for it to answer 200, and only then moves to the next. SIGTERM is the fallback, for a restart nobody drained. It starts the same drain, aborts every running turn with the reason shutdown and cancels every running run, then waits up to NODETOOL_SHUTDOWN_GRACE_MS (default 240000, under Fly’s 300 s cap) for them to settle before flushing telemetry and exiting. A turn aborted this way writes Stopped: server restarting into its thread — the one abort the user did not ask for, and so the one that says so. Client → Server Commands Commands use a command and data envelope. An optional request_id string is echoed back by the RPC commands. A frame with a command the server does not implement is answered with {"error": "Unknown command"}. { "command": "<name>", "data": { }, "request_id": "<optional>" } Command Purpose run_job Start a workflow run cancel_job Cancel a run reconnect_job Reattach to a run and replay missed frames stream_input, end_input_stream Feed a streaming input node update_node_properties Change node properties on a live run get_status Report active runs set_mode Choose binary or text frames clear_models Unload cached models chat_message, resume_chat, list_chat_turns, set_permission_mode Chat turns, see Chat API inference, stop Direct streaming inference and stopping work RPC commands list_workflows, get_workflow, list_assets, get_asset, list_nodes, get_node, generate_media, generate_text, transcribe_audio, lookup_generations A deployed app’s visitor reaches this socket through a scoped session and can use only run_job, reconnect_job, cancel_job, stream_input, end_input_stream, and get_status. Any other command is refused with This command is not available for a published app, and a job command for a run that does not belong to the app is refused with That run does not belong to this app. Command replies Every command except the RPC commands gets an acknowledgement frame right away. It has no type field. It is either an error object or a short status: { "message": "Job started", "workflow_id": "<uuid>" } { "error": "job_id is required" } The acknowledgement says the command was accepted, not that the work finished. Results arrive later as the typed messages below. Frames that fail validation get {"error": "invalid_frame" | "invalid_message" | "invalid_command", ...} with a message or details string. A non-command frame without a type the server knows is answered with invalid_message. run_job Start a new workflow execution. { "command": "run_job", "data": { "workflow_id": "<uuid>", "params": { "<input_name>": "<value>" }, "job_id": "<uuid>", "job_name": "Nightly render", "graph": { "nodes": [], "edges": [] }, "explicit_types": false } } Field Type Description workflow_id string UUID of the saved workflow to run. Omit it when graph carries the whole run params object Input parameter values keyed by input name job_id string \| null Optional client-generated id. The server generates one when it is absent job_name string \| null Title stored on the job row user_id string User the run is recorded under. Defaults to the connection’s user graph object \| null Optional graph override ({ nodes, edges }). A malformed graph is rejected with invalid_command concurrent boolean Let this run start while the same workflow already has a run in flight. Without it a workflow runs one at a time explicit_types boolean When false, let the server infer types project_id string \| null Project the run belongs to execution_options object \| null persistence ("job" or "session"), event_detail ("full", "outputs", or "terminal"), asset_persistence ("auto" or "temporary"). Defaults are job, full, auto require_terminal_result boolean SDK opt-in: a completed job_update counts as terminal only when it carries result.outputs supervise boolean Let an LLM supervisor answer node failures. Off unless set. See the supervisor design supervisor object Supervisor options: provider, model, max_decisions, max_retries_per_node, decision_timeout_ms, cost_cap_usd. Ignored unless supervise is true application_id, application_version, operation_id   Set by a mini app run so the server can check the app’s spend budget. application_version is the released version, absent for a draft The server keeps at most MAX_CONCURRENT_JOBS runs in flight per connection (default 4, a setting) and at most one per workflow unless concurrent is set (MAX_CONCURRENT_RUNS_PER_WORKFLOW, default 4, caps concurrent runs of one workflow). A run that has to wait gets a job_update with status: "queued" and a 1-based queue_position, and starts on its own when a slot frees. api_url, job_type, type, execution_strategy, and resource_limits appear in older clients and in the RunJobRequest type. The server reads none of them. execution_strategy is a column on the jobs table that nothing writes, and no threaded, subprocess, or docker execution mode exists. See Execution Strategies. cancel_job Cancel a running job. { "command": "cancel_job", "data": { "job_id": "<uuid>", "workflow_id": "<uuid>" } } reconnect_job Reattach to an in-flight job (for example after a page reload). The server sends a job_resumed header, then replays the frames the client missed. { "command": "reconnect_job", "data": { "job_id": "<uuid>", "workflow_id": "<uuid>", "last_seq": 41 } } Every frame of a run carries a job_seq number. last_seq is the highest one the client already holds, and 0 or an absent value means none. The header reports what the server holds: { "type": "job_resumed", "job_id": "<uuid>", "workflow_id": "<uuid>", "status": "running", "last_seq": 57, "replay_count": 16, "replay_incomplete": false } status is "running" (live frames follow the replay) or "finished" (the replay is the whole tail). replay_incomplete is true when last_seq is older than the bounded buffer still holds. When no run holds a replayable session, the server sends a plain job_update with the stored outcome instead of the header. stream_input Push a value into a streaming input node while a job is running. { "command": "stream_input", "data": { "job_id": "<uuid>", "workflow_id": "<uuid>", "input": "<input_name>", "value": "<any>", "handle": "<handle_name | null>" } } end_input_stream Signal that a streaming input is complete. { "command": "end_input_stream", "data": { "job_id": "<uuid>", "workflow_id": "<uuid>", "input": "<input_name>", "handle": "<handle_name | null>" } } update_node_properties Push property changes into the node executors of a running job, for example synth knobs while a patch plays. job_id, node_id, and properties are required. { "command": "update_node_properties", "data": { "job_id": "<uuid>", "node_id": "<uuid>", "properties": { "<name>": "<value>" } } } The reply is { "applied": true } when a running node took the change and { "applied": false } when it did not. A miss is not an error, because the canvas already holds the value for the next run. get_status Report active runs on this connection. Without job_id the reply lists them. With one it reports that run. { "command": "get_status", "data": { "job_id": "<uuid>" } } { "status": "running", "job_id": "<uuid>", "workflow_id": "<uuid>" } { "status": "not_found", "job_id": "<uuid>" } { "active_jobs": [{ "job_id": "<uuid>", "workflow_id": "<uuid>", "status": "running" }] } set_mode Choose the frame format for everything the server sends on this connection. mode is "binary" (MessagePack, the default) or "text" (JSON). Any other value gets { "error": "mode must be binary or text" }. { "command": "set_mode", "data": { "mode": "text" } } clear_models Accepted for compatibility. The server replies with a message and unloads nothing, because model lifetime belongs to the provider implementations. { "command": "clear_models", "data": {} } chat_message Start one chat turn on a thread. thread_id is required. The server saves the message, runs the turn, and streams chunk, message, tool, and approval frames tagged with the same thread_id. The acknowledgement is { "message": "Chat message processing started", "thread_id": "<id>" }. { "command": "chat_message", "data": { "thread_id": "<thread-id>", "role": "user", "content": "Summarize this workflow", "provider": "openai", "model": "gpt-5-mini", "permission_mode": "default" } } A plain chat turn without a provider and model pair is answered with an error frame before anything is saved. The other fields the turn reads: Field Description content A string or an array of content blocks workflow_id, project_id Context for the turn. A bare workflow_id does not run the workflow workflow_target "workflow" runs the workflow as a chatbot. It also takes workflow_message_input_name and workflow_messages_input_name media_generation { "mode": ... } where a mode other than "chat" produces an image, video, or audio message instead of an LLM reply permission_mode "plan", "default" (the fallback), or "auto". Governs whether gated tool calls run, ask, or are blocked system_prompt Extra system prompt text layered after the base prompt ui_context Which documents the user has open and which has focus A new chat_message on a thread supersedes a turn still running there. Turns are detached from the socket. If the connection drops, the turn keeps running and the client reattaches with list_chat_turns and resume_chat. resume_chat Reattach to a thread’s latest turn. The server sends a chat_resumed header and then the frames after last_seq, which is the highest chat_seq the client holds. 0 or an absent value declares a fresh client, and the server then replays only what the stored thread history cannot provide and sets replay_incomplete so the client reloads history over REST. { "command": "resume_chat", "data": { "thread_id": "<thread-id>", "last_seq": 12 } } { "type": "chat_resumed", "thread_id": "<thread-id>", "status": "running", "last_seq": 30, "replay_count": 18, "replay_incomplete": false } status is "running", "finished", or "unknown". "unknown" means no turn is held, either because none ran or because the retention window passed. Fetch the thread’s history instead. list_chat_turns Ask which of the caller’s turns are still running. The server sends one chat_turn_active frame per turn ({ "type": "chat_turn_active", "thread_id": "<id>", "status": "running", "last_seq": 30 }) and acknowledges with { "message": "Chat turns listed", "count": 1 }. set_permission_mode Change the permission mode of a thread’s turn while it runs. thread_id and permission_mode ("plan", "default", or "auto") are required. The change applies to a turn already running, and "auto" also approves every approval request that is waiting on the thread. { "command": "set_permission_mode", "data": { "thread_id": "<thread-id>", "permission_mode": "auto" } } inference Stream one model call with no thread and no saved messages. data takes provider, model (both default to the server’s), messages (each with role and content), and tools (each with name, description, and inputSchema). The server streams chunk frames and tool_call frames, each stamped with a seq number, and ends with { "type": "inference_done", "seq": n }. The acknowledgement is { "message": "Inference started" }. stop Stop generation. It cancels the connection’s running chat turn or inference, any pending RPC calls, and the run named by job_id or the turn named by thread_id. { "command": "stop", "data": { "thread_id": "<thread-id>", "job_id": "<uuid>" } } The server sends { "type": "generation_stopped", "message": "Generation stopped by user", "job_id": ..., "thread_id": ... } followed by the usual acknowledgement. RPC commands These commands return a single rpc_response frame that carries the request_id of the request. request_id is required. Without it the reply is { "error": "request_id is required for RPC commands" }. Command data Result list_workflows limit, run_mode, tag, cursor Same as the tRPC workflows.list procedure get_workflow id workflows.get list_assets parent_id, content_type, workflow_id, node_id, job_id, page_size assets.list get_asset id assets.get list_nodes namespace, query, fields ("summary" or "full"), limit nodes.list get_node node_type nodes.get generate_text provider, model, prompt, system, messages, max_tokens, schema, schema_name, schema_description { "text": string, "data": object \| null }. With schema the model answers through one forced tool and data carries the parsed object generate_media mode (image, image_edit, inpaint, upscale, outpaint, video, video_edit, video_extend, audio, music), provider, model, prompt (optional for upscale and outpaint), plus size, seed, scale, padding, reference, and voice fields { "asset_ids": string[] }. Creates no thread or message row transcribe_audio provider, model, asset_id, language Transcript of a stored audio asset lookup_generations request_ids { "generations": [...] } with request_id, generation_id, status, asset_ids, and error for each id that has a row. A client that reloaded mid-render uses it to recover results the closed socket never received { "type": "rpc_response", "request_id": "<id>", "command": "get_workflow", "result": { } } On failure the frame carries error instead of result, with code, message, retryable, and the optional apiCode and trpcCode. Other client frames Some frames have no command envelope. They answer a request the server sent, so a client that handles chat or the editor bridge sends them. type Answers Key fields ping   Server replies pong tool_result tool_call tool_call_id, result tool_approval_response tool_approval_request approval_id, decision ("allow", "allow_for_chat", or "deny") plan_approval_response   approval_id. Resolves a waiting approval by id. The server sends no plan approval request frame today secret_request_response secret_request approval_id, status ("saved" or "declined"). The secret value never travels in this frame client_tools_manifest   tools, the frontend tools this editor offers renderer_tool_result renderer_tool_call See Live Editor Renderer Bridge Server → Client Messages Every server message contains a type field used for dispatch. Messages also include routing fields (workflow_id, job_id, or thread_id) so the client can multiplex updates across concurrent workflows. job_update Reports overall job status changes. { "type": "job_update", "status": "running", "job_id": "<uuid>", "workflow_id": "<uuid>", "message": "optional status text", "error": null, "traceback": null, "result": null, "duration": null, "queue_position": null, "validation_issues": null, "error_code": null } Field Type Description status string "queued", "running", "completed", "failed", or "cancelled" job_id string \| null Job UUID workflow_id string \| null Workflow UUID for routing message string \| null Human-readable status message error string \| null Error description on failure traceback string \| null Stack trace on failure result object \| null Final result map on completion duration number \| null Execution duration in seconds queue_position number \| null 1-based place in the pending-run queue, present with status: "queued" validation_issues array \| null Per-property issues from pre-flight graph validation, present when a failed run was refused by validation instead of crashing error_code string \| null Machine-readable reason for a failed status. BUDGET_EXCEEDED means an app’s spend budget refused the run node_update Reports per-node status changes during execution. { "type": "node_update", "node_id": "<uuid>", "node_name": "GenerateImage", "node_type": "nodetool.image.Generate", "status": "running", "error": null, "result": null, "properties": null, "workflow_id": "<uuid>" } Field Type Description node_id string Node UUID node_name string Display name of the node node_type string Fully qualified node type status string "running", "completed", "error", or "warning" error string \| null Error message if the node failed, or the warning text error_detail object \| null Structured cause behind error, when the failure has one result object \| null Node output on completion. Constant and input nodes omit it, because the client already holds those values properties object \| null Updated node properties provider_cost object \| null Provider charge for the last completed run, when the node reports one workflow_id string \| null Workflow UUID for routing job_id string \| null Run id, so a client watching several runs can tell them apart node_progress Reports progress for long-running nodes (e.g. image generation steps). { "type": "node_progress", "node_id": "<uuid>", "progress": 3, "total": 20, "chunk": "", "workflow_id": "<uuid>" } Field Type Description node_id string Node UUID progress number Current step total number Total steps chunk string Optional text chunk for streaming output workflow_id string \| null Workflow UUID for routing job_id string \| null Run id output_update Delivers a final output value from an output node. { "type": "output_update", "node_id": "<uuid>", "node_name": "ImageOutput", "output_name": "image", "value": { "type": "image", "data": "<binary>" }, "output_type": "image", "metadata": {}, "workflow_id": "<uuid>" } Field Type Description node_id string Node UUID node_name string Display name output_name string Name of the output handle value any The output value (see Value Types) output_type string Type descriptor (e.g. "image", "string") metadata object Additional metadata disposition "append" \| "replace" append means value is a chunk to add to the live display buffer and replace means it is a whole snapshot. Absent means append done boolean Marks the last chunk of an append stream workflow_id string \| null Workflow UUID for routing job_id string \| null Run id edge_update Reports data flow status on a connection between nodes. { "type": "edge_update", "workflow_id": "<uuid>", "edge_id": "<edge_id>", "status": "active", "counter": 5 } Field Type Description workflow_id string Workflow UUID edge_id string Edge identifier job_id string \| null Run id. Edge animation is scoped per run status string "active", "completed", or "drained" (a target handle was left without input when the run ended) counter number \| null Number of items that have passed through log_update Streams log output from a running node. { "type": "log_update", "node_id": "<uuid>", "node_name": "RunModel", "content": "Loading model weights...", "severity": "info", "workflow_id": "<uuid>" } Field Type Description node_id string Node UUID node_name string Display name content string Log text severity string "info", "warning", or "error" workflow_id string \| null Workflow UUID for routing notification Server-initiated notification to display to the user. { "type": "notification", "node_id": "<uuid>", "content": "GPU memory low", "severity": "warning", "workflow_id": "<uuid>" } Field Type Description node_id string Originating node UUID content string Notification text severity string "info", "warning", or "error" workflow_id string \| null Workflow UUID for routing planning_update Reports agent planning phases. { "type": "planning_update", "phase": "analyzing", "status": "running", "node_id": "<uuid>", "content": "Determining approach...", "workflow_id": "<uuid>" } task_update Reports agent task progress. { "type": "task_update", "task": { "...task object..." }, "step": { "...step object..." }, "event": "step_started", "node_id": "<uuid>", "workflow_id": "<uuid>" } event is one of task_planned, task_removed, task_created, step_started, entered_conclusion_stage, step_completed, step_failed, task_completed, or task_failed. step is absent for task-level events. tool_call_update Reports when an agent node invokes a tool. { "type": "tool_call_update", "name": "web_search", "args": { "query": "example" }, "node_id": "<uuid>", "tool_call_id": "<id>", "workflow_id": "<uuid>" } tool_result_update Delivers the result of a tool call. { "type": "tool_result_update", "node_id": "<uuid>", "tool_call_id": "<id>", "name": "web_search", "result": { "...result data..." }, "is_error": false, "workflow_id": "<uuid>" } prediction Reports prediction/inference status from an external provider. { "type": "prediction", "id": "<prediction_id>", "node_id": "<uuid>", "status": "running", "user_id": "<user>", "workflow_id": "<uuid>", "logs": "Downloading model...", "error": null, "duration": null } chunk Streams incremental text/media content from a node. { "type": "chunk", "content": "Hello ", "content_type": "text", "content_metadata": {}, "done": false, "thinking": false, "node_id": "<uuid>", "workflow_id": "<uuid>" } Field Type Description content string Content fragment content_type string "text", "audio", "image", "video", "document", "tool_call", or "agent_status" content_metadata object Extra metadata for the content done boolean true on the final chunk thinking boolean true when the model is in reasoning/thinking mode node_id string \| null Node UUID workflow_id string \| null Workflow UUID for routing system_stats Reports the server’s CPU and memory load. Unlike every message above, this is not tied to a run: the server samples on a wall-clock cadence and pushes to every connected socket — one frame ~1s after connect (long enough for the CPU delta to mean something), then every 5s — whether or not a workflow is running. Clients that record a run’s frame stream should drop it as connection control, alongside ping/pong and resource_change. A server that enforces auth (SUPABASE_URL + SUPABASE_KEY — a shared deployment) sends this message never: its CPU and RAM belong to a container the user does not own. Clients must render the readout only once a frame arrives. { "type": "system_stats", "stats": { "cpu_percent": 23.4, "memory_percent": 61.2, "memory_used": 10522669056, "memory_total": 17179869184, "memory_used_gb": 9.8, "memory_total_gb": 16.0 } } Field Type Description cpu_percent number Whole-machine CPU use, 0–100, from the delta since the previous sample memory_percent number Used memory as a percentage of total memory_used / memory_total number Bytes memory_used_gb / memory_total_gb number The same figures in GB vram_total_gb / vram_used_gb number \| null Present only when the host samples a GPU; the default sampler omits them Other server messages type Sent when Key fields error A turn or run failed, or the server refused work (for example while draining) message, thread_id, workflow_id generation_complete A generator node committed one complete artifact node_id, node_name, node_type, outputs, index, job_id save_update A node saved a value node_id, name, value, output_type, metadata binary_update A node produced raw bytes node_id, output_name, binary step_result An agent step finished step, result, error, is_task_result todo_update An agent changed its to-do list todos (each with content and status of pending, in_progress, or completed) llm_call A node made an LLM call node_id, provider, model, messages, response, tool_calls, tokens_input, tokens_output, cost, duration_ms, error, timestamp provider_call_failed A provider call failed provider, model, operation, kind, status, message, request_id, duration_ms, timestamp supervisor_escalation A supervised run parked on a failed node node_id, node_name, escalation supervisor_decision The supervisor answered an escalation node_id, escalation, verdict, decided_by, cost message A chat turn persisted a message (assistant or tool) role, content, thread_id, tool_calls, tool_call_id, provider, model tool_call A tool the client owns must run thread_id, tool_call_id, name, args. Answer with a tool_result frame. Waits 300 s tool_approval_request A gated tool call needs approval thread_id, approval_id, tool_name, category, message, description, args. Answer with tool_approval_response secret_request Sandboxed code needs a credential the install lacks thread_id, approval_id, key, description, reason, help_url. Answer with secret_request_response generation_stopped A stop command was processed job_id, thread_id inference_done An inference call finished seq job_resumed, chat_resumed, chat_turn_active Replies to reconnect_job, resume_chat, list_chat_turns See those commands rpc_response Reply to an RPC command request_id, command, result or error resource_change A stored resource changed, from any tab, agent, or CLI call event (created, updated, or deleted), resource_type, resource, and optional ops ping, pong Heartbeat ts sdk_execution_target Connection open, when the server runs the SDK execution registry runner_id Chat frames carry a chat_seq number and workflow frames a job_seq number. Those numbers drive replay after a reconnect. Value Types Output values are plain JSON values or objects with a type discriminator. Image, audio, and video outputs are asset references. The server converts the bytes before it sends the frame, and the form depends on the connection mode: Mode Asset reference Binary (MessagePack, the default) { "type": "image", "uri": "<temporary URL>" }. The bytes are uploaded to temporary storage and fetched over HTTP Text (JSON, after set_mode) { "type": "image", "uri": "data:image/png;base64,..." }. The bytes are inline The same applies to audio and video. Other values keep their JSON shape: Type Shape Text "plain string" Number 42 or 3.14 Boolean true / false Object { ... } Other binary fields (Uint8Array) travel as raw byte arrays in MessagePack frames. In text mode they become arrays of numbers, not Base64. In-process audio chunks that carry native samples are encoded as base64 f32le in both modes. Message Routing The server tags each message with one or more routing keys: workflow_id — primary key for workflow execution updates. job_id — fallback when workflow_id is not present. thread_id — used for chat/conversation streams. The GlobalWebSocketManager in the main web app multiplexes a single connection and dispatches messages to per-workflow handlers based on these keys. The standalone workflow runner uses a simpler approach, handling all messages in a single callback. Typical Message Sequence Client Server | | |--- run_job ---------------------->| | | |<------------- job_update (queued) | |<------------ job_update (running) | | | |<---- node_update (node A running) | |<--- node_progress (node A 1/10) | |<--- node_progress (node A 5/10) | |<-- node_update (node A completed) | | | |<---- node_update (node B running) | |<-------------- output_update (B) | |<-- node_update (node B completed) | | | |<---------- output_update (final) | |<-------- job_update (completed) | | | Error Handling A command that fails validation or is refused gets an acknowledgement frame with an error field. See Command replies. Run failures arrive as job_update with status: "failed" and an error field, and chat failures arrive as error frames with a thread_id. Per-node errors arrive via node_update with an error field set. Reconnection is the client’s job. The server never reopens a connection. The reference client in examples/workflow_runner applies a 120-second timeout to each run_job call and rejects the promise if no terminal job_update arrives. Other WebSocket Endpoints /ws/download Model downloads. The endpoint exists only when NODETOOL_ENV is not production. Frames are JSON text in both directions. { "command": "start_download", "repo_id": "<hf-repo>", "path": null, "model_type": null } { "command": "cancel_download", "repo_id": "<hf-repo>" } start_download also takes allow_patterns, ignore_patterns, cache_dir, and scope. A scope of "worker" relays the download to a connected worker. A model_type that starts with tjs. downloads a Transformers.js model. The server streams progress frames until the download ends: { "status": "progress", "repo_id": "<hf-repo>", "path": null, "model_type": null, "downloaded_bytes": 1048576, "total_bytes": 4194304, "downloaded_files": 0, "current_files": ["model.safetensors"], "total_files": 3 } status is "idle", "start", "progress", "completed", "cancelled", or "error". An error frame carries an error string. A protocol failure is sent as { "status": "error", "error": "<message>" }. For REST-style downloads see Downloading a Model over the SDK Routes. Quick Start Examples Binary mode (MessagePack) // Add ?api_key=YOUR_TOKEN when the server enforces auth const socket = new WebSocket("ws://localhost:7777/ws"); socket.binaryType = "arraybuffer"; socket.onopen = () => { socket.send( msgpack.encode({ command: "run_job", data: { workflow_id: "YOUR_WORKFLOW_ID", params: { prompt: "hello world" } } }) ); }; socket.onmessage = (event) => { const data = msgpack.decode(new Uint8Array(event.data)); console.log(data.type, data); }; Text mode (JSON) const socket = new WebSocket("ws://localhost:7777/ws"); socket.onopen = () => { // Replies are binary until the client asks for text socket.send(JSON.stringify({ command: "set_mode", data: { mode: "text" } })); socket.send( JSON.stringify({ command: "run_job", data: { workflow_id: "YOUR_WORKFLOW_ID", params: { prompt: "hello world" } } }) ); }; socket.onmessage = (event) => { const data = JSON.parse(event.data); console.log(data.type, data); };--- # WebSocket Runner Refactor Plan Source: https://docs.nodetool.ai/websocket-runner-refactor-plan.html WebSocket Runner Refactor Plan Split packages/websocket/src/unified-websocket-runner.ts (10,115 lines, one class) into a small connection host plus four domain classes behind one seam interface, without changing wire behavior. Problem UnifiedWebSocketRunner is one class doing six jobs. Measured layout of the file: Region Lines Content Module-level helpers 1–2,150 Error/text sanitization, autoSaveAssets, audio chunk encoding, chat system prompt, tool-name plumbing, serverModelInterfaces — no this dependency at all Job execution 3,135–4,860 Admission gates, cost estimation, refuseRun, concurrency queue, startJob, streamJobMessages, reconnect, cancel, terminal-status persistence Chat 4,937–8,327 dbMessageToProviderMessage, handleChatMessage (1,300 lines), history/skill/plan assembly, cost handlers, createWorkflowResponseContent Direct inference 8,327–8,640 handleInference media/text generation RPC and commands 8,640–9,760 entityRefResolver, getTrpcCaller, the 540-line handleCommand switch Socket plumbing rest connect/disconnect/send/receive, heartbeat, stats broadcast, renderer bridge, ToolBridge Costs of this shape: Every private member is reachable from all 10k lines, so nothing documents what belongs to what. activeJobs is mutated from three regions. A change to chat and a change to job admission conflict in the same file. The regions cannot be tested apart. The existing suites (unified-websocket-runner*.test.ts) drive everything through the socket. The file is too large to review or navigate as a unit. The this.* usage per region was counted before designing the split. The overlap is narrow: every region uses send, identity (userId, appSession), logError, and the resolver factories from UnifiedWebSocketRunnerOptions. Region-specific state (chatRequestSeq, jobQueue, rpcAborts) never crosses regions, with one exception: chat and RPC read activeJobs directly. That shared subset is the seam. Solution overview One interface, ClientSession, carries the shared subset. It is the only thing extracted classes know about the connection. Four classes own the four domains: JobExecutionManager, ChatTurnHandler, DirectInferenceHandler, CommandRouter. The remaining class — renamed WebSocketClientSession — implements ClientSession, owns the socket lifecycle, constructs the four classes, and routes inbound frames to them. Target size: 600–800 lines. The ~1,900 lines of this-free helpers become plain-function modules. Everything stays in packages/websocket/src/; no new npm package. New files live in packages/websocket/src/session/. Old names (UnifiedWebSocketRunner, UnifiedWebSocketRunnerOptions) and old export paths survive as aliases until callers and tests migrate, so no step forces a repo-wide rename. Behavior is unchanged throughout: same frames in, same messages out, same queue and abort semantics. The existing test suites keep passing at every step because they test through the public surface. Architecture WebSocketClientSession (implements ClientSession; composition root) ├─ owns: socket, send lock, outbound validation, heartbeat, stats, │ renderer registry/bridge, ToolBridge instances, frame router ├─ JobExecutionManager run_job / reconnect / cancel / status / queue ├─ ChatTurnHandler handleChatMessage, turn lifecycle, permissions ├─ DirectInferenceHandler handleInference └─ CommandRouter handleCommand dispatch, tRPC glue, RPC aborts session/sanitize.ts pure: error + large-text sanitization session/asset-autosave.ts pure: autoSaveAssets + image/audio helpers session/chat-prompt.ts pure: system prompt, permission prompts, tool-name plumbing, formatUiContext session/chat-history.ts pure: db→provider message conversion, context/skill sections, response content session/model-interfaces.ts serverModelInterfaces, createRuntimeContext Frame routing: run_job/reconnect_job/cancel_job → jobs; chat messages and tool_result → chat; inference frames → inference; everything else → CommandRouter.handle, which also composes the other three for commands that touch them (stop, get_status, set_permission_mode). Design ClientSession (interface — the seam) // session/client-session.ts export interface ClientSession { readonly userId: string | null; readonly appSession: AppSessionScope | null; readonly mode: WebSocketMode; /** Serialized, validated, seq-stamped send. The host keeps the lock. */ send(message: Record<string, unknown>): Promise<void>; /** Fire-and-forget send for messages that must not await the lock. */ sendDetached(message: Record<string, unknown>): void; logError(context: string, error: unknown): void; readonly resolveExecutor: ResolveExecutor; readonly resolveProvider?: ResolveProvider; readonly resolveNodeType?: ResolveNodeType; readonly getNodeMetadata?: GetNodeMetadata; readonly validateNode?: ValidateNode; readonly workspaceResolver?: WorkspaceResolver; readonly nodeRegistry?: NodeRegistry; readonly pythonBridge?: PythonBridge; } Facts a caller must know beyond the types: send resolves once the frame is accepted for delivery — written to the socket, or buffered into the replay session of the chat turn or run it belongs to, which may deliver it to a later connection or to no one, since that buffer is bounded and expires (it never throws for a dropped socket — that is the current sendMessage contract); userId is the app owner when appSession is set, so authorization decisions must consult both; userId and mode both change during a connection’s life, so a class reads them at call time rather than capturing them. Tests get a FakeClientSession that records sent messages — the assertion surface the existing suites already use, one level up. JobExecutionManager Owns everything between “run_job arrived” and “terminal status persisted”. // session/job-execution.ts export class JobExecutionManager { constructor(session: ClientSession, deps: { beforeRunJob?: BeforeRunJob; getMaxConcurrentJobs: () => Promise<number>; }); runJob(req: RunJobRequest): Promise<void>; reconnectJob(jobId: string, lastSeq: number): Promise<void>; cancelJob(jobId: string): Promise<Record<string, unknown>>; getStatus(jobId?: string): Record<string, unknown>; resolveJobControl(jobId: unknown): JobControl | null; getSdkExecutionCapacitySnapshot(input: SdkCapacityInput): SdkExecutionCapacitySnapshot; cancelAll(): Promise<void>; // disconnect path get slotCounters(): { activeJobs: number; startingJobs: number }; } Private, and after the move reachable from nowhere else: activeJobs, startingJobs, jobQueue, dequeuedJobs, drainQueue, broadcastQueuePositions, startJob/startJobInner, streamJobMessages, persistTerminalJobStatus, settleApplicationInvocation, settleRunAgainstLostConnection, refuseRun, admission gates (confineAppSessionRun, admitApplicationRun, admitCreditRun), cost estimation (estimateRunCost, estimateGraphCost, estimateNodetoolSpend), cost handlers (_handleNodeProviderCost, _handlePredictionCost, runMeasuredCost), graph hydration (getRawGraph, normalizeGraph). Invariants it owns (currently documented on the runner’s fields, moved with them): the synchronous slot reservation that keeps two back-to-back run_job commands from both passing the concurrency gate, and the slotCounters leak accounting the reliability harness reads. ChatTurnHandler Owns the turn lifecycle and all chat-only state: chatRequestSeq, chatAbort, chatSessionAllow, chatTurnPermissionMode, chatCapabilityRun. // session/chat-turn.ts export class ChatTurnHandler { constructor(session: ClientSession, deps: { jobs: JobExecutionManager; // chat runs workflows as tools toolBridge: ToolBridge; // created by the host — frames land there approvalBridge: ToolBridge; clientTools: () => Record<string, Record<string, unknown>>; beforeRunJob?: BeforeRunJob; defaults: { provider: string; model: string }; }); handleChatMessage(data: Record<string, unknown>, requestSeq?: number, signal?: AbortSignal): Promise<void>; cancel(): void; // stop + disconnect resolveToolResult(toolCallId: string, payload: Record<string, unknown>): void; setPermissionMode(threadId: string, mode: PermissionMode): void; getCapabilityRun(): CapabilityRun | null; } handleChatMessage (1,300 lines) is itself split along its sequential phases: thread setup, history conversion, toolbelt assembly, the provider stream loop, output persistence. History conversion and response-content building are near-pure and move to session/chat-history.ts as functions (dbMessageToProviderMessage, appendContextToLastUser, invokedSkillsSection, attachPlanApproval, toolResultDisplayText, createWorkflowResponseContent — the last is 1,300 lines of value→content mapping that touches no connection state). DirectInferenceHandler // session/inference.ts export class DirectInferenceHandler { constructor(session: ClientSession, deps: { defaults: { provider: string; model: string }; /** Registers an abort controller; returns the deregistration. Backed by the host's rpcAborts set so `stop` and disconnect reach it. */ registerAbort: (c: AbortController) => () => void; }); handleInference(data: Record<string, unknown>, requestSeq?: number): Promise<void>; } Private: runDirectMediaGenerationInner, runDirectTextGenerationInner, the direct-generation request types, estimateDirectTextSpend (already a module-level export; stays one). CommandRouter The 540-line handleCommand switch becomes a dispatch table. It is the one place that composes the other classes. // session/commands.ts export class CommandRouter { constructor(deps: { session: ClientSession; jobs: JobExecutionManager; chat: ChatTurnHandler; inference: DirectInferenceHandler; trpc: () => TrpcCaller; // lazy, as getTrpcCaller is today apiOptions?: HttpApiOptions; registerAbort: (c: AbortController) => () => void; }); handle(command: string, data: Record<string, unknown>, requestSeq?: number): Promise<Record<string, unknown> | null>; } Private: the per-command handlers, runRpc, entityRefResolver, retrieveSourceAssetBytes, tRPC caller construction. WebSocketClientSession (the host; renamed from UnifiedWebSocketRunner) Keeps: connect/disconnect/run/receiveMessages, the send lock and sendToSocket with outbound validation (the implementation behind ClientSession.send), MsgPack/JSON mode handling, heartbeat, stats broadcast, observer registration, renderer registry and rendererToolBridge, clientToolsManifest intake, the rpcAborts set, and the frame router. Constructor builds the four domain classes and the bridges. Aliases for the transition: export { WebSocketClientSession as UnifiedWebSocketRunner }; export type { WebSocketClientSessionOptions as UnifiedWebSocketRunnerOptions }; Design decisions D1 — one seam, not five. The measured overlap (send, identity, log, resolvers in every region) justifies one shared ClientSession; region-specific needs go in each constructor’s deps so the interface stays small. Handlers taking their exact dependencies individually was considered and rejected: nine-parameter constructors restate UnifiedWebSocketRunnerOptions four times. D2 — activeJobs gets one owner. Chat and RPC currently reach into the map; afterwards they go through resolveJobControl/getStatus. This is the only call-shape change in the refactor; everything else is a move. D3 — no new package. packages/websocket/src/session/ next to the existing job-queue.ts, job-control.ts, chat-turn-registry.ts. A package split can come later if another server ever hosts these classes; today one adapter means a hypothetical seam. D4 — bridges stay with the socket. ToolBridge instances are created by the host (frames arrive there) and injected into ChatTurnHandler, so a dropped socket can cancelAll() without asking the chat handler. D5 — rename via alias, delete later. The rename ships with re-exports; the ~10 test files and index.ts migrate in a follow-up mechanical PR, and the aliases are removed once nothing imports them. Task plan One PR per task, in order. After each: npm run test:affected, npm run typecheck, npm run lint. The existing unified-websocket-runner*.test.ts suites must pass unmodified through T1–T7 (T8 may move them). T1 — extract pure modules. Move the this-free helpers to session/sanitize.ts, session/asset-autosave.ts, session/chat-prompt.ts, session/model-interfaces.ts; re-export current names from the runner file. ~1,900 lines out, zero behavior risk. T2 — introduce ClientSession. Define the interface, have the runner implement it (send delegating to sendMessage), add FakeClientSession to the test utilities. No extraction yet. T3 — extract JobExecutionManager. Largest and riskiest step; includes the D2 call-shape change for chat/RPC access to jobs. Verify with the Ring 0 reliability journeys (nodetool harness gate --base main selects them for this diff) in addition to the standard three checks — the slotCounters leak invariant is theirs. T4 — extract chat. session/chat-history.ts (pure functions) first, then ChatTurnHandler. Split handleChatMessage into phase methods in the same PR only if the diff stays reviewable; otherwise a follow-up. T5 — extract DirectInferenceHandler. Small; also introduces registerAbort over the host’s rpcAborts. T6 — extract CommandRouter. Switch → dispatch table; move RPC glue. T7 — rename. UnifiedWebSocketRunner → WebSocketClientSession, file → websocket-client-session.ts, aliases per D5. T8 — migrate callers and tests to the new names; delete aliases. Rollback story: every task is a move with re-exports, so reverting any single PR restores the previous shape without touching the others.--- # Worker Deployment Source: https://docs.nodetool.ai/worker-deployment.html NodeTool runs most graphs on your machine. When a node needs a GPU you don’t have — large image/video models, HuggingFace pipelines — you can rent one for the duration of the work instead of buying hardware. A worker is a remote box running the lean NodeTool Python worker image; your local NodeTool instance attaches to it, runs the Python nodes there, and tears it down when you’re done. This is a different subsystem from server deployment. A server is long-lived infrastructure that humans connect into. A worker is an ephemeral, billing-sensitive box that one NodeTool instance connects out to. Because a rented GPU bills while it exists, teardown is the headline feature. Read the cost guard below, including the difference between stopping and terminating a worker. The model: profiles → instances Two concepts, deliberately split:   Worker profile Worker instance What A declarative, reusable preset A live, running worker Lifetime Permanent until you delete it Ephemeral — spin up, attach, tear down Holds target, image, GPU, vCPU, volume size, optional SSH public key, token policy, idle timeout, max lifetime the provider’s pod/instance id, the ws:// or wss:// URL, the bearer token, status, estimated cost Stored worker_profiles table (DB) worker_instances table (DB) A profile is the recipe (“an A40 RunPod pod running the HuggingFace worker image, idle-stop after 15 minutes”). Provisioning a profile launches an instance. Instances are never written to deployment.yaml — nothing should ever be able to resurrect a torn-down GPU pod from declarative config. Both tables live in NodeTool’s SQLite DB so the UI and CLI share one source of truth. Prerequisites A RunPod account and API key (runpod.io console → settings), a Vast.ai account and API key, or a Verda cloud API client id and secret (API credentials). Verda’s cloud credentials are separate from its inference key, and are deleted when the team member who created them is removed from the project. A worker container image — the published NodeTool worker image, or your own built from a NodeTool Python package. It must run python -m nodetool.worker on port 7777 (msgpack RPC, bearer-token auth). Worker images Image Contents Use for ghcr.io/nodetool-ai/nodetool-worker:latest The lean Python worker Python nodes, HuggingFace pipelines, LLM providers ghcr.io/nodetool-ai/nodetool-worker-comfy:latest Worker + a co-located, loopback-only ComfyUI Everything above plus the Run ComfyUI Workflow (Worker) node The worker image installs PyTorch 2.14 from PyPI, whose Linux build uses CUDA 13.0. A GPU host therefore needs NVIDIA driver 580 or newer. On an older driver the HuggingFace nodes run on the CPU or fail their VRAM checks. NodeTool does not filter hosts by CUDA version, so check the driver of the host you rent. A worker started from the ComfyUI image fronts ComfyUI over the worker bridge (ComfyUI itself is never exposed outside the container) and reports worker.status.comfy.enabled: true. The Run ComfyUI Workflow (Worker) node runs an API-format ComfyUI workflow on such a worker. Pick the ComfyUI image from the Worker image preset dropdown in the Profiles editor, or pass --image ghcr.io/nodetool-ai/nodetool-worker-comfy:latest on the CLI. See ComfyUI for the node’s properties, input handles, and the comfy.* bridge surface. Store the API key in the secret store so the manager can read it: nodetool secrets store RUNPOD_API_KEY # prompts for the value nodetool secrets store VAST_API_KEY # for Vast.ai nodetool secrets store VERDA_CLIENT_ID # for Verda (both are required) nodetool secrets store VERDA_CLIENT_SECRET If the secret store is unreachable (headless/sandboxed), the manager falls back to the environment variables of the same names. Supported targets Target Provider URL form Teardown runpod RunPod pod (REST rest.runpod.io/v1/pods) wss://<podid>-7777.proxy.runpod.net deletes the pod and its volume vast Vast.ai instance (console.vast.ai/api/v0) ws://<ip>:<port> destroys the instance and its disk verda Verda VM (REST api.verda.com/v1) ws://<ip>:7777 deletes the instance and permanently deletes its OS volume The target column shows what terminate does. Each target also has a cheaper stop (pause), described under Cost guard. All three run the same worker image — there is no per-provider image work. Local or LAN workers are also supported, but unmanaged: run the worker container yourself and point NODETOOL_WORKER_URL (and NODETOOL_WORKER_TOKEN) at it. There is no provisioning provider for local Docker — you start and stop it. Verda Verda rents virtual machines, not containers, so a Verda profile needs two images: the guest OS (osImage, a CUDA + Docker image from GET /v1/images) and the worker container (image, as on every target). Leave osImage unset and the provider picks a CUDA+Docker image from the live catalog. Its gpu field is a Verda instance type — 1H100.80S.30V, not a GPU name. List the live ids with GET /v1/instance-types; availability is per-location, and a type absent from every location has no capacity right now. Three behaviours differ from RunPod and Vast, all of them cost-relevant: Verda bills a shut-down instance at the full rate, and removed its hibernate action for that reason. So NodeTool’s pause does not shut the machine down: it deletes the instance and retains the OS volume, which ends the GPU charge while keeping the model cache. Resuming boots a new machine from that volume, so its IP — and its provider handle — change. Terminate deletes the OS volume permanently. A trashed volume still holds storage quota and can be restored for a charge covering the deleted interval, which is not what teardown promises. The cached models are gone for good. Only Pay As You Go is used. Long-term contracts are prepaid, and spot instances can be discontinued mid-workflow; neither is chosen for you. A Verda worker publishes port 7777 on a public IP, and Verda’s own security guide warns that a UFW rule does not block a Docker-published port. The provider therefore refuses to launch a worker with no bearer token — keep token_policy on generate unless you have a specific reason not to. Pay As You Go bills in prepaid ten-minute increments, and running out of balance can discontinue instances and delete volumes. A low balance is not harmless billing metadata. Quick start (CLI) # 1. Store your provider API key nodetool secrets store RUNPOD_API_KEY # 2. Create a reusable profile nodetool worker profile add hf-a40 \ --target runpod \ --image ghcr.io/nodetool-ai/nodetool-worker:latest \ --gpu "NVIDIA A40" \ --disk 100 \ --idle-timeout 15 \ --max-lifetime 120 # 3. Provision an instance from it and attach in one step nodetool worker create --profile hf-a40 --attach # 4. Watch what's live (and what it's costing) nodetool worker list # 5. Pause it when you're done (keeps the volume and model cache) nodetool worker stop <instance-id> # or pause every live worker: nodetool worker stop --all worker stop pauses. The CLI has no terminate command. To destroy a worker and its volume, use Terminate in the Workers panel, or let --max-lifetime destroy it. worker create prints the new instance id, its wsUrl, a redacted bearer token, and its status. With --attach it also points your bridge at the worker and prints export NODETOOL_WORKER_URL=…, followed by a commented hint for the token (the raw token is never printed to stdout): export NODETOOL_WORKER_URL=wss://… # NODETOOL_WORKER_TOKEN was redacted; set it with: # export NODETOOL_WORKER_TOKEN=$(nodetool worker token <instance-id>) Retrieve the full token on demand with nodetool worker token <id>, which prints only the token so it pipes cleanly into NODETOOL_WORKER_TOKEN. CLI reference nodetool worker profile add <name> --target <runpod|vast|verda> --image <img> \ [--gpu <type>] [--vcpu <n>] [--disk <gb>] [--ssh-key <pubkey-file>] \ [--token-policy <generate|fixed>] \ [--idle-timeout <minutes>] [--max-lifetime <minutes>] nodetool worker profile list [--json] nodetool worker profile rm <name> nodetool worker create --profile <name> [--attach] nodetool worker create --target <t> --image <img> [--gpu <g>] [--attach] # inline, one-off # the inline form also takes --vcpu, --disk, --ssh-key, # --token-policy, --idle-timeout, --max-lifetime nodetool worker list [--json] nodetool worker status <id> # refresh status from the provider nodetool worker token <id> # print the decrypted bearer token (pipeable) nodetool worker stop <id> # pause: release the GPU, keep the volume nodetool worker stop --all # pause every live worker nodetool worker models list [worker-id] [--json] nodetool worker models download [worker-id] --repo-id <owner/name> \ [--file-path <path>] [-a <glob>]... [-i <glob>]... nodetool worker models delete [worker-id] --repo-id <owner/name> The inline form of create synthesises a throwaway profile named inline-<target>-<timestamp> from the flags, so you don’t have to define one first. Flag notes: --disk <gb> sizes the persistent volume. The worker’s HF_HOME is /workspace/huggingface, so the Hugging Face cache lives there and survives pause and resume. The default is 100. --ssh-key <file> takes an OpenSSH public key file. It exposes port 22 and passes the key to the worker as PUBLIC_KEY, so you can log in to debug. Without it, the bridge is the only way in. --token-policy generate (the default) mints a random bearer token for each instance. fixed mints none. --idle-timeout and --max-lifetime take positive whole minutes. worker models manages the Hugging Face cache on a worker over its bridge. Pass a worker id, or omit it to use the attached worker. download sends your HF_TOKEN (from the secret store, then the environment) because the worker has no Hugging Face credential of its own, and -a and -i filter files by glob. The worker image must report bridge protocol version 2 or later. Attaching from the UI The Workers tab in the bottom panel is the main surface: Manage Profiles — pick a provider, name, GPU, vCPU, disk (GB), idle timeout, max lifetime, worker image, and token policy, and save a reusable profile. The dialog warns when the provider’s API key is missing. Start Worker — choose a profile and launch an instance. Progress runs provisioning → running → attached. Attach stays disabled until the worker answers a health probe. Instance rows — each live or paused worker shows its status, uptime, and estimated cost, with Attach, Detach, Resume, Stop (pause), and Terminate (destroy) actions. Reconcile and Stop All — Reconcile runs the orphan check on demand. Stop All pauses every listed worker. Status-bar indicator — when a worker is attached, a badge shows it and offers Stop attached worker, which pauses it. Attaching re-points NodeTool’s Python bridge at the worker’s wss:// URL and bearer token without a restart; detaching reverts to the local stdio worker. The active worker is a single DB pointer (active_worker_instance_id in the settings table) that any NodeTool instance reads — which is why a self-hosted Docker server can adopt a worker by the same mechanism. Cost guard GPU pods and Vast instances bill continuously — there’s no scale-to-zero in pod mode. The laptop sleeps, the app crashes, a tab gets forgotten. Two operations release a worker, and they differ in what keeps billing: Operation Effect Still billing Stop (pause) Releases the GPU, keeps the volume and cached models. The instance is marked stopped and can be resumed. The volume, at a lower rate Terminate Destroys the worker and its volume. The instance is marked terminated. Nothing worker stop, Stop, Stop All, and the status-bar quick-stop all pause. Only Terminate and the hard TTL destroy. Four mechanisms keep a stray worker from quietly billing for days: Guard What it does Terminate Issues the provider’s real delete/destroy for the worker and its volume, never just a status flip. Available in the Workers panel. Idle auto-pause The reaper pauses an instance after its profile’s idle_timeout_minutes of bridge inactivity (the bridge tracks last_activity_at). Attaching or resuming resets the clock. Hard TTL Optional max_lifetime_minutes. The reaper terminates an instance older than this regardless of activity. If both limits fire, the TTL wins. Orphan reconcile At server startup and on Reconcile, NodeTool diffs the DB’s running and attached instances against the provider’s live list. Workers killed out-of-band are marked stopped. Running provider workers the DB doesn’t track are reported as orphans with a live count and estimated cost. The panel banner tells you to stop them in your provider console, because NodeTool holds no record to act on. The reaper runs inside the NodeTool server and checks every 60 seconds. If you provision from the CLI with no server running, nothing enforces the idle timeout or TTL, so terminate by hand. A failed database write during provision also terminates the new worker, so no untracked resource is left billing. Set both --idle-timeout and --max-lifetime on every profile you provision from. A profile that sets neither opts its instances out of the reaper entirely — do that only deliberately. How provisioning works WorkerManager.provision(profileName) looks up the profile and resolves the target’s provider (RunPod, Vast or Verda). If the profile’s token_policy is generate, it mints a high-entropy bearer token for the worker. The provider launches the image on the chosen GPU/spec with persistent storage for the model cache (HF_HOME=/workspace/huggingface), polls until the box is running, and derives the WebSocket URL. A worker_instances row is written and transitioned provisioning → running. On attach, the manager writes the active-worker pointer, marks the instance attached, and hands the { wsUrl, token } to the bridge. All state is persisted through the DB — the manager never holds instance state only in memory, so a forgotten pod is always recoverable from the registry. Related Self-Hosted Deployment — run the NodeTool server on your own infra with Docker. Deployment Guide — overview of server self-hosting and GPU workers. CLI Reference — full nodetool command reference. </content> </invoke>--- # Workflow API Guide Source: https://docs.nodetool.ai/workflow-api.html NodeTool exposes workflow REST endpoints under /api/workflows from a single server (nodetool serve): /api/workflows for CRUD and query operations, and POST /api/workflows/{id}/run to run a workflow. This page collects the basics from the project README. See API Reference for the canonical endpoint list and auth requirements. Send Authorization: Bearer <token> on every request except the public routes noted below. A server in Local mode trusts loopback callers without a token, and a server in Supabase mode requires one for everything else. See Authentication. Loading Workflows const response = await fetch("http://localhost:7777/api/workflows/"); const { workflows, next } = await response.json(); GET /api/workflows lists the caller’s workflows, full graphs included. Query parameters: Parameter Effect limit Page size. Defaults to 100 and caps at 500 cursor The next value from the previous page run_mode Only workflows saved with this run mode project_id Only workflows in this project GET /api/workflows/{id} returns one workflow. The caller must own it, hold a collaborator grant, or find it marked access: "public". Anything else is a 404 with { "detail": "Workflow not found" }. Creating, Updating, and Deleting POST /api/workflows creates a workflow and returns it. name and a graph with nodes and edges arrays are required, and a missing one is a 400. curl -X POST "http://localhost:7777/api/workflows" \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "Greeting", "description": "Says hi", "graph": { "nodes": [], "edges": [] }, "access": "private" }' Optional body fields are description, tags, tool_name, package_name, path, thumbnail, thumbnail_url, access ("public" or anything else for private), settings, run_mode (default "workflow"), workspace_id, project_id (default "default"), html_app, and app_doc. A project_id the caller does not own is refused. To start from a shipped example, add ?from_example_name=<name> and optionally &from_example_package=<package> (default nodetool-base). When the body carries no graph nodes, the example’s graph and app UI are copied in. PUT /api/workflows/{id} replaces the workflow’s name, description, tags, tool_name, package_name, and graph. It takes the same body, and name and graph are required again. Other fields change only when the body includes them. The call creates the workflow under that id when none exists. An owner or an editor collaborator may update, and only the owner changes access and project_id. Saves are optimistic. Send expected_updated_at with the updated_at you last read, and a stale value is a 409. An expected_updated_at for a workflow that does not exist is a 404. DELETE /api/workflows/{id} removes a workflow the caller owns, along with its collaborator and share rows, and answers 204 with no body. It is a 404 for a workflow the caller does not own. Workflow responses carry the stored fields plus an etag. Running a Workflow HTTP API curl -X POST "http://localhost:7777/api/workflows/<workflow_id>/run" \ -H "Authorization: Bearer YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "params": { "param_name": "param_value" } }' const response = await fetch( "http://localhost:7777/api/workflows/<workflow_id>/run", { method: "POST", headers: { "Content-Type": "application/json", "Authorization": "Bearer YOUR_TOKEN", }, body: JSON.stringify({ params: params, }), } ); const body = await response.json(); // body has the shape: // { // "job_id": "<uuid>", // "workflow_id": "<uuid>", // "status": "completed" | "cancelled" | "failed", // "outputs": { /* one property per output node, keyed by node name */ }, // "error": null, // "message_count": 42, // "background": false // } // outputs values can be a string, image, audio, etc. POST /api/workflows/{id}/run runs the workflow to completion and returns a single JSON response — it does not stream. For real-time progress (job and node updates, incremental output), run the workflow over the WebSocket endpoint instead. The body is optional. Its fields: Field Effect params Input values keyed by input node name background Return a receipt at once and keep running. The response has status: "running", background: true, job_id, id, and a poll hint. Read the settled job and its outputs through the jobs tRPC procedures interactive Park on a failed node and hand the decision to the caller. See Interactive Runs max_decisions, max_retries_per_node, decision_timeout_ms Bounds for an interactive run project_id Must match the workflow’s project, or the call is a 400 node_ids Run only these nodes and what they need upstream. See Running Part of a Workflow reuse_results With node_ids: false runs every upstream node again instead of reusing saved generations. Default true POST /api/workflows/{id}/debug takes the same body. It returns a debug report instead of the run summary: job_id, workflow_id, status, outputs, error, a per-node summary (status, errors, logs, edges, LLM calls), and a verdict with a headline. Use it when a caller needs to know what happened inside the run. Failures before the run starts: Status Meaning 404 The caller has no workflow with that id 400 The workflow’s run_mode is not "workflow", a provider the graph needs has no key (the detail names the secret), project_id names another project, node_ids is not a non-empty list of strings or names a node the graph does not have, or reuse_results is not a boolean 429 An interactive run was refused because the caller already holds the maximum of 8 live debug sessions Running Part of a Workflow node_ids is the API form of the editor’s Run Node and Run selected. The run executes the named nodes and the upstream nodes they need. Unrelated branches and downstream nodes do not run and are not billed. An upstream node that saves its generations, such as an image, video, audio or LLM generator, does not run again when the workflow already has a generation of it. Its saved output is fed to the node that consumes it. The node’s pinned generation (selected_generation) is used when the editor set one, otherwise the newest. The named nodes themselves always run. Runs from the editor and from this endpoint save these generations as assets. The response carries partial_run: { "node_ids": ["caption"], "ran": ["caption"], "reused": [ { "node_id": "image", "job_id": "<job that made it>", "asset_ids": ["<asset>"] } ] } Agents pass the same node_ids and reuse_results to the run_workflow, debug_workflow and start_background_job tools. A node that fails during the run does not change the HTTP status. The response is 200 with status: "failed" and an error string. Listing Names and Tools Two lightweight GET routes answer “what workflows exist?” without paying for full graphs. Both read the caller’s own library, so both stay behind auth. GET /api/workflows/names returns an id → name object for up to 1000 of the caller’s workflows. Use it to label a workflow id you already hold — a job record, a saved reference — without fetching the workflow. curl "http://localhost:7777/api/workflows/names" \ -H "Authorization: Bearer YOUR_TOKEN" { "21fddc0c2c46458493287b151b790cc4": "Greeting" } GET /api/workflows/tools returns only the workflows saved with run_mode: "tool" — the ones an agent may call as a tool — reduced to what a tool picker needs. limit defaults to 100 and is capped at 500. curl "http://localhost:7777/api/workflows/tools?limit=50" \ -H "Authorization: Bearer YOUR_TOKEN" { "workflows": [ { "name": "Summarize", "tool_name": "summarize", "description": "Summarize a block of text" } ], "next": null } A workflow with a tool_name but no run_mode: "tool" does not appear here. Exporting a Workflow as DSL GET /api/workflows/{id}/dsl-export returns the workflow’s graph as TypeScript DSL source with content-type: text/plain; charset=utf-8 — the same source nodetool workflows export-dsl writes. Use it to put a workflow under version control, or to hand an agent an editable form of the graph. curl "http://localhost:7777/api/workflows/<workflow_id>/dsl-export" \ -H "Authorization: Bearer YOUR_TOKEN" \ -o workflow.ts import { constant, workflow } from "@nodetool-ai/dsl"; // 1 — nodetool.constant.String const string = constant.string({ value: "hi" }); export const greetingWorkflow = workflow(string); The route answers 404 unless the workflow is yours or marked access: "public", and 400 when the workflow has no graph or the graph cannot be expressed as DSL. Public Workflows A workflow saved with access: "public" is readable without a token. These two routes and the example routes below are the only /api/workflows paths exempt from auth — every other one serves the caller’s private library, graph included, so it stays behind a token. # Every public workflow (limit defaults to 100, caps at 500) curl "http://localhost:7777/api/workflows/public" # One public workflow, full graph included curl "http://localhost:7777/api/workflows/public/<workflow_id>" Both return the normal workflow shape (id, name, description, graph, access, …). Asking for a workflow that exists but is not public gets the same 404 as one that does not exist: { "detail": "Workflow not found" } Example Templates The example workflows NodeTool ships are served from disk rather than the database, so they need no token and exist on a fresh install. curl "http://localhost:7777/api/workflows/examples" curl "http://localhost:7777/api/workflows/examples/search?query=chat" search filters the same list on query against each example’s name, description, and tags, case-insensitively; omitting query returns everything. Both responses carry metadata only — graph comes back empty: { "workflows": [ { "id": "A Boolean Constant.json", "name": "A Boolean Constant", "description": "The smallest possible graph, and a real one: …", "tags": ["example"], "package_name": "nodetool-base", "thumbnail": "A Boolean Constant.jpg", "thumbnail_url": "/api/workflows/examples/thumbnails/A%20Boolean%20Constant.jpg?v=a6dce6b4", "graph": { "nodes": [], "edges": [] } } ], "next": null } Fetch the image at the thumbnail_url the list hands back: curl "http://localhost:7777/api/workflows/examples/thumbnails/A%20Boolean%20Constant.jpg" \ -o thumb.jpg Only .jpg and .png are served — any other extension is a 400 — and the filename is reduced to its basename, so it cannot escape the examples assets directory. WebSocket API For real-time streaming and job control over WebSocket, see the dedicated WebSocket API page. The reference client in examples/workflow_runner/js/workflow-runner.js shows how to consume job_update, node_update, and node_progress messages. API Demo Grab the example runner (examples/workflow_runner). Open index.html in a browser locally. Select the endpoint (local or api.nodetool.ai for alpha users). Enter an API token from the NodeTool settings dialog. Select a workflow and run it.--- # Workflow Debugging Guide Source: https://docs.nodetool.ai/workflow-debugging.html This guide teaches you how to debug NodeTool workflows when they don’t work as expected. You’ll learn to inspect intermediate data, read error logs, and systematically isolate problems. Quick Debugging Checklist When a workflow isn’t working: Look for a moving ring (running), a Completed in badge, or a Failed in badge on each node Read the red error panel on failed nodes Add Preview nodes after suspicious nodes Verify all required inputs are connected Check data types match between connections Review the Logs panel for detailed errors Test problem nodes in isolation with Run Node Run nodetool validate on the graph Understanding Node Status Nodes show their execution status on the canvas: Sign Status Meaning Moving colored ring around the node Running Node is currently executing. A progress bar appears when the node reports progress. Completed in 1.2s badge above the node Completed Node finished successfully Failed in 1.2s badge above the node, and a red error panel Failed Node encountered an error Model is booting, taking minutes. Booting A model is starting up. This is not a failure. When a node fails, the error text appears on the node. The panel has View logs (opens the [Logs panel]({{ ‘/editor-panels’ relative_url }}#logs) filtered to that node and run), Report (starts a bug report), and a copy button. Select the node and open the Inspector (i) to see validation issues for its properties before you run. Using Preview Nodes Preview nodes are your primary debugging tool. They show exactly what data is flowing through your workflow at any point. Adding Preview Nodes Press Space to open the node search Type “Preview” and select the Preview node Connect the output you want to inspect to the Preview node Run the workflow A faster route: drop a connection on empty canvas, or right-click an output port, and choose Preview in the menu. It creates a Preview node already connected. Preview Node Strategies After every major transformation: Input → Process → Preview(1) → Transform → Preview(2) → LLM → Preview(3) → Output Before and after suspicious nodes: ... → Preview(before) → SuspiciousNode → Preview(after) → ... At branch points: → Preview(A) → ProcessA Input → → Preview(B) → ProcessB What Preview Shows Preview nodes render the value by type: Data Type Preview Display Text/String Text content, scrollable Image Rendered image Audio Playable audio widget Video Video player List The items, rendered per type Object/Dict Formatted JSON DataFrame Table Number, Boolean The value HTML, 3D model, sketch, timeline, chart A rendered view of each Inspecting Workflow JSON Every NodeTool workflow is stored as JSON. Inspecting this JSON can help debug complex issues. Exporting Workflow JSON Open your workflow in the editor Open the composer’s ⋮ menu and choose Download JSON. The command menu (Ctrl/⌘ + K) has Download Workflow as JSON and Copy Workflow as JSON. Save the .json file Open in any text editor or JSON viewer Workflow JSON Structure The file is the workflow with its graph under graph. Download JSON blanks the id. This trimmed example leaves out several fields, such as access, tags, and settings: { "id": "", "name": "My Workflow", "description": "...", "graph": { "nodes": [ { "id": "node_1", "type": "nodetool.input.StringInput", "data": { "name": "text", "value": "Hello world" }, "ui_properties": { "position": { "x": 100, "y": 100 }, "width": 280 } }, { "id": "node_2", "type": "nodetool.agents.Agent", "data": { "prompt": "..." }, "ui_properties": { "position": { "x": 400, "y": 100 }, "width": 280 } } ], "edges": [ { "id": "edge_1", "source": "node_1", "sourceHandle": "output", "target": "node_2", "targetHandle": "prompt", "edge_type": "data" } ] } } Properties fed by a connection are left out of a node’s data, because the edge supplies the value. ui_properties also holds the title, color, size, and a bypassed flag for disabled nodes. What to Look For Missing connections: Check that edges correctly connect outputs to inputs Incorrect values: Look at data for each node to verify settings. A node with "bypassed": true in ui_properties is disabled. Node types: Ensure type matches expected node (typos happen in programmatic workflows) Position issues: If nodes overlap or are off-canvas, check ui_properties.position Reading Error Logs In the Editor Open the [Logs panel]({{ ‘/editor-panels’ relative_url }}#logs) with l. It lists log lines for the open workflow, newest first. Filter by Info, Warn, or Error. The [Trace panel]({{ ‘/editor-panels’ relative_url }}#trace) (Ctrl/⌘ + Shift + T) shows per-node timing for a run. The Trace panel also has an Ask the agent button. It opens a chat tab titled “Inspect run” with the run id attached, and the span id too when a span is selected, and drafts “Help me diagnose the selected run.” (or “Help me diagnose this span in the selected run.”) for you to send. Desktop App Logs Open Tools → Log Viewer to read the backend log in its own window. The desktop app writes the log file here: Windows: %LOCALAPPDATA%\nodetool\logs\nodetool.log macOS and Linux: ~/.local/share/nodetool/logs/nodetool.log For renderer errors, open the developer tools from the View menu. CLI/Server Logs When running NodeTool from the command line, serve accepts only --host and --port. The log level comes from the environment (NODETOOL_LOG_LEVEL, falling back to LOG_LEVEL, default info). Valid levels are debug, info, warn, and error: # Debug logging NODETOOL_LOG_LEVEL=debug nodetool serve # Write the log to a file instead of stderr NODETOOL_LOG_FILE=/tmp/nodetool.log nodetool serve Understanding Error Messages Messages you are likely to meet: Select a model → An Agent node has no model chosen. Pick one in the Inspector. COHERE_API_KEY is not configured → A provider key is missing. The name in the message is the key to set. Add it in Settings → Models & Providers. Missing required workflow input "prompt" → Give the input node a value, or pass the input when you run the workflow. Cycle detected in graph; a cycle may only close on the "next" or "condition" input of a Loop node. → Remove the connection that loops back. Only a Loop node’s next and condition inputs may close a cycle. Python node "<type>" cannot execute: Python worker is not connected. → The node needs the Python bridge. Check that Python is installed and the worker started. See [Troubleshooting]({{ ‘/troubleshooting’ | relative_url }}). The editor refuses a connection between incompatible types, so a type mismatch shows up as a wire that will not connect, not as a run error. Add a conversion node between the two. Debugging Specific Issues LLM Not Responding Check model availability Open More → Model Manager on the left rail Verify model is installed (local) or API key is set (cloud) Test with Preview Add Preview after the Agent/LLM node Run and check if any output appears Check prompt Is the prompt template valid? Are all template variables being filled? Verify connectivity For cloud models: check internet connection For local models: check if Ollama/llama.cpp is running Image Generation Fails Check model installation Open More → Model Manager Ensure Flux/Qwen Image/etc. is downloaded Verify VRAM Check GPU memory with nvidia-smi Close other GPU apps if memory is low Inspect parameters Valid dimensions? (usually multiples of 8 or 64) Reasonable step count? (20-50 typical) Valid CFG scale? (5-15 typical) Add Preview before generation Confirm prompt text is correct Check that conditioning inputs are valid RAG/Search Returns Nothing Verify collection exists Check that index workflow was run first Collection name matches between index and search Check embedding model Same embedding model for index and search? Model available and running? Test query Try a simpler query Increase top_k to retrieve more documents Check if documents were chunked appropriately Inspect indexed content Use Preview to see what was indexed Check chunk sizes aren’t too small/large Workflow Runs Forever Check for loops The validator rejects cycles except one closing on a Loop node’s next or condition input, but a Loop with a condition that never ends still runs forever Look for conditional nodes that might never exit Monitor node status Which node still shows the moving running ring? That node is the bottleneck Check network calls API timeouts can appear as hangs Add timeout parameters where available Resource constraints Is the system running out of memory? Is disk full (for large file operations)? Isolating Problems Binary Search Debugging When you have a complex workflow and something’s wrong: Disable half the workflow Select the downstream nodes and press B (or right-click → Disable All) A disabled node and its connections are left out of the run, so only the first half executes Check results Working? Problem is in second half Broken? Problem is in first half Repeat Continue halving until you find the problem node Testing Nodes in Isolation Create a minimal test workflow: Right-click the node → Run Node. It runs as its own job using previous results as inputs. Run Selected does the same for a selection. Or make a new workflow with just the suspicious node, add input nodes with known good test data, add Preview/Output, and run it Or, from a terminal, nodetool node run <node_type> --props '{...}' runs one node with no workflow If the node works in isolation, the problem is with the data it receives in the full workflow. Comparing Working vs Broken If a workflow used to work: Export both versions (working and broken) as JSON, or compare two entries in the Versions panel Diff the files to see what changed Focus on changed nodes/edges diff working_workflow.json broken_workflow.json Command-Line Debugging Three nodetool commands cover the same ground without the editor. See the [CLI reference]({{ ‘/cli’ relative_url }}) for all options. Command Use nodetool validate <workflow_id_or_file> Static check in under a second: unknown node types, missing required properties, dangling or mis-typed edges, bad model references. Run it before an expensive run. nodetool debug <workflow_id_or_file> Run the workflow and write a bundle with every message, log line, node input and output, and error, plus a verdict. Add --trace for timing and cost, --watch to re-run a file on save. nodetool node run <node_type> --props '{...}' Run one node in isolation. Debugging Tools Summary Tool When to Use How to Access Preview nodes See intermediate data Space → search “Preview” Node error panel See error messages On the failed node Logs panel Run log lines by severity l Trace panel Per-node timing Ctrl/⌘ + Shift + T Run Node / Run Selected Test part of a graph Right-click a node or selection Disable Node Exclude part of a graph from a run B Versions panel Compare or restore earlier saves Bottom panel → Versions JSON export Inspect workflow structure ⋮ menu → Download JSON Log Viewer Backend log (desktop) Tools → Log Viewer Debug logging CLI debugging NODETOOL_LOG_LEVEL=debug nodetool serve nodetool validate / debug Headless checks Terminal Common Fixes Problem Solution Wire will not connect Types are incompatible. Add a conversion node between them “Select a model” Choose a model on the Agent node, or set a default in Settings → Default Models “… is not configured” Add the named key in Settings → Models & Providers Model not available Install it in More → Model Manager or configure its API key Empty output Add Preview to find where data is lost Wrong format Use the Format Text node or a conversion node Large run warning Review the cost, or change Large-Run Threshold in Settings → General → Execution Getting More Help If you’re still stuck after debugging: Click Report on the failed node. The bug-report form collects the error, the node’s settings, the workflow and the logs for you. The hosted app sends it to the NodeTool team. A local install saves a zip you drag into a pre-filled GitHub issue. See Report a Bug from inside NodeTool. Ask on Discord if you would rather talk it through first. Bring the workflow JSON, a screenshot, and your OS and NodeTool version. Related Documentation Troubleshooting Guide – Broader troubleshooting for common issues Key Concepts – Understanding nodes, edges, and data flow Workflow Editor – Using the visual editor effectively Cookbook – Working workflow patterns to learn from--- # Workflow Editor Source: https://docs.nodetool.ai/workflow-editor.html The canvas: place nodes, connect ports, run, debug. Covers basic navigation through disabling nodes and auto layout. New here? Start with Getting Started, then come back. For panel-by-panel detail, see Editor Panels. Where this fits The workflow editor is NodeTool’s automation layer. A workflow reads assets, calls AI models, and generates new assets. These new assets can then be: Sent to the Sketch Editor or Video Editor for manual adjustments. Packaged as a Mini-App for non-technical users. Everything connects back to the same shared asset store and AI models. See Key Concepts → How everything fits together for the full loop. Editor Layout Area Where What It Does Canvas Center Place and connect nodes Left panel Left Documents, chats, library, and nodes, plus a More panel with favorites, recent nodes, workflow settings, and app pages Inspector Right Properties, I/O, and help for the selected node Bottom panel Bottom Logs, queue, workers, versions, and trace Composer Floating over the canvas Chat and media prompt, add node, auto-layout, save, run Canvas Basics Your infinite workspace. Navigate: Do This How Pan Left-drag on empty canvas (default on Windows and Linux). On macOS, two-finger scroll, or right or middle-button drag Zoom Scroll wheel, or pinch. On macOS, pinch Fit everything F Reset zoom (to 50%) Ctrl/⌘ + 0 Zoom in or out Ctrl/⌘ + = / Ctrl/⌘ + - Zoom to 50%, 100%, 200% Ctrl/⌘ + Alt + 5 / 1 / 2 Left-Click Drag in Settings → General → Canvas swaps the mouse roles. With Select nodes (box), left-drag draws a selection box and panning moves to right or middle-button drag. The default is Pan canvas on Windows and Linux and Select nodes (box) on macOS. Node Selection Mode decides whether a box must enclose nodes (Full) or only touch them (Partial, the default). The grid helps align nodes. Turn on Snap to Grid in the View menu of the desktop app, from the command menu (Ctrl/⌘ + K), or in Settings → General → Canvas. The grid size is the Grid Snap Precision setting, and Connection Snap Range sets how close a dragged connection must get to a port to snap. The ⋮ menu on the composer has a Mini Map toggle. Working with Nodes Each node does one thing. Add Nodes Space bar: Press Space anywhere Type what you want (“image”, “text”) Click to add Double-click: Double-click empty space Opens node menu Smart connect: Drag from a node’s output Drop on empty space Pick from the compatible nodes, or choose Preview, Reroute, or a Save node for that type Node Structure Header (top) - Name, drag to move Inputs (left circles) - Data in Outputs (right circles) - Data out Properties - Settings panel Select Nodes Do This How One Click it Multiple Ctrl/⌘ + click, or draw a selection box (Shift + drag, or plain drag when Left-Click Drag is set to select) All Ctrl/⌘ + A Connected nodes Shift + C (inputs and outputs), Shift + I (inputs), Shift + O (outputs) None Click canvas Move Nodes Drag header to move Arrow keys to nudge by 10 px Auto Layout button to organize C collapses or expands the selected nodes Disable Nodes Exclude nodes from a run without deleting them: Right-click the node (or select it and press B) Select Disable Node The node dims A disabled node and every connection to or from it are excluded from the submitted graph. Data does not pass through the node. Good for: Testing - Run a smaller part of the graph Debugging - Exclude a failing or costly branch Editing - Keep unfinished nodes on the canvas without executing them Re-enable: right-click → Enable Node, or press B again. With several nodes selected, the context menu offers Disable All or Enable All. Connections Connections are the lines between nodes that show how data flows through your workflow. Data flows left to right, from output ports (right side of a node) to input ports (left side of another node). Make Connections Click output circle (right side) Drag the line to an input circle (left side of another node) Release to connect Connection Rules Types must match: You can only connect compatible types (text to text, image to image) One input, multiple outputs: Each input accepts one connection; outputs can connect to many Color coding: Connection colors indicate data type Removing Connections Right-click a connection and choose Delete Edge, or Insert Reroute to add a reroute node at that point Drag the connection away from its target and release Smart Connections When you drag a connection and release on empty space, a connection menu appears: A search box over the nodes that can receive (or produce) that data type Shortcuts for Preview, Reroute, and a Save node matching the type Esc cancels Releasing on a node’s body instead connects to a matching input on that node. On a node that supports dynamic inputs, it creates a new input named after the source node. Lineage Links A lineage link records that one node relates to another, such as “this clip comes from this shot” or “this sketch is the reference for that frame”. It is a dotted line that carries no data and does not change the order nodes run in. A run gives the same result with or without it. Select exactly two nodes and click Link in the selection toolbar. The link runs from the node on the left to the node on the right. Click a link to select it, then press Delete or Backspace. You can also right-click it and choose Delete Link, or middle-click it. Deleting a node removes its links. Copy, paste and duplicate keep a link when both of its nodes are included. Agents add and remove links with the add_workflow_link and remove_workflow_link tools. get_workflow lists them under graph.links. Running Workflows Starting a Run Method How Button Click Run entire workflow in the bottom toolbar Keyboard Ctrl/⌘ + Enter Command menu Select Run Entire Workflow All three methods run the entire enabled graph. They use the same checks for missing models, search-provider setup, large runs, and additional concurrent runs. The large-run warning is controlled by Warn Before Large Runs and Large-Run Threshold in Settings → General → Execution. To run part of a graph, right-click a node and choose Run Node, or right-click a multi-node selection and choose Run Selected. Each runs as its own job and uses previous results as inputs. Watching Progress Streaming nodes show output as it’s generated Preview nodes display intermediate results A moving colored ring around a node marks it as running Completed in and Failed in badges above a node show how long it took Edge animations show data flowing between nodes The main run control shows Starting, Queued, Running, Stopping, or Error without requiring a tooltip Stopping a Run Method How Button Click Stop while a run is starting, queued, or running Keyboard Esc Organizing Your Workflow Auto Layout Click the Auto Layout button in the composer bar to arrange your nodes in a readable layout. The editor also auto-arranges nodes when Chat creates or modifies workflows. There is no keyboard shortcut for auto-layout. It is a toolbar button only. Grouping Nodes Select multiple nodes and press Ctrl/⌘ + G to group them. Groups keep related nodes together and move as a unit. The node context menu also offers Group into Subgraph, and the canvas context menu offers Add Group, Add Comment, Add Subgraph, Add Workflow, and Add App. An App node runs a mini app’s workflow: the app’s inputs become the node’s inputs, its outputs become the node’s outputs, and an output the app streams reaches downstream nodes one value at a time. Aligning Nodes Shortcut Action A Align selected nodes Shift + A Align and distribute evenly Shift + ←/→/↑/↓ Align left, right, top, or bottom edges Shift + H / Shift + E Align centers horizontally or vertically Shift + D Distribute horizontally V Stack selected nodes in one column G Arrange selected nodes in a grid Adding Nodes from the Keyboard Shortcut Node added at the cursor Shift + P Prompt Shift + G Text to Image Shift + V Text to Video Shift + L Agent Left Panel The left rail has Documents, Chats, Library, Nodes, and More. See Editor Panels → Left Panel for details on each. Right Panel (Inspector) Properties for the selected node, on the Params tab Input and output slots on the I/O tab, and node documentation on the Help tab Validation errors for the selected node The Inspector opens automatically when you select a node. With several nodes selected it edits their shared properties. Logs, Queue, Trace, and Version History live in the Bottom Panel. Finding Nodes The Node Menu Press Space to open, then: Search: Just start typing (“whisper”, “image”, “agent”) Browse: Explore the category tree on the left Filter: Use the input and output type chips next to the search box (All, Image, Text, Audio, Video, Number) to show only nodes that take or produce a type Move: Drag the menu header to reposition it, and drag its edges to resize it Close: Esc, Space on an empty search box, or click outside Find in Workflow Ctrl/⌘ + F searches the workflow you already have: node names, types, and property values. Matches highlight on the canvas and the view pans to the one you pick. Node Documentation Get help on any node: In the Node Menu: Hover over a node to see its description Inspector: Select a node and view full documentation in the right panel Context Menus Right-click for options anywhere: Location Options Canvas Paste, Fit Screen, your favorite nodes, Add Constant Node, Add Input Node, Add Comment, Add Group, Add Workflow, Add Subgraph, Add App Node Copy, Cut, Copy Node as JSON, Duplicate, Duplicate Vertical, Run Node, Disable or Enable Node, Collapse or Expand Node, Add Comment, Group into Subgraph, Convert to Input or Constant Node, Show Templates, select all nodes of the same type, Delete Node Selection Duplicate, Copy, Cut, Run Selected, Align, Arrange, Disable All, Collapse / Expand, Surround With Group, Group into Subgraph, Select All Connected, Select Inputs, Select Outputs, Delete Input or output port Searchable list of compatible nodes, plus Preview, Reroute, and Save shortcuts Connection Insert Reroute, Delete Edge Lineage link Delete Link Copy Node as JSON puts the node’s data — its property values, title, and dynamic slots — on the clipboard as formatted JSON. Paste it into a bug report, or hand it to an agent. Built-in Editors NodeTool includes professional editing tools for creative work. Sketch Editor Open a blank canvas (+ New → New sketch in the workspace tab bar) or edit an existing image to use the full layered editor: Layers: Control blend modes, opacity, and visibility. Painting: Brush, pencil, eraser, fill, gradient, blur, and clone stamp tools. Shapes & Transform: Add shapes (rectangle, ellipse, line, arrow), crop, or freely transform layers. AI Generation: Create a layer directly from a text prompt or connect it to a workflow. History: Unlimited undo and redo steps. Full guide: See Sketch Editor for complete documentation with tool reference, shortcuts, and workflows. Text and Code Editor Long text or code properties open in a full editor featuring syntax highlighting, line numbers, and search. This is the same editor used for code nodes. DataFrame Editor Edit tabular data just like a spreadsheet: add or remove rows and columns, edit cell values, and sort your data. Image Comparer Preview nodes with before/after images show a slider. Drag it left or right to compare the two images. Color Picker The color picker appears when selecting colors in properties: Visual Selection: Saturation/brightness picker with hue slider Multiple Formats: Enter values as HEX, RGB, HSL, or CMYK Harmony Modes: Complementary, triadic, analogous color suggestions Gradient Builder: Create and edit color gradients Swatches: Save and reuse favorite colors Contrast Checker: Verify accessibility compliance Eyedropper: Pick colors from anywhere on screen Keyboard Shortcuts Essential Shortcuts Shortcut Action Space Open node menu Ctrl/⌘ + Enter Run entire workflow Ctrl/⌘ + S Save Ctrl/⌘ + Z Undo F Fit view Esc Stop / Cancel All Editor Shortcuts Shortcut Action Ctrl/⌘ + C Copy Ctrl/⌘ + V Paste Ctrl/⌘ + X Cut Ctrl/⌘ + D Duplicate horizontally Ctrl/⌘ + Shift + D Duplicate vertically Ctrl/⌘ + G Group selection Ctrl/⌘ + 0 Reset zoom to 50% Ctrl/⌘ + = / Ctrl/⌘ + - Zoom in / out Ctrl/⌘ + Alt + 1 / 2 / 5 Zoom to 100% / 200% / 50% Ctrl/⌘ + 1-9 Switch to tab 1-9 Ctrl + PageUp / PageDown Previous / next tab Ctrl/⌘ + F Find in workflow Ctrl/⌘ + I Node info panel Ctrl/⌘ + / Show keyboard shortcuts Ctrl/⌘ + , Open Settings B Disable or enable selected nodes C Collapse or expand selected nodes Shift + C / Shift + I / Shift + O Select connected / input / output nodes Shift + ←/→/↑/↓ Align left, right, top, bottom Shift + H / Shift + E Align centers horizontally / vertically Shift + D Distribute horizontally V / G Stack / grid-arrange selected nodes Shift + P / G / V / L Add Prompt / Text to Image / Text to Video / Agent node at the cursor A Align selected nodes Shift + A Align and distribute Arrow keys Nudge selected nodes Delete / Backspace Delete selection i Toggle Inspector w Toggle Workflow Settings panel l Toggle Logs panel Ctrl + Shift + T Toggle Trace panel Alt-based keyboard navigation moves focus between nodes without the mouse: Ctrl/⌘ + Alt + N and P step to the next and previous node, Alt + arrows move focus to the nearest node in that direction, Enter selects the focused node, Esc leaves navigation mode, and Ctrl/⌘ + Alt + B goes back. Ctrl/⌘ + T (new workflow) and Ctrl/⌘ + W (close tab) work in the desktop app only. Tips Design Principles Left to right — Arrange nodes so data flows left to right across the canvas for readability Preview often — Add Preview nodes after each major step to inspect intermediate results Name clearly — Rename nodes (double-click the header) to describe their purpose, e.g., “Resize to 512px” instead of “Resize” Group logically — Keep related nodes together and use Groups (Ctrl/⌘ + G) to visually organize complex workflows Debugging Add Preview nodes between steps to see exactly what data each node produces Check connections — verify data types match (connection colors indicate type) Look at node badges — a Failed in badge and an error panel mark a failed node, and a Completed in badge marks a finished one Test incrementally — disable downstream nodes and run smaller graphs to isolate problems Use the Inspector — press i to see validation warnings for the selected node. Run errors appear on the node itself Performance Local models — slower but work offline and are free to use Cloud models — faster response times, require internet and API keys Streaming nodes — show progress during long-running operations (look for the streaming indicator) Parallel branches — NodeTool automatically runs independent branches in parallel for faster execution To give someone else access to a workflow, see Sharing Workflows. Next Steps Cookbook – Workflow patterns and best practices Workflow Examples – Ready-to-use workflows Tips & Tricks – Power user features Node Reference – All available nodes--- # Workflow Graph View Source: https://docs.nodetool.ai/workflow-graph-view.html The Workflow Graph View is a read-only rendering of a workflow. Use it for visual snapshots, screenshots for documentation, and showing a graph without handing over the editor. Opening the View Two pages render the graph, and both take the same inline-data option. Page Source Use /graph/:workflowId in the app Fetches the workflow with the signed-in session Look at a stored workflow graph.html (the standalone viewer) ?data=<base64 JSON> only, no backend Screenshots and static embeds Unlike the [Workflow Editor]({{ ‘/workflow-editor’ relative_url }}), the graph view: Does not load the Node Menu, Inspector, or panel drawers. Does not allow editing. Nodes can’t be added, moved, connected, selected, or deleted. Lays the graph out automatically instead of using the saved node positions. There is no link to it in the editor. Build the URL yourself. Interactions The view is a snapshot. Panning, scroll-zoom, pinch-zoom, and double-click-zoom are disabled. The view fits the whole graph to the viewport once when it loads. Running the workflow is not possible here. Open it in the editor first. Query parameters: Parameter Default Effect data none Base64-encoded workflow JSON. Renders it instead of fetching a stored workflow. Accepts a full workflow (graph.nodes and graph.edges) or { nodes, edges } at the top level. bg #1a1a2e Background color (in-app route only) padding 60 Fit-view padding, as a percentage (in-app route only) For /graph/json, data is required. Without a workflow ID or data, the page shows an error. Once the graph is laid out and fitted, the container gets data-ready="true" (and data-ready="error" on failure), so headless Chrome can wait on [data-ready="true"]. The in-app route also sets data-workflow-name. The standalone viewer uses a fixed 15% fit padding. To screenshot a workflow JSON file, run npx tsx scripts/screenshot-workflow.ts <workflow.json> [output.png] from web/. It accepts --width, --height, --bg, and --port. Access Control /graph/:workflowId loads the workflow through the same authenticated workflows.get call the editor uses. You need at least viewer access to the workflow. On a local install, loopback requests are trusted, so any workflow on your own machine opens. Inline ?data= payloads need no access check because nothing is fetched. Next Steps [Workflow Editor]({{ ‘/workflow-editor’ relative_url }}) — the full editor where you author workflows [Chain Editor]({{ ‘/chain-editor’ relative_url }}) — the linear alternative [Authentication]({{ ‘/authentication’ relative_url }}) — how access is enforced--- # Sharing Workflows Source: https://docs.nodetool.ai/workflow-sharing You share a workflow by creating a link and sending it to someone. The link carries a permission level. When the other person opens it, they get that level of access to your workflow. Only the owner of a workflow can create links or change who has access. Sharing depends on user accounts, so it matters which authentication mode your server runs in. Which deployments support sharing Mode Sharing Supabase mode (SUPABASE_URL and SUPABASE_KEY set) Works. Each person signs in with their own account, and a link gives that account access. Local mode, one person on loopback Not useful. Loopback connections all run as the same user ("1"), so you are always the owner of your own links. Opening a share link requires being signed in to the server. The one exception is a public link, which anyone can open to view the workflow without an account. See Authentication and Supabase Deployment for setting up accounts. Permission levels There are three kinds of link. Link Who can open it What it grants Can view Anyone signed in to the server Viewer access to the workflow Can edit Anyone signed in to the server Editor access to the workflow, including version history Public view Anyone, without an account A read-only page showing the workflow. Grants no access to your account or workflows A viewer can open the workflow. An editor can also save changes and see its version history. Only the owner can open the share dialog, create or revoke links, change roles, and remove people. Create a link Open the workflow in the editor. Open the command menu and choose Share Workflow…. In the Share “workflow name” dialog, click Copy view link, Copy edit link, or Copy public link. The link is copied to your clipboard. If the browser blocks clipboard access, NodeTool tells you the link was created, and you can copy it from the Active links list in the dialog. Each role has one active link per workflow. Clicking the same button again returns the existing link instead of making a new one. The links look like this: Link type Path Can view, Can edit /share/<token> Public view /view/<token> Revoke a link Every active link appears under Active links with a role label, a copy button, and Revoke. Revoking a link stops anyone from using it again. For a view or edit link, people who already joined keep their access and stay listed under People with access. To take their access away, remove them as described below. For a public link, revoking stops the public page and copying. People with access After someone opens a view or edit link, they appear under People with access. Each row shows their user ID, a role selector, and a remove button. Change the selector between Can view and Can edit to change their role. Click the remove button (tooltip Remove access) to take their access away. Until someone redeems a link, the list shows No collaborators yet. If you open your own link, nothing changes, because you are already the owner. A collaborator can also remove themselves from a workflow they were given access to. Accept a share link Opening a view or edit link takes you to a page that says “Opening shared workflow…”. NodeTool adds you as a collaborator with the link’s role and sends you to the workflow in the editor. If the link is invalid or revoked, you see “This share link is invalid or has been revoked. Ask the workflow owner for a new one.” with a Go to editor button. Public links cannot be accepted this way, since they do not grant access. Public links A public link opens a read-only page showing the workflow’s graph, name, and description. It needs no account and shows nothing about the owner. A signed-in visitor sees Duplicate to my workflows, which copies the workflow into their own account as a new private workflow. A visitor who is not signed in sees Sign in to duplicate. The original stays unchanged. If the link has been revoked, the page shows “This workflow is not available”. Shared with me The server keeps a list of every workflow shared with an account, with the role on each. Workflows whose owner deleted them are left out. This list is available through the API as workflows.sharing.sharedWithMe. The web app does not include a “Shared with me” section in the workflow list. Open a shared workflow again from its original link. Related Workflow Editor Authentication--- # Workflow Supervisor — Technical Design Source: https://docs.nodetool.ai/workflow-supervisor-design.html Workflow Supervisor — Technical Design Author: Matti Georgi Status: Draft — for review Last updated: 2026-08-01 PRD: workflow-supervisor-prd.md Implementation plan: workflow-supervisor-implementation-plan.md 1. Summary One primitive: a failing node invocation raises an escalation; a handler returns a verdict — retry, substitute, skip, or fail. The hook lives in NodeActor’s catch path and defaults to fail, so a runner without a supervisor behaves exactly as today. The supervisor itself is not new machinery: it is a StepExecutor with a verdict output schema and two read-only tools. The kernel stays the executor. The graph stays the plan. The agent supplies judgment only at the moment a step breaks, which is what makes a supervised workflow “run like an agent” without giving up deterministic steps. 2. Design goals Zero cost on clean runs. No supervisor code executes unless an invocation throws after its own error handling is exhausted. Fail-closed everywhere. Any failure of the supervisor (timeout, invalid verdict, provider down) resolves as fail — today’s behavior. One hook, no scheduler changes. The correlation system, streaming actors, control edges, and EOS semantics are untouched. The supervisor cannot rewire the graph. Interventions are data. Every escalation and verdict is a ProcessingMessage and lands in RunResult, so audit, replay, and UI are consumers of one record, not three features. 3. Architecture NodeActor (packages/kernel/src/actor.ts) process() throws └─ ctx.escalate({nodeId, invocationKey, detail, inputs}) ← the only new kernel seam │ (default handler: () => ({action: "fail"})) ▼ SupervisorHandle (packages/kernel/src/supervisor.ts) serialized queue · sticky-verdict cache · bounds · timeout │ one escalation at a time ▼ SupervisorAgent (packages/agents/src/supervisor/) = StepExecutor + Verdict outputSchema + {get_run_state, read_node_output} │ finish_step(verdict) ▼ NodeActor applies the verdict (after checking it against the kernel-computed allowed set): retry → call process() again, identical inputs and properties substitute → runtime value validation, then emit skip → lineage_done + scope closure per slot, key pruned downstream end_stream → keep emitted values, close output slots fail → today's error path The kernel package defines the SupervisorHandle interface and ships a no-op default; the LLM-backed implementation lives in @nodetool-ai/agents. The dependency arrow stays kernel ← agents, unchanged. 4. Types In @nodetool-ai/protocol (they cross the kernel/agents boundary and appear in messages): export type VerdictAction = "retry" | "substitute" | "skip" | "end_stream" | "fail"; export interface Escalation { nodeId: string; correlationLineage: string[]; // full lineage chain of the failing invocation invocationKey: string; // leaf token of correlationLineage; "" for single-fire nodes allowedActions: VerdictAction[]; // kernel-computed (§5.3/§5.4); schema and enforcement both derive from it nodeType: string; detail: string; // the thrown error, stringified failureSignature?: string; // stable categorical code; absent ⇒ stickiness disabled candidateOutput?: unknown; // from RecoverableNodeError — the malformed value, redacted+truncated inputs: Record<string, unknown>; // the invocation's input values, redacted+truncated declaredOutputs: Record<string, string>; // the node's declared output types, per slot attempt: number; // 1-based, per (nodeId, invocationKey) spentCostUsd: number; // provider cost recorded by this invocation createdAssets: boolean; retrySafe: boolean; // node metadata opt-in; unknown ⇒ false ⇒ no retry emitted: boolean; // streaming: the invocation already sent something downstream (§5.4) } export type Verdict = | { action: "retry" } // no property overrides in v1 | { action: "substitute"; outputs: Record<string, unknown> } | { action: "skip"; applyTo?: "invocation" | "signature" } // default "invocation" | { action: "end_stream" } // streaming only: keep emitted, close slots | { action: "fail"; reason?: string; applyTo?: "invocation" | "signature" }; retry deliberately carries no property overrides. A free-form Record<string, unknown> would let the supervisor rewrite URLs, prompts, code, or paths — a nominally read-only HTTP node could be redirected to exfiltrate its inputs — and it wouldn’t even work reliably: updateNodeProperties mutates the executor, but the next invocation reassigns edge-driven inputs over it. If re-parameterized retry earns its way back, it will be via explicitly declared, typed retryMutable properties applied to a local invocation copy (§10); v1 retries the invocation exactly as it was. end_stream is a first-class verdict, not a skip alias: the actor must distinguish “keep what was emitted and close the slots” from “drop this invocation’s outputs,” and one action cannot encode both. In the product surface it is presented under Skip — “skipped the rest of this step, kept what it had produced” — so the PRD’s four user verbs hold while the kernel action stays distinct; the intervention record always carries the real action. fail.reason is not decoration: when a supervised run ends in fail, the runner surfaces reason as the job’s user-facing error summary (the PRD’s “one-sentence reason written by the agent”) while the original thrown error is preserved in the intervention record and node error detail. The pseudocode’s rethrow carries the original error; the runner attaches the reason at job-status level. applyTo: "signature" is the sticky form: the handle caches the verdict keyed by (nodeId, failureSignature) and resolves later escalations that match without waking the agent (PRD scenario 2 — 7 identical failures, 1 LLM call). Keying by signature rather than node keeps one login-required error from silently skipping later, unrelated timeouts on the same node. A signature exists only when the error carries a stable categorical code — HTTP status, provider error code, validation path — extracted by a small registry of error-shape recognizers. A plain Error gets no signature (error class alone is not one; every generic throw would collide), and with no signature stickiness is simply off: each such failure escalates individually. Signatures never derive from message text with embedded values. Only skip and fail may stick; a sticky retry or substitute would blindly replay a decision made against different inputs. declaredOutputs is what makes substitute authorable at all: a repair has to be typed per slot, and the escalation is the only thing the supervisor sees. It is also what the agent-side acceptance hook validates against before the verdict is allowed to become terminal, so the model learns its repair was the wrong shape while its conversation is still open. Type metadata, so nothing to redact. RecoverableNodeError (node-sdk) is how a node hands the supervisor the thing that needs repairing: a parser that throws on broken JSON attaches the raw response as candidateOutput instead of losing it. No candidateOutput, no substitute — without the broken value in hand, “repair” is fabrication: the model would invent a structurally valid output the runtime validator cannot semantically vet. A plain thrown error offers retry/skip/fail only. New ProcessingMessage variants: supervisor_escalation and supervisor_decision (escalation + verdict + decidedBy: "agent" | "sticky" | "bounds" | "default" + cost). Both flow through the existing _emit path. RunResult gains: interventions?: Array<{ escalation: Escalation; verdict: Verdict; decidedBy: string }>; Status stays the existing enum. “Completed — supervised” is status: "completed" with a non-empty interventions — a derived display state, not a fifth status, so every existing consumer of status keeps working. 5. The kernel hook 5.1 Where it sits WorkflowRunner takes the handle as an option and hands it to each NodeActor, alongside the callback that records interventions. The handle stays in the kernel rather than on ProcessingContext: node code has no business raising or answering escalations, and the actor already holds everything the escalation constructor needs. What the context does gain is the resolved-secret registry the constructor masks against (§6.1) and per-invocation cost/asset accounting (§5.3). export interface WorkflowRunnerOptions { // ...existing supervisor?: SupervisorHandle; } export interface SupervisorHandle { decide(e: Escalation, signal: AbortSignal): Promise<DecisionOutcome>; close(): void; } /** The verdict, who made it, and what it cost — the intervention record minus its escalation. */ export interface DecisionOutcome { verdict: Verdict; decidedBy: "agent" | "sticky" | "bounds" | "default"; costUsd?: number; } NodeActor wraps each invocation’s execution. Pseudocode for the buffered/correlated path (_executeWithInputs): const allowed = this.allowedActions(node, invocation); // kernel-computed, §5.3/§5.4 let result; for (;;) { try { result = await this.invoke(inputs); // node execution ONLY break; } catch (err) { if (err instanceof WorkflowSuspendedError) throw err; // suspend is not failure let verdict = await this.escalateOrFail(err, inputs, allowed); if (!allowed.has(verdict.action)) verdict = { action: "fail" }; // kernel enforces, always switch (verdict.action) { case "retry": continue; // same inputs, same properties case "substitute": result = this.validated(verdict.outputs); break; case "skip": this.dropOutputs(); return; // lineage_done + scope closure case "end_stream": this.completeSlots(); return; case "fail": throw err; } break; } } this.sendOutputs(result); // routing is OUTSIDE the recovery loop — see below Two boundaries here are load-bearing: The kernel computes and enforces allowed. Schema narrowing (§5.3, §5.4) shapes what the agent can express, but the actor independently rejects any verdict outside the allowed set — a buggy or third-party SupervisorHandle returning retry for an unsafe node, or substitute for an unsupported output, gets fail. The allowed set travels as Escalation.allowedActions so both layers derive from one computation. Defense in depth: the schema is UX, the kernel check is the guarantee. Only node execution is recoverable. sendOutputs routes values to downstream inboxes sequentially; if edge 1 delivered and edge 2 throws, a retry would duplicate edge 1’s delivery. Routing failures therefore fail the node without escalation — the catch wraps invoke(), never the send. Retry is a plain loop re-entry: the invocation’s inputs are still in scope, so nothing is reconstructed and no scheduler state is touched. The same wrapper applies to the controlled path (_runControlled, per control event) and to input dispatch. 5.2 What each verdict means to the kernel retry — re-invoke with the identical inputs and properties (no overrides, §4). Bounded by maxRetriesPerNode (counted per (nodeId, invocationKey)). substitute — offered only when the escalation carries a candidateOutput (§4) and every declared output type has a validator rule. Outputs pass a runtime value validator, not a TypeMetadata compatibility check: graph.validate() compares declared edge types and the existing runtime accepts any object for custom types, so type-compatibility would wave through a fabricated {type:"image"} blob. The validator checks actual values per declared output type: primitives and enums strictly, lists/objects structurally, and reference types (ImageRef, AudioRef, …) as well-formed refs whose uri/asset id resolves against the run’s storage. Resolution is asynchronous, which constrains where validation runs: it cannot happen after decide() returns, because the StepExecutor conversation is over and there is nobody left to bounce an error to — and finish_step’s built-in loop only covers synchronous JSON-schema checks. So the verdict tool’s acceptance is an async hook: when the agent calls finish_step with a substitute, the handle awaits the full runtime validation (storage resolution included) before the call is accepted as terminal; a failure returns as an ordinary tool error and the conversation continues. After MAX_REPAIR_ROUNDS (3) → fail. Valid → emitted through the normal sendOutputs path with the invocation’s lineage, so correlation downstream is indistinguishable from a real emit. skip — the actor does not simply return. For a correlated invocation, plain non-emission leaves siblings buffered against the skipped key at downstream joins — the kernel grew per-lineage lineage_done signaling precisely because global EOS can’t express this. NodeOutputs.drop(slot, envelope) covers that half. It does not cover the other half: drop() emits only lineage_done, and a skipped invocation of an iteration node would have opened child scopes that an aggregate downstream is waiting to see closed. Skip therefore needs new kernel plumbing — per-parent lineage_scope_closed emission for the scopes the invocation would have created (the signaling primitive exists in NodeInbox; what’s new is the actor-side emission path). The iterator-feeding-an-aggregate case is a named PR 1 test. The skip is recorded in interventions unconditionally — the audit is the feature (PRD scenario 3). fail — rethrow; identical to today. 5.3 Retry safety The handle — not the agent’s judgment — decides whether retry appears in the verdict schema at all, so an unsafe retry is unrepresentable rather than merely discouraged. retry is offered only when all hold: retrySafe === true — the node opts in via node metadata (node-sdk); spentCostUsd === 0 — no provider cost recorded by this invocation; createdAssets === false — no asset writes. (1) is an opt-in, not an opt-out, because the failure mode is asymmetric. An unset flag on a custom node, a legacy package, a nested WorkflowNode (whose inner graph can contain anything), or an overlooked write node must not gain retry by omission — cost tracking cannot see external writes (Publish, Upsert, Notify, payment-shaped nodes can complete their side effect and then throw, recording nothing). So unknown defaults to unsafe: a node nobody classified loses a safe retry (annoying, visible, fixable with one flag) rather than gaining an unsafe one (silent duplicate effects). PR 1 declares retrySafe across the pure-compute and read-only node categories in the shipped packages; WorkflowNode/SubgraphNode are never retry-safe in v1. A side-effecting node can re-earn retry later via an idempotency key — out of scope, noted in §10. (2) and (3) require invocation-scoped tracking, and it cannot be a stack on the shared context: every actor holds the same ProcessingContext and actors complete out of order, so push/pop attribution would credit actor A’s cost to actor B whenever A finishes first — exactly the corruption that would re-expose retry after a billed call. The scope must travel with the invocation’s async execution: AsyncLocalStorage around invoke(), or an explicit per-invocation accounting object threaded through the provider/asset call paths. PR 2 builds it either way, and reverse-order completion of concurrent invocations is a named test. 5.4 Streaming nodes Streaming breaks the “inputs still in hand, just re-invoke” premise in two distinct ways. Streaming output (is_streaming_output, genProcess): the coherent verdict set depends on whether anything was emitted (PRD scenario 6): State Allowed verdicts threw before first emit retry* · skip · fail threw after emitting end_stream (close slots, keep what was emitted) · fail No substitute in either state — the PRD’s rule (“Repair never” for streams) holds: there is no single invocation value for a substitution to stand in for. Streaming input (is_streaming_input, run() nodes): the node drains its inbox itself and may have consumed messages long before throwing — so even “threw before first emit” does not mean the invocation is replayable. Worse, a run() failure has no single invocation lineage: consumed-but-unresolved envelopes span arbitrary lineages, so there is nothing coherent for skip’s drop() to target either. In v1 the verdict set for run() nodes is end_stream · fail — nothing else is representable without a kernel-level input replay buffer or per-envelope lineage retention, both explicitly out of scope (§10). *retry additionally requires §5.3’s conditions, as everywhere. Excluded verdicts are removed from the agent’s schema and from the kernel’s allowed set (§5.1) — the schema is UX, the allowed set is the guarantee. 5.4a Content-filter refusals in an unsupervised run One failure class degrades without a supervisor: a provider content filter refusing to generate. Veo 3.1 filtered one shot of a five-shot trailer on an ordinary cinematic shot description, the same prompt passed on retry, and the blocked take was not charged — so the refusal is non-deterministic and provider-side, not a defect in the request. It is answered at two levels. ProcessingContext.runProviderPrediction retries a refusal CONTENT_FILTER_MAX_RETRIES times with backoff, so most never reach the node. One that survives reaches the actor as a ContentFilterRefusal (@nodetool-ai/runtime), and the actor takes the skip path — retiring that invocation’s lineage, exactly as the verdict does — when the invocation is one item of a fan-out and nothing has been emitted yet. Everything else still fails: a single-shot invocation has no siblings whose spend a failure would discard, and an emitted stream has partial output downstream that a retired lineage would contradict. A supervised run is unchanged. The refusal escalates like any other failure, because putting an agent on the failure path is what --supervise is for; skip is already in its allowed set (§5.4). 5.5 Cancellation and timeouts Each decide() call gets its own AbortSignal, derived from the run’s signal plus a decision timeout (default 60s). The signal is not decoration around the await: it threads through StepExecutor into provider.generateLoop({signal}), so abort actually terminates the in-flight LLM request rather than orphaning it to keep consuming tokens after cancel(). Abort or timeout → fail. This is what keeps cancel() instant and free (PRD scenario 8): a pending decision is killed, not merely un-awaited. 6. The supervisor agent packages/agents/src/supervisor/supervisor-agent.ts implements SupervisorHandle: One StepExecutor per decision, outputSchema = the verdict JSON schema (narrowed to allowedActions), maxIterations small (6). This does require extending StepExecutor: its current finish_step validation checks only top-level type and required keys — no enums, nested fields, additionalProperties, or discriminated unions, which a verdict schema needs — and its acceptance closure is private and synchronous, while substitute acceptance must await storage resolution (§5.2). PR 3 adds a full host-side JSON-schema validator and an injectable async acceptance callback to StepExecutor proper (both generally useful, neither supervisor-specific). Two tools, both read-only, backed by state the runner already holds: get_run_state() (per-node status, error counts, cost — the digest) and read_node_output(nodeId) — scoped to the failed invocation’s causal lineage. In a fan-out, “last emitted value” may belong to a different item; an unscoped read hands the agent another item’s output as repair material (a poisoned repair) and quietly widens the data shared with the model beyond the failing item. The tool resolves through Escalation.correlationLineage and refuses reads outside it. Progressive disclosure, same pattern as read_shared. Serialized queue. One decision at a time. Consecutive failures are usually related; the agent sees them in order, and a sticky verdict drains the queue for free. Head-of-line cost is bounded by the decision timeout. Bounds. maxDecisions (default 10), maxRetriesPerNode (default 2), and maxSupervisorCostUsd (default $0.50). Decisions and retries alone do not cap dollars, and checking spent cost before a call doesn’t either — the next call’s cost is only known afterward, so $0.49 spent could admit a $0.30 call and land at $0.79. The ceiling is enforced by reservation: worst-case cost (input tokens × input price + max output tokens × output price, from the model-pricing catalog), admitted only if spent + reserved ≤ cap, committed to actuals afterward. Two consequences the plumbing must respect. First, the interception point must sit inside the provider layer, not around it: a wrapper around generateLoop() sees one call while the base provider makes its turns internally, and the Claude Agent SDK provider overrides the whole loop — so neither a kernel-side check at decide() granularity nor an outer wrapper can reserve per turn. The mechanism is a runtime-level budget object (TurnBudget: reserve(estimate) → admit | reject, commit(actual)) accepted by generateLoop options and honored before every model turn by the base loop and by native loop overrides — an explicit contract obligation on providers that own their loop. Decision/retry counting stays in the kernel’s BoundedHandle (PR 2); the budget object is constructed by the supervisor handle and threaded per decision (PR 3, with the runtime change). Second, reservation needs a known output bound, so the supervisor sets an explicit supervisorMaxOutputTokens (default 2048) on every call — StepExecutor.maxTokens being optional is not good enough here. A model with no usable pricing entry fails closed — the supervisor won’t run on it. At any boundary the handle stops calling the agent and returns fail with decidedBy: "bounds" — degradation is deterministic, not a cheaper model. System prompt carries the verb semantics from the PRD, including skip-distrust and the itemization duty. It is a preamble over the standard execution contract, per the established StepExecutor rule. Decisions and their one-line rationales are also written to context.memory under supervisor: keys so a downstream CompilerAgent (in the workflows-as-agents path) can reference them in its synthesis. 6.1 What the supervisor model sees Escalations carry node inputs, error text, and (via read_node_output) prior outputs — real user data, sent to whichever model supervises. Truncation is not redaction; the boundary is explicit: Redaction at construction, not at the prompt. Escalation is a public record: it becomes a supervisor_escalation message, lands in RunResult.interventions, and crosses the websocket — all before any agent exists. So the redaction layer lives in the kernel and runs when the escalation is built (PR 1, the same PR that mints the type): the actor hands raw error + inputs to a constructor that masks secret values wherever they appear (headers, URLs, bodies) and drops fields matching the sensitive-name list (password, token, authorization, …). Raw values never leave the actor’s stack frame; there is no unredacted Escalation anywhere in the system to leak. The agent-side handle applies the same layer to tool results (read_node_output), the one supervisor input not derived from an Escalation. Masking “every secret-store value” presupposes knowing which values are secrets, and today’s ProcessingContext doesn’t: it resolves one secret by key and forgets. So redaction has a prerequisite deliverable — a resolved-secret registry on the context that records every value handed out by the secret resolver for the run’s lifetime, which the escalation constructor masks against. PR 1 builds it alongside the constructor; without it the masking promise is empty. Provider policy. The supervisor model must be restricted to providers the workspace approves — and this, too, is infrastructure that does not exist yet: current allow-listing is global cloud-node pruning and frontend filtering, with no runtime enforcement per workspace. A workspace provider allow-list (schema + API + a runtime check in provider resolution) is a concrete deliverable on the path to default-on, not an assumed property. Until it ships, the supervisor honors the existing global restrictions only, and the gate treats the allow-list as unmet. Consent. Enabling supervision is the consent act, and the toggle’s help text says what will be shared, with which model, when. Consent is per-trigger and forward-looking only: existing triggers are never silently migrated when the default flips — the new default applies to newly created triggers, and existing ones surface a one-time prompt instead. This is why default-on is gated (§ plan PR 8): defaults must not expand data flow before the boundary above has shipped and been evaluated. 7. Entry points Surfaces do not construct WorkflowRunner directly anymore — CLI, debug, headless, and websocket execution all go through the ExecutionSession facade (packages/execution), with a handful of grandfathered direct-construction sites. Supervision therefore plumbs through the facade: ExecutionSessionOptions.supervisor is the one integration point every surface shares, and it forwards to WorkflowRunnerOptions.supervisor internally. Wiring the surfaces around the facade would either reintroduce direct construction or make supervision behave differently per surface — both are non-goals. WorkflowRunnerOptions.supervisor — the kernel primitive. Used directly only by kernel tests and by ExecutionSession itself. CLI — nodetool run --supervise [--max-decisions N] [--max-retries N] [--supervisor-cost-cap USD] [--supervisor-model id] and the same flags on nodetool debug. Interventions print inline (⛨ lines) and appear in --json reports. Agent({ graph }) — a fourth branch in Agent._executeImpl alongside executeScriptPlan/executeGraphPlan: hydrate the graph, start a WorkflowRunner with itself as supervisor, forward the runner’s message stream, return run outputs from getResults(). No planning phase — the graph is the plan. API/web — supervise: true on the run request; the websocket runner constructs the handle and forwards supervisor_* messages to the client for the intervention feed. Trigger rows carry the flag, off by default until the plan’s PR 8 gate passes; the eventual flip covers newly created triggers only (§6.1 consent). Interactive (agent tools) — interactive: true on POST /api/workflows/:id/run|debug makes the calling agent the supervisor: InteractiveEscalationHandle (packages/execution/src/service/debug-sessions.ts) parks each decide() until a verdict arrives on POST /api/debug/sessions/:id/verdict, and the escalated HTTP response carries the Escalation record itself. The agent-facing run_workflow/debug_workflow tools expose the flag and resolve_workflow_escalation answers; the handle is wrapped in the same BoundedHandle (caps, sticky verdicts, a timeout sized for a tool round trip — 10 min — that fails closed), and the kernel still enforces allowedActions whatever arrives. 8. Replay supervisor_decision messages carry everything the decision consumed (the escalation, the digest snapshot hash, the verdict). A ReplayHandle implementing SupervisorHandle resolves escalations from a recorded run’s decision list, matching on (nodeId, invocationKey, attempt); a miss falls through to fail. This is a consumer of the intervention record, not a new capture pipeline — deferred to the eval-suite PR, where it doubles as the deterministic test double. 9. Alternatives considered Compile the graph to a TaskPlan and run it on ParallelTaskExecutor. Rejected: Step.dependsOn cannot express correlation lineage, streaming, control edges, or per-handle EOS. The kernel is the better scheduler; supervise it instead of replacing it. Per-node SupervisionPolicy struct (onError/retries/checkpoint/budget). Rejected: everything in it except retry counts is better expressed as graph structure (Assert nodes, branches), where users can see and edit it. Retry counts became a single run-level bound. A run ledger / observation pipeline. Rejected: the runner already holds run state in memory; two pull tools replace a push pipeline. Waking the agent on checkpoints was cut with it — a clean run must cost zero. Whole-run suspend (WorkflowSuspendedError) as the intervention mechanism. Rejected for this purpose: it tears down in-flight actors. escalate parks one invocation while the rest of the DAG proceeds. Suspend remains what it is — a deliberate pause for triggers and human waits. Cross-node repair (“retry the producer with a better prompt”). Rejected: needs a control channel from supervisor to arbitrary nodes, reopening most of the complexity this design removed. Expressible as a graph loop where wanted. 10. Open questions inputs truncation policy in Escalation — per-value byte cap vs. schema-aware summarization. Start with the byte cap (MAX_TOOL_RESULT_CHARS sibling), revisit if verdict quality suffers. (Redaction per §6.1 happens regardless; this question is only about size.) Provider-level backoff audit. PRD scenario 1 assumes transient retry exists below the hook. Audit packages/runtime/src/providers/ retry behavior in PR 1; if absent for a provider, that gap is fixed in the runtime layer, not compensated for in the supervisor prompt. Idempotency keys for side-effecting nodes. §5.3 gates retry on an explicit retrySafe opt-in. A side-effecting node that accepts an idempotency key could safely re-earn it. Deferred until a real node needs it. Input replay buffer for run() nodes. §5.4 leaves streaming-input nodes with only end_stream · fail because consumed inbox messages are gone and no single lineage exists to drop. A bounded replay buffer (or per-envelope lineage retention) would re-earn retry and skip; kernel-level work, deferred until an eval case shows it matters. Typed retryMutable properties. §4 removes property overrides from retry entirely. If re-parameterized retry proves necessary, the path is node-declared, typed retryMutable properties applied to a local invocation copy — never a free-form record, never updateNodeProperties.--- # Implementation Plan: Workflow Supervisor Source: https://docs.nodetool.ai/workflow-supervisor-implementation-plan.html Implementation Plan: Workflow Supervisor Author: Matti Georgi Status: Draft — for review Last updated: 2026-08-01 PRD: workflow-supervisor-prd.md Design: workflow-supervisor-design.md PRs marked ∥ can proceed in parallel once their listed dependency lands. 0. Ground rules Every PR passes npm run check and lands behavior-preserving unless its description says otherwise. PR 1 and PR 2 must be invisible to existing runs. The kernel never imports from @nodetool-ai/agents. The SupervisorHandle interface lives in kernel; implementations live above it. No UI work before the intervention record is stable in --json output (PR 4) — the web surface consumes the record, it does not define it. Verdict semantics change only via the design doc, in the same PR. Phase A — the mechanism (kernel, no LLM) — shipped PR 1 and PR 2 landed together: PR 1’s allowed-set enforcement is only as good as PR 2’s invocation-scoped cost tracking, so shipping the first without the second would have meant a window where retry was offered after a billed call. PR 1 — Escalation types and the actor hook The seam, fail-closed. Zero behavior change. packages/protocol/src/supervisor.ts: Escalation (incl. optional failureSignature, candidateOutput, retrySafe), Verdict, message schemas supervisor_escalation / supervisor_decision; add to the ProcessingMessage union. packages/kernel/src/supervisor.ts: SupervisorHandle interface (decide(e, signal)), FailClosedHandle default. Resolved-secret registry on ProcessingContext (design §6.1): record every value the secret resolver hands out for the run’s lifetime. Prerequisite for masking — today the context resolves one key and forgets. Redaction at escalation construction (design §6.1): the kernel-side constructor masks registry values and drops sensitive-named fields before the Escalation exists as a record — Escalation is public (messages, RunResult, websocket), so no unredacted instance may be constructed. Test: a planted secret in a node input never appears in any emitted message or RunResult. Kernel-side allowed-action enforcement (design §5.1): the actor computes the allowed set per escalation, sends it on the Escalation, and rejects (→ fail) any verdict outside it. Test: a hostile scripted handle returning retry for an unsafe node is rejected without the schema layer’s help. Recovery catch wraps invoke() only (design §5.1): output routing stays outside; a routing failure after partial delivery fails the node without escalation. Test: edge-2-throws-after-edge-1-delivered never re-invokes. Scope-closure plumbing for skip (design §5.2): per-parent lineage_scope_closed emission for scopes the skipped invocation would have opened — drop() alone signals only lineage_done. Test: skipping one iterator invocation feeding an aggregate still lets the aggregate collapse. node-sdk: RecoverableNodeError (carries the malformed candidate output, redacted at construction like everything else) and the retrySafe node-metadata opt-in; declare it across the pure-compute and read-only node categories in shipped packages. WorkflowNode/SubgraphNode stay unsafe. Unknown ⇒ no retry. packages/kernel/src/runner.ts: WorkflowRunnerOptions.supervisor, wire escalate onto the execution context, interventions on RunResult. packages/kernel/src/actor.ts: wrap invocation execution per design §5.1 — buffered/correlated path first, controlled path second. WorkflowSuspendedError passes through untouched. Skip = drop() per output slot (design §5.2), never a bare return. Audit while in here: provider-level transient retry in packages/runtime/src/providers/ (design §10.3). Done — workflow-supervisor-provider-retry-audit.md. Gemini and Ollama have no backoff at all and would flood the supervisor with escalations a 2-second wait would fix; the fixes land in runtime, not here. Tests (packages/kernel/tests/): scripted SupervisorHandle doubles driving every verdict against fan-out, controlled, and single-fire nodes; retry re-invokes with identical inputs and properties; skip on a correlated key prunes a downstream join whose other input already buffered (the lineage_done case) plus the iterator→aggregate scope-closure case; no-supervisor runs byte-identical to before (message-stream snapshot). PR 2 — Bounds, retry safety, streaming carve-out Still no LLM. Depends on PR 1. Invocation-scoped cost/asset tracking in ProcessingContext — via AsyncLocalStorage around invoke() or an explicit per-invocation accounting object, not a stack on the shared context: actors complete out of order and push/pop would cross-attribute (design §5.3). New plumbing — today cost aggregates per run in one slot and no asset-created flag exists. Test: two concurrent invocations completing in reverse order attribute correctly. BoundedHandle wrapper in kernel: maxDecisions, maxRetriesPerNode, per-decision timeout; every decide() gets a derived AbortSignal (run signal + timeout); decidedBy: "bounds" | "default" stamping. (Dollar reservation is agent-side, PR 3 — the kernel handle cannot see individual provider turns.) Allowed-set computation (consumed by PR 1’s enforcement): retry only when retrySafe ∧ cost-free ∧ asset-free (design §5.3); substitute only with candidateOutput ∧ full validator coverage; streaming sets per design §5.4 — run() nodes get end_stream · fail only. Runtime substitute validator (design §5.2): per-output-type value rules — strict primitives, structural lists/objects, reference types must be well-formed refs that resolve against run storage; async, exposed for PR 3’s acceptance hook. Sticky-verdict cache keyed by (nodeId, failureSignature) (skip/fail only), decidedBy: "sticky"; signatures come from a registry of error-shape recognizers (HTTP status, provider codes, validation paths) — a plain Error yields no signature and no stickiness. Tests: bounds degrade to deterministic fail; cancel during a pending decision resolves instantly; an undeclared node’s allowed set has no retry; a plain-Error escalation has no substitute; a run() node’s set is exactly {end_stream, fail}; a fabricated {type:"image"} ref with an unresolvable uri fails validation; sticky skip resolves the 7-of-200 case with one decide() call while a plain-Error failure on the same node escalates every time. Phase B — the agent and the CLI PR 3 — SupervisorAgent — shipped Depends on PR 2. Two additions the plan did not name, both forced by what the agent has to see: Escalation.declaredOutputs (a repair cannot be typed, or accepted, without the node’s declared output types) and a kernel-side RunStateReader handed to the handle via SupervisorHandle.attach() — the tools are pull-based over runner state, and a handle is configured before the runner exists. Output recording for read_node_output is bounded per node and happens only on a supervised run. Host-side schema validation checks the schema as the caller wrote it, not the sanitized copy sent to the provider: the sanitizer injects additionalProperties: false for strict structured-output modes, and enforcing that would reject results no author ever forbade. packages/agents/src/supervisor/supervisor-agent.ts: SupervisorHandle via one StepExecutor per decision, verdict outputSchema (mirroring the escalation’s allowed set), serialized queue; the decide() signal threads through StepExecutor into provider.generateLoop({signal}) so abort kills the in-flight LLM request. TurnBudget in the runtime provider layer (design §6): reserve/commit object accepted via generateLoop options, honored before every model turn by the base loop and by native loop overrides (Claude Agent SDK provider explicitly). Worst-case reservation from the pricing catalog with explicit supervisorMaxOutputTokens (default 2048, never unset); no pricing entry ⇒ fail closed. This is a packages/runtime change with its own provider-contract test, not a wrapper in the agents package — a wrapper around generateLoop() cannot see individual turns. StepExecutor extensions (design §6): full host-side JSON-schema validation for finish_step (enums, nested fields, additionalProperties, discriminated unions — today it checks top-level type and required keys only) and an injectable async acceptance callback so a substitute becomes terminal only after the runtime validator (PR 2, storage resolution included) resolves; failures return as tool errors into the still-open conversation, MAX_REPAIR_ROUNDS (3) then fail. Both extensions are general StepExecutor features with their own tests. Tool-result redaction: read_node_output returns are the one supervisor input not derived from an already-redacted Escalation (that layer landed in PR 1, kernel-side); apply the same layer here. packages/agents/src/supervisor/tools.ts: get_run_state, read_node_output (read-only, redacted, truncated, lineage-scoped — reads resolve through the escalation’s correlation_lineage and refuse anything outside it). packages/agents/src/supervisor/prompt.ts: verb semantics, skip-distrust, one-line rationale requirement. supervisor: memory keys for decisions + rationales. Tests: mock provider returning scripted tool-calls/verdicts; malformed verdict exercises the finish_step bounce loop; an unresolvable substitute ref bounces back into the open conversation; supervisor exceptions and timeouts resolve as fail; abort mid-decision cancels the provider call; a secret value planted in a read_node_output result never appears in the outbound prompt; in a fan-out, read_node_output refuses a sibling item’s output; reservation blocks the $0.49+$0.30 overshoot mid-decision. PR 4 — CLI surface ∥ (with PR 5) — shipped Depends on PR 3. Two deviations the plan did not anticipate. nodetool run is a DSL command, not a session command. `nodetool run ` executes through `@nodetool-ai/dsl`'s own `run()`, which constructs a `WorkflowRunner` directly — one of the grandfathered sites this PR is forbidden to grow supervision on. So `--supervise` switches that command onto `ExecutionSession` (`packages/cli/src/run-dsl-supervised.ts`) and the unsupervised path stays byte-identical on the old one. One user-visible consequence: the supervised path does not rethrow the first node error the way the DSL path does. A node error the supervisor resolved is not a run failure; only the run's terminal status decides. **The summary line's item counts do not exist at run level.** "198/200 items" presumes a batch the kernel never names: it counts invocations, not the items a user thinks in. `formatSupervisedSummary` takes an optional item total for a caller that knows one and otherwise reports decisions — `⛨ supervised: 2 skipped, 1 retried, 3 decisions, +$0.0200`. The intervention record shipped as `Intervention` from `@nodetool-ai/protocol` (minted in PR 1), unchanged — `RunResult.interventions`, the `supervisor_decision` message, and the `--json` block are the same record, so PR 6 has one shape to consume. The rollup and the two printed lines live in `@nodetool-ai/execution` beside the debug reducer, so the CLI, the debug bundle, and the HTTP debug endpoint cannot drift. - **`ExecutionSessionOptions.supervisor`** in `packages/execution` — the single integration point; CLI and websocket surfaces configure the facade, never `WorkflowRunner` directly (design §7). Grandfathered direct-construction sites are out of scope but must not gain supervision ad hoc. - `--supervise`, `--max-decisions`, `--max-retries`, `--supervisor-cost-cap`, `--supervisor-model` on `nodetool run` and `nodetool debug` (the latter through the shared debug service in `@nodetool-ai/execution`, not around it). - Inline `⛨` intervention lines; interventions block in `--json`; supervised summary line ("198/200 items, 2 skipped, 3 decisions, +$0.02"). - Supervisor cost attributed through the existing cost tracking so `nodetool costs` sees it (PRD open question 3 resolved: attributed to the run, tagged `supervisor`). - Docs: CLI section in root `CLAUDE.md` + `docs/cli.md`. ### PR 5 — Workflows as agents ∥ (with PR 4) — **shipped** Depends on PR 3. Three things the plan did not anticipate, all forced by the websocket half. A `supervise` flag alone cannot start a supervisor: something has to name the model. The run request therefore carries an optional `supervisor` (`SupervisorRunOptions`: provider, model, the three bounds, cost cap) next to the flag, falling back to the connection's configured default model and then to `NODETOOL_SUPERVISOR_PROVIDER` / `NODETOOL_SUPERVISOR_MODEL` — the last exists because trigger-driven headless runs have no connection defaults at all. A request that asks for supervision it cannot get runs **unsupervised** rather than failing, which is the same fail-closed rule every other supervisor failure follows. The trigger flag is a real column (`trigger_registrations.supervise`, default `0`, migration `20260801_000001`), read by the dispatcher into the headless run. Registration sync mutates existing rows in place, so re-syncing a workflow never resets it. The supervisor gets a **dedicated provider instance** (`getProvider`, not the context's cached one) and a listener-free context copy: per-turn spend is reconciled from the provider's own running cost, so a second caller on the same instance would corrupt the dollar cap, and the decision's own provider traffic is not the run's message stream. The escalation and the verdict still cross the websocket — they are emitted by the kernel on the run's context. `ExecutionSessionOptions.supervisor` is PR 4's deliverable and landed here because PR 5 needs it; the two branches carry the same three-line change. - `AgentOptions.graph?: GraphData | { workflowId: string }`; fourth branch in `Agent._executeImpl`: hydrate, run through `ExecutionSession` with self as supervisor, forward messages, `getResults()` returns run outputs. The branch adopts the common `AgentPolicy` object (`packages/agents/src/agent-policy.ts`); it does not add a fifth ad-hoc policy. - Websocket: `supervise` flag on run requests; forward `supervisor_*` messages to clients. Trigger rows carry the flag but it **defaults to off** — the flip to default-on belongs to PR 8's gate, nowhere earlier. - Tests: `Agent({graph})` returns identical outputs to a bare runner on a clean graph; interventions surface in the message stream. Plus what makes the branch trustworthy: a scripted `skip` completes a run that otherwise fails and a scripted `fail` does not, a clean supervised run emits no `supervisor_*` message at all, an aborted signal cancels the run, and `createRunSupervisor` returns no handle without an explicit flag and a resolvable model. ## Phase C — the product surface ### PR 6 — Editor and run report Depends on PR 4 (record shape frozen). - Run-bar **Supervisor** toggle + settings (model, max decisions, max retries, cost cap); off by default everywhere until PR 8. Toggle help text states what failure context is shared and with which model (the consent surface, design §6.1). - Shield badges per verdict on nodes; "supervisor deciding…" node state; intervention feed panel (appears on first intervention only). - Run report **Interventions** section with itemized skips; "Completed — supervised" derived display state. - All via `ui_primitives`; feed cards reuse the chat agent-step card family. ### PR 7 — Assert node ∥ (with PR 6) Depends on PR 1 only. - `nodetool.control.Assert` in `packages/core-nodes/`: **deterministic in v1** — expression condition over its input; throws on failure, pass-through otherwise. - The natural-language criterion variant is deferred: it needs a model, and `WorkflowRunnerOptions` has no provider/model policy — inventing one inside a node would sidestep the cost caps, consent copy, and provider allow-list this whole design routes model access through. It returns as a follow-up once a run-level model policy exists (it can then reuse the supervisor's own budget and boundary machinery). - The PRD's "graph is the policy" counterpart: docs steer foreseeable checks here, not into supervisor config. ### PR 8 — Eval suite, replay, enable by default Depends on PR 5–7. - `ReplayHandle` (design §8) as the deterministic test double. - **Workspace provider allow-list** (design §6.1): schema + API + runtime enforcement in provider resolution. Does not exist today (current allow-listing is global cloud pruning and frontend filtering); it is a named precondition of the default flip, so it lands here at the latest. - `nodetool eval supervisor` in `packages/agents/src/evals/`: fault-injection cases over the three PRD reference shapes (transient error, malformed output, poisoned item, mid-stream failure, budget exhaustion). Metrics measure **unsafe recovery, not just recovery**: recovery rate, but also incorrect-skip rate (item skipped that a scripted verdict recovers), poisoned-repair rate (substitution passes validation but flunks the case's semantic check), duplicate-effect count (side-effect probe nodes counting executions), and cost-overrun count. `--min-success` CI gate like the other suites. - Flip the trigger-run default only when **all** gates pass: recovery ≥ the PRD's 60% bar; incorrect-skip, poisoned-repair, duplicate-effect, and cost-overrun each under an explicit threshold set in the suite (duplicate effects and cost overruns: zero); and the data-boundary requirements (design §6.1) verified. The flip applies to **newly created triggers only** — existing triggers are never silently migrated; they get a one-time prompt (consent is forward-looking, design §6.1). Until then everything ships opt-in. - Docs: user guide page; PRD/design status → Shipped. --- ## Sequencing ``` PR 1 → PR 2 → PR 3 → PR 4 ∥ PR 5 → PR 6 ∥ PR 7* → PR 8 (*PR 7 only needs PR 1) ``` Two people: one takes 1→2→3 (kernel/agents), the other starts PR 7 after PR 1 and PR 6 mocks against the PR 4 record shape. ## Risks to the plan - **The actor catch-path has three execution modes** (correlated, controlled, streaming) and the hook must behave identically in each. Mitigation: PR 1 lands correlated + controlled with the streaming carve-out stubbed to `fail`; PR 2 completes streaming. Snapshot tests on unsupervised runs guard regressions. - **Verdict quality is unproven until PR 8.** The eval suite arrives last but gates the only risky default (trigger default-on). Everything before it ships opt-in. - **Provider retry gaps** (design §10.3) would flood the supervisor with transient escalations. The PR 1 audit exists to surface this before PR 3 makes it expensive. - **Invocation-scoped cost tracking is new plumbing** (confirmed: `ProcessingContext` aggregates per run, no asset flag). It is a first-class PR 2 deliverable; if the stacked-scope approach fights the context's structure, PR 2 grows rather than the retry-safety rule weakening. - **Cost/asset tracking cannot see external writes.** The `retrySafe` opt-in is the defense, and unknown defaults to unsafe — so the residual risk is a node *wrongly declared* `retrySafe`, not an undeclared one. Mitigation: PR 1's sweep declares only pure-compute/read-only categories; the review checklist for new nodes gains the question.--- # PRD: Workflow Supervisor Source: https://docs.nodetool.ai/workflow-supervisor-prd.html PRD: Workflow Supervisor Author: Matti Georgi Status: Draft — for review Last updated: 2026-08-01 Design doc: workflow-supervisor-design.md Implementation plan: workflow-supervisor-implementation-plan.md 1. Summary A NodeTool workflow is deterministic: the graph says what happens, in what order, with what data. That is its strength — and the reason a single flaky HTTP call at 3am kills a 200-item pipeline that was 97% done. The Workflow Supervisor gives a run a safety net. When a node fails after its own retries are spent, the run doesn’t die: an agent wakes up, looks at what the node was doing, and makes one of four calls — Retry, Repair, Skip, or Stop. Then it goes back to sleep. On a clean run the supervisor costs nothing: zero LLM calls, zero latency, zero dollars. The user-facing promise is one sentence: your workflow runs like it always did, but a failure becomes a decision instead of a funeral. This is also what makes a workflow behave like an agent. An agent improvises its steps; a supervised workflow keeps its deterministic steps and borrows only the part of agency that’s actually valuable — judgment at the moment something breaks. 2. Problem 2.1 Failures are total Today one node error fails the whole job (RunResult.status = "failed"). The kernel’s suspended state is a different animal — a deliberate pause for triggers and human waits, not failure handling. For failures there is no middle ground between “everything worked” and “you get nothing”: A 200-URL content pipeline dies on URL #147. The 146 finished summaries are in the message log, but the run is failed, the CSV never writes, and the user’s mental model is “it broke.” An overnight image batch fails on one malformed provider response after $6 of successful generations. A nightly ingest dies mid-run; nobody sees it until someone notices the vector store is stale two days later. 2.2 The existing answers don’t fit Agent mode (AgentNode, plan mode) replaces determinism with planning. Users who drew a graph did so because they want these steps, not whatever a planner invents tonight. Manual babysitting works only when someone is watching. The whole point of long and scheduled runs is that no one is. Defensive graph-building (try/catch subgraphs, fallback branches everywhere) is possible today and nobody does it, because it triples the graph size to handle failures that occur on 2% of runs. 2.3 Why now Triggers (triggers-prd.md) turn workflows into things that run while the user sleeps. Unattended runs make failure handling the difference between an automation and a liability. The shared constraint from that PRD applies verbatim here: a broken automation that silently spends money is worse than no automation. The supervisor must be safe before it is clever. 3. Users Same population as triggers: solo freelancers and small agencies. What they add to the picture: They are not on-call. A failure at 3am must resolve itself or wait politely until morning — never spend unboundedly trying. They don’t read logs. Whatever the supervisor did must be legible in one glance from the run page or a notification. They trust the graph. Anything the supervisor changes about an execution must be visibly marked, or trust in every green checkmark evaporates. 4. Use cases Ten failure scenarios were reasoned through against three reference workflows; these shaped the requirements below. 4.1 Reference workflows # Workflow What it exercises A 200 URLs → Fetch → Extract → Summarize → Collect → CSV fan-out, cheap items, unattended, cost per item ≈ nothing B Prompt → Enhance → Image gen ($) → Quality check → Upscale ($) → Publish expensive sequential steps, quality gates, retries cost real money C Nightly: List docs → Chunk (streaming) → Embed → Upsert → Notify streaming, external side effects, nobody watching 4.2 Scenarios and what they demand # Scenario Requirement it produced 1 Transient 429 mid-batch (A) Provider-level backoff handles this below the supervisor. The supervisor wakes only when a node’s own error handling is exhausted. 2 7 of 200 items fail the same way (A) One decision, applied to the rest. First failure wakes the agent; its verdict can stick for matching failures on that node (same error signature), so failures 2–7 resolve instantly and free — while a different error on the same node still gets its own decision. 3 Skipped items vanish silently (A) Skips are always audited. The run ends “Completed — supervised”, never plain “Completed”, and the report says exactly what was skipped. 4 Repair produces a wrongly-shaped value (B) Repaired values are type-checked against the node’s declared outputs before entering the graph; invalid repairs bounce back to the agent. 5 Retry after money was spent or an external write happened (B, C) Retry exists only for steps known safe to redo: the node type opts in, and the failed attempt recorded no cost and wrote no asset. A step nobody classified gets no retry — external writers (publish, upsert, notify) can complete their side effect and then fail without recording anything, so “unknown” must mean “unsafe.” The agent can’t double-charge or double-post. 6 Streaming node fails mid-stream (C) Retry only if nothing was emitted yet; Repair never. Otherwise the choice is Stop or end-the-stream. 7 One bad repair cascades downstream (A) Decisions carry lineage. The report shows one cause, not twelve symptoms, and attributes downstream failures to the verdict that caused them. 8 The supervisor itself hangs or dies Escalation has a timeout and honors Cancel. Any supervisor failure resolves as Stop. Cancel always works, instantly. 9 Tuesday’s run behaves differently from Monday’s (C) Any run where a verdict fired is marked supervised in the run record. Deterministic runs stay visibly deterministic. 10 Quality gate fails, root cause is upstream (B) An Assert node’s value can be repaired in place (it’s a pass-through). “Re-run the producer with a better prompt” is out of scope — expressible as a graph loop, not a supervisor power. 5. Product principles Silent on success. A clean run with supervision on is byte-for-byte the run you’d get with it off, at zero added cost. The supervisor is a smoke detector, not a co-pilot. Four verbs, user words. Retry, Repair, Skip, Stop. No configuration matrix, no policy DSL. If a behavior can’t be explained in one of these four words, it doesn’t ship. Never silent, never sneaky. Every intervention is visible in the run status, on the node, and in the report. “Completed — supervised” is a distinct state the user learns to read in a week. Money is sacred. The supervisor can never cause a second charge or a duplicate external write for work already done, and its own spending sits under a hard dollar cap the user can see. The graph is the policy. Want a quality bar? Add an Assert node. Want a fallback? Draw the branch. The supervisor handles the failures you didn’t plan for; the ones you can foresee belong in the graph where they’re visible and editable. 6. User experience 6.1 Turning it on One toggle, next to the Run button: Supervisor. The end state: on by default for triggered (unattended) runs, off by default for interactive runs — when you’re watching, you are the supervisor. Until the evaluation and data-safety gates pass (§10 phase 4), it ships off by default everywhere — and when the default flips, it applies to newly created triggers only; existing triggers are never switched on silently. Turning it on is also a data decision: failure context — the failing step’s inputs and error, and, if the agent digs deeper, earlier outputs belonging to the same failing item — is sent to the supervisor model, with secrets masked. The toggle’s help text says so in one sentence. Expanding the toggle shows four settings: Setting Default Meaning Model workspace default which model makes the calls Max decisions 10 after this, remaining failures fail normally Max retries per step 2 per failed step, across all verdicts Cost cap $0.50 hard ceiling on the supervisor’s own spending, checked before every call No per-node configuration. No policy editor. (Deliberately — see §9.) 6.2 During a run (editor) A failed-then-handled node shows a shield badge in place of the error badge, colored by verdict: retry (blue), repair (amber), skip (gray). While the agent is deciding, the node pulses with a “supervisor deciding…” state — distinct from “running.” Typical decision time is seconds; work that doesn’t depend on the failed node keeps executing (its own downstream naturally waits). A slim intervention feed appears in the run panel only when the first intervention happens: one line per decision, in plain language. “Summarize failed on item 147 (empty response) → retried → succeeded.” Cancel behaves exactly as today. A pending decision is abandoned, not awaited. 6.3 After a run Terminal states, in the order a user scans them: State Meaning ✅ Completed clean run, fully deterministic, same as today 🛡️ Completed — supervised finished, but the agent intervened; the report says where ❌ Failed the agent judged it unrecoverable (Stop), or bounds were hit — with a one-sentence reason written by the agent, not a stack trace ⏹️ Cancelled as today The run report gets an Interventions section: what failed, what the agent saw, what it decided, what it cost, and — for skips — precisely which items are missing from the output (“193 of 200 URLs in the CSV; 7 skipped: …”). One click expands any decision to the raw error and the repaired value. For triggered runs, the notification line carries the state: “Nightly ingest: completed — supervised (2 retries, $0.03).” A user who reads only notifications still knows everything material. 6.4 CLI nodetool run workflow.json --supervise nodetool run workflow.json --supervise --max-decisions 5 Interventions print inline as they happen, prefixed ⛨: ⛨ fetch_url [item 147/200] failed: timeout → retry (1/2) → ok ⛨ extract [item 152/200] failed: malformed HTML → skip (verdict applies to node) ── run completed (supervised): 198/200 items, 2 skipped, 3 decisions, +$0.02 supervisor cost --json reports include the full intervention record. nodetool debug gains --supervise so the edit→verify loop can exercise recovery paths. 6.5 Workflows as agents (chat and API) An agent-facing entry point runs a workflow as an agent: same streaming interface, same result contract as Agent.execute(), no planning phase — the graph is the plan. In chat, “run my ingest workflow and deal with any hiccups” does exactly what it says, and the interventions render as the same feed cards chat already uses for agent steps. 6.6 The four verbs, as the user meets them Verb Report line reads like When the agent may use it Retry “retried → succeeded on the second attempt” transient-looking failures, and only on steps known safe to redo — never after the step spent money or wrote anything, never on steps that weren’t classified as redo-safe. The step reruns exactly as it was; the supervisor can’t quietly change what it does Repair “provider returned broken JSON; supervisor extracted the valid fields” only when the failing step handed over the broken value — repair fixes what exists, it never invents a plausible output from nothing; the repaired value must pass validation Skip “skipped item 147 (page requires login)” the item is dispensable and the run is more valuable finished; always itemized in the report Stop “stopping: the API key is invalid, so all remaining items would fail” continuing would waste money or produce garbage One boundary case wears Skip’s clothes: a streaming step that fails partway is ended early, keeping what it already produced — reported as “skipped the rest of this step, kept the 400 chunks it had finished.” Under the hood this is a distinct action (the run record says so), but as a user verb it stays Skip: something was given up, and the report itemizes exactly what. Skip is the verb the agent is instructed to distrust most: silent data loss dressed up as success is the one outcome this feature must never produce. 7. Non-goals No plan changes. The supervisor never adds, removes, or rewires nodes. The graph the user drew is the graph that runs. No per-node policy configuration. Foreseeable failure handling belongs in the graph (Assert nodes, branches), not in a settings matrix. No cross-node repair. “Retry the producer with a better prompt” is a graph loop, not a supervisor power. No human-in-the-loop pauses in v1. The existing plan-approval machinery is the natural extension point later. No replacement for provider retry. Backoff for rate limits lives in the provider layer; if it’s missing there, it gets built there first. 8. Success metrics Recovery rate: ≥60% of runs that would have failed today complete (fully or supervised) with supervision on — measured by the nodetool eval supervisor suite (graphs with injected faults), gated in CI like the other eval suites. Silence on success: 0 LLM calls, 0 added latency on clean runs. Measured, not assumed. False-wake rate: <5% of decisions are ones a deterministic rule would have made identically (e.g. retrying an obvious transient) — the signal that layering below the supervisor is right. Trust proxy: skips per completed-supervised run trend toward itemized-and-small; a run reporting success while missing un-itemized output is a P0 bug, full stop. 9. Open questions Default-on for triggered runs — right call, or should the first supervised recovery be opt-in with a “turn this on for future runs?” prompt after showing the user what it would have done (a dry-run verdict on the failed run’s report)? Verdict stickiness across runs. Scenario 2 makes verdicts stick within a run. Should a nightly trigger remember last night’s verdict for the same failure signature? Powerful for cost, risky for drift. Where does the supervisor’s cost land in nodetool costs — attributed to the run, or broken out as a separate line? 10. Phasing Phase Ships User sees 1 Escalation hook in the kernel, fail-closed default; audited statuses nothing changes; groundwork only, behavior-preserving 2 Supervisor agent + four verbs; CLI --supervise; intervention record in --json CLI users get the full feature 3 Editor toggle, shield badges, intervention feed, run-report section (all opt-in) the product surface 4 Workflows-as-agents entry point; chat integration; eval suite in CI; then trigger default-on, gated on the eval bar and the data-safety checklist “run it and deal with hiccups”--- # Provider Transient-Retry Audit Source: https://docs.nodetool.ai/workflow-supervisor-provider-retry-audit.html Provider Transient-Retry Audit Status: findings only — fixes land in packages/runtime, not in the supervisor. Last updated: 2026-08-01 Design: workflow-supervisor-design.md §10.3 Plan: workflow-supervisor-implementation-plan.md PR 1 The supervisor wakes only when a node’s own error handling is exhausted (PRD scenario 1). That premise is load-bearing: a provider without backoff turns every 429 into an LLM decision costing real money, on a failure a 2-second wait would have fixed. This audit establishes where the premise holds today. Where retry lives OpenAICompatClient.post (packages/runtime/src/providers/openai-compat/client.ts:114) is the only shared mechanism: 2 retries over 408/409/429/5xx and network errors, jittered exponential backoff 500ms → 8s, Retry-After honored and bounded to 60s. Every OpenAI-compatible provider inherits it, and no subclass overrides the budget. The vendor SDKs supply their own: openai and @anthropic-ai/sdk default to maxRetries = 2, replicate to 5. NodeTool overrides this in exactly one place — anthropic-provider.ts:470 disables SDK retry for models.list and hand-rolls a better loop for it. Inference paths keep the defaults. Three files re-implement fetchWithRetry and Retry-After parsing independently (openai-compat/client.ts:51, topaz-provider.ts:60, atlascloud-provider.ts:73, anthropic-provider.ts:62). Gaps Provider Transient retry Mechanism openai, codex, anthropic yes vendor SDK, maxRetries = 2 the OpenAI-compat family (groq, mistral, xai, openrouter, together, vllm, lmstudio, …) chat only OpenAICompatClient.post atlascloud, topaz yes compat client + local fetchWithRetry replicate partial SDK; non-GET retries 429 only claude agent sdk opaque SDK subprocess retries; api_retry events unobserved gemini, ollama none raw fetch cohere, jina, voyage, huggingface, elevenlabs, fal, kie, reve, meshy, rodin none raw fetch / SDK without retry Two structural holes beyond the table: No provider resumes a stream that dies after the first byte (openai-compat/client.ts:107). Retry covers connection setup only, so design §5.4’s “threw after emitting” case is permanently non-retryable below the hook — which is why end_stream exists rather than a retry. base-provider.ts:1369 isRateLimitError is dead code — tested, never called. Nothing classifies a 429 before it reaches the escalation path, and generateLoop (base-provider.ts:844) has no rate-limit handling at all. Follow-up work (runtime, not supervisor) Extract the compat client’s retry into one exported helper and re-point the three duplicate fetchWithRetrys and four Retry-After parsers at it. Wrap Gemini’s and Ollama’s chat entry points in it. These are first-class chat providers reachable by the supervisor, and today every 429 from them becomes an escalation. Extend to the embedding and media providers with none, keeping non-idempotent POSTs excluded the way atlascloud-provider.ts:188 documents. Use or delete isRateLimitError. Surface retry attempts as telemetry — including the agent SDK’s api_retry — so a supervised run can tell “waited and recovered” from “failed at once”.--- # Examples Source: https://docs.nodetool.ai/workflows/ NodeTool ships examples for every editor, not only workflows. Open the Examples page from the More panel, or choose Browse examples on the new-project screen. Each tab adds an editable copy to your project, so the shipped original never changes. The Templates Gallery covers the page itself. Tab What you get Opens in Apps A screen with inputs and a Run button, plus the workflows behind it Mini Apps Workflows A node graph you can run and edit Workflow editor Storyboards A board of shots with briefs, ready to render Storyboard Timelines A finished film with every clip, text layer, and keyframe editable Video editor Sketches A 1200 × 900 image document with five vector layers Sketch editor 3D models Low-poly glTF models in six packs 3D editor Games A playable 2D or 3D game Game editor To make something new from a brief rather than open an example, see the Cookbook. It names the guided flow or shipped skill for each kind of project. Apps Add to your apps installs the app and the workflows it runs. The apps fall into two groups. Studios put a whole production chain behind one screen. App What it does Film Studio One brief drives a directed short, an editable rough cut, and its poster. Trailer Room Rewrite the beat sheet and shot list, then render and cut the trailer. Concept Studio Fan a brief into concept art, mix creatures, and develop the pick. Directed Campaign Kit One product image becomes a brief, three directions, and campaign formats. Product Launch Kit Stage a product photo into lifestyle mockups, then into video. SKU Factory One product photo becomes a cutout, studio shots, and a listing set. Viral Ad Engine One offer drives the ad copy, the hooks, and the video ad. UGC Product Video A native-audio testimonial with captions and a branded close. Ad Maker Compare copy routes and headline angles, then direct a campaign hero. Product Price Drop A layered six-second vertical ad from your product, logo, and prices, with no generated video. Dubbing Desk Transcribe a presenter clip, translate it, and voice it in a new language. AI Video Post House Restyle, grade, trim, and pull a cover still from one uploaded clip. Podcast Production Desk Clean a recording, then write show notes, a newsletter, and social posts. Brand & Social An agent reads your website and fills in your brand. Photo Studio A live photo editor and a batch retoucher. Vary Image Change one thing in an image, such as lighting or pose, and keep the rest. Model Arena Three models answer side by side, and a judge model scores them. Ad formats are sixteen apps that each build an editable 8 to 15 second vertical ad from your images, logo, and copy. Examples include Kinetic offer wall, Product cutout shuffle, Editorial image panels, and Integration puzzle that snaps together. The ad library shows each format with a beat sheet and a sample cut. Recipes A recipe is a named outcome built from one of the apps above and the ordered workflows it runs. Each recipe page on the website walks through the steps and shows a real run. Recipe Outcome Directed campaign kit One product image to a campaign in four formats UGC product video A 15-second creator testimonial with captions Product photo to video ad Compared copy and thumbnails, then an animated product shot Dub your video for a new market A translated, re-voiced version of your clip One packshot, a full product set Catalogue and campaign images from one photo Turn your story into a trailer A storyboard, voiceover, and cut trailer Impossible product worlds A three-shot commercial around one impossible setting Workflows The Workflows tab lists graph templates by category: Image, Video, Audio, Multimodal, Agents, and Data & Web. The showcase graphs below have their own walkthroughs. Workflow Walkthrough Movie Trailer Generator Use case Product Video Generator Use case Movie Posters Use case Podcast Repurposing Studio Use case SEO Content Engine Use case Direct a Short Film, Three Ratios, Localized Explainer, and other creative graphs Workflow patterns Workflow walkthroughs These smaller graphs each have a node-by-node walkthrough and a diagram. Most load from the Workflows tab. Categorize Mails, Creative Story Ideas, Fetch Papers, and Summarize RSS do not ship as templates, and each page says so at the top. Kind Walkthroughs Image Movie Posters, Image Enhance Video Color Boost Video Audio and voice Transcribe Audio, Image To Audio Story Writing Creative Story Ideas, Flashcard Generator Documents Chat with Docs, Fetch Papers Automation Meeting Transcript Summarizer, Categorize Mails, Summarize RSS Data Data Generator Storyboards Each board holds a brief, a style, and a shot list, ready to render stills and then clips. Most are single-moment commercials and film scenes, such as Cold Brew Pour, Puddle Strike, Silent Arrival, Fox in Snow, and The Interrogation. Three are multi-shot openings to explore: Sneaker Drop, First Light, and Lighthouse Keeper. The storyboard gallery shows each board with its rendered shots. AI Video Production covers rendering and assembling a board. Timelines Each timeline is a finished film in which every clip, text layer, and keyframe stays editable. Film What it shows Serein A 26-second software launch film from scene clips, text, and motion Kite A 15-second app ad made from motion graphics alone Tidewater A 16-second animated risograph poster Cadence A vertical year-in-review data story with animated charts Prism An 18-second running-shoe ad on a 120 BPM grid Voltra A 23-second motorcycle launch ad cut to the beat T minus 30 A 30-second launch broadcast with moving cameras and a live inset The agent can also insert stored compositions, such as a lower third, a logo sting, a title card, or an end card, into any timeline. See Timeline Motion Design. Sketches Sketches are layered image documents, filtered by category: NodeTool, Ads, Movies, and Art studies. Open sketch creates your own copy. They need no model or API key to open. See Sketch Editor. 3D models The 3D models tab groups low-poly glTF models into six packs: Adventure, Dungeon, Platformer, Seaside, Sci-fi, and Sci-fi outpost. Open model copies the model into your project and opens it in the 3D editor. Games Play and edit adds a copy you can play in the editor and change. Game Genre Kindle A jump-and-run through a drowned temple Lumen A firefly flight through a night forest Neon Drift A neon arena shooter AETHER // Skybound A platform climb across floating islands BLACKSITE A first-person mission through a reactor facility The game development page shows Kindle, Lumen, and Neon Drift. See Game Editor. More examples Use CasesFlagship projects end to end: trailers, product videos, ad batches, posters. CookbookWhere to start for each kind of project, and when to turn it into a workflow. ComfyUI RecipesGraphs that run on a ComfyUI backend. Shipped SkillsThe procedures the agent follows for each kind of project.--- # Categorize Mails Source: https://docs.nodetool.ai/workflows/categorize-mails.html Overview Classifies emails into categories (Newsletter, Work, Family, Friends) with an LLM and applies the matching Gmail label. This tutorial does not ship as an importable template. Build it manually by following the steps below. It needs the Google Workspace integration, which is available only on installs with a login (Supabase auth mode) or with NODETOOL_GOOGLE_WORKSPACE=1, and a connected Google account. See Google Workspace and Email. Demo Workflow Steps Gmail Search (nodetool.code.Code) - Imports gmail_search from @nodetool-ai/sandbox-nodetool/google. It takes a Gmail query (for example is:unread newer_than:7d) and max_results (1 to 50, default 10), and returns messages with subject, sender, date, and body. Prompt (nodetool.text.Prompt) - Formats each email into a prompt with subject, sender, and body. Classifier (nodetool.agents.Classifier) - Picks one of the categories you list. It takes text and a model. Add Label (nodetool.code.Code) - Imports gmail_modify_labels and calls it with the message’s message_id and add_label_ids. Label ids come from gmail_list_labels. Tags email, start Workflow Diagram graph TD search["Code (gmail_search)"] prompt["Prompt"] classifier["Classifier"] label["Code (gmail_modify_labels)"] search --> prompt prompt --> classifier classifier --> label search -->|message_id| label--- # Chat with Docs Source: https://docs.nodetool.ai/workflows/chat-with-docs.html Overview Retrieval-augmented question answering over your own documents. The template indexes three short documents into a vector collection, retrieves passages, and answers from those passages only, citing the source of each claim. It is the “Chat With Your Documents” template under Examples. Inputs - question, search (the text used for retrieval), and three document texts (doc_specs, doc_charging, doc_warranty), all nodetool.input.StringInput. Collection (vector.Collection) - A collection named aurora_docs that uses the Ollama nomic-embed-text embedding model. Run Ollama with that model available. Indexing (vector.IndexTextChunk, three nodes) - Each document goes into the collection with a document_id and a source metadata entry. Retrieval (vector.QueryText) - Queries the collection with the search text and returns the top 5 documents. Context - A Code node joins the passages, and a nodetool.text.Prompt combines them with the question. Answer (nodetool.agents.Agent) - Answers strictly from the passages, cites the passage heading after each claim, and replies “I couldn’t find that in the provided documents.” when the answer is absent. The template selects gpt-5-mini. Outputs - Answer and Retrieved Passages. The search input is separate from the question. The shipped value, Aurora, appears in every document, so retrieval returns the whole corpus. Narrow it, for example to warranty, to retrieve one topic. Replace the three document inputs to use your own corpus. Demo Tags rag, vectorstore, retrieval, llm, citations Workflow Diagram graph TD collection["Collection (aurora_docs)"] docs["StringInput x3 (documents)"] index["IndexTextChunk x3"] search["StringInput (search)"] query["QueryText"] join["Code (join passages)"] question["StringInput (question)"] prompt["Prompt"] agent["Agent"] answer["Output (Answer)"] passages["Output (Retrieved Passages)"] docs -->|text| index collection --> index collection --> query search -->|text| query query -->|documents| join join -->|CONTEXT| prompt question -->|QUESTION| prompt prompt --> agent --> answer join --> passages--- # Color Boost Video Source: https://docs.nodetool.ai/workflows/color-boost-video.html Overview Splits a video into frames, grades each frame, and reassembles the frames into a new video. A Grading Intensity input drives the saturation pass, so you can bind it to a slider in the app view. Video Input (nodetool.input.VideoInput, named source_video) - The clip to grade. The template ships with a short demo clip. Frame Iteration (nodetool.video.ForEachFrame) - Extracts frames with ffmpeg and emits frame, index, and fps. Start and End are frame numbers and End is inclusive. The template sets End to 2, so only three frames are graded. Set End to -1 to grade the whole clip. Exposure (lib.image.color_grading.Exposure) - Applies exposure 0.2, contrast 0.1, highlights -0.1, and shadows 0.1. Saturation and Vibrance (lib.image.color_grading.SaturationVibrance) - Vibrance is 0.4. Its saturation property comes from the Grading Intensity input (nodetool.input.FloatInput, range -1 to 1, default 0.5). Reassembly (nodetool.video.FrameToVideo) - Recompiles the graded frames at the fps that ForEachFrame reports, and sends the result to the graded_video output. Demo Tags video, start Workflow Diagram graph TD video["VideoInput (source_video)"] intensity["FloatInput (grading_intensity)"] frames["ForEachFrame"] exposure["Exposure"] saturation["SaturationVibrance"] assemble["FrameToVideo"] out["Output (graded_video)"] video --> frames frames -->|frame| exposure exposure --> saturation intensity -->|saturation| saturation saturation --> assemble frames -->|fps| assemble assemble --> out--- # Creative Story Ideas Source: https://docs.nodetool.ai/workflows/creative-story-ideas.html Overview A beginner workflow that fills a prompt from three inputs and streams story ideas one at a time. This tutorial does not ship as an importable template. Build it manually by following the steps below. How it works: Set inputs - Three nodetool.input.StringInput nodes: Genre, Character Type, and Setting. Fill a prompt - A nodetool.text.Prompt node holds text such as Give five one-line story ideas in the {{ genre }} genre featuring {{ character }}. Each {{ name }} becomes an input on the node that you connect to a StringInput. Generate ideas - A nodetool.generators.ListGenerator sends the prompt to the language model you select and streams one idea per item. Key concepts: Connect nodes to create workflows A Prompt node turns {{ variables }} into inputs, so one wording serves many runs ListGenerator streams its item output, so downstream nodes start before the list is complete Demo Tags start, beginner, tutorial, creative, writing Workflow Diagram graph TD genre["StringInput (Genre)"] character["StringInput (Character Type)"] setting["StringInput (Setting)"] prompt["Prompt"] ideas["ListGenerator"] genre -->|genre| prompt character -->|character| prompt setting -->|setting| prompt prompt --> ideas How to Use Add the nodes above and connect them as in the diagram Select a language model on the ListGenerator node Fill in the input nodes: Genre: “Sci-fi”, “Fantasy”, “Mystery”, etc. Character Type: “Hero”, “Villain”, “Detective”, etc. Setting: “Space station”, “Medieval castle”, “Future city”, etc. Press Ctrl/⌘ + Enter or click Run Add a Preview node on the item output to watch the ideas arrive Tips: Run multiple times to get different ideas Try unusual combinations for unique results Next Steps Movie Posters - Generate images from ideas Image Enhance - Chain image transformations--- # Data Generator Source: https://docs.nodetool.ai/workflows/data-generator.html Overview Generates a structured dataset from a one-line topic. You define the columns once on the DataGenerator node and get validated rows back instead of prose. Inputs - topic (nodetool.input.StringInput) and row_count (nodetool.input.IntegerInput, 3 to 30, default 8). Prompt (nodetool.text.Prompt) - Fills {{TOPIC}} and {{ROW_COUNT}} into the generation instructions. DataGenerator (nodetool.generators.DataGenerator) - Asks the selected language model for rows that match the columns. The template defines two string columns, name and color, and selects gpt-5-mini. Edit the columns to change the shape. Output - The dataframe output goes to generated_data. The node also streams each row as a record output. Demo Tags agents Workflow Diagram graph TD topic["StringInput (topic)"] count["IntegerInput (row_count)"] prompt["Prompt"] generator["DataGenerator"] out["Output (generated_data)"] topic -->|TOPIC| prompt count -->|ROW_COUNT| prompt prompt --> generator generator -->|dataframe| out--- # Fetch Papers Source: https://docs.nodetool.ai/workflows/fetch-papers.html Overview Fetches the README of a GitHub list of research papers, keeps the links that point at PDFs, and downloads each PDF into the run workspace. This tutorial does not ship as an importable template. Build it manually by following the steps below. NodeTool has no HTTP request or download node, so both network steps are Code nodes that call the sandbox fetch. Demo Workflow Steps Get README (nodetool.code.Code) - Calls fetch(url) on the raw README URL and reads text(). The same body pulls the link targets that end in .pdf and returns them as a list. Use the @nodetool-ai/sandbox-html pack’s extractLinks instead when the source is an HTML page. For Each (nodetool.control.ForEach) - Emits one URL at a time from input_list. Download (nodetool.code.Code) - Fetches each URL and saves it with workspace.writeBytes(name, await res.bytes()). Collect (nodetool.control.Collect) - Gathers the saved file names into a list for a Preview node. fetch refuses loopback, link-local, and private addresses. See the JavaScript Sandbox reference for its limits. Tags automation Workflow Diagram graph TD readme["Code (get README, keep PDF links)"] each["ForEach"] download["Code (download)"] collect["Collect"] preview["Preview"] readme --> each each -->|output| download download --> collect collect --> preview--- # Flashcard Generator Source: https://docs.nodetool.ai/workflows/flashcard-generator.html Overview Generates study flashcards as structured rows, then computes a study plan from them: no repeated questions, categories interleaved, and a review schedule per card. Inputs - topic (nodetool.input.StringInput) and num_cards (nodetool.input.IntegerInput, 3 to 15, default 5). Prompt (nodetool.text.Prompt) - Fills {{TOPIC}} and {{NUM_CARDS}} into the card-writing instructions. DataGenerator (nodetool.generators.DataGenerator) - Returns rows with front, back, and category columns. The dataframe output goes to the Flashcards output and to the next step. Code (nodetool.code.Code) - Drops repeated questions, deals one card from each category in turn, and assigns each card review_after_days of [0, 2, 5]. Outputs - Flashcards and study_plan. Demo Tags education, structured-data, ai, flashcards, learning Workflow Diagram graph TD topic["StringInput (topic)"] count["IntegerInput (num_cards)"] prompt["Prompt"] generator["DataGenerator"] plan["Code (study plan)"] cards["Output (Flashcards)"] planOut["Output (study_plan)"] topic -->|TOPIC| prompt count -->|NUM_CARDS| prompt prompt --> generator generator -->|dataframe| cards generator -->|dataframe| plan plan -->|study_plan| planOut--- # Image Enhance Source: https://docs.nodetool.ai/workflows/image-enhance.html Overview A photo-editing chain of four filters. Each filter exposes numeric properties that the mini app binds to sliders. The template uses no model and no API key. The pipeline: Image Input (nodetool.input.ImageInput, named image) - Your photo. The template ships with a demo image so Run works at once. Denoise (lib.image.filter.GaussianBlur) - Radius and sigma start at 0, so it changes nothing until you raise them. Brightness and Contrast (lib.image.color.BrightnessContrast) - Brightness 0.05, contrast 1.15. Saturation (lib.image.color.HSB) - Hue 0, saturation 1.2, brightness 1. Sharpen (lib.image.filter.UnsharpMask) - Amount 1, threshold 0.02. Output - The enhanced_image result. Reorder the filters or add another from the lib.image namespace, and the app picks up the new controls. Demo Tags image, start, example Workflow Diagram graph TD image["ImageInput (image)"] denoise["GaussianBlur (Denoise)"] tone["BrightnessContrast"] color["HSB (Saturation)"] sharpen["UnsharpMask"] enhanced["Output (enhanced_image)"] image --> denoise --> tone --> color --> sharpen --> enhanced How to Use Open NodeTool, click Examples in the app menu, and load “Image Enhance” Click the Image Input node and upload a photo Adjust the filter properties on each node Press Ctrl/⌘ + Enter or click Run Tips: Raise Denoise before Sharpen on a noisy photo More sharpness isn’t always better Next Steps Movie Posters - Combine enhancement with AI generation Color Boost Video - Apply similar enhancements to video--- # Image To Audio Story Source: https://docs.nodetool.ai/workflows/image-to-audio-story.html Overview Turns an image into a narrated story. A vision agent writes the story and a text-to-speech model reads it aloud. How it works: Image Input (nodetool.input.ImageInput, named image) - Your photo, artwork, or any visual Agent (nodetool.agents.Agent) - Receives the image on its image input and writes a 150 to 250 word story with a title line. The template selects gpt-5-mini. Text-to-Speech (nodetool.audio.TextToSpeech) - Reads the agent’s text output. The template selects OpenAI tts-1 with the alloy voice. Output - The narration audio you can save or share Prompt examples: “Describe this image as if you’re a museum curator” “Write a short poem inspired by this artwork” “Create a brief backstory for this scene” Demo Tags start, multimodal, example Workflow Diagram graph TD image["ImageInput (image)"] agent["Agent"] tts["TextToSpeech"] narration["Output (narration)"] image --> agent agent -->|text| tts tts -->|audio| narration How to Use Open NodeTool, click Examples in the app menu, and load “Image To Audio Story” Load your image Edit the prompt in the Agent node (the default asks for a short story with a beginning, middle, and end) Choose your voice in the TextToSpeech node Press Ctrl/⌘ + Enter to run Related Workflows Movie Posters - Create visual content Creative Story Ideas - Generate story concepts--- # Meeting Transcript Summarizer Source: https://docs.nodetool.ai/workflows/meeting-transcript-summarizer.html Overview Summarizes a meeting from a recording or from a pasted transcript, and returns action items as table rows instead of prose. Inputs - Recording (nodetool.input.AudioInput) and Transcript (nodetool.input.StringInput, multiline). Fill in either one. The template ships with a sample recording, so clear it when you paste a transcript. Routing - A Code node tests whether the pasted transcript is empty. Two If nodes pick the transcribed audio when it is empty and the pasted text otherwise. A second Code node joins the chosen text. Transcription (nodetool.text.AutomaticSpeechRecognition) - Transcribes the recording with openai/whisper-large-v3 on fal.ai. Summary (nodetool.agents.Summarizer) - Writes a Markdown summary of about 150 words with Overview, Key discussion points, and Open questions. Action items (nodetool.text.Prompt into nodetool.generators.DataGenerator) - Returns a table with action, owner, and due_date columns. Outputs - Transcript, Summary, and Action Items. Demo Tags audio, llm, dataframe Workflow Diagram {% mermaid %} graph TD recording[“AudioInput (Recording)”] pasted[“StringInput (Transcript)”] asr[“AutomaticSpeechRecognition”] empty[“Code (is empty)”] useAudio[“If (use transcribed audio)”] usePasted[“If (use pasted transcript)”] join[“Code (join)”] summarizer[“Summarizer”] prompt[“Prompt (action items)”] generator[“DataGenerator”] transcriptOut[“Output (Transcript)”] summaryOut[“Output (Summary)”] actionsOut[“Output (Action Items)”] recording –> asr pasted –> empty empty –> useAudio empty –> usePasted asr –> useAudio pasted –> usePasted useAudio –> join usePasted –> join join –> summarizer –> summaryOut join –> prompt –> generator –> actionsOut join –> transcriptOut {% endmermaid %}--- # Movie Posters Source: https://docs.nodetool.ai/workflows/movie-posters.html Overview A poster generator. An art-direction step writes a creative strategy, then an image model renders one poster per concept. How it works: Inputs - Three StringInput nodes: Movie Title, Genre, and Visual Style. Strategy prompt (nodetool.text.Prompt) - Frames the art-director brief from the three inputs. Structured output (nodetool.generators.StructuredOutputGenerator) - Returns four fields: positioning, audience_insight, design, and core_visual_concept. Concept list (nodetool.text.Prompt into nodetool.generators.ListGenerator) - Writes one line per poster concept, using the title, genre, positioning, and audience insight. Each line streams out as item. Poster prompt (nodetool.text.Prompt) - Combines each concept with the genre, title, design, and core visual concept. Image generator (nodetool.image.TextToImage) - Renders each poster at 3:4 and 2K. The template selects fal-ai/flux/schnell. Output - The Poster output receives each rendered poster. Demo Tags start, image, example Workflow Diagram {% mermaid %} graph TD title[“StringInput (Movie Title)”] genre[“StringInput (Genre)”] style[“StringInput (Visual Style)”] strategyPrompt[“Prompt (strategy)”] strategy[“StructuredOutputGenerator”] pitch[“Prompt (concepts)”] list[“ListGenerator”] posterPrompt[“Prompt (poster)”] image[“TextToImage”] out[“Output (Poster)”] title –> strategyPrompt genre –> strategyPrompt style –> strategyPrompt strategyPrompt –> strategy strategy –>|positioning, audience_insight| pitch title –> pitch genre –> pitch pitch –> list list –>|item| posterPrompt strategy –>|design, core_visual_concept| posterPrompt title –> posterPrompt genre –> posterPrompt posterPrompt –> image –> out {% endmermaid %} How to Use Open NodeTool, click Examples in the app menu, and load “Movie Posters” Fill in your movie details: Movie Title: “Singularity” (or your movie name) Genre: “Sci-Fi Thriller” (choose any genre) Visual Style: “Neo-noir, high-contrast, cinematic” Press Ctrl/⌘ + Enter or click Run View the posters in the Poster output Tips: Try different genres for the same title to see how styles change Be specific about the visual style to get more targeted designs Next Steps Image Enhance - Enhance your poster designs Creative Story Ideas - Generate movie concepts--- # Summarize RSS Source: https://docs.nodetool.ai/workflows/summarize-rss.html Overview Fetches an RSS feed, gathers its entries into one text, and summarizes it. This tutorial does not ship as an importable template. Build it manually by following the steps below. NodeTool has no RSS node, so the fetch is a Code node. Demo Workflow Steps Fetch feed (nodetool.code.Code) - Calls fetch(url) on the feed URL and parses the XML with parse from @nodetool-ai/sandbox-xml. Read the entries from feed.rss.channel.item and wrap a single entry with [].concat(x ?? []), because the parser returns one object for a one-item feed. Emit each entry’s title and description as one string. Collect (nodetool.control.Collect) - Gathers the streamed entries into a list. Join (nodetool.code.Code) - Joins the list into one string, because the Summarizer’s text input takes a string. Summarizer (nodetool.agents.Summarizer) - Condenses the text with the language model you select. Tags rss, llm Workflow Diagram {% mermaid %} graph TD fetch[“Code (fetch and parse feed)”] collect[“Collect”] join[“Code (join)”] summarizer[“Summarizer”] fetch –> collect collect –> join join –> summarizer {% endmermaid %}--- # Transcribe Audio Source: https://docs.nodetool.ai/workflows/transcribe-audio.html Overview Convert speech to text with Whisper. The workflow returns plain text, with no timestamps. Audio Input (nodetool.input.AudioInput) - Record your voice or upload an audio file Automatic Speech Recognition (nodetool.text.AutomaticSpeechRecognition) - Transcribes the audio. The template selects openai/whisper-large-v3 on fal.ai. The node also takes a language code (empty detects it), a guiding prompt, and a temperature. Output - The transcript text For timed segments or words, use openai.audio.Transcribe, which returns text, segments, and words. Demo How to Use Record your voice or upload a file using the audio input Click Run Read the transcript in the output Tags start, audio, asr Workflow Diagram {% mermaid %} graph TD audio[“AudioInput (audio)”] asr[“AutomaticSpeechRecognition”] transcript[“Output (transcript)”] audio –> asr asr –>|text| transcript {% endmermaid %}--- # Workspaces Source: https://docs.nodetool.ai/workspaces.html A workspace is a folder that NodeTool can safely access for reading and writing files during workflow runs, agent tasks, and chat sessions. Why workspaces exist Workspaces make local file access explicit and controlled: Keep project files organized under known roots Limit file operations to approved directories Share the same folder roots across workflows and agent tools Manage workspaces in the app Workspaces have their own full-screen page at /workspaces (not a Settings tab): Open the app menu (click the logo at the top of the left rail). Choose Workspaces. Click Add Workspace and pick a directory. The path must be absolute, exist, be a directory, and be writable. Use the star on a row to make it the default workspace. The default workspace has no remove button. After adding a workspace, NodeTool can browse files, list folders, and read/write files inside the configured root. A row marked Inaccessible points at a folder the server can no longer reach. Workspaces also appear in the Workspace view of the left panel, which browses the current one as a file tree, and under Advanced in the workflow form, which pins a workflow to one. Every user gets a default workspace the first time the list loads. What a cloud deployment allows Pointing a workspace at a host folder browses the server’s filesystem, so NODETOOL_ENV=production refuses create, update, and delete (packages/websocket/src/trpc/routers/workspace.ts). Reading stays open: every user gets a server-managed workspace under the data dir, and listing and downloading from it reach nothing the deployment did not create itself. The server says which of the two applies in can_manage on workspace.list, and the UI reads it through useWorkspaces(). So every workspace surface is present on a cloud deployment, and the only missing control is Add Workspace, which would open a folder picker with nothing to browse. Where workspaces are used Agents and chat tools for file operations The Code node, whose workspace.* API (read, write, list, stat, mkdir, remove, copy, move) is confined to the workspace unless Allow Host Filesystem is on (the cloud profile ignores that switch) API The tRPC workspace router has list, create, update, delete, listFiles, readFile, and writeFile (packages/websocket/src/trpc/routers/workspace.ts). Paths passed to the file procedures are workspace-relative. Absolute paths are refused and traversal answers 403. Binary reads use GET /api/workspaces/:id/download/:path (packages/websocket/src/workspace-api.ts). Related docs Workflow Editor Storage Guide Node Reference: nodetool.code — workspace file access now goes through the Code node’s workspace.* API