Skip to content

Conversation

@BinaryMuse
Copy link
Member

@BinaryMuse BinaryMuse commented Oct 11, 2025

Overview

This PR implements a persistent document context system that fundamentally changes how runbook execution state is managed. Instead of maintaining global state across executions, each block now derives its context from the blocks above it in the document, creating a clear "chain of causality" for execution.

Motivation

Previously, execution context was scoped to a single execution flow (#135), requiring global state management across runs. This made issues like #82 difficult to resolve and created ambiguity about which blocks set contextual data.

The new architecture:

  • Replaces execution-scoped context with document-level context
  • Enables context recovery across tab closes and app restarts (e.g., preserving mktemp -d output)
  • Creates foundation for future features like multiple execution "runs" per runbook
  • Enables potential realtime shared execution by serializing context to collaborators

Architecture

Core Runtime System (backend/src/runtime)

Document & DocumentActor

  • Document contains a vector of Block objects with associated BlockContexts
  • DocumentActor manages document lifecycle and handles commands via message passing
  • Accessed through DocumentHandle which sends DocumentCommands
  • Updates reactively when BlockNote document changes from frontend

Block Execution Model

Replaced the BlockHandler pattern with BlockBehavior trait:

  • async fn passive_context(&self, &ContextResolver, Option<&BlockLocalValueProvider>) -> Result<Option<BlockContext>> — Defines passively set context (cwd, env vars, ssh host, template variables). Uses BlockLocalValueProvider for local-only storage access.
  • async fn execute(self, ExecutionContext) -> Result<Option<ExecutionHandle>> — Executes block using ExecutionContext, consuming self. Returns optional ExecutionHandle for long-running operations.

BlockContext

  • Type map pattern storing context structs (DocumentCwd, DocumentEnvVar, SshHost, etc.)
  • Each block's context = union of all BlockContexts from blocks above it
  • Maintains causality: changes below a block don't affect blocks above
  • Enables predictable, deterministic context resolution
  • Each block has a passive context (set when the document updates) and an active context (set when the block executes). The active context is cleared each time the block beings execution.

ContextResolver

  • Lightweight context reduction for resolving template strings within passive_context()
  • Prevents circular dependencies during context building
  • Provides clean API for template interpolation
┌────────────────────────────────────────────────────┐
│                      Document                      │            ┌─────────────────┐
│                  (DocumentActor)                   │            │ Block Execution │
│                                                    │            └─────────────────┘
│   ┌──────────────────┐      ┌──────────────────┐   │                     │
│   │      Block       │─────▶│   BlockContext   │   │                     │
│   │  (BlockBehavior) │      │                  │   │                     │
│   └──────────────────┘      └──────────────────┘   │                     │
│                                       ▲            │                     ▼
│   ┌──────────────────┐      ┌──────────────────┐   │       ┌──────────────────────────┐
│   │      Block       │─────▶│   BlockContext   │◀──┼───────│     ExecutionContext     │
│   │  (BlockBehavior) │      │                  │   │       │                          │
│   └──────────────────┘      └──────────────────┘   │       └──────────────────────────┘
│                                       ▲            │
│   ┌──────────────────┐      ┌──────────────────┐   │
│   │      Block       │─────▶│   BlockContext   │   │
│   │  (BlockBehavior) │      │                  │   │
│   └──────────────────┘      └──────────────────┘   │
└────────────────────────────────────────────────────┘

Frontend Integration

ClientMessageChannel

  • Message channel for runtime -> frontend communication
  • Sends ResolvedContext updates to frontend per block as document changes
  • Enables reactive UI updates without polling
  • Foundation for realtime collaborative execution

BlockLocalValueProvider

  • Abstraction for blocks that need local-only storage (credentials, local paths)
  • Key-value storage scoped per block and runbook
  • Used by local-var and local-directory blocks
  • Keeps sensitive data out of shared document props

Frontend Pattern

  • Blocks use useBlockContext(blockId) hook to receive resolved context
  • useBlockKvValue() for local-only data via BlockLocalValueProvider
  • Blocks simplified to prop management; runtime handles all context resolution
  • No direct backend state manipulation from UI components

TODO:

  • Add workspace namespace to string templating
  • Add document/block data to string templating
  • Log executions in exec_log

@BinaryMuse BinaryMuse added rust Pull requests that update rust code in-progress internal Internal implementation, neither a feature nor a bug labels Oct 11, 2025
@BinaryMuse BinaryMuse self-assigned this Oct 11, 2025
@BinaryMuse BinaryMuse force-pushed the mkt/realtime-context branch from 852aff0 to 8714476 Compare October 16, 2025 16:37
@BinaryMuse BinaryMuse changed the title Persistent document context chore: Update backend to use a persistent document context for execution Oct 16, 2025
@ellie ellie force-pushed the ellie/backend-execution branch from 4e5e389 to 99fb965 Compare October 16, 2025 20:28
Base automatically changed from ellie/backend-execution to main October 16, 2025 20:43
BinaryMuse and others added 17 commits October 17, 2025 09:21
Add PartialEq and Eq trait implementations to all block types and the
Block enum to enable comparison for change detection in document updates.

HttpResponse and HttpOutput only implement PartialEq (not Eq) due to
their f64 duration field, which cannot implement Eq.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Sculptor <sculptor@imbue.com>
Implement document update logic that efficiently handles insertions,
deletions, moves, and updates in a single pass:

- Drain existing blocks into HashMap for O(1) lookup
- Iterate new blocks once, building final list in correct order
- Detect content changes and position moves
- Track minimum rebuild index for passive context updates
- Handle deletions by checking remaining HashMap entries

Also make rebuild_passive_contexts synchronous by spawning event
emission in a separate tokio task, avoiding blocking.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Sculptor <sculptor@imbue.com>
Update open_document and update_document commands to use the new
DocumentHandle::put_document() method. Remove BlockChange enum
and related code, replacing granular change tracking with full
document state updates.

Update frontend to send complete document state on changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-authored-by: Sculptor <sculptor@imbue.com>
@BinaryMuse BinaryMuse changed the title chore: Update backend to use a persistent document context for execution feat: Introduce new Runbook execution engine Oct 29, 2025
Copy link
Member Author

@BinaryMuse BinaryMuse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ellie Reqwest offers several features for TLS handling, which should we set it to in order to match the behavior of the other TLS stuff?

https://docs.rs/reqwest/latest/reqwest/index.html#optional-features

@ellie
Copy link
Member

ellie commented Nov 1, 2025

native-tls is the one!

Can elaborate later but it was really difficult to get enterprise cert chains working properly with rustls

@BinaryMuse BinaryMuse force-pushed the mkt/realtime-context branch from bd3d3a1 to 299d9a9 Compare November 2, 2025 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

in-progress internal Internal implementation, neither a feature nor a bug rust Pull requests that update rust code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants