Baobab Lacuna · Use-Case 1
LACUNA ENGINEERING · INTERNAL + DESIGN-PARTNER DRAFT

Use-Case 1 — News-Class Retrieval at the Edge

baobab-worker-rs + Cloudflare Workers + Firecrawl + Loam: a one-page-load, audit-grade pipeline for diaspora-scale publishing.

Doc: lacuna-usecase-1 · v0.1-draft · 2026-05-12 · Owner: Lacuna Engineering
Audience: SRE / Architecture review · Design partners (publishers, civic newsrooms)
Classification: technical · neutral · no marketing copy

Contents

  1. Executive summary
  2. The customer
    1. Baobab as the reference deployment
    2. Customer shape beyond Baobab
    3. What the existing market gets wrong
  3. The five substrates, briefly
  4. baobab-worker-rs: the execution layer
    1. Why a Rust worker at all
    2. Two targets, one source
    3. Mailboxes: Durable Objects vs tokio
    4. PlanResult + audit log
  5. Plans as contract: why Scheme
  6. Tool registry & capability model
  7. The plan, line by line
  8. Cloudflare deployment
  9. Fly.io native deployment
  10. baobab-worker-rs vs Firecrawl (and how they compose)
  11. Where Loam fits
  12. Lacuna's value-add surface
  13. Tiering & commercial model
  14. Security model
  15. SRE: SLOs, SLIs, error budgets
  16. Observability & the audit-log discipline
  17. Capacity planning
  18. Incident response & runbooks
  19. Unit economics
  20. Roadmap to general availability
  21. FAQ
  22. Glossary

1. Executive summary

Lacuna ships a stack for publishing-class retrieval: pulling hundreds of structured sources every hour, normalising and enriching the output, and serving a fast, image-rich, audit-traceable surface to a global audience. The execution layer is a Rust worker — baobab-worker-rs — that compiles to two targets (Cloudflare Workers / WASM and a Fly native binary) and runs Scheme plans emitted by the Lacuna brain. Tools are capability-gated. Every side effect is audited. Plans are textual contracts, versionable and replayable.

This document is the SRE-grade view of one use case — the news refresh path that powers Baobab — and treats it as a worked example of the larger Lacuna playbook. The same shape applies to every other content-heavy operator we'd onboard: civic dashboards, vertical newsrooms, market-data sites, internal knowledge portals.

Three claims this document defends:

  1. The worker is the right shape for retrieval at the edge: cheap, fast, safe, and auditable in ways a generic Node-on-Fly box is not.
  2. Firecrawl is a capability, not a competitor. The worker calls Firecrawl when it needs a real browser; the worker itself is the orchestration layer that decides when.
  3. Loam — the world-building cache between crawlers and the LLM — is what keeps the per-operator cost model viable as the surface scales.
Where we are. The crate scaffold compiles. The plan runner, tool registry, mailbox surface, and one canonical Scheme plan (news-refresh.scm) are checked in. The Steel-VM body, the Cloudflare Workers deploy, and the Lacuna Python integration are the remaining work for v1. Baobab today still runs a Node single-file API and a browser-side Web Worker; the Rust worker is the next migration target.

2. The customer

2.1 Baobab as the reference deployment

Baobab (baobab.lacunalabs.ai) is a pan-diaspora portal: news, markets, library, rooms, and a small wallet. It pulls ~150 RSS feeds across five continents, surfaces them in a gazette-style layout, and runs a client-side tagging pass against ~150 diaspora-domain vocabulary terms. It is the canonical first deployment for the Lacuna stack — a real product with a real audience, not a demo.

Production shape, today:

SurfaceHostStack
Static sitebaobab-lacuna.fly.devCaddy + HTML/JS
Node APIbaobab-api.fly.devNode 20, single-file
News taggingbrowserWeb Worker, JS
LLM opsbaobab-llmDeepSeek tool-calling proxy

Baobab works. It also makes a strong case for the Rust worker, because the seams where it strains today — image-proxy fan-out, RSS fan-out, tagging budget on the device — are exactly what an edge-resident Scheme worker relieves.

2.2 Customer shape beyond Baobab

Beyond Baobab itself, the realistic Lacuna customer is any team that:

The shortlist: civic-tech newsrooms, vertical trade publications, market-data dashboards, internal corporate knowledge portals, NGO field reports, sports stat sites, cultural archives. The same five-substrate stack serves all of them; the operator brings the source list and the brand.

2.3 What the existing market gets wrong

Three failure modes we see in the wild:

  1. Generic LLM "scrapers" that hallucinate provenance. The output looks structured. The citations don't survive a click. Operators discover this when their auditors do.
  2. Hand-rolled fan-out on a single Node box. Works for ten feeds. Cracks at fifty. The image proxy starts returning 415s under load, the worker hangs on a slow CDN, the page renders without pictures. Familiar?
  3. Heavyweight ETL frameworks. Airflow, Dagster — great for batch warehouses, wrong shape for a sub-second-latency reader-facing surface. The ops cost dominates.

Lacuna's answer to all three is a small, fast, auditable execution layer at the edge, fed by a brain that emits a textual plan the worker runs.

3. The five substrates, briefly

Use-case 1 sits inside a five-substrate architecture. The other four are context, not the subject of this document; you'll see them referenced.

SubstrateTierRoleEngine
CortexL0 (device)Per-operator personality graph; preferences and correctionsCortex / Kuzu
AtlasL1 (cloud)Shared reference graph; canonical entitiesCortex / Neo4j Aura
LoamL1 (cache)World-building cache between crawlers + LLMMemgraph
EngramL1+Operator-private long-term store(decision pending)
SakuraL0 device LLMPersonal model; runs locally, knows the operatorllama.cpp / MLX

The worker is the execution layer that feeds — and is fed by — these substrates. It is not itself a substrate; it is the cheap, fast actor that makes the substrates load-bearing.

4. baobab-worker-rs: the execution layer

4.1 Why a Rust worker at all

The retrieval shape we want for Baobab — and any Lacuna customer — has four properties:

  1. Concurrent. A hundred-feed refresh should run in under ten seconds, not a hundred sequential seconds.
  2. Cheap. Sub-cent per refresh cycle. Edge compute, not a long-running VM.
  3. Sandboxed. The worker should be unable to take an action the operator didn't grant. No clever LLM completion can talk it into one.
  4. Replayable. Given the plan, the time, and the audit log, we should be able to reconstruct what happened — for debugging, for compliance, for the operator's own memory.

A Rust crate compiled to WASM at the Cloudflare edge, embedding a Scheme interpreter, hits all four. Node-on-Fly hits maybe two of them; Lambda hits three but loses the audit story. We picked the shape that buys the most properties for the least surface area.

4.2 Two targets, one source

baobab-worker-rs is a single Rust crate with two compilation targets:

TargetTripleWhere it runsMailbox
WASMwasm32-unknown-unknown Cloudflare Workers (edge)Durable Object
Nativex86_64 / aarch64 Fly.io, local laptoptokio::mpsc

The library exposes run_plan(src: &str) -> Result<PlanResult, WorkerError>. The mailbox surface and the tool registry live behind mailbox and tools modules. Both targets produce the same PlanResult shape — the operator's downstream code does not know (and should not know) which backend ran the plan.

Crate layout

workers/baobab-worker-rs/ ├── Cargo.toml # crate-type ["cdylib","rlib"]; two profiles ├── README.md ├── src/ │ ├── lib.rs # run_plan(), PlanResult, AuditEntry, WorkerError │ ├── plan.rs # Steel VM wiring, plan runner │ ├── tools.rs # tool registry, Capabilities, STANDARD_CAPS │ ├── mailbox.rs # Message, MessageResponse, Mailbox │ └── bin/cli.rs # native CLI: plan, mailbox-serve, version └── plans/ └── news-refresh.scm # canonical example plan

Cargo profile choices

[profile.release]
opt-level    = "z"    # optimize for size — WASM target dominates
lto          = true
codegen-units = 1
strip        = true

WASM size is the binding constraint: Cloudflare Workers has a hard cap, and every kilobyte spent on bloat is a kilobyte we cannot spend on the Steel VM and the tool registry. opt-level = "z" with full LTO and one codegen unit gives the smallest binary; we accept slightly slower native performance in exchange.

4.3 Mailboxes: Durable Objects vs tokio

The mailbox is the actor abstraction. Each named mailbox processes messages serially, in arrival order, and emits one response per message. The two backends:

The wire shape is identical in both cases:

{
  "id":       "<uuid>",
  "from":     "lacuna://baobab-news",
  "to":       "actor://baobab-news-fetcher",
  "plan":     "(begin (for-each refresh-source sources))",
  "caps":     ["net", "cache", "firecrawl", "audit"],
  "deadline": 1747920000000
}

The Lacuna brain owns "plan" and "caps". The mailbox owns ordering and resource limits. The worker owns execution and the audit log.

4.4 PlanResult + audit log

Every plan run returns the same shape:

pub struct PlanResult {
    pub ok:          bool,
    pub value:       serde_json::Value,
    pub audit:       Vec<AuditEntry>,
    pub duration_ms: u64,
}

pub struct AuditEntry {
    pub ts:     u64,
    pub op:     String,
    pub detail: serde_json::Value,
}

value is whatever the plan's last expression evaluated to. audit is the list of (audit ...) calls the plan made during execution. Lacuna writes both to JSONL alongside the plan source. Given those three artefacts — plan text, audit log, value — you can reconstruct the run.

This is the bit that's hard to retrofit. Adding an audit log on top of an existing Node scraper is doable but lossy: you only capture what the author thought to log. Building the audit log into the permission boundary — every gated tool emits one — makes the audit complete by construction.

5. Plans as contract: why Scheme

Three reasons. Each one is non-obvious until you've felt the absence.

  1. Auditable by reading. The plan is a few dozen lines of parenthesised text. A new engineer (or, more importantly, a customer's compliance reviewer) can read it. They cannot read a JIT-compiled Node program with the same confidence. Auditability that depends on "trust the framework" is auditability that fails the first audit.
  2. Portable by construction. The same plan runs on every backend. We can move a customer from Cloudflare to Fly to a local laptop without changing the plan. The plan is the contract; the backend is an implementation detail. (This is the same property Terraform plans have, and for the same reason.)
  3. Dumb workers, smart brain. Reasoning lives in Lacuna (Python, where model integrations are mature). Execution lives in the worker (Rust, where the safety story is mature). The worker cannot reason on its own and cannot be talked into autonomous behaviour by clever Scheme — it can only run the tools it has been given capabilities for.

We embed Steel (steel-core 0.7) as the VM. It's a mature R7RS-ish interpreter, WASM-compatible, with a Rust embedding model that matches our tool-registry plan. Alternatives considered: Janet (smaller but less mature WASM story), Lua (different syntax tradition, harder to express homoiconic plans), WASM-itself (right answer for the long run, wrong answer for the year-one DX). Steel is the right answer for v1.

6. Tool registry & capability model

The tool registry is the only side-effect surface Scheme can reach. Every primitive is a Rust function bound to a Scheme symbol; every primitive declares the capability it requires; every plan declares the capabilities it claims. The worker rejects plans at parse time that ask for capabilities they have not been granted.

6.1 Capabilities

CapWhat it grants
:netAny outbound HTTP/HTTPS
:cacheKV / Firestore / D1 reads + writes
:storageR2 / S3 / local fs object storage
:browserHeadless-browser rendering (was :firecrawl)
:auditAppend to audit log
:fsFilesystem read/write (native only)
:llmUpstream LLM (summarise / classify / translate / chat)
:secretsRead named secrets
:threadsSpawn cooperative threads via fork/join
:channelsUnbounded inter-thread channels

6.2 Tools (selected)

Scheme primitiveCapReturns
(fetch-url url):netstring body
(fetch-rss url):netxml string
(post-json url body):netjson
(crawl-url url):net :browsermarkdown
(parse-rss xml)list of items
(parse-html-meta s)alist
(kv-get key):cachestring or #f
(kv-put k v ttl):cache#t
(fork lambda):threadsthread-handle
(parallel-map fn xs):threadslist
(sha256 s)hex string
(summarize text):llmstring
(audit op detail):audit#t
(now)unix-ms int

The full registry is in src/tools.rs. The discipline is: pure helpers (parsing, encoding, hashing) require no capability; anything that talks to the outside world or persists state requires one.

6.3 Why the boundary is in Rust, not Scheme

If the capability check lived in Scheme — say, as a wrapping macro — a sufficiently clever plan could rebind the wrapper and break out. By checking in Rust before the tool runs, the Scheme program physically cannot reach the network without the cap; the function it would call is gated by a Rust caps.require(":net")?; at the top of the body. There is no Scheme expression that can bypass that check.

This is the same insight that makes capability-based OS designs (KeyKOS, seL4) defensible: the permission boundary lives one layer below the language the user can program.

7. The plan, line by line

The canonical plan for the news refresh, as checked in at workers/baobab-worker-rs/plans/news-refresh.scm:

;; news-refresh.scm
;;
;; The Scheme plan Lacuna sends to a baobab-worker-rs instance to
;; refresh the news cache. Stable, inspectable, replayable, audit-
;; loggable — the textual contract between brain and worker.
;;
;; Required capabilities:
;;   :net :cache :audit :threads :channels

(define sources
  '(("AllAfrica"           "https://allafrica.com/tools/headlines/rdf/latest/headlines.rdf" africa     news)
    ("Daily Maverick"      "https://www.dailymaverick.co.za/section/business-maverick/feed/" africa     news)
    ("Premium Times"       "https://www.premiumtimesng.com/feed"                              africa     news)
    ("Jeune Afrique"       "https://www.jeuneafrique.com/feed/"                                africa     news)
    ("BBC Africa"          "https://feeds.bbci.co.uk/news/world/africa/rss.xml"               africa     news)
    ("BBC Caribbean"       "https://feeds.bbci.co.uk/news/world/latin_america/rss.xml"        caribbean  news)
    ("Africa Is a Country" "https://africasacountry.com/feed"                                 africa     longform)
    ("Jacobin"             "https://jacobin.com/feed"                                         diaspora   longform)
    ("Aeon"                "https://aeon.co/feed.rss"                                         diaspora   longform)
    ("Monthly Review"      "https://monthlyreview.org/feed/"                                  diaspora   longform)
    ("Africa at LSE"       "https://blogs.lse.ac.uk/africaatlse/feed/"                        africa     longform)
    ("ROAPE"               "https://roape.net/feed/"                                          africa     longform)
    ("The Root"            "https://www.theroot.com/rss"                                      diaspora   diaspora)
    ("Black Agenda Report" "https://www.blackagendareport.com/rss.xml"                        diaspora   diaspora)
    ("Haitian Times"       "https://haitiantimes.com/feed/"                                   caribbean  diaspora)))

;; ---- Stage 1: fetch + parse, in parallel across all sources ----

(define (refresh-one src)
  (let* ((name  (car   src))
         (url   (cadr  src))
         (cont  (caddr src))
         (lane  (cadddr src))
         (t0    (now))
         (xml   (fetch-rss url))
         (items (parse-rss xml))
         (took  (- (now) t0)))
    (audit 'fetch-rss
           (list (cons 'src   name)
                 (cons 'count (length items))
                 (cons 'took-ms took)))
    (map (lambda (it)
           (append it (list (cons 'source name)
                            (cons 'continent cont)
                            (cons 'lane lane))))
         items)))

(define all-items
  (apply append (parallel-map refresh-one sources)))

;; ---- Stage 2: og:image fallback ----

(define needing-image
  (filter (lambda (it) (not (alist-ref 'image it))) all-items))

(define (enrich-image it)
  (let* ((og (fetch-og-image (alist-ref 'url it))))
    (if og
        (cons (cons 'image og) it)
        it)))

(define enriched
  (parallel-map enrich-image (take needing-image 80)))

;; ---- Stage 3: classify + cache ----

(define classified
  (map classify-entities all-items))

(define cached-count
  (cache-news classified))

;; ---- Final return: summary to Lacuna ----

(list (cons 'sources  (length sources))
      (cons 'items    (length all-items))
      (cons 'enriched (length enriched))
      (cons 'cached   cached-count)
      (cons 'at       (now)))

7.1 What the plan says, in English

  1. Stage 1. For each of 15 sources, fetch the RSS feed and parse it; tag every item with its source, continent, and lane. All 15 fetches run in parallel up to the worker's concurrency cap (default 8; bump via :concurrency=N). Each fetch audits a small record: source, count, milliseconds.
  2. Stage 2. Of the resulting items, filter to those missing an image; cap at 80 (the worker's per-cycle time budget); fetch the og:image for each and patch it in. Also parallel.
  3. Stage 3. Run a classifier pass over every item, then write the enriched list into KV (or, for the Cloudflare target, Workers KV).
  4. Return. A small alist with the counts and the timestamp. Lacuna logs this alongside the plan source and the audit entries.

7.2 Why this is the right factoring

The plan is forty lines. It describes a behaviour any operator can read. There is no hidden control flow, no buried setTimeout chain, no leaking event listener. If the cycle takes too long, you read the audit log: every fetch-rss entry has its source and its millisecond count. You find the slow source. You either drop it from the list, raise the per-source timeout, or push it onto its own less-frequent schedule. The change is one edit to the plan, not a code deploy.

7.3 What it does not say

The plan does not describe retry policy, deadline-budget enforcement, or output schema. Those are properties of the worker, not the plan. A plan written by a customer cannot accidentally redefine the deadline budget — the budget is set by the message envelope, not the plan body. The plan is the what; the worker enforces the how-much.

8. Cloudflare deployment

8.1 Bindings

Cloudflare-side, the worker needs four binding categories:

BindingPurposev1 instance
Durable ObjectMailboxes (one per actor)BAOBAB_DO
Workers KVNews cache, dedup keysBAOBAB_KV
R2Image and large-object cacheBAOBAB_R2
SecretsAPI keys (Firecrawl, LLM)per-secret-name

8.2 wrangler.toml

name = "baobab-worker"
main = "build/worker/shim.mjs"
compatibility_date = "2026-05-01"

[build]
command = "worker-build --release"

[[durable_objects.bindings]]
name = "BAOBAB_DO"
class_name = "Mailbox"

[[migrations]]
tag = "v1"
new_classes = ["Mailbox"]

[[kv_namespaces]]
binding = "BAOBAB_KV"
id      = "<kv-namespace-id>"

[[r2_buckets]]
binding     = "BAOBAB_R2"
bucket_name = "baobab-images"

[triggers]
crons = ["0 * * * *"]  # hourly news refresh

8.3 Cron-triggered refresh

A Cloudflare Cron Trigger fires every hour. The handler enqueues the news-refresh plan into the baobab-news-fetcher mailbox:

// Pseudocode, post-cargo-build
export default {
  async scheduled(event, env) {
    const plan = await env.BAOBAB_R2.get("plans/news-refresh.scm").then(o => o.text());
    const mailbox = env.BAOBAB_DO.get(env.BAOBAB_DO.idFromName("baobab-news-fetcher"));
    await mailbox.fetch("https://internal/run", {
      method: "POST",
      body: JSON.stringify({
        id:       crypto.randomUUID(),
        from:     "cron://hourly",
        to:       "actor://baobab-news-fetcher",
        plan,
        caps:     ["net","cache","audit","threads","channels"],
        deadline: Date.now() + 60_000,
      }),
    });
  },
}

8.4 What we get from Cloudflare specifically

9. Fly.io native deployment

The same crate, compiled native, runs as baobab-worker on Fly:

cargo build --release --bin baobab-worker
./baobab-worker mailbox-serve --addr 0.0.0.0:8080

The mailbox-serve mode accepts the same JSON message envelope over HTTP POST, processes it on a tokio task with an mpsc inbox, and returns a MessageResponse. Same plan, same contract.

9.1 When Fly is the right target

9.2 When Cloudflare is the right target

We expect most production deployments to be Cloudflare-default with Fly as the dev-and-staging path and as the escape hatch for operators whose plans don't fit the WASM budget.

10. baobab-worker-rs vs Firecrawl

This is the question that needs the clearest answer, because it's the one the customer will ask before any other.

10.1 They are different things

Firecrawlbaobab-worker-rs
ShapeHosted APIEdge runtime (you deploy)
LayerSingle capability: render and clean a URLOrchestration: fan-out, audit, schedule, compose
What it does wellRenders JS-heavy pages to markdown; pay-per-callCoordinates many sources, enforces caps, replays plans
What it doesn'tDoesn't fan out across 150 RSS feeds, doesn't enforce capabilities, doesn't audit, doesn't cacheDoesn't render JS-heavy pages itself; doesn't dodge anti-bot
Cost shapePer-call to a third partyPer-CPU-second on infra you control

10.2 The composition

In a real deployment, the worker calls Firecrawl when it needs a real browser:

(define (refresh-one src)
  (let* ((url     (cadr src))
         (kind    (caddr src))         ; 'rss | 'browser
         (body    (cond ((eq? kind 'rss)     (fetch-rss   url))
                        ((eq? kind 'browser) (crawl-url   url))
                        (else                (fetch-url   url)))))
    ...))

(crawl-url url) is the tool that goes to Firecrawl behind the scenes; it requires :browser :net. The Scheme plan stays clean; the operator decides at the source-list level whether a given source is RSS-fetchable or browser-rendered. The worker absorbs the difference.

10.3 Why this is good news commercially

Firecrawl is a partner, not a competitor. We pay Firecrawl for browser rendering when we need it (a small fraction of the source list, the JS-heavy ones), and we don't pay anyone for the 90% of sources that are plain RSS or static HTML. The operator's per-cycle bill is dominated by the small browser fraction; the long tail of plain RSS is essentially free.

And: an operator who only has the budget for plain RSS still gets the audit log, the capability boundary, the parallelism, and the edge latency. Firecrawl as an add-on, not a prerequisite.

11. Where Loam fits

Loam is the world-building cache between the crawlers and the LLM. It sits on the read path:

sources ─┬─► fetch-rss / fetch-url / crawl-url │ │ │ ▼ │ parse + normalise │ │ │ ▼ │ ┌───────────┐ │ │ LOAM │ world-building cache │ │ (Memgraph) │ - entity dedup │ │ │ - cross-source link │ │ │ - confidence accrual │ │ │ - freshness bookkeeping │ └─────┬──────┘ │ │ │ ▼ │ LLM (summarise / classify / translate) │ │ ▼ ▼ kv / r2 / response to Lacuna

11.1 Why Loam is load-bearing

Without Loam, the LLM is asked to re-discover the world on every call. "Who is the Premier of South Africa?" gets re-derived every cycle from the raw headlines. With Loam, the entity exists as a node; subsequent references resolve to it; the LLM is asked to summarise or classify, not to remember.

The effects on the cost model are large:

11.2 The Loam-worker boundary

The worker writes into Loam via a tool primitive ((loam-ingest ...), gated by :loam) and reads out via (loam-query ...). Both go over a Memgraph Bolt connection or, for the WASM target, a small JSON RPC the worker performs over HTTPS. The plan does not know it is talking to Memgraph; it sees Scheme primitives.

11.3 What Loam is not

Loam is not the operator's personal memory (Cortex) and not the shared canonical reference graph (Atlas). It is the intermediate layer where the world is built up en route between crawl and model. Operators who never look at Loam directly still benefit from it; their LLM bills shrink and their provenance gets crisp. Operators who want to look — there's an inspection surface for that.

12. Lacuna's value-add surface

What does the operator pay Lacuna for, beyond Cloudflare + Firecrawl + Memgraph at retail? Eight things; each is a real engineering surface and not a marketing slogan.

#Value-addWhat it costs to build yourself
1The Rust worker (this crate)3-6 weeks of senior Rust + WASM time
2The Scheme plan registry and the canonical plans2-4 weeks of careful design
3The capability boundary + audit log2 weeks if you've never built one; longer if you have
4Loam ingestion adapters and confidence scoring4-8 weeks; graph people are expensive
5The Lacuna brain (plan emission, tool calling)2-3 months of LLM-integration work
6Per-source health monitoring + failover plans1 week per source; long tail of edge cases
7The operator console (plan editor, audit viewer, source list)4-6 weeks of frontend
8SRE: runbooks, alerts, on-callOngoing; never finished

The operator brings the source list, the brand, and the editorial decisions. We bring the rest. The deal is: you operate a publishing-class retrieval system without staffing a retrieval team.

12.1 Extensions the operator can plug in

13. Tiering & commercial model

The tier shape we're proposing for v1, subject to design-partner feedback:

TierVolumeSourcesLLM opsLoamSupport
Hobby1 plan / hour≤ 20shared keyread-onlycommunity
Operator4 plans / hour≤ 100bring-your-ownread + writeemail, 48h
Newsroom1 plan / 15min≤ 500bring-your-own + Lacuna fallbackread + write + dedicated namespaceemail, 8h
Enterpriseunlimitedunlimitedmulti-key, per-plandedicated clusteron-call

Cost shape: the Hobby tier is roughly Cloudflare's free / starter tier plus a small Lacuna fee for the brain. The Newsroom tier is dominated by Firecrawl calls and LLM ops; the worker itself is a small fraction. The Enterprise tier is what we negotiate.

14. Security model

14.1 Threat model

The realistic attackers, in priority order:

  1. A malicious plan author. Someone the operator trusted enough to give plan-edit rights, who is now trying to exfiltrate secrets or pivot to other operator systems.
  2. A compromised upstream source. A scraped page or RSS feed whose author is now trying to ship a payload through the worker into Loam or the LLM.
  3. A compromised LLM response. A prompt-injection attempt that tries to coax the brain into emitting a plan with escalated capabilities.
  4. A network observer. Standard TLS-only edge story.

14.2 Defences, mapped

AttackerPrimary defenceSecondary
Malicious plan author Capability boundary in Rust; plan rejected at parse time if it requests caps the message envelope did not grant Audit log; secrets behind :secrets, never inlined in plans; per-plan deadline
Compromised upstream Tools sanitise their outputs (RSS parser is XML-safe; HTML parser is DOM-bounded; bytes never eval'd) Loam ingestion does schema validation; entries that fail schema do not enter the graph
Compromised LLM The brain emits a plan; the plan declares its caps; the worker only honours the caps the operator pre-granted to that mailbox, not the caps the plan claims Out-of-band plan review for any new plan that requests new caps; default-deny on cap escalation
Network observer TLS 1.3 throughout; secrets at rest in Cloudflare Secrets / Fly Secrets / Vault Audit log redacts secret values by default; only ops metadata visible

14.3 Key separation

The worker holds three classes of key material, with three different lifecycles:

14.4 The plan-replay property

Replay protection on the message envelope: every Message carries an id (a UUID) and a deadline (a unix ms timestamp). The mailbox rejects messages whose id it has seen within the last 2 × max-deadline, and rejects any message whose deadline is already past. Combined: replay-resistant without a heavy nonce database.

14.5 What the audit log does not contain

By default, the audit log captures: tool name, source URL (host only), result size, duration, and a hash of the response body. It does not capture: full request bodies, full response bodies, secret values, user-identifying information from upstream content. If an operator wants the verbose audit (for incident forensics), they enable :audit-verbose and accept the storage cost.

15. SRE: SLOs, SLIs, error budgets

15.1 Service level indicators

SLIDefinitionSource
Plan-run availability fraction of scheduled plan runs that return ok: true Mailbox response log
Plan-run latency p95 wall-clock from message enqueue to response Mailbox response log
Source-fetch success fraction of fetch-rss / fetch-url calls returning 2xx Audit log
Image-proxy success fraction of /api/img requests returning 2xx with non-empty bytes Edge access log
Render success fraction of news cells in the rendered gazette that have a non-fallback image Client-side telemetry

15.2 Service level objectives

SLOTargetWindow
Plan-run availability≥ 99.5%rolling 30d
Plan-run latency (p95)≤ 12s for news-refreshrolling 30d
Source-fetch success≥ 90% (with fallback to last cache)rolling 7d
Image-proxy success≥ 95%rolling 7d
Render success≥ 90% of cells with non-fallback artrolling 7d

15.3 Error budgets

At 99.5% availability, the monthly error budget is ~3.6 hours. We split that 50/50: 1.8h for planned-change burn (deploys, schema migrations), 1.8h for unplanned burn (incidents, upstream outages). If the planned-burn half is exhausted, the deploy gate trips and we freeze plan-registry changes until the window rolls.

Source-fetch success at 90% is deliberately loose because the upstream RSS world is unreliable; we treat falling under 90% as a signal to either drop the source from the plan, raise its individual timeout, or move it to a less-frequent schedule.

16. Observability & the audit-log discipline

16.1 Three tiers of signal

TierSignalCardinality
Metrics counters & histograms; per-plan, per-tool, per-source Low; emitted to OTLP every 60s
Logs structured JSON; one per plan-run Medium; one per plan-run × number of mailboxes
Audit log JSONL; one per gated tool invocation High; tens-to-hundreds per plan-run

16.2 The audit log is the trace

Unlike a generic OpenTelemetry trace, the audit log is the contract. Every gated tool emits one entry; the entries are ordered; the plan source plus the entries reconstruct the run. A distributed tracing system gives you the same information for the network calls; the audit log gives you the same information for the semantic operations (an entity was classified, an image was enriched, a cache was written).

16.3 Sample audit-log entry

{
  "ts":     1747920003012,
  "op":     "fetch-rss",
  "detail": {
    "src":     "AllAfrica",
    "url":     "https://allafrica.com/tools/headlines/rdf/latest/headlines.rdf",
    "host":    "allafrica.com",
    "status":  200,
    "count":   42,
    "took-ms": 311,
    "hash":    "sha256:9b18..."
  }
}

16.4 Dashboards we'll ship

17. Capacity planning

17.1 Per-cycle worst-case

For an operator with 500 sources and 80 image enrichments per cycle:

ResourcePer cycleDaily (24 cycles)
Outbound HTTPS~580 requests~13.9k
Firecrawl calls~50 (10% of sources)~1.2k
KV writes~500~12k
R2 writes (images)~80~1.9k
LLM tokens (summarise)~50k~1.2M
CPU-ms (worker)~6,000~144k

At those numbers the Cloudflare bill is dominated by Workers KV writes and outbound bandwidth, not Workers CPU. The LLM bill is dominated by the summarise step, which Loam is designed to suppress over time (repeat entities skip the LLM).

17.2 Headroom & throttles

18. Incident response & runbooks

18.1 Standing runbooks

SymptomFirst checkAction
Plan-run availability falls below 99% in an hour Per-source health dashboard Identify the failing source; drop from plan or raise timeout
Image proxy returns > 5% non-2xx Edge access log; sample the failing CDNs Add the offending CDN to the Referer-strip list; redeploy
LLM token spend > 2x daily baseline Cost-burn dashboard; check for runaway summarise Inspect plan; check Loam dedup rate; freeze the plan
Audit log volume drops to zero Mailbox response log Plan is silently throwing before any audit; check WorkerError
Cloudflare Durable Object errors spike CF status; wrangler tail If platform: wait. If our code: redeploy with previous build

18.2 The "two instances" incident pattern

A specific failure mode worth calling out: the operator running two browser tabs against the same Baobab deployment. Each tab has its own Web Worker, its own IntersectionObserver, its own AbortController; they share localStorage. Concurrent writes from two tabs can stomp the cortex.v1 key, retag mid-render, and produce a "now you see it, now you don't" experience for image cells. Mitigations:

18.3 Postmortem template

  1. Window: when did it start, when did it end, who saw it.
  2. Impact: SLI delta in the window.
  3. Cause: the plan, the audit log, the offending entry.
  4. Detection: which alert fired, or which human noticed.
  5. Response: what we did, in order, with timestamps.
  6. Action items: changes to the plan, the worker, the dashboards, the runbook.
  7. What we got right.

19. Unit economics

A rough back-of-envelope for the Operator tier (≤100 sources, 24 plan runs/day, BYO LLM key):

Cost lineMonthlyNotes
Cloudflare Workers$5–15Paid plan, request volume
Workers KV + R2$3–10News + image cache
Durable Objects$2–8Mailbox compute
Firecrawl$10–40~10% of sources
Memgraph (Loam)$0–30Shared cluster initially
LLM tokens$0BYO; operator-paid directly
Total Lacuna pass-through~$50–100Per operator, infra-only
Lacuna margin (Operator tier)$100–200Negotiable

The Hobby tier is a leader-priced loss-leader. The Operator tier is profitable at modest volume. The Newsroom tier carries the company.

20. Roadmap to general availability

Week 1–2 — Worker bodies

Week 3 — WASM target

Week 4 — Lacuna integration

Week 5–6 — Loam

Week 7–8 — Second customer

Week 9–10 — SRE polish

Week 11–12 — GA

21. FAQ

Why not Lua / Python / WASM-direct?

Lua: viable, but its parenless syntax fights the homoiconicity we want (plans-as-data, plans-as-text). Python: too big for the WASM target. WASM-direct (handing operators a wasm module per plan): right answer for v3 or v4; wrong answer for v1 DX — operators don't write WASM.

Why not just write the worker in Node?

Three reasons. (1) Audit-log discipline is much easier to enforce in Rust where we can put the cap check on the function body, not in a linter. (2) WASM size budget matters and Rust wins it. (3) The threading model — fork/join/channels — is straightforward in tokio and awkward in Node's single-threaded event loop.

Isn't a custom DSL a maintenance burden?

The DSL is R7RS Scheme. We didn't invent it; we picked it because the operator's plan is small enough that they should be able to read it, and Scheme is the smallest readable thing. The tool registry is our custom surface; the language is not.

What happens when a plan exceeds its deadline?

The plan run is cancelled cooperatively. Tools whose body is in flight see their deadline drop to now and may abort their I/O. The mailbox response is ok: false with a deadline-exceeded error. The audit log up to the cancellation is preserved.

How do we test plans?

The CLI runs plans against a real worker. We also ship a --dry-run mode that runs the plan against a mocked tool registry; tools return canned values, and the audit log shows the call shape. CI runs every canonical plan in dry-run on every push to the plan registry.

Can a plan call another plan?

Yes — via (send mailbox-name message). This is how compound workflows are built: the news refresh sends a message to the Loam-ingest mailbox; the Loam-ingest mailbox sends a message to the LLM-summarise mailbox; results trickle back. Each mailbox has its own capabilities; cross-mailbox capability escalation is not possible.

What if Cloudflare goes down?

The Fly native target is the failover. Operators on the Newsroom and Enterprise tiers can be configured for active-active across both backends; the Lacuna brain picks the healthy one per cycle. Hobby and Operator tiers degrade to "next cycle will retry."

22. Glossary

TermMeaning
AtlasShared canonical reference graph (L1). Distinct from Loam.
Audit logAppend-only JSONL record of every gated tool invocation in a plan run.
BaobabThe reference deployment; a pan-diaspora portal.
CapabilityA named permission that gates a tool. Granted by the message envelope.
CortexPer-operator personality graph (L0).
Durable ObjectCloudflare's named, single-writer, persistent state primitive.
EngramOperator-private long-term store (L1+).
FirecrawlHosted headless-browser rendering service. A capability inside the worker.
Lacuna brainThe Python service that emits plans for workers to run.
LoamWorld-building cache between crawlers and the LLM (L1, Memgraph).
MailboxActor-style inbox; DO on Cloudflare, tokio mpsc on native.
PlanA Scheme program; the textual contract a worker runs.
PlanResultThe struct a plan run returns: ok, value, audit, duration_ms.
SakuraThe per-operator device LLM.
SteelThe Scheme interpreter we embed (steel-core).
ToolA Rust function bound to a Scheme primitive name.