jesterky runs agent workflows as replayable programs
An open Rust workflow substrate for research engineers: fan out model work into seeded sandboxes, capture artifacts, and replay the orchestration path without pretending future model calls are deterministic.
Dynamic model workflows fan out, call tools, mutate state, and become difficult to inspect after the run. jesterky records the specification, runtime events, and sandbox outputs so research engineers can inspect the path a workflow took and check a replay against that record.
Headline results
Three workloads, run live, replayed
Every number is read from a committed run manifest or score artifact. Each card names its evidence and its boundary.
Blog quality scan
replay okgpt-5.5 audits 8 production posts, live
One blog corpus and one recorded run; this is a scan, not a quality benchmark.
Dev-port bench
per-cell score artifactsmodels port Python game engines to Rust in a seeded sandbox
One run per cell; repeat attempts swing mid-ladder cells, so no model ranking is claimed.
SMR ReportBench
replay okworkflow verdicts vs. autograde over real graded runs
Four graded lanes in the captured packet; this does not establish general evaluator reliability.
Workflow record
What jesterky captures from a run
Read the evidence at the right level
The claims here are about workflow execution and artifacts, not deterministic model behavior. A replay verifies the recorded orchestration path against recorded outputs. It does not guarantee that a future call to a hosted model will make the same choice unless that call is recorded or otherwise controlled.
The quality scan, live
expand → map → reduce, live
One run audited every published post on this blog
The workflow expands the blog corpus into per-post audit jobs, maps a gpt-5.5 auditor over them concurrently, and reduces the verdicts into a report. These are its real findings — including about this blog.
strong launch proof; routing metadata mostly absent
clear beta post with smoke proof; routing metadata missing
measured benchmark claims lack required proof metadata
required routing metadata largely absent
measured claims lack reproducible proof
measured lifts lack visible proof metadata
launch metadata and proof links incomplete
launch metadata and release custody missing
Same spec, two actors: the fake actor echoes deterministically — what the tests run on. Swap in the codex actor and the identical workflow drives gpt-5.5 on ChatGPT-bundle auth, no API key in the process. The spec doesn't change; the actor behind the seam does.
Models run a real environment
sandbox → port → verify → capture → score
Five engines, three models, one cliff
Each model ports a working Python game engine to a Rust crate inside a seeded workspace, iterating against a 4-scenario train oracle. The captured crate is scored on ALL scenarios — the tick marks the train subset: bars that stop there memorized the diff and generalized zero.
workflow and single-session runs
Two models, two execution modes
Sokoban (15 scenarios), identical seeded files, porter brief, and 10-minute cap. The comparison is Gemini Flash Lite and GPT-5.6 Luna at low effort, each run through the workflow and a single Codex session.
gemini-3.1-flash-lite
Workflow
4/15
0:56 · 303k tok · $0.087 est.
Single harness
0/15
5:48
gpt-5.6-luna · low
Workflow
4/15
1:47
Single harness
2/15
3:49 · 59.6k tok
The workspace is seeded by jesterky-sandbox (local or Docker); capture globs pull the built crate back into the run manifest. Gemini and DeepSeek drive the same loop through jesterky-proxy — Responses⇄chat tool-call translation plus Gemini's thought-signature round-trip — so one agentic harness runs any model. The ports above are manifests you can replay.
Workflows against deep standards
engineering soundness, continuously
57 modules graded against 7 house standards
The same expand → map → reduce shape pointed at code: modules across five repos, each graded 1–10 on typed seams, no-fallbacks, error legibility, exhaustive enums, unique nouns, earned abstraction, and explicit-over-implicit — with caps-at-hold verdicts and file-level evidence.
HttpObject = dict[str, object] … ConfigDict(extra="allow") — open JSON contracts at a public seam
runtime tests bypass seams with casts: `tools: tools as never`, `let mockCurrentWorktreeSession: any = null`
typed StrEnums exist, but escape hatches remain: `schedule: dict[str, object] = field(default_factory=dict)`
A standards document only matters when a workflow can apply it consistently. The scan above runs the engineering-soundness rubric against each module, records quoted evidence, and applies a caps-at-hold verdict. The manifest gives a reviewer the inputs, outputs, and evidence behind each grade.
Verify a captured run
replay verification
Check a captured run against the runtime
Events use a logical address: run_id, node_path, iteration, and local_seq. During replay, the runtime must reproduce the recorded address, kind, and payload.
Replay tests the workflow runtime separately from a new model call. It reads a captured manifest, re-runs the scheduler against recorded outputs, and fails when an event's logical address, kind, or payload changes. It does not claim that a future hosted-model call will return the same answer.
jesterky run examples/quality_scan_blogs.json --actor codex --out run.manifest.json
jesterky replay run.manifest.json --spec examples/quality_scan_blogs.json
# replay ok: events=52 recorded=9In the products
- Optimizers: GELO and GEPA take a
[jesterky_workflow]config block — traces export, a workflow clusters failure themes, and the themes materialize into the proposer before each propose. GELO's arm with the workflow improved Craftax reward over baseline by +0.977 vs +0.339 without. The same hook inside GEPA is wired but showed no uplift, and we report it that way. - Stack: agent workers get five MCP verbs —
stack_jesterky_register,launch,inspect,replay,compare— verified end-to-end against the published binary, including a full Stack effort eval — launch → inspect → replay → compare withnode_diff_count=0and A0–A3 acceptance recorded pass. - Cloud: SMR ReportBench runs are graded through the same trace-evaluate workflow; gpt-5.5's verdicts agreed with autograde on all four graded lanes (report score 0.926, replay ok).
Where it falls short
A flaky hosted model still produces a flaky run — jesterky classifies the failure, it can't prevent it. Mid-ladder dev-port cells swing between runs, so the bench packet claims the cliff, not model rankings. GEPA's workflow arm has no measured uplift. The ReportBench A/B ran as a post-hoc revision pass (+0.294 on the one non-ceiling lane, with two checks recovered by rubric gaming — the packet says which); in-run guidance is the next rung.
Resources
- jesterky repository — source, examples, CLI, and the runtime contract.
- Proof artifacts — committed manifests, score files, ablations, and run notes behind the claims above.
- jesterky on crates.io and jesterky on PyPI — install surfaces for the 0.1.1 release.
- Claude Code auto mode — the workflow-orchestration pattern that motivated this substrate.
- GELO: Go-Explore for long-horizon agents — the adjacent Synth optimizer work that consumes workflow evidence.
- GEPA platform — the optimizer surface where the workflow hook is wired, including the negative result reported here.
Citation
If you use the workflow substrate or the replayable-agent-workflow pattern in research, cite the post and repository:
@misc{jesterky2026,
title = {jesterky runs agent workflows as replayable programs},
author = {{Synth Team}},
year = {2026},
howpublished = {\url{https://www.usesynth.ai/blog/jesterky-dynamic-workflows}},
note = {Open-source Rust workflow substrate},
url = {https://github.com/synth-laboratories/jesterky}
}Start with the recorded quality-scan example. cargo install jesterky-cli, run the scan, then inspect the manifest and replay report. The evidence linked above lives in the proof/ tree.