Inside Union: The Replay Log That Makes Flyte 2.0 Durable

How ordinary Python becomes a durable, replayable log of actions, so that a lost node, a bug fix, or a failing model API costs one step and not the whole run.

This is part 2 of Inside Union, a five-part series on the engine behind Flyte 2.0. It follows part 1 on the durable AI runtime and continues with part 3 on leases and the scheduling engine, part 4 on resource-aware scheduling, and part 5 on observability.

"There are only two hard things in Computer Science: cache invalidation and naming things."
— Phil Karlton

A log written while the code runs

Flyte 1 compiled a workflow into a static graph before anything ran. The control plane always knew what came next and could restart from any node, and we paid for that certainty with a DSL that users found hard to learn and we found hard to extend, because a pseudo-language built inside Python is hard on both sides of it. Flyte 2 gave the graph up on purpose. A run is plain Python: loops, conditionals, an `asyncio.gather`, a `while` around an LLM call that decides when it's done, a wait for an external signal or approval in the middle of the control flow. There's nothing to compile, because the graph is whatever the code turns out to do. That's what makes agents natural to write, for a person who knows Python and equally for a model generating code, since nothing is inverted and there's no framework shape to learn first. It also means the only complete description of a run lives inside a process that is allowed to die.

The previous post listed nine requirements we set before building the engine. This one covers how the SDK and the per-run controller meet four of them: Python with a minimal decorator, full recovery including after a code change, no finished work redone, and infrastructure errors that carry their cause. It also lays the base for observability, because everything the UI shows about an action hangs off the name this post explains.

Making the code feel simple created a new problem, which is that durability became hard to add. We solved it with a write-ahead log. Every child task call, every traced function, and every wait for a human is recorded as an action in the control plane, under a name that a fresh process running the same code computes again. The set of actions under a parent is a log in the sense Jay Kreps wrote about: an append-only, ordered record of what happened. Once you have the log, replay, recover, and fork all work the same way. You run the code, and at each call you look up the entry before doing any work. Every call does this lookup, so recovery doesn't need a special mode.

The controller

The controller is a library that runs inside every task's process. Its core is Rust with Python bindings, and a pure-Python implementation with the same behavior exists for local runs and development. The Rust core is also the seed of the multi-language promise: it speaks the wire protocol and owns naming, so a task written in another language gets the same replay log by linking the same core, and a run can mix languages without the engine noticing. When your code calls a child task, the controller serializes the inputs canonically and uploads them, computes the child's name and the output path that follows from it, submits the action with its full specification, and then waits on an event that the watch stream will fire.

The watching is shaped like a Kubernetes controller rather than a client library. There's one stream per parent action, a local cache of every child's state, a work queue, and a small pool of workers that reconcile whatever the cache says: not started, launch it; terminal, wake the caller; anything else, keep watching. It never polls. And the control plane sends a parent only the updates that change what it should do next, terminal and final-attempt, rather than every queued-to-running transition, so a parent with a hundred thousand children sees a hundred thousand messages rather than millions.

Naming

A durable log only works if a restarted process can find its place in it, and names are how it does. The same code, run again, computes the same name for every call, so finished work is found by name and reused instead of redone. Each name is computed from five ingredients and never assigned:

name = base36(md5(
    parent_action_name          # where in the tree
  + hash(canonical_inputs)      # what it was given
  + task_identity               # what code, with what interface
  + call_sequence               # the nth identical call from this parent
  + group                       # optional, user-declared
))
output_path = run_dir / name

Each ingredient answers the same question: should this call, made again in a fresh process, get the same name?

Inputs are converted and serialized with Flyte's type system, which gives data types interoperability across languages, renders them in the UI, and automatically offloads files and directories and dataframes to object storage, passing references with content hashes in their place. So the log stores names and hashes, never the data. The same wire format is what keeps names language-agnostic, since any language that speaks it computes the same names. We took particular care to keep names consistent across input types, unordered dictionaries for example, and to stay fast with multi-gigabyte values: a large input carries its own content hash, so naming it costs nothing.

Task identity is the task's fully qualified name, its interface, and a hash of its function body. It deliberately leaves out the container image, the code-bundle version, resource requests, environment variables, and plugin configuration. Rebuild the image or ask for more memory on a retry and nothing is renamed. Edit the function body and it is, so the edited task re-runs instead of replaying a stale result. Traced functions get the same treatment, name plus body hash.

The call sequence distinguishes the same task called twice with the same inputs from the same parent. It exists for side-effect tasks, the bread and butter of data engineering teams arriving from Airflow: a load, a notify, a write to a warehouse, called twice on purpose with identical arguments and expected to run twice. The counter is keyed on the call itself, meaning the task identity, the inputs hash, and the group, and not on time or on the order the event loop happened to resume coroutines. A hundred identical calls under `asyncio.gather` get sequence numbers one to a hundred whatever the interleaving. Calls that share a counter are byte-identical and interchangeable, which is exactly when order shouldn't matter.

A group is a label you can put on a set of calls. It's folded into the name so grouped and ungrouped invocations of the same call never collide.

One consequence took us a while to see. The run's start time is passed into containers as an argument, and arguments feed the hash, so on recovery the start time is pinned to a constant rather than `now()`. Anything that varies between attempts has to be kept out of the name, and the list of such things turned out to be short.

Replay, recover, fork

Take the case where a parent dies and restarts. The node was preempted, and the control plane starts a new attempt of the parent. The new process opens its watch stream before it does anything else. The stream begins with a snapshot of every child that already exists, ending in a sentinel, and the controller submits nothing until the sentinel arrives. Then the code runs. At each call the controller computes the name and looks in the cache: succeeded, return the recorded outputs without submitting; running, wait on it; absent, submit. A submit that races with an existing action gets "already exists", which is treated as success. From the user's side, the run continued.

Traced functions work the same way at a finer grain. `@flyte.trace` turns a function call into a log entry whose value is whatever the function returned, or the error it raised. On replay the recorded value is returned and the body is skipped. This is how a non-deterministic agent loop replays deterministically: the log records what the model said the first time, and the second attempt reads the log instead of asking again.

@flyte.trace
async def plan(history: list[str]) -> str:
    ...  # LLM call; recorded once, replayed on recovery

@env.task
async def agent(goal: str) -> list[str]:
    history = [goal]
    while (step := await plan(history)) != "done":
        history.append(await act(step))   # child action, named by (inputs, identity, seq)
    return history

Errors are recorded with their cause. A task that raises records a user error. A container that is OOM-killed or a node that is preempted records a system error, and the distinction travels with the action all the way to the retry policy and to the UI, so "what happened to this step" has an answer that names the infrastructure when the infrastructure was at fault. We read the same distinction in aggregate: a system that runs arbitrary code sees user errors constantly, so system errors are monitored as their own class, driven down over time, and in many cases auto-recovered before a user ever notices. A recorded error replays as an error unless it was marked recoverable, in which case the function runs again. And because a trace's identity includes its body hash, fixing a bug in `plan` and forking the run re-executes `plan` while every `act` that already succeeded is reused.

Recovering a whole run is the same lookup at the top level. `flyte.rerun("r1", recover=True)` starts a new run that matches succeeded actions from the source run by name, marks them recovered pointing at the source outputs, and executes only what failed or never ran. Change an input and the root's input hash changes, which changes the names of everything downstream, which re-runs exactly the actions that depended on the changed value. We didn't build dependency tracking; it fell out of putting the input hash in the name. Forking is the same mechanism with your current code: edited task bodies get new names and run again, and everything else is reused. This is the what-if workflow from the previous post. Change the system prompt at one step of an agent and only that step and everything downstream of it runs again. Repair a corrupted intermediate result and everything that consumed it is recomputed while everything before it is reused. There's an escape hatch, `force_rerun_actions`, for when you want a succeeded action to run again anyway.

What was hard

Determinism in a dynamic language came first. Python offers dict ordering, float formatting, set iteration, and event-loop interleaving as ways for two runs of the same code to drift. The rule we settled on is to hash canonical bytes, never Python objects, and to key sequence numbers on the call rather than on when it happened.

Deciding what identity means was the harder judgment call. Every ingredient in the name is a decision about what should invalidate a result. Include too much, the image say, and nothing ever recovers. Include too little, drop the body hash, and edited code replays stale results. The line we drew is the code and its contract, and nothing about where it runs, and that line is what lets a retry ask for more memory without losing its place in the log. The out-of-memory example from the previous post, where one source needs more memory than all the others, is exactly this case: the retry of that one step asks for a larger container, its name doesn't change, and the rest of the log is untouched.

Traces must not fragment the tree. A child task called from inside a traced function parents to the real task action, never to the trace's pseudo-action. Otherwise a trace boundary would split the log and a recovery would find nothing under the parent it expects.

The control plane will be unavailable sometimes. The controller rate-limits itself, keeps a bounded number of children in flight per parent, and retries with backoff long enough to ride out twenty to twenty-five minutes of continuous unavailability before a run fails. We've put a lot of work into handling the range of networking and infrastructure failures underneath that, so the controller stays up through them.

The controller is a client we can't upgrade. It ships inside the customer's environment and has to stay compatible as the control plane revs underneath it. The wire contract is versioned and additive, and in years of operating it we've barely ever had to force an upgrade on a user.

Local had to equal remote. The pure-Python controller has behavior parity with the Rust core, so a run can be written, tested, and debugged locally and then behave identically on the platform.

And data flow is its own subsystem. Abstracting data movement is the property Flyte is built around, and underneath the controller sits a full data subsystem, written in Rust, that handles offloading, references, and streaming. That one deserves its own post.

Where it stands

• 0 succeeded actions re-executed on recovery of a parent, by construction
• 1M actions in one run, under one replay log
• 20–25 min of control-plane unavailability a running task rides out
• 1 stable address per action, shared by the UI, the scheduler, and runtime history

The zero is by construction: a parent that restarts consumes every child that finished, waits on every child still running, and submits only what's new. The last one is the quiet enabler for the rest of the series and for the UI. Because the name is stable and the address is shared, the console can show a million-action run as a tree that updates in real time, with the logs, the error cause, and the cost of each action hanging off the same key. Every action has an address, `org/project/domain/run/action`, that everything agrees on, and the scheduler later uses it to look up how long this task usually takes before deciding where to put it.

What customers say

"V2 looks and feels great. It is very trivial to build awesome experimentation pipelines with V2. The SDK is excellent. I have thrown the brick wall at the type transformers for input/output types; some of the craziest inputs/outputs I can think of, and it all works perfectly fine. Magically shows up in the UI, works from CLI and caches correctly."
— Grantham Taylor, Principal Scientist, ATPCO

Next up

Everything above happens before an action is placed anywhere. The next post follows an action after it's submitted: how the scheduling engine hands it to a worker as a lease, what fencing and heartbeats look like at hundreds of thousands of actions, and a durability model where the memory of a single leader-elected process is the truth. If you'd rather see the whole path from the user's side first, the life of a run page walks it end to end. And if you'd rather run it than read about it: `pip install flyte` and `flyte start devbox`.

The Inside Union series

1. The Durable AI Runtime for Flyte 2.0
2. The Replay Log That Makes Flyte 2.0 Durable (this post)
3. Leases, the Scheduling Engine Behind Flyte 2.0
4. Resource-Aware Scheduling for Flyte 2.0
5. Observability at a Million Actions

Appendix: controller defaults

Every number in this post that comes from the controller, with the default it ships with. These are per-process settings inside your task, not platform limits, and most runs never change them. They fall into three groups: how fast one parent can push work, how the controller behaves when the control plane is slow or gone, and what travels inline versus by reference.

Throughput and backpressure

Surviving a slow or absent control plane

Payloads