Union.ai achieves 9.8x ROI according to analysts.

View the report
GPU Cost & Utilization

GPU Idle Tax

Hidden inefficiencies in your ML pipelines can cost thousands in wasted GPU spend.

GPU Infrastructure

See the cost of legacy orchestration.

?Per GPU — × 64 GPUs = 2,560 cold-booting tasks/day

Sets a typical throughput & cold-start profile. Adjust anything after.

64
$4.00
A100 · $2.50 H100 · $4.00 H200 · $6.00 Custom

Loss

GPU time lost to container startup, data loading and pipeline steps

$104K

That's 4.6% of your $2.24M/yr GPU fleet.

3.0
Idle GPUs per year

Recoverable with Union ↓

~$65K

Using reusable containers, cold boot optimization, and data batching

Savings with Union

Turn on Union features

?In Union, a plain Python function fans out into many parallel workers with one call — .map() over your inputs, or a replicas setting on the task environment. Union spins up that many copies of the function and runs them at once, so one function becomes N parallel processing functions. With Reusable Containers on, each worker pays the cold boot once and then serves many batches — so 8-way fan-out over 1,000 batches means 8 cold boots, not 1,000, plus 8-way parallelism.

Reclaimed each year

$87K

Union reclaims 83.8% of the tax.

320×
fewer cold boots with Reusable Containers

By mechanism

Reusable Containers (cold boot, once per replica)$61K
Image Cold Boot Optimization (image pull removed)$109
DynamicBatcher + JsonlDir (data, streamed)$26K
Total reclaimed$87K
Where the tax comes from, and how Union removes it

Four leaks, four mechanisms

Leak · cold start on every task

A fresh container per task

Platforms like Kubeflow start a new container (with a multi-GB image pull) on a billing GPU for every task. Reusable Containers keep replicas live across tasks via ReusePolicy: the model loads once and stays resident, so repeat work dispatches in <100ms instead of cold-starting again. Docs ↗

Leak · slow cold boot

When a boot is unavoidable, it's slow

The first launch (or a scale-up) still has to boot a container and pull its image. Cold Boot Optimization cuts that cold boot from minutes to seconds, so even the boots you can't skip stop burning GPU wall-clock while nothing computes. Docs ↗

Leak · Data Gravity

GPU allocated, work waiting

Tasks sit idle behind scheduling. The DynamicBatcher packs concurrent submissions into batched GPU calls, and queues of data keep the accelerator fed instead of parked between actions. Docs ↗

Leak · failed tasks

OOM burns the whole run

A crash throws away everything computed so far back to your last checkpoint. Self-healing retries fix that: an OOM retries with more memory, and checkpoints resume at the exact line of code, so failure costs seconds, not a re-run. Docs ↗

Go deeper: the architecture that removes the tax

How Union keeps the GPU saturated

The full walkthrough (100M+ inference calls at near-maximal GPU utilization) is in the blog Batch Inference at Scale: How to Maximize GPU Utilization ↗.

Four primitives do the work:

Reusable Containers

The container loads once and the replica stays live across tasks via ReusePolicy. Dispatch in <100ms using Cold Boot Optimization, with no per-task cold boot. Docs ↗

DynamicBatcher

Accumulates concurrent submissions into batched model calls: one forward pass over 256 samples instead of 256 separate calls. Docs ↗

JsonlDir

Streams JSONL shards from object storage record by record, so memory stays constant whether you process 1M or 100M samples. Docs ↗

Replica-level parallelism

Tasks run with multiple replicas, effectively a pool of identical GPU workers. Combined with the concurrency setting on Reusable Containers, this determines how many shards run in parallel: at most replicas × concurrency shards in flight at once.

See it in action: benchmarks + production results

Public benchmark · one identical batch-inference job, reuse vs cold-start

4.1×faster end-to-end (404s vs 1,665s) 16% → 100%GPU actually computing (cold-start → warm) 3.3×peak decode throughput

Qwen2.5-7B under vLLM on one NVIDIA L4, Reusable Containers vs a fresh pod per call. Flyte Systems Group technical report FSG-26-04, Jul 2026. View report ↗

Life-sciences company · GPU-heavy diagnostic inference
up to 60%lower GPU-node time
3 min → 800 msimage-pull time
>$50K/yrsaved, and growing

Runs 8,000-node inference pipelines across multiple AWS regions. The image-pull tax (the between-jobs leak above) is removed by Reusable Containers + Cold Boot Optimization, with self-healing retries at massive scale.

Token cost · agent-authoring benchmark
1.8× fewertokens to a working pipeline
5× fewerrun → fix iterations

A coding agent reaches a passing pipeline in 1.8× fewer tokens on Union than DAG-based orchestration, and solves control-flow patterns DAGs can't express at all. Read the benchmark →

Results measured on Union in production; company names are withheld until each case study is public, then added here by logo.

Stop paying the idle tax

Spin up a DevBox and run a durable workflow with Reusable Containers in under an hour: GPUs kept live across actions, self-healing retries, and action-level resource visibility, in your own cloud.

Get DevBox free
Talk to us about GPU efficiency

How this is calculated: cold start per task = image pull (container GB × ~4s/GB effective) + container start + data load. Tasks/yr = GPUs × pipelines-per-day × active days. Idle today = tasks × cold-start ÷ 3600 × price/GPU-hour; fleet cost = GPUs × 24 × active days × price. With Union: Image Cold Boot Optimization removes the image pull; Reusable Containers keep containers running and swap out code, so the container startup is paid once per replica (N replicas = N cold boots, not one per task) while adding N-way parallelism; DynamicBatcher + JsonlDir stream data so load overlaps compute. Reclaim is attributed to these mechanisms and reconciles to the baseline. Proof figures reflect production results pending named-customer clearance; Union scale/latency claims reflect internal observability systems.