Our Workflow Engine

When we started Nimbus, every engineer we knew told us the same thing: don't build a workflow engine. Use Temporal. Use Airflow. Use Argo. Use anything but yours.

We didn't listen. Two years later, that decision is the single biggest reason Nimbus exists in its current form. Here's what we built, and why every off-the-shelf option failed our requirements.

The four constraints that broke every existing engine

Workflow engines come from one of three worlds: data engineering (Airflow, Prefect, Dagster), microservices orchestration (Temporal, Cadence), or business automation (Zapier, Make, n8n).

Each is excellent at the world it was designed for. None of them solved our problem.

1. Workflows must survive every kind of failure

Most workflow engines handle the happy path beautifully and the failure path badly. Temporal is excellent at retries but assumes you control the entire call chain. Airflow can resume from a failure but requires manual intervention.

We needed workflows that handle: third-party API outages, partial responses, race conditions, customer-side schema changes, and (critically) human-introduced errors. Without manual intervention.

2. Latency must be sub-second for synchronous workflows

Business automation tools like Zapier average 1-3 seconds per step. For a 10-step workflow that's 10-30 seconds. Acceptable for nightly batch processing. Not acceptable for an approval flow.

We needed cold-start times under 200ms and steady-state per-step latency under 50ms.

3. Schema-less by default, schema-aware when needed

Data engineering tools demand schemas. They want you to define a typed dataframe before you can run anything. That's correct for analytics workloads where consistency matters more than flexibility.

For business workflows, the opposite is true. The customer's HubSpot field is a string today and an array tomorrow. The Slack message has new fields next month. The workflow needs to keep running.

4. The visual editor must be authoritative

Most workflow engines have a visual editor that's a "view" of the underlying code or YAML. Edit the visual, edit the code, both change. That sounds good in theory and breaks in practice — the two representations drift.

We wanted the visual editor to BE the workflow. No YAML to maintain separately. What you see is exactly what runs.

The architecture we landed on

We built a deterministic event-driven engine with three layers:

Layer 1: The graph

Every workflow is a directed acyclic graph (DAG) of steps. Each step is a pure function that takes input data and returns output data + a status.

Importantly: steps cannot mutate global state. Every operation is functional. This makes retries trivial — re-running a step with the same input always produces the same output.

Layer 2: The runtime

The runtime traverses the graph, handles retries, and persists checkpoints between steps. We wrote it in Rust for performance.

The runtime knows nothing about HTTP, Slack, HubSpot, or any specific integration. It just knows: "execute step X with input Y, get output Z, persist, continue."

Layer 3: The action layer

This is where integrations live. Each action — "send Slack message", "create HubSpot deal", "check Stripe charge" — is implemented as a small package conforming to a common interface.

Adding a new integration takes a few hours. We have 50+ integrations and the layer hasn't required architectural changes in 18 months.

The trade-offs we made

Determinism over performance

Our engine recomputes the graph from scratch on every retry rather than caching intermediate states aggressively. This is slower than what Temporal does but means our retries are absolutely reproducible.

Simplicity over expressiveness

You can't do arbitrary Turing-complete computation in a Nimbus workflow. No loops over unbounded ranges. No recursion. We have a step type called "iterate" that's bounded, and that's the only way to repeat anything.

This limits expressiveness. It also means workflows are statically analysable — we can predict worst-case runtime before running it. Customer support agents love this.

JSON over typed schemas

Every step accepts JSON in and emits JSON out. Internally we use Rust enums but at the boundary it's all JSON.

This makes our engine slower than a strongly-typed system. It also makes integrations trivial to write and means we don't break when third-party APIs change shape.

What we got from building it ourselves

• A workflow language designed for business users. Most engines have query languages designed for engineers. Ours has primitives like "when this happens, do this" that read naturally.

• Deterministic retries. Every step is pure. Every retry is identical. Customer support never has to ask "are you sure this ran twice?"

• Sub-200ms cold start. Workflows feel synchronous even when they're not.

Would we do it again?

Yes. But I want to be honest about the cost: building this took 14 months of senior engineering time before we shipped a single customer feature. If you're a startup considering building your own engine — please don't, unless you have constraints similar to ours.

For 95% of workflows, Temporal is the right answer. Use Temporal.

— James Okafor