Skip to content

oryxflow vs Dagster: a lightweight research loop vs an asset platform

Dagster and oryxflow both build DAGs of Python steps and both persist step outputs automatically — so they get compared. But one is a production asset platform for a data team, and the other is a pip install for the research loop. Picking well starts with being honest about which problem you're solving.

If you searched "Dagster for machine learning experimentation," here's the short answer up front: Dagster is an excellent, mature orchestrator for a team running production data assets — and oryxflow is a small, local-first library for the fast, messy iteration that happens before anything is a production asset. oryxflow is a zero-infrastructure Python library that makes the data-science scripts you edit all day trustworthy and reproducible: a change to your code, data, or parameters reruns exactly what it affects, so no result quietly sits on stale inputs, and every run is tied to the code and inputs that made it. It stays cheap because nothing recomputes twice. Below is the honest breakdown, including where Dagster clearly wins.

What each tool is actually for

Dagster is an asset-oriented data orchestrator. You declare software-defined assets, and Dagster gives you a rich web UI, scheduling, sensors, partitions and backfills, and — importantly — IO managers that persist each asset's output automatically. It's built for a data team running and observing data assets in production: you can see every asset's freshness, backfill a partition, and trigger runs off sensors. The full experience assumes a running Dagster instance and its UI.

oryxflow is a research-loop tool. Its job is to make an analysis you're actively editing trustworthy and reproducible: reruns follow what your code and parameters actually changed, so you can't evaluate new logic against an old output, and every run is logged so you can say later which code and inputs produced any result. It's a pip install with no server, no database, no account, and no telemetry — the state lives in a local data/ folder and a plain lineage log. Every task's output is cached, which is what keeps all of that free rather than a tax on your rerun time.

Neither is a worse version of the other. They sit at different layers of the same project's life.

Doesn't Dagster already persist outputs automatically?

Yes — and this is the honesty checkpoint. Dagster's IO managers do persist asset outputs automatically. oryxflow did not invent automatic persistence; Dagster, Metaflow, Kedro, and Ploomber all do it too. So auto-persistence is not the differentiator, and any comparison that pretends otherwise is selling you something.

The real difference is how much you wire and what environment it assumes:

  • In Dagster, you declare assets and choose and configure an IO manager (or accept a default and configure where it writes). That's a powerful, explicit contract — and it's exactly right for a platform where storage, partitioning, and environments matter. It's also configuration you own.
  • In oryxflow, the task's base class decides the format — TaskPqPandas writes a DataFrame to parquet, TaskPickle pickles a model, TaskJson writes JSON. There's no IO manager to pick, no catalog to maintain, and no server. You write self.save(obj) and the next task calls self.inputLoad(); the format and the path are handled by type, keyed on (task, params).

That's the precise oryxflow differentiator: type-driven, zero-config I/O plus automatic code-change invalidation, entirely local. Task identity is native Python — a class with parameters — not a YAML asset definition or a catalog entry.

What about reruns when I change my code?

This is the move oryxflow is built around, and it's a trust feature before it's a speed one. It tracks each task's code — and every helper file it imports — comparing what your code does, not how it's written, so comments and reformatting don't count. Edit one task, or a helper it imports, and on the next run() oryxflow reruns exactly that task and everything downstream, so you can never read a number that was produced by logic you've since changed. The expensive upstream stays cached, and if you changed nothing, nothing recomputes.

import oryxflow
class LoadData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(load_data())            # cached once, keyed on (code, params)

@oryxflow.requires(LoadData)
class Features(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()
        self.save(add_features(df))       # reruns only when this code (or a helper) changes

@oryxflow.requires(Features)
class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.ChoiceParameter(choices=['ols', 'gbm'])
    def run(self):
        df = self.inputLoad()
        m = fit(df, self.model)
        self.save(m)
        self.saveMeta({'score': m.score})

# compare models — shared upstream (LoadData, Features) computes once
flow = oryxflow.WorkflowMulti(TrainModel, {'ols': {'model': 'ols'},
                                           'gbm': {'model': 'gbm'}})
flow.run()
flow.outputLoadMeta()                      # {'ols': {'score': ...}, 'gbm': {'score': ...}}

Dagster tracks asset materializations and code versions and can help you reason about staleness, but the everyday oryxflow loop — edit a function, rerun, only the affected tasks recompute, no server involved — is the thing it optimizes for. One honest caveat that applies to both tools: oryxflow makes your results reproducible, not correct. It guarantees the same inputs and code produce the same outputs; it does not check that your analysis is right. That's still your job.

Side by side

oryxflow Dagster
Built for a solo/small-team research loop you can trust production data assets for a team
Rerun on a code edit automatic, per-symbol, downstream too code versions + staleness in the platform
Task identity native Python class + params software-defined assets
Setup pip install, no server a running Dagster instance + UI for the full experience
Auto-persist outputs ✅ type-driven, zero config ✅ via IO managers you configure
Configuring I/O none — base class picks the format declare assets, choose/configure an IO manager
Scheduling / sensors / backfills ❌ (not its job) ✅ its core strength
Web UI / observability queryable lineage log (no UI) rich web UI
Partitions at scale ❌ ✅

Read the ❌ rows the right way: they aren't oryxflow "losing." They're Dagster being a platform. If you need a UI, schedules, sensors, and partitioned backfills, you need Dagster — a caching library has no business pretending to replace them.

They're complementary

The mature pattern uses both, in sequence:

  • Iterate with oryxflow. During research — where you change code constantly and need every rerun to be correctly scoped — a local trust layer pays off and a server is overhead.
  • Promote to Dagster when the pipeline stabilizes and needs to become an observed, scheduled, partitioned production asset for the team.

Because your oryxflow logic is already decomposed into tasks with explicit dependencies, wrapping the stable pipeline as Dagster assets later is mechanical, not a rewrite.

Which do you need?

  • You're iterating on a model or analysis, locally, editing all day, and you need each result to be one you can believe and regenerate → oryxflow. (Slow upstream steps stop rerunning too.)
  • A team needs to run, schedule, observe, and backfill production data assets with a UI → Dagster.
  • Both, at different stages → the common case. Iterate in oryxflow; promote to Dagster when it's production.

Takeaway

Dagster and oryxflow both auto-persist outputs, so don't choose on that. Choose on the layer: oryxflow is a zero-infrastructure, type-driven way to keep the research loop trustworthy and reproducible — nothing runs on stale inputs, and every result names the code that made it; Dagster is a server-backed asset platform for production. For experimentation you can defend, pip install and iterate — it costs you nothing in rerun time, because nothing recomputes twice — then hand the stable pipeline to Dagster when it's ready.

pip install oryxflow

Frequently asked questions

Is there a lighter-weight alternative to Dagster for local analysis?

Yes. oryxflow is a lightweight alternative to Dagster for local analysis: a pip install with no server, database, or UI. Where Dagster is an asset platform a team runs in production, oryxflow makes the research loop you edit all day trustworthy and reproducible — a code, data, or parameter change reruns exactly what it affects, and every run lands in a plain local lineage log. Because unchanged work is reused rather than recomputed, none of that costs you rerun time. Promote the stable pipeline to Dagster later.

Do I need a Dagster instance and UI just to cache pipeline steps locally?

No. Dagster's IO managers persist outputs, but the full experience assumes a running Dagster instance and its UI. oryxflow persists every task output locally with zero configuration — the task's base class picks the format and keys it on task and params — so you get dependency-aware steps that rerun when your code changes, from a pip install alone, with no server or catalog to maintain.