Is Rust a Viable Alternative to Python for ETL, Data Engineering, and Data Projects? (2025–2026)

Audience: senior engineer fluent in Rust, Python, and Elixir, deciding whether to invest in Rust for data work. Research date: 11 September 2026. Every factual claim below links to a page located or fetched during this research session.

Bottom line up front: Rust has decisively won the engine layer of the data stack — dataframes, query engines, file/format I/O, log pipelines, streaming databases, and even the new dbt parser are Rust today. It has not won the workflow layer: orchestration, exploration, ML, and visualization remain Python’s, and the pure-Rust equivalents are either absent or years behind. The honest 2026 answer is “yes, viable — as the engine and service layer of a hybrid stack,” not “yes, replace Python.”


1. State of the Rust data ecosystem

1.1 Polars — the flagship

Polars describes itself as an “extremely fast Query Engine for DataFrames, written in Rust,” with multi-threaded SIMD execution, lazy and eager modes with a query optimizer, a streaming engine for larger-than-RAM data, plugin extensibility, bindings for Python/Rust/Node.js/R/SQL, optional NVIDIA GPU acceleration, and Apache Arrow memory layout for zero-copy interop; the repo shows 39.7k stars and 749 contributors, with Python Polars 1.44.2 released 9 September 2026 (pola-rs/polars). The same README notes real limits: streaming can let you process a 250 GB dataset on a laptop, but advanced install guidance is required beyond ~2^32 (4.2 billion) rows or on pre-2011 CPUs lacking AVX2 (pola-rs/polars).

Architecture direction (2026): Polars 2.0 entered release-candidate stage on 2 September 2026, and its headline change is that all LazyFrame queries run on the streaming engine by default, which the team expects to be “easily 5x faster” in aggregate versus the in-memory engine — at the cost of no longer guaranteeing row order for join, group_by, unpivot and friends unless you opt into maintain_order=True (Pre-release of Polars 2.0). The streaming engine is built on morsel-driven parallelism, and — an important caveat for production — unsupported query fragments silently fall back to the in-memory engine (Polars in Aggregate, Dec 2025).

Published benchmark (PDS-H, derived from TPC-H, May 2025), run on an AWS c7a.24xlarge (96 vCPU / 192 GB), totals in seconds (Polars benchmarks):

SolutionSF-10 total (s)FactorSF-100 total (s)Factor
polars[streaming] 1.30.03.891.023.941.2
duckdb 1.3.05.871.519.651.0
polars[in-memory] 1.30.09.682.5152.277.7
dask 2025.5.146.0211.8548.5227.9
pyspark 4.0.0120.1130.9312.4315.9
pandas 2.2.3365.7194.0excluded (OOM / 2 orders of magnitude slower)—

Polars itself notes the honest asterisks: PDS-H results are not comparable to official TPC-H numbers, DuckDB edges out Polars at SF-100, and Polars loses query 21 because range joins are not yet implemented in the streaming engine (Polars benchmarks).

Rust library vs Python package — the adoption gap is stark. The Python polars package logged 58.35M downloads in the last 30 days (807.8M all-time) (pepy: polars), while the Rust polars crate logged 845,803 downloads per month, latest 0.53.0 (8 Feb 2026) (lib.rs: polars). Polars is also explicit that the Rust surface is the less stable one: “Breaking changes to the Rust API are not deprecated first,” because supporting deprecations “would slow down development too much at this point in time” (Polars versioning). If you write Rust against Polars, expect churn the Python users never see.

Commercial backing: Polars raised an €18M Series A led by Accel with BCV participating, and states adoption grew “from 250k to over 23M monthly users” (Polars Series A). TechCrunch puts the round at ~21M** (announced 29 Sept 2025), following a **4M seed led by Bain Capital in 2023, with 24M+ downloads, production use in finance, life sciences and logistics, and most of the new money going to Polars Distributed — an explicit run at Spark’s market (TechCrunch). Polars Cloud is now GA on AWS, with the distributed engine in public/open beta, spinning compute up and down in the customer’s own account (Launch of Polars Cloud, Polars in Aggregate Dec 2025).

1.2 Apache Arrow / arrow-rs — the substrate

arrow-rs is the official Rust implementation of Arrow plus Parquet: 3.6k stars, 843 contributors, “used by” 11K repos, latest arrow 59.3.0 on 1 September 2026, with releases roughly monthly and breaking major versions at most quarterly (apache/arrow-rs). This cadence is the single most important operational fact about the Rust data stack: everything downstream (DataFusion, delta-rs, iceberg-rust) inherits a fast-moving, frequently-major-versioned core.

The object_store crate — the cloud-storage abstraction used across the stack (S3, GCS, Azure, local, memory) — was donated to Apache Arrow by InfluxData (InfluxData: Rust Object Store Donation) and now lives in its own repo (apache/arrow-rs-object-store); it ships ~5.58M downloads/month, latest 0.14.0 (22 June 2026) (lib.rs: object_store).

1.3 Apache DataFusion — the embeddable query engine

DataFusion became an ASF Top-Level Project on 11 June 2024, described as “a fast, extensible query engine for building high-quality data-centric systems in Rust,” usable unmodified as an embedded SQL engine or as the foundation for new systems, with built-in Parquet/CSV/JSON/Avro, extension traits, a real optimizer, streaming async I/O from S3/Azure/GCS, and Substrait support (ASF announcement). Repo metrics: 7.9k stars, 883 contributors, used by 3,217 repos (apache/datafusion).

Velocity is extreme: DataFusion 55.0.0 (25 August 2026) landed 877 commits from a record 175 contributors in ~10 weeks, with representative wins including 4.2× faster TopK via runtime row-group pruning, up to 14.7× faster IN-list evaluation, approx_distinct 101× faster (1723 ms → 17 ms at 50k groups), and a production Comet query whose Parquet reads dropped from 1.35 TB to 30.9 GB through nested-column leaf pruning (DataFusion 55.0.0).

On raw single-node performance, DataFusion 43 became the fastest engine for querying Parquet in ClickBench, beating DuckDB, chDB and ClickHouse on identical hardware (c6a.4xlarge, 16 vCPU/32 GB, 14 GB partitioned Parquet, 100 files, “hot” run, results 16 Nov 2024) — the first Rust engine to hold that spot; the same post concedes DuckDB was slightly faster on the cold run (DataFusion blog).

Who runs it in production. InfluxData — whose InfluxDB 3.0 is built on the Rust “FDAP” stack (Flight, DataFusion, Arrow, Parquet) — reports that “our multi-tenant production systems alone execute 10s of millions of DataFusion plans daily,” notes 94+ contributors to a single release, and frames 2025 as DataFusion entering a “hardening” phase while admitting real pain: downstream upgrade friction from release velocity, feature gaps and “increasingly esoteric bugs” on complex queries, and review capacity as the project’s scarcest resource (InfluxData: 2025, the Year of 1,000 DataFusion-Based Systems). InfluxData also explains the language choice and architecture in Flight, DataFusion, Arrow, Parquet: the FDAP stack.

DataFusion’s own “Known Users” list is the best single proof of ecosystem gravity — it includes Arroyo, ArkFlow, Auron, Ballista, CnosDB, Comet, Cube Store, dbt Fusion engine, delta-rs, Feldera, Funnel, GlareDB, GreptimeDB, HoraeDB, iceberg-rust, InfluxDB, Kamu, Kubeflow Trainer, LakeSoul, Lance, OpenObserve, ParadeDB, Parseable, Cloudflare’s R2 Query Engine, rerun.io, Restate, ROAPI, Sail, SedonaDB, Spice.ai, Synnada, VegaFusion and Vortex (DataFusion introduction docs, rendered at datafusion.apache.org). GreptimeDB confirms independently that “GreptimeDB’s query engine is built on Apache DataFusion” (GreptimeDB docs).

DataFusion Comet is the Rust-native accelerator for Spark (originally built at Apple and donated to the ASF, per InfluxData). Its README reports 22 TPC-H queries at 100 GB Parquet on a single executor with 8 cores: Spark 652 s vs Comet 268 s = 2.4× speedup, aims at 100% Spark compatibility with no code changes, and needs no GPUs/FPGAs (1k stars, 83 contributors) (apache/datafusion-comet). Comet’s project docs also advertise ~2× on TPC-DS SF1000 (1 TB) ≈ ~50% cost savings, per the same README.

1.4 Distributed compute in Rust: Ballista, Daft, Polars Distributed, Sail

Ballista distributes DataFusion across nodes and reports an overall 2.9× speedup vs Apache Spark on TPC-H-derived queries at scale factor 100 (single node, single executor, 8 concurrent tasks) — while stating plainly that “there is a gap between DataFusion and Ballista” that may bring incompatibilities (1.8k stars, 30 tags) (apache/datafusion-ballista). It is still shipping in lockstep with DataFusion: 53.0.0 in May 2026 (release notes) and 54.0.0 in July 2026 (release notes).

Daft is the most Python-forward of the Rust engines: “a distributed query engine for large-scale data processing using Python or SQL, implemented in Rust,” 46.4% Rust / 36.1% Python by source, 5.8k stars, 198 contributors, v0.7.24 (14 Aug 2026), scaling from local to Ray/Kubernetes, with multimodal (image/audio/video) and LLM/embedding operators as first-class citizens (Eventual-Inc/Daft).

Polars Distributed (public beta) and Polars Cloud (GA on AWS) are the commercial distributed path, reusing the open-source streaming engine on distributed workers (Polars Cloud launch). Sail (“unifying stream, batch and AI workloads with Apache Spark compatibility”) is another DataFusion-based Spark-replacement effort (DataFusion known users).

1.5 Database → dataframe loading: ConnectorX

ConnectorX (“Fastest library to load data from DB to DataFrames in Rust and Python”, 2.5k stars, v0.4.4 on 26 Aug 2025) uses zero-copy, schema-preallocated, partition-parallel loading into pandas/PyArrow/Modin/Dask/Polars (sfu-db/connector-x). Its README benchmark — loading a 10× TPC-H lineitem table (8.6 GB) from Postgres with 4-core parallelism — reports 13× less time and 3× less memory than pandas read_sql, and up to 21× less time / 3× less memory versus the slowest Python alternatives (README). Caveats from the repo itself: partitioning only on non-null numerical columns for SPJA queries, federated queries are experimental, ODBC support is WIP, and release cadence is slow (13 releases total, last in Aug 2025) (sfu-db/connector-x).

1.6 Lakehouse formats: delta-rs and iceberg-rust

delta-rs is “a native Rust library for Delta Lake, with bindings into Python” — 3k stars, 237 contributors, 98 releases, latest python-v1.2.1 (20 Oct 2025) — covering create/read/vacuum/delete/optimize (compaction, Z-order)/merge/update/constraints/restore across S3, Azure, GCS, HDFS, OneLake and LakeFS (delta-io/delta-rs). Independent benchmarking found delta-rs v1’s Rust engine materially better than the old PyArrow path (it fixed OOMs at 2–4 vCores for DuckDB) and significantly faster at VACUUM than Spark, but also flagged that delta-rs lacks deletion-vector write support, forcing “lowest common denominator” table configs (Miles Cole, The Small Data Showdown ‘25).

iceberg-rust is real but visibly behind Java: release 0.9.0 covered Jan–Mar 2026 with 109 PRs from 28 contributors (Apache Iceberg Rust 0.9.0), and Iceberg’s own implementation-status matrix marks Rust as not supporting update-schema/update-partition-spec on Table Spec V2, expire/manage snapshots, rewrite files, rewrite manifests, overwrite files, row delta, delete files, or ORC/Puffin/Avro data files — all of which Java has (Iceberg implementation status). If your lakehouse work needs full table maintenance, Rust is not yet a complete substitute for the JVM implementation.

1.7 File formats and serialization

serde remains the backbone of Rust data plumbing at 79.48M downloads/month (1.0.228, Sept 2025) (lib.rs: serde). Parquet ships inside arrow-rs alongside arrow-flight and parquet_derive (apache/arrow-rs). For spreadsheets, calamine reads xls/xlsx/xlsm/xlsb/xla/xlam and ODS in pure Rust at 769,830 downloads/month (0.35.0, 10 May 2026), with a published benchmark against Go’s excelize, C#‘s ClosedXML and Python’s openpyxl on a 1,000,001-row × 41-column workbook (lib.rs: calamine, docs.rs: calamine).

1.8 Log/telemetry pipelines: Vector (Datadog)

Vector is the strongest “Rust replaced an incumbent” story in data movement: 22.5k stars, 674 contributors, >100,000 downloads per day, a largest user pushing over 500 TB daily, and named adopters including Atlassian, T-Mobile, Comcast, Zendesk, Discord, Fastly, CVS, Visa, Instacart and Trivago (vectordotdev/vector). Datadog maintains it as a high-performance agent/aggregator for logs, metrics and traces (Datadog Open Source Hub), and it is marketed as “blistering fast, memory efficient” (vector.dev). Its own published numbers are refreshingly non-uniform: Vector leads TCP-to-blackhole (86 MiB/s vs FluentBit 64.4, Logstash 40.6, Fluentd 27.7) and file-to-TCP (76.7 vs Filebeat 7.8, Logstash 3.1), but loses regex parsing to FluentBit (13.2 vs 20.5 MiB/s) and ties/loses TCP-to-TCP against Splunk UF (vectordotdev/vector).

1.9 Streaming / real-time engines written in Rust

  • RisingWave — PostgreSQL-compatible streaming database in Rust: 9.3k stars, 228 contributors, v3.0.3 (17 Aug 2026), claiming sub-100 ms end-to-end freshness, 10–20 ms p99 serving latency, and state on object storage “roughly 100× cheaper than RAM” (risingwavelabs/risingwave); the team documented its migration from C++ to Rust (RisingWave blog).
  • Arroyo — distributed stream processing in Rust, SQL-first, checkpointed state, “scales to millions of events per second”: 5.0k stars, 45 contributors, v0.15.0 (1 Dec 2025), still on a 0.x line with active mainline commits in Sept 2026 (ArroyoSystems/arroyo).
  • Feldera, Synnada, ArkFlow, Kamu, Streamling, Supermetal — all DataFusion-based streaming/CDC systems (DataFusion known users).
  • Quickwit — Rust log/trace search decoupled from storage (Quickwit).
  • Lance / LanceDB — Rust-implemented columnar format for ML with Python and Java bindings (lance contributing docs).

1.10 tokio and async I/O for I/O-bound ETL

For the “thousands of concurrent API/DB calls” shape of ETL, tokio is mature infrastructure: 33.1k stars, 1,024 contributors, v1.53.1 (20 July 2026), a work-stealing multithreaded scheduler over epoll/kqueue/IOCP, MSRV 1.71 with a rolling ≥6-month MSRV policy, and explicit backpressure/cancellation handling (tokio-rs/tokio). DataFusion and object_store build directly on this async foundation for streaming reads from object stores (ASF DataFusion announcement, object_store docs). This is the area where Rust’s advantage over Python is most structural: no GIL, no per-task interpreter overhead, and compile-time enforcement of Send/Sync.

1.11 The orchestration gap (Rust’s clearest weakness)

There is no Rust-native equivalent of Airflow/Dagster/Prefect with comparable adoption. The closest credible project, Windmill, has a Rust backend but is positioned as a developer platform / “simplified Temporal” whose scripts are written in Python, TypeScript and other languages; it claims to be the “fastest workflow engine — 13x vs Airflow” on flows of 40 lightweight or 10 long-running tasks, has 17.9k stars, 175 contributors, v1.810.0 (11 Sept 2026), and ships a Community Edition under AGPLv3 that also contains proprietary, non-public code (windmill-labs/windmill). Kestra — the other frequently cited alternative — is declarative/YAML and JVM-based, not Rust (Kestra). Community “Rust orchestrator” projects exist (e.g. IronFlow, taskgraph-rs) but are early-stage hobby/utility efforts (r/rust: IronFlow, lib.rs: taskgraph-rs). Meanwhile the incumbent Python orchestrator ecosystem is consolidating and investing heavily (see §2 and §5). Practical consequence: even a Rust-heavy shop schedules its Rust binaries from a Python (or YAML/JVM) orchestrator.

1.12 PyO3 / maturin — how Rust actually reaches data teams

The dominant delivery mechanism for Rust in data is embedding into Python, not replacing it. PyO3 (16.1k stars, 465 contributors, “used by” 32,492 repos, v0.29.2 on 5 Aug 2026, requires Rust ≥1.83, supports CPython 3.9+/PyPy/GraalPy) provides the bindings (PyO3/pyo3), and maturin builds and publishes the wheels for Python 3.8+ across Windows/Linux/macOS/FreeBSD with minimal configuration (PyO3/maturin). PyO3’s own front page advertises the pattern’s payoff with links titled “Parsing Python ASTs 20x Faster with Rust” and “Making Python 100x faster with less than 100 lines of Rust” (PyO3/pyo3).

The 2025–2026 evidence that this is the winning pattern:

  • Polars: a Rust engine whose users are overwhelmingly Python (58.35M/month PyPI vs 0.85M/month crates.io — see §1.1).
  • dbt: dbt Labs open-sourced the Fusion runtime as dbt Core v2.0 under Apache 2.0, whose Rust-based engine gives up to 10× faster parse times; a Rust parser also shipped in dbt Core v1.12 beta for teams not ready for a major upgrade. dbt itself has >1 billion PyPI downloads and >100,000 weekly active projects (Fivetran/dbt blog). The dbt Fusion engine is DataFusion-based (DataFusion known users).
  • Python tooling generally: ruff is “an extremely fast Python linter and code formatter, written in Rust… 10–100x faster than existing linters (like Flake8) and formatters (like Black)” (Astral docs), and Rust-written uv was the most admired technology in the 2025 Stack Overflow survey at 74%, with Rust’s own Cargo the most admired cloud/infra tool at 71% (Stack Overflow 2025).

1.13 Named companies and case studies using Rust for data infrastructure

OrganizationWhat Rust does thereSource
DatadogMaintains Vector, the Rust logs/metrics/traces pipeline (>100k downloads/day; one user >500 TB/day)Datadog OSS hub, repo
InfluxDataInfluxDB 3.0 rebuilt in Rust on the FDAP stack; 10s of millions of DataFusion plans/dayFDAP post, DataFusion 2025
Apple → ASFBuilt a native Spark execution replacement, donated as DataFusion CometInfluxData, Comet repo
CloudflareR2 Query Engine (SQL over Iceberg catalogs) built on DataFusionDataFusion known users
dbt Labs / Fivetrandbt Fusion / dbt Core v2.0 Rust engine, up to 10× faster parsingFivetran blog
GreptimeDBRust time-series DB with DataFusion query engine; Postgres wire compat via pgwiredocs, blog
DiscordRewrote Read States service Go→Rust to eliminate GC latency spikesDiscord blog
MetaRust is a primary supported server-side language (with C++, Python, Hack); in use since 2016Meta Engineering 2022, 2024
Eventual (Daft), Polars B.V., RisingWave, Arroyo, ParadeDB, LanceDB, Spice.ai, Cube, FelderaVenture-backed data products with Rust coresDaft, Polars, RisingWave, Arroyo, DataFusion users

2. State of the Python data ecosystem (the incumbent)

Raw usage. pandas has been downloaded 15.61 billion times total, 734.06M in the last 30 days, latest 3.0.3 (11 May 2026) (pepy: pandas) — roughly 12.6× Polars’ 58.35M/30 days (pepy: polars). Independent survey data agrees on the shape: in the DataTalks.Club 2025–2026 data engineering survey, pandas 70%, Spark 58%, Polars 8%, Flink 6% for processing frameworks; Airflow 46%, Step Functions 15%, Prefect 10%, Kestra 6%, and 19% using no orchestrator at all; dbt 46%, dlt 20%, Airbyte 9%, Fivetran 7% for integration/transformation (DataTalks.Club survey).

Orchestration is where Python’s moat is deepest. Astronomer’s State of Airflow 2026 — 5,800+ respondents across 122 countries, “the largest survey of data engineers ever conducted” — reports Airflow 3 (April 2025, first major in over four years, 300+ contributors) already adopted by 26% of Airflow users, 48% of Astro customers, 60% of Astro’s largest enterprises, with 84% of 2.x users planning to upgrade; 32% of Airflow users run GenAI/MLOps in production; and OpenAI standardized on Airflow in 2023 with roughly 7,000 pipelines (State of Airflow 2026, press release).

DuckDB is Python’s (and SQL’s) other answer to Polars — and it is not Rust. DuckDB is 92.7% C++ by source, 41.2k stars, 682 contributors, v1.5.5 (22 July 2026) (duckdb/duckdb), with v2.0 alpha in September 2026 ahead of an October release (DuckDB blog). DuckCon #7 (June 2026) put adoption at roughly 40M downloads/month (up from 4M in Aug 2024) and 39k GitHub stars (State of the Duck, DuckCon #7). Importantly for a Rust-vs-Python decision: much of the “escape pandas” performance win is available without Rust, via DuckDB’s SQL in-process engine, and DuckDB beat Polars on the SF-100 PDS-H run (Polars benchmarks).

Spark still wins the big end. In an independent, reproducible Fabric-based ELT benchmark, Spark with vectorized native execution was the only engine to complete a ~127 GB-compressed workload at every compute size; Polars OOM’d minutes in, Daft failed after an hour, and DuckDB (the only single-node survivor) was ~3.5× slower than Spark at 32 vCores and ~6× slower at 64 vCores (Miles Cole).

Why Python dominated, restated in evidence terms: the breadth (pandas/NumPy/scikit-learn/PyTorch), the orchestration and transformation layer (Airflow, Dagster, Prefect, dbt with >1B downloads), notebooks, and the fact that the fastest engines all ship first-class Python bindings — Polars, DuckDB, Daft, DataFusion (via datafusion-python), delta-rs — so Python users capture Rust’s performance without writing Rust (pola-rs/polars, apache/datafusion, delta-io/delta-rs, Daft).


3. Head-to-head comparison

DimensionRustPythonEvidence
Single-node dataframe throughputPolars streaming totalled 3.89 s (SF-10) / 23.94 s (SF-100) on PDS-Hpandas 365.71 s at SF-10 (94×) and excluded at SF-100 for OOM / 2-orders-of-magnitude slowness; PySpark 120.11 s / 312.43 s; Dask 46.02 s / 548.52 sPolars benchmarks
SQL engine performanceDataFusion was fastest ClickBench Parquet engine (hot run, c6a.4xlarge), first Rust engine to top itDuckDB (C++) slightly faster cold; DuckDB won PDS-H SF-100DataFusion, Polars
Accelerating existing SparkComet: 652 s → 268 s (2.4×) on TPC-H 100 GB, 8 cores; ~2× TPC-DS 1 TB ≈ 50% costSpark/JVM baseline; Fabric’s C++ Velox path gets similar wins without RustComet, Miles Cole
Distributed compute maturityBallista 2.9× vs Spark but “gap between DataFusion and Ballista”; Polars Distributed in public beta; Daft on RaySpark completed a 127 GB workload at all compute sizes where Polars/Daft failedBallista, TechCrunch, Miles Cole
DB → dataframe extractionConnectorX: 13× less time, 3× less memory vs pandas read_sql (8.6 GB Postgres, 4 cores)pandas read_sql baseline; ConnectorX is itself consumed from PythonConnectorX README
I/O-bound concurrencytokio: work-stealing scheduler, backpressure/cancellation, no GIL, Send/Sync checked at compile timeasyncio + GIL; parallelism usually via processes or native extensionstokio
Memory safety / reliabilityarrow-rs holds an explicit “Soundness Pledge” — intent to be free of soundness bugsRuntime type errors surface late; Polars’ own goal is to “fail fast” because errors “20 minutes into a pipeline” are the pain being fixedarrow README, Polars 2.0
Compile-time guarantees vs flexibilityStatic typing + borrow checker; schema errors still runtime for dynamic framesFully dynamic; collect_schema()-style checks are opt-inPolars 2.0
Developer velocity / iterationSlow compile times are still the top-cited non-trivial Rust problem; Polars breaks Rust APIs without deprecation cyclesREPL/notebook loop; pandas/Polars Python APIs get deprecation warnings2025 State of Rust, Polars versioning
Learning curveComplexity concerns persist in the official survey; 30% of surveyed Rust users started <1 month ago and 52% are still learningPython is the top language for learning to code per SO 2025 podcast discussion; adoption +7 pp 2024→20252025 State of Rust, JetBrains, SO 2025
Hiring pool / talent45% of respondents’ orgs made non-trivial use of Rust (2024, +7 pp); only 26% use Rust in professional projects (JetBrains 2025); official survey confirms a hiring trend for Rust devsPython usage rose 7 pp in one year and is the language of AI/data science2024 State of Rust, JetBrains, 2025 State of Rust, SO 2025
Ecosystem breadth: MLburn 15.9k stars but “active development… breaking changes… no guarantees at this stage,” limited ONNX ops, LibTorch backend deprecated; candle 21.0k stars, explicitly “minimalist,” aimed at serverless inference; linfa 4.3k stars, 0.8.0 (Sept 2025), used by 752 reposPyTorch/scikit-learn/XGBoost ecosystem; Airflow users running GenAI/MLOps in production at 32%burn, candle, linfa, State of Airflow 2026
Notebooks / interactive explorationevcxr Jupyter kernel: 6.2k stars, latest release v0.21.1 in July 2025, 50 contributors, “used by 167”Jupyter is the default surface; Daft ships 16.3% notebooks in-repoevcxr, Daft
VisualizationServer-side acceleration exists (VegaFusion) but rendering/BI stays elsewherematplotlib/plotly/Vega ecosystemDataFusion known users
OrchestrationNo mainstream Rust-native orchestrator; Windmill (Rust backend, polyglot scripts, AGPLv3 + proprietary bits) claims 13× AirflowAirflow 46% adoption; Airflow 3 with 300+ contributors; Dagster/Prefect now one companyWindmill, DataTalks, Astronomer, Dagster
Lakehouse table-format completenessiceberg-rust missing V2 schema/partition updates, snapshot expiry, rewrite/overwrite/delete files, ORC/Puffin/Avro; delta-rs cannot write deletion vectorsPyIceberg + Java cover far more of the spec; Spark implements native VACUUM/OPTIMIZEIceberg status, Miles Cole
InteroperabilityArrow columnar format enables zero-copy sharing; PyO3/maturin ship Rust as wheels; Substrait moves plans across languagesSame mechanisms, consumed from Python — 32,492 repos already depend on PyO3Polars, PyO3, ASF DataFusion
Deployment / packagingSingle static binary (Vector ships as one binary, >100k downloads/day); maturin builds cross-platform wheelsInterpreter + dependency graph; uv (Rust) is now the most admired tool for fixing exactly thisVector, maturin, SO 2025
Operational observability of the engineIndependent benchmarker: with single-node Rust/C++ engines “there’s zero visibility” into a running query, vs Spark UI/task metricsSpark UI, Airflow UI, Dagster asset lineageMiles Cole
Release-churn burdenarrow-rs majors up to quarterly; DataFusion 877 commits/10 weeks; InfluxData names “downstream upgrade friction” as a real costpandas/Airflow move slowly and deprecate loudlyarrow-rs, DataFusion 55, InfluxData

4. When Rust makes sense vs when Python still wins

Rust is the right call when…

  1. You are building reusable data infrastructure, not a pipeline. Every marquee Rust data success is a product: Vector, InfluxDB 3, RisingWave, Polars, dbt Fusion, Cloudflare R2 Query Engine, GreptimeDB (Vector, InfluxData, RisingWave, Fivetran/dbt, DataFusion users). If your artifact will be run by other teams thousands of times a day, Rust’s cost amortizes.
  2. High-throughput streaming and log/telemetry movement. Vector’s file-to-TCP throughput is ~10× Filebeat’s and ~25× Logstash’s, and one deployment moves >500 TB/day (Vector).
  3. Latency-sensitive paths where GC or interpreter pauses are the bug. Discord’s Go→Rust rewrite was motivated precisely by GC latency spikes (Discord); RisingWave targets 10–20 ms p99 serving (RisingWave).
  4. Cost-sensitive cloud compute at scale. Comet’s ~2× on TPC-DS 1 TB is stated as ~50% cost savings with no Spark code changes (Comet); DataFusion 55 cut one production query’s Parquet reads from 1.35 TB to 30.9 GB (DataFusion 55).
  5. Massively concurrent I/O-bound extraction. tokio + object_store + ConnectorX is a genuinely better substrate than asyncio for thousands of concurrent object/DB reads (tokio, object_store, ConnectorX).
  6. Embedded / edge / single-binary deployment. Vector ships as one binary (vector.dev); InfluxData explicitly markets its Rust client for “edge gateways, embedded systems, and high-throughput ingest pipelines” (InfluxData); Burn is the only Rust DL framework with a no_std-capable backend (burn).
  7. You want to speed up an existing Python data tool. Write a PyO3 extension, not a rewrite (PyO3, maturin) — the same play dbt, Astral and Polars ran (Fivetran/dbt, Ruff).

Python still wins when…

  1. Exploratory analysis and notebook-driven work. Rust’s flagship notebook kernel, evcxr, last released July 2025 and reports 50 contributors — versus Jupyter as the industry default (evcxr).
  2. ML / data science glue. The Rust DL frameworks self-describe as in-flux (“breaking changes… no guarantees at this stage,” limited ONNX operator coverage) or deliberately minimalist for inference (burn, candle), and classical ML (linfa) is small: 0.8.0, 55 contributors, 752 dependent repos (linfa).
  3. Orchestration, scheduling, lineage, catalogs. Airflow 46% adoption, Airflow 3 with 300+ contributors and OpenAI’s ~7,000 pipelines; nothing in Rust is close (DataTalks, State of Airflow 2026).
  4. Rapid prototyping and one-off transforms. Compile times remain the top-ranked non-trivial Rust problem in the official survey (2025 State of Rust).
  5. You need the mature, boring path. An independent benchmarker who loves Polars and DuckDB still recommended Spark for most small-data scenarios, citing maturity (Spark code “just ran” a version later, 2× faster, zero changes), triage visibility, composable-stack version management, and Delta feature gaps (Miles Cole).
  6. Full lakehouse table maintenance. Rust Iceberg lacks V2 schema/partition updates, snapshot expiry and file rewrite/overwrite/delete; delta-rs can’t write deletion vectors (Iceberg status, Miles Cole).
  7. Data beyond a single fat node. Polars OOM’d and Daft failed where Spark completed a 127 GB-compressed workload (Miles Cole).

5. Recent developments (2024–2026)

Governance and engine maturity

  • DataFusion became an ASF Top-Level Project on 11 June 2024 (developed inside Apache Arrow since 2019) (ASF, Arrow blog); Wikipedia summarizes the 2017 origin → 2019 Arrow donation → 2024 TLP arc (Wikipedia).
  • Nov 2024: DataFusion 43 became the fastest single-node ClickBench Parquet engine (DataFusion).
  • 2025: InfluxData’s Andrew Lamb publicly predicted “the year of 1,000 DataFusion-based systems” and documented the project’s move past early adopters into hardening (InfluxData).
  • Aug 2026: DataFusion 55.0.0 — 877 commits, record 175 contributors (release); Nov 2025 had already shipped 51.0.0 (release).
  • 2026: Ballista 53.0.0 (May) and 54.0.0 (July) shipped with production-deployment, shuffle and REST/observability work (53.0.0, 54.0.0).
  • arrow-rs reached arrow 59.3.0 (Sept 2026) on a monthly cadence with 60.0.0 in flight (repo); iceberg-rust shipped 0.8.0 (144 PRs / 37 contributors) and 0.9.0 (109 PRs / 28 contributors) between Nov 2025 and Mar 2026 (0.8.0, 0.9.0).

Money and market structure

  • **Polars raised €18M/~21M Series A (Accel, BCV) in Sept 2025**, after a 4M 2023 seed, and shipped Polars Cloud GA on AWS plus Polars Distributed in public beta — openly targeting Spark (Polars, TechCrunch, Polars Cloud, RTInsights).
  • Sept 2026: Polars 2.0 RC makes the streaming engine the default (announcement).
  • Oct 2025: Fivetran and dbt Labs agreed to merge in an all-stock deal (Fivetran press release, Reuters), later completing (Fivetran) — and the combined company open-sourced the Rust Fusion runtime as dbt Core v2.0 (Fivetran blog).
  • 13 July 2026: Prefect acquired Dagster Labs, with ~40 Dagster staff joining and Dagster OSS/Dagster+ continuing (Dagster, Prefect, BusinessWire). Net effect: the Python orchestration layer consolidates rather than cedes ground.
  • DuckDB hit ~40M downloads/month by June 2026 and is heading into v2.0 (DuckCon #7, DuckDB blog).

Community sentiment and adoption data

  • 2024 State of Rust: 45% of respondents’ organizations made non-trivial use of Rust (+7 pp YoY), 38% write Rust for the majority of their work coding (up from 34%), and 82% said Rust helped their company achieve its goals — with popularity concentrated in server backends, web/networking services and cloud, not data science (2024 results).
  • 2025 State of Rust (7,156 completed responses, 17 Nov–17 Dec 2025, 10th edition): results “closely match” 2024 with sub-point differences; resource usage — slow compile times and storage — remains the top non-trivial problem; complexity concerns persist; and the report notes a confirmed hiring trend among organizations looking for more Rust developers (2025 results).
  • JetBrains State of Rust 2025: 65% use Rust for side/hobby projects, 52% are currently learning it, only 26% use it in professional projects, 30% started within the last month, three-quarters target Linux (JetBrains).
  • Stack Overflow 2025: Rust-written uv is the most admired technology overall (74%) and Cargo the most admired cloud/infra tool (71%), while Python adoption rose 7 percentage points in a single year on the strength of AI/data science (SO 2025, technology section).
  • Practitioner sentiment, from the HN thread on “Why Rust is taking the data engineering world by storm,” captures the consensus well: “Rust is the language if you have a well-defined data crunching problem to automate. It’s not so great for exploratory work but for production…” (Hacker News); an earlier “Show HN: Data Engineering with Rust” thread argued Rust is “a very serious and good option for data engineering workloads” while acknowledging Java/Scala/Python dominance (Hacker News).

6. Verdict

Rust is viable — as the engine, not the ecosystem. The evidence for the engine layer is now overwhelming and primary-sourced: a Rust engine holds (or contests) the single-node ClickBench Parquet crown (DataFusion); a Rust dataframe is 94× faster than pandas on PDS-H SF-10 and finishes SF-100 where pandas OOMs (Polars); a Rust accelerator halves Spark’s TPC-H runtime with zero code changes (Comet); a Rust log router moves >500 TB/day at Datadog-scale users (Vector); and the canonical Python transformation tool now runs a Rust parser (dbt Core v2.0).

But “pure-Rust stack” is not a real option in 2026. The gaps are specific, not hand-wavy: no mainstream Rust-native orchestrator (Windmill is polyglot and partly proprietary; Kestra is JVM/YAML) against Airflow’s 46% share and Airflow 3’s 300+ contributors (DataTalks, Astronomer); a Jupyter kernel whose last release was July 2025 (evcxr); ML frameworks that self-describe as breaking and minimalist (burn, candle); table-format implementations with documented feature holes (Iceberg status); zero query-level observability compared to the Spark UI (Miles Cole); and a maintenance tax from arrow-rs/DataFusion release velocity that even DataFusion’s biggest production user calls out as “downstream upgrade friction” (InfluxData).

The most telling single number: the Rust polars crate sees ~0.85M downloads/month while the Python polars package sees ~58M/month (lib.rs, pepy). The market has voted for Rust engines with Python front-ends — and Polars reinforces that by refusing to deprecate Rust APIs before breaking them (versioning policy).

Also worth internalizing: much of the performance case for “leaving Python” does not require Rust at all. DuckDB (C++) beat Polars on PDS-H SF-100 and was the most robust single-node engine in independent ELT testing (Polars benchmarks, Miles Cole), and Fabric’s Spark gains come from C++ Velox/Gluten (Miles Cole). Choose Rust for what only Rust gives you — memory safety with C++-class performance, fearless concurrency, single-binary deploys — not for “fast,” which is available in several forms.

  1. Adopt Rust engines from Python first (week 1). Polars lazy + streaming, DuckDB, ConnectorX, delta-rs. This captures most of the performance for none of the risk (Polars, ConnectorX, delta-rs). If you’re on Spark, try Comet before rewriting anything (Comet).
  2. Write your first Rust in a PyO3 extension, targeting a profiled hot loop or parser, and ship it with maturin as a wheel your Python team already knows how to install (PyO3, maturin). This is precisely the path dbt, Polars and Astral took (dbt, Ruff).
  3. Then take one standalone, long-lived Rust service where the payoff is structural: a high-throughput ingest/CDC/telemetry service (tokio + object_store + arrow/parquet), or a custom query surface embedding DataFusion (tokio, object_store, ASF DataFusion). Budget for quarterly arrow-rs/DataFusion major upgrades from day one (arrow-rs).
  4. Keep orchestration, notebooks, ML and BI in Python. Schedule Rust binaries from Airflow/Dagster/Prefect; do not attempt to replace them (Astronomer, Dagster).
  5. Do not use Rust for exploration. Prototype in Polars-Python or DuckDB SQL; port only the parts that must be a binary. Compile times remain the #1 complaint of Rust’s own users (2025 State of Rust).
  6. Hiring reality check: only 26% of surveyed Rust developers use it professionally, and roughly half are still learning it — a Rust-heavy data platform is a real staffing constraint, even though org-level adoption and Rust hiring are both trending up (JetBrains, 2024 State of Rust, 2025 State of Rust).

For your Elixir instincts specifically: the closest analogue is that Rust plays the role of NIF-grade infrastructure — you reach for it where a supervised, GC’d, dynamically-typed runtime cannot meet a throughput, latency, or memory-footprint requirement. That is a large and growing slice of data engineering (ingest, streaming, query engines, format I/O), and a small slice of data analysis. Invest accordingly: deep in Rust for the engines and services you’ll operate for years, and keep Python as the coordination and exploration layer.


Notes on evidence quality and gaps

  • Polars’ PDS-H and Comet’s TPC-H/TPC-DS results are vendor/project-run and derived from, not compliant with, official TPC benchmarks (Polars, Ballista). The most useful independent counterweight found this session is Miles Cole’s LakeBench ELT benchmark, which is reproducible and reaches a more Spark-favorable conclusion.
  • The Rust Project’s 2025 survey publishes several key breakdowns (organizational adoption, domain mix) only in charts/PDF; the org-adoption and work-usage percentages cited here are the officially stated 2024 figures, with the 2025 post confirming results “closely match” the prior year (2024, 2025). Secondary write-ups quoting 45.5%/48.8% for 2025 were not used as primary evidence.
  • Star counts, contributor counts, release versions and download figures are point-in-time as of 11 September 2026 and move quickly.