Status & limitations
What works, what doesn’t, and what is known to be wrong
marrow is experimental. Both Apache Arrow and Mojo are moving targets. This page lists what works and what does not, with a pointer to the source each line comes from, so it can be checked rather than taken on trust.
How correctness is measured
Two independent checks, neither of which grades marrow against itself:
- The golden corpus —
golden/, 285 query cases. Each records a SQL query, the answer generated from DuckDB, and the marrow spelling. The expectations are never regenerated from marrow. The corpus currently carries zeroxfails: every case that compiles, passes. - Arrow archery integration — the official cross-implementation suite, run against the C++, Rust and Go implementations in both directions (
pixi run -e integration integration), skipping the layouts marrow does not implement: unions, list views, run-end encoding and extension types. Marrow reads the suite’s JSON, writes and reads IPC, and exports and imports C Data itself; pyarrow only compares the results. See Spec compliance for what is checked, type by type.
Known-wrong answers
Integer arithmetic overflow wraps where SQL raises. The edges fixture carries int64 max/min against the day a checked mode exists; the machinery is already in cast, which takes safe as a comptime parameter.
Division by zero is no longer one of these: floats answer inf, -inf and nan, and pc.divide raises on an integer zero divisor as pyarrow.compute.divide does.
One latent trap is worth knowing too: count(*) desugars to count(lit(1)), whose referenced-column set is empty, so anything that prunes by “what does this read” would prune every column. Column pruning handles it by never narrowing a source to zero columns.
What the corpus measures as missing
51 of the 286 cases carry -- skip mojo: marrow has no API for them, so the SQL and the DuckDB answer are recorded and the marrow body is a proposal rather than a verified spelling.
| cases | area |
|---|---|
| 12 | aggregates — median, quantile, first/last, arg_min/arg_max, string_agg, corr/covar, mode, skewness, bool_and/bool_or |
| 9 | temporal — date_diff, age, strftime/strptime, make_date, epoch seconds, month intervals, timestamp literals |
| 8 | nested — struct field, map lookup, list element/slice, unnest |
| 7 | math — greatest/least, rounding to digits, bitwise operators, gcd/lcm, signbit, trigonometry |
| 2 | string — concat_ws, null-skipping concat |
| 4 | joins — cross, non-equi, asof |
| 3 | GROUPING SETS / ROLLUP / CUBE |
| 3 | filters — SQL NOT IN null semantics, .is_in as a method |
| 2 | subqueries |
| 1 | DISTINCT ON |
golden/COVERAGE.md is the authoritative, machine-checked version of this table.
Capability by layer
| Status | |
|---|---|
| Layouts | bool, numeric, string/large_string, binary/large_binary, fixed_size_binary, list/large_list/fixed_size_list, struct, map, dictionary, decimal (32/64/128/256), temporal — implemented. Union, run-end-encoded and view layouts — not |
| Sources | A lazy scan reads one Parquet file, local or at an object-store URI through OpenDAL (whose C library is installed separately), or an in-memory batch, and one newline-delimited JSON file (marrow.read_json). Arrow IPC is read and written eagerly, NDJSON eagerly through marrow.json. No CSV, and no globs, directories or hive partitions |
| Parquet | native reader and writer, snappy/zstd/lz4, data page v1 & v2, statistics, page index. Bloom filters are implemented but off by default and unreachable from Python. No modular encryption |
| Optimizer | 16 rules plus column pruning. No cost model, so no join reordering and no build-side selection |
| Execution | in-memory only — no spilling anywhere, so every aggregate and join is bounded by RAM |
| Joins | equijoins only: inner, left, right, full, semi, anti. No nested-loop, range or asof joins |
| GPU | element-wise kernels only (arithmetic, comparison, boolean, cast, hashing), and off unless built with -D MARROW_GPU=true |
| UDFs | none |
Python-specific limits
The Python bindings are narrower than the Mojo core, and some of the gaps are CPython-binding limits rather than missing features:
- Most aggregates are not on the eager surface.
marrow.computehasanyandall, but nosum/min/max/mean; those reductions live in the query layer. - Row groups are skipped only in an optimized plan.
.optimize()pushes filters into the Parquet scan;collect()on a plan that was not optimized reads every row group. ==on anArrayis identity, not element-wise. Usemc.equalto compare values. Dtypes, fields and schemas do compare by value.- Some
pyarrow.computeoptions are missing.skip_nulls=False, multi-keysort_keys,nan_is_null=Trueandboundscheck=FalseraiseNotImplementedError, andmemory_poolis ignored.
Errors
There are 300+ raise Error sites and zero typed exceptions, so failures cannot yet be caught by kind.
The open work, with complexity estimates and what blocks what, is tracked in backlog.md at the repository root.