PyArrow interop

The same memory, exchanged without copying

Marrow and PyArrow implement the same Apache Arrow columnar format. This page covers how they relate and how to exchange data between them — in Python and in Mojo — over the Arrow C Data Interface.

Same data model

Both libraries represent the same Arrow memory layout. An int64 array in Marrow and in PyArrow are bitwise identical in memory.

import marrow as ma
import pyarrow as pa

# Marrow
ma_arr = ma.array([1, 2, 3, None, 5])
print("marrow:", ma_arr)
print("type:  ", ma_arr.type)

# PyArrow
pa_arr = pa.array([1, 2, 3, None, 5])
print("pyarrow:", pa_arr)
print("type:   ", pa_arr.type)

Performance

NoteBenchmark numbers pending

The comparison harnesses are in the tree (python/marrow/tests/bench_*.py, covering marrow vs PyArrow vs Polars vs DuckDB) but no results are recorded yet, so no speed claim is published here rather than an unverified one.

Reproduce them yourself:

pixi run -e bench bench-engines

Two performance numbers are measured and CI-gated: binary size — see Architecture — and the comptime lane’s speed against the runtime lane — see Comptime vs runtime lane.

Zero-copy C Data Interface

Marrow arrays implement the Arrow C Data Interface protocol (__arrow_c_array__ / __arrow_c_schema__), so they exchange data with PyArrow with zero copies at both the Python and Mojo level.

Python level

import sys
sys.path.insert(0, "../../python")   # the built extension
import marrow as ma
import pyarrow as pa

# PyArrow -> marrow, via the __arrow_c_array__ protocol
pa_arr = pa.array([1, 2, 3, None, 5])
ma_arr = ma.array(pa_arr)
print(ma_arr)

# marrow -> PyArrow, the same way back
round_trip = pa.array(ma_arr)
print(round_trip, "|", round_trip.type)
PrimitiveArray[int64]([1, 2, 3, NULL, 5])
[
  1,
  2,
  3,
  null,
  5
] | int64

Neither direction copies the values — what crosses is a pointer, a length and a release callback.

Mojo level

At the Mojo level the underlying CArrowSchema and CArrowArray structs are directly accessible for low-level interop.

from std.python import Python
from marrow.c_data import CArrowArray, CArrowSchema

var pa = Python.import_module("pyarrow")
var pyarr = pa.array([1, 2, 3, 4, 5], mask=[False, False, False, False, True])

# Import from PyArrow via the Arrow C Data Interface capsule protocol
var capsules = pyarr.__arrow_c_array__()
var dtype = CArrowSchema.from_pycapsule(capsules[0]).to_dtype()  # int64
var data  = CArrowArray.from_pycapsule(capsules[1])^.to_array(dtype)
ref typed = data.as_int64()

print(typed[0].is_valid())   # True
print(typed[4].is_valid())   # False — null
print(typed[0].value())      # 1

# Export from Mojo back to PyArrow — __arrow_c_array__ protocol is supported
var pa_result = pa.array(data.copy())
Back to top