Pandera

Pandera 0.33: a CLI for validation without writing Python, and native PyArrow table support

No items found.

Every Pandera feature until now has assumed one thing: you're writing Python code. That's a fine assumption when validation lives inside a pipeline, but a lot of validation doesn't. It lives in a CI job that gates a data drop, a shell script that sanity-checks a vendor file, a Makefile target someone runs before a demo. For those, "import pandera and write a script" is real friction for what should be a one-liner.

Pandera 0.33 removes that assumption. Here are the headline features of v0.33.*:

  • A `pandera` command-line interface: validate, infer, and generate data against serialized schemas with no Python code
  • First-class validation of `pyarrow.Table` objects

Both of these are built on the unified Narwhals backend that shipped in 0.32: once validation logic is written once against a portable expression layer, pointing new frontends at it gets a lot easier.

The Pandera CLI: validation as a shell command

Install it with the `cli` extra (it's built on Typer):

pip install 'pandera[cli,io]'

You'll also want the `pandera[io]` extra for YAML schema support, plus whichever dataframe library your schema targets. The CLI works with Pandera's serialized schema format — the same YAML/JSON you get from `schema.to_yaml()` — and exposes three commands.

pandera validate

The workhorse. Point it at a schema and a data file:

pandera validate --schema schema.yaml --data data.csv

On success you get a summary of the schema- and data-level checks that ran (rendered as Rich tables if you have Rich installed) and a zero exit code. On failure you get a breakdown of which checks passed and failed, with details — and a non-zero exit code. That last part is the point: it makes Pandera a drop-in CI gate.

Copied to clipboard!
# .github/workflows/data-checks.yml (excerpt)
- name: Validate incoming data
  run: pandera validate -s schemas/transactions.yaml -d data/transactions.parquet

No wrapper script, no `python -c` incantation. The job fails when the data does.

Serialized schemas can carry a top-level api field (pandas, polars, ibis, pyspark.sql, and friends) that tells the CLI which dataframe API to validate with; it defaults to pandas when absent. You can also override it at the command line:

Copied to clipboard!
pandera validate -s schema.yaml -d data.csv --backend polars
pandera validate -s schema.yaml -d data.csv --backend narwhals

The `--backend narwhals` option runs validation through the unified Narwhals engine, and the report tells you which backend actually ran (Backend: `narwhals` vs. Backend: `pandas`), so there's no guessing about which code path checked your data.

pandera infer

Bootstrapping a schema from scratch is the tedious part of adopting any validation library. `infer` reads a data file and writes a schema — as YAML, JSON, or a runnable Python module:

Copied to clipboard!
# Infer a YAML schema from a CSV
pandera infer -d data.csv -o schema.yaml

# Infer from Parquet using the Polars backend
pandera infer --data table.parquet --output schema.json --backend polars

# Emit a Python DataFrameModel class instead
pandera infer -d data.csv -o model.py --format py --script-type model

That last form is a nice on-ramp: run it once against representative data, get a `DataFrameModel` class you can check into your repo, then tighten the generated constraints by hand. Inference is a starting point, not a finished schema — but it beats transcribing forty column names.

pandera generate

The inverse of validate: produce synthetic data from a schema, using Pandera's hypothesis-backed strategies (requires `pandera[strategies]`):

Copied to clipboard!
pandera generate -s schema.yaml
    -o sample.csv/pandera generate
    --schema ds_schema.json
    --output data.nc
    --size 5

This currently supports pandas DataFrame schemas and xarray DataArray/Dataset schemas, and it's handy anywhere you need schema-conformant fixtures — seeding a dev environment, generating test data in CI, or handing a downstream team example files before the real data exists.

The three commands compose into a workflow that never touches a Python file: `infer` a schema from known-good data, commit it, `validate` every future drop against it in CI, and `generate` fixtures for tests along the way.

Native PyArrow table validation

Arrow is the interchange format of the modern data stack — it's what crosses the boundary between DuckDB and pandas, between Polars and your feature store, between a Flight server and its clients. A pyarrow.Table already carries its own schema, but that schema tells you types, not truths: it knows price is an int64, not that a price should be positive. Pandera 0.33.0 closes that gap by validating pyarrow.Table objects directly:

pip install 'pandera[pyarrow]'

The API is the same one you already know, imported from `pandera.pyarrow`:

Copied to clipboard!
import pyarrow
import pandera.pyarrow as pa

schema = pa.DataFrameSchema(
    {
        "state": pa.Column(str),
        "city": pa.Column(str),
        "price": pa.Column(int, pa.Check.in_range(5, 20)),
    }
)

table = pyarrow.table({
    "state": ["FL", "FL", "CA", "CA"],
    "city": ["Orlando", "Miami", "Los Angeles", "San Francisco"],
    "price": [8, 12, 10, 16],
})

schema.validate(table)  # returns the pyarrow.Table

`validate()` returns the `pyarrow.Table` itself, so it slots into a pipeline without any conversion. Class-based `DataFrameModel`s and the `@pa.check_types` decorator work too:

Copied to clipboard!
from pandera.typing.pyarrow import Table

class Schema(pa.DataFrameModel):
    state: str
    city: str
    price: int = pa.Field(in_range={"min_value": 5, "max_value": 20})

@pa.check_types
def transform(df: Table[Schema]) -> Table[Schema]:
    return df

Dtypes can be declared as native PyArrow types (`pyarrow.int64()`), Python builtins (`int`), or string aliases (`"int64"`) — they all resolve to the same thing — and parametrized types like `pyarrow.timestamp("us")`, `pyarrow.decimal128(10, 2)`, and `pyarrow.list_(pyarrow.int32()`) are supported. Custom checks get a PyArrowData container holding the native table and the column key, so you can drop down to pyarrow.compute:

Copied to clipboard!
import pyarrow.compute as pc

schema = pa.DataFrameSchema(
    {"price": pa.Column(int, pa.Check(lambda data: pc.greater(data.table[data.key], 0)))}
)

Under the hood, PyArrow validation runs on the Narwhals backend, which installs automatically with the pyarrow extra. Because an Arrow table is fully materialized (unlike a lazy Polars or Ibis frame), both schema-level and data-level checks run by default; set `PANDERA_VALIDATION_DEPTH=SCHEMA_ONLY` if you only want the cheap structural checks.

There are two limitations to be addressed in the future: `coerce=True` is not yet implemented for PyArrow (you get a `SchemaWarning` and a `WRONG_DATATYPE` error instead of a silent cast), and data synthesis strategies aren't available yet — same as the Polars and Ibis backends.

Conclusion

Pandera 0.33 pushes validation to two new surfaces: the shell, where pandera validate turns any serialized schema into a CI gate with an exit code, and Arrow, where the same `DataFrameSchema` and `DataFrameModel` APIs now run directly against `pyarrow.Table` — the format your data is probably already in when it crosses a system boundary. Both are dividends of the unified Narwhals backend: write the validation engine once, and new frontends get cheap.

Install it with `pip install -U 'pandera[cli,pyarrow]'`, and see the CLI guide and PyArrow guide for the full reference. If you hit an edge, open an issue.

Try the devbox

A free, local sandbox to explore the Union.ai platform.

Chat with an engineer
No items found.