read_resource_batches

Warning

🚧 Sprout is still in active development and evolving quickly, so the documentation and functionality may not work as described and could undergo substantial changes 🚧

read_resource_batches(
    resource_properties: ResourceProperties,
    paths: list[Path] | None = None,
)

Reads all the batch resource file(s) into a list of (Polars) DataFrames.

This function takes the Parquet file(s) given by paths, reads them in as Polars DataFrames as a list and does some checks on each of the DataFrames in the list based on the resource_properties. The resource_properties object is used to check the data and ensure it is correct. While Sprout generally assumes that the files stored in the resources/<id>/batch/ folder are already correctly structured and tidy, this function still runs checks to ensure the data are correct by comparing to the properties.

Parameters

resource_properties : ResourceProperties

The ResourceProperties object that contains the properties of the resource you want to check the data against.

paths : list[Path] | None = None

A list of paths for all the files in the resource’s batch/ folder. Use path_resource_batch_files() to help provide the correct paths to the batch files. Defaults to the batch files of the given resource.

Returns

list[pl.DataFrame]

Outputs a list of DataFrame objects from all the batch files.

Raises

FileNotFoundError

If a file in the list of paths doesn’t exist.

ValueError

If the batch file name is not in the expected pattern.

ValueError

If the timestamp column name matches an existing column in the DataFrame.

Examples

import seedcase_sprout as sp

with sp.ExamplePackage():
    resource_properties = sp.example_resource_properties()
    sp.write_resource_batch(sp.example_data(), resource_properties)

    sp.read_resource_batches(resource_properties)

PosixPath('/tmp/tmpddw4hod0/example-package/resources/example-resource/batch/2025-05-30T105346Z-4821eb5b-2be5-4501-b92a-8d738b27ff93.parquet')

[shape: (3, 4)
 ┌─────┬──────────┬──────────┬────────────────────────┐
 │ id  ┆ name     ┆ value    ┆ _batch_file_timestamp_ │
 │ --- ┆ ---      ┆ ---      ┆ ---                    │
 │ i64 ┆ str      ┆ f64      ┆ str                    │
 ╞═════╪══════════╪══════════╪════════════════════════╡
 │ 34  ┆ Helly R  ┆ 123.123  ┆ 2025-05-30T105346Z     │
 │ 99  ┆ Mark S   ┆ 9988.0   ┆ 2025-05-30T105346Z     │
 │ 100 ┆ Ms Casey ┆ -76.0009 ┆ 2025-05-30T105346Z     │
 └─────┴──────────┴──────────┴────────────────────────┘]