read_resource_batches

read_resource_batches(
    resource_properties: ResourceProperties,
    paths: list[Path] | None = None,
)

Read all batch resource file(s) into a list of DataFrames.

Use this function to read the Parquet file(s) specified in paths into a list of Polars DataFrames, and perform checks on each of the DataFrames against the resource_properties. The resource_properties object is used to check the data and ensure it is correct. While Sprout generally assumes that the files stored in the resources/<name>/batch/ folder are already correctly structured and tidy, this function still runs checks to ensure the data are correct by comparing to the properties.

Parameters

resource_properties : ResourceProperties

The ResourceProperties object that contains the properties of the resource you want to check the data against.

paths : list[Path] | None = None

A list of paths for all the Parquet files in the resource’s batch/ folder. Use PackagePath().resource_batch_files() to help provide the correct paths to the batch files. Defaults to the batch files of the given resource.

Returns

list[pl.DataFrame]

A list of DataFrame objects from all the batch files.

Raises

ValueError

If the batch file name is not in the expected pattern.

ValueError

If the timestamp column name matches an existing column in the DataFrame.