Skip to main content

Python interface

This document describes the design of the Python functions forming the CLI and the library layer, as well as tracking implementation status.

We use symbols to indicate the status of implementation (see table below). For planned or in-progress work, we might include signatures, docstrings, and pseudocode to clarify the design. Once the interface is implemented, these are replaced by links to the reference documentation.

A table showing the symbols used to indicate the status of interface components, along with their descriptions.
Status Description

Interface that has been implemented.

Interface that is currently being worked on.

Interface that is planned, but isn’t being worked on currently.

CLI

extract_metadata()

extract_metadata() has two arguments: parquet_path and output_path. The parquet_path argument is the path to a Parquet file in, e.g. the staging/ directory, and the output_path argument is the path to the Python script that will have the ResourceProperties() dataclass filled in with the extracted metadata.

flowchart
    parquet[/"Parquet data<br>[PARQUET-PATH]"/]
    read_parquet["read_parquet()"]
    output_opt[/"Output path<br>[--output]"/]
    output[/"Python script<br>[File]"/]
    data[("DataFrame<br>(Tidy)")]
    extract_metadata["extract_column_metadata()"]
    metadata[("ResourceProperties<br>(extracted)")]
    init_metadata["init_resource_metadata()"]

    parquet --> read_parquet
    read_parquet --> data
    data --> extract_metadata
    extract_metadata --> metadata -->|render<br>from<br>template| init_metadata
    output_opt --> write["write_file()"]
    init_metadata -->|string| write
    write --> output
Figure 1: Diagram showing the flow of objects and functions to extract a resource’s metadata from data.

init_metadata()

init_metadata() has two arguments: output_path and type. The output_path argument is the path to where the Python script will be created, which will contain an empty PackageProperties() or ResourceProperties() dataclass. The type argument is the type of metadata to create: package or resource.

flowchart
    output_path[/"Output path<br>[output_path]"/]
    type[/"Type of metadata<br>[type]"/]
    template[/"Template<br>[File]"/]
    init_metadata["init_*_metadata()"]
    write_file("write_file()")
    output[/"Python script<br>[File]"/]

    type -->|package<br>or resource| init_metadata
    template --> init_metadata -->|string| write_file
    output_path --> write_file --> output
Figure 2: Diagram showing the flow of objects and functions to compose the metadata in a Python script.

build-resources

The build-resources command converts sets of staged Parquet files into the final resources.

If any step errors, the build stops immediately without any cleanup. The displaying of errors is handled by run_without_tracebacks() from Soil. It is the responsibility of each internal function to include enough information in the error message to identify the point of failure (e.g., which resource failed at which step).

The following classes are used to represent the resources throughout the build pipeline:

@dataclass(frozen=True)
class StagingFile:
    """The data from one staged file, with the timestamp taken from its file name."""

    path: Path
    timestamp: datetime
    data: pl.LazyFrame


@dataclass(frozen=True)
class BuildTarget:
    """The build details for one resource, including where to read and write it."""

    resource_metadata: ResourceProperties
    input_dir: Path
    output_path: Path


@dataclass(frozen=True)
class BuildData[T]:
    """A resource's build target together with its data at one stage of the build."""

    build_target: BuildTarget
    data: T
flowchart TD
    %% Config and metadata
    project_dir[/"project_dir"/]:::io --> read_config
    read_config["read_config()"] --> Config[/"Config"/]
    Config --> resolve_paths["resolve_paths()"] --> config_resolved[/"Config"/]
    project_dir --> resolve_paths
    config_resolved -->|metadata_file| read_metadata["read_metadata()"] --> metadata[/"PackageMetadata"/]
    metadata --> check_metadata["check_metadata()"] --> metadata_checked[/"PackageMetadata"/]

    %% Deleted IDs
    config_resolved -->|delete_obs_units_file| read_csv["pl.read_csv()"] --> deleted_ids[/"pl.DataFrame"/]
    metadata_checked --> check_deleted_ids["check_deleted_ids()"] --> deleted_ids_checked[/"pl.DataFrame"/]
    deleted_ids --> check_deleted_ids

    %% Resources: each step runs over the whole list
    config_resolved -->|resources| resource_configs[/"list[ResourceConfig]"/]
    resource_configs -->|map| get_build_target["get_build_target()"] --> build_targets[/"list[BuildTarget]"/]
    metadata_checked --> get_build_target

    build_targets -->|map| read_staged_resource["read_staged_resource()"] --> staged[/"list[BuildData[list[StagingFile]]]"/]
    staged -->|map| check_schema["check_schema()"]
    check_schema -->|map| join_staging_files["join_staging_files()"]
    join_staging_files --> joined[/"list[BuildData[pl.LazyFrame]]"/]

    joined -->|map| drop_deleted_ids["drop_deleted_ids()"] --> joined_after_delete[/"list[BuildData[pl.LazyFrame]]"/]
    deleted_ids_checked -->|"if any"| drop_deleted_ids
    joined_after_delete -->|map| collect["collect()<br><i>errors on lazy frames surface here</i>"]

    collect --> collected[/"list[BuildData[pl.DataFrame]]"/]
    collected -->|map| check_schema_joined["check_schema()"]
    check_schema_joined -->|map| check_constraints["check_constraints()"]
    check_constraints -->|map| write_parquet["write_parquet()"]
    write_parquet --> outputs[/"output Parquet files"/]:::io

    classDef io fill:#FFF
Figure 3: The internal flow of functions and objects in the build-resources CLI command.

Library

Metadata classes

These classes contain an explicit, structured set of official metadata defined for a data package. The main purpose of these classes is to give programmatic access to the structured metadata. That way the metadata can be generated from writing Python code that, if it’s possible, can processes some input source of metadata into Sprout’s metadata structure. The classes also make it easier to get an overview or learn about what fields are part of the metadata in Sprout (e.g. optional vs required fields) and what will be eventually generated into the final data package.

See the help documentation with help(Package()) for more details on the metadata classes.

Class architecture

The class architecture follows the structure of the TOML metadata file with two differences:

  • Columns with different data types are represented by different classes, so that each class only defines the constraints applicable to its data type.

  • Package does not have an extend attribute because extensions are resolved when the TOML metadata file is read.

The metadata classes exposed in the library interface are sketched out below. The aim is to show the attributes and their types for each class, rather than provide a full implementation plan. All classes are implemented in Rust and exposed in the Python library through PyO3 bindings.

class DataType(StrEnum):
    integer = "integer"
    string = "string"
    boolean = "boolean"
    float = "float"
    date = "date"
    datetime = "datetime"
    time = "time"
    duration = "duration"

class IntegerColumn:
    name: NonEmptyString
    title: NonEmptyString
    description: NonEmptyString
    required: bool = False
    unique: bool = False
    enum: Optional[NonEmptySet[int]] = None
    minimum: Optional[int] = None
    maximum: Optional[int] = None

class DurationColumn:
    # In microseconds
    ...
    # Same fields as IntegerColumn

class FloatColumn:
    name: NonEmptyString
    title: NonEmptyString
    description: NonEmptyString
    required: bool = False
    unique: bool = False
    minimum: Optional[float] = None
    maximum: Optional[float] = None

class BooleanColumn:
    name: NonEmptyString
    title: NonEmptyString
    description: NonEmptyString
    required: bool = False
    unique: bool = False

class StringColumn:
    name: NonEmptyString
    title: NonEmptyString
    description: NonEmptyString
    required: bool = False
    unique: bool = False
    enum: Optional[NonEmptySet[str]] = None
    pattern: Optional[str] = None
    min_length: Optional[int] = None
    max_length: Optional[int] = None

class DateColumn:
    name: NonEmptyString
    title: NonEmptyString
    description: NonEmptyString
    required: bool = False
    unique: bool = False
    enum: Optional[NonEmptySet[date]] = None
    # In a specified format language
    format: Optional[NonEmptyString] = None
    minimum: Optional[date] = None
    maximum: Optional[date] = None

class DatetimeColumn:
    ...
    # Same fields as DateColumn, but using datetime instead of date

class TimeColumn:
    ...
    # Same fields as DateColumn, but using time instead of date

class ForeignKey:
    target_resource: NonEmptyString
    columns: NonEmptyList[NonEmptyString]

class Resource:
    name: NonEmptyString
    title: NonEmptyString
    description: NonEmptyString
    primary_key: NonEmptyList[NonEmptyString]
    foreign_keys: Optional[NonEmptyList[ForeignKey]] = None
    columns: NonEmptyList[any column type]

class Contributor:
    name: NonEmptyString
    email: NonEmptyString
    orcid: Optional[NonEmptyString] = None
    organisation: Optional[NonEmptyString] = None
    roles: Optional[NonEmptyList[NonEmptyString]] = None

class License:
    name: NonEmptyString
    file: Optional[Path] = None

class Package:
    name: NonEmptyString
    title: NonEmptyString
    description: NonEmptyString
    version: NonEmptyString = "0.1.0"
    homepage: Optional[NonEmptyString] = None
    repository: Optional[NonEmptyString] = None
    readme: Optional[Path] = None
    keywords: Optional[NonEmptyList[NonEmptyString]] = None
    observation_key: NonEmptyList[NonEmptyString]
    contributors: NonEmptyList[Contributor]
    licenses: NonEmptyList[License]
    resources: NonEmptyList[Resource]

Reading TOML files into metadata objects

When the TOML metadata file is read, the resource, license, and contributor extensions under [extend] are resolved into their corresponding metadata objects as follows:

  1. The metadata file at the given path is parsed in Rust.
  2. The extend property, if it exists, is checked for correctness.
  3. Each path in extend.resources, extend.licenses, and extend.contributors is resolved relative to the directory containing the metadata file.
  4. Each extension file is read into a Resource, License, or Contributor object.
  5. These objects are combined with the objects defined inline in the original metadata file.