Python interface

This document describes the design of the Python functions forming the CLI and the library layer, as well as tracking implementation status.

We use symbols to indicate the status of implementation (see table below). For planned or in-progress work, we might include signatures, docstrings, and pseudocode to clarify the design. Once the interface is implemented, these are replaced by links to the reference documentation.

A table showing the symbols used to indicate the status of interface components, along with their descriptions.

Status	Description
	Interface that has been implemented.
	Interface that is currently being worked on.
	Interface that is planned, but isn’t being worked on currently.

CLI

extract_metadata()

extract_metadata() has two arguments: parquet_path and output_path. The parquet_path argument is the path to a Parquet file in, e.g. the staging/ directory, and the output_path argument is the path to the Python script that will have the ResourceProperties() dataclass filled in with the extracted metadata.

flowchart
    parquet[/"Parquet data<br>[PARQUET-PATH]"/]
    read_parquet["read_parquet()"]
    output_opt[/"Output path<br>[--output]"/]
    output[/"Python script<br>[File]"/]
    data[("DataFrame<br>(Tidy)")]
    extract_metadata["extract_column_metadata()"]
    metadata[("ResourceProperties<br>(extracted)")]
    init_metadata["init_resource_metadata()"]

    parquet --> read_parquet
    read_parquet --> data
    data --> extract_metadata
    extract_metadata --> metadata -->|render<br>from<br>template| init_metadata
    output_opt --> write["write_file()"]
    init_metadata -->|string| write
    write --> output

Figure 1: Diagram showing the flow of objects and functions to extract a resource’s metadata from data.

init_metadata()

init_metadata() has two arguments: output_path and type. The output_path argument is the path to where the Python script will be created, which will contain an empty PackageProperties() or ResourceProperties() dataclass. The type argument is the type of metadata to create: package or resource.

flowchart
    output_path[/"Output path<br>[output_path]"/]
    type[/"Type of metadata<br>[type]"/]
    template[/"Template<br>[File]"/]
    init_metadata["init_*_metadata()"]
    write_file("write_file()")
    output[/"Python script<br>[File]"/]

    type -->|package<br>or resource| init_metadata
    template --> init_metadata -->|string| write_file
    output_path --> write_file --> output

Figure 2: Diagram showing the flow of objects and functions to compose the metadata in a Python script.

Library

Metadata classes

These classes contain an explicit, structured set of official metadata defined for a data package. The main purpose of these classes is to give programmatic access to the structured metadata. That way the metadata can be generated from writing Python code that, if it’s possible, can processes some input source of metadata into Sprout’s metadata structure. The classes also make it easier to get an overview or learn about what fields are part of the metadata in Sprout (e.g. optional vs required fields) and what will be eventually generated into the final data package. We currently only have metadata classes for the Data Package spec.

See the help documentation with help(SproutProperties()) for more details on the metadata classes.