Inputs

This page documents the inputs to Sprout’s CLI and Python library. The inputs to the library are mainly to fill out Metadata classes within Python scripts as part of the data package build pipeline.

In general, the data and metadata inputs come from files in a “staging” directory. For more detail on what is kept within a staging directory and why to use this structure, check out our Building Data Packages guide. Briefly, this directory contains data for data resources that are tidied and processed from the raw data as well as metadata that has been edited (manually or programmatically) to accurately reflect the data. The files in staging are bulk collections of data that are processed together. This staging/ directory is where the data is stored before it is processed and saved in the resources/ directory. Using the staging directory makes the history between raw data and the final data resource more transparent and reproducible. It can also help with troubleshooting issues that may arise during data entry or processing or for auditing purposes.

Configuration file

The paths to all the important files, like the data in staging, are kept within a .config/sprout.toml file. Other locations include sprout.toml, .sprout.toml, or kept within the pyproject.toml file. Most of the below input go through this configuration file. Details about what exactly is contained within this file are described in the sections for each individual CLI command in the CLI interface design document.

Metadata

This is either the Python classes or the TOML files that contain the TOML representation of the Python classes. The TOML files are used to manually edit or add information on the metadata, while the Python classes are used when programmatic processing of metadata is possible.

Data

The data input are Parquet files kept within the staging directory. These files are created when the batch raw data has been tidied and processed in preparation for being added to a data resource. These files are usually (but not required to be) kept in a path with a format that looks like:

staging/<data-source>/<resource-name>/<timestamp>.parquet

Sprout reads these files into Polars DataFrame objects to continue the build process into the final data package structure. The input data should be in a tidy format, though Sprout doesn’t have any direct way of enforcing this. Tidy data is a conceptual framework around how data should be structured to be optimally usable for analysis. This is similar to what is called third normal form in relational database development, excluding the emphasis on creating additional tables to reduce redundancy.

Tidy data has the following properties:

  1. Each variable forms a column.
  2. Each observation forms a row.
  3. Each cell has a single value that represents both the column and row entity.

This tidy structure makes it easier to process and use the data for later analysis. If it is already in this form, there is less need to transform it later.

Observational units (for deletion)

An observational unit is the level of detail on the entity (e.g. human, animal, event) that the data was collected on at a given point in time for a specific analysis. This is also called the statistical unit. Many countries, most prominently in the EU with the GDPR, have some legal regulations or framework around the “right to be forgotten”. This legal regulation requires that if a person has had their data collected, they can request that any personally identifiable and sensitive data of theirs is deleted. In order to fulfill this legal requirement, the data must at a minimum contain some identifiers to that person to be able to delete the correct data.

Within the configuration file (e.g. .config/sprout.toml) is a field to give a path to a CSV that has a column or columns of the identifiers that link to people who want their data deleted. So if anyone requests their data to be deleted, the owner or other user would need to include the identifier(s) in this file so that during the build process Sprout can remove that person’s data.

Note

An example of a more granular observational unit might be a person in a research study who came to the clinic at a specific time in May 2024 to have their blood collected and filled out a survey. This person may only want the data from that May 2024 visit to be removed.

A “coarser” observational unit might be a person who participated in a research study and had data collected for many different measurements at multiple points. This person may want all their data removed, from all periods of time and all types of data.

Technically, it is near-impossible to completely remove all traces of data on a specific observational unit (e.g. a person). That’s because data can be saved within any backups done by the data owner or automatically in the server, or when stored within source management systems (e.g. like Git). The only legal requirement is that the data that is requested to “be forgotten” is no longer used for any other purposes aside from what it has already been used for. As long as that person’s data is no longer used for further distribution and use, the law would be upheld.