Outputs
Sprout is one of several tools used to develop and build data packages. This document describes the outputs Sprout generates during this process.
In brief, a data package consists of one or more files (called data resources) that contain data specific to a given set of conditions or research questions. For instance, a data package could be a research study that collects information from people older than 50 who live in Denmark and who have diabetes. In this example data package, there might be data collected on basic demographics like gender, ethnicity, or education, or blood sample data like blood glucose, cholesterol, or blood pressure. Here, the basic demographics could be one data resource, while the blood sample data could be another data resource within the data package.
For more details about data packages and what they mean, check out our guide.
Data packages are called many things in other fields. For instance, a data package could be called “cohort study data”, a “data resource”, a “clinical trial dataset”, a “dataset”, a “data product”, a “study dataset”, simply “data”, or any other combination of words that include the word “data”.
Data resources
Sprout builds staged and tidy data into the final Parquet data resources, by default following these patterns:
resources/<name>.parquetfor resources with only one tabular data file.resources/<name>/part-*.parquetfor resources that benefit from being in a Parquet partition format.resources/<name>/data.parquetwithresources/<name>/<single-file>.<extension>for resources where data for a single subject is also a single file, such as an image or audio file. For example,resources/liver-scan/id-101.dcmfor a DCM image of a liver for participant 101.
What data is included in a resource must be defined by a domain expert. In general, a data resource encompasses data collected in a specific way and for a specific set of measurements from a specific set of input sources for a specific purpose. The data resource(s) will always be within one specific package, with each data resource in its own folder having a unique resource name <name>, e.g. resources/blood-samples.parquet.
For example, a data package may have two data resources, one for demographics and another for blood samples. Why are these data divided into these specific data resources? Because they represent two distinct approaches to collecting and interpreting the data. While they are both on the same group of people, they collect the data in different ways. Demographic data is usually collected with a survey by a researcher or technician (or the participants themselves if online) in a relatively consistent way using some survey software. Meanwhile, blood sample data is usually taken at a specific time and analyzed by a specific machine by a specific laboratory in a specific way that is output in a specific format. The two types of data are collected in different ways and should be recognized as separate entities and thus separate data resources, even though they contain data about the same group of people.
Metadata
The current metadata format output by Sprout is the datapackage.json from the Data Package Standard. This machine-readable JSON file is the (current) foundation for what makes a set of files and folders a data package. This file contains metadata on the package (like title, description, and contributors) as well as metadata on the data resources (like column names, data types, descriptions, and paths to the Parquet file(s) containing the resource data).