SpaceDataModel.jl

DOI version

A lightweight data model package for space/heliospheric science: datasets with selector domains and sources that materialize them over a time range.

Quick Start

using Pkg; Pkg.add("SpaceDataModel")
using SpaceDataModel

pattern = FilePattern("https://data.elfin.ucla.edu/{probe}/l2/{probe}_l2_epd_{t:yyyymmdd}_v{version}.cdf")
epd = Dataset("epd", Archive(pattern);  # a reader(paths, t0, t1) opens and trims, e.g. Archive(pattern, cdfopen)
    selectors = (; probe = ("ela", "elb")))
reg = Registry("EPD", [epd]; defaults = (; probe = "ela"))

filter(reg; probe = "elb")    # discovery path: keeps matching rows
ds = reg[probe = "elb"]       # select the only matching dataset, or an error listing the domains
keys(reg); reg["epd"]         # dataset names; one by name
getdata(ds, "2021-08-08", "2021-08-09")    # enumerate remote files, cache, open

Data Sources

getdata(x, t0, t1) is the one fetch verb; x is a SpaceDataModel.DataSource or a function with (t0, t1) -> data. AbstractDataset <: DataSource adds indexable variables: ds[var] isa DataSource.

A package adds a dataset type, or a DataSource for a standalone variable:

struct MyDataset <: SpaceDataModel.AbstractDataset; id::String; end
SpaceDataModel.getdata(ds::MyDataset, t0::DateTime, t1::DateTime; kw...) = ...    # all variables, indexable by name
SpaceDataModel.getdata(p::Product{MyDataset}, t0::DateTime, t1::DateTime; kw...) = ...  # optional: fetch `p.variable` directly
Base.keys(ds::MyDataset) = ...      # variable names
SpaceDataModel.getmeta(ds::MyDataset) = ...  # dataset attributes

getdata converts time strings ("2020-001", "2020-01-01 12:00") and Dates to DateTime before dispatching on a DataSource.

Contract: data covers [t0, t1) (half-open); getdata(ds, t0, t1)[v] equals getdata(ds[v], t0, t1). A range without data should return an empty variable (Archive throws instead: it has no file to build one from). Check a dataset against the contract in its tests with using Test; SpaceDataModel.Testing.test_dataset(ds, var, t0, t1; empty).

Dataset is the AbstractDataset for file archives: a name templated over selector domains, and a source such as Archive, see Quick Start.

SpaceDataModel.OverlayDict(base, overlay) layers writable metadata over (read-only) attributes (e.g. CDF file) without copying.

Metadata Schemas

Resolve semantic attributes (:name, :unit, :desc, …) against heterogeneous metadata formats (ISTP, HAPI, Madrigal).

using SpaceDataModel: ISTPSchema, get_schema

# data is anything carrying metadata (raw Dict, DimArray, DataVariable, …)
schema = ISTPSchema()
attrs = schema(data)
attrs[:unit]
attrs[:desc]

# Or auto-detect the schema from the metadata content
get_schema(data)

See docs/src/schema_guide.md for lookup patterns and extending to new formats.

Remote Files

Describe an archive's file names with a template, then resolve and download by time range.

using SpaceDataModel: FilePattern, remotefiles, localize

p = FilePattern("https://data.elfin.ucla.edu/{probe}/{level}/fgm/survey/{t:yyyy}/{probe}_{level}_fgs_{t:yyyymmdd}_v{version}.cdf")
fgs = p(; probe="ela", level="l1")              # fill keywords now or at the call
fgs("2020-10-01"; version="03")                 # one file name
urls = remotefiles(fgs, "2020-10-01", "2020-10-03")  # `version` defaults to "*": the latest listed
paths = localize(urls)                          # cached local copies

Placeholder syntax and cadence: docstring of FilePattern in src/filepattern.jl; fetching in src/remote.jl.

References