GitHub

CommonDataFormat.jl

DOI version

A Julia package for reading Common Data Format (CDF) files, widely used in space physics for storing multidimensional data arrays and metadata. See CDFDatasets.jl for a high-level interface.

Installation

using Pkg
Pkg.add("CommonDataFormat")

Quick Start

using CommonDataFormat

# Load a CDF file
omni_file = joinpath(pkgdir(CommonDataFormat), "data/omni_coho1hr_merged_mag_plasma_20240901_v01.cdf")
ds = CDFDataset(omni_file)
CDFDataset{CommonDataFormat.NoCompression, Int64}
path: /home/runner/work/CommonDataFormat.jl/CommonDataFormat.jl/data/omni_coho1hr_merged_mag_plasma_20240901_v01.cdf
variables:
  Epoch : (720,) support_data CDF_EPOCH [2024-09-01T00:00:00 … 2024-09-30T23:00:00]
  heliographicLatitude : (720,) data CDF_REAL4 [7.2 … 6.8]
  heliographicLongitude : (720,) data CDF_REAL4 [261.6 … 291.0]
  BR : (720,) data CDF_REAL4 [6.7 … -2.6]
  BT : (720,) data CDF_REAL4 [-8.2 … 2.1]
  BN : (720,) data CDF_REAL4 [-3.6 … 3.5]
  ABS_B : (720,) data CDF_REAL4 [11.4 … 6.7]
  V : (720,) data CDF_REAL4 [377.0 … 424.0]
  elevAngle : (720,) data CDF_REAL4 [-2.5 … 5.6]
  azimuthAngle : (720,) data CDF_REAL4 [1.7 … -1.9]
  N : (720,) data CDF_REAL4 [4.5 … 3.6]
  T : (720,) data CDF_REAL4 [82095.0 … 107645.0]
CDR (CDF Descriptor Record):
  Version: 3.9.0
  Encoding: 1
  Flags: 0x00000003
    - Majority: Row
    - Single File Format: true
    - Checksum Used: false
    - MD5 Checksum: false
  Identifier: -1

attributes: Dict{String, Vector} with 28 entries:
  "Rules_of_use"            => ["Public"]
  "TITLE"                   => ["Near-Earth Heliosphere Data (OMNI)"]
  "Instrument_type"         => ["Plasma and Solar Wind", "Magnetic Fields (spac…
  "Project"                 => ["SPDF"]
  "alt_logical_source"      => ["Combined_OMNI_1AU-MagneticField-Plasma-Particl…
  "Discipline"              => ["Space Physics>Interplanetary Studies"]
  "Time_resolution"         => ["1 hour"]
  "Logical_file_id"         => ["omni_coho1hr_merged_mag_plasma_00000000_v01"]
  "LINK_TEXT"               => ["COHO dataset", "Additional analysis tools for …
  "PI_name"                 => ["J.H. King, N. Papatashvilli"]
  "spase_DatasetResourceID" => ["spase://NASA/NumericalData/OMNI/COHO/MergedMag…
  "TEXT"                    => ["Hourly averaged definitive multispacecraft int…
  "Data_type"               => ["COHO1HR>Definitive Hourly data from cohoweb"]
  "PI_affiliation"          => ["AdnetSystems, NASA GSFC"]
  "LINK_TITLE"              => ["Documentation", "COHOWeb service"]
  "Data_version"            => ["1"]
  "Acknowledgement"         => ["NSSDC"]
  "Source_name"             => ["OMNI (1AU IP Data)>Merged 1 Hour Interplantary…
  "Generation_date"         => ["Ongoing"]
  ⋮                         => ⋮

API Reference

CommonDataFormat.ADR — Type

Attribute Descriptor Record (ADR)

Contains a description of an attribute in a CDF. There will be one ADR per attribute. The ADRhead field of the ADR contains the file offset of the first ADR.

source
CommonDataFormat.ADR — Method
ADR(buf, offset, RecordSizeType)

Load an Attribute Descriptor Record from the buffer at the specified position.

source
CommonDataFormat.AEDR — Type
AEDR

Attribute g/r Entry Descriptor Record. Describes a global entry (for global attributes) or rVariable entry (for variable attributes).

source
CommonDataFormat.CDR — Type

CDF Descriptor Record (CDR) - the main file header record Contains version, encoding, format information, and pointer to GDR

source
CommonDataFormat.CDR — Method
CDR(buffer, pos, FieldSizeT)

Load a CDF Descriptor Record from the IO stream at the specified offset. This follows the CDF specification for CDR record structure.

source
CommonDataFormat.Epoch16 — Type
Epoch16

Picoseconds since Year 0 (01-Jan-0000 00:00:00.000.000.000.000) Represented as two doubles (seconds and picoseconds).

source
CommonDataFormat.GDR — Type

Global Descriptor Record (GDR) - contains global information about the CDF file Points to variable and attribute descriptor records

source
CommonDataFormat.GDR — Method
GDR(buffer::Vector{UInt8}, pos, FieldSizeT)

Load a Global Descriptor Record from the buffer at the specified offset.

source
CommonDataFormat.TT2000 — Type
TT2000

Nanoseconds since J2000 (01-Jan-2000 12:00:00.000.000.000) with leap seconds, represented as an 8-byte integer.

source
CommonDataFormat.VDR — Method
VDR(io::IO, FieldSizeT)

Load a Variable Descriptor Record from the IO stream at the specified offset.

source
CommonDataFormat.VXR — Method
VXR(source, offset, RecordSizeType)

Load a Variable Index Record from the source at the specified offset.

source
CommonDataFormat.rVDR — Method
rVDR(io::IO, FieldSizeT)

Load a Variable Descriptor Record from the IO stream at the specified offset.

source
CommonDataFormat.attrib — Method
attrib(cdf::CDFDataset, attribute_name::String)

Retrieve all entries for a named attribute from the CDF file.

source
CommonDataFormat.decode_cdr_flags — Method
decode_cdr_flags(flags::UInt32)

Decode the CDR flags field into individual boolean flags.

CDF Flags (from CDF specification):

  • Bit 0: Majority (1=row-major, 0=column-major)
  • Bit 1: File format (1=single-file, 0=multi-file)
  • Bit 2: Checksum used (1=checksum present, 0=no checksum)
  • Bit 3: MD5 checksum method (requires bit 2=1)
source
CommonDataFormat.majority_swap! — Method
majority_swap!(data, dims_without_record)

Convert row-major data layout to column-major (Julia's native layout) in-place. For row-major CDF files, this reverses the dimension order (except record dimension).

source
CommonDataFormat.@read_be_fields — Macro
@read_be_fields buffer pos T1 T2 ...

Unrolls sequential big-endian reads starting at pos within buffer. Returns a tuple of the parsed values and the updated position, mirroring read_be_i but without the runtime ntuple/offset bookkeeping.

Example

values, next = @read_be_fields buf pos UInt32 Int16
source