All data submissions to EBAS need to be in the EBAS Nasa Ames fileformat. EBAS Nasa Ames is a FFI 1001 Nasa Ames format with several extensions specific to EBAS (additional standardized metadata).
In addition to the file format definition, EBAS provides templates for all measurement types. The file format and all specific templates are documented in the EBAS Data Submission Manual.
Regular Data Submision
Regular data submissions are usual quality assured annual files with data from one instrument. In all cases, we expect at least data level 2 (final, manually quality assured data, usually at hourly resolution). Some frameworks require additionally the submission of lower data levels (0 and 1) in separate files.
All regular data submissions should be uploaded using the EBAS Data Submission Tool. This tool checks the file format, metadata consistency and value ranges. For data submitters which provide data on a network level (many instruments) we provide the possibility for direct upload (please contact EBAS support if needed).
Near Realtime Data Submission
Near realtime data er usually submitted from the observatory directly to the EBAS data center. Usually, the instruments send one file per hour (data level 0). See the EBAS Data Submission Manual for the level 0 file templates.
Technically, the files are transmitted using sftp and public key authentication is supported.
Setting up the NRT dataflow will need direct interaction between the data submitter and the EBAS datacenter. Please contact EBAS support if you want to set up such a dataflow.
Tools for generating EBAS Nasa Ames files
The least sophisticated way to generate an EBAS Nasa Ames file for submitting data to EBAS is to open a an example file in a text editor and change it to your needs. This example file can either be a similar file downloaded from EBAS (or last year’s submission) or the official template file from the EBAS Data Submission Manual).
You will have to
- change all metadata in the file header
- add the time series (sample start and end times as well as measurement data)
There are some challenges using this approach:
- The file header contains some inter-dependencies which can easily become inconsistent when edited manually
- Encoding sample start and end times as well as measurement data (especially missing values) is not easily done manually
However, this method can still be the easiest when submitting simple datasets for a very limited amount of instruments).
EBAS File Generation Tool
The EBAS File Generation Tool is an online web based tool to generate a file for submission to EBAS.
EBAS I/O Python Module
Especially for setting up an operational environment for either submitting data for many instruments in parallel or submitting data very often (e.g. near realtime), it is crucial to generate EBAS Nasa Ames files automatically by using dedicated software.
ebas-io is a python module for reading and writing EBAS Nasa Ames files and can help setting up such data production software.
The ebas-io repository provides the python module for download. The repository includes documentation and some example scripts for generating files for different instrument types.
The idea is that data providers can easily produce software to upload their data operational.