Ti trovi in : ArpaWeb / Idro-Meteo-Clima / Ricerca guidata / Documenti /

Documenti

Arkimet

sim_software

Introduction

Arkimet is a set of tools to archive, dispatch and distribute data files.

Features

General

  • All arkimet functionality besides metadata extraction and dataset recovery is file format agnostic
  • Support for new metadata types, match functions and file formats can be added with plugins
  • Data is treated like an opaque, read only binary string, never modified to guarantee integrity
  • Archive data files are only accessed using append operations, to avoid the risk of accidentally corrupting existing data
  • Metadata is archived in a compact, binary representation to save space, but it can be handled in easy to read, easy to parse, standard formats.

Metadata

  • The extraction of metadata is flexible and customisable, and for generic formats like GRIB or NetCDF it can be scripted using the simple and well known LUA scripting language
  • Metadata is stored in a compact binary representation to save space, but can be converted to standard formats when required
  • Metadata can have timestamped annotations to track its workflow
  • Metadata can be summarised to allow to know what data can be found in a big dataset without scanning its contents.

Remote access

  • Remote data access is provided through arki-server, an HTTP server application
  • arki-server can serve data from local datasets, as well as from remote datasets served by other arki-server instances (this allows, for example, to provide a single arki-server external front-end to various internal arki-servers in an organisation)
  • arki-server can be run behind apache mod-proxy to provide encrypted (SSL) or authenticated access
  • Client data access is done using libCURL, and can access the server over SSL or HTTP proxies
  • Summary of query results can be obtained quickly before transfering the result data
  • Postprocessing chains can be provided by the server to transfer only the postprocessed data (e.g. an average value instead of a big grid)

Archive

  • File layout can be customised depending on data volumes (one file per day, one file per month, etc.)
  • It is possible to select what metadata is indexed for faster queries
  • It is possible to know what metadata fields are used to identify a data item for the purpose of catching duplicate inserts
  • It is possible to insert, replace or remove items from a dataset
  • Datasets are self-contained, so it is possible to store the dataset in offline media, and query it right away as soon as the offline media is mounted online
  • Dataset summaries are small, can easily be shared across the Internet and can be used to keep track of offline datasets
  • Dataset summaries can be indexed in a second level index that implements complex data manipulations like spatial queries

User interfaces

  • A powerful and flexible suite of commandline tools allows to easily integrate arkimet into automated data processing chains.
  • arki-server not only allows remote access to the datasets, but it also provides a low-level, web-based query interface
  • ArkiMEOW is a web-based front-end to arkimet that provides simple and powerful browsing and data retrieval for end users.

Factsheet

The metadata currently supported are:

  • Reference time
  • Origin
  • Product
  • Level
  • Time Range
  • Area
  • Ensemble run information

The formats used to handle metadata are:

  • Arkimet-specific compact binary representation
  • Human-readable, easy to parse YAML representation http://www.yaml.org/
  • XML/RDF can be implemented as soon as a suitable standard is defined

The file formats that can be scanned are:

  • WMO GRIB (edition 1 and edition 2), any template can be supported via grib_api and LUA scripts
  • WMO BUFR (edition 3 and edition 4), via DB-All.e
  • NetCDF (to be implemented soon), any data layout can be suppoted via LUA scripts
  • More can easily be added via plugins

The match expressions currently available are:

  • Exact or partial match on any types of metadata
  • Advanced matching of reference times: 
    # Open or close intervals
reftime:>=2005
reftime:>=2005,<2007
# Exact years, month, days, hours, etc.
reftime:=2007
reftime:=2007-05
# Time extremes can vary in precision
reftime:>=2005-06-01 12:00
# Also interval of time of day can be specified
reftime:>=12:00,<18:00
# And repetitions during the day
reftime:>=2007-06/6h
reftime:=2007,>=12:00/30m,<18:00
  • Configurable aliases of subexpressions for easy recall via mnemonic names

Licensing and acquisition

Software dependencies

Examples

To extract metadata:

$ arki-scan --yaml --annotate file.grib


To create a summary of the content of many GRIB files:

$ arki-scan --summary *.grib > summary


To view the contents of the generated summary file:

$ arki-dump --yaml summary


To select GRIB messages from a file:

$ arki-scan file.grib | arki-grep --data ´origin:GRIB1,98;reftime:>=2008´ > file1.grib


To configure a dataset interactively:

$ arki-config datasets/name


To dispatch grib messages into datasets:

$ arki-mergeconf datasets/* > allowed-datasets
$ arki-scan --dispatch=allowed-datasets file.grib > dispatched.md


To query:

$ arki-query ´origin:GRIB1,98;reftime:>=2008´ datasets/* --data > output.grib


To get a preview of the results without performing the query:

$ arki-query ´origin:GRIB1,98;reftime:>=2008´ datasets/* --summary --yaml --annotate


To export some datasets for remote access:

$ arki-mergeconf datasets/* > datasets.conf
$ arki-server --port=8080 datasets.conf


To query a remote dataset:

$ arki-mergeconf http://host.name > datasets.conf
$ arki-query ´origin:GRIB1,98;reftime:>=2008´ datasets.conf --data

Free software with contribute of Arpa-SIM

Anno pubblicazione 2008

Materia: Meteo e clima


Tipologia: Tecnico/istituzionali



Allineamento Verticale