User Tools

Site Tools


open:wp4:dadiproductrepository

This is an old revision of the document!


ASTERICS DADI Product Repository

Introduction

The Data Access, Discovery and Interoperability (DADI) Work Package of the ASTERICS project maintains a repository of its “products”, with a first delivery at the project mid-term (April 2017, D4.8). The repository will then be regularly maintained with a final delivery at the end of the project. DADI has three strands of work:

  • Support to ESFRIs and pathfinders in their uptake of the Virtual Observatory framework, including workshops involving the wider European data provider community
  • Support to the scientific community in the usage of VO-enabled data and tools
  • Technological activities to update the VO framework to optimize it to include ESFRI and pathfinder data and taking into account feedback from the ESFRIs and pathfinders and science users

Content

DADI has several kind of “products”, including some which have been developed totally or partially with DADI support and some which are key components of the project with respect to the ESFRI and science needs. They can be broadly classified in several categories:

  • Scientific tutorials describing real use cases of VO-enabled data and tools, prepared and updated in particular in the framework of the annual DADI Schools
  • VO-enabled tools in support to the ESFRI and science needs identified by the project
  • VO standards in support to the ESFRI needs identified by the project
  • Tools in support to data publishing in the VO
  • Other results of collaborative work

Presentations of DADI participants in the Interoperability meetings organised twice a year by the International Virtual Observatory Alliance IVOA can also be considered as DADI “products”. A list is provided in this repository

DADI team members' contributions to the final ASTERICS event, the New Era of Multi-Messager Astrophysics, is also one of the final products of DADI. The list is provided, with links to the slides or poster.

DADI also organised many workshops which are direct products of its activities. The list of the Workshops, with a link to the meeting agenda and all the material presented, is on DADI wiki site here. Some were formal deliverables included in the project initial description of work, others were organised at the request of the ESFRI/pathfinder partners on topics of specific interest to them, with in particular a series of Provenance Workshops, or co-organised with other ASTERICS Work Packages.

Scientific tutorials

The scientific tutorials prepared for the DADI Schools are provided on a sustainable web page from the Euro-VO web site http://www.euro-vo.org/?q=science/scientific-tutorials. Their last version to date, with the list of the updates if any, as well as additional tutorials prepared for different occasions by the DADI teams, are listed below.

Tutorial name Link Description Dates
The CDS tutorial pdf This tutorial describes the basis of the VO program hosted at CDS. The three major VO programs are described: SIMBAD (astronomical database), VizieR (catalog service) and Aladin (interactive sky atlas). The user gets familiar with the programs 1) searching for the galaxy NGC4039 through the CDS portal to get direct access to SIMBAD, VizieR and Aladin, 2) comparing the sky coverage between SDSS and GALEX surveys using Aladin and 3) selecting interacting galaxies with Aladin. School Nov. 2018, School Nov. 2017, Jan. 2017, School Nov. 2016, School Dec. 2015
Discovery of Brown Dwarfs mining the 2MASS and SDSS databases pdf This tutorial uses the advanced VO functionalities of Aladin (interactive sky atlas), TOPCAT (tools to work on catalogs) and STILTS to find brown dwarfs in 2MASS and SDSS surveys using three differents methods. The user learns about the filtering, cross-matching and visualization functions of Aladin and TOPCAT as well as about the implementation of scripts for Aladin and STILTS to repeat several time the same steps leading to the identification of the brown dwarfs in these surveys. Schools Nov. 2017, Nov. 2016, Dec. 2015
Determination of stellar physical parameters using VOSA pdf This tutorial uses the advanced VO functionalities of VOSA (VO Sed Analyzer) and TOPCAT to determine empirically the masses and radii of stars surrounded by planets. The user needs to register to get access to the functionalities of VOSA (online tool). They can then upload a list of objects to study, build their SEDs and analyze them (by fitting models). Using the interoperability between VOSA and TOPCAT, the user can compare the empiral values obtained with VOSA to those published in papers. Schools Nov. 2018, Nov. 2017, Nov. 2016, Dec. 2015
Multi-instrument, multi-wavelength study of high energy sources with the virtual Observatory pdf This tutorial makes extensive use of the cross-matching and selection (based on criteria) functionalities of the VO tools to study high energy sources observed at different photon energy bands and with various instruments. The user queries SIMBAD to get HESS sources and overlays them on a Fermi LAT image in Aladin. They then cross-match HESS sources with the Fermi LAT catalog that they download from VizieR via its interoperability with TOPCAT. Applying selection criteria with TOPCAT functionalities, they can identify different classes of objects in a color-color diagram to finally get SEDs for the objects of interest. Schools Nov. 2016, Dec. 2015
Exploring Gaia with TopCAT and STILTS pdf This tutorial uses TOPCAT and STILTS to study the Pleiades open star cluster. The user starts with getting TGAS data for the Pleiades and identifies its as a comoving subset. In a second step, the user matches HST data with Gaia observations using the interoperability of TOPCAT with VizieR to access the catalogs. The cross-match is refined using a color-magnitude diagram. The user can also use the TAP (Table Access Protocole) service of TOPCAT to run scripts for Gaia data. Finally, the user can upload the full TGAS catalog and investigate it with STILTS. Schools Nov. 2018, Nov. 2016
Accessing and cross matching big datasets with ADQL webpage This online tutorial permits the user to get familiar with ADQL (Astronomical Data Query Language) and TAP (Table Access Protocol) through lots of examples. ADQL and TAP are widely used in VOs to handle large datasets that cannot be handled locally. Schools 2018, Nov. 2017, Nov. 2016
Advanced usage of HiPS and MOCs pdf This is a hands-on tutorial demonstrating an advanced usage of Hierarchical Progressive Surveys (HiPS) and Multi-Order Coverage (MOC) maps in Aladin. Using this document, you will learn how to handle a problem like : I have an image survey. I would like to select regions in my observations that are above a given threshold in another survey (e.g. at low extinction), retrieve objects from very large catalogs (e.g. Gaia + WISE) in these non-trivial shapes and not-necessarily-connected regions, and combine them to visualise some quantities (e.g. color magnitude diagram). School Nov. 2018 (June 2018), June 2017 (Aladin V10), Feb. 2017
Electromagnetic follow-up of gravitational wave events github page This online tutorial uses mostly Aladin functionalities to locate the sources of latest gravitational wave events on the sky. Schools Nov. 2018, Nov. 2017
Handling gravitational-wave sky maps for EM-followUP observations pdf This tutorial uses the functionalities of Aladin to visualize probability sky maps. It also makes extensive use of the MOC (Multi-Order Coverage) capabilities of Aladin to query a given region of the sky and access the catalogs with information in that region via the interoperability with VizieR. Finally the additional interoperability with TOPCAT permits sending the data available in that region to the latter for further studies. School Nov. 2016
Classifying the SEDs of Herbig Ae/Be stars pdf The overall goal of this tutorial is to become familiar with VOSpec, by building the Spectral Energy Distribution (SED) of a Herbig Ae/Be star. Schools Nov. 2017, Dec. 2015
Advanced functionalities in TOPCAT pdf Advanced TOPCAT tutorial (December 2015), covering of the desktop GUI tool TOPCAT and its command-line evil twin STILTS. These tools are designed to do things with tabular data - typically source catalogues. TOPCAT and STILTS can do basically the same things, but are used in different ways. TOPCAT is easier to learn, and good for interactive use, especially exploring data to get a feel for what's there. For production work, it is sometimes better to move on to STILTS, which can be scripted for repeated or reproducible work. STILTS can also, for some purposes, be used for larger datasets. School Dec. 2015
Compute Redshifts of Quasars Using SPLAT-VO pdf SPLAT-VO to search for spectra of galaxies and quasars in the VO, and compute their redshifts and velocity, using the Hydrogen Lyman Alpha line. School Dec. 2015
Abell 1656: The Coma Cluster of Galaxies pdf This tutorial uses the advanced VO functionalities of Aladin (interactive sky atlas), TOPCAT (tools to work on catalogs) and Cassis (interactive spectrum analyzer) to study interactively the Coma cluster of galaxies. The user can visualize the Coma cluster of galaxies and build a subset of these galaxies with Aladin. With TOPCAT, they can analyze this subset. Finally, they can study an HST power spectrum with Cassis. Jan. 2017

VO-enabled tools

The tools listed here are in particular used in the science tutorials. A more complete list of VO-enabled applications can be found in the IVOA site at http://www.ivoa.net/astronomers/applications.html.

Tool name Link Short description Producer
Aladin web site The interactive sky atlas Aladin allows one to visualize astronomical images and to superimpose entries from different catalogs and databases, including SIMBAD and VizieR, as well as to upload own tables or images. There are two versions of Aladin: Aladin desktop and Aladin lite which runs in the web browser. CDS
CASSIS web site CASSIS is a tool for the scientific analysis of high resolution spectral observations, which includes line identification, prediction of spectra from telescopes, comparison of observations and models and determination of physical parameters of the source. IRAP
GWSky web site Interactive Python script to generate a sequence of pointings given a specific Gravitational Wave Field of View (FoV). Interactive Detection Sky Map INFN with CDS support
SIMBAD web site The SIMBAD astronomical database contains more than 8 million objects. For each object it provides basic measurements (type, coordinates, proper motion, radial velocity, spectral type, distance, magnitude), cross-correlations and bibliography. CDS
SkyMap viewer web site SkyMap Viewer is a sky atlas for understanding LIGO-Virgo skymaps. LIGO with CDS support
SPLAT-VO web page SPLAT is a graphical tool for displaying, comparing, modifying and analysing astronomical spectra stored in several file formats. Spectra can be read from local files or retrieved through VO protocols. Maintained by GAVO
STILTS web site STILTS (STARLINK Tables Infrastructure Library Tool Set) offers many of the same facilities as TOPCAT (and some additional ones) in the form of command-line tools, which can be invoked from the Unix/DOS command-line prompt, or from Jython. University of Bristol
TOPCAT web site TOPCAT (Tool for OPerations on Catalogues And Tables) is an interactive graphical viewer and editor for tabular data. Its aim is to provide most of the facilities that astronomers need for analysis and manipulation of source catalogues and other tables, though it can be used for non-astronomical data as well. University of Bristol
VizieR web site The VizieR catalogue service provides access to about 15,000 catalogues, including very large surveys, and datasets attached to publications. CDS
VOSA web site VOSA (VO SED analyser) is a tool which provides a workflow to derive physical properties of stars by fitting synthetic spectra or photometry extracted from theoretical models to observational data. Spanish VO
VOSpec web site VOSpec is a multi-wavelength spectral analysis tool with access to spectra, theoretical models and atomic and molecular line databases registered in the VO. ESAC

VO standards

The VO standards and their history can be found from the IVOA web page here: http://www.ivoa.net/documents/. Several are particularly linked to DADI, either because they tackle DADI initial priorities, multi-dimensional data and time domain, or because they deal with topics which appeared as priorities for the ESFRIs, in particular HiPS/MOC and Provenance. A short description of these standards of particular interest for DADI is given below. The status of the standard is given at the end of ASTERICS (April 2019). The link goes to the current version of the standard when it is activated.

IVOA categories of standards: App: Applications, DM: Data Model, DAL: Data Access Layer, G&WS: Grid & Web Services, Semantics, VoE: VO Event - currently dealt with by DM and DAL

Standard status in IVOA: REC: IVOA Recommendation, PR: IVOA Proposed Recommendation, WD: Working Draft, see IVOA Document standards for more details)

Topic IVOA Category Standard name Link Short description Status in IVOA in April 2019
The multi-dimensional “DAL caravan” DAL DataLink current version DataLink describes the linking of data discovery metadata to access to the data itself, further detailed metadata, related resources, and to services that perform operations on the data. V1.0 REC, 17 June 2015
DAL SIAP current version The Simple Image Access protocol (SIA) provides capabilities for the discovery, description, access, and retrieval of multi-dimensional image datasets, including 2-D images as well as datacubes of three or more dimensions. V2.0 REC, 23 Dec. 2015
DAL SODA current version The Server-side Operations for Data Access (SODA) web service capability is a low-level data access capability or server side data processing that can act upon the data files, performing various kinds of operations: filtering/subsection, transformations, pixel operations, and applying functions to the data. V1.0 REC, 17 May 2017
Related standard DM ObsCore Current version ObsCore defines the core components of the Observation data model that are necessary to perform data discovery when querying data centers for astronomical observations of interest. ObsCore has been selected as the core data model for data discovery by the Simple Image Access protocol version 2 (SIAv2). V1.1 REC, 09 May 2017
Time Domain VoE VOEvent current version VOEvent defines the content and meaning of a standard information packet for representing, transmitting, publishing and archiving information about a transient celestial event, with the implication that timely follow-up is of interest. V2.0 REC, 11 July 2011
DAL VTP current version VOEvent Transport Protocol formalizes a TCP-based protocol for VOEvent transportation that has been in use by members of the VOEvent community for several years and discusses the topology of the event distribution network. It is intended to act as a reference for the production of compliant protocol implementations. V2.0 REC, 20 March 2017
App VOTable current version VOTable is the IVOA XML standard for interchange of data represented by a set of tables. V1.4 includes the TIMESYS element, a simple means to furnish time in VOTables with the necessary metadata V1.4 WD, 3 March 2019
Space coverage App MOC current version The standard describes Multi-Order Coverage maps (MOC) to specify arbitrary sky regions. The goal is to be able to provide a very fast comparison mechanism between coverage maps. The mechanism is based on the HEALPix sky tessellation algorithm. V1.1 PR, 04 April 2019
Sky tessellation App HiPS current version HiPS is a hierarchical scheme for the description, storage and access of sky survey data, based on hierarchical tiling of sky regions at finer and finer spatial resolution which facilitates a progressive view of a survey, and supports multi-resolution zooming and panning. HiPS uses the HEALPix tessellation of the sky as the basis for the scheme and is implemented as a simple file structure with a direct indexing scheme that leads to practical implementations. V1.0 REC, 19 May 2017
Provenance DM ProvenanceDM current version ProvenanceDM describes how provenance information for astronomical datasets can be modeled, stored and exchanged within the astronomical community in a standardized way. The definition of provenance as proposed by the W3C is applied. V1.0 PR, 19 October 2018
Authentication G&WS SSO current version Approved client-server authentication mechanisms are described for the IVOA single-sign-on profile. Normative rules are given for the implementation of these mechanisms, mainly by reference to pre-existing standards. V2.0 REC, 24 May 2017
Semantics for quantities Semantics UCD1+ current version The document contains the list of controlled terms used to build the Unified Content Descriptors, which describe quantities in the Virtual Observatory. V1.3 REC, 27 May 2018
Semantics UCDListMaintenance current version Description of the procedures used to update the UCD list V2.0 PR, 23 August 2018

Data publication tools

The list includes the data publication tools demonstrated during the Training sessions of DADI Workshops. An excellent introduction to publishing in the VO was presented at the Heidelberg European Data Provider Forum & Training event by Grégory Mantelet (pdf).

Name Link Description Producer
CDS-ARI libraries ADQL Library, TAP Library, UWS Library a set of four generic Java libraries, each of them implementing an IVOA standard: ADQL-Lib for ADQL, UWS-Lib for UWS (a protocol for asynchronous execution of computations), TAP-Lib for TAP (Table Access Protocol) and SCS-Lib for Simple Cone Search. Maintained by CDS
DaCHS web site The GAVO Data Center Helper Suite (DaCHS) is a publishing infrastructure for the Virtual Observatory, including a flexible component for ingesting and mapping data, integrated metadata handling with a publishing registry, and support for many VO protocols and standards. UHEI
INAF IA2 VO Publishing tools VO-Dance, IA2TAP, TAP Schema Manager A set of tools to deploy and manage VO services based upon S*AP and TAP recommendations. Tools are all developed as Java web applications, connecting to MySQL RDBMS back ends. INAF IA2
SAADA web site Saada builds databases from data files: no code to write - heterogeneous dataset storage - can host multiple data collections - support persistent links between data - metadata tagging - access by Web interface, VO protocols or API. OAS
SVO Publishing Tools web site SVOCat and MySpecMyImg are tools developed respectively to publish a web interface and access through the relevant VO protocols, respectively ConeSearch and SIAP or SSAP. No code to write and little technical expertise is required. Spanish VO
VizieR web site VizieR can be used to publish tables and data (e.g. images, spectra) in the Virtual Observatory, taking advantage of all the service functionalities. CDS

Other products of DADI work

Title Link Short description Producer
Maintenance of PGSphere web site PGSphere, a popular extension of PostgreSQL for astronomy used by many data providers, is now maintained by UHEI. This includes in particular its extension to MOC data and using computation exclusively in the database back-end (an exemplar of the code-to-data paradigm) to create a publication-ready data product. UHEI
Publication of ANTARES data in the VO query form The time integrated search for point sources of cosmic neutrinos was performed using the data collected from January 2007 to November 2010 by the ANTARES neutrino telescope (a Km3NeT pathfinder) was published in the VO by GAVO following the First Trieste ESFRI Forum & Training Event GAVO and LPPM
Inclusion of the GLADE catalogue in VizieR VizieR catalogue entry GLADE (Galaxy List for the Advanced Detector Era) is a value-added full-sky galaxy catalog with high completeness for identifying gravitational wave (GW) sources in order to support future electromagnetic (EM) follow-up projects of the LIGO/Virgo Collaboration. It was included in VizieR as a result of the DADI Gravitational wave meeting. CDS and Eötvös University
Proper inclusion of the Gravitational Wave Events in SIMBAD list from SIMBAD The way to include Gravitational Wave events in SIMBAD was discussed between CDS and APC during the DADI Gravitational wave meeting and the Third DADI Technology Forum CDS and APC

The first IVOA meeting during ASTERICS was held in May 2015 just after the beginning of the project, too early to report on “ASTERICS products” for this meeting. The other meetings held since then have allowed significant input from DADI:

List of talks related to topics discussed in DADI presented by participants working in European laboratories during the meetings, with a link to the slides:

DADI contribution to ASTERICS final event

DADI contributions to the ASTERICS final event, The New Era of Multi-Messenger Astrophysics, which was held in Groningen 26-29 March 2019, are an excellent illustration of DADI results, impact and legacy.

open/wp4/dadiproductrepository.1556108766.txt.gz · Last modified: 2019/04/24 14:26 by genova