User Tools

Site Tools


open:wp4:wp4techforum2

ASTERICS DADI Technology Forum 2

The Forums enable technology discussion involving the framework developers and ESFRI project staff, to develop technical collaboration and exchange good practice - establishing a European strategy for IVOA and RDA activities

(Top level DADI page: WP4: Data Access, Discovery and Interoperability (DADI))

Dates

Meeting: 7 & 8 March, 2016

Location

The meeting will be held on the Pollock Halls campus in the Pollock Room in St Leonard's Hall

Hotels

Please book your own accommodation.

To book campus accommodation, visit the booking page, and quote promotion code: CONF16. This will provide 15% off accommodation at Masson House and Salisbury Green Hotel at Pollock Halls.

There are many other Hotels and Guest Houses in the vicinity of Pollock Halls. More information can be found here and here

Dinner

A working dinner will be held Monday evening. This needs to be confirmed as soon as possible so please indicate whether you wish to attend.

Agenda

Time Title Speaker Materials
Day 1 Monday, March 7th
10:00 Coffee
10:15 Welcome and round table intros Keith / Mark A
10:40 DADI status Françoise DADI Status
11:00 EM follow-up of gravitational wave candidates Giuseppe Gravitational Waves
11:20 HiPS - Hierarchical Progressive Survey Pierre HiPS
11:40 A&A systems developed for CTA Mathieu A&A for CTA
12:00 Applying A&A to server/client VO scenario Marco A&A Server-client
12:20 Applicable conditions for A&A (web, non web resources, etc.) Marco A&A applicable conditions
12:40 VODML from a VODSL perspective Paul VODSL
13:00 Lunch
14:00 Links between Provenance Data Model prototypeand the IVOA Dataset Metadata data Model Mireille Provenance
14:15 DAL protocols for ESFRI projects / current status Francois / Chaitra DAL Protocols
- IVOA DAL context status
- current development at CDS : DataLink integration within Aladin
14:50 Aladin Lite and usage of the MOCServer Thomas MOCServer
15:10 SAMP over HTTPS Mark T tlsamp
15:30 Afternoon tea
15:50 Intro to Hack-a-thon Keith
16:00 Hack-a-thon
17:30 Close
19h Tech Forum Dinner To be held in the Nelson Room, St Leonard's Hall
Day 2 Tuesday, March 8th
09:30 Description and discovery of VizieR resources in VO registries) Sébastien VizieR resource discovery
09:50 Registry Markus Registry Notes
10:50 Coffee
11:10 SODA – what's all the fuss about on the DAL list? Markus SODA Notes
11:30 Madrid School Françoise School2015
11:50 Engaging with major astronomy projects at the next IVOA meeting Mark A Engagement
12:10 Discussion : Engagement & IVOA Interop prep
13:00 Lunch
14:00 Hack-a-thon
15:30 Afternoon tea
15:50 Hack-a-thon
17:00 Conclusions Keith / Mark A
17:30 Close
Breakout sessions

Monday

1. Provenance data model (Notes author : Mireille)
minutes by Mireille Louys draft to be checked with group.
Francois Bonnarel, Markus Nullmeier, Mireille Louys, Matthieu Servillat, Julien LeFaucheur, Chaitra ( CDS contract)
(i). Which existing generic tools could we re-use?
Prov-N used in the RAVE use-case
Prov Store (Southampton) offers a software suite to handle the entity/activity/agent pattern. It is based on the Prov package in Python that can be used directly. Will be explored further by Luth ( Mathieu, Julien )
(ii). Format for serialising the metadata
JSON
prov-XML
TAP-Schema
Fits- keywords??? extra header
Need for a bundle to wrap all together: data (fits) +Instrument files+ IVOA metadata
(iii). Modeling
- which part to re-use from Obscore
- how to bind IRF , instrument config, etc.
- need for a version tag in the Method metadata
(iv). Usecases to clarify
need for practical use-cases for all projects:
*CTA Julien & Mathieu& Catherine ?
a possible use-case: spectrum reconstruction as UWS service in LUTH.
(v). RAVE / IAP
check with Kristin
Pollux Michèle
SVOM starting the effort(Laurent michel , XMM-High energy physics team, Strasbourg Observatory)
(vi). Questions to discuss
Mireille: how could we disentangle the different layers:
- science use-case: what the scientist needs to see , and provide as input data and parameters
- chain of algorithms : description of methods and params.
- workflow execution : job stalled, job terminated, files in , files out , config , etc.
(vii). Proposed meetings
Wish to have an IVOA tech meeting together :
Paris? Strasbourg? Heidelberg?
difficult to plan before IVOA :may be in June-July?
(viii). IVOA DM session
ask for a provenance DM session at next interop in Cape town:
think about targeting some didactic presentation for the audience :
many newcomers to the VO are expected.
2. HiPS / Registry discussion (Notes author : Pierre)
4 participants : Thomas Boch, Sebastien Derrière, Pierre Fernique & Markus Deimleitner
A good debate, with a lot of exchanges, lots of semantic explorations (creator vs publisher, resource vs service, …)
We did a beautiful loop to come back to the agreement that we had at the beginning of the discussion (publisher on App mailing list and presented in the Pierre F's HiPS talk)
3. GWsky discussion (Notes author : Giuseppe)
Interesting discussions on the possibility of implementing MOC - dedicated to the coverage manipulations - and HiPS to manipulate data (pixels, catalog, sources, etc.). This will be done using the Aladin script commands via SAMP. In particular, the HEALPix pixels contained in a certain probability will be considered for faster loading of catalogs or images.

In addition, we have discussed to develop a graphical user interface (Tkinter?) and to redirect the output of GWsky to a printable page. In general, the use of SAMP has been highly appreciated and encouraged for any future development

4. ADQL (Notes author : Dave)
Participants : Dave Moris & Marco Molinaro
Planning meeting concerning next steps for the ADQL specification.
OFFSET/LIMIT
The current consensus on the mailing list is to accept OFFSET without LIMIT.
TODO - Dave : Write a page on the IVOA wiki summarising the OFFSET/LIMIT syntax for each of the main platforms, PostgreSQL, MySQL, SQLServer, Oracle etc.
If a database platform requires LIMIT, the proposed work around is to specify either (MAX_ROWS + 1) or (MAX_LONG).
However, many of the text books and online manuals describe using OFFSET with LIMIT.
TODO - Dave : Propose to the mailing list that we accept OFFSET with LIMIT, allowing LIMIT as a synonym for TOP.
The only reason for this is to make ADQL compatible with 90% of the text books and online manuals.
Hopefully this should be a short yes/no discussion and then accept the consensus for the v2.1 specification.
TODO - Dave : Update the text of the v2.1 specification to include OFFSET (with/without LIMIT as agreed)
Hexadecimal literals
Adding hexadecimal literals to the BNF grammar has raised some issues.
The simple way to add them would be to extend numeric literal to allow both decimal and hexadecimal values.
If we follow a strict interpretation of the BNF, then allowing both decimal and hexadecimal values for numeric literals causes some odd side effects, particularly with scientific notation and exponents.
TODO - Dave : Write a page on the IVOA wiki detailing the issues.
TODO - Dave : Propose to the mailing list that we accept
a) Numeric literals may contain both decimal and hexadecimal values but there are places where hexadecimal values do not make sense, and developers should do the right thing.
and
b) The BNF grammar is not a definitive machine readable specification and there may be some unintended side effects in it.
TODO - Dave : If consensus is to accept a flexible interpretation of the BNF, update the text of the v2.1 specification to reflect this.
If consensus is for a strict interpretation of the BNF, then discussion on validating a strict BNF grammar will need to be continued at the May interop.
Crossmatch function
The mailing list has yet to agree on a consensus for the crossmatch function.
TODO - Marco : Write an email to the mailing list summarising the options and starting a new discussion thread.
TODO - Dave : If consensus is reached, update the text of the v2.1 specification to include the crossmatch function.
If no consensus is reached, then discussion of the crossmatch function can be continued at the May interop.
Specification review
The plan is to try to resolve the outstanding issues during March, and have an updated version of the specification ready for review in April.
If we are not able to reach consensus on an issue, then it should be deferred to the next version of the specification.
5. Rendering plots server-side (Notes author : Mark)
Discuss options for server-side rendering of plots using topcat/plot2 infrastructure and a web browser with small(?) amounts of javascript Looks like it should be possible to make a start on it without too much difficulty. Mark to have a first go, Stelios may find time for some work on it in future if javascript expertise is required Future progress depends on our other commitments.
Discuss some technical issues related to auto-completion of ADQL queries.

Tuesday

6. A&A (Notes author : Marco)
Participants:
Giuliano Taffoni (skype)
Mark Taylor
Stelios Voutsinas
Pierre Fernique
Paul Harrison
Keith Noddle
Dave Morris
Mathieu Servillat
Thomas Boch
IVOA SSO-2.0 and CDP-1.0 standards are what is available in the VO to secure resources/services and delegate user certification. This standards seem to cover the needs on the server side (maybe CDP needs a refurbish due to technology changes) but the user authentication and authorization from the client side point of view is an open question.
Currently everything has worked against public services, and HTTPBasic authentication proved to be enough when data policies where of concern from data providers.
The participants discussed about whether it would be needed to introduce more complex solutions for authentication at present time and what the uses cases for this were.
The outcome was that: use cases probably are variations on:
- a user that authenticates and then tries to access to different services: how does the identification works in this case
- a user consumes a secured server and tries to push the results to a different one (again secured)
In a sense there are ongoing projects that try to solve this, in terms of provider federation, but not from the client side.
It seems to be no urgent matter to solve, nonetheless thinking of solutions to these use cases in advance can be useful to be prepared when the issue will became a critical one, and may also be useful if some ESFRI partners (instead of the full VO community) decide to pursue authentication when providing data and services (some of them are already developing AuthN&AuthZ infrastructures with identity providers of their own, e.g. CTA and SKA).
A solution for the client side, i.e. for applications that run on the user's desktop, could be, e.g., having a local authentication tool to act as a proxy to handle the SSL and authentication delegation towards the accessed resources. This would act as a sort of secondary HUB to the SAMP one, letting applications work mainly as they do now, with this proxy taking care of passing on the credential tokens or certificates to the requesting resources.
It could also be a way to solve the SAMP issue with the web profile and mixed HTTP/HTTPS connections.
Oncoming Northern Spring Interop (Cape Town, May 2016) will probably be a good place to continue on this discussion.
7. pgSphere (Notes author : Markus N.)
minutes by Markus Nullmeier
Dave Morris, Keith Noddle, Markus Nullmeier, Mathieu Servillat, Mike Sipior (via Skype), Paul Harrison, Stefano Zampieri, Stelios Voutsinas
(i). An update on the state of pgSphere
- Markus joined forces with the developers from Moscow, his work on the code was merged by Alexander Korotkov into the master branch of https://github.com/akorotkov/pgsphere, which is the official pgSphere repository for the time being.
- A new official release of pgSphere is imminent, hopefully in a few weeks at latest. This release will support PostgreSQL releases 9.1 and later (i.e., all supported ones). It also uses the 9.1+ “CREATE EXTENSION pg_sphere;” SQL syntax to activate pgSphere for a database.
- There will be a new home page for the project soon, together with a repository and a mailing list for both users and developers (everyone present accepted the invitation to become a member of this list). Then, a pointer to the new website will be put on the old project page on PgFoundry.
- The Russian developers are still active in further improving the already much faster indexing code (one order of magnitude in the present code base) that was released on Github in 2014 (A. Korotkov: "A new double sorting-based node splitting algorithm for R-tree", Programming and Computer Software 38(3), 109-118).
- The pgSphere documentation explains how to use its spatial indexing, which uses 3d R-trees on the unit sphere.
- If somebody should require commercial support for pgSphere, Postgres Professional (info@postgrespro.ru) would certainly be up to the task.
- Extending pgSphere to provide MOC as an additional data type for objects on the surface of the unit sphere (such as circles and polygons that were in pgSphere from the start) was delayed by several other things, but now it has very high priority for Markus. There already exists a concept to do proper (“high-order”) indexing support to accelerate SELECT operations on MOCs to be faster than sequential scan (just by viewing a MOC as a set of numbers), but this requires PostgreSQL core support (written by Alexander Korotkov), which is hopefully going to be accepted for inclusion into PostgreSQL 9.6 (due to be released in 2016).
(ii). Installing pgSphere
- compiling the latest official release 1.1.1 from 2010 works only up to PostgreSQL release 9.1, this has created serious problems for at least one of the attendants (a patched version for later PostgreSQL releases is available as https://github.com/mnullmei/pgsphere/archive/fixes-1-1-1.tar.gz , users of Debian 8 (Jessie) may install a packaged version of this as “postgresql-9.4-pgsphere” from http://soft.g-vo.org/repo
– additional information provided after the session)
- Users of Redhat-based Linux distributions may install pgSphere 1.1.1 for a PostgreSQL from the PostgreSQL Yum Repository as the package “pgsphere95” for PostgreSQL 9.5 / etc.
- Packaging pgSphere for the Debian Linux distribution (which will proliferate into Ubuntu), and also for the PostgreSQL Apt Repository (analogous to the PostgreSQL Yum Repository) is planned for the time after the upcoming pgSphere release
- for pgSphere on Docker see further below.
(iii). pgSphere for DaCHS users
- DaCHS support for the upcoming pgSphere release is under way.
- DaCHS users should not upgrade to pgSphere from https://github.com/akorotkov/pgsphere just yet.
(iv). Use cases
- image services, a feature that most other spatial database extensions lack
- using the Java taplib libraries for TAP and ADQL to re-use existing Java code
- (potential) replacement of non-PostgreSQL data base systems
(v). pgSphere on Docker
- Dave gave an demonstration of effortlessly running different PostgreSQL releases under Docker.
- Availability of pgSphere for Docker would be of considerable help to several people.
- There is also some activity to provide DaCHS images for Docker.
(vi). Science-based common test cases
- Dave instigated the idea to jointly create a set of common science-based test cases for ADQL, cone search, etc., with known results in VOTable files.
- These would certainly also help pgSphere.
- Markus mentioned http://skyview.gsfc.nasa.gov/xaminblog/index.php/tag/pgsphere – it is unknown if the bugs described there would still show up with the present codebase of pgSphere.
8. SODA(Notes author : François)
SODA is the last “open” element of the “multi-dimensional standard caravan”, a top priority of the IVOA as well as ASTERICS. The debate on architecture for expanding the requirements beyond those defined by the IVOA Committee for Scientific Priorities continued during the Hack-A-Thon and it was not possible to reach a consensus at that time.
NB (added after the Cape Town Interop): The discussion continued in the IVOA DAL Working Group after the meeting and a consensus was finally reached on an incremental approach. It was presented in Cape Town and is the basis of SODA Proposed Recommendation.

Organisation

Program Organising Committee:Keith Noddle, Mark Allen (co-chairs)

Local Organising Committee: Keith Noddle, Nathalie Dupin, Andrew Lawrence

Contact: ktn (at) roe.ac.uk

Participants

Name Affiliation Menu
Andrew Lawrence IfA, Edinburgh
Bob Mann (excused) IfA, Edinburgh n/a
Carlos Rodrigo INTA
Chaitra Suvarna CDS, Observatoire de Strasbourg
Clive Davenhall IfA, Edinburgh n/a
Dave Morris IfA, Edinburgh
François Bonnarel CDS, Observatoire de Strasbourg
Francoise Genova CDS, Observatoire de Strasbourg
Giuliano Taffoni INAF - OATs
Giuseppe Greco Università degli Studi di Urbino and INFN-Firenze n/a
Julien Lefaucheur LUTH / Observatoire de Paris-Meudon
Keith Noddle IfA, Edinburgh V
Marco Molinaro INAF - OATs
Mark Allen CDS, Observatoire de Strasbourg
Mark Holliman IfA, Edinburgh n/a
Mark Taylor University of Bristol V
Markus Demleitner ARI, UHEI, Heidelberg V
Markus Nullmeier ARI, UHEI, Heidelberg V
Mathieu Servillat LUTH / Observatoire de Paris-Meudon
Mike Sipior (excused) ASTRON/LOFAR
Mireille Louys CDS, Observatoire de Strasbourg
Paul Harrison University of Manchester
Pierre Fernique CDS, Observatoire de Strasbourg
Rob van der Meer ASTRON/LOFAR
Sebastien Derriere CDS, Observatoire de Strasbourg
Stefano Zampieri
Stelios Voutsinas IfA, Edinburgh
Thomas Boch CDS, Observatoire de Strasbourg

28|24 participants


  • Hot Scottish smoked salmon, lemon, capers & black pepper
  • Supreme of chicken stuffed with haggis in thyme scented whisky cream sauce
  • Drambuie & stem ginger cranachan in a brandy snap basket
  • Filter coffee & petits fours

Vegetarian Starter

  • Sweet & sour vegetable spring roll with a fresh mango salsa

Vegetarian Main

  • Sweet potato, squash & coriander tart with crispy sweet pickled vegetable salad

open/wp4/wp4techforum2.txt · Last modified: 2016/12/20 09:11 by genova