Changes between Version 11 and Version 12 of SoftWare/ProjectMeta


Ignore:
Timestamp:
Apr 29, 2018, 7:55:21 PM (6 years ago)
Author:
g7moreau
Comment:

Clean source

Legend:

Unmodified
Added
Removed
Modified
  • SoftWare/ProjectMeta

    v11 v12  
    11= SoftWare / !ProjectMeta - Meta project for open data management =
     2
    23== Aim ==
    3 Project-Meta is a software to help you to manage your open data, using the protocol OpenDAP. The initiative is supported by the European Commission as part of the project Hydralab+ of the Horizon 2020 programme. This programme  requests that research data are open access, that is providing online access free of charge to the end-user and reusable. Furthermore access must allow the right to copy, distribute, search, link, crawl and mine the data. In addition to these general requests, we aim at achieving the following goals: 1) Allow the end user to scan and visualise the data without downloading. 2) Integrate the process in the data analysis procedure, with minimal additional work.
    44
    5 == OpenDap ==
    6 The protocol OPeNDAP (Open-source Project for a Network Data Access Protocol). This includes standards for encapsulating structured data, annotating the data with attributes and adding semantics that describe the data.  OPeNDAP is widely used by governmental agencies such as NASA and NOAA to serve satellite, weather and other observed earth science data.
     5Project-Meta is a software to help you to manage your open data, using the protocol OpenDAP.
     6The initiative is supported by the European Commission as part of the project Hydralab+ of the Horizon 2020 programme.
     7This programme  requests that research data are open access, that is providing online access free of charge to the end-user and reusable.
     8Furthermore access must allow the right to copy, distribute, search, link, crawl and mine the data.
     9In addition to these general requests, we aim at achieving the following goals:
     10 1. Allow the end user to scan and visualise the data without downloading.
     11 2. Integrate the process in the data analysis procedure, with minimal additional work.
    712
    8 The protocol is based on http, so that data can be scanned with an ordinary web browser. However added functionality of data visualization is provided by graphics programs (like Matlab, GrADS, Ferret or ncBrowse). Compared to ordinary file transfer protocols (e.g. FTP) a major advantage using OPeNDAP is the ability to retrieve subsets of files, so it is possible to work remotely without downloading whole data files.  Although any file format can be use, data are often in HDF or NetCDF formats. The older NetCDF format is limited to arrays of numbers, while HDF provides wider possibilities of data structures (and it contains NetCDF as a particular case).  We choose the NetCDF format which is sufficient for most experimental data and can be more easily read with a variety of software.
     13
     14== OPeNDAP ==
     15
     16The protocol OPeNDAP (Open-source Project for a Network Data Access Protocol).
     17This includes standards for encapsulating structured data, annotating the data with attributes and adding semantics that describe the data.
     18OPeNDAP is widely used by governmental agencies such as NASA and NOAA to serve satellite, weather and other observed earth science data.
     19
     20The protocol is based on http, so that data can be scanned with an ordinary web browser.
     21However added functionality of data visualization is provided by graphics programs (like Matlab, GrADS, Ferret or ncBrowse).
     22Compared to ordinary file transfer protocols (e.g. FTP) a major advantage using OPeNDAP is the ability to retrieve subsets of files,
     23so it is possible to work remotely without downloading whole data files.
     24Although any file format can be use, data are often in HDF or NetCDF formats.
     25The older NetCDF format is limited to arrays of numbers, while HDF provides wider possibilities of data structures (and it contains NetCDF as a particular case).
     26We choose the NetCDF format which is sufficient for most experimental data and can be more easily read with a variety of software.
     27
    928
    1029== Description ==
    11 The creation of the OpenDAP repository is done by the script [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/project-meta.html project-meta] using unix commands, scripts in Perl and C++ .
    1230
    13 The first step is to make a list of the data folders to display. This list needs to be introduced in a text file, complemented by some information about the authors of the work and about related publications. This text file must be put in your current folder with the name `PROJECT-META.yml`. The text structure must follow some simple rules consisting in the YAML format. An example could be found in the Project-Meta repository or online [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
     31The creation of the OpenDAP repository is done by the script
     32[http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/project-meta.html project-meta]
     33using UNIX commands, scripts in Perl and C++.
     34
     35The first step is to make a list of the data folders to display.
     36This list needs to be introduced in a text file, complemented by some information about the authors of the work and about related publications.
     37This text file must be put in your current folder with the name `PROJECT-META.yml`.
     38The text structure must follow some simple rules consisting in the YAML format.
     39An example could be found in the Project-Meta repository or online
     40[http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
     41
    1442
    1543== `PROJECT-META.yml` meta file ==
    16 This file is at the core of the procedure. A first task is to list the data folder to publish. For that purpose a good practice is to organise the data and the procedures of analysis such that the final data to publish are contained in folders named with specific extensions. Then search tools can be used to list all the selected folders. For instance the following unix command lists all the folders with extension '`*.mproj*`' and append it to the file `PROJECT-META.yml` (and creates the file if it does not exist yet):
    1744
     45This file is at the core of the procedure. A first task is to list the data folder to publish.
     46For that purpose a good practice is to organise the data and the procedures of analysis such that the final data to publish are contained in folders named with specific extensions.
     47Then search tools can be used to list all the selected folders.
     48For instance the following unix command lists all the folders with extension '`*.mproj*`' and append it to the file `PROJECT-META.yml`
     49(and creates the file if it does not exist yet):
    1850{{{
    1951find . -name '*.mproj*' -a -type d | sed 's/^/    - /;' >> PROJECT-META.yml
     
    2153The `find` command only search the folders with the right extension (recursively)  under the current one (.) and the `sed` command add 4 spaces and the dash at the beginning of each line in order to respect the YAML format.
    2254
    23 The YAML file has to be complemented by general information about the authors and the related publications, following the template  [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
     55The YAML file has to be complemented by general information about the authors and the related publications,
     56following the template  [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
     57
    2458
    2559== Debian package ==
    2660
    27 Debian package for amd64 arch could be download on: http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/download.
     61Debian is a GNU/Linux distribution.
     62Debian (and certainly Ubuntu) package for amd64 arch could be download on: http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/download.
    2863
    2964You can then install it with
     
    3166sudo dpkg -i project-meta_*_amd64.deb
    3267}}}
    33 (replace * with the version you have donwloaded).
     68(just replace * with the version you have donwloaded).
     69
    3470
    3571== Software repository ==