Changes between Version 7 and Version 8 of SoftWare/ProjectMeta


Ignore:
Timestamp:
Apr 27, 2018, 6:53:11 PM (6 years ago)
Author:
sommeria
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • SoftWare/ProjectMeta

    v7 v8  
    11= SoftWare / !ProjectMeta - Meta project for open data management =
    2 
    32== Aim ==
    4 Project-Meta is a set of software to help you to manage your open data, using the protocol OpenDAP. The initiative is supported by the European Commission as part of the project Hydralab+ of the Horizon 2020 programme. This programme  requests that research data are open access, that is providing online access free of charge to the end-user and reusable. Furthermore access must allow the right to copy, distribute, search, link, crawl and mine the data. In addition to these general requests, we aim at achieving the following goals:
    5 1) Allow the end user to scan and visualise the data without downloading.
    6 2) Integrate the process in the data analysis procedure, with minimal additional work.
     3Project-Meta is a software to help you to manage your open data, using the protocol OpenDAP. The initiative is supported by the European Commission as part of the project Hydralab+ of the Horizon 2020 programme. This programme  requests that research data are open access, that is providing online access free of charge to the end-user and reusable. Furthermore access must allow the right to copy, distribute, search, link, crawl and mine the data. In addition to these general requests, we aim at achieving the following goals: 1) Allow the end user to scan and visualise the data without downloading. 2) Integrate the process in the data analysis procedure, with minimal additional work.
    74
    85== OpenDap ==
    9 The protocol OPeNDAP (Open-source Project for a Network Data Access Protocol). This includes standards for encapsulating structured data, annotating the data with attributes and adding semantics that describe the data.  OPeNDAP is widely used by governmental agencies such as NASA and NOAA to serve satellite, weather and other observed earth science data. 
     6The protocol OPeNDAP (Open-source Project for a Network Data Access Protocol). This includes standards for encapsulating structured data, annotating the data with attributes and adding semantics that describe the data.  OPeNDAP is widely used by governmental agencies such as NASA and NOAA to serve satellite, weather and other observed earth science data.
    107
    11 The protocol is based on http, so that data can be scanned with an ordinary web browser. However added functionality of data visualization is provided by graphics programs (like Matlab, GrADS, Ferret or ncBrowse). Compared to ordinary file transfer protocols (e.g. FTP) a major advantage using OPeNDAP is the ability to retrieve subsets of files, so it is possible to work remotely without downloading whole data files.
    12 Although any file format can be use, data are often in HDF or NetCDF formats. The older NetCDF format is limited to arrays of numbers, while HDF provides wider possibilities of data structures (and it contains NetCDF as a particular case).  We choose the NetCDF format which is sufficient for our applications and can be more easily read with a variety of software.   
     8The protocol is based on http, so that data can be scanned with an ordinary web browser. However added functionality of data visualization is provided by graphics programs (like Matlab, GrADS, Ferret or ncBrowse). Compared to ordinary file transfer protocols (e.g. FTP) a major advantage using OPeNDAP is the ability to retrieve subsets of files, so it is possible to work remotely without downloading whole data files.  Although any file format can be use, data are often in HDF or NetCDF formats. The older NetCDF format is limited to arrays of numbers, while HDF provides wider possibilities of data structures (and it contains NetCDF as a particular case).  We choose the NetCDF format which is sufficient for most experimental data and can be more easily read with a variety of software.
    139
    1410== Description ==
     11The creation of the OpenDAP repository is done by the script [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/project-meta.html project-meta] (**which langage). The first step is to make a list of the data folders to display. This list needs to be introduced in a text file, complemented by some information about the authors of the work and about related publications. This text file must be put in your current folder with the name `PROJECT-META.yml`. The text structure must follow some simple rules consisting in the YAML format. An example could be found in the Project-Meta repository or online [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
    1512
    16 The main script is [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/project-meta.html project-meta].
    17 Before using it, you will need a METADATA file in your current folder with the name {{{PROJECT-META.yml}}}.
    18 This file is in the YAML format.
    19 An example could be found in the Project-Meta repository or online [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
     13At this stage, Project-Meta is at a early stage of develepment. Many aspect of it will be improve in the future.
    2014
    21 At this stage, Project-Meta is at a early stage of develepment.
    22 Many aspect of it will be improve in the future.
     15== `PROJECT-META.yml` meta file ==
     16This file is the core of the project. One difficult task is to list all your open data. Sometime, it's very easy, sometime it's could be painful and time consuming, especially if there is a lot of data
    2317
    24 == {{{PROJECT-META.yml}}} meta file ==
     18Here an example to append all the folder of kind '`*.mproj*`' (Coriolis example for some open data) at the end of the `PROJECT-META.yml` file.
    2519
    26 This file is the core of the project.
    27 One difficult task is to list all your open data.
    28 Sometime, it's very easy,
    29 sometime it's could be painful and time consuming,
    30 especially if there is a lot of data
    31 
    32 Here an example to append all the folder of kind '{{{*.mproj*}}}' (Coriolis example for some open data)
    33 at the end of the {{{PROJECT-META.yml}}} file.
    3420{{{
    3521find . -name '*.mproj*' -a -type d | sed 's/^/    - /;' >> PROJECT-META.yml
    3622}}}
    37 The {{{find}}} command only search the right folders under the current one (.)
    38 and the {{{sed}}} command add 4 spaces and the dash at the beginning of each line
    39 in order to respect the YAML format.
     23The `find` command only search the right folders under the current one (.) and the `sed` command add 4 spaces and the dash at the beginning of each line in order to respect the YAML format.
    4024
    4125== Repository ==
     26All code is under '''free license'''. Scripts in `bash` are under GPL version 3 or later (http://www.gnu.org/licenses/gpl.html), `C++` sources are under GPL version 2 or newer, the `perl` scripts are under the same license as `perl` itself ie the double license GPL and Artistic License (http://dev.perl.org/licenses/artistic.html).
    4227
    43 All code is under '''free license'''.
    44 Scripts in {{{bash}}} are under GPL version 3 or later (http://www.gnu.org/licenses/gpl.html),
    45 {{{C++}}} sources are under GPL version 2 or newer,
    46 the {{{perl}}} scripts are under the same license as {{{perl}}} itself
    47 ie the double license GPL and Artistic License (http://dev.perl.org/licenses/artistic.html).
     28All sources are available on the LEGI forge: http://servforge.legi.grenoble-inp.fr/svn/soft-trokata/trunk/project-meta
    4829
    49 
    50 All sources are available on the LEGI forge:
    51 http://servforge.legi.grenoble-inp.fr/svn/soft-trokata/trunk/project-meta
    52 
    53 The sources are managed via subversion (http://subversion.tigris.org/).
    54 It is very easy to stay synchronized with these sources
     30The sources are managed via subversion (http://subversion.tigris.org/). It is very easy to stay synchronized with these sources
    5531
    5632 * initial recovery
    57    {{{
     33{{{
    5834svn checkout http://servforge.legi.grenoble-inp.fr/svn/soft-trokata/trunk/project-meta soft-project-meta
    5935}}}
    6036 * the updates thereafter
    61    {{{
     37{{{
    6238svn update
    6339}}}
    6440
    65 It is possible to have access to writing at the forge
    66 on reasoned request to [mailto:Gabriel.Moreau(A)legi.grenoble-inp.fr Gabriel Moreau].
    67 For issues of administration time and security,
    68 the forge is not writable without permission.
    69 For the issues of decentralization of the web,
    70 autonomy and non-allegiance to the ambient (and North American) centralism,
    71 we use our own forge...
     41It is possible to have access to writing at the forge on reasoned request to [mailto:Gabriel.Moreau(A)legi.grenoble-inp.fr Gabriel Moreau]. For issues of administration time and security, the forge is not writable without permission. For the issues of decentralization of the web, autonomy and non-allegiance to the ambient (and North American) centralism, we use our own forge...
    7242
    73 You can propose an email patch of a particular file via the {{{diff}}} command.
    74 Note that {{{svn}}} defaults to the unified format ({{{-u}}}).
    75 Two examples:
     43You can propose an email patch of a particular file via the `diff` command. Note that `svn` defaults to the unified format (`-u`). Two examples:
     44
    7645{{{
    7746diff -u project-meta.org project-meta.new > project-meta.patch
    … …  
    7948}}}
    8049We apply the patch (after having read and read it again) via the command
     50
    8151{{{
    8252patch -p0 < project-meta.patch