Changes between Version 16 and Version 17 of SoftWare/ProjectMeta


Ignore:
Timestamp:
Jun 22, 2018, 5:50:14 PM (2 years ago)
Author:
g7moreau
Comment:

Update the page

Legend:

Unmodified
Added
Removed
Modified
  • SoftWare/ProjectMeta

    v16 v17  
    11= SoftWare / !ProjectMeta - Meta project for open data management =
     2
    23== Aim ==
    3 Project-Meta is a software to help you to manage your open data, using the protocol OpenDAP. The initiative is supported by the European Commission as part of the project Hydralab+ of the Horizon 2020 programme. This programme  requests that research data are open access, that is providing online access free of charge to the end-user and reusable. Furthermore access must allow the right to copy, distribute, search, link, crawl and mine the data. In addition to these general requests, we aim at achieving the following goals:
     4
     5Project-Meta is a software to help you to manage your open data, using the protocol OpenDAP.
     6The initiative is supported by the European Commission as part of the project Hydralab+ of the Horizon 2020 programme.
     7This programme  requests that research data are open access, that is providing online access free of charge to the end-user and reusable.
     8Furthermore access must allow the right to copy, distribute, search, link, crawl and mine the data.
     9In addition to these general requests, we aim at achieving the following goals:
    410
    511 1. Allow the end user to scan and visualise the data without downloading.
     
    814The attached document [attachment:OpenDAP_GM.pdf] describes the wider motivation of the project.
    915
     16
    1017== OPeNDAP ==
    11 The protocol OPeNDAP (Open-source Project for a Network Data Access Protocol). This includes standards for encapsulating structured data, annotating the data with attributes and adding semantics that describe the data. OPeNDAP is widely used by governmental agencies such as NASA and NOAA to serve satellite, weather and other observed earth science data.
    1218
    13 The protocol is based on http, so that data can be scanned with an ordinary web browser. However added functionality of data visualization is provided by graphics programs (like Matlab, GrADS, Ferret or ncBrowse). Compared to ordinary file transfer protocols (e.g. FTP) a major advantage using OPeNDAP is the ability to retrieve subsets of files, so it is possible to work remotely without downloading whole data files. Although any file format can be use, data are often in HDF or NetCDF formats. The older NetCDF format is limited to arrays of numbers, while HDF provides wider possibilities of data structures (and it contains NetCDF as a particular case). We choose the NetCDF format which is sufficient for most experimental data and can be more easily read with a variety of software.
     19The protocol OPeNDAP (Open-source Project for a Network Data Access Protocol).
     20This includes standards for encapsulating structured data, annotating the data with attributes and adding semantics that describe the data.
     21OPeNDAP is widely used by governmental agencies such as NASA and NOAA to serve satellite, weather and other observed earth science data.
     22
     23The protocol is based on http, so that data can be scanned with an ordinary web browser.
     24However added functionality of data visualization is provided by graphics programs (like Matlab, GrADS, Ferret or ncBrowse).
     25Compared to ordinary file transfer protocols (e.g. FTP) a major advantage using OPeNDAP is the ability to retrieve subsets of files, so it is possible to work remotely without downloading whole data files.
     26Although any file format can be use, data are often in HDF or NetCDF formats.
     27The older NetCDF format is limited to arrays of numbers, while HDF provides wider possibilities of data structures (and it contains NetCDF as a particular case).
     28We choose the NetCDF format which is sufficient for most experimental data and can be more easily read with a variety of software.
     29
    1430
    1531== Description ==
    16 The creation of the OpenDAP repository is done by the script [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/project-meta.html project-meta] using UNIX commands, scripts in Perl and C++.
    1732
    18 The first step is to make a list of the data folders to display. This list needs to be introduced in a text file, complemented by some information about the authors of the work and about related publications. This text file must be put in your current folder with the name `PROJECT-META.yml`. The text structure must follow some simple rules consisting in the YAML format. An example could be found in the Project-Meta repository or online [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
     33The creation of the OpenDAP repository is done
     34by the script [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/project-meta.html project-meta]
     35using UNIX commands, scripts in Perl and C++.
    1936
     37The first step is to make a list of the data folders to display.
     38This list needs to be introduced in a text file,
     39complemented by some information about the authors of the work and about related publications.
     40This text file must be put in your current folder with the name {{{PROJECT-META.yml}}}.
     41The text structure must follow some simple rules consisting in the YAML format.
     42An example could be found in the Project-Meta repository or online
     43[http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
    2044{{{
    2145project-meta help
    2246man project-meta
    2347}}}
    24 == `PROJECT-META.yml` meta file ==
    25 This file is at the core of the procedure. A first task is to list the data folder to publish. For that purpose a good practice is to organise the data and the procedures of analysis such that the final data to publish are contained in folders named with specific extensions. Then search tools can be used to list all the selected folders. For instance the following unix command lists all the folders with extension '`*.mproj*`' and append it to the file `PROJECT-META.yml` (and creates the file if it does not exist yet):
    2648
     49
     50== {{{PROJECT-META.yml}}} meta file ==
     51
     52This file is at the core of the procedure.
     53A first task is to list the data folder to publish.
     54For that purpose a good practice is to organise the data and the procedures of analysis
     55such that the final data to publish are contained in folders named with specific extensions.
     56Then search tools can be used to list all the selected folders.
     57For instance the following unix command lists all the folders with extension {{{*.mproj*}}}
     58and append it to the file {{{PROJECT-META.yml}}} (and creates the file if it does not exist yet):
    2759{{{
    2860find . -name '*.mproj*' -a -type d | sed 's/^/    - /;' >> PROJECT-META.yml
    2961}}}
    30 The `find` command only search the folders with the right extension (recursively)  under the current one (.) and the `sed` command add 4 spaces and the dash at the beginning of each line in order to respect the YAML format.
     62The `find` command only search the folders with the right extension (recursively)  under the current one (.)
     63and the `sed` command add 4 spaces and the dash at the beginning of each line in order to respect the YAML format.
    3164
    32 The YAML file has to be complemented by general information about the authors and the related publications, following the template  [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
     65The YAML file has to be complemented by general information about the authors and the related publications,
     66following the template [http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/PROJECT-META.sample.yml PROJECT-META.sample.yml].
     67
    3368
    3469== Debian package ==
    35 Debian is a GNU/Linux distribution. Debian (and certainly Ubuntu) package for amd64 arch could be download on: http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/download.
     70
     71Debian is a GNU/Linux distribution.
     72Debian (and certainly Ubuntu) package for amd64 arch could be download on: http://servforge.legi.grenoble-inp.fr/pub/soft-trokata/project-meta/download.
    3673
    3774You can then install it with
     
    4279(just replace * with the version you have donwloaded).
    4380
     81
    4482== Software repository ==
    45 All code is under '''free license'''. Scripts in `bash` are under GPL version 3 or later (http://www.gnu.org/licenses/gpl.html), `C++` sources are under GPL version 2 or newer, the `perl` scripts are under the same license as `perl` itself ie the double license GPL and Artistic License (http://dev.perl.org/licenses/artistic.html).
     83
     84All code is under '''free license'''.
     85Scripts in `bash` are under GPL version 3 or later (http://www.gnu.org/licenses/gpl.html),
     86`C++` sources are under GPL version 2 or newer,
     87the `perl` scripts are under the same license as `perl` itself ie the double license GPL and Artistic License (http://dev.perl.org/licenses/artistic.html).
    4688
    4789All sources are available on the LEGI forge: http://servforge.legi.grenoble-inp.fr/svn/soft-trokata/trunk/project-meta
    4890
    49 The sources are managed via subversion (http://subversion.tigris.org/). It is very easy to stay synchronized with these sources
     91The sources are managed via subversion (http://subversion.tigris.org/).
     92It is very easy to stay synchronized with these sources
    5093
    5194 * initial recovery
     
    58101}}}
    59102
    60 It is possible to have access to writing at the forge on reasoned request to [mailto:Gabriel.Moreau(A)legi.grenoble-inp.fr Gabriel Moreau]. For issues of administration time and security, the forge is not writable without permission. For the issues of decentralization of the web, autonomy and non-allegiance to the ambient (and North American) centralism, we use our own forge...
     103It is possible to have access to writing at the forge on reasoned request to [mailto:Gabriel.Moreau(A)legi.grenoble-inp.fr Gabriel Moreau].
     104For issues of administration time and security, the forge is not writable without permission.
     105For the issues of decentralization of the web, autonomy and non-allegiance to the ambient (and North American) centralism, we use our own forge...
    61106
    62 You can propose an email patch of a particular file via the `diff` command. Note that `svn` defaults to the unified format (`-u`). Two examples:
    63 
     107You can propose an email patch of a particular file via the `diff` command.
     108Note that `svn` defaults to the unified format (`-u`).
     109Two examples:
    64110{{{
    65111diff -u project-meta.org project-meta.new > project-meta.patch
     
    67113}}}
    68114We apply the patch (after having read and read it again) via the command
    69 
    70115{{{
    71116patch -p0 < project-meta.patch