Changes between Version 54 and Version 55 of ProjectDocumentation


Ignore:
Timestamp:
Jun 16, 2018, 10:59:02 PM (6 years ago)
Author:
sommeria
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • ProjectDocumentation

    v54 v55  
    108108== EXP02 ==
    109109== EXP03 ==
    110 
    111110= 8 - Data management = #Datamanagement
    112111We can distinguish three steps:
    113  * Raw data: as given by the different instruments. They are generally stored in local disks, instrument units (volts, pixels for images...). The available formats are often limited and proprietary to the company selling the instrument. For long data sets, avoid text formats which are longer to read and occupy more disk space than binary formats.
    114  * Processed data: those are the physical quantities of interest, obtained after calibration of the raw data and processing linked to the instruments, leading for instance to velocity fields form the images by Particle Imaging Velocimetry. They should be understandable by researchers who did not participate in the project, so that physical units and standard formats are needed. They are stored in storage bays with backup system, as local instrument disks are quickly full and less safe. 
    115  * Published data: they contain a selection of the most "interesting" data, and involve various analysis of the processed data, like statistics, plots, which are very dependent on the project. We consider two levels of published data.
    116    * Publication on the OpenDAP server of LEGI: this allows for flexible modification and online consultation, see [https://servforge.legi.grenoble-inp.fr/projects/soft-trokata/wiki/SoftWare/ProjectMeta] for details.
    117    * Publication as a Zenodo archive: its advantage is the perennity of the storage and its support by European authorities. However, the data of a project are limited in size (50 Go), and can be only downloaded as a .zip archive, without online consultation. It is appropriate as a final step after the OpenDAP. [[BR]]
     112
     113 * '''Raw data''': as given by the different instruments. They are generally stored in local disks, using instrument units (volts, pixels for images...). The available formats are often proprietary and limited by the constraints of fast disk writting. For long data sets, avoid text formats which are longer to read and occupy more disk space than binary formats.
     114 * '''Processed data''': those are the physical quantities of interest, obtained after calibration of the raw data and processing linked to the instruments, leading for instance to velocity fields form the images by Particle Imaging Velocimetry. They should be understandable by researchers who did not participate in the project, so that physical units and standard formats are needed. They are stored in storage bays with backup system, as local instrument disks are quickly full and less safe.
     115 * '''Published data''': they contain a selection of the most "interesting" data, and involve various analysis of the processed data, like statistics, plots, which are very dependent on the project. We consider two levels of published data.
     116   * Publication on the '''OpenDAP''' server of LEGI: this allows for flexible modification and online consultation, see [https://servforge.legi.grenoble-inp.fr/projects/soft-trokata/wiki/SoftWare/ProjectMeta] for details.
     117   * Publication as a '''Zenodo''' archive ([https://zenodo.org]): its advantage is the perennity of the storage and its support by European authorities. However, the data of a project are limited in size (50 Go), and can be only downloaded as a .zip archive, without online consultation. It is appropriate as a final step after the OpenDAP.
    118118
    119119= 9 - Data format =
     120== Images ==
     121 We use different imaging systems that provide their own proprietary image format.Successive frames, and even frames from different cameras,can be packed together in a single file for faster disk writting. Therefore an operation of extraction is often done, to provide a set of properly indexed images in a standard format, png(portable network graphics). It is a binary format for images with lossless (reversible) compression (like .zip) recommended by w3c (http://www.w3.org/Graphics/PNG). It is an open source patent-free replacement of GIF. It can be read directly by all standard programs of image visualisation and processing. Compressing a raw binary image to its png form typically saves disk storage by a factor of 3.
     122
     123== Data from instruments ==
     124Like imaging systems, instruments provide various proprietary formats. Text formats are often used for interoperability and human reading. However automatic reading of text files is slow and it is often inpaired by pecularities for text heading and data separators. Text files are also not efficient in terms of disk storage.