Fairdata IDA Command Line Tools: Uploading and Validation of Data

views

Discover how to upload data efficiently, use the 'validate' action, and ensure the integrity of your files with checksums. More information: https://github.com/CSCfi/ida2-command-line-tools/

Commands and actions used in this tutorial:

UPLOADING AND MODIFYING DATA IN STAGING

1.Show local example data

find ./data | sort

2. Show that no files have yet been uploaded to IDA

ida info / ida info -f /

3. Show use of verbose flags, to display more information

ida info -v / ida info -V /

4. Upload all data to staging

ida upload -v /data ./data

5. Show that data has been uploaded to staging

ida info / ida info /data ida info /data/Experiment_1 ida info /data/Experiment_1/run01.dat

6. Show that renaming, moving, and copying files in staging is allowed

ida move /data/Experiment_1/run01.dat /data/Experiment_1/run01.dat.old ida copy /data/Experiment_1/run02.dat /data/Experiment_1/run02.dat.old

7. Show file changes

ida info /data/Experiment_1

VALIDATION AND RESTORATION OF DATA

8. Show missing file that was renamed

ida validate -v /data ./data

9. Show only one file uploaded and all existing files skipped

ida upload -v /data ./data

10. Show all files in staging OK

ida validate -v /data ./data

11. Show that the moved and copied files still exist and ignored by validation since not local

ida info /data/Experiment_1

12. Simulate corrupted file

ida upload -v -F /data/Experiment_2/run02.dat /dev/null

13. Show invalid file reported

ida validate /data ./data

14. Show invalid file skipped and warning of size mismatch

ida upload -v /data ./data

15. Replace invalid file with new copy

ida upload -v -F /data/Experiment_2/run02.dat ./data/Experiment_2/run02.dat

16. Show all files in staging now OK

ida validate -v /data ./data