Feature #5464
openHelios integration with Appion pipeline
0%
Description
Using the DSynchronize program installed on the Helios Microscope PC, we are able to synchronize the "TiffImages" folder with the mounted data drive on the support PC. I will install Python2.7 on the Microsope and Support PC's in the near future so we can write a more sophisticated transfer script.
NOTE: Using the auto-synchronize utility in DSynchronize, if you delete a file from the monitored Microscope directory, it will delete it from the monitored Support PC directory. Be careful!
Files
Updated by Carl Negro over 6 years ago
I installed Python2.7 on both the Helios microscope and support PC's. There is a file monitoring library called watchdog (https://pypi.python.org/pypi/watchdog) that I also installed with pip, that we may be able to use to find changes in the microscope data storage directories.
https://www.michaelcho.me/article/using-pythons-watchdog-to-monitor-changes-to-a-directory
I moved the following packages over to the microscope PC with a USB drive and installed them individually:
argh==0.26.2
pathtools==0.1.2
PyYAML==3.12
watchdog==0.8.3
Updated by Sargis Dallakyan over 6 years ago
- Related to Feature #4973: Upload Helios images to appion added
Updated by Sargis Dallakyan over 6 years ago
- File uploadSEM.sh uploadSEM.sh added
- File SEM_upload.py SEM_upload.py added
Created 2 new scripts on node01 for automatically uploading Helios images from support PC. uploadSEM.sh finds upload.cfg files in /mnt/SharedData (mounted from support PC) modified within last 30 days. It then calls SEM_upload.py that parses upload.cfg, uploads images and stores session name. I've just runs uploadSEM.sh, it has found /mnt/SharedData/vgarcia/vgarcia_support20180105T170429/TiffImages/upload.cfg that Ashleigh created and uploaded images to 18feb13r13 session.
We can discuss how often to call this script. If we run it too often (i.e. more than one time per hour), it would use hard drive on support PC more often which can lead to a premature failure of the drive on support PC. Maybe we can call it once an hour or once a day.
Updated by Bridget Carragher over 6 years ago
Great to see this moving forward!
If we need to limit the number of times this launches it would be good to be able to have Ash or Bill or whoever is collecting the data force a push on demand so that they can tell a remote end user to look at the data.
Updated by Sargis Dallakyan over 6 years ago
Added cron job on node01 to run uploadSEM.sh every hour:0 * * * * flock -n /root/.upload /root/bin/uploadSEM.sh
Updated by Sargis Dallakyan over 6 years ago
Added a new entry InstrumentData table called SEM. Also, created corresponding entry in ScopeEMData table. Committed commit:d39fa796. Starting tomorrow all images uploaded for SEM will have this ScopeEMData and can be easily found in image viewer.