Task #3343
openAdd protomo and opencv-python to centosAutoInstallation.py
0%
Description
Will need to modify auto-installation tool to include protomo and opencv-python for the next release (3.2).
Files
Updated by Sargis Dallakyan over 9 years ago
r19003 adds opencv-python. centosAutoInstallation.py exits with an error:
Could not connect: Access denied for user 'root'@'localhost' (using password: NO)
Working on fixing this...
Updated by Sargis Dallakyan about 9 years ago
Adding the list of packages to be included in centosAutoInstallation.py
- Protomo 2.4.1
- i3
- Imagemagick
- FFMPEG
- numpy
- scipy
- PIL
- pylab
Updated by Alex Noble about 9 years ago
Hi Sargis,
Can you update us on the progress of getting Protomo auto-installed by default? Thanks, -Alex
Updated by Sargis Dallakyan about 9 years ago
Hi Alex,
I'm committed r19105 that makes it possible for auto-installer to create databases and tables without root password for mysql. The good news that auto-installer now finishes successfully and it opens up working myamiweb and leginon after it's done.
On getting Protomo auto-installed by default, I have 80% of the code written. I hope to finish remaining 20% and commit it before the next Appion hangout.
Updated by Sargis Dallakyan about 9 years ago
r19121 implements Protomo installer. I've tested this on one of our servers (fly) and it works as expected.
There are a couple of things related to opencv and mysql, that I can discuss with Anchi before closing this issue.
- opencv.x86_64 in the current CentOS 6 distro is version 2.0.0-12.el6 and it's called cv instead of cv2. Can we change openCVcaller.py#L18 as follows and see if this works?
try: import cv2 except: import cv as cv2
- mysqld_safe --skip-grant-tables & in the current centosAutoInstallation.py works fine and it allows creating databases and tables without root password, however, it won't survive a reboot. I'm thinking to change auto-installation tool wiki page and ask users to run mysql_secure_installation to set mysql root password and then update their sinedon and myamiweb config files.
Updated by Anchi Cheng about 9 years ago
- Status changed from New to Assigned
Carl tested the autoinstallor and then handed to me to testrun.
I got the following error while running a mock protomo2aligner.py :
sh: i3cut: command not found ... Alignment Correlation Peak Images and/or Videos could not be generated. Make sure i3, ffmpeg, and imagemagick are in your $PATH. Make sure that pyami and scipy are in your $PYTHONPATH.
Also got an error from the same script on FFTmpeg
FFmpeg version 0.6.5, Copyright (c) 2000-2010 the FFmpeg developers built on Jan 29 2012 17:52:15 with gcc 4.4.5 20110214 (Red Hat 4.4.5-6) configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --incdir=/usr/include --disable-avisynth --extra-cflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' --enable-avfilter --enable-avfilter-lavf --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libx264 --enable-gpl --enable-nonfree --enable-postproc --enable-pthreads --enable-shared --enable-swscale --enable-vdpau --enable-version3 --enable-x11grab libavutil 50.15. 1 / 50.15. 1 libavcodec 52.72. 2 / 52.72. 2 libavformat 52.64. 2 / 52.64. 2 libavdevice 52. 2. 0 / 52. 2. 0 libavfilter 1.19. 0 / 1.19. 0 libswscale 0.11. 0 / 0.11. 0 libpostproc 51. 2. 0 / 51. 2. 0 Unrecognized option 'framerate'
Updated by Anchi Cheng about 9 years ago
- Priority changed from Normal to Urgent
elevate it to urgent since we need to release 3.2
Updated by Alex Noble about 9 years ago
If i3cut isn't found, then i3 isn't installed or isn't in the path.
Here's my FFmpeg configuration, which works without issue:
configuration: --enable-gpl --enable-version3 --disable-shared --disable-debug --enable-runtime-cpudetect --enable-libmp3lame --enable-libx264 --enable-libx265 --enable-libwebp --enable-libspeex --enable-libvorbis --enable-libvpx --enable-libfreetype --enable-fontconfig --enable-libxvid --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-gray --enable-libopenjpeg --enable-libopus --enable-libass --enable-gnutls --enable-libvidstab --cc=gcc-4.9
I'm using a 2015 ffmpeg version N-45094-g1932f7e-
Updated by Sargis Dallakyan about 9 years ago
- Status changed from Assigned to Closed
- Assignee changed from Sargis Dallakyan to Anchi Cheng
r19286 fixes i3cut and other issues. I use repoforge to install ffmpeg. Perhaps there are other repos that have FFmpeg compiled with the options Alex wrote, but for now this is good for shipping. Docker containers would make it easier to compile and distribute third party packages with the options we need.
Closing this issue for now; we can revisit this in the next release, if needed.
Updated by Carl Negro about 9 years ago
- Status changed from Closed to Assigned
- Assignee changed from Anchi Cheng to Alex Noble
Alex,
Can you provide installation instructions for your working FFmpeg version? The autoinstaller currently uses the FFmpeg from the CentOS yum repository, which still causes the same framerate error Anchi mentioned as well as
i3cut: could not load libi3tiffio.so, TiffioModule disabledduring protomo2align.py.
Updated by Alex Noble about 9 years ago
The i3cut error is not a problem. i3 has a list of image libraries that it tries to load in a particular order; if the first image library isn't present then it goes to the next one. That's all that error signifies - it found the next one and continued working.
It's been many months since I installed the ffmpeg I use so I don't remember what I did exactly... I'll install it again on my end and report back.
Updated by Alex Noble about 9 years ago
I use the static git version of ffmpeg:
http://johnvansickle.com/ffmpeg/
I just tried the latest 64bit build and it runs perfectly.
Updated by Anchi Cheng about 9 years ago
Alex,
When protomo2align.py what files should we have and how big relative to the images in raw directory ? Our test run, apart from not getting media files had small files in croase_align and recon. I also only used 12 tilts (1 deg increment centered at 0). would that be a problem ?
Updated by Alex Noble about 9 years ago
The cache directory should be as large as the raw directory.
It shouldn't matter how many tilts you work with as long as it's more than maybe 3 or 5, I'm not sure.
Here is the complete terminal output for a Coarse Alignment run with all options as default (except I set frame_aligned=True): http://pastie.org/private/6hjmylrcctsuo8rezx7tya
Here is the corresponding log file: http://pastie.org/private/cd9ajzxxpw7vvy99szejq
Updated by Anchi Cheng about 9 years ago
Looks like the output size is reasonable with our small tilt series. Now we are only waiting for a good fftmpeg installation through autoinstaller.
Updated by Carl Negro about 9 years ago
I rewrote the i3 paths in the autoinstaller so that protomo2aligner.py finishes, but throws the 'i3cut: could not load libi3tiffio.so, TiffioModule disabled'. Still working on this
Updated by Alex Noble about 9 years ago
- File series000108_cor.img series000108_cor.img added
If you can take this attached .img file and run the following commands:
i3cut -fmt mrc series000108_cor.img series000108_cor.mrc
proc3d series000108_cor.mrc series000108_cor.mrc norm
3dmod series000108_cor.mrc
and you can see something, then nothing needs to be fixed. Even we get the "i3cut: could not load libi3tiffio.so, TiffioModule disabled" warning and we're using a Hanspeter installation!
Updated by Carl Negro almost 9 years ago
Alex,
Here's the console output from your instructions:
[root@vagrant-centos65 local]# wget http://emg.nysbc.org/attachments/download/4677/series000108_cor.img
--2015-11-14 16:31:41-- http://emg.nysbc.org/attachments/download/4677/series000108_cor.img
Resolving emg.nysbc.org... 172.85.57.157
Connecting to emg.nysbc.org|172.85.57.157|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: “series000108_cor.img.1”
[ <=> ] 3,788,920 1.47M/s in 2.5s
2015-11-14 16:31:44 (1.47 MB/s) - “series000108_cor.img.1” saved [3788920]
[root@vagrant-centos65 local]# i3cut -fmt mrc series000108_cor.img series000108_cor.mrc
i3cut: could not load libi3tiffio.so, TiffioModule disabled
[root@vagrant-centos65 local]# proc3d series000108_cor.mrc series000108_cor.mrc norm
EMAN 1.9 Cluster ($Date: 2009/02/18 05:12:22 $).
Run 'proc3d help' for detailed help.
Original image : 160x160x37 Mean=3.2e-05 Sigma=0.00427 Min=-0.0205 Max=0.0394
Sampling : 1 A/pixel Origin = 0,0,0
Final image : 160x160x37 Mean=4.38e-09 Sigma=1
Sampling : 1 A/pixel Origin = 0,0,0
[root@vagrant-centos65 local]# 3dmod series000108_cor.mrc
-bash: 3dmod: command not found
[root@vagrant-centos65 local]#
Is this the output you were anticipating?
Also, could you post the contents of 'echo $PATH', 'echo $PYTHONPATH', and 'echo $LD_LIBRARY_PATH' on your machine? Thanks
Updated by Alex Noble almost 9 years ago
Yes it works fine, you just don't have 3dmod (ie. IMOD) installed.
i3cut -fmt mrc input.img output.mrc just takes the Protomo-specific img file format and converts it to mrc. proc3d then normalizes it so that you can open it up and see it in 3dmod: http://bio3d.colorado.edu/imod/doc/3dmodguide.html#ZapWindow
But you don't have 3dmod installed (which is fine). Open it up in any EM image viewer that you prefer to convince yourself.
Updated by Alex Noble almost 9 years ago
bash-4.2$ echo $PATH
/panfs/storage.local/imb/home/ajn10d/bin/screen-4.2.1:/panfs/storage.local/imb/home/ajn10d/bin/automatch_v0.3.1_withdata/bin:/panfs/storage.local/imb/home/ajn10d/bin/Komodo-Edit-8.5.4/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/bin:/panfs/storage.local/imb/stagg/software/myami_python/bin:/panfs/storage.local/imb/home/ajn10d/myami3/redux/bin:/panfs/storage.local/imb/home/ajn10d/myami3/appion/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/bin:/panfs/storage.local/imb/home/ajn10d/bin/ffmpeg-git-20150311-64bit-static:/panfs/storage.local/imb/home/sstagg/Komodo-Edit-8/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/spider18.15/spider/bin/:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/simple_linux_120521/apps:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/simple_linux_120521/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/relion-1.3/bin:/panfs/storage.local/imb/scratch/software/i3-0.9.6/bin:/panfs/storage.local/imb/scratch/software/i3-0.9.6/bin/linux/x86-64:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/bin:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/bin/linux/x86-64:/panfs/storage.local/imb/home/sstagg/hsource_rss_20100325:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/grigorieff/frealign_v9.11/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/bin:/opt/hpc/gnu/xalt/bin:/panfs/storage.local/opt/hpc/gnu/openmpi/bin:/opt/hpc/gnu/openmpi/bin:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/opt/hpc/gnu:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/matlab/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/matlab/src:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/cuda/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/mpi:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/examples:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/doc
bash-4.2$ echo $PYTHONPATH
/panfs/storage.local/imb/home/ajn10d/myami3/appion:/panfs/storage.local/imb/home/ajn10d/myami3/leginon:/panfs/storage.local/imb/home/ajn10d/myami3:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/lib:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/lib/linux/x86-64:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/lib:/opt/hpc/gnu/xalt/site:/opt/hpc/gnu/xalt/libexec:/usr/lib64/python2.6/site-packages/openmpi
bash-4.2$ echo $LD_LIBRARY_PATH
/usr/local/cuda-7.0/lib64:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/relion-1.3/lib:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/lib/linux/x86-64:/panfs/storage.local/imb/scratch/software/lib/linux/x86-64:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/lib:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/lib:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/lib:/panfs/storage.local/opt/hpc/gnu/openmpi/lib64:/opt/hpc/gnu/openmpi/lib64:/usr/lib64/openmpi/lib:/panfs/storage.local/opt/hpc/gnu/lib64:/opt/hpc/gnu/lib64:/opt/hpc/gnu/lib64/netcdf-3:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/runtime/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/bin/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/os/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64/server:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64
Updated by Sargis Dallakyan almost 9 years ago
r19387 merges protomo related r19374 through r19383 from trunk to 3.2 brunch.