Leginon

Google numpytest, and you will find the answer. numpy and scipy need to be the compatible version. Since you had to upgrade one for the scipy bug, you need the other one to match. Don't try to change inside scipy yourself, there may be other compatiability bugs down the road.

What are the compatible versions? The latest version of numpy and scipy?

So, in trying to get around this ctffind problem, I updated scipy and numpy. Unfortunatley, this has completely halted all appion-based scripting and leginon. Now, when I try to run leginon:

$ start-leginon.py 
Traceback (most recent call last):
  File "/usr/bin/start-leginon.py", line 9, in ?
    from leginon import start
  File "/usr/lib/python2.4/site-packages/leginon/start.py", line 13, in ?
    import leginon.gui.wx.Manager
  File "/usr/lib/python2.4/site-packages/leginon/gui/wx/Manager.py", line 20, in ?
    from leginon import manager
  File "/usr/lib/python2.4/site-packages/leginon/manager.py", line 19, in ?
    import launcher
  File "/usr/lib/python2.4/site-packages/leginon/launcher.py", line 13, in ?
    import node
  File "/usr/lib/python2.4/site-packages/leginon/node.py", line 26, in ?
    import correctorclient
  File "/usr/lib/python2.4/site-packages/leginon/correctorclient.py", line 13, in ?
    import scipy.ndimage
  File "/usr/lib64/python2.4/site-packages/scipy/ndimage/__init__.py", line 39, in ?
    from numpy.testing import NumpyTest
ImportError: cannot import name NumpyTest

Both versions of scipy and numpy were installed through:

yum install python-scipy python-numpy

Giving numpy version 1.6.1 and scipy version 0.6.0.

All of this is on redhat running centos 5.

Considering that I managed to hinder appion/leginon, I'd appreciate any help to get around this problem.

Thanks,
mike

Is this on the same machine that you had to update scipy above 0.9? This means the installation in python site-package might not be the same as that is installed by yum now. Do this simple test to find out where things come from:

In python command line

import numpy
numpy.__version__
numpy

import scipy
scipy.__version__
scipy

If these run (which might not be, because scipy will want to do numpy test just like what you showed), numpy.__version__ will give you the actual version that python loaded, and the same for scipy.

We have a few combination that we have and work:
numpy 1.4.1 - scipy 0.7.2 (CentOS 6)
numpy 1.0.4 - scipy 0.6.0 (CentOS 5)
numpy 1.6.2 - scipy 0.10.1 (Ubumtu 12.10)

I found in scipy release notes that scipy 1.10.0 needs numpy 1.5 or above
http://docs.scipy.org/doc/scipy/reference/release.0.10.0.html

Also that scipy 0.7.0 starts supporting numpy 1.2 which suggests that your combination of numpy 1.6.1 with scipy 0.6.0 would not have worked.
http://docs.scipy.org/doc/scipy/reference/release.0.7.0.html

See if an yum update of scipy would help you if your installation is indeed the old 0.6.0

Jim also suggests that you do yum uninstall before reinstall to clean up the installation.

Hi Anchi,

Thank you for your response. I appreciate your help!

This is what I get when I do the commands that you suggested:

import numpy
numpy.__version__

'1.6.1'

numpy

<module 'numpy' from '/usr/lib64/python2.4/site-packages/numpy/__init__.pyc'>

import scipy
scipy.__version__

'0.6.0'

scipy

<module 'scipy' from '/usr/lib64/python2.4/site-packages/scipy/__init__.pyc'>

scipy.test()

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib64/python2.4/site-packages/scipy/__init__.py", line 74, in test
from numpy.testing import NumpyTest
ImportError: cannot import name NumpyTest

After doing this, I uninstalled python-scipy and python-numpy using 'yum remove' and then reinstalled both packages. This gave me the same versions and same errors.

Then, given that you were able to successfully run numpy 1.0.4 and scipy 0.6.0, I went and downloaded the tarball for numpy 1.0.4 to go with my scipy 0.6.0. After a seemingly successful manual installation (no obvious error messages), I now get a different error when I try to import numpy OR scipy:

>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib64/python2.4/site-packages/numpy/__init__.py", line 137, in ?
    import add_newdocs
  File "/usr/lib64/python2.4/site-packages/numpy/add_newdocs.py", line 9, in ?
    from numpy.lib import add_newdoc
  File "/usr/lib64/python2.4/site-packages/numpy/lib/__init__.py", line 4, in ?
    from type_check import *
  File "/usr/lib64/python2.4/site-packages/numpy/lib/type_check.py", line 8, in ?
    import numpy.core.numeric as _nx
  File "/usr/lib64/python2.4/site-packages/numpy/core/__init__.py", line 8, in ?
    import numerictypes as nt
  File "/usr/lib64/python2.4/site-packages/numpy/core/numerictypes.py", line 429, in ?
    _set_up_aliases()
  File "/usr/lib64/python2.4/site-packages/numpy/core/numerictypes.py", line 412, in _set_up_aliases
    allTypes[alias] = allTypes[t]
KeyError: 'timedelta'

Have you seen this error before? I'm searching the web for other instances of this error but I'm not finding much that are helpful.

Apologies for the double posting in a row. I just realized that I should've unstalled numpy before re-installing it manually. When I do this, I got this error:

>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/lib64/python2.4/site-packages/numpy/__init__.py", line 39, in ?
    import core
  File "/usr/lib64/python2.4/site-packages/numpy/core/__init__.py", line 13, in ?
    from defmatrix import *
  File "/usr/lib64/python2.4/site-packages/numpy/core/defmatrix.py", line 6, in ?
    from numerictypes import issubdtype
ImportError: cannot import name issubdtype

Any thoughts on what is causing this error?

Ok! I fixed the problem. It came down to selecting the correct version AND building --> installing numpy version 1.0.4 to go along with scipy 0.6.0.

The commands I used to install numpy and scipy on redhat/centos v. 5 (just to put these into redmine).

Download the tarball for numpy version 1.0.4: http://sourceforge.net/projects/numpy/files/

Then:

$ tar xfz numpy-1.0.4.tar.gz
$ cd numpy-1.0.4
$ BLAS=None LAPACK=None ATLAS=None python setup.py build
$ python setup.py install

For scipy:

$ sudo yum install python-scipy

Which installs scipy version 0.6.0.

Note, this setup for numpy and scipy doesn't work with ctffind.

Mike,

Glad it is half solved.

As for the setup doesn't work with ctffind, could you be more specific? You may have mentioned before, then just point me to the right place.

I just tried it, if I am using version 2.2, which is what you need to use on CentOS 5, Appion should work with ctffind. If I run the trunk version,
which you should use with CentOS 6, the new Appion dependency of matlabplot does not work with this early version of numpy.

Hi Anchi,

Since there were members of this lab trying to develop spot-scanning approaches within leginon (Brandy Pappas) in the past, we are running on the trunk version of appion. However, now that no one is developing, I would like to switch to the branch. This would likely fix the ctffind problem, as you just showed.

Considering that I don't want to put my lab members through more appion/leginon downtime, do you think it should be a seamless transition from the trunk to the branch? I'm still getting a feeling for appion, so I just want to make sure I don't really mess things up (if possible).

A few basic questions:
1) If I download the branch version to my workstation, this will NOT affect my current appion installation, correct? It will only affect my current appion installation if I copy over all of these new files into the folder containing the current appion directory.

2) Can I run the website/database on the trunk but call commands from the branch? This would leave the the web interface as-is, running out of the trunk appion directory, but all of the appion commands would come from the bin directory for the appion branch.

3) Or, should I just bite the bullet and switch the branch?

Thank you again for all of your help!
michael

Answers to your questions:
(1) I believe you are talking about downloading in your personal directory (sandbox) not the group installation that reside on on python's site-package. If your PATH, PYTHONPATH put your sandbox ahead of the default while everyone else uses the default, you certainly do not affect them in the running from your sandbox. However, if you all write into the same database, you do affect them in that sense. That said, if you work on different projects, the conflict is much smaller than if you work on the same project.

(2) myamiweb is the gui for the commands in Appion, therefore, they do need to match in operation. Leginon has less of trunk/branch conflict but it can still be a problem if a major change has been made. Could you find out how recent has the trunk version the group uses being updated? if you can get the svn revision number, it would be most useful. From that, I can tell you if you switch to 2.2 branch, which is more than half-year old would cause problem to the others.

Anchi

Hi Anchi,

The revision number for appion/trunk is 17420. We just updated it a few weeks ago.

Thanks for looking into this.
Mike

MIchael,

This is unusual. trunk version 17420 has a lot of new Leginon/Appion changes that 2.2 branch does not have. We have not attempt to backward converting them. One particular problem is the defocus convention change I am sure you are aware of.

Can you tell me how the package is used so far? If it is limited to Leginon, it might be easier.

Also, I will need to know the setup better. For example, Is the webserver run on CentOS 6 with the same trunk revision? The revision you said is updated would not have worked with CentOS 5 on the webserver side. Maybe your webserver myamiweb is on an older revision that is still compatible with 2.2 branch. Same conflict would be true with the Leginon/pyscope installed on the microscope. Therefore, please give me the version of Leginon on the microscope computer as well if it is not always updated with the others.

Your idea of switching to branch is a very good one for stability especially if a frequent svn update is done. We can not guarantee the stability of the trunk. I hope you are not discouraged by the obstacles. We will help you as much as we can.

Hi Anchi,

The version number I told you for appion is incorrect. I just realized yesterday that the previous lab members who installed appion copied all of the appion scripts into the bin directory. Therefore, I'm not sure what version they came from. I could try to figure it out, but I'm going to leave everything as-is for now since it all works, and I don't want to take it down again.

Instead, I'm going to start my own sandbox. I'll install appion from scratch and, after a successful set up, I'll push the changes onto the rest of the users in lab. This will give me more control and also a working version of ctffind (hopefully).

When/if I have more questions, I'll probably start them in a different thread specific to installation.

Thanks again for all of your help!
mike

Mike,

If you starts your own sandbox, and have control on which OS version you use, I would recommend you take 2.2-redux branch that uses CentOS 6. This will allow you to use the default numpy/scipy that are compatible with each other.

As for the old installation, if the last installator copied the content of the whole folder, the .svn files would still be there so you could still find the svn info. If not, look into leginon/version.py and look for the function getTextVersion. You will at least know if the general use one is on trunk or on branch. The trunk one would say pre-xxx.