Project

General

Profile

SOLVED: ~/appion/imageloader.py is throwing errors

Added by Patrick Goetz almost 7 years ago

First, a note about our system configuration. For various reasons we ended up investing in a very beefy 4 GPU rackmount system with 700TB+ of storage which will be used for motion correction, myamiweb, and image and database storage. The appion software is installed on this machine, but will be run from peripheral workstations. In order to simplify software installation and upgrades, I installed all the appion stuff in /usr/local/bin on this machine, and the /usr/local directory is then NFS mounted out to the workstations on /local.

We started testing running the software on some existing images from one of these workstations and ran into error messages. I assumed that this was because some python modules were installed as packages from the distro, and consequently got installed in /usr/lib/python instead of /usr/local/lib/python, but as a reality check, we tried running imageloader.py on the system which has everything installed, and still got error messages:

taylor@kraken:/data2$  imageloader.py --projectid=1 --session=17nov28z --cs=2 --batchparams=/data2/appion/MR+N_C8_0001.mrc --description="test" --jobtype=uploadimage

 ... Time stamp: 17nov28q31
 ... Function name: imageloader
 ... Appion directory: /usr/local/lib/python2.7/dist-packages
 ... Processing hostname: kraken
 ... Using split database
Traceback (most recent call last):
  File "/usr/local/bin/appion/imageloader.py", line 783, in <module>
    imgLoop = ImageLoader()
  File "/usr/local/bin/appion/imageloader.py", line 36, in __init__
    appionLoop2.AppionLoop.__init__(self)
  File "/usr/local/lib/python2.7/dist-packages/appionlib/appionLoop2.py", line  27, in __init__
    appionScript.AppionScript.__init__(self)
  File "/usr/local/lib/python2.7/dist-packages/appionlib/appionScript.py", line 75, in __init__
    newdbname = apProject.getAppionDBFromProjectId(self.params['projectid'])
  File "/usr/local/lib/python2.7/dist-packages/appionlib/apProject.py", line    93, in getAppionDBFromProjectId
    apDisplay.printError("could not find appion db name for project %d          "%(projectid))
  File "/usr/local/lib/python2.7/dist-packages/appionlib/apDisplay.py", line    65, in printError
    raise Exception, colorString("\n *** FATAL ERROR ***\n"+text+"\n\a","red")
Exception:
 *** FATAL ERROR ***
could not find appion db name for project 1

Exception AttributeError: "'NoneType' object has no attribute 'startswith'" in  <bound method ImageLoader.__del__ of <__main__.ImageLoader object at            0x7f4cc48a2710>> ignored
taylor@kraken:/data2$

This is of less importance, but notice the

 *** FATAL ERROR ***
could not find appion db name for project 1

The users told me they often got this error message when running the software on their Berkeley systems, but it didn't interfere with their work (so this must be an unrelated problem?). Regarding just this issue, I'm assuming there's some step where the image must be associated with a project. I checked the mysql projectdb database and the table which seemed most likely to store such information was empty. Is this a user error with the use of myamiweb?

Assuming this issue is resolved, my current plan is to re-install python 2.7.14 (+ all necessary python modules) in /usr/local on the main machine so that the entire python environment is self-contained. This way they can set up a workstation which only has python3 installed and will still be able to use appion, assuming some environment variables are properly set. Please let me know if this is a bad idea, for some reason.


Replies (6)

RE: ~/appion/imageloader.py is throwing errors - Added by Patrick Goetz almost 7 years ago

Just reporting that I upgraded to the latest myami-beta; moved the appion executables to /usr/local/bin, and am getting the same error messages. It seems to be looking for a project ID that didn't get created:

cnsit@kraken:~$ imageloader.py --projectid=1 --session=17nov28z --cs=2 --batchparams=/data2/appion/MR+N_C8_0001.mrc --description="test" --jobtype=uploadimage

 ... Time stamp: 17nov29q08
 ... Function name: imageloader
 ... Appion directory: /usr/local/lib/python2.7/dist-packages
 ... Processing hostname: kraken
 ... Using split database
Traceback (most recent call last):
  File "/usr/local/bin/imageloader.py", line 5, in <module>
    app = apImageUpload.ImageLoader()
  File "/usr/local/lib/python2.7/dist-packages/appionlib/apImageUpload.py", line 36, in __init__
    appionLoop2.AppionLoop.__init__(self)
  File "/usr/local/lib/python2.7/dist-packages/appionlib/appionLoop2.py", line 28, in __init__
    appionScript.AppionScript.__init__(self)
  File "/usr/local/lib/python2.7/dist-packages/appionlib/appionScript.py", line 75, in __init__
    newdbname = apProject.getAppionDBFromProjectId(self.params['projectid'])
  File "/usr/local/lib/python2.7/dist-packages/appionlib/apProject.py", line 93, in getAppionDBFromProjectId
    apDisplay.printError("could not find appion db name for project %d "%(projectid))
  File "/usr/local/lib/python2.7/dist-packages/appionlib/apDisplay.py", line 65, in printError
    raise Exception, colorString("\n *** FATAL ERROR ***\n"+text+"\n\a","red")
Exception: 
 *** FATAL ERROR ***
could not find appion db name for project 1 

Exception AttributeError: "'NoneType' object has no attribute 'startswith'" in <bound method ImageLoader.__del__ of <appionlib.apImageUpload.ImageLoader object at 0x7f34b0262e10>> ignored

RE: ~/appion/imageloader.py is throwing errors - Added by Anchi Cheng almost 7 years ago

Your specific error is:

could not find appion db name for project

Each project is assigned with a processing database and its database name recorded in the projectdb.

It looks like you did not assign one before you start to upload. This means you missed this step:

Create_a_Processing_Database

RE: ~/appion/imageloader.py is throwing errors - Added by Patrick Goetz almost 7 years ago

So I know almost nothing about how to use the software (is there a step by step tutorial?), but does appear that we have set up an initial processing database. When I go to

http://kraken.biosci.utexas.edu/myamiweb/project/project.php

I see the project "DNA Repair Complexes". This indicates that it must be stored in a table somewhere, but which table?

Since there is only one project recorded so far, the --projectid=1 in the imageloader.py command line would presumably be correct.

RE: ~/appion/imageloader.py is throwing errors - Added by Anchi Cheng almost 7 years ago

Patrick,

As indicated in the wiki page, you should see a line showing the linked processing database name if you have created the processing database and linked it with your project. Just seeing the project name in project.php does not mean you have a project processing database.

Since you are more familiar with SQL, here is the explanation in database terms: A project is recorded in projectdb.projects table as an entry. A processingdb is recorded in projectdb.processingdb table which has a field that reference the project entry in projectdb.projects.

RE: ~/appion/imageloader.py is throwing errors - Added by Patrick Goetz almost 7 years ago

Thank you. that solved that problem. Now imageloader.py is writing rawdata output to the leginon folder, but is still terminating with a (new) fatal error:

 *** FATAL ERROR ***
Bad batch file parameters

It would probably be helpful to have some text on this page:

    ~myamiweb/project/project.php

Telling people they need to click on the project link in order to create a project database. One would assume that such a project database would be created automatically, which makes it kind of confusing.

Should I mark this issue as solved and log the FATAL ERROR above as a new problem ticket?

RE: SOLVED: ~/appion/imageloader.py is throwing errors - Added by Patrick Goetz almost 7 years ago

Turns out this issue was 100% user error and has been resolved. We were filling in some fields on the myamiweb page incorrectly. imageloader.py works as advertised. On to the next problem.

    (1-6/6)