SOLVED: ~/appion/imageloader.py is throwing errors
Added by Patrick Goetz almost 7 years ago
First, a note about our system configuration. For various reasons we ended up investing in a very beefy 4 GPU rackmount system with 700TB+ of storage which will be used for motion correction, myamiweb, and image and database storage. The appion software is installed on this machine, but will be run from peripheral workstations. In order to simplify software installation and upgrades, I installed all the appion stuff in /usr/local/bin on this machine, and the /usr/local directory is then NFS mounted out to the workstations on /local.
We started testing running the software on some existing images from one of these workstations and ran into error messages. I assumed that this was because some python modules were installed as packages from the distro, and consequently got installed in /usr/lib/python instead of /usr/local/lib/python, but as a reality check, we tried running imageloader.py on the system which has everything installed, and still got error messages:
taylor@kraken:/data2$ imageloader.py --projectid=1 --session=17nov28z --cs=2 --batchparams=/data2/appion/MR+N_C8_0001.mrc --description="test" --jobtype=uploadimage ... Time stamp: 17nov28q31 ... Function name: imageloader ... Appion directory: /usr/local/lib/python2.7/dist-packages ... Processing hostname: kraken ... Using split database Traceback (most recent call last): File "/usr/local/bin/appion/imageloader.py", line 783, in <module> imgLoop = ImageLoader() File "/usr/local/bin/appion/imageloader.py", line 36, in __init__ appionLoop2.AppionLoop.__init__(self) File "/usr/local/lib/python2.7/dist-packages/appionlib/appionLoop2.py", line 27, in __init__ appionScript.AppionScript.__init__(self) File "/usr/local/lib/python2.7/dist-packages/appionlib/appionScript.py", line 75, in __init__ newdbname = apProject.getAppionDBFromProjectId(self.params['projectid']) File "/usr/local/lib/python2.7/dist-packages/appionlib/apProject.py", line 93, in getAppionDBFromProjectId apDisplay.printError("could not find appion db name for project %d "%(projectid)) File "/usr/local/lib/python2.7/dist-packages/appionlib/apDisplay.py", line 65, in printError raise Exception, colorString("\n *** FATAL ERROR ***\n"+text+"\n\a","red") Exception: *** FATAL ERROR *** could not find appion db name for project 1 Exception AttributeError: "'NoneType' object has no attribute 'startswith'" in <bound method ImageLoader.__del__ of <__main__.ImageLoader object at 0x7f4cc48a2710>> ignored taylor@kraken:/data2$
This is of less importance, but notice the
*** FATAL ERROR *** could not find appion db name for project 1
The users told me they often got this error message when running the software on their Berkeley systems, but it didn't interfere with their work (so this must be an unrelated problem?). Regarding just this issue, I'm assuming there's some step where the image must be associated with a project. I checked the mysql projectdb database and the table which seemed most likely to store such information was empty. Is this a user error with the use of myamiweb?
Assuming this issue is resolved, my current plan is to re-install python 2.7.14 (+ all necessary python modules) in /usr/local on the main machine so that the entire python environment is self-contained. This way they can set up a workstation which only has python3 installed and will still be able to use appion, assuming some environment variables are properly set. Please let me know if this is a bad idea, for some reason.
Replies (6)
RE: ~/appion/imageloader.py is throwing errors - Added by Patrick Goetz almost 7 years ago
Just reporting that I upgraded to the latest myami-beta; moved the appion executables to /usr/local/bin, and am getting the same error messages. It seems to be looking for a project ID that didn't get created:
cnsit@kraken:~$ imageloader.py --projectid=1 --session=17nov28z --cs=2 --batchparams=/data2/appion/MR+N_C8_0001.mrc --description="test" --jobtype=uploadimage ... Time stamp: 17nov29q08 ... Function name: imageloader ... Appion directory: /usr/local/lib/python2.7/dist-packages ... Processing hostname: kraken ... Using split database Traceback (most recent call last): File "/usr/local/bin/imageloader.py", line 5, in <module> app = apImageUpload.ImageLoader() File "/usr/local/lib/python2.7/dist-packages/appionlib/apImageUpload.py", line 36, in __init__ appionLoop2.AppionLoop.__init__(self) File "/usr/local/lib/python2.7/dist-packages/appionlib/appionLoop2.py", line 28, in __init__ appionScript.AppionScript.__init__(self) File "/usr/local/lib/python2.7/dist-packages/appionlib/appionScript.py", line 75, in __init__ newdbname = apProject.getAppionDBFromProjectId(self.params['projectid']) File "/usr/local/lib/python2.7/dist-packages/appionlib/apProject.py", line 93, in getAppionDBFromProjectId apDisplay.printError("could not find appion db name for project %d "%(projectid)) File "/usr/local/lib/python2.7/dist-packages/appionlib/apDisplay.py", line 65, in printError raise Exception, colorString("\n *** FATAL ERROR ***\n"+text+"\n\a","red") Exception: *** FATAL ERROR *** could not find appion db name for project 1 Exception AttributeError: "'NoneType' object has no attribute 'startswith'" in <bound method ImageLoader.__del__ of <appionlib.apImageUpload.ImageLoader object at 0x7f34b0262e10>> ignored
RE: ~/appion/imageloader.py is throwing errors - Added by Anchi Cheng almost 7 years ago
Your specific error is:
could not find appion db name for project
Each project is assigned with a processing database and its database name recorded in the projectdb.
It looks like you did not assign one before you start to upload. This means you missed this step:
RE: ~/appion/imageloader.py is throwing errors - Added by Patrick Goetz almost 7 years ago
So I know almost nothing about how to use the software (is there a step by step tutorial?), but does appear that we have set up an initial processing database. When I go to
http://kraken.biosci.utexas.edu/myamiweb/project/project.php
I see the project "DNA Repair Complexes". This indicates that it must be stored in a table somewhere, but which table?
Since there is only one project recorded so far, the --projectid=1 in the imageloader.py command line would presumably be correct.
RE: ~/appion/imageloader.py is throwing errors - Added by Anchi Cheng almost 7 years ago
Patrick,
As indicated in the wiki page, you should see a line showing the linked processing database name if you have created the processing database and linked it with your project. Just seeing the project name in project.php does not mean you have a project processing database.
Since you are more familiar with SQL, here is the explanation in database terms: A project is recorded in projectdb.projects table as an entry. A processingdb is recorded in projectdb.processingdb table which has a field that reference the project entry in projectdb.projects.
RE: ~/appion/imageloader.py is throwing errors - Added by Patrick Goetz almost 7 years ago
Thank you. that solved that problem. Now imageloader.py is writing rawdata output to the leginon folder, but is still terminating with a (new) fatal error:
*** FATAL ERROR *** Bad batch file parameters
It would probably be helpful to have some text on this page:
~myamiweb/project/project.php
Telling people they need to click on the project link in order to create a project database. One would assume that such a project database would be created automatically, which makes it kind of confusing.
Should I mark this issue as solved and log the FATAL ERROR above as a new problem ticket?
RE: SOLVED: ~/appion/imageloader.py is throwing errors - Added by Patrick Goetz almost 7 years ago
Turns out this issue was 100% user error and has been resolved. We were filling in some fields on the myamiweb page incorrectly. imageloader.py works as advertised. On to the next problem.