Bug #1086
closedFrealign prep bug
0%
Description
When prepping a Frelaign run, it start on guppy and then the last lines of the file
[see : name: frealign_recon2.appionsub.job
user: bcarr
appion path: /ami/data00/appion/10may20c/recon/frealign_recon2
submit time: 2010-12-14 20:24:47
status: Done ]
are:
... Time stamp: 10dec14u24
... Function name: prepFrealign
... Appion directory: /usr/lib64/python2.4/site-packages
... Load average is high 4.24
!!! WARNING: Using split database
Connected to database: 'ap6'
... Committing data to database
... Getting stack data for stackid=1496
Old stack info: 'no ctf'
... Stack 1496 pixel size: 0.83
... Looking up symmetry for input: D7
... Selected symmetry group: D7 -- D7 (z)
... Selected symmetry d7 with id 19
... Run directory: /ami/data00/appion/10may20c/recon/frealign_recon2
!!! WARNING: directory '/ami/data00/appion/10may20c/recon/frealign_recon2'
already exists.
... Writing function log to: prepFrealign.log
... Found 8 processors on this machine
... Running Appion version 'r15117'
... IBLOW set to 1, requiring 8.3 GB memory
... Getting stack data for stackid=1496
Old stack info: 'no ctf'
... requesting 8.3 GB of memory
... Current symmetry: 7-fold symmetry along the z axis, 2-fold rotational axis 90 degrees from z
EMAN: proc3d /ami/data17/appion/09nov02c/models/accepted/09nov02p03/groel-initmodel.mrc
/am ... =320,320,320 scale=1.9639
EMAN 1.9 Cluster ($Date: 2009/02/18 05:12:22 $).
Run 'proc3d help' for detailed help.
Original image : 192x192x192 Mean=0.0659 Sigma=0.248 Min=-1.32e-05 Max=1.68
Sampling : 1.63 A/pixel Origin = 0,0,0
Final image : 320x320x320 Mean=0.108 Sigma=0.309
Sampling : 0.829981 A/pixel Origin = 23.683,23.683,23.683
... Creating parameter file: /ami/data00/appion/10may20c/recon/frealign_recon2/params.iter000.par
... querying stack particles from stackid=1496 at Tue Dec 14 20:25:18 2010
... sorting particles
... received 4800 stack particles in 2.11 sec
... Writing out particle parameters
... particle 200 -- 46.68 sec remain
... particle 400 -- 44.54 sec remain
... particle 600 -- 42.28 sec remain
... particle 800 -- 40.15 sec remain
... particle 1000 -- 38.16 sec remain
... particle 1200 -- 36.18 sec remain
... particle 1400 -- 34.19 sec remain
... particle 1600 -- 32.2 sec remain
... particle 1800 -- 30.2 sec remain
... particle 2000 -- 28.2 sec remain
... particle 2200 -- 26.2 sec remain
... particle 2400 -- 24.19 sec remain
... particle 2600 -- 22.18 sec remain
... particle 2800 -- 20.17 sec remain
... particle 3000 -- 18.16 sec remain
... particle 3200 -- 16.14 sec remain
... particle 3400 -- 14.13 sec remain
... particle 3600 -- 12.12 sec remain
... particle 3800 -- 10.1 sec remain
... particle 4000 -- 8.09 sec remain
... particle 4200 -- 6.06 sec remain
... particle 4400 -- 4.04 sec remain
... particle 4600 -- 2.02 sec remain
... particle 4800 -- 0.0 nsec remain
Initial files:
Stack: start.hed
Volume: initmodel.hed
Params: params.iter000.par
... Multi node run, iteration 1
... Multi node run, iteration 2
... Multi node run, iteration 3
... Multi node run, iteration 4
... Multi node run, iteration 5
... Multi node run, iteration 6
... Multi node run, iteration 7
... Multi node run, iteration 8
... Multi node run, iteration 9
... Multi node run, iteration 10
tar --exclude=*.tar -cf frealign_recon2.tar *
Traceback (most recent call last):
File "/opt/appion/bin/prepFrealign.py", line 830, in ?
frealign.start()
File "/opt/appion/bin/prepFrealign.py", line 824, in start
self.prepareForCluster()
File "/opt/appion/bin/prepFrealign.py", line 776, in prepareForCluster
frealignq['refineIter'] = appiondata.ApRefineIterData.direct_query(self.params['reconiterid'])
File "/usr/lib64/python2.4/site-packages/sinedon/data.py", line 403, in direct_query
result = db.direct_query(cls, dbid, **kwargs)
File "/usr/lib64/python2.4/site-packages/sinedon/dbdatakeeper.py", line 66, in direct_query
raise RuntimeError('direct_query should only return a single result')
RuntimeError: direct_query should only return a single result
Files
Updated by Amber Herold almost 14 years ago
- Priority changed from Normal to High
- Target version set to Appion/Leginon 2.2.0
- Deliverable set to 2.2 Bug Reduction
Updated by Neil Voss almost 14 years ago
Looks like there is a duplicate DEF_id entry in the database table ApRefineIterData. You'll need to fix this in phpMyAdmin, but I am more curious as to how this happened.
Updated by Scott Stagg almost 14 years ago
I want to take this opportunity to draw everyone's attention to a prototype I started for an alternative to our current system with prepFrealign. A link to the discussion about this can be found here [[http://emg.nysbc.org/issues/397]] . The problem with prepFrealign is that if you want to change any parameters in the refinement, you have to run prepFrealign again. Moreover there is no way to change params from one iteration to the next. I wrote a program called runFrealign.py that takes over the role of job creation in prepFrealign. Currently you still have to run prepFrealign one time to get the particle parameters in Frealign format, but then you launch runFrealign to do the refinement. I am anxious to get feedback on this program, because if people like it, I will modify prepFrealign and uploadFrealign to work with runFrealign.
Updated by Amber Herold almost 14 years ago
- File Screenshot-1.png Screenshot-1.png added
- Assignee set to Amber Herold
Neil, I'm looking at the DB (see screenshot), and there is a warning that UNIQUE and INDEX keys should not both be set for column `REF|ApRefineRunData|refineRun`. I noticed that you made some changes to this table in source:root/trunk/dbschema/schema-r13713.py, perhaps there is a problem there. I'll take a look and see if I can correct it. Let me know if you see an obvious mistake in the schema change file, it might take me a while since I'm not familiar with how the refinement tables should be related. Should REF|ApRefineRunData|refineRun be Unique OR Index? and where did REF|ApRefinementRunData|refine_2 come from?
Updated by Neil Voss almost 14 years ago
ap6 is one of the original databases and it probably came up with some early experimenting. I would just remove the UNIQUE index, because, in general, we don't support that. I do no think this would cause the problem. The '_2' was automatically added by phpMyAdmin because a unique name is required for all indexes.
Updated by Amber Herold almost 14 years ago
Thanks, yeah...looks like newer dbs dont have the Unique REF|ApRefinementRunData|refine_2 key...will remove it.
Updated by Anchi Cheng almost 14 years ago
It looks like the problem is that the parameter 'reconiterid' is None. This can be fixed with a if cause before the direct_query.
Neil, does frealign implementation works without previous refinement, though?
Updated by Neil Voss almost 14 years ago
The None value would also make sense. Maybe we should have sinedon reject None values for direct_queries or at least have a better error message.
I have not tested frealign without EMAN in a long time, but it did work at one point. Dmitry tried it out and ran into some problems and I never got around to testing it.
Updated by Anchi Cheng almost 14 years ago
- Status changed from New to In Code Review
r15155 keeps frealignq['refineIter'] from assigning if self.params['reconiterid'] is None.
r15161 raises specific exception in sinedon ( by Jim).
Unique REF|ApRefinementRunData|refine_2 key in older ap databases do not need to be removed since they are not the problem.
Updated by Amber Herold almost 14 years ago
- Assignee changed from Amber Herold to Jim Pulokas
Jim, will it be clear to the user what 'id must be specified, not None' means if this exception is thrown? Id seems very vague to me but perhaps in context it makes more sense?
It looks like it is a general purpose function so maybe we can't specify which id but maybe we can add something like "A database query has failed in direct_query() due to an invalid input parameter. The 'id' must be specified but is currently set to 'None'."?
That's my only comment, it can go into test if you don't want to change the message. Bridget should be able to test.
Updated by Lauren Fisher over 13 years ago
- Status changed from In Code Review to Duplicate
Updated by Lauren Fisher over 13 years ago
- Status changed from Duplicate to Assigned
Updated by Lauren Fisher over 13 years ago
- Status changed from Assigned to In Code Review
Updated by Amber Herold over 13 years ago
- Status changed from In Code Review to Duplicate