Bug #1155
closedspider stacks
0%
Description
The ability to create a SPIDER-formatted stack in Appion is completely broken, perhaps we should remove this option.
The "runMakeStack2.php" web page doesn't output the correct filename if you select a "SPIDER" stack - it is hard coded to set the output to "start.hed".
This is an easy fix, but what is very difficult is changing makestack2.py to accommodate SPIDER stack creation. I spent some time trying to fix this, and it's a pretty big mess. Many of the functions are hard coded to use "start.hed", and ignore the input filename option (self.params['single']). Many other functions assume an imagic-formatted stack to work properly. The averaging & montage creation simply won't work on a spider stack, for example. I tried changing it to first create the stack in IMAGIC format, and then do a final conversion to a SPIDER stack, but this totally breaks the "continue" option, and there are lots of checks that go on during the stack creation that check the filename in the database, so that having "start.spi" messes it up.
Do we remove the file-format option entirely, or should we spend a significant amount of time reworking the makestack2.py script?
I bring this up because I'm thinking about incorporating an EMAN2/SPARX reconstruction script into Appion, which requires an hdf-formatted file as input. I wanted to add "hdf" as another output format option in makestack, but maybe we only create imagic stacks, and always deal with conversion afterwards? This becomes problematic when we're dealing with stacks that are dozens/hundreds of GB in size.
What are people's thoughts?
Updated by Anchi Cheng almost 14 years ago
My feeling on this is mixed. There is advantage using one format so that small functions like making average and calculate stats can be written easily, and then there is the problem of making a bad choice in the long run.
A few questions for those more familiar with SPIDER and hd5 format:
1. Can images be appended to these, if in stack format?
2. Do they have higher limit in accommodating larger number of images?
If first is true, I think it would be possible to modify makestack to make continue function to work with the conversion either in the loop parallel to our chosen standard stack format or as a post-loop function by keeping track of what need to be added.
As for accomodating larger number of images, I suppose eventually the stack approach will not be a good option.
Since I have not had to program stack functions, I have to admit that the above arguments are only philosophical, though.
Updated by Neil Voss almost 14 years ago
There was a bug in EMAN 1.8 that prevented SPIDER stacks from having more than some limit of particles (~80,000? May have depended on the box size). This bug was fixed in EMAN 1.9, but when makeStack2 was being developed we did not trust the SPIDER stacks, so we never added the function.
IIRC, I think the option is still on the website, but the feature is not available in makestack2???
Updated by Amber Herold almost 14 years ago
Neil, Gabe and Scott, would you be available to call into our Appion meeting this Wednesday at 11:00am? We would like to have a discussion on this topic and reach a conclusion.
Updated by Gabriel Lander almost 14 years ago
I'm free for the meeting, should I call the conference room at 11? What's the number there again?
Updated by Scott Stagg almost 14 years ago
I'm available then. Please post that phone number here so I can get it too.
Updated by Neil Voss almost 14 years ago
I have class from 11:00-1:20 (central) or 9:00-11:20 (pacific), so can call in late. Well, my lectures tend to run short, so I maybe able to call on time. Let me know how to proceed...
Updated by Amber Herold almost 14 years ago
Ok, Scott, Gabe, will 11:20am work for you guys as a start time?
Here is the number to call:
858-784-2663
Thanks!
Updated by Neil Voss almost 14 years ago
My number is (312) 322-7172 just in case. Another issue to consider is that if you create a SPIDER stack will it break all subsequent functions. It probably will.
Updated by Amber Herold almost 14 years ago
- Status changed from New to Closed
1. Agreed to have makestack just make the stack in imagic format and then write a converter that can convert to any other format.
2. User will have to option to create a spider stack when creating a normal stack. User will also be prompted to create the stack before running an operation that needs a missing stack format.
3. Gabe and Anchi will write the converter and may get input from Neil on particle correspondence
The converter work will be logged in #1163.
Thanks for everyone's participation!