Bug #2635
openrefinements on garibaldi - issue 1 - saving command
0%
Description
"runjob.py ... ", the command for setting up and executing the refinement protocol, is not saved anywhere. It creates the file "[*].commands", which contains some csh / python scripts, but the command itself is not saved anywhere. Therefore if it fails, it is not clear how to rerun the job after the setup finished.
Updated by Dmitry Lyumkis almost 11 years ago
Amber, any chance you can take care of this bug? We are trying to go through the Xmipp refinement pipeline with Natalia, and can't get commands to rerun.
Updated by Amber Herold almost 11 years ago
Here is a thought for implementing this.
We could create a "Project Notebook". In the database, there would be a new table with at least a "date/time" field and a "Note" text field and possibly a "category" field. Categories could be "shown commands", "run commands", "custom commands" maybe "custom script"...basically any type of text based notes that would be useful to save with the project.
Then we could add a GUI page that just displays the entries in the Notebook, and they are perhaps editable and deletable. Another GUI page would allow the user to add a custom entry. All commands that are shown to the user or run for the user are saved in this table by default. The entries could eventually be sorted by category or searched for keywords.
Updated by Dmitry Lyumkis almost 11 years ago
My feeling is that this is going to be too complicated. All we need to do is save the command so that it can be rerun. But however you think is best would work.
Updated by Amber Herold almost 11 years ago
OK, I can look into saving it in a text file on the python side into the run directory.
Updated by Amber Herold almost 11 years ago
Dmitry, if I save the command to a file on the python side, it would happen right before creating the job file. So it seems to me that the only case this would help for is if there was an issue writing out the job file. Once the job file is written, you can just run the job file with qsub. Am I missing something? Is it that you want to be able to change the refinement parameters like the number of iterations to run more easily?
The .commands file is needed because the .job file is overwritten during the upload step and the upload step needs the info. Is this the issue? Do you need the runJob.py command or the job file after an upload has already been run?
Updated by Bridget Carragher almost 11 years ago
Will this plan work?
How do we get this resolved?
Updated by Amber Herold almost 11 years ago
- Category set to Python scripting
- Status changed from New to In Code Review
- Assignee changed from Amber Herold to Dmitry Lyumkis
- Target version set to Appion/Leginon 3.1.0
r18084 adds code to save any commands run with runJob.py to a text file called run_commands.log.
Commands are appended, so they are never overwritten. Each entry has a date/time stamp.
Dmitry, we also talked about making a copy of the created job file. I can still do that, but getting this portion checked in first.
To save off the job file, I'll need to generate a unique name for it so that it is not overwritten. Perhaps adding the date/time to the name would work.
I'll create a new issue for the job file feature. (#2647)
Updated by Dmitry Lyumkis almost 11 years ago
- Status changed from In Code Review to In Test
looks good amber, thanks. We will test with Natalia. Along the lines of this bug, we were looking at the refinement job from Mark Walter, trying to figure out his initial stack filtering parameters during the preparation step. i couldn't find them anywhere, nor could I find the "prepare_refinement" command saved anywhere. Am I missing something? might be worth logging that as well.
Updated by Amber Herold almost 11 years ago
This will create a command log for every feature of appion as long as it is run using runJob.py. That includes the refinement prep step.