Bug #2635
open
refinements on garibaldi - issue 1 - saving command
Added by Dmitry Lyumkis over 10 years ago.
Updated over 10 years ago.
Category:
Python scripting
Affected Version:
Appion/Leginon 2.1.0
Description
"runjob.py ... ", the command for setting up and executing the refinement protocol, is not saved anywhere. It creates the file "[*].commands", which contains some csh / python scripts, but the command itself is not saved anywhere. Therefore if it fails, it is not clear how to rerun the job after the setup finished.
Related issues
1 (1 open — 0 closed)
Amber, any chance you can take care of this bug? We are trying to go through the Xmipp refinement pipeline with Natalia, and can't get commands to rerun.
Here is a thought for implementing this.
We could create a "Project Notebook". In the database, there would be a new table with at least a "date/time" field and a "Note" text field and possibly a "category" field. Categories could be "shown commands", "run commands", "custom commands" maybe "custom script"...basically any type of text based notes that would be useful to save with the project.
Then we could add a GUI page that just displays the entries in the Notebook, and they are perhaps editable and deletable. Another GUI page would allow the user to add a custom entry. All commands that are shown to the user or run for the user are saved in this table by default. The entries could eventually be sorted by category or searched for keywords.
My feeling is that this is going to be too complicated. All we need to do is save the command so that it can be rerun. But however you think is best would work.
OK, I can look into saving it in a text file on the python side into the run directory.
Dmitry, if I save the command to a file on the python side, it would happen right before creating the job file. So it seems to me that the only case this would help for is if there was an issue writing out the job file. Once the job file is written, you can just run the job file with qsub. Am I missing something? Is it that you want to be able to change the refinement parameters like the number of iterations to run more easily?
The .commands file is needed because the .job file is overwritten during the upload step and the upload step needs the info. Is this the issue? Do you need the runJob.py command or the job file after an upload has already been run?
Will this plan work?
How do we get this resolved?
- Category set to Python scripting
- Status changed from New to In Code Review
- Assignee changed from Amber Herold to Dmitry Lyumkis
- Target version set to Appion/Leginon 3.1.0
r18084 adds code to save any commands run with runJob.py to a text file called run_commands.log.
Commands are appended, so they are never overwritten. Each entry has a date/time stamp.
Dmitry, we also talked about making a copy of the created job file. I can still do that, but getting this portion checked in first.
To save off the job file, I'll need to generate a unique name for it so that it is not overwritten. Perhaps adding the date/time to the name would work.
I'll create a new issue for the job file feature. (#2647)
- Status changed from In Code Review to In Test
looks good amber, thanks. We will test with Natalia. Along the lines of this bug, we were looking at the refinement job from Mark Walter, trying to figure out his initial stack filtering parameters during the preparation step. i couldn't find them anywhere, nor could I find the "prepare_refinement" command saved anywhere. Am I missing something? might be worth logging that as well.
This will create a command log for every feature of appion as long as it is run using runJob.py. That includes the refinement prep step.
Also available in: Atom
PDF