How to add a new refinement method » History » Revision 8
Revision 7 (Dmitry Lyumkis, 08/01/2011 01:11 PM) → Revision 8/38 (Dmitry Lyumkis, 08/01/2011 01:28 PM)
h1. How to add a new refinement method h2. database architecture for refinement methods The current database scheme for every refinement method (both single-model and multi-model) is shown below: "database architecture for refinements":http://emg.nysbc.org/attachments/955/database_scheme_diagram.pdf For reference, below is a diagram of the modifications to the refinement pipeline that have been performed for the refactoring. Color coding is as follows: "changes to the database architecture for refinements":http://emg.nysbc.org/attachments/954/database_scheme_diagram_changes.pdf * all previous database tables / pointers that have remained unchanged during refactoring are blue. * database tables that are completely new are outlined AND filled in red * database tables that have existed, but are modified are outlined in red, filled in white. The new additions are highlighted * new pointers to other database tables are red; unmodified pointers are blue * pointers to other database tables are all combined under "REFS"; if "REFS" is highlighted, this means that new pointers have been added h2. How to add a new refinement !database_scheme_diagram_changes.pdf! # determine the name of the new table in the database. In most cases, this will only be called "ApYourPackageRefineIterData." Unless there are specific parameters for each particle that you would like to save, this should probably contain all of your package-specific parameters. # write a job setup script in python (see example below). # write an upload script in python (see example below). Another option would be to have a script that converts your parameters into Appion / 3DEM format (see below), then upload as an external package. h2. Upload refinement script in python The script should be titled 'uploadYourPackageRefine.py' This script performs all of the basic operations that are needed to upload a refinement to the database, such that it can be displayed in AppionWeb. The bulk of the job is performed with the ReconUploader.py base class, which is inherited by each new uploadYourPackageRefine.py subclass script. this means that the developer's job is simply to make sure that all of the particle / package parameters are being passed in a specific format. Effectively, the only things that need to be written to this script are: # def start(self): this will setup basic package parameters and call on converter functions. In the single-model refinement case: <pre> def start(self): ### database entry parameters package_table = 'ApXmippRefineIterData|xmippParams' ### set projection-matching path self.projmatchpath = os.path.abspath(os.path.join(self.params['rundir'], self.runparams['package_params']['WorkingDir'])) ### check for variable root directories between file systems apXmipp.checkSelOrDocFileRootDirectoryInDirectoryTree(self.params['rundir'], self.runparams['cluster_root_path'], self.runparams['upload_root_path']) ### determine which iterations to upload lastiter = self.findLastCompletedIteration() uploadIterations = self.verifyUploadIterations(lastiter) ### upload each iteration for iteration in uploadIterations: apDisplay.printColor("uploading iteration %d" % iteration, "cyan") ### set package parameters, as they will appear in database entries package_database_object = self.instantiateProjMatchParamsData(iteration) ### move FSC file to results directory oldfscfile = os.path.join(self.projmatchpath, "Iter_%d" % iteration, "Iter_%d_resolution.fsc" % iteration) newfscfile = os.path.join(self.resultspath, "recon_%s_it%.3d_vol001.fsc" % (self.params['timestamp'],iteration)) if os.path.exists(oldfscfile): shutil.copyfile(oldfscfile, newfscfile) ### create a stack of class averages and reprojections (optional) self.compute_stack_of_class_averages_and_reprojections(iteration) ### create a text file with particle information self.createParticleDataFile(iteration) ### create mrc file of map for iteration and reference number oldvol = os.path.join(self.projmatchpath, "Iter_%d" % iteration, "Iter_%d_reconstruction.vol" % iteration) newvol = os.path.join(self.resultspath, "recon_%s_it%.3d_vol001.mrc" % (self.params['timestamp'], iteration)) mrccmd = "proc3d %s %s apix=%.3f" % (oldvol, newvol, self.runparams['apix']) apParam.runCmd(mrccmd, "EMAN") ### make chimera snapshot of volume self.createChimeraVolumeSnapshot(newvol, iteration) ### instantiate database objects self.insertRefinementRunData(iteration) self.insertRefinementIterationData(package_table, package_database_object, iteration) ### calculate Euler jumps self.calculateEulerJumpsAndGoodBadParticles(uploadIterations) ### query the database for the completed refinements BEFORE deleting any files ... returns a dictionary of lists ### e.g. {1: [5, 4, 3, 2, 1]} means 5 iters completed for refine 1 complete_refinements = self.verifyNumberOfCompletedRefinements(multiModelRefinementRun=False) if self.params['cleanup_files'] is True: self.cleanupFiles(complete_refinements) </pre> in the multi-model refinement case (example XmippML3D): <pre> def start(self): ### database entry parameters package_table = 'ApXmippML3DRefineIterData|xmippML3DParams' ### set ml3d path self.ml3dpath = os.path.abspath(os.path.join(self.params['rundir'], self.runparams['package_params']['WorkingDir'], "RunML3D")) ### check for variable root directories between file systems apXmipp.checkSelOrDocFileRootDirectoryInDirectoryTree(self.params['rundir'], self.runparams['cluster_root_path'], self.runparams['upload_root_path']) ### determine which iterations to upload lastiter = self.findLastCompletedIteration() uploadIterations = self.verifyUploadIterations(lastiter) ### create ml3d_lib.doc file somewhat of a workaround, but necessary to make projections total_num_2d_classes = self.createModifiedLibFile() ### upload each iteration for iteration in uploadIterations: ### set package parameters, as they will appear in database entries package_database_object = self.instantiateML3DParamsData(iteration) for j in range(self.runparams['package_params']['NumberOfReferences']): ### calculate FSC for each iteration using split selfile (selfile requires root directory change) self.calculateFSCforIteration(iteration, j+1) ### create a stack of class averages and reprojections (optional) self.compute_stack_of_class_averages_and_reprojections(iteration, j+1) ### create a text file with particle information self.createParticleDataFile(iteration, j+1, total_num_2d_classes) ### create mrc file of map for iteration and reference number oldvol = os.path.join(self.ml3dpath, "ml3d_it%.6d_vol%.6d.vol" % (iteration, j+1)) newvol = os.path.join(self.resultspath, "recon_%s_it%.3d_vol%.3d.mrc" % (self.params['timestamp'], iteration, j+1)) mrccmd = "proc3d %s %s apix=%.3f" % (oldvol, newvol, self.runparams['apix']) apParam.runCmd(mrccmd, "EMAN") ### make chimera snapshot of volume self.createChimeraVolumeSnapshot(newvol, iteration, j+1) ### instantiate database objects self.insertRefinementRunData(iteration, j+1) self.insertRefinementIterationData(package_table, package_database_object, iteration, j+1) ### calculate Euler jumps self.calculateEulerJumpsAndGoodBadParticles(uploadIterations) ### query the database for the completed refinements BEFORE deleting any files ... returns a dictionary of lists ### e.g. {1: [5, 4, 3, 2, 1], 2: [6, 5, 4, 3, 2, 1]} means 5 iters completed for refine 1 & 6 iters completed for refine 2 complete_refinements = self.verifyNumberOfCompletedRefinements(multiModelRefinementRun=True) if self.params['cleanup_files'] is True: self.cleanupFiles(complete_refinements) </pre> #