How to add a new refinement method¶

database architecture for refinement methods¶

The current database scheme for every refinement method (both single-model and multi-model) is shown below:

database architecture for refinements

For reference, below is a diagram of the modifications to the refinement pipeline that have been performed for the refactoring. Color coding is as follows:

changes to the database architecture for refinements

• all previous database tables / pointers that have remained unchanged during refactoring are blue.
• database tables that are completely new are outlined AND filled in red
• database tables that have existed, but are modified are outlined in red, filled in white. The new additions are highlighted
• new pointers to other database tables are red; unmodified pointers are blue
• pointers to other database tables are all combined under "REFS"; if "REFS" is highlighted, this means that new pointers have been added

How to add a new refinement¶

1. determine the name of the new table in the database. In most cases, this will only be called "ApYourPackageRefineIterData." Unless there are specific parameters for each particle that you would like to save, this should probably contain all of your package-specific parameters.
2. write a job setup script in python (see example below).
3. write an upload script in python (see example below). Another option would be to have a script that converts your parameters into Appion / 3DEM format (see below), then upload as an external package.

Upload refinement script in python¶

The script should be titled 'uploadYourPackageRefine.py'

This script performs all of the basic operations that are needed to upload a refinement to the database, such that it can be displayed in AppionWeb. The bulk of the job is performed with the ReconUploader.py base class, which is inherited by each new uploadYourPackageRefine.py subclass script. this means that the developer's job is simply to make sure that all of the particle / package parameters are being passed in a specific format. Effectively, the only things that need to be written to this script are:

1. def start(self): this will setup basic package parameters and call on converter functions. In the single-model refinement case:
   

   def start(self):
   
       ### database entry parameters
       package_table = 'ApXmippRefineIterData|xmippParams'
   
       ### set projection-matching path
       self.projmatchpath = os.path.abspath(os.path.join(self.params['rundir'], self.runparams['package_params']['WorkingDir']))
   
       ### check for variable root directories between file systems
       apXmipp.checkSelOrDocFileRootDirectoryInDirectoryTree(self.params['rundir'], self.runparams['cluster_root_path'], self.runparams['upload_root_path'])
   
       ### determine which iterations to upload
       lastiter = self.findLastCompletedIteration()
       uploadIterations = self.verifyUploadIterations(lastiter)    
   
       ### upload each iteration
       for iteration in uploadIterations:
   
           apDisplay.printColor("uploading iteration %d" % iteration, "cyan")
   
           ### set package parameters, as they will appear in database entries
           package_database_object = self.instantiateProjMatchParamsData(iteration)
   
           ### move FSC file to results directory
           oldfscfile = os.path.join(self.projmatchpath, "Iter_%d" % iteration, "Iter_%d_resolution.fsc" % iteration)
           newfscfile = os.path.join(self.resultspath, "recon_%s_it%.3d_vol001.fsc" % (self.params['timestamp'],iteration))
           if os.path.exists(oldfscfile):
               shutil.copyfile(oldfscfile, newfscfile)
   
           ### create a stack of class averages and reprojections (optional)
           self.compute_stack_of_class_averages_and_reprojections(iteration)
   
           ### create a text file with particle information
           self.createParticleDataFile(iteration)
   
           ### create mrc file of map for iteration and reference number
           oldvol = os.path.join(self.projmatchpath, "Iter_%d" % iteration, "Iter_%d_reconstruction.vol" % iteration)
           newvol = os.path.join(self.resultspath, "recon_%s_it%.3d_vol001.mrc" % (self.params['timestamp'], iteration))
           mrccmd = "proc3d %s %s apix=%.3f" % (oldvol, newvol, self.runparams['apix'])
           apParam.runCmd(mrccmd, "EMAN")
   
           ### make chimera snapshot of volume
           self.createChimeraVolumeSnapshot(newvol, iteration)
   
           ### instantiate database objects
           self.insertRefinementRunData(iteration)
           self.insertRefinementIterationData(package_table, package_database_object, iteration)
   
       ### calculate Euler jumps
       self.calculateEulerJumpsAndGoodBadParticles(uploadIterations)    
   
       ### query the database for the completed refinements BEFORE deleting any files ... returns a dictionary of lists
       ### e.g. {1: [5, 4, 3, 2, 1]} means 5 iters completed for refine 1
       complete_refinements = self.verifyNumberOfCompletedRefinements(multiModelRefinementRun=False)
       if self.params['cleanup_files'] is True:
           self.cleanupFiles(complete_refinements)
   

   
   in the multi-model refinement case (example XmippML3D):
   

   def start(self):
   
       ### database entry parameters
       package_table = 'ApXmippML3DRefineIterData|xmippML3DParams'
   
       ### set ml3d path
       self.ml3dpath = os.path.abspath(os.path.join(self.params['rundir'], self.runparams['package_params']['WorkingDir'], "RunML3D"))
   
       ### check for variable root directories between file systems
       apXmipp.checkSelOrDocFileRootDirectoryInDirectoryTree(self.params['rundir'], self.runparams['cluster_root_path'], self.runparams['upload_root_path'])
   
       ### determine which iterations to upload
       lastiter = self.findLastCompletedIteration()
       uploadIterations = self.verifyUploadIterations(lastiter)                
   
       ### create ml3d_lib.doc file somewhat of a workaround, but necessary to make projections
       total_num_2d_classes = self.createModifiedLibFile()
   
       ### upload each iteration
       for iteration in uploadIterations:
   
           ### set package parameters, as they will appear in database entries
           package_database_object = self.instantiateML3DParamsData(iteration)
   
           for j in range(self.runparams['package_params']['NumberOfReferences']):
   
               ### calculate FSC for each iteration using split selfile (selfile requires root directory change)
               self.calculateFSCforIteration(iteration, j+1)
   
               ### create a stack of class averages and reprojections (optional)
               self.compute_stack_of_class_averages_and_reprojections(iteration, j+1)
   
               ### create a text file with particle information
               self.createParticleDataFile(iteration, j+1, total_num_2d_classes)
   
               ### create mrc file of map for iteration and reference number
               oldvol = os.path.join(self.ml3dpath, "ml3d_it%.6d_vol%.6d.vol" % (iteration, j+1))
               newvol = os.path.join(self.resultspath, "recon_%s_it%.3d_vol%.3d.mrc" % (self.params['timestamp'], iteration, j+1))
               mrccmd = "proc3d %s %s apix=%.3f" % (oldvol, newvol, self.runparams['apix'])
               apParam.runCmd(mrccmd, "EMAN")
   
               ### make chimera snapshot of volume
               self.createChimeraVolumeSnapshot(newvol, iteration, j+1)
   
               ### instantiate database objects
               self.insertRefinementRunData(iteration, j+1)
               self.insertRefinementIterationData(package_table, package_database_object, iteration, j+1)
   
       ### calculate Euler jumps
       self.calculateEulerJumpsAndGoodBadParticles(uploadIterations)            
   
       ### query the database for the completed refinements BEFORE deleting any files ... returns a dictionary of lists
       ### e.g. {1: [5, 4, 3, 2, 1], 2: [6, 5, 4, 3, 2, 1]} means 5 iters completed for refine 1 & 6 iters completed for refine 2
       complete_refinements = self.verifyNumberOfCompletedRefinements(multiModelRefinementRun=True)
       if self.params['cleanup_files'] is True:
           self.cleanupFiles(complete_refinements)
   

   #