Appion

h1. How to add a new refinement method

h2. database architecture for refinement methods

The current database scheme for every refinement method (both single-model and multi-model) is shown below:

"database architecture for refinements":http://emg.nysbc.org/attachments/955/database_scheme_diagram.pdf

For reference, below is a diagram of the modifications to the refinement pipeline that have been performed for the refactoring. Color coding is as follows: 

"changes to the database architecture for refinements":http://emg.nysbc.org/attachments/954/database_scheme_diagram_changes.pdf

* all previous database tables / pointers that have remained unchanged during refactoring are blue. 

* database tables that are completely new are outlined AND filled in red

* database tables that have existed, but are modified are outlined in red, filled in white. The new additions are highlighted

* new pointers to other database tables are red; unmodified pointers are blue

* pointers to other database tables are all combined under "REFS"; if "REFS" is highlighted, this means that new pointers have been added

h2. How to add a new refinement

# determine the name of the new table in the database. In most cases, this will only be called "ApYourPackageRefineIterData." Unless there are specific parameters for each particle that you would like to save, this should probably contain all of your package-specific parameters. 

# write a job setup script in python (see example below). 

# write an upload script in python (see example below). Another option would be to have a script that converts your parameters into Appion / 3DEM format (see below), then upload as an external package. 

h2. Upload refinement script in python

The script should be titled 'uploadYourPackageRefine.py'

This script performs all of the basic operations that are needed to upload a refinement to the database, such that it can be displayed in AppionWeb. The bulk of the job is performed with the ReconUploader.py base class, which is inherited by each new uploadYourPackageRefine.py subclass script. this means that the developer's job is simply to make sure that all of the particle / package parameters are being passed in a specific format. Effectively, the only things that need to be written to this script are: 

# def start(self): this will setup basic package parameters and call on converter functions. In the single-model refinement case:

<pre>

def start(self):

	### database entry parameters

	package_table = 'ApXmippRefineIterData|xmippParams'

	### set projection-matching path

	self.projmatchpath = os.path.abspath(os.path.join(self.params['rundir'], self.runparams['package_params']['WorkingDir']))

	### check for variable root directories between file systems

	apXmipp.checkSelOrDocFileRootDirectoryInDirectoryTree(self.params['rundir'], self.runparams['cluster_root_path'], self.runparams['upload_root_path'])

	### determine which iterations to upload

	lastiter = self.findLastCompletedIteration()

	uploadIterations = self.verifyUploadIterations(lastiter)	

	### upload each iteration

	for iteration in uploadIterations:

		apDisplay.printColor("uploading iteration %d" % iteration, "cyan")

		### set package parameters, as they will appear in database entries

		package_database_object = self.instantiateProjMatchParamsData(iteration)

		### move FSC file to results directory

		oldfscfile = os.path.join(self.projmatchpath, "Iter_%d" % iteration, "Iter_%d_resolution.fsc" % iteration)

		newfscfile = os.path.join(self.resultspath, "recon_%s_it%.3d_vol001.fsc" % (self.params['timestamp'],iteration))

		if os.path.exists(oldfscfile):

			shutil.copyfile(oldfscfile, newfscfile)

		### create a stack of class averages and reprojections (optional)

		self.compute_stack_of_class_averages_and_reprojections(iteration)

		### create a text file with particle information

		self.createParticleDataFile(iteration)

		### create mrc file of map for iteration and reference number

		oldvol = os.path.join(self.projmatchpath, "Iter_%d" % iteration, "Iter_%d_reconstruction.vol" % iteration)

		newvol = os.path.join(self.resultspath, "recon_%s_it%.3d_vol001.mrc" % (self.params['timestamp'], iteration))

		mrccmd = "proc3d %s %s apix=%.3f" % (oldvol, newvol, self.runparams['apix'])

		apParam.runCmd(mrccmd, "EMAN")

		### make chimera snapshot of volume

		self.createChimeraVolumeSnapshot(newvol, iteration)

		### instantiate database objects

		self.insertRefinementRunData(iteration)

		self.insertRefinementIterationData(package_table, package_database_object, iteration)

	### calculate Euler jumps

	self.calculateEulerJumpsAndGoodBadParticles(uploadIterations)	

	### query the database for the completed refinements BEFORE deleting any files ... returns a dictionary of lists

	### e.g. {1: [5, 4, 3, 2, 1]} means 5 iters completed for refine 1

	complete_refinements = self.verifyNumberOfCompletedRefinements(multiModelRefinementRun=False)

	if self.params['cleanup_files'] is True:

		self.cleanupFiles(complete_refinements)

</pre>

in the multi-model refinement case (example XmippML3D):

<pre>

def start(self):

	### database entry parameters

	package_table = 'ApXmippML3DRefineIterData|xmippML3DParams'

	### set ml3d path

	self.ml3dpath = os.path.abspath(os.path.join(self.params['rundir'], self.runparams['package_params']['WorkingDir'], "RunML3D"))

	### check for variable root directories between file systems

	apXmipp.checkSelOrDocFileRootDirectoryInDirectoryTree(self.params['rundir'], self.runparams['cluster_root_path'], self.runparams['upload_root_path'])

	### determine which iterations to upload

	lastiter = self.findLastCompletedIteration()

	uploadIterations = self.verifyUploadIterations(lastiter)				

	### create ml3d_lib.doc file somewhat of a workaround, but necessary to make projections

	total_num_2d_classes = self.createModifiedLibFile()

	### upload each iteration

	for iteration in uploadIterations:

		### set package parameters, as they will appear in database entries

		package_database_object = self.instantiateML3DParamsData(iteration)

		for j in range(self.runparams['package_params']['NumberOfReferences']):

			### calculate FSC for each iteration using split selfile (selfile requires root directory change)

			self.calculateFSCforIteration(iteration, j+1)

			### create a stack of class averages and reprojections (optional)

			self.compute_stack_of_class_averages_and_reprojections(iteration, j+1)

			### create a text file with particle information

			self.createParticleDataFile(iteration, j+1, total_num_2d_classes)

			### create mrc file of map for iteration and reference number

			oldvol = os.path.join(self.ml3dpath, "ml3d_it%.6d_vol%.6d.vol" % (iteration, j+1))

			newvol = os.path.join(self.resultspath, "recon_%s_it%.3d_vol%.3d.mrc" % (self.params['timestamp'], iteration, j+1))

			mrccmd = "proc3d %s %s apix=%.3f" % (oldvol, newvol, self.runparams['apix'])

			apParam.runCmd(mrccmd, "EMAN")

			### make chimera snapshot of volume

			self.createChimeraVolumeSnapshot(newvol, iteration, j+1)

			### instantiate database objects

			self.insertRefinementRunData(iteration, j+1)

			self.insertRefinementIterationData(package_table, package_database_object, iteration, j+1)

	### calculate Euler jumps

	self.calculateEulerJumpsAndGoodBadParticles(uploadIterations)			

	### query the database for the completed refinements BEFORE deleting any files ... returns a dictionary of lists

	### e.g. {1: [5, 4, 3, 2, 1], 2: [6, 5, 4, 3, 2, 1]} means 5 iters completed for refine 1 & 6 iters completed for refine 2

	complete_refinements = self.verifyNumberOfCompletedRefinements(multiModelRefinementRun=True)

	if self.params['cleanup_files'] is True:

		self.cleanupFiles(complete_refinements)

</pre>

#