Project

General

Profile

Actions

Bug #328

closed

Major Appion database schema change

Added by Neil Voss over 14 years ago. Updated over 14 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Amber Herold
Category:
-
Target version:
Start date:
04/07/2010
Due date:
% Done:

100%

Estimated time:
Affected Version:
Show in known bugs:
Workaround:

Description

We talked about this at our appion meeting. I want to change the refine tables to be more general for other processing packages.


Related issues 5 (1 open4 closed)

Related to Appion - Bug #333: Make all tables compatible with sinedonNew04/08/2010

Actions
Related to Appion - Feature #344: Create database conversion scriptClosedAmber Herold04/12/2010

Actions
Related to Appion - Bug #347: Remove spaces from all db fields in AppionClosedAnchi Cheng04/13/2010

Actions
Related to Appion - Bug #362: appion job names should sync with appion program namesClosedAnchi Cheng04/19/2010

Actions
Related to Appion - Feature #345: automatic database conversion on release updateClosedAmber Herold04/12/2010

Actions
Actions #1

Updated by Neil Voss over 14 years ago

  • Changing Table Names:
    ApRefinementData -> ApRefineIterData #fix confusing name
    ApRefinementRunData -> ApRefineRunData #shorten name
    ApParticleClassificationData -> ApRefineParticleData #fix confusing name
    ApRefinementParamsData -> ApEmanRefineIterData  #EMAN only params
    
    ApStackParticlesData -> ApStackParticleData #rm plural
    ApAlignParticlesData -> ApAlignParticleData #rm plural
    ApClusteringParticlesData -> ApClusteringParticleData #rm plural
    
  • Columns Moving to Different Table:
    ApRefinementParamsData.symmetry -> ApRefineIterData.symmetry #these are global params not EMAN only
    ApRefinementParamsData.mask -> ApRefineIterData.mask #these are global params not EMAN only
    ApRefinementParamsData.imask -> ApRefineIterData.imask #these are global params not EMAN only
    
  • Merging Columns:
    ApRefinementData.classAverage
    ApRefinementData.emanClassAvg
    ApRefinementData.MsgPGoodClassAvg #barely used
    ApRefinementData.SpiCoranGoodClassAvg
    
    There are 4 columns for class averages, I want to have a more generic 2 column system of:
    ApRefinementData.classAverage
    ApRefinementData.classAverage2
    
    Or we can call it finalClassAverage, filteredClassAverage
ApParticleClassificationData.coran_keep -> ApRefineParticleData.filter_keep
ApParticleClassificationData.msgp_keep -> ApRefineParticleData.filter_keep
ApParticleClassificationData.thrown_out ~~ ApRefineParticleData.refine_keep #need to invert value
It is good to keep track of which particle EMAN gets rid of "thrown_out" and which particles are rejected for other reason
"msgp_keep", "coran_keep", but they can be unified into "filter_keep", "refine_keep"
  • Changing Column Names:
    ApRefinementData.refinementParams -> ApRefineIterData.emanParams #eman only table
    ApRefinementData.refinementRun -> ApRefinementData.refineRun #shorten name
    ApRefinementRunData.name -> ApRefineRunData.runname #make consistent with other tables
    
  • New Calculated Columns:
    ApRefineRunData.num_iter #number of iterations
    ApStackData.boxsize #final boxsize of stack, makes it easier to get/change rather than calculate
    
  • Project columns:
    *.project|projects|project -> *.project
    
    Can we do this?
Actions #2

Updated by Neil Voss over 14 years ago

  • Priority changed from Normal to High
Actions #3

Updated by Neil Voss over 14 years ago

  • Target version set to Appion/Leginon 2.0.0
Actions #4

Updated by Anchi Cheng over 14 years ago

what is *.project? I don't see what that accomplishes since the table and column for it is not in sinedon format.

Merging columns
ApRefinementData.classAverage
ApRefinementData.emanClassAvg
ApRefinementData.MsgPGoodClassAvg #barely used
ApRefinementData.SpiCoranGoodClassAvg

I thought the last three were divided to make it easier to query individually. I don't remember the details.
If we are sure that package name is sufficient for knowing which ones were generated from what method, generic names are indeed simpler.

Actions #5

Updated by Neil Voss over 14 years ago

Actually the sinedon format would be:

REF|project|projects|projectId

NOT
project|projects|project

And it probably should be
REF|project|projects|DEF_id

to be really compatible with sinedon.

Currently the project Table is not compatible with Sinedon and all the fields linking to it are just an integers.

Actions #6

Updated by Neil Voss over 14 years ago

Comment on merging columns by merging we get some advantages:

  • Programs that make substack new volumes based on class averages only need to link to one column not 3 or 4.
  • The Recon Report page/particledata.inc will have less IF statements
  • If type is of concern we can create a new field, classAverage2type, that will fix this
Actions #7

Updated by Anchi Cheng over 14 years ago

Actually sinedon format would be `REF|projectdata|projects|xxxx` xxxx refers to the key you used in the appiondata.py
where projectdata refers to the module name in sinedon.cfg not the database name.

About merging columns, I agree that it should work fine.

Actions #8

Updated by Neil Voss over 14 years ago

  • % Done changed from 0 to 70
Actions #9

Updated by Neil Voss over 14 years ago

MyAMI Branch with Schema changes has been created.

svn co http://emg.nysbc.org/svn/myami/branches/may2010schema/ may2010schema/
Actions #10

Updated by Neil Voss over 14 years ago

  • Status changed from Assigned to In Test
  • Priority changed from High to Normal
  • % Done changed from 70 to 100
Actions #11

Updated by Neil Voss over 14 years ago

  • Assignee changed from Neil Voss to Anchi Cheng

test on Scott's data and see if it works.

Actions #12

Updated by Anchi Cheng over 14 years ago

  • Status changed from In Test to Assigned
  • Assignee changed from Anchi Cheng to Neil Voss

Scott's ApRefineGoodBadParticleData is empty after upgrade.

Actions #13

Updated by Neil Voss over 14 years ago

Added schema-14077.py to do this. It can be done any time, but the code is already ready written.

source:trunk/dbschema/schema-14077.py

Actions #14

Updated by Neil Voss over 14 years ago

  • Status changed from Assigned to In Code Review
  • Assignee changed from Neil Voss to Anchi Cheng

Added schema-14077.py to do this. It can be done any time, but the code is already ready written for the latest version.

source:trunk/dbschema/schema-14077.py

Actions #15

Updated by Anchi Cheng over 14 years ago

  • Status changed from In Code Review to In Test

added version and dbcheck in r14096. Tested at the same time as testing bug #479

Actions #16

Updated by Anchi Cheng over 14 years ago

Why can't I update this issue to closed?

Actions #17

Updated by Anchi Cheng over 14 years ago

  • Assignee changed from Anchi Cheng to Amber Herold

I can't close this issue even though it is tested.

Actions #18

Updated by Amber Herold over 14 years ago

You must close #345 before you can close this one because it is blocking this.

Actions #19

Updated by Amber Herold over 14 years ago

  • Status changed from In Test to Closed
Actions

Also available in: Atom PDF