Task #2071
closedBug #2068: Resolve CTF problem effort by NRAMM local team
Ctf testing matrix
0%
Description
- We need to identify all test data needed to test all features of CTF Estimation found in appion.
- We need to know what the output from these data should be.
- We need to identify all the tests that need to be run.
CTF Estimation Test Matrix¶
Ace2 Tests
Note: After running command, results are found in ...imagename.mrc.ctf.txt
Astigmatism | Over/Under Focus | Command | Image Name | Defocus | Ang Astigmatism | Amplitude Contrast | Confidence |
---|---|---|---|---|---|---|---|
none | over | ace2.exe -i 12oct01e_00059en.mrc -b 2 -c 2.0 -k 200.0 -a 1.3656 -e 10.0,0.001 -r 0.0 | 12oct01e_00059en.mrc | 5.568862e-07 6.559742e-07 | -70.235983 | 0.091264 | 0.815827 |
none | under | ace2.exe -i 12oct01e_00055en.mrc -b 2 -c 2.0 -k 200.0 -a 1.3656 -e 10.0,0.001 -r 0.0 | 12oct01e_00055en.mrc | 1.280421e-06 1.318144e-06 | 15.267624 | 0.192086 | 0.947226 |
direction 1 | over | ace2.exe -i 12oct01e_00041ea.mrc -b 2 -c 2.0 -k 200.0 -a 1.3656 -e 10.0,0.001 -r 0.0 | 12oct01e_00041ea.mrc | 1.337478e-06 1.482890e-06 | 72.485569 | 0.050000 | 0.863265 |
direction 1 | under | ace2.exe -i 12oct01e_00032ea.mrc -b 2 -c 2.0 -k 200.0 -a 1.3656 -e 10.0,0.001 -r 0.0 | 12oct01e_00032ea.mrc | 1.773057e-06 1.867818e-06 | -66.550039 | 0.182214 | 0.957607 |
direction 2 | over | ace2.exe -i 12oct01e_00050eb.mrc -b 2 -c 2.0 -k 200.0 -a 1.3656 -e 10.0,0.001 -r 0.0 | 12oct01e_00050eb.mrc | 1.477478e-06 1.674565e-06 | -64.236799 | 0.050000 | 0.876410 |
direction 2 | under | ace2.exe -i 12oct01e_00054eb.mrc -b 2 -c 2.0 -k 200.0 -a 1.3656 -e 10.0,0.001 -r 0.0 | 12oct01e_00054eb.mrc | 1.533405e-06 1.649353e-06 | 29.912167 | 0.194930 | 0.958123 |
Ace1 Tests
Note: After running command, results are found in
Astigmatism | Over/Under Focus | Command | Image Name | Defocus | Ang Astigmatism | Amplitude Contrast | Confidence |
---|---|---|---|---|---|---|---|
none | over | ||||||
none | under |
CTFFind Tests
Note: After running command, results are found in rundir/logfiles/imagename-ctfprog.log if run in appion. Output files end in -pow.mrc and are attached to this issue.
Astigmatism | Over/Under Focus | Command | Image Name | Defocus | Ang Astigmatism | Cross Correlation |
---|---|---|---|---|---|---|
none | over | ctffind64.exe 12oct01e_059en.mrc 12oct01e_059en-pow.mrc 2.0,200.0,0.15,109841.827768,15.0,2 512,40.0,8.0,1000.0,30000.0,1000.0,100.0 | 12oct01e_00059en.mrc | 6072.85 5544.12 | -17.61 | 0.47946 |
none | under | ctffind64.exe 12oct01e_055en.mrc 12oct01e_055en-pow.mrc 2.0,200.0,0.15,109841.827768,15.0,2 512,40.0,8.0,1000.0,40000.0,1000.0,100.0 | 12oct01e_00055en.mrc | 13154.68 13285.13 | -18.97 | 0.40741 |
direction 1 | over | ctffind64.exe 12oct01e_037ea.mrc 12oct01e_037ea-pow.mrc 2.0,200.0,0.15,109841.827768,15.0,2 512,40.0,8.0,1000.0,35000.0,1000.0,100.0 | 12oct01e_00037ea.mrc | 8318.40 8167.91 | 50.30 | 0.48868 |
direction 1 | under | ctffind64.exe 12oct01e_032ea.mrc 12oct01e_032ea-pow.mrc 2.0,200.0,0.15,109841.827768,15.0,2 512,40.0,8.0,1000.0,40000.0,1000.0,100.0 | 12oct01e_00032ea.mrc | 18052.81 18536.83 | 65.26 | 0.36839 |
direction 2 | over | ctffind64.exe 12oct01e_048eb.mrc 12oct01e_048eb-pow.mrc 2.0,200.0,0.15,109841.827768,15.0,2 512,40.0,8.0,1000.0,30000.0,1000.0,500.0 | 12oct01e_00048eb.mrc | 4500.14 6016.58 | 64.72 | 0.45517 |
direction 2 | under | ctffind64.exe 12oct01e_054eb.mrc 12oct01e_054eb-pow.mrc 2.0,200.0,0.15,109841.827768,15.0,2 512,40.0,8.0,1000.0,40000.0,1000.0,500.0 | 12oct01e_00054eb.mrc | 15613.98 16342.15 | -28.88 | 0.37626 |
Xmipp CTF Tests
Note: After running command, results are found in
Astigmatism | Over/Under Focus | Command | Image Name | Defocus | Ang Astigmatism | Amplitude Contrast | Confidence |
---|---|---|---|---|---|---|---|
none | over | ||||||
none | under | ||||||
direction 1 | over | ||||||
direction 1 | under | ||||||
direction 2 | over | ||||||
direction 2 | under |
CTF Correction Test Matrix¶
Estimation Package | Correction Method | Astigmation | Over/Under Focus | Command | Image Name |
---|---|---|---|---|---|
Ace2 | ACE2 wiener | none | over | 12oct01e_00059en.mrc | |
Ace2 | ACE2 wiener | none | under | 12oct01e_00055en.mrc | |
Ace2 | ACE2 wiener | direction 1 | over | 12oct01e_00041ea.mrc | |
Ace2 | ACE2 wiener | direction 1 | under | 12oct01e_00032ea.mrc | |
Ace2 | ACE2 wiener | direction 2 | over | 12oct01e_00050eb.mrc | |
Ace2 | ACE2 wiener | direction 2 | under | 12oct01e_00054eb.mrc | |
Ace2 | ACE2 phaseflip | none | over | ||
Ace2 | ACE2 phaseflip | none | under | ||
Ace2 | ACE2 phaseflip | direction 1 | over | ||
Ace2 | ACE2 phaseflip | direction 1 | under | ||
Ace2 | ACE2 phaseflip | direction 2 | over | ||
Ace2 | ACE2 phaseflip | direction 2 | under | ||
Ace2 | SPIDER phaseflip | none | over | ||
Ace2 | SPIDER phaseflip | none | under | ||
Ace2 | SPIDER phaseflip | direction 1 | over | ||
Ace2 | SPIDER phaseflip | direction 1 | under | ||
Ace2 | SPIDER phaseflip | direction 2 | over | ||
Ace2 | SPIDER phaseflip | direction 2 | under | ||
Ace2 | EMAN phaseflip whole image | none | over | ||
Ace2 | EMAN phaseflip whole image | none | under | ||
Ace2 | EMAN phaseflip whole image | direction 1 | over | ||
Ace2 | EMAN phaseflip whole image | direction 1 | under | ||
Ace2 | EMAN phaseflip whole image | direction 2 | over | ||
Ace2 | EMAN phaseflip whole image | direction 2 | under | ||
Ace2 | EMAN phaseflip by boxed stack per image | none | over | ||
Ace2 | EMAN phaseflip by boxed stack per image | none | under | ||
Ace2 | EMAN phaseflip by boxed stack per image | direction 1 | over | ||
Ace2 | EMAN phaseflip by boxed stack per image | direction 1 | under | ||
Ace2 | EMAN phaseflip by boxed stack per image | direction 2 | over | ||
Ace2 | EMAN phaseflip by boxed stack per image | direction 2 | under | ||
Ace2 | EMAN phaseflip by tilt location | none | over | ||
Ace2 | EMAN phaseflip by tilt location | none | under | ||
Ace2 | EMAN phaseflip by tilt location | direction 1 | over | ||
Ace2 | EMAN phaseflip by tilt location | direction 1 | under | ||
Ace2 | EMAN phaseflip by tilt location | direction 2 | over | ||
Ace2 | EMAN phaseflip by tilt location | direction 2 | under | ||
Ace1 | ACE2 wiener | none | over | ||
Ace1 | ACE2 wiener | none | under | ||
Ace1 | ACE2 phaseflip | none | over | ||
Ace1 | ACE2 phaseflip | none | under | ||
Ace1 | SPIDER phaseflip | none | over | ||
Ace1 | SPIDER phaseflip | none | under | ||
Ace1 | EMAN phaseflip whole image | none | over | ||
Ace1 | EMAN phaseflip whole image | none | under | ||
Ace1 | EMAN phaseflip by boxed stack per image | none | over | ||
Ace1 | EMAN phaseflip by boxed stack per image | none | under | ||
Ace1 | EMAN phaseflip by tilt location | none | over | ||
Ace1 | EMAN phaseflip by tilt location | none | under | ||
CTFFind | ACE2 wiener | none | over | 12oct01e_00059en.mrc | |
CTFFind | ACE2 wiener | none | under | 12oct01e_00055en.mrc | |
CTFFind | ACE2 wiener | direction 1 | over | 12oct01e_00037ea.mrc | |
CTFFind | ACE2 wiener | direction 1 | under | 12oct01e_00032ea.mrc | |
CTFFind | ACE2 wiener | direction 2 | over | 12oct01e_00048eb.mrc | |
CTFFind | ACE2 wiener | direction 2 | under | 12oct01e_00054eb.mrc | |
CTFFind | ACE2 phaseflip | none | over | ||
CTFFind | ACE2 phaseflip | none | under | ||
CTFFind | ACE2 phaseflip | direction 1 | over | ||
CTFFind | ACE2 phaseflip | direction 1 | under | ||
CTFFind | ACE2 phaseflip | direction 2 | over | ||
CTFFind | ACE2 phaseflip | direction 2 | under | ||
CTFFind | SPIDER phaseflip | none | over | ||
CTFFind | SPIDER phaseflip | none | under | ||
CTFFind | SPIDER phaseflip | direction 1 | over | ||
CTFFind | SPIDER phaseflip | direction 1 | under | ||
CTFFind | SPIDER phaseflip | direction 2 | over | ||
CTFFind | SPIDER phaseflip | direction 2 | under | ||
CTFFind | EMAN phaseflip whole image | none | over | ||
CTFFind | EMAN phaseflip whole image | none | under | ||
CTFFind | EMAN phaseflip whole image | direction 1 | over | ||
CTFFind | EMAN phaseflip whole image | direction 1 | under | ||
CTFFind | EMAN phaseflip whole image | direction 2 | over | ||
CTFFind | EMAN phaseflip whole image | direction 2 | under | ||
CTFFind | EMAN phaseflip by boxed stack per image | none | over | ||
CTFFind | EMAN phaseflip by boxed stack per image | none | under | ||
CTFFind | EMAN phaseflip by boxed stack per image | direction 1 | over | ||
CTFFind | EMAN phaseflip by boxed stack per image | direction 1 | under | ||
CTFFind | EMAN phaseflip by boxed stack per image | direction 2 | over | ||
CTFFind | EMAN phaseflip by boxed stack per image | direction 2 | under | ||
CTFFind | EMAN phaseflip by tilt location | none | over | ||
CTFFind | EMAN phaseflip by tilt location | none | under | ||
CTFFind | EMAN phaseflip by tilt location | direction 1 | over | ||
CTFFind | EMAN phaseflip by tilt location | direction 1 | under | ||
CTFFind | EMAN phaseflip by tilt location | direction 2 | over | ||
CTFFind | EMAN phaseflip by tilt location | direction 2 | under | ||
Xmipp CTF | ACE2 wiener | none | over | ||
Xmipp CTF | ACE2 wiener | none | under | ||
Xmipp CTF | ACE2 wiener | direction 1 | over | ||
Xmipp CTF | ACE2 wiener | direction 1 | under | ||
Xmipp CTF | ACE2 wiener | direction 2 | over | ||
Xmipp CTF | ACE2 wiener | direction 2 | under | ||
Xmipp CTF | ACE2 phaseflip | none | over | ||
Xmipp CTF | ACE2 phaseflip | none | under | ||
Xmipp CTF | ACE2 phaseflip | direction 1 | over | ||
Xmipp CTF | ACE2 phaseflip | direction 1 | under | ||
Xmipp CTF | ACE2 phaseflip | direction 2 | over | ||
Xmipp CTF | ACE2 phaseflip | direction 2 | under | ||
Xmipp CTF | SPIDER phaseflip | none | over | ||
Xmipp CTF | SPIDER phaseflip | none | under | ||
Xmipp CTF | SPIDER phaseflip | direction 1 | over | ||
Xmipp CTF | SPIDER phaseflip | direction 1 | under | ||
Xmipp CTF | SPIDER phaseflip | direction 2 | over | ||
Xmipp CTF | SPIDER phaseflip | direction 2 | under | ||
Xmipp CTF | EMAN phaseflip whole image | none | over | ||
Xmipp CTF | EMAN phaseflip whole image | none | under | ||
Xmipp CTF | EMAN phaseflip whole image | direction 1 | over | ||
Xmipp CTF | EMAN phaseflip whole image | direction 1 | under | ||
Xmipp CTF | EMAN phaseflip whole image | direction 2 | over | ||
Xmipp CTF | EMAN phaseflip whole image | direction 2 | under | ||
Xmipp CTF | EMAN phaseflip by boxed stack per image | none | over | ||
Xmipp CTF | EMAN phaseflip by boxed stack per image | none | under | ||
Xmipp CTF | EMAN phaseflip by boxed stack per image | direction 1 | over | ||
Xmipp CTF | EMAN phaseflip by boxed stack per image | direction 1 | under | ||
Xmipp CTF | EMAN phaseflip by boxed stack per image | direction 2 | over | ||
Xmipp CTF | EMAN phaseflip by boxed stack per image | direction 2 | under | ||
Xmipp CTF | EMAN phaseflip by tilt location | none | over | ||
Xmipp CTF | EMAN phaseflip by tilt location | none | under | ||
Xmipp CTF | EMAN phaseflip by tilt location | direction 1 | over | ||
Xmipp CTF | EMAN phaseflip by tilt location | direction 1 | under | ||
Xmipp CTF | EMAN phaseflip by tilt location | direction 2 | over | ||
Xmipp CTF | EMAN phaseflip by tilt location | direction 2 | under |
Files
Updated by Bridget Carragher about 12 years ago
The matrix is a good idea in theory but seems we will never get all these tests done in practice. So how about just picking some obvious ones and making sure those work. We certainly never did these exhaustive tests in the past. I would also assume for now during this "emergency" that all we should test are underfocus as that would be 99% of the images we take and I would just post a warning to users that we are unsure of the behavior for overfocus images.
Updated by Anchi Cheng about 12 years ago
This is for the future testing and the over/under only need to be done at the algorithm level and one image each.
I will aquire one image for each case for this today.
Updated by Neil Voss about 12 years ago
Currently, the pipeline is only designed to handle underfocus. Overfocus should fail, which may be a good test nonetheless.
Updated by Amber Herold about 12 years ago
- File 12oct01e_00054eb.mrc 12oct01e_00054eb.mrc added
- File 12oct01e_00055en.mrc added
- File 12oct01e_00059en.mrc 12oct01e_00059en.mrc added
- File 12oct01e_00032ea.mrc 12oct01e_00032ea.mrc added
- File 12oct01e_00037ea.mrc 12oct01e_00037ea.mrc added
- File 12oct01e_00041ea.mrc 12oct01e_00041ea.mrc added
- File 12oct01e_00048eb.mrc 12oct01e_00048eb.mrc added
- File 12oct01e_00050eb.mrc 12oct01e_00050eb.mrc added
Updated by Amber Herold about 12 years ago
- File 12oct01e_059en-pow.mrc 12oct01e_059en-pow.mrc added
- File 12oct01e_032ea-pow.mrc 12oct01e_032ea-pow.mrc added
- File 12oct01e_037ea-pow.mrc 12oct01e_037ea-pow.mrc added
- File 12oct01e_048eb-pow.mrc 12oct01e_048eb-pow.mrc added
- File 12oct01e_054eb-pow.mrc 12oct01e_054eb-pow.mrc added
- File 12oct01e_055en-pow.mrc 12oct01e_055en-pow.mrc added
Updated by Amber Herold about 12 years ago
- File 12oct01e_00055en.mrc 12oct01e_00055en.mrc added
Updated by Amber Herold about 12 years ago
r17167 adds testAce2.py to myami. It uses the standard python unittest class to run all the tests outlined in the Ace2 CTF Estimation test matrix and compare the results to what is expected.
### This class is used to test the Ace2 program. It will dowload the needed test images from the AMI website, ### then run ace2.exe and compare the results to what is ecpected. It will remove results files from a prior run ### when running subsequent tests. It does not remove the files after a test is run so that the usr can review ### the result files if needed. Any functions in this class that begin with "test_" is a seperate test that ### will be executed by unittest. ### To run this test: ### 1. Create a temporary test directory and change directories to it. ex. mkdir ace2test, cd ace2test ### 2. run this file by providing the path to it. ex. python ../workspace/myami/appion/test/testAce2.py ### 3. review the results printed to the terminal.
Updated by Amber Herold almost 12 years ago
- Status changed from Assigned to Closed