Appion

8/27 Appion cofnerence call: Scott has a working prototype at FSU. Could work at TSRI but needs different geometric transformations. Scott hasn't been able to figure this out yet. Created particle and whole frame alignment. Need launcher web page too.

Scott,
I may have time to implement the launcher in September if you have trouble finding the time.

Amber, here is an example of a command line. There are a ton of other options, but they are all advanced options. If you can make a page that has normal and advanced options, I can fill in the rest.

de_aligner.py \
  --runname run7 -R /data/lustre/appiondata/14may19b/framealignment/run7 \
  --stackid=971 --projectid 20 --darkreference_transform=3 --gainreference_transform=3 \
  --defects_columns=690 --defects_rows=1093,1094,3369,3370,3371 \
  --alignment_correct=1 --boxes_transform=2 --radiationdamage_compensate=1 \
  --radiationdamage_exposurerate=2.92 --radiationdamage_apix=1.25813 \
  --output_rotation=90 --alignment_target=boxes

Scott, where in the pipeline menu should this be added to?

It depends on stacks, so I would make it under the stack menu

Can I override that and suggest it goes in the direct detector tools section?

After talking to Anchi, it sounds like the new aligner does both particles and images, so it may need to show up in both GUI locations. The addition to makestack is similar to what needs to be done to add Relion movie processing, however I have not done that yet, and it would take a somewhat long while to do. Makestack is a beast on both the GUI and python side.

I will bow out of this one since it will take much longer than the 1 day I have left, and wish you good luck Scott! Hopefully Anchi will be available to help.

There are actually two different scripts. One depends on a preexisting particle stack and aligns frames of particles, and the other processes whole frames. These scripts are somewhat philosophically different than Anchi's scripts, and so for now I'd like them to have separate launcher pages. They really won't be nearly as complicated as you outline above. If you can set up the particle frame alignment page (at least the outline of it), I'm sure I can finish it up relatively quickly. The launcher page could go under the stacks heading or the direct detector tools, but keep in mind the particle aligner depends on a preexisting stack of particles which is why I was suggesting it go under the stack heading.

Scott,
I've added a GUI for de_aligner.py based on the command you listed above. I put it under "more stack tools" in the stacks section of the pipeline. The location can easily be moved if needed. See r18596 for the code.

Scott ran into file server problems. Too slow at FSU. Wants to use HPC but can't get job launcher too work. Started appionpbs loop class instead of appionloop2 that enables a new method for submitting to a cluster. The frame alignment script will use this.

I just added the appionPBS.py module to the repository. I forgot to add the associate the revision to this thread. I don't know how to add it after the fact, but the revision is 18823. I'm pretty excited about the new module. I have it working on a test copy of makeDEAlignedSum.py, and it has increased the throughput for aligning frames, by as many processors I have access to on the cluster. The nice thing is that it is launching single processor jobs, so they start super quick. I think we can use this for parallelizing a lot of different processes.

I got the DE aligner script to work at NYSBC and added a web launch page.

added support for sym links and better support for serial mode.

The script requires EMAN2. It is in my environment on guppy. Sargis, if you become my user, you can see where it is installed. The script requires the DE_aligner python script to be in both your PYTHONPATH and your PATH. It also requires that the script be called deProcessFrames.py just like you figured out. I made a module file in my environment that sets it up properly.

Anchi, that is fine with me. There are still some things that need to be worked out with appionPBS, but I'm happy to see how you modify it.

committed wrapper prepend in r19079.

Scott,

What does checkJob wait for ? an output file that gets added in the scratch directory named image.sh.xxxx ? Doesn't this depends on the job management system ?

It looks for the job outfile. It should appear in the directory from which the job was launched. Right now it is specific for PBS or MOAB clusters because that was all that I could test on. I'm working on adding the SLURM option this week. I will likely change this part substantially in the next weeks. Instead of checking for the job outfile, I'm thinking of having the job create a unique done file that checkJob would look for. I chose to have it look for a file instead of interacting with the resource manager because the manager sometimes gets overloaded and that could cause the whole thing to fail.

The python part now work at nramm cluster with commit as run and aligned image with r19080 and r19081 changes

r19084 add mechanism to shorten defect column and row list to within the cropped area of the border. Otherwise, my testdata with all border taken out takes too long (54 min) to process, instead of (17 min). Scott, is 17 min per image reasonably matchs your experience ?

The border should not be included in the bad rows and columns. It doesn't affect the frame alignment and it is cropped out later. I'm going to crop that part out of your code if that's OK. With ~40 frames, it takes us around 5 minutes, I believe. I can get you a better estimate after I get slurm up and running. I should be committing that later today.

Scott,

Please don't take that code out. It is doing the opposite.

My Leginon correction plan in the test dataset includes columns and/or rows in the border part. For example, column 0 and 5119. With border option, DE program really does not need to include that, which is why I want to filter them out before feeding into DE program.

Oh, I see. I'll leave it in.

Hey Anchi and Sargis,

Since Anchi added an option to automatically determine the bad rows and columns, I'm wondering if I could get one of you to add an advanced section to the web launch page and move the defective rows and columns options to the advanced section. I would do it myself, but I don't know how. Once I see how it's done, I can fill it up with some other options.

r19086 put defects in advanced display. I also add standard appionLoop on the left side.

r19090 filled out job submission web page with relevant options. Note, file handling is set to copy. NYSBC users will want to set that to link. It will work with copy but will be slower for them.

r19093 adds wrapper parameter and default it according to config.php. Also found better way of setting selection parameter based on what is done for presets in appionLoopTable.inc

We are having a problem that I don't know how to fix. The problem has to do with database replication. My dbemdata database is replicated between two different hosts. When the frame alignment script inserts the new resulting image into the db on one host, that entry isn't getting replicated to the other host. We don't have this problem for data collected with Leginon. My system admin has seen this type of problem before and thinks the problem is that the database to which the data is being inserted isn't explicit enough, so it isn't being replicated. In other words, if we were to use the regular mysql interface like so: use dbemdata; insert ...; then the entry would be replicated, but if dbemdata wasn't explicitly selected, the entry wouldn't be replicated. He said the concept is analogous to the difference between a full path and a relative path. For this script, I think it has to do with sinedon making inserts into both the leginon and the appion database. I don't know how to get this working properly. Sargis or Anchi, do you have any ideas?

Scott Stagg wrote:

We are having a problem that I don't know how to fix. The problem has to do with database replication. My dbemdata database is replicated between two different hosts. When the frame alignment script inserts the new resulting image into the db on one host, that entry isn't getting replicated to the other host. We don't have this problem for data collected with Leginon. My system admin has seen this type of problem before and thinks the problem is that the database to which the data is being inserted isn't explicit enough, so it isn't being replicated. In other words, if we were to use the regular mysql interface like so: use dbemdata; insert ...; then the entry would be replicated, but if dbemdata wasn't explicitly selected, the entry wouldn't be replicated. He said the concept is analogous to the difference between a full path and a relative path. For this script, I think it has to do with sinedon making inserts into both the leginon and the appion database. I don't know how to get this working properly. Sargis or Anchi, do you have any ideas?

I don't have experience with master-master replication. I did master-slave replication when we were upgrading our db server.

I came across this link about Replication Features and Issues https://dev.mysql.com/doc/refman/5.6/en/replication-features-create-select.html

Now that you mentioned it, our auto installer was having similar issue; it was trying to insert without selecting the database. Maybe you can try older version of mysql (<5.1) or the latest version to see if that helps. I'll try to get to the bottom of this issue with auto installer when I get a chance.

I fixed the replication error that we were having. Like Anchi suggested, all it required was inserting the leginondata separately from the appiondata. Anchi, can you merge this into the 3.2 release if you find the change satisfactory?

Looks good. Will merge to 3.2 by tonight. Can we close this issue as a completed 3.2 feature ? You can open a separate one for per-particle implementation if needed.