Leginon

There isn't supposed to be Falcon4 or TEMScripting/Advanced Scripting changes that would affect speed in the TUI versions you have. But to be sure, let's try a few basic tests here.

Please run these two tests from pyscope subpackage on the TEM computer and post the results here.
1. Test camera acquisition: The test script is under pyscope/testing. falcon4ec_speed_test.py
2. Test tem function SET speed: In the same directory, run scope_functions_test.py There will be some failure since it is an automated attribute method test. It will give out set time on the set methods.

Please also send screenshot of PresetsManager Logger that covers two preset changes while collecting tilt series. If there is a suspicious slow down, we are likely going to see it there.
Without setting changes, Tomography node timing should follow the results of the two tests above since the alignment/tracking algorithm is only python dependent.

Hi Anchi,

1)

C:\Users\Supervisor\Downloads\Leginon\trunk_June2023\pyscope\testing>C:\Python27\python.exe "falcon4ec_speed_test - Copy.py" 
Falcon Image Storage Server Path is  Z:\TemScripting\EF-Falcon\
Scheduled job ATS20230707120124 for camera EF-Falcon
('returned image shape', (8, 8))
('returned image mean', 3999.9998026171875)
image at binning 1 took 5.521000 seconds
1666
Scheduled job ATS20230707120130 for camera EF-Falcon
('returned image shape', (8, 8))
('returned image mean', 3999.9998026171875)
image at binning 1 took 2.479000 seconds
1666
Scheduled job ATS20230707120132 for camera EF-Falcon
('returned image shape', (8, 8))
('returned image mean', 3999.9998026171875)
image at binning 1 took 2.513000 seconds
1666
Scheduled job ATS20230707120135 for camera EF-Falcon
('returned image shape', (8, 8))
('returned image mean', 3999.9998026171875)
image at binning 1 took 2.488000 seconds
1666
Scheduled job ATS20230707120138 for camera EF-Falcon
('returned image shape', (8, 8))
('returned image mean', 3999.9998026171875)
image at binning 1 took 2.502000 seconds
1666
2.49550002813
0.111000061035
press return key to quit

2)

C:\Users\Supervisor\Downloads\Leginon\trunk_June2023\pyscope\testing>C:\Python27\python.exe scope_functions_test.py
Loading EFGlacios
Error testing getFlashingAdvised: getFlashingAdvised() takes exactly 2 arguments (1 given)
setMagnifications
time (s): 0.000000
setBeamBlank
time (s): 0.000000
setBeamShift
time (s): 0.012000
setBeamTilt
time (s): 0.007000
setBeamstopPosition
time (s): 0.000000
setColdFegFlashing
time (s): 0.000000
setColumnValvePosition
time (s): 0.000000
setCorrectedStagePosition
time (s): 0.000000
setDarkFieldMode
time (s): 0.001000
setDiffractionShift
time (s): 0.009000
setDefocus
time (s): 0.105000
setProjectionMode
time (s): 0.000000
setEmission
Error testing setEmission: Error testing setEmission with setEmission: gun emission state can not be set on this instrument
setFocus
time (s): 0.004000
setGunShift
time (s): 2.761000
setGunTilt
time (s): 2.748000
setHighTension
time (s): 1.373000
setImageShift
time (s): 0.015000
setIntensity
Error testing setIntensity: Error testing setIntensity with setIntensity: 'EFGlacios' object has no attribute 'need_normalize_all'
setMagnification
time (s): 0.001000
setMainScreenPosition
time (s): 0.001000
setProbeMode
time (s): 0.000000
setProjectionMode
time (s): 0.000000
setRawImageShift
time (s): 0.018000
setSpotSize
time (s): 0.000000
setStagePosition
time (s): 6.794000
setStigmator
time (s): 0.021000
setProjectionSubModeMap
time (s): 0.001000
----------------------
Number of error found: 3
Finished. Hit return to quit

3) See attached.

What are the main changes that occur when implementing support for newer TEM server versions? I wonder if an update by TFS may have changed the network parameters or computational overhead on either the MPC or SPC.

Best regards,
Trishant

Here is a partial result from a Glacios with Falcon4 TFS Talos software 2.14.1

!Screenshot 2023-07-10 at 8.02.01 PM.png!

First, I can see that SetStagePosition took longer on your scope than mine. this could indicate some problem in the API. The scope_function_test.py just send the same stage position as it just got, so it should not take long to do. However, I don't think this is not the main problem.

Second, from your PresetsManager node logger, I see that you have cycling presets on in this node since it goes through all presets before setting the final value. Each of the preset change takes several seconds, indicating that lens normalization was taken place. This adds up a lot of time. My guess is that before the TUI upgrade you have deactivated all auto lens normalization in TUI and used Leginon preset cycling to remove hysteresis. TFS engineer probably turned it on so now you have the added lens normalization that duplicates Leginon's preset cycling effect.

You can either turn off all lens normalization again, or remove preset cycling. These two do very similar thing but the state they settle in might be slightly different, so you might need to realign the presets after the changes. You can try both to decide which you like better.

You should also take a look of this https://emg.nysbc.org/issues/14158 about the expected timing.

Hi Anchi,

Changing FORCE_NORMALIZE_ALL_AFTER_SETTING from True to False seems to have fixed this. Curiously, there is no longer the option to change auto lens normalization in our TUI using the Supervisor account.

Thanks for the help.

Trishant

Hi, Thishant,

Glad to know it works. Please do read description of #7076 and make sure that your scope does not have side effect from changing the configuration. New version TUI might have hidden the option from everyone except service since TFS no longer expect even superuser to go through column alignment procedure. It is a good thing since the column and environment are stable with minimal use case difference.

Hi Anchi,

Sorry to open this issue again but after some more time collecting data, I've found other causes for throughput loss.

Targets are being skipped due to move errors in navigator.
The Navigator Target Tolerance is 1e-08 m, the Navigator Acceptable Tolerance is 1e-07 m. Larger acceptable tolerance somehow caused off-target acquisition (4096*0.19 nm = ~780*780 nm field of view, 63,000 nominal magnification)
The tomography node, if skipping a target, displays the error "Move failed. skipping acquisition at this target". This will occur after a message in Navigator "error got worse".

It would be interesting to know if this is solvable in Leginon or the problem stems from a problem associated with TEM server or drifting stage/optics.

It's worth noting that I had to switch Leginon's own preset cycler back on to even be able to achieve consistent preset alignment.

The facility manager is on leave until later in the week, so I may update on any hardware related information then.

Best,
Trishant

Since exceeding the Navigator Acceptable Tolerance is meant to signal abort, this is the correct behavior of the software. There are two possible reasons for you to have the problem often, neither has a software solution within standard leginon, I am afraid.

1. Stage not behaving according to the assumption we made "The closer the distance it needs to move, the more likely it will get there".

You can use Navigation node to find at which distance, the targeting (click on the feature to move it to the center) assumption fails on your hl image mag. You should then set Navigator Target Tolerance around or slightly larger than that value. By doing so, you reduce the number of trials the iterative move process do. If it turns out to be around 1e-07 m, you should probably ask to have the stage serviced.

2. The correlation peak is too weak for leginon to pick up consistently. Watch the correlation panel and the location of the correlation peak Leginon found in Navigation node while it is making the iterative move. The iterative move use a predictive local correlation that consider about half of the length of image instead of the whole so that it gives strong correlation when the movement is large. If the target you pick is around a featurless area, there is more chance of it be picking up the bad peak. We've found that it fails more when there is a strong gradient in the image. If you find a filter that will help improving the success rate, please let me know. We should add it.

It is unfortunate that you still need preset cycling on top of the auto lens normalization.