Leginon

numextension.allstats missing problem on Windows PC is caused by the numextension for py2.5 is older than the trunk requires. Jim will need to recompile this.

Error sending request error is not very specific, could you print the request in leginon/datatransport.py in the function _send before it raises TransportError in the last line of the function?

Please find the Request object was causing error

Origin: Manual
Node: Instrument
Name: TIA
Attributename: Image
Type: r
args:
()
kwargs: {}

Jian,

I just convinced Jim to work on this immediately. We have two possible route. One, use SerialEM plugin directly which you said works already, Two, compile the new version of numextension in python2.5 that you have. Each has pros and cons. Jim is sure the current problem you have is just numextension version. but we've lost the computer where we had the old visual studio for it, so we will have to recreate it. The SerialEM plugin would be good for the future because we don't have support that part, but there is more rewriting code which we can't test ourselves.

For the short term, we will try option two.
Could you give us specific on your Windows PC that Falcon and TIA is run from?
1. Windows version (including 32 or 64-bit version specification).
2. Python version

new numextension (current svn revision) for python 2.5 is now available. click "files" tab to find it.

Hi Jim,

I updated numpy to 1.7 on Windows PC, and start Leginon main and was able to take images from Falcon camera without problem. Great! Thanks,

But when I try Leginon main from Linux PC, I got following errors. It looks it is due to old scipy version. But I can't find right rpm for python2.6.6 since I am runing CentOS 6. I saw you generate an install package for Ubuntu, can you create a rpm of scipy 0.10.1 for CentOS 6 as well. Otherwise I think I have to wipe the CentOS and move to fedora.

********Error Message*********************************
[cbis@LeginonCBIS Jian]$ start-leginon.py
/usr/lib/python2.6/site-packages/pyami/tifffile.py:903: UserWarning: Failed to import _tifffile.decodepackbits
warnings.warn("Failed to import %s" % module_function)
/usr/lib/python2.6/site-packages/pyami/tifffile.py:903: UserWarning: Failed to import _tifffile.decodelzw
warnings.warn("Failed to import %s" % module_function)
Leginon version: pre3.0
Origin: Manual
Node: Instrument
Name: TIA
Attributename: Image
Type: r
args:
()
kwargs: {}

The error message is exactly what it said. tifffile module does not have what it needs to decode the said compression. However, since tifffile is needed only for reading tiff raw frame from DE series camera which is not compressed, the error message has no impact on any Leginon/Appion operation. You don't have to do anything about it. CentOS 6 default numpy scipy should work fine.

Anchi,

But I still got "Error acquring image: Error sending request" message same as attached falconLeginonOnLinux.png in this thread. And no image is displayed in Leginon-Main window. I don't have any problem while running Leginon-Main on Windows PC.

Jian

Before when you had the "error sending request", it was also showing the error about numextension on the Windows client. With the current "error sending request" is there a corresponding error on windows? Also, can you try some other functions, such as getting and setting presets in one of the other applications (calibrations or MSI). I wonder if it is only image acquisition that is having problems, or other functions as well.
Jim

Well, as I can remember according to post 2 in this thread, the windows error and Linux error are not related, I was running leginon main on either OS respectively. So I think you have solved issue 2 I listed in post #2, and now we down to issue 3 in post 2. I also managed to print out the request object in datatransport.py._send(), in post 4 as Anchi suggested. Please suggest more test I can run for diagnosis. I will try other application, but I thought image acquisition is shared by all applications.

Jim and I are writing this together. He thinks that it may be a network problem since it works on PC (I assume it means that you get an image through Leginon and it displays in its gui).

Linux tests:

• This will rule out general installation problem.

• If there is a problem communicating, it may show an error at this stage.

• This is to see if the size is too big to transfer over.

If none works, run leginon/syscheck.py and give us the output that shows the
version of the packages, and check the numextension that you installed. It should
have allstats function. If in doubt, just reinstall it with setup.py.

If all work, there is something about the image gotten from Falcon makes it not
transferable. Please save an image through Leginon on the Windows PC and attach that here.

So it seems to be something about the size of the image. It seems to be cutting off the data stream before it sends the whole image over the network.
Next thing to try on the linux side, make an edit to leginon/socketstreamtransport.py:

At line 167 there is a try/except block:

    try:
      result = pickle.load(sfile)
    except Exception, e:
      raise TransportError('error unpickling response, %s' % e)

    try:
      result = pickle.load(sfile)
    except Exception, e:
      raise
      #raise TransportError('error unpickling response, %s' % e)

Jian,
In addition to the debugging in the previous post, I have one more question and one more debug test.

As I mentioned above,I thought it was the image size that causes the problem, but are you able to get a 4k x 4k image from a Gatan camera? (assuming your Gatan camera is 4kx4k, right?) If you can get 4k image from Gatan, but not from Falcon, then there is something else particular about the Falcon image. My next guess is that it is a combination of the size and the type of the image. Can you make a modification in your pyscope installation on the Falcon Windows PC. I want to have it print the type of the image array. In pyscope/tia.py, line 178, the original code looks like:

      arr = comarray.prop(self.im.Data, 'Array')
      arr = numpy.flipud(arr)

      arr = comarray.prop(self.im.Data, 'Array')
      print arr.dtype
      arr = numpy.flipud(arr)

Jim

Jim, you are right, Falcon image IS 32bit, the counts can be way over 65000.
I will try your debug code next week. Thanks,

I just want to summarize the information I need to understand the problem, so can you fill in the following when you have time:

1. the error that shows up from the test in post 15 above (probably a pickle error)
2. The data type that is printed from the test in post 16 above (32 bit int or float?)
3. exact Windows version (XP, Vista, 7, etc...), and I assume 32 bit, but want to make sure.
4. exact version of Python you have on Windows: 2.5.? (need the minor number)
5. exact version of Python on linux. I saw 2.6.6 in post 8 above, but want to make sure this has not changed.

I was able to do a simulation here with a 32 bit 4kx4k image and did not have the problem you are having, so with all of the above information you provide, I should be able to simulate this more accurately.

An observation may or may not related to this issue: that once I select TIA in the camera list for the first time in imaging setting window, the correct camera size (upto 4K for Falcon) will not be initialized until I close the image setting window and reopen again.

That is an unfortunate known issue that has to do with gui updating. We see that with other camera as well but should not be related with the transfer.

Hi Jim,

I downloaded and installed the latest from trunk, version 17558. I tried the test with different size from 1024 to 8096, but no error comes out on either PC.

------On Windows PC, the terminal shows:

import socketpickle

Running Server
host: Titan52331950
port: 55555
Handling request...
Size requested: 2048
Sending result ...
Handling request...
Size requested: 4096
Sending result ...
Handling request...
Size requested: 8196
Sending result ...

------On Linux PC, the terminal shows:
python socketpickle.py Titan52331950 8196
Connecting to server...
Sending request...
Getting result...

++++But terminal won't exit without further output at this stage until ctrl+C. Then the following message show up.

^CTraceback (most recent call last):
File "socketpickle.py", line 72, in <module>
run_client()
File "socketpickle.py", line 65, in run_client
im = c.getImage(size)
File "socketpickle.py", line 47, in getImage
result = pickle.load(sfile)
File "/usr/lib64/python2.6/socket.py", line 383, in read
data = self._sock.recv(left)
KeyboardInterrupt

However, I still have the same problem when I tried get 4K image out of Falcon in Leginon MSI-T.

Jian

Jian, Here is the next thing to try:
On the Windows computer, edit leginon/socketstreamtransport.py
At about line 108, there is a try/except statement. We need to make sure the exception is raised, so add an immediate raise in the except clause before it does anything else:

    try:
      pickle.dump(result, self.wfile, pickle.HIGHEST_PROTOCOL)
      self.wfile.flush()
    except Exception, e:
      raise

Hi Jim, Anchi

Anything else I may try? The system quite free in May, so it is good opportunity.

Jian

Jian,
How much memory do you have on your Windows machine? There is a slight chance that the error is caused by system running low on memory while trying to send the image across the network. If it is easy for you, you could try adding more memory to the Windows machine. This is only a guess, and it may not help. Otherwise, I am working on a software solution that does better buffering of the data that is sent to the network. I will send you the new code when it is complete.
Jim

Jian,
I just made a new version of leginon/socketstreamtransport.py (r17585). It has a parameter at line 126 called "wbufsize" which is set to a default of 0 (no buffering). Can you install this new file in leginon on your Windows machine. Then try different values for the buffer size. Each time you modify the value in the file, restart Leginon and try to acquire a 4kx4k image across the network.

First try setting it to -1, which allows lower level code to select a default value. If that does not work, try setting it to 16777216 (size of 2048x2048 image, which seems to work). Otherwise, try some smaller values to see if anything works.

Let me know if you find a value that works without giving you the "no buffer space available".
Jim

Hi Jim,
I installed latest trunk on windows PC and tried different value of wbufsize from -1, 2kX2k,1kX1k, 512x512, 128x128, none of them worked :(
The memory of microscope PC is 3.5 GB. In fact, the microscope PC communicate with Falcon through TCP/IP protocol as well. I am wondering if it could be culprit which occupy all the buffer resources while taking 4k images. And cause no problem while Legion running on the microscope PC.

"10055 No buffer space available" seems quite common on google, I will try to dig some possible solution out.

Thanks,
Jian

This makes me think that setting that buffer size in high level code is still not having an effect on the low level buffering. I may have to dig deeper into the python socket code to override buffering. We can also just create the pickle into local memory, then manually break it into small strings that we send over the socket.

Jian,
I determined that the parameter "wbufsize" that I had you test is not effective because it only specifies a minimum buffer size. If we send a large image, than it still tries to use a buffer that is the full size of the image. Since there is no easy way to restrict it to a maximum buffer size, I made the change to just break up the data and make each write to the socket be a smaller chunk.

Please update leginon/socketstreamtransport.py from svn on both windows and linux and try it now. There is one parameter you can set at the top of that module: CHUNK_SIZE. The default should work for you, but you can always try smaller values if necessary.
Let me know how this works.
Jim

Yeah! It is working now!What's next

That is great! There is one more thing you can test if you have a chance:

In the latest revision of pyscope (starting with r17589) there is a new tia.py module. This one is completely different because it no longer uses the comarray package. Instead it requires a different package called comtypes. If you want to try this, make a backup of your tia.py, then install the new one from svn. Then install comtypes as described here: http://emg.nysbc.org/projects/leginon/wiki/Installing_comtypes
The instructions also explain that you need to make one modification to comtypes.

Let me know how that works.
Jim

And try to take some images through Leginon and its gui and let us know if we need to modify gain/dark reference requirement etc.

Anchi

In terms of gain/dark reference, Falcon manages its own references through TUI and user is supposed to take references everyday using their image condition. The final images appear on TIA is already corrected by Falcon reference image manager.
I tried to take images through Leginon as well, the result seems to me that no need to take dark/gain references in leginon since it looks same as the one in TIA, which is corrected already.

Jian,

Regarding the correction, We will make a feature to turn off correction but it will take a bit of time. In the mean time, I think only a handful of places in Leginon requires correction or it would not run. Most places probably just give a warning. If you run into situation that absolutely demanded correction on Leginon side, let me know.

As for pyScope Error. I am a bit confused. NewpyScopeError.JPG on PC shows that you put all nodes on that computer. Is that how you intend to run? Or is it a test that you would normally have the main on Linux which gave you the other error.

I am not sure why TEM is not showing up, but this is probably why all the other errors happen.

First try a basic pyscope test to see if you get any errors when initializing the TEM. On the TEM computer, start a python command line and try this:

import pyscope.tecnai
t = pyscope.tecnai.Tecnai()
t.getMagnification()

import pyscope.tia
t = pyscope.tia.TIA()
t.getImage()

If the basic tests do not give an error message, next try Leginon again, but first edit the instruments.cfg and only configure the TEM, not the cameras. So put # in front of all the camera lines to comment them out. Then start Leginon and see if the TEM is initialized. Then try again with each camera put back into the instruments.cfg. Maybe this will determine what is causing the problem.

Jim

Can you attach here your old tecnai.py (the one that works). I am trying to figure out how old it is.

Also, what happens if you install the new pyscope, then run the pyscope/updatecom.py script. Make sure to run the updatecom.py in the new pyscope. Maybe that makes it work right?

Jian,

I guess my ctf estimation algorithm used in beam-tilt coma calibration is still buggy. Try doing the calibration with a smaller image if Falcon allows or just skip it now. I will debug it when I get time. Please give me your numpy scipy version. I think the bug might be version dependent.

As for MagCalibrationError, you just need to do it with the "Scale and Rotation Adjustment Only" UNCHECKed to get rid of that error. At least one model is needed before you can active that option.

It looks like you've gone past pscope error. Is this r17589 or the current ? Are there remaining problem in acquiring or transferring image?

numpy version: 1.4.1 Scipy 0.7.2-8.el6

I just realized that I only did stage model at 480x with Orius camera and Falcon is not allowed to take images in LM range, so I got the MagCalibrationError when I tried to model the stage for hl preset using Falcon.

I am still running r17589, not the new version using comtype.

The latest pyscope works fine now. It was my bad. I had done before but forgot this time: that is I had to copy TecnaiCCD.dll and stdscript.dll into pyscope folder before run updatecom.py. Then all instrument working now.

But there are errors in 17652 when I try to load calibration application. Pls see attached pics. Version 17624 works fine.

In addition, I am running calibration on EFCCD, but find that I can't update the energyfilter either to microscope or from microscope.

Jian,
The energy filter problem sounds similar to the same problem a few years ago in this issue: #1166
Try running the gatantest.py script that is attached to that issue to see if the energy filter functions are found in the scripting interface. Maybe the names have changed again?
Jim

Correction about my estimated sq image area requirement. We have used ~ 30 um in RCT sq and it is fine, too.

Jian,

This is very encouraging.

1. I didn't know that when it was mentioned before that "Falcon could only be used in SA mode" meant that it could get down to 1700x. I thought we would be stuck at ~ 5000x. This gives us a lot more flexibility.

2. Do I understand correctly that Orius pushes Falcon out when you set the camera to Orius in your test script and then tell it to insert? If we then set the camera to Falcon, getInserted() would retrun False, right? Do each camera retract on its own? In other words, If I want to make sure both are out of the beam path when Falcon is already inserted, do I need to go through the following:

detector.ccd.Camera = 'BM-Orius'
detector.setInserted(True)
detector.setInserted(False)

detector.ccd.Camera = 'EM-Falcon'
detector.setInserted(False)

3. How did you find out the name string used for setting the camera?

4. With your success, we can safely say that it is very likely we can get Falcon to work with Orius once we find the right sequence to insert one and to retract one or both. Therefore the last remaining hurdle is the protector program. Could you describe what the issue there is? I need some help refreshing my memory about the details there. Thanks.

Jian,

Thanks for the update. Very good news indeed. Here are my comments.
(1) We will have to do some work with your help on this.
(3) You can set available binnings in tia.py for Falcon as follows:

class TIA_Falcon(TIA):
camera_name = 'BM-Falcon'
binning_limits = [1,2,4]

Have a good time in Taiwan.

More update on testing MSI-T on quantafoil grids.

1) I can calibrate beam-tilt and done the defocus measurement well in beam-tilt node. But during the auto-run, I always got blank images in focus node so that the focus always failed. In focus simulation, I sometime got black images and sometime normal images. I think some pause is needed but not sure where yet.

2) In general, phase-correlation does not work well, such as Z-focus, hole targeting. The correlation is very smeared and hard to be found correctly. I am not sure it is due to the grids or property of falcon, but the grids seems typical to me. I can send some images for your guys to tune correlation algorithms.

3) For micro/nano-probe, it is available in stdscript.dll as
"Instrument.Illumination.Mode" "Mode of the illumination system (either nanoprobe or microprobe). (Nearly) no effect for low magnifications (LM)."
See temscripting.chm for description. It should be easy.

Thanks,
Jian

(1) I see that we don't have any gui chanable sleep time set for autofocus step. You will have to try to do this in the code of leginon/focuser.py for now. Change the assignment of "settle" variable in this line (currently at 0.25 second):

correction = self.btcalclient.measureDefocusStig(btilt, correct_tilt=True, correlation_type=setting['correlation type'], stig=setting['stig correction'], settle=0.25, image0=lastdriftimage)

(2) Yes, please send images.

(3) We have added nanoprobe/microprobe call in pyscope and tested it. Next is to add them to preset and database.

There is one more issue I failed to mention, say I have gr at 100x, Orius camera, sg at 1700x, falcon camera.

Because I found the sg targeting wasn't very accurate based on atlas of Orius images, I choosed to use Navigator to center for gr imaging. However the very first correction gr image out of Orius was always blank, then the second and later ones were OK. Surprisingly the Navigator still runs and most time get gr target correctly. I increases the pause everywhere on interface, but the issue still persist but only limited to the very first one.
In addition, align-preset process between sg and gr also have the same issue, maybe due to some shared code. I am not sure where I should insert some pause in the code.

Anchi, the Krios will start autofill about every 12hours, it would be nice to pause data-collection (particularly for tomography) during autofill, that takes ~10-20mins. Maybe you can open another thread for this feature. I don't know it have been done already.

I will see what I can repeat here on our K2/Orius combo. We are talking to orius through DM in this case, though.

Autofill pause has been asked multiple times so I created an issue already yesterday #2520

I played a bit with Orius/K2. Orius can only insert when K2 is not inserted. Therefore, it behaves the same as in your case, I think. I put Orius zplane smaller than K2 as in Instrumentscfg_for_22, then I have no problem in Navigation nor in Preset Alignment tool.

If you only have problem with the first image, it is possible that tia.py setInserted function returns before it is done insertion.
I committed pyscope/tia.py as r17902. You can play with the sleeptime in the function "setInserted" to see if you can give enough sleep time allow Orius to complete its insertion.

My guess for Navigation iterative move not failing on the first black image is that it would just give random error and continue with the iteration to get the next image in an attempt to improve it.

Beam tilt coma calibration needs very consistent ctf estimation and not to fail with large astigmation. 95% success rate is the minimum. I will write down my best protocol for this. We have not used the calibration much ourselves, though. And I hide the calibrations and its application in the 2.2 release because it is not reliable enough for me to teach others. Our coma-free alignment is usually done through Beam Beam Tilt Imager manually looking at Zemlin tableau. If we want to continue on this dicussion, it is best as Forum thread so others can find it easier.

I looked at the Falcon images you sent, they do look rather hard. Will need a bit of time on that.

Sleeptime worked. I don't have black image issue anymore in camera switching scenarios. Align-preset and Navigator all works fine now.

I think we have completed the job for the purpose of this thread. Only thing left is the cross-correlation of hl images. I can mitigate it with more dose, larger defocus and smaller tilt.

Jian,

Great. Give me the sleep time you finally use so that I can commit them.

I just used 5, sorry didn't test other values. I got frustrated of bad compustage today.