Leginon

Try saving frames when the binning is 1, use the full sensor area as well. When we had K2, frame saving only works with full size unbinned image even in DM.

We did use the full sensor area and binning 1. I just double checked. The only change is the we are using super resolution now and before we were using counting. Could it be a timing issue since it takes almost a minute to save these files to the other server?

Found a bug in my bin scaling. committed as r17476. Just pyscope/gatank2.py is changed. Sorry about it. It might even fix the binning problem in super resolution.

Where can I find the new gatank2.py?

under Repository tab of redmine, or directly http://emg.nysbc.org/projects/leginon/repository/revisions/17476

By now, redmine also added a link where I wrote r17476 in my message.

I just tried it and it is still not saving frames. I will look over more carefully tomorrow.

Why is there always a second binned image after the frame saving one in your log? Are you doing defocal pair? If so, try doing only one image. It maybe as you said a timing issue since the second one would set DM to non-frame saving, possibly before the frame images were saved. The values passed to DM all look fine.

You should also try to use the same parameters and acquire in dose fraction mode in DM gui in case that there are bugs related to the new version.

We have been collecting with ef removed from the exposure node and the result is no frames saved. I then added ef back so that it would collect it with the hope the timing would work if it was collecting an ef after the en. That did not work either but it saved a non frame en and ef. I then eliminated ef from the presets all together thinking if it was not there it would not switch formats. This time it did not save frames and also did not save a non frame en file either. I think that Leginon is switching formats from fraame mode en to non frame mode ef when ef is present and that image is saved although I am not sure what the dose for that image would be.

As for using the same parameters in DM, this is how we are collecting the data manually already so we know that this works.

I strongly suspect a timing issue since frames were saved previously when we were using counting mode and collecting 10 frames the frames were saaved along with many empty folders. Now that we are using super resolution and 20 frames the frames are not saved and as I stated above the switch over to non movie mode happens if there is a ef present and a en is savd without frames. Is there a command that checks if frames are saved before issuing the next preset setting? I will check when I get some more scope time when this problem occurs i,e, what is the data transfer limit.

We are out of idea for you to try here. Jim went and bought a new computer yesterday and is setting up a simulation machine with DM and SerialEM plugin. Hopefully we will be able to reproduce it. If it is purely a timing in term of giving next command too quickly, you should be o.k. if you only ask for one image with saved frames.
There are some plug-in function for getting the file save result back. We should be able to see what it does from the simulator.

1. Go back to basic: saving the frames on Gatan's local disk and save only one frame.
2. Since we are saving frames as individual files in a folder, you can try not to do so. Maybe it will reduce the time spent.
   Although I can't tell you if it will breaks right there without testing it here:

The boolean that determines this is in pyscope/gatansocket.py and the function SetupFileSaving
Change

bools = True # filePerImage=True

bools = False # filePerImage=False

I modified gatansocket.py as discribed above but making frames 1 file did not save them when sent to external disk.

I modified the filesave to a local disk and one big file and the frames failed to be written there either.

I changed gatansocket.py back to save individual frames and on a local disk and this time it created a folder and saved the frames. With this configuration I was able to save 20 super res frames. It creates multiple empty folders along with the folder containing the frames.

Bob- This could turn out to be a stupid python problem. In looking at your original post above, I noticed the missing backslash after V: in your frame saving path. This is because the os.path.join python function does not handle Windows drive letters properly (as I just learned today). I just modified gatank2.py to fix this issue. Now you can try that new version of gatank2.py, but you still need to go in and change D: to V:. The difference now is to just create a literal path string instead of using os.path.join. This has not been tested here, so let us know if it works. Still not sure why it worked for local disk but not v:, so maybe this is not the only problem here.

Fix is in r17492.

I tried the changed file and got 21 empty folders on the external disk. they are named 20130402_18125100 to _18231808. No frames in the folders. DM output below

An error occurred acquiring an image for SerialEM:
V:Image\20130402_10475800\frame.mrc

An error occurred acquiring an image for SerialEM:
Failed to open above file: No such file or directory

An error occurred acquiring an image for SerialEM:
Y:Image\20130402_11055800\frame.mrc

An error occurred acquiring an image for SerialEM:
Failed to open above file: No such file or directory
Failed to load acquisition tags from 'X:\2013-04-02_11.31.46 .gtg'.

Bob,

I am working with UC Berkeley people saving frames to their K2, and with the current svn version frame saving works fine saving to their Samba mounted network drive in both counted and super-resolution mode.

On their system, if I use simulate target tool to initiate the Exposure node in MSI-T or equivalent with one preset that save frames, I get one folder created named similar to 20130402_10475800 only. There are n images saved in there name frame_001.mrc to frame_00n.mrc. We used 0.3 s frame_exposure_time and 6.0 second exposure.

Just noticed one thing from your last post. It prints the file name as V:Image\20130402_10475800\frame.mrc
As above mentioned, the file names should carry a number such as frame_001.mrc

Could it be that the path passing screwed up some how after all the changes? Instead of saving to 21 files, it ended up creating 21 directories.

Could you give us your current pyscope gatank2 related files that you have made changes, and the leginon.cfg on the gatan computer. Send them through e-mail if you want them to stay private.

Were you able to glean anything from the files I sent you last month? Is the path wrong? How can I correct it. Thanks

This is embarrassing. You should have reminded me earlier. I got too many gatank2.py files from different people sent to me at that time, I can't tell which one is yours. As far as I can tell none of them were wrong except Jim's discovery. Do you mind send them again just to be sure?

We did find out in the last few weeks since we had our k2 again that the idea we had about trying to saving raw frames as a stack was broken when you tried it. We had it fixed now r17543. We also found out why directories were saved even though the frames weren't saved. We get around that with r17537. However, nothing about your problem. I have some scope time tomorrow. I will see what I can test with the files you send.

OK I just sent them to you. Which one of the dmsem.py files should I use? I assume the r17543 is newer with all the fixes correct?

I looked at the files. Yes, I have seen them before : ). Except of Jim's change in r17492 which you have not applied, the files all looked fine. Jim's change might be all you need.

dmsem.py (Digital Micrograph Serieal EM) extends and replace gatank2.py so that we can support Orius (still need refining). If you start using them, you will need to do a full update because the change need to be synced with leginon if r17492's change does not solve the problem. Give me a few hours to write down what files need changing.

I just sent the file that I lost track of with Jim's modification he made for us. We will try it again in the morning and change the drive letter to V: instead of D:.

Please add a print statement after path is set if it still does not work so that we can check if the output of the code line is right.

I've got the documentation on the update ready in Gatan K2 Support Update of trunk from before 17514 to higher than 17561

After updating svn to r17617, we ran the schema_update.py script and got the following error message:

c:\Program Files\myami\dbschema>c:\Python27\python.exe schema_update.py
←[32mconnected to db 'projectdb' on server 'oahu'←[0m
←[35mGetting release revision from myamiweb/xml/projectDefaultValues.xml←[0m
Traceback (most recent call last):
File "schema_update.py", line 11, in <module>
updatelib_inst = updatelib.UpdateLib(project_dbupgrade)
File "c:\Program Files\myami\dbschema\updatelib.py", line 13, in init
self.checkout_revision = self.getCheckOutRevision()
File "c:\Program Files\myami\dbschema\updatelib.py", line 110, in getCheckOutR
evision
release_revision = self.getReleaseRevisionFromXML(module_path)
File "c:\Program Files\myami\dbschema\updatelib.py", line 229, in getReleaseRe
visionFromXML
raise ValueError('unable to parse XML file "%s"' % xmlfilepath)
ValueError: unable to parse XML file "myamiweb/xml/projectDefaultValues.xml"