Project

General

Profile

new network or other problem

Added by Anonymous almost 20 years ago

I've noticed Leginon appear to hang when I turn automatic checks off and I am running Square Targeting - with the busy wheels spinning as if it is still busy but nothing is actually done. Further investigation has revealed that the Leginon client generally has crashed in these cases. This time I ran the Leginon client in a DOS window; so I could get the output error message, which appears to be network related. Any suggestions what I can do? I've pasted the error message below,

William

C:\Python23\Lib\site-packages\Leginon>launcher.py


  • DataManager size reached, removing data as needed **


Exception happened during processing of request from ('129.11.140.56', 51813)

Traceback (most recent call last):

File "C:\Python23\lib\SocketServer.py", line 463, in process_request_thread

self.finish_request(request, client_address)

File "C:\Python23\lib\SocketServer.py", line 254, in finish_request

self.RequestHandlerClass(request, client_address, self)

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 47

, in init

server_address, server)

File "C:\Python23\lib\SocketServer.py", line 521, in init

self.handle()

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 78

, in handle

p = localHack(o)

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 19

, in localHack

location = obj['manager location']

File "C:\Python23\lib\site-packages\Leginon\data.py", line 563, in getitem

return self.special_getitem(key, dereference=True)

File "C:\Python23\lib\site-packages\Leginon\data.py", line 547, in special_get

item

value = super(Data, self).__getitem__(key)

KeyError: 'manager location'



Exception happened during processing of request from ('129.11.140.56', 51931)

Traceback (most recent call last):

File "C:\Python23\lib\SocketServer.py", line 463, in process_request_thread

self.finish_request(request, client_address)

File "C:\Python23\lib\SocketServer.py", line 254, in finish_request

self.RequestHandlerClass(request, client_address, self)

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 47

, in init

server_address, server)

File "C:\Python23\lib\SocketServer.py", line 521, in init

self.handle()

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 78

, in handle

p = localHack(o)

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 19

, in localHack

location = obj['manager location']

File "C:\Python23\lib\site-packages\Leginon\data.py", line 563, in getitem

return self.special_getitem(key, dereference=True)

File "C:\Python23\lib\site-packages\Leginon\data.py", line 547, in special_get

item

value = super(Data, self).__getitem__(key)

KeyError: 'manager location'



Exception happened during processing of request from ('129.11.140.56', 51986)

Traceback (most recent call last):

File "C:\Python23\lib\SocketServer.py", line 463, in process_request_thread

self.finish_request(request, client_address)

File "C:\Python23\lib\SocketServer.py", line 254, in finish_request

self.RequestHandlerClass(request, client_address, self)

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 47

, in init

server_address, server)

File "C:\Python23\lib\SocketServer.py", line 521, in init

self.handle()

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 78

, in handle

p = localHack(o)

File "C:\Python23\lib\site-packages\Leginon\socketstreamtransport.py", line 19

, in localHack

location = obj['manager location']

File "C:\Python23\lib\site-packages\Leginon\data.py", line 563, in getitem

return self.special_getitem(key, dereference=True)

File "C:\Python23\lib\site-packages\Leginon\data.py", line 547, in special_get

item

value = super(Data, self).__getitem__(key)

KeyError: 'manager location'


C:\Python23\Lib\site-packages\Leginon>


Replies (6)

Square Targeting - Added by Anchi Cheng almost 20 years ago

Could you be specific on what option you turned off in Square Targeting? I am not aware of an "automated check" option.

Re: Square Targeting - Added by Anonymous almost 20 years ago

"anchi" wrote: Could you be specific on what option you turned off in Square Targeting? I am not aware of an "automated check" option.

Sorry, I mean to say that manual checks are turned off (not automated checks). I'm going through the steps in MSI quick start up in the manual and I'm on step 39 which is to turn off manual checks, pick squares and then just let data come in. The manual checks which are turned off are the manual focus check in the Z Focus node (previously turned on in step 28), "allow for user verification of picked holes" in the Hole Targeting and Exposure Targeting nodes (previously turned on in step 29) and the manual focus check in the Focus node (previously turned on in step 35).

I ran Leginon again - up to step 39 of the MSI quick start up - and I got basically the same error message when the Leginon client crashed. (The only differences were in the port numbers in the error message - the IP numbers and everything else were the same as before. I can send you a copy of this error message as well if you like.) When this happened this time, Leginon gave errors in the Exposure node (error message - "6:53:03 Process target failed: (104, 'Connection reset by peer')" ) and the Presets Manager node (error message - "6:53:05 Move failed: unable to set instrument" and information message - "6:53:06 preset request to "hl" failed").

By the way, when I pick squares in step 39 do I need to press the submit targets button again in the Square Targeting node to submit them as acquisition targets in addition to the sacrificial square that was already submitted in step 30?

William

- Added by Jim Pulokas almost 20 years ago

Here are my suggestions for picking square targets:

Each time you are about to select more square targets, first click the refresh button on the square targetting node to make it update the displayed image and targets. Any targets are are done should show up in red. If there is a square target that you expect to be done, but it is not red after doing a refresh, then there has been a problem and that square is actually not considered done. If you click submit at this point, all green targets are submitted (maybe even targets that have already been submitted previously) Clicking the refresh button will totally clear all targets from the displayed image and then reload them from the database (both targets that are done, and targets that have been submitted but not done), so be sure to do this before you select more targets. That targets you click are not stored in the database until you submit them, so they will be deleted when you click on the refresh button.

As for the errors when you switch to full automation, can you give me some more details:

- Which version of Leginon and pyScope are you using?

(my first suggestion is to upgrade to the latest of each if you haven't already: Leginon 1.1.1, pyScope 1.0.5)

- Are you running in our most common setup, with the leginon client running on the Tecnai and the main Leginon program running on a linux machine? Or the whole thing running on the Tecnai? The automated hole finder is known to have problems if it is running on windows.

- Can you determine at what point exactly the errors show up on both the client and the main program? Is it a direct result of you clicking on something, or does it happen a few seconds or minutes later.

- Added by Anonymous almost 20 years ago

"pulokas" wrote: As for the errors when you switch to full automation, can you give me some more details:
- Which version of Leginon and pyScope are you using?
(my first suggestion is to upgrade to the latest of each if you haven't already: Leginon 1.1.1, pyScope 1.0.5)
- Are you running in our most common setup, with the leginon client running on the Tecnai and the main Leginon program running on a linux machine? Or the whole thing running on the Tecnai? The automated hole finder is known to have problems if it is running on windows.
- Can you determine at what point exactly the errors show up on both the client and the main program? Is it a direct result of you clicking on something, or does it happen a few seconds or minutes later.

I'm still using the versions of Leginon and pyScope which I installed at the last workshop (which was Leginon 1.0.1-1 and pyScope 1.0.2-1 I think - at least this is what "rpm --query" says).

Yes, I am running in the most common setup - the Leginon client runs on the Tecnai PC and the main Leginon program on a Linux machine (which is the laptop which Leginon was installed on at the workshop).

It's difficult to determine at what point exactly the errors show up on both the client and the main program. It isn't a direct result of me clicking on something - it happens a number of minutes later (i.e. on the order of half an hour or more).

I'm concerned network speed might be a problem. The notes recommend running on gigabit ethernet with 100 Mbps a minimum. At present the Tecnai PC and its local firewall are on gigabit ethernet. The Leginon laptop is plugged into an ethernet jack in the microscope room that we thought was 100 Mbps (and which goes onto the regular campus/ departmental network); but it may have a lot of contention or even be only 10 Mbps (which we're checking with the network administrator). The laptop only has 100 Mbps at present. The plan is to get all of the relevant systems onto gigabit ethernet.

- Added by Jim Pulokas almost 20 years ago

Hi William,

I am thinking this is a memory leak in pyScope 1.0.2 that was fixed in 1.0.3

Try pyScope 1.0.3 from here:

http://emg.nysbc.org/software/pyscope/index.php

That should be enough to fix the problem, but I still recommend that when you have time, do the full upgrade to Leginon 1.1.1 and pyScope 1.0.5.

Let me know if this helps,

Jim

- Added by Jim Pulokas almost 20 years ago

just another note: you might be able to confirm that this is a memory leak. If you watch the memory usage on the Tecnai, it should increase slowly (after each image acquisition) until the leginon client starts to fail.

    (1-6/6)