Leginon

Rasmus,

You can also start the complete main Leginon program on the Tecnai PC without starting the client. This is a good way to test out leginon without worrying about the network connections. You will still need to have the network connection to the MySQL database, but I think that part is working for you. This should allow you to test if you have the instrument configured properly. I think you need to set the scope type to "Tecnai" with a capital T. But even if the instrument is not configured right, the network connection should still work, so this should be investigated independently.

For the network problems, try the attached scripts. They do a simplified version of the connections between Leginon and the Leginon client. This is how you do the test:

1. Run test1.py on the Tecnai PC (this represents the leginon client)

2. Note the host name and port reported by test1.py, it will say "ACCEPTING CONNECTIONS AT:..."

with the host and port number

3. Run test2.py on your laptop, but give it two arguments, the host and port number reported in step 1, like this:

test2.py <host> <port>

4. This should establish a short dialog between the two scripts, which should conclude when test2.py reports:

"REMOTE CLIENT RESPONDED..."

Copy the output of both scripts to me and you may also be able to follow what is going on yourself. Basically, test1.py waits for a connection. test2.py connects to it, then test1.py send some info to test2.py. Through all of this, they are both reporting info about host names and port numbers. It is possible that the host names are not being resolved properly, or the ports are not open due to a firewall or something.

Let me know how it goes.

Jim

test1.py

#!/usr/bin/env python
import databinder
import time
import data, event
import socket
import datatransport

class Logger(object):

def info(self, stuff):

print 'INFO', stuff

def exception(self, stuff):

print 'EXCEPTION', stuff

def warning(self, stuff):

print 'WARNING', stuff

def printData(d):

manlocation = d['location']

managerhost = manlocation['TCP transport']['hostname']

managerport = manlocation['TCP transport']['port']

print 'MANAGER:  %s:%s' % (managerhost, managerport)

print 'connecting to manager...'

client = datatransport.Client(manlocation, Logger())

myloc = db.location()

print 'MYLOC', myloc

del myloc['local transport']

e = event.NodeAvailableEvent(location=myloc, destination=managerhost)

client.send(e)

myhostname = socket.gethostname().lower()

for myport in range(49152,65536):

try:

db = databinder.DataBinder(myhostname, Logger(), tcpport=myport)

break

except:

continue

db.addBinding(myhostname, event.SetManagerEvent, printData)

print 'ACCEPTING CONNECTIONS AT:  %s:%s' % (myhostname, myport)

time.sleep(1000000)

test2.py

#!/usr/bin/env python
import data
import databinder
import socket
import event
import datatransport
import sys
import time

if len(sys.argv) != 3:

print 'usage:   test2.py <remote_host> <remote_port>'

sys.exit(1)

tecnaihost = sys.argv[1]
tecnaiport = int(sys.argv[2])

def printData(d):

remotehost = d['location']['TCP transport']['hostname']

remoteport = d['location']['TCP transport']['port']

print 'REMOTE CLIENT RESPONDED:  %s:%s' % (remotehost, remoteport)

class Logger(object):

def info(self, stuff):

print 'INFO', stuff

def exception(self, stuff):

print 'EXCEPTION', stuff

def warning(self, stuff):

print 'WARNING', stuff

myhostname = socket.gethostname().lower()

for myport in range(49152,65536):

try:

db = databinder.DataBinder(myhostname, Logger(), tcpport=myport)

break

except:

continue
print 'ACCEPTING CONNECTIONS AT:  %s:%s' % (myhostname, myport)

db.addBinding(myhostname, event.NodeAvailableEvent, printData)

mylocation = &#123;'TCP transport': &#123;'hostname': myhostname, 'port': myport&#125;&#125;
yourlocation = &#123;'TCP transport': &#123;'hostname': tecnaihost, 'port': tecnaiport&#125;&#125;

e = event.SetManagerEvent(destination=tecnaihost, location=mylocation)
print 'CONNECTING TO:  %s:%s' % (tecnaihost, tecnaiport)
client = datatransport.Client(yourlocation, Logger())

## this will connect to the tecnai
client.send(e)

time.sleep(1000000)

Pawels Penczek's group at UT Houston was having similar problems. Some things to note about how it was solved:

- Getting the hosts files configured properly (or nameserver if available) is critical, this is /etc/hosts on linux and c:\windows\system32\ldrivers\etc\hosts on windows

- You should be able to ping tecnai host from remote host and vice versa using host names, not IPs

- You should be able to run the test1.py and test2.py scripts successfully to prove that the two hosts can communicate using host names, not IPs

In the case of Houston, all of that was working, but there was one final discovery (probably a bug in Leginon) that was causing all the trouble:

- The host name of an instrument should be entered into the database in all lowercase letters. Case doesn't seem to matter in the hosts file. You can call it "TECNAI" or "tecnai" and you can still ping it either way. But in the leginon database, if it is not all lower, it will not work.

The group at Leeds is experiencing a problem in which test1.py fails with an IOError "unable to push data"

My suggestion when this happens is:

Leave test2.py running on the remote host, and on the tecnai host where test1.py was running, try this short script to uncover more low level details of the exception:

import socket
s = socket.socket()
s.connect(('test2host', test2port))

where test2host and test2port are reported by test2.py in the "ACCEPTING CONNECTIONS AT" message

Most likely, there will be one of the following two exceptions raised:

a) socket.gaierror: (-2, 'Name or service not known')

which means the tecnai cannot resolve the hostname. Add a hostname<->IP mapping to the hosts file or the nameserver

b) socket.error: (111, 'Connection refused')

which means the port is not open, either due to firewall or server is not running (test2.py in this case)

thanks to this thread our connection problems have been resolved. :)

apparently, we had the same problems as leeds and rasmus, couldn't see the scope in leginon main, couldn't connect to microscope, same error message with the scripts jim provided. seems the only thing missing was the entry for the dedicated leginon machine in the etc/hosts on the windows machine. slaps himself silly

thanks for your help. :)

justus

(edit: solved up to a point, that is. the laptop works fine, leginon machine doesn't. oh well, we'll get it sorted out now, i hope)

leginon is running, finally. looks like the hosts file on the leginon server got overwritten by an update script, no wonder the miscroscope hostname couldn't be resolved. the second problem was the domainname (local subnet 10.0.0.0, connected to leginon server 141.14.xx.xx) of the miscroscope machine. i've changed the domainname (winnt\system32\drivers\etc\hosts) on the microscope machine from localdomain to .molgen.mpg.de (domain of the server, which it technically isn't in, but whatever....) and things are running fine now.

thanks for all the help!

j.

(btw, this is on a very nonstandard home-made distribution written for our institute. guess i'm adventurous out of necessity. but things look quite good, no major problems yet. :)

I just edited test1.py and test2.py to update for the latest Leginon release. The calls to "push" are now "send".

test1.py and test2.py are now included in the leginon package as of the 1.5 beta release.

This has come up in other threads and e-mails, but I want to add it to this thread since this is the largest collection of network related troubleshooting.

The question is: What ports does Leginon use?

This is important because you may have a firewall installed on one or more of the machines that are running Leignon, or you may have a firewall between two machines that are running Leginon. You want to keep everything secure and only open up ports that are necessary.

The typical case is that you are running the main Leginon process on a linux host that is communicating with the TEM host, which is running a Leginon launcher (aka Leginon client). You are also running a MySQL server either on the same linux host or maybe a different one. Optionally, you may be running a NFS file server on another linux host. Those are the only hosts/servers that are involved in running Leginon. I exclude the web server because it is not participating in actually running Leginon. It is only there to view the stored data, and Leginon can run without it.

MySQL normally uses port 3306 (but configurable in /etc/my.cnf)

NFS uses port 2049 (maybe depends on others also?)

Leginon Client running on TEM host: port 55555

Leginon main program on linux host: dynamically assigned ports between 49152 and 65535. There is no strict port assignment since we could potentially have more than one Leginon process running on the same linux host talking to different TEM hosts.

It is probably good enough if you only worry about opening up the first few of those ports in your firewall (maybe 49152 through 49160, or something like that).

Someday I hope to clean up the network issue in Leginon so that it only needs a single socket between the main program and the client instead of multiple sockets using different ports. Then you only need to worry about opening port 55555 on the TEM host and nothing else.

This is a description of the problem sent by Zhongjun through e-mail.

I'v masked out most of the ip address.

The operating system of host computer is centos 5.4. I install all

the softwares following the install manual. Another computer is
windows.
I test the communication between
the server and client computers. I find something wrong in the test.
My client computer's host name is Scanner , Ip is xxx.xxx.x.155
My server computer's host name is xray3 , Ip is xxx.xxx.x.55
I find that in the client computer, when I run the test1.py , the
result is right. But when I run the test2.py in my server computer,
Something is wrong.
Further more, there is also somewrong when I run the test1.py in server
computer. I don't know why. Could you help me to solve these problem ?

D:\Python25\Lib\site-packages\Leginon>python test1.py
INFO localtransport server created at location {'instance': <localtransport.Serv
er object at 0x00C82E30>}
INFO tcptransport server created at location {'hostname': 'scanner', 'port': 491
52}
INFO <class 'event.SetManagerEvent'> binding added for destination scanner, meth
od <function printData at 0x01240870>
ACCEPTING CONNECTIONS AT:  scanner:49152
hit enter to kill
INFO localtransport server exited
INFO tcptransport server exited
INFO Exited
INFO Handler loop exited

[zhongjun@xray3] 206% python test2.py xxx.xxx.x.155 49152
INFO localtransport server created at location {'instance': <localtransport.Server obj
ect at 0xb7d11cec>}
INFO localtransport server created at location {'instance': <localtransport.Server obj
ect at 0xb7d181ac>}
INFO tcptransport server created at location {'hostname': 'xray3.xxxx.ac.cn', 'por
t': 49153}
ACCEPTING CONNECTIONS AT:  xray3.xxxx.ac.cn:49153
INFO <class 'event.NodeAvailableEvent'> binding added for destination xray3.xxxx.a
c.cn, method <function printData at 0xb7f85764>
CONNECTING TO:  xxx.xxx.x.155:49152
WARNING localtransport client add failed
INFO tcptransport client added
INFO server location set to to {'TCP transport': {'hostname': 'xxx.xxx.x.155', 'port':

49152}}
Traceback (most recent call last):

File "test2.py", line 49, in ?

client.send(e)

File "/usr/lib/python2.4/site-packages/Leginon/datatransport.py", line 59, in send

result = self._send(request)

File "/usr/lib/python2.4/site-packages/Leginon/datatransport.py", line 56, in _send

raise TransportError('Error sending request')
datatransport.TransportError: Error sending request

There is also an error if test1.py is run on the linux box xray3:

[zhongjun@xray3] 197% python test1.py
INFO localtransport server created at location {'instance': <localtransport.Server object at 0xb7cd2d0c>}
INFO tcptransport server created at location {'hostname': 'xray3.xxxx.ac.cn', 'port': 49152}
INFO <class 'event.SetManagerEvent'> binding added for destination xray3.xxxx.ac.cn, method <function printData at 0xb7f46764>
ACCEPTING CONNECTIONS AT:  xray3.xxxx.ac.cn:49152
hit enter to kill
INFO localtransport server exited
Traceback (most recent call last):

File "test1.py", line 43, in ?

db.exit()

File "/usr/lib/python2.4/site-packages/Leginon/databinder.py", line 43, in exit

self.server.exit()

File "/usr/lib/python2.4/site-packages/Leginon/datatransport.py", line 88, in exit

self.servers[t].exit()

File "/usr/lib/python2.4/site-packages/Leginon/tcptransport.py", line 60, in exit

socketstreamtransport.Server.exit(self)

File "/usr/lib/python2.4/site-packages/Leginon/socketstreamtransport.py", line 140, in exit

client.send(ExitException())

File "/usr/lib/python2.4/site-packages/Leginon/socketstreamtransport.py", line 151, in send

s = self.connect()

File "/usr/lib/python2.4/site-packages/Leginon/tcptransport.py", line 78, in connect

raise TransportError(string)
tcptransport.TransportError: Name or service not known

This could be a firewall problem. Try starting test1.py on scanner, then run the "nmap" command on one of your linux systems to see which ports are opened on scanner. Here is the nmap command that will search for open ports on scanner:

nmap -p49152- scanner

test1.py
INFO localtransport server created at location {'instance': <localtransport.Server object at 0x2aaaaaabe810>}
INFO localtransport server created at location {'instance': <localtransport.Server object at 0x2aaaaaabefd0>}
INFO tcptransport server created at location {'hostname': 'bnc16', 'port': 49153}
INFO <class 'event.SetManagerEvent'> binding added for destination bnc16, method <function printData at 0x2aaaaaac8578>
ACCEPTING CONNECTIONS AT:  bnc16:49153
hit enter to kill

nmap -p49152- bnc16

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2010-01-12 20:19 PST
Interesting ports on bnc16 (xxx.xxx.xxx.xxx):
Not shown: 16382 closed ports
PORT      STATE SERVICE
49152/tcp open  unknown
49153/tcp open  unknown

Nmap finished: 1 IP address (1 host up) scanned in 0.853 seconds

If nmap does not find the open port, then there is probably a firewall blocking the port. This could be part of the security settings in Windows, or it could be the SELinux system on the CentOS system. Try disabling any of these security features to see if it helps.

Hi all,

I am also experiencing network problems...Leginon works when started locally (Leginon main on microscope PC) and over the network (main/client on distinct workstations with simulation camera) when not firewalled.

However, our actual network setup is:
Microscope PC (Leginon client) <=> gateway (acts as a port forwarder) <=> server A (leginon main and mysql)

There is NO direct routing from microscope PC to institute network and vice-versa...all should work through port-forwarding to the gateway.
All relevant host names are in /etc/hosts (or its Windows equivalent) and resolve appropriately (always to gateway IP).

Port forwarding is setup in the following way and works (checked with telnet):

gateway:3306 --> server A: 3306
Microscope PC : 55555 <-- gateway:55555

gateway:49152--> server A: 49152

gateway:49153 --> server A: 49153

gateway:49154 --> server A: 49154

....

mysql port forwarding works for sure, because leginon main runs successfully on the microscope PC. However, when starting leginon client on the microscope PC and leginon main on server A, the microscope PC does not appear in the dropdown list when assigning the nodes...
When I dump the network traffic with wireshark I see that the connection to port microscope PC:55555 is successfully initiated by leginon main. However, server A never sees any traffic to its open ports (which are certainly one of 49152, 49153, 49154, as checked by netstat).

Has anyone ever got Leginon to work with port forwarding (without direct routing)? I suppose setting the DNS names is critical, to "trick" both sides into believing that the gateway is the relevant communication partner. Or is, at some point, an IP address transferred in the first connection to port 55555 - then, of course, it would not work?

What also puzzles me is that Leginon client also opens a port 49152 (apart from 55555), which is AFAIK not mentioned in the documentation (or in this thread)? When testing with test1.py/test2.py I get the first connection (microscope PC to server A) but the second one back fails.

I would appreciate your help.

Cheers
Matthias Brunner

I am using a leginon SVN checkout from 2009-11-20.

Hello veryone:

I am Zhongjun. I have solved the problem. Thanks Anchi and Pulokas !