Python – Why does Paramiko hang if you use it while loading a module

modulemultithreadingparamikopythonssh

Put the following into a file hello.py (and easy_install paramiko if you haven't got it):

hostname,username,password='fill','these','in'
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect(hostname=hostname, username=username, password=password)
i,o,e = c.exec_command('ls /')
print(o.read())
c.close()

Fill in the first line appropriately.

Now type

python hello.py

and you'll see some ls output.

Now instead type

python

and then from within the interpreter type

import hello

and voila! It hangs! It will unhang if you wrap the code in a function foo and do import hello; hello.foo() instead.

Why does Paramiko hang when used within module initialization? How is Paramiko even aware that it's being used during module initialization in the first place?

Best Solution

Paramiko uses separate threads for the underlying transport. You should never have a module that spawns a thread as a side effect of importing. As I understand it, there is a single import lock available, so when a child thread from your module attempts another import, it can block indefinitely, because your main thread still holds the lock. (There are probably other gotchas that I'm not aware of too)

In general, modules shouldn't have side effects of any sort when importing, or you're going to get unpredictable results. Just hold off execution with the __name__ == '__main__' trick, and you'll be fine.

[EDIT] I can't seem to create a simple test case that reproduces this deadlock. I still assume it's a threading issue with import, because the auth code is waiting for an event that never fires. This may be a bug in paramiko, or python, but the good news is that you shouldn't ever see it if you do things correctly ;)

This is a good example why you always want to minimize side effects, and why functional programming techniques are becoming more prevalent.

Related Solutions

Python – How to reload a Django model module using the interactive interpreter via “manage.py shell”

Well, I think I have to answer to this. The problem is that Django caches its models in a singleton (singleton like structure) called AppCache. Basically, to reload Django models you need to first reload and re-import all the model modules stored in the AppCache. Then you need to wipe out the AppCache. Here's the code for it:

import os
from django.db.models.loading import AppCache
cache = AppCache()

curdir = os.getcwd()

for app in cache.get_apps():
    f = app.__file__
    if f.startswith(curdir) and f.endswith('.pyc'):
        os.remove(f)
    __import__(app.__name__)
    reload(app)

from django.utils.datastructures import SortedDict
cache.app_store = SortedDict()
cache.app_models = SortedDict()
cache.app_errors = {}
cache.handled = {}
cache.loaded = False

I've put all of this in a separate file called reloadmodels.py in the root directory of my Django site. Using IPython I can reload everything by running:

%run ~/mysite/reloadmodels.py

Python – Why does this python code hang on import/compile but work in the shell

That's indeed a bad idea to execute this kind of code at import time, although I am not sure why it hangs - it may be that import mechanism does something strange which interacts badly with paramiko (thread related issues maybe ?). Anyway, the usual solution is to implement the functionality into a function:

def my_expensive_function(args):
    pass

if __name__ == '__main__':
    import sys
    my_expensive_functions(sys.args)

This　way, just importing the module will not do anything, but running the script will execute the function with the given arguments at command line.

Best Solution

Related Solutions

Python – How to reload a Django model module using the interactive interpreter via “manage.py shell”

Python – Why does this python code hang on import/compile but work in the shell

Related Question