Leginon

Hello,

A couple notes about using redux with the current version of fs (https://pypi.python.org/pypi/fs/2.0.10a1).

I'm currently using myami-3.2 on RHEL 7.

While testing redux according to the instructions I received the following error.

*** Using custom copy of fftw3 wrapper
calc_fftw3: 12 CPUs found, setting threads=12
**power thread lock in effectTraceback (most recent call last):
  File "/usr/bin/redux", line 3, in <module>
    import redux.client
  File "/usr/lib/python2.7/site-packages/redux/client.py", line 8, in <module>
    import redux.pipeline
  File "/usr/lib/python2.7/site-packages/redux/pipeline.py", line 20, in <module>
    results = redux.cache.Cache(disk_cache_path, disk_cache_size, size_max=mem_cache_size)
  File "/usr/lib/python2.7/site-packages/redux/cache.py", line 19, in __init__
    self.diskcache = cachefs.CacheFS(disk_cache_path, disk_cache_size)
  File "/usr/lib/python2.7/site-packages/redux/cachefs.py", line 108, in __init__
    files = list(self.walkfiles())
AttributeError: 'CacheFS' object has no attribute 'walkfiles'

After looking at the trunk version of myami it looks like there is a fix for this error which I then incorporated into the local version of the code.

I then received the following error:

Traceback (most recent call last):
  File "/usr/bin/redux", line 5, in <module>
    redux.client.run()
  File "/usr/lib/python2.7/site-packages/redux/client.py", line 141, in run
    result = client.process_kwargs(**kwargs)
  File "/usr/lib/python2.7/site-packages/redux/client.py", line 57, in process_kwargs
    return pl.process(**kwargs)
  File "/usr/lib/python2.7/site-packages/redux/pipeline.py", line 147, in process
    results.put(done, result)
  File "/usr/lib/python2.7/site-packages/redux/cache.py", line 39, in put
    return self._put(pipeline, result)
  File "/usr/lib/python2.7/site-packages/redux/cache.py", line 34, in _put
    self.file_put(pipeline, result)
  File "/usr/lib/python2.7/site-packages/redux/cache.py", line 79, in file_put
    self.diskcache.makedir(path, recursive=True, allow_recreate=True)
TypeError: makedir() got an unexpected keyword argument 'recursive'

I did not see a fix for this in the trunk version so I thought I'd bring it to your attention.
I appears they do not accept the recursive or allow_recreate parameter in the latest fs package. Here is a link to the makedir method documentation: https://pyfilesystem2.readthedocs.io/en/latest/reference/base.html#fs.base.FS.makedir

Here is the version information for the fs python module on my machine.

# pip show fs
---
Metadata-Version: 2.0
Name: fs
Version: 2.0.9
Summary: Python's filesystem abstraction layer
Home-page: https://github.com/PyFilesystem/pyfilesystem2
Author: Will McGugan
Author-email: will@willmcgugan.com
Installer: pip
License: MIT
Location: /usr/lib/python2.7/site-packages
Requires: enum34, appdirs, setuptools, pytz, six
Classifiers:
  Development Status :: 5 - Production/Stable
  Intended Audience :: Developers
  License :: OSI Approved :: MIT License
  Operating System :: OS Independent
  Programming Language :: Python
  Programming Language :: Python :: 2.7
  Programming Language :: Python :: 3.3
  Programming Language :: Python :: 3.4
  Programming Language :: Python :: 3.5
  Programming Language :: Python :: 3.6
  Topic :: System :: Filesystems
Entry-points:
  [fs.opener]
  file = fs.opener.osfs:OSFSOpener
  ftp = fs.opener.ftpfs:FTPOpener
  mem = fs.opener.memoryfs:MemOpener
  osfs = fs.opener.osfs:OSFSOpener
  siteconf = fs.opener.appfs:AppFSOpener
  sitedata = fs.opener.appfs:AppFSOpener
  tar = fs.opener.tarfs:TarOpener
  temp = fs.opener.tempfs:TempOpener
  usercache = fs.opener.appfs:AppFSOpener
  userconf = fs.opener.appfs:AppFSOpener
  userdata = fs.opener.appfs:AppFSOpener
  userlog = fs.opener.appfs:AppFSOpener
  zip = fs.opener.zipfs:ZipOpener

Hopefully this information helps someone.