PDA

View Full Version : Python retrieve a list of available modules



giuspen
October 9th, 2009, 02:16 PM
Hi,
I'm searching for a function that lists all the available modules to import but I found no way, anybody can help me?
Thanks in advance,
Giuseppe.

Pyro.699
October 9th, 2009, 02:59 PM
Does this help? http://www.wellho.net/resources/ex.php4?item=y108/listmethods

giuspen
October 9th, 2009, 03:06 PM
Does this help? http://www.wellho.net/resources/ex.php4?item=y108/listmethods

This explains how to retrieve the attributes of an object, so also the attributes of a module (being the module an object too) but does not explain how to list the modules that I could import.

unutbu
October 9th, 2009, 04:33 PM
Python makes no distinction between python scripts and python modules. They both can be imported using the same syntax. So really any file in any directory in your sys.path that ends in .py is a module as far as import is concerned.

This should list them all: (*)

#!/usr/bin/env python
import sys
import os
files=[]
for directory in sys.path:
files.extend([filename.replace('.py','') for filename in os.listdir(directory)
if filename.endswith('.py')])
print files


(*) Edit: Hm. I forgot about directories containing __init__.py...

giuspen
October 12th, 2009, 10:57 AM
Python makes no distinction between python scripts and python modules. Thy both can be imported using the same syntax. So really any file in any directory in your sys.path that ends in .py is a module as far as import is concerned.

This should list them all: (*)

#!/usr/bin/env python
import sys
import os
files=[]
for directory in sys.path:
files.extend([filename.replace('.py','') for filename in os.listdir(directory)
if filename.endswith('.py')])
print files


(*) Edit: Hm. I forgot about directories containing __init__.py...

Thanks for your help, I created this script according to your lead:
#!/usr/bin/env python
import sys, os

files=[]

for directory in sys.path:
for dirpath, dirnames, filenames in os.walk(directory):
if dirpath == directory and "__init__.py" in filenames:
for filename in filenames:
if filename.endswith('.py'): files.append(filename.replace('.py',''))

files.sort()

print files

but in the resulting list I can't find for example the module "gtk" while I can do "import gtk".

unutbu
October 12th, 2009, 01:43 PM
To understand your code I cut it down to


#!/usr/bin/env python
import sys, os
files=[]
for directory in sys.path:
for dirpath, dirnames, filenames in os.walk(directory):
if dirpath == directory and "__init__.py" in filenames:
print(dirpath)

and found on my installation that it only prints


/usr/lib/python2.5/site-packages/PIL

I think the criterion is only getting satisfied when dirpath is itself in sys.path *and* dirpath contains a file called __init__.py. This criterion is too restrictive.

Here is an script which attempts to cull modules including those buried in directories containing __init__.py.



#!/usr/bin/env python
import sys
import os
import operator

def find_modules(path,prefix=[]):
modules=[]
if not os.path.isdir(path):
return []
listing=os.listdir(path)
for filename in listing:
fullname=os.path.join(path,filename)
# print('Checking %s'%fullname)
if os.path.isdir(fullname):
initfile=os.path.join(fullname,'__init__.py')
# print(' Checking for %s'%initfile)
if os.path.exists(initfile):
# print(' %s is a module'%filename)
modules.append('.'.join(prefix+[filename]))
modules.extend(
find_modules(fullname,prefix+[filename]))
elif (os.path.isfile(fullname) and filename.endswith('.py') and
filename!='__init__.py'
):
modules.append('.'.join(prefix+[filename.replace('.py','')]))
return modules

modules=list(
set(reduce(operator.add,[find_modules(directory) for directory in sys.path])))
modules.sort()
print(modules)


Please tell me if you find a bug.

giuspen
October 13th, 2009, 06:21 PM
Hi and thanks for your help.
Running your script in windows xp I have the following result:

Traceback (most recent call last):
File "D:\BEPPE\cherrytree\modules_list.py", line 27, in <module>
set(reduce(operator.add,[find_modules(directory) for directory in sys.path])))
File "D:\BEPPE\cherrytree\modules_list.py", line 8, in find_modules
listing=os.listdir(path)
WindowsError: [Error 3] The system cannot find the path specified: 'C:\\WINDOWS\\system32\\python25.zip/*.*'
That's why I replaced os.listdir() which seems to cause problems.
I needed this in windows xp as I wanted to package one my application (cherrytree (http://open.vitaminap.it/en/cherrytree.htm)) for windows in addition to ubuntu but have a problem installing the package pygtksourceview in windows (after installation, the "import gtksourceview" causes an error telling module unknown) and wanted to understand better the modules import criterion.

unutbu
October 13th, 2009, 07:22 PM
I edited the script in my previous post with this:


def find_modules(path,prefix=[]):
modules=[]
if not os.path.isdir(path):
return []

Perhaps that will help, though I'm not sure.

It appears that under Windows, sys.path includes
'C:\\WINDOWS\\system32\\python25.zip/*.*'.

I don't understand what Windows is trying to do here.
First, the *.* suggests this path is not a directory. I thought all elements of sys.path were directories.

Second, this path mixes Windows-style backslashes with unix-style forward-slashes. Is that allowed?

Since I don't understand what this path is supposed to do, I added some code to cause the script to ignore such paths. If that is a mistake, then the script won't find all the modules.

giuspen
October 14th, 2009, 05:41 PM
Your script works perfectly now, thanks for the help!

unutbu
October 14th, 2009, 05:43 PM
Great! Glad I could help.

engla
October 14th, 2009, 09:22 PM
Python has lots of awesome tools for modules and whatnot hidden away in the standard library. I discovered pkgutil -- that generalizes package listing and module loading for Python.

I wanted to make it as general as possible so that my application could still load if packages in a .zip package, yes that python2.5.zip in sys.path is no strange thing, python also supports modules in zips, but I can't tell you why. Anyway, pkgutil helped me generalize my problem -- extract and eval a module partially. Basically you can get the source of any module as a compiled code object with pkgutil, then eval that.


loader = pkgutil.get_loader(modulename)
attributes = {}
try:
eval(loader.get_code(), attributes)
except Exception, exc:
pass

The module may raise an exception on import, but all functions, constants etc that were evaluated before the exception are stored in attributes with this code snippet -- pretty cool, that's "rescue mode" module importing.

----

However, what I wanted to say was that I found a function in pkgutil now, which seems to be close to what you were looking for previously.. pkgutil.iter_modules() ; it returns a generator of all toplevel modules in your path (together with an importer object should you wish to import it programmatically).

giuspen
October 15th, 2009, 10:42 AM
It's interesting but I can't make it work.
I tried this script:

import pkgutil

loader = pkgutil.get_loader("pickle")
attributes = {}
try:
eval(loader.get_code(), attributes)
except Exception, exc:
pass
but I always catch the exception

About

pkgutil.iter_modules()
I can't understand how to use it, can you provide a short example?
Thanks.

Can+~
October 15th, 2009, 06:23 PM
pkgutil.iter_modules()
I can't understand how to use it, can you provide a short example?
Thanks.

It returns an iterable, whenever you declare a class, a method, or even a function to be "iterable", then the for loop can iterate over it (and other functions, like reduce, that demand iterables):



import pkgutil

for pkg in pkgutil.iter_modules():
print pkg

giuspen
October 16th, 2009, 07:59 PM
It returns an iterable, whenever you declare a class, a method, or even a function to be "iterable", then the for loop can iterate over it (and other functions, like reduce, that demand iterables):



import pkgutil

for pkg in pkgutil.iter_modules():
print pkg

Really cool, thanks a lot.