Import hooks for PyQt4.QtCore

Question

I'm am attempting to setup some import hooks through sys.meta_path, in a somewhat similar approach to this SO question. For this, I need to define two functions find_module and load_module as explained in the link above. Here is my load_module function,

import imp

def load_module(name, path):
    fp, pathname, description = imp.find_module(name, path)

    try:
        module = imp.load_module(name, fp, pathname, description)
    finally:
        if fp:
             fp.close()
    return module

which works fine for most modules, but fails for PyQt4.QtCore when using Python 2.7:

name = "QtCore"
path = ['/usr/lib64/python2.7/site-packages/PyQt4']

mod = load_module(name, path)

which returns,

Traceback (most recent call last):
   File "test.py", line 19, in <module>
   mod = load_module(name, path)
   File "test.py", line 13, in load_module
   module = imp.load_module(name, fp, pathname, description)
SystemError: dynamic module not initialized properly

The same code works fine with Python 3.4 (although imp is getting deprecated and importlib should ideally be used instead there).

I suppose this has something to do with the SIP dynamic module initialization. Is there anything else I should try with Python 2.7?

Note: this applies both with PyQt4 and PyQt5.

Edit: this may be related to this question as indeed,

cd /usr/lib64/python2.7/site-packages/PyQt4
python2 -c 'import QtCore'

fails with the same error. Still I'm not sure what would be a way around it...

Edit2: following @Nikita's request for a concrete use case example, what I am trying to do is to redirect the import, so when one does import A, what happens is import B. One could indeed think that for this it would be sufficient to do module renaming in find_spec/find_module and then use the default load_module. However, it is unclear where to find a default load_module implementation in Python 2. The closest implementation I have found of something similar is future.standard_library.RenameImport. It does not look like there is a backport of the complete implementation of importlib from Python 3 to 2.

A minimal working example for the import hooks that reproduces this problem can be found in this gist.

Answer 1

UPD: This part in not really relevant after answer updates, so see UPD below.

Why not just use importlib.import_module, which is available in both Python 2.7 and Python 3:

#test.py

import importlib

mod = importlib.import_module('PyQt4.QtCore')
print(mod.__file__)

on Ubuntu 14.04:

$ python2 test.py 
/usr/lib/python2.7/dist-packages/PyQt4/QtCore.so

Since it's a dynamic module, as said in the error (and the actual file is QtCore.so), may be also take a look at imp.load_dynamic.

Another solution might be to force the execution of the module initialization code, but IMO it's too much of a hassle, so why not just use importlib.

UPD: There are things in pkgutil, that might help. What I was talking about in my comment, try to modify your finder like this:

import pkgutil

class RenameImportFinder(object):

    def find_module(self, fullname, path=None):
        """ This is the finder function that renames all imports like
             PyQt4.module or PySide.module into PyQt4.module """
        for backend_name in valid_backends:
            if fullname.startswith(backend_name):
                # just rename the import (That's what i thought about)
                name_new = fullname.replace(backend_name, redirect_to_backend)
                print('Renaming import:', fullname, '->', name_new, )
                print('   Path:', path)


                # (And here, don't create a custom loader, get one from the
                # system, either by using 'pkgutil.get_loader' as suggested
                # in PEP302, or instantiate 'pkgutil.ImpLoader').

                return pkgutil.get_loader(name_new) 

                #(Original return statement, probably 'pkgutil.ImpLoader'
                #instantiation should be inside 'RenameImportLoader' after
                #'find_module()' call.)
                #return RenameImportLoader(name_orig=fullname, path=path,
                #       name_new=name_new)

    return None

Can't test the code above now, so please try it yourself.

P.S. Note that imp.load_module(), which worked for you in Python 3 is deprecated since Python 3.3.

Another solution is not to use hooks at all, but instead wrap the __import__:

print(__import__)

valid_backends = ['shelve']
redirect_to_backend = 'pickle'

# Using closure with parameters 
def import_wrapper(valid_backends, redirect_to_backend):
    def wrapper(import_orig):
        def import_mod(*args, **kwargs):
            fullname = args[0]
            for backend_name in valid_backends:
                if fullname.startswith(backend_name):
                    fullname = fullname.replace(backend_name, redirect_to_backend)
                    args = (fullname,) + args[1:]
            return import_orig(*args, **kwargs)
        return import_mod
    return wrapper

# Here it's important to assign to __import__ in __builtin__ and not
# local __import__, or it won't affect the import statement.
import __builtin__
__builtin__.__import__ = import_wrapper(valid_backends, 
                                        redirect_to_backend)(__builtin__.__import__)

print(__import__)

import shutil
import shelve
import re
import glob

print shutil.__file__
print shelve.__file__
print re.__file__
print glob.__file__

output:

<built-in function __import__>
<function import_mod at 0x02BBCAF0>
C:\Python27\lib\shutil.pyc
C:\Python27\lib\pickle.pyc
C:\Python27\lib\re.pyc
C:\Python27\lib\glob.pyc

shelve renamed to pickle, and pickle is imported by default machinery with the variable name shelve.

Answer 2

When finding a module which is part of package like PyQt4.QtCore, you have to recursively find each part of the name without .. And imp.load_module requires its name parameter be full module name with . separating package and module name.

Because QtCore is part of a package, you shoud do python -c 'import PyQt4.QtCore' instead. Here's the code to load a module.

import imp

def load_module(name):
    def _load_module(name, pkg=None, path=None):
        rest = None
        if '.' in name:
            name, rest = name.split('.', 1)
        find = imp.find_module(name, path)
        if pkg is not None:
            name = '{}.{}'.format(pkg, name)
        try:
            mod = imp.load_module(name, *find)
        finally:
            if find[0]:
                find[0].close()
        if rest is None:
            return mod 
        return _load_module(rest, name, mod.__path__)
    return _load_module(name)

Test;

print(load_module('PyQt4.QtCore').qVersion())  
4.8.6