Python3 C-libs: How do I serialize a class that depends on C .dlls for use in a bare instance?

Question

For example:

// file foo
def my_func():
   print("foo!")

// file bar
class Bar:
   def __init__(self):
      pass 
   def hello():
      import foo.my_func
      my_func()

import dill
dill.dump(Bar(), open("bar.dpkl",mode='wb+'))

$: python3 -m venv foobar
$: source activate foobar/bin/activate
(foobar) $: python3
>>> import dill
>>> obj = dill.load(open("bar.dpkl",mode='rb')) # just fine
>>> obj.hello()
foo!

But, suppose I throw in a twist:

class Bar:
  def __init__(self):
     import numpy

Now,

>>> obj = dll.load(open("bar.dpkl",mode='rb'))
ModuleNotFoundError: No module named 'numpy'

It appears as if the linked C-binaries are not bundled up in the pickle file. However, if the code is genuine, native python code, the pickle file output by dill is essentially a dynamic archive.

Is there any way to extend this dynamic archive behavior to include the libraries like numpy et al.? Or must these sort of C + python-shell libraries always be installed in the executing python environment?

Basically, I think the thing I am asking for is a pythonic version of a jar that can be generated straight from python code: is this possible? Or is a container with requisite dependencies the jar I am looking for?

I'm the `dill` author. `dill` is the most aggressive serializer in that it can serialize entire modules, if warranted, as long as they are pure-python. For anything with C, you are correct that the assumption is that the C-dependency is installed. — Mike McKerns, Feb 12 '21 at 12:44

Chris · Accepted Answer · 2022-03-03T18:08:48.353

A possible solution:

import dill

class Bar:
  def __init__(self, foo):
    self.basename = foo.name
    self.deps = deepcopy(foo.deps)
    self.foo = '-'.join([self.basename,'foo'])
    self.bar = '-'.join([self.basename,'bar'])

  def install(self):
    import pip
    pip.main(["install"] + self.deps) # this method will be deprecated soon

  def load(self):
    self.install()
    foo = dill.load(open(self.foo,'rb'))
    return foo
 
  def store_self():
    with open(self.bar + '.dpkl','wb+') as out:
      dill.dump(out,self)

  @classmethod
  def store(cls, foo):
    bar = cls(foo)
    bar.store_self()
    bar.install()
    with open(self.foo + '.dpkl','wb+') as out:
      dill.dump(out,foo)

class Foo:
  def __init__(self,name,*arg,**kwargs):
    self.name = name
    self.deps = [ "numpy" ]

  def hello():
    import numpy
    print("foo")
    ...


foo = Foo("Hello! I'm a foo!")
Bar.store(foo)

$: python3 -m venv venv-foo
$: source venv-foo/bin/activate
$: python3 -m pip install dill
>>> import dill
>>> bar = dill.load(open('bar.dpkl','rb'))
>>> foo = bar.load()
>>> foo.hello()
Hello! I am a foo!

No warranty of copy-pasteability, but the gist of it is there. Might have to tinker with it a bit to get the Bar.dpkl working. It is also sensitive to python3's version.

With some more work, native pickle can probably be used to store Bar, which can then be used to install dill.

Python3 C-libs: How do I serialize a class that depends on C .dlls for use in a bare instance?

1 Answers1