Writing and reading namedtuple into a file in python

Question

I need to write a data structure stored as namedtuple to file and read it back as a namedtuple in python.

Solutions here suggest using Json.load/loads or pickle which write the variable as json key-value pair in the form of strings. However, all my field accesses/dereferences are of the form struct.key (rather than struct["key"] which is the way to access json values), and it is unfeasible to correct this in the whole code.

I want to store this to a file because the struct is huge and takes a lot of time to generate.

Answer 1

Since the standard JSON modules in Python generally use dict to work with JSON objects, you need to convert to and from a dict.

For a little setup, let's say I've created this namedtuple:

>>> from collections import namedtuple
>>> import json
>>> X = namedtuple('X', ['x', 'y', 'z'])
>>> x = X(1,2,3)
>>> x
X(x=1, y=2, z=3)

1. Use _asdict() to convert to a dict that you can dump as JSON:

   >>> j = json.dumps(x._asdict())
   >>> j
   '{"x": 1, "y": 2, "z": 3}'
   

   Now you have a JSON representation.

2. To get it back into an object, use ** to convert a dict into keyword arguments:

   >>> x2 = X(**json.loads(j))
   >>> x2
   X(x=1, y=2, z=3)
   

   Done.

You can of course read/write that JSON out to a file any way you wish. (For example, the JSON modules have methods that work with files directly.)

Answer 2

Just addressing your pickling difficulty, it seems that for pickle.dumps() to work it is required that the typename argument to namedtuple match the name to which the returned class is bound.

import pickle
from collections import namedtuple

group_t = namedtuple('group_t', 'field1, field2')
Myobj = group_t(field1=1, field2=2)

>>> pickle.dumps(Myobj)
'ccopy_reg\n_reconstructor\np0\n(c__main__\ngroup_t\np1\nc__builtin__\ntuple\np2\n(I1\nI2\ntp3\ntp4\nRp5\n.'

Compare with this:

mismatched_group_t = namedtuple('group_t', 'field1, field2')
Myobj = mismatched_group_t(field1=1, field2=2)
>>> pickle.dumps(Myobj)
Traceback (most recent call last):
.
.
pickle.PicklingError: Can't pickle <class '__main__.group_t'>: it's not found as __main__.group_t

The difference between the two classes is:

>>> group_t.__name__
'group_t'
>>> mismatched_group_t.__name__
'group_t'

I'd say that that's what is throwing pickle off.

Answer 3

I wrote a library for doing this: https://github.com/ltworf/typedload

It supports rather complicated types, which include enums, unions, tuples, sets, lists.

import typedload
typedload.dump(some_namedtuple)

Your namedtuple could be something more complicated like this

class B(NamedTuple):
    name: str
class A(NamedTuple):
    values: List[Union[str, int]]
    other: Dict[str, str]
    points: Tuple[Tuple[float, float, float], ...]
    more: Optional[B] = None

And it can do dump of objects and then load them back.