Python - Object Serialization

Object serialization is the process of converting state of an object into byte stream. This byte stream can further be stored in any file-like object such as a disk file or memory stream. It can also be transmitted via sockets etc. Deserialization is the process of reconstructing the object from the byte stream.

Python refers to serialization and deserialization by terms pickling and unpickling respectively. The 'pickle' module bundled with Python’s standard library defines functions for serialization (dump() and dumps()) and deserialization (load() and loads()).

The data format of pickle module is very Python specific. Hence, programs not written in Python may not be able to deserialize the encoded (pickled) data properly. In fact it is not considered to be secure to unpickle data from unauthenticated source.

pickle protocols

Protocols are the conventions used in constructing and deconstructing Python objects to/from binary data. Currently pickle module defines 5 different protocols as listed below:

To know highest and default protocol version of your Python installation use following constants defined in pickle module

>>> import pickle
>>> pickle.HIGHEST_PROTOCOL
4
>>> pickle.DEFAULT_PROTOCOL
3

The dump() and load() functions of pickle module respectively perform pickling and unpickling of Python data. The dump() function writes pickled object to a file (or file like object) and load() function unpickles data from file back to Python object.

Following program pickle a dictionary object into a binary file.

import pickle
f=open("pickled.txt","wb")
dct={"name":"Rajeev", "age":23, "Gender":"Male","marks":75}
pickle.dump(dct,f)
f.close()

When above code is executed, the dictionary object’s byte representation will be stored in 'pickled.txt' file. The file must have 'write and binary' mode enabled.

On the other hand load() function unpickles or deserializes data from binary file back to Python dictionary. Run following script

import pickle

f=open("pickled.txt","rb")
d=pickle.load(f)
print (d)
f.close()

Python console shows the dictionary object read from file

{'age': 23, 'Gender': 'Male', 'name': 'Rajeev', 'marks': 75}

Note that the dictionary object doesn't retain order of insertion of keys. Hence the k-v pairs in retrieved dictionary may not be in original order.

The pickle module also consists of dumps() function that pickles Python data to a string representation.

>>> from pickle import dump
>>> dct={"name":"Rajneesh", "age":23, "Gender":"Male","marks":75}
>>> dctstring=dumps(dct)
>>> dctstring
b'\x80\x03}q\x00(X\x04\x00\x00\x00nameq\x01X\x04\x00\x00\x00Raviq\x02X\x03\x00\x00\x00ageq\x03K\x17X\x06\x00\x00\x00Genderq\x04X\x01\x00\x00\x00Mq\x05X\x05\x00\x00\x00marksq\x06KKu.'

Use loads() function to unpickle the string and obtain original dictionary object.

>>> from pickle import load
>>> dct=loads(dctstring)
>>> dct
{'name': 'Rajneesh', 'age': 23, 'Gender': 'Male', 'marks': 75}

In addition to above convenience functions, the pickle module also defines Pickler and Unpickler classes. Pickler class has dump() method that writes pickle data to a binary file. Unpickler class reads binary data from file and constructs Python object.

To write Python object’s pickled data

from pickle import pickler

f=open("pickled.txt","wb")
dct={'name': 'Rajneesh', 'age': 23, 'Gender': 'Male', 'marks': 75}
Pickler(f).dump(dct)
f.close()

To read back data by unpickling binary file from pickle import Unpickler

f=open("pickled.txt","rb")
dct=Unpickler(f).load()
print (dct)
f.close()

Just as objects of Python standard data types are picklable, similarly, objects of custom class can also be pickled and unpickled.

from pickle import *

class student:

       def __init__(self):
               self.name="XYZ"
               self.age=22
       def show(self):
               print ("name:", self.name, "age:", self.age)
p1=person()
f=open("pickled.txt","wb")
dump(p1,f)
f.close()
print ("unpickled data")
f=open("pickled.txt","rb")
p1=load(f)
p1.show()

Python library also contains 'marshal' module that is used b Python interpreter itself for internal serialization of Python objects.

Internal Python object serialization (marshal)

The marshal module in Python’s standard library provides object serialization features (similar to pickle module). However it is not really useful for general purpose data persistence in files or transmission of Python objects through sockets etc. This module is internally used by Python interpreter itself to perform read/write operations on compiled versions of Python modules (.pyc files). The data format used by marshal module keeps on changing and is not compatible across Python versions (not even sub versions). That’s the reason a compiled Python script (.pyc file) of one version most probably won’t execute on another. The marshal module is thus used for Python’s internal object serialization.

Similar to pickle module, marshal module also defines load() and dump() functions for reading and writing marshalled objects from / to file. Also loads() and dumps() function deal with string representation of marshalled object.

• dumps() : returns a byte like object my marshalling a Python object. Only objects of standard data types are supported for marshalling. Unsupported types raise ValueError exception.
• loads() : This function converts the byte like object to corresponding Python object. If the conversion doesn’t result in valid Python object, ValueError or TypeError may be raised.

Following code shows a Python dictionary object marshalled using dumps(). The byte representation is converted back to dictionary by loads() function.

import marshal
person={"name":"Krishna", "age":22, "marks":[45,56,78]}
data=marshal.dumps(person)
obj= marshal.loads(data)
print (obj)

• dump() : This function writes byte representation of supported Python object to a file. The file itself should be a binary file with write permission
• load(): This function reads the byte data from a binary file and converts it to Python object.

As mentioned above marshal module is used by interpreter to process .pyc files. Following example demonstrates use of dump() and load() functions to handle Python code objects, which are used to store precompiled Python modules.

The code uses built-in compile() function to build a code object out of a source string which embeds Python instructions.

compile(source, file, mode)

The file parameter should be the file from which the code was read. If it wasn’t read from a file pass any arbitrary string.

The mode parameter is set to ‘exec’ if the source contains sequence of statements or ‘eval’ if there is a single expression or ‘single’ if it contains a single interactive statement.

The compile code object is then stored in a .pyc file using dump() function

import marshal

string = """
a=10
b=20
print ('multiplication=',a*b)
"""
code = compile(string, "script", "exec")
f=open("marshal.pyc","wb")
marshal.dump(code, f)
f.close()

To deserialize the object from .pyc file use load() function. Since it returns a code object, it can be run using exec(), another built-in function.

import marshal
f=open("marshal.pyc","rb")
data=marshal.load(f)
exec (data)

The output will be result of code block embedded in source string

multiplication= 200

Python object persistence (shelve)

Another module called 'shelve' in Python’s standard library is a simple yet effective tool for persistent data storage when you don't really need a relational database solution. The 'Shelf' object defined in this module is dictionary-like object which is persistently stored in a disk file. This creates a file similar to dbm database on UNIX like systems. However, object of only string type is allowed as key in this special dictionary object, value however can be any picklable object.

The shelve module defines three classes as follows:

Easiest way to form a Shelf object is to use open() function defined in shelve module which return a DbfilenameShelf object.

shelve.open(filename, flag='c', protocol=None, writeback=False)

The filename parameter is used as name of the database created.

Default value for flag parameter is ‘c’ for read/write access. Other flags are ‘w’ (write only) ‘r’ (read only) and ‘n’ (new with read/write)

Protocol parameter denotes pickle protocol enlisted earlier.

writeback parameter by default is false. If set to true, the accessed entries are cached. Every access calls sync() and close() operations hence process may be slow.

Following code creates a database and stores dictionary entries in it.

import shelve

s=shelve.open("test")
s['name']="Ajay"
s['age']=23
s['marks']=75
s.close()

This will create test.dir file in current directory and store key-value data in hashed form. The Shelf object has following methods available:

To access value of a particular key in shelf:

>>> s=shelve.open('test')
>>> s['age']
23
>>> s['age']=25
>>> s.get('age')
25

The items(), keys() and values() methods return view objects.

>>> list(s.items())
[('name', 'Ajay'), ('age', 25), ('marks', 75)]
>>> list(s.keys())
['name', 'age', 'marks']
>>> list(s.values())
['Ajay', 25, 75]

To remove a key-value pair from shelf

>>> s.pop('marks')
75
>>> list(s.items())
[('name', 'Ajay'), ('age', 25)]

Notice that key-value pair of marks-75 has been removed.

To merge items of another dictionary with shelf use update() method

>>> d={'salary':10000, 'designation':'manager'}
>>> s.update(d)
>>> list(s.items())
[('name', 'Ajay'), ('age', 25), ('salary', 10000), ('designation', 'manager')]

There is another popular serialization/deserialization format called JSON which stands for JavaScript Object Notation. It is a lightweight data interchange format. It is similar to pickle. However, pickle serialization is Python specific whereas JSON format is implemented by many languages. The 'json' module in Python’s standard library defines functions for object serialization.

Just as in pickle module, the json module also provides dumps() and loads() function for serialization of Python object into JSON encoded string. The dump() and load() functions write and read serialized Python objects to/from file.

dumps():

This function converts the object into JSON format.

loads():

This function converts a JSON string back to Python object.

Following example demonstrates basic usage of these functions:

>>> import json
>>> data=['Rakhee',{'marks':(50,60,70)}]
>>> s=json.dumps(data)
>>> s
'["Rakhee", {"marks": [50, 60, 70]}]'
>>> json.loads(s)
['Rakhee', {'marks': [50, 60, 70]}]

Th dumps() function can take optional sort_keys argument. By default it is False. If set to True, the dictionary keys appear in sorted order in the JSON string.

>>> data={"marks":50, "age":20, "rank":5}
>>> s=json.dumps(data, sort_keys=True)
>>> s
'{"age": 20, "marks": 50, "rank": 5}'
>>> json.loads(s)
{'age': 20, 'marks': 50, 'rank': 5}

The dumps() function has another optional parameter called indent which takes a number as value. It decides length of each segment of formatted representation of json string, similar to pprint output.

>>> data=['Rakesh',{'marks':(50,60,70)}]
>>> s=json.dumps(data, indent=2)
>>> print (s)
[
 "Rakesh",
 {
   "marks": [
     50,
     60,
     70
   ]
 }
]

The json module also has object oriented interface corresponding to above convenience functions. There are two classes defined in the module – JSONEncoder and JSONDecoder.

JSONEncoder class:

Object of this class is encoder for Python data structures. Each Python data type is converted in corresponding JSON type as shown in following table:

The JSONEncoder class is instantiated by JSONEncoder() constructor. Following important methods are defined in encoder class:

encode(): serializes Python object into JSON format

iterencode(): Encodes the object and returns an iterator yielding encoded form of each item in the object.

indent: Determines indent level of encoded string

sort_keys: is either true or false to make keys appear in sorted order or not.

Check_circular: if True, check for circular reference in container type object

Following example encodes Python list object.

>>> e=json.JSONEncoder()
>>> e.encode(data)
'["Rakhee", {"marks": [50, 60, 70]}]'

Using iterencode() method, each part of the encoded string is displayed as below:

>>> for obj in e.iterencode(data):
print (obj)
["Rakhee"
,
{
"marks"
:
[50
, 60
, 70
]
}
]

JSONDEcoder class:

Object of this class helps in decoded in json string back to Python data structure. Main method in this class is decode(). Following example code retrieves Python list object from encoded string in earlier step.

>>> d=json.JSONDecoder()
>>> d.decode(s)
['Rakhee', {'marks': [50, 60, 70]}]

JSON with files

The json module defines load() and dump() functions to write JSON data to a file like object – which may be a disk file or a byte stream and read data back from them.

dump(): 

This function encodes Python object data in JSON format and writes it to a file. The file must be having write permission.

>>> data=['Rakhee', {'marks': (50, 60, 70)}]
>>> fp=open('json.txt','w')
>>> json.dump(data,fp)
>>> fp.close()

This code will create ‘json.txt’ in current directory. It shows the contents as follows:

["Rakhee", {"marks": [50, 60, 70]}]

load() : 

This function loads JSON data from the file and constructs Python object from it. The file must be opened with read permission.

>>> fp=open('json.txt','r')
>>> ret=json.load(fp)
>>> ret
['Rakhee', {'marks': [50, 60, 70]}]
>>> fp.close()