In general, pickling a dict
will fail unless you have only simple objects in it, like strings and integers.
Python 2.7.9 [default, Dec 11 2014, 01:21:43]
[GCC 4.2.1 Compatible Apple Clang 4.1 [[tags/Apple/clang-421.11.66]]] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> type[globals[]]
>>> import pickle
>>> pik = pickle.dumps[globals[]]
Traceback [most recent call last]:
File "", line 1, in
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
Pickler[file, protocol].dump[obj]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save[obj]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f[self, obj] # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems[obj.iteritems[]]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
save[v]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
rv = reduce[self.proto]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle module objects
>>>
Even a really simple dict
will often fail. It just depends on the contents.
>>> d = {'x': lambda x:x}
>>> pik = pickle.dumps[d]
Traceback [most recent call last]:
File "", line 1, in
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
Pickler[file, protocol].dump[obj]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
self.save[obj]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f[self, obj] # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems[obj.iteritems[]]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
save[v]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
f[self, obj] # Call unbound method with explicit self
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
[obj, module, name]]
pickle.PicklingError: Can't pickle : it's not found as __main__.
However, if you use a better serializer like dill
or cloudpickle
, then most dictionaries can be pickled:
>>> import dill
>>> pik = dill.dumps[d]
Or if you want to save your dict
to a file...
>>> with open['save.pik', 'w'] as f:
... dill.dump[globals[], f]
...
The latter example is identical to any of the other good
answers posted here [which aside from neglecting the picklability of the contents of the dict
are good].
Serialization is a technique used to save the state of an object from any process. We can later use this state by deserialization, to continue the process. Pickle is a python module that makes it easy to serialize or save variables and load them when needed. Unlike JSON serialization, Pickle converts the object into a binary string. JSON is text specific, but Pickle is python specific, and it can serialize the custom classes which JSON fails to serialize. Due to this feature, it is heavily used in training machine learning models. This article discusses how variables can be saved and loaded in python using pickle.
Functions used:
- In python, dumps[] method is used to save variables to a pickle file.
Syntax:
pickle.dumps[obj, protocol=None, *, fix_imports=True, buffer_callback=None]
- In python, loads[] is used to load saved data from a pickled file
Syntax:
pickle.loads[data, /, *, fix_imports=True, encoding=”ASCII”, errors=”strict”, buffers=None]
Saving a variable:
- Method 1: Passing the variable
In dumps[] method, we can pass the variable, and it will return us the binary string for the same. We can then transmit it to other python modules or save in a database.
Example:
Python3
import
pickle
myvar
=
[{
'This'
:
'is'
,
'Example'
:
1
},
'of'
,
'serialisation'
, [
'using'
,
'pickle'
]]
serialized
=
pickle.dumps[myvar]
print
[serialized]
Output:
b’\x80\x04\x95K\x00\x00\x00\x00\x00\x00\x00]\x94[}\x94[\x8c\x04This\x94\x8c\x02is\x94\x8c\x07Example\x94K\x01u\x8c\x02of\x94\x8c\rserialisation\x94]\x94[\x8c\x05using\x94\x8c\x06pickle\x94ee.’
- Method 2: We can directly save the variable in a file itself.
Example:
Python3
import
pickle
myvar
=
[{
'This'
:
'is'
,
'Example'
:
2
},
'of'
,
'serialisation'
, [
'using'
,
'pickle'
]]
with
open
[
'file.pkl'
,
'wb'
] as
file
:
pickle.dump[myvar,
file
]
Loading a Variable:
- Method 1:
The loads[] method takes a binary string and returns the corresponding variable. If the string is invalid, it throws a PickleError.
Example:
Python3
import
pickle
binary_string
=
b
'\x80\x04\x95K\x00\x00\x00\x00\x00\x00\x00]\x94[}\x94[\x8c\x04This\x94\x8c\x02is\x94\x8c\x07Example\x94K\x01u\x8c\x02of\x94\x8c\rserialisation\x94]\x94[\x8c\x05using\x94\x8c\x06pickle\x94ee.'
myvar
=
pickle.loads[binary_string]
print
[myvar]
Output:
[{‘This’: ‘is’, ‘Example’: 1}, ‘of’, ‘serialisation’, [‘using’, ‘pickle’]]
- Method 2:
The load[] method loads a pickled file and returns a deserialized variable.
Example:
Python3
import
pickle
with
open
[
'file.pkl'
,
'rb'
] as
file
:
myvar
=
pickle.load[
file
]
print
[myvar]
Output:
[{‘This’: ‘is’, ‘Example’: 2}, ‘of’, ‘serialisation’, [‘using’, ‘pickle’]]