Save data to pickle file python

In general, pickling a dict will fail unless you have only simple objects in it, like strings and integers.

Python 2.7.9 (default, Dec 11 2014, 01:21:43) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> type(globals())     

>>> import pickle
>>> pik = pickle.dumps(globals())
Traceback (most recent call last):
  File "", line 1, in 
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle module objects
>>> 

Even a really simple dict will often fail. It just depends on the contents.

>>> d = {'x': lambda x:x}
>>> pik = pickle.dumps(d)
Traceback (most recent call last):
  File "", line 1, in 
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle  at 0x102178668>: it's not found as __main__.

However, if you use a better serializer like dill or cloudpickle, then most dictionaries can be pickled:

>>> import dill
>>> pik = dill.dumps(d)

Or if you want to save your dict to a file...

>>> with open('save.pik', 'w') as f:
...   dill.dump(globals(), f)
... 

The latter example is identical to any of the other good answers posted here (which aside from neglecting the picklability of the contents of the dict are good).

Serialization is a technique used to save the state of an object from any process. We can later use this state by deserialization, to continue the process. Pickle is a python module that makes it easy to serialize or save variables and load them when needed. Unlike JSON serialization, Pickle converts the object into a binary string. JSON is text specific, but Pickle is python specific, and it can serialize the custom classes which JSON fails to serialize. Due to this feature, it is heavily used in training machine learning models. This article discusses how variables can be saved and loaded in python using pickle.

Functions used:

• In python, dumps() method is used to save variables to a pickle file.

Syntax:

pickle.dumps(obj, protocol=None, *, fix_imports=True, buffer_callback=None)

• In python, loads() is used to load saved data from a pickled file

Syntax:

pickle.loads(data, /, *, fix_imports=True, encoding=”ASCII”, errors=”strict”, buffers=None)

Saving a variable:

• Method 1: Passing the variable

In dumps() method, we can pass the variable, and it will return us the binary string for the same. We can then transmit it to other python modules or save in a database.

Example:

Python3

import pickle

myvar = [{'This': 'is', 'Example': 1}, 'of',

         'serialisation', ['using', 'pickle']]

serialized = pickle.dumps(myvar)

print(serialized)

Output:

b’\x80\x04\x95K\x00\x00\x00\x00\x00\x00\x00]\x94(}\x94(\x8c\x04This\x94\x8c\x02is\x94\x8c\x07Example\x94K\x01u\x8c\x02of\x94\x8c\rserialisation\x94]\x94(\x8c\x05using\x94\x8c\x06pickle\x94ee.’

• Method 2:  We can directly save the variable in a file itself.

Example:

Python3

import pickle

myvar = [{'This': 'is', 'Example': 2}, 'of',

         'serialisation', ['using', 'pickle']]

with open('file.pkl', 'wb') as file:

    pickle.dump(myvar, file)

Loading a Variable: 

• Method 1: 

The loads() method takes a binary string and returns the corresponding variable. If the string is invalid, it throws a PickleError.

Example:

Python3

import pickle

binary_string = b'\x80\x04\x95K\x00\x00\x00\x00\x00\x00\x00]\x94(}\x94(\x8c\x04This\x94\x8c\x02is\x94\x8c\x07Example\x94K\x01u\x8c\x02of\x94\x8c\rserialisation\x94]\x94(\x8c\x05using\x94\x8c\x06pickle\x94ee.'

myvar = pickle.loads(binary_string)

print(myvar)

Output:

[{‘This’: ‘is’, ‘Example’: 1}, ‘of’, ‘serialisation’, [‘using’, ‘pickle’]]

• Method 2: 

The load() method loads a pickled file and returns a deserialized variable.

Example:

Python3

import pickle

with open('file.pkl', 'rb') as file:

    myvar = pickle.load(file)

    print(myvar)

Output:

[{‘This’: ‘is’, ‘Example’: 2}, ‘of’, ‘serialisation’, [‘using’, ‘pickle’]]

How do you save things with pickles?

How do you load data in pickles?

How do I save a dict in pickle?

Can you pickle dictionaries?