Serializers#

By default, responses are serialized using pickle, but some alternative serializers are also included. These are mainly intended for use with FileCache, but are compatible with the other backends as well.

Note

Some serializers require additional dependencies

Specifying a Serializer#

Similar to Backends, you can specify which serializer to use with the serializer parameter for either CachedSession or install_cache().

JSON Serializer#

Storing responses as JSON gives you the benefit of making them human-readable and editable, in exchange for a minor reduction in read and write speeds.

Usage:

>>> session = CachedSession('my_cache', serializer='json')

Example JSON-serialized Response

{
  "url": "https://httpbin.org/get",
  "status_code": 200,
  "reason": "OK",
  "_content": "dkP>RB4Ki8b0Rt*dwnb*3LqdNXk}q!WpZ;OIv{%rARr(hB0*zgWpH#NIv^q{FDfD|APOKLARr<^V`F7-bS*`0V{c?>Zf7DoAR=daX>cqcWMyV-VRU68EFcOXARr(jNN;m=B03-<XmoUNVrgzJZ*pfMEFcOXARr(jRdZ!>EkS2xZge6#AR=&ibZBpGEplaXb!BsOb1yP3GBz$SA}k;ZARr(hB3La!ZF+7kRB~ZsWi3f$B03-<Qg3f`JuxjdFlIPmF)(2*GB7hXGcjZ_Gh<>hGc{o{G%#g2VK_22A_^cNeJmgfARr=da%pF2ZX!A$A~82JE-^VSF*GqQGBq+HEFcOXAR={gY$7@!B4~7UaC15@FKBdhaAIk0E^l&YFK1<RA_{#9",
  "cache_key": "4dc151d95200ec91fa77021989f5194e9be47e87f8f228306f3a8d5434b9e547",
  "created_at": "2021-07-21T22:34:50.343095",
  "elapsed": 0.242198,
  "encoding": "utf-8",
  "headers": {
    "Date": "Wed, 21 Jul 2021 22:34:50 GMT",
    "Content-Type": "application/json",
    "Content-Length": "308",
    "Connection": "keep-alive",
    "Server": "gunicorn/19.9.0",
    "Access-Control-Allow-Origin": "*",
    "Access-Control-Allow-Credentials": "true"
  },
  "request": {
    "body": "PH%2y",
    "headers": {
      "User-Agent": "python-requests/2.26.0",
      "Accept-Encoding": "gzip, deflate",
      "Accept": "*/*",
      "Connection": "keep-alive"
    },
    "method": "GET",
    "url": "https://httpbin.org/get"
  },
  "raw": {
    "decode_content": false,
    "reason": "OK",
    "status": 200,
    "version": 11
  }
}

This will use ultrajson if installed, otherwise the stdlib json module will be used. You can install the optional dependencies for this serializer with:

pip install requests-cache[json]

YAML Serializer#

YAML is another option if you need a human-readable/editable format, with the same tradeoffs as JSON.

Usage:

>>> session = CachedSession('my_cache', serializer='yaml')

Example YAML-serialized Response

url: https://httpbin.org/get
status_code: 200
reason: OK
_content: !!binary |
  ewogICJhcmdzIjoge30sIAogICJoZWFkZXJzIjogewogICAgIkFjY2VwdCI6ICIqLyoiLCAKICAg
  ICJBY2NlcHQtRW5jb2RpbmciOiAiZ3ppcCwgZGVmbGF0ZSIsIAogICAgIkhvc3QiOiAiaHR0cGJp
  bi5vcmciLCAKICAgICJVc2VyLUFnZW50IjogInB5dGhvbi1yZXF1ZXN0cy8yLjI2LjAiLCAKICAg
  ICJYLUFtem4tVHJhY2UtSWQiOiAiUm9vdD0xLTYwZjhhMDcxLTBiN2JmN2VjNGMyZTdmNjA2YWI4
  ZDYyNCIKICB9LCAKICAib3JpZ2luIjogIjE3My4xOS4xNDEuMjUyIiwgCiAgInVybCI6ICJodHRw
  czovL2h0dHBiaW4ub3JnL2dldCIKfQo=
cache_key: 4dc151d95200ec91fa77021989f5194e9be47e87f8f228306f3a8d5434b9e547
created_at: '2021-07-21T22:32:17.592974'
elapsed: 0.187586
encoding: utf-8
headers:
  Access-Control-Allow-Credentials: 'true'
  Access-Control-Allow-Origin: '*'
  Connection: keep-alive
  Content-Length: '308'
  Content-Type: application/json
  Date: Wed, 21 Jul 2021 22:32:17 GMT
  Server: gunicorn/19.9.0
request:
  method: GET
  url: https://httpbin.org/get
  body: !!binary |
    Tm9uZQ==
  headers:
    Accept: '*/*'
    Accept-Encoding: gzip, deflate
    Connection: keep-alive
    User-Agent: python-requests/2.26.0
raw:
  decode_content: false
  reason: OK
  status: 200
  version: 11

You can install the extra dependencies for this serializer with:

pip install requests-cache[yaml]

BSON Serializer#

BSON is a serialization format originally created for MongoDB, but it can also be used independently. Compared to JSON, it has better performance (although still not as fast as pickle), and adds support for additional data types. It is not human-readable, but some tools support reading and editing it directly (for example, bson-converter for Atom).

Usage:

>>> session = CachedSession('my_cache', serializer='bson')

You can install the extra dependencies for this serializer with:

pip install requests-cache[mongo]

Or if you would like to use the standalone BSON codec for a different backend, without installing MongoDB dependencies:

pip install requests-cache[bson]

Serializer Security#

See Security for recommended setup steps for more secure cache serialization, particularly when using pickle.

Custom Serializers#

If the built-in serializers don’t suit your needs, you can create your own. For example, if you had a imaginary custom_pickle module that provides dumps and loads functions:

>>> import custom_pickle
>>> from requests_cache import CachedSession
>>> session = CachedSession(serializer=custom_pickle)

Serializer Pipelines#

More complex serialization can be done with SerializerPipeline. Use cases include text-based serialization, compression, encryption, and any other intermediate steps you might want to add.

Any combination of these can be composed with a SerializerPipeline, which starts with a CachedResponse and ends with a str or bytes object. Each stage of the pipeline can be any object or module with dumps and loads functions. If the object has similar methods with different names (e.g. compress / decompress), those can be aliased using Stage.

For example, a compressed pickle serializer can be built as:

Example code

>>> import gzip
>>> from requests_cache import CachedSession, SerializerPipeline, Stage, pickle_serializer
>>> compressed_serializer = SerializerPipeline(
...     [
...         pickle_serializer,
...         Stage(dumps=gzip.compress, loads=gzip.decompress),
...     ],
...     is_binary=True,
... )
>>> session = CachedSession(serializer=compressed_serializer)

Text-based Serializers#

If you’re using a text-based serialization format like JSON or YAML, some extra steps are needed to encode binary data and non-builtin types. The cattrs library can do the majority of the work here, and some pre-configured converters are included for several common formats in the preconf module.

For example, a compressed JSON pipeline could be built as follows:

Example code

>>> import json, gzip
>>> from requests_cache import CachedSession, SerializerPipeline, Stage, json_serializer, utf8_encoder
>>> comp_json_serializer = SerializerPipeline([
...     json_serializer, # Serialize to a JSON string
...     utf8_encoder, # Encode to bytes
...     Stage(dumps=gzip.compress, loads=gzip.decompress), # Compress
... ])

Note

If you want to use a different format that isn’t included in preconf, you can use CattrStage as a starting point.

Additional Serialization Steps#

Some other tools that could be used as a stage in a SerializerPipeline include:

Class	loads	dumps
`codecs.*`	encode	decode
`bz2`	compress	decompress
`gzip`	compress	decompress
`lzma`	compress	decompress
`zlib`	compress	decompress
`pickle`	dumps	loads
`itsdangerous.signer.Signer`	sign	unsign
`itsdangerous.serializer.Serializer`	loads	dumps
`cryptography.fernet.Fernet`	encrypt	decrypt