Request Matching#

Requests are matched according to the request method, URL, parameters and body. All of these values are normalized to account for any variations that do not modify response content.

There are some additional options to configure how you want requests to be matched.

Selective Parameter Matching#

By default, all normalized request parameters are matched. In some cases, there may be request parameters that you don’t want to match. For example, an authentication token will change frequently but not change response content.

Use the ignored_parameters option if you want to ignore specific parameters.

Note

Many common authentication parameters are already ignored by default. See Removing Sensitive Info for details.

Request Parameters:

In this example, only the first request will be sent, and the second request will be a cache hit due to the ignored parameters:

>>> session = CachedSession(ignored_parameters=['auth-token'])
>>> session.get('https://httpbin.org/get', params={'auth-token': '2F63E5DF4F44'})
>>> r = session.get('https://httpbin.org/get', params={'auth-token': 'D9FAEB3449D3'})
>>> assert r.from_cache is True

Request Body Parameters:

This also applies to parameters in a JSON-formatted request body:

>>> session = CachedSession(allowable_methods=('GET', 'POST'), ignored_parameters=['auth-token'])
>>> session.post('https://httpbin.org/post', json={'auth-token': '2F63E5DF4F44'})
>>> r = session.post('https://httpbin.org/post', json={'auth-token': 'D9FAEB3449D3'})
>>> assert r.from_cache is True

Request Headers:

As well as headers, if match_headers=True is used:

>>> session = CachedSession(ignored_parameters=['auth-token'], match_headers=True)
>>> session.get('https://httpbin.org/get', headers={'auth-token': '2F63E5DF4F44'})
>>> r = session.get('https://httpbin.org/get', headers={'auth-token': 'D9FAEB3449D3'})
>>> assert r.from_cache is True

Note

Since ignored_parameters is most often used for sensitive info like credentials, these values will also be removed from the cached request parameters, body, and headers.

Matching Request Headers#

Note

In some cases, request header values can affect response content. For example, sites that support i18n and content negotiation may use the Accept-Language header to determine which language to serve content in.

The server will ideally also send a Vary header in the response, which informs caches about which request headers to match. By default, requests-cache respects this, so in many cases it will already do what you want without extra configuration. Not all servers send Vary, however.

Use the match_headers option if you want to specify which headers you want to match when Vary isn’t available:

>>> session = CachedSession(match_headers=['Accept'])
>>> # These two requests will be sent and cached separately
>>> session.get('https://httpbin.org/headers', {'Accept': 'text/plain'})
>>> session.get('https://httpbin.org/headers', {'Accept': 'application/json'})

If you want to match all request headers, you can use match_headers=True.

Custom Request Matching#

If you need more advanced behavior, you can implement your own custom request matching.

Cache Keys#

Request matching is accomplished using a cache key, which uniquely identifies a response in the cache based on request info. For example, the option ignored_parameters=['foo'] works by excluding the foo request parameter from the cache key, meaning these three requests will all use the same cached response:

>>> session = CachedSession(ignored_parameters=['foo'])
>>> response_1 = session.get('https://example.com')          # cache miss
>>> response_2 = session.get('https://example.com?foo=bar')  # cache hit
>>> response_3 = session.get('https://example.com?foo=qux')  # cache hit
>>> assert response_2.cache_key == response_3.cache_key

Recreating Cache Keys#

There are some situations where request matching behavior may change, which causes previously cached responses to become obsolete:

  • You start using a custom cache key, or change other settings that affect request matching

  • A new version of requests-cache is released that includes new or changed request matching behavior (typically, most non-patch releases)

In these cases, if you want to keep using your existing cache data, you can use the recreate_keys method:

>>> session = CachedSession()
>>> session.cache.recreate_keys()

Cache Key Functions#

If you want to implement your own request matching, you can provide a cache key function which will take a PreparedRequest plus optional keyword args for request(), and return a string:

def create_key(request: requests.PreparedRequest, **kwargs) -> str:
    """Generate a custom cache key for the given request"""

You can then pass this function via the key_fn param:

session = CachedSession(key_fn=create_key)

**kwargs includes relevant BaseCache settings and any other keyword args passed to CachedSession.send(). If you want use a custom matching function and the existing options ignored_parameters and match_headers, you can implement them in key_fn:

def create_key(
    request: requests.PreparedRequest,
    ignored_parameters: list[str] = None,
    match_headers: list[str] = None,
    **kwargs,
) -> str:
    """Generate a custom cache key for the given request"""

Reference:

  • See create_key() for the reference implementation.

  • See the rest of the cache_keys module for some useful helper functions.

  • See Examples for a complete example of custom request matching.

Tip

As a general rule, if you include less information in your cache keys, you will have more cache hits and use less storage space, but risk getting incorrect response data back.

Warning

If you provide a custom key function for a non-empty cache, any responses previously cached with a different key function will be unused, so it’s recommended to clear the cache first.

Custom Header Normalization#

When matching request headers (using match_headers or Vary), requests-cache will normalize minor header variations like order, casing, whitespace, etc. In some cases, you may be able to further optimize your requests with some additional header normalization.

For example, let’s say you’re working with a site that supports content negotiation using the Accept-Encoding header, and the only variation you care about is whether you requested gzip encoding. This example will increase cache hits by ignoring variations you don’t care about:

from requests import PreparedRequest
from requests_cache import CachedSession, create_key


def create_custom_key(request: PreparedRequest, **kwargs) -> str:
    # Don't modify the original request that's about to be sent
    request = request.copy()

    # Simplify values like `Accept-Encoding: gzip, compress, br` to just `Accept-Encoding: gzip`
    if 'gzip' in request.headers.get('Accept-Encoding', ''):
        request.headers['Accept-Encoding'] = 'gzip'
    else:
        request.headers['Accept-Encoding'] = None

    # Use the default key function to do the rest of the work
    return create_key(request, **kwargs)


# Provide your custom request matcher when creating the session
session = CachedSession(key_fn=create_custom_key)