SQLite#

../_images/sqlite.png

SQLite is a fast and lightweight SQL database engine that stores data either in memory or in a single file on disk.

Despite its simplicity, SQLite is a powerful tool. For example, it’s the primary storage system for a number of common applications including Dropbox, Firefox, and Chrome. It’s well suited for caching, and requires no extra configuration or dependencies, which is why it’s the default backend for requests-cache.

Cache Files#

  • See Cache Files for general info on specifying cache paths

  • If you specify a name without an extension, the default extension .sqlite will be used

In-Memory Caching#

SQLite also supports in-memory databases. You can enable this (in “shared” memory mode) with the use_memory option:

>>> session = CachedSession('http_cache', use_memory=True)

Or specify a memory URI with additional options:

>>> session = CachedSession(':file:memdb1?mode=memory')

Or just :memory:, if you are only using the cache from a single thread:

>>> session = CachedSession(':memory:')

Performance#

When working with average-sized HTTP responses (< 1MB) and using a modern SSD for file storage, you can expect speeds of around:

  • Write: 2-8ms

  • Read: 0.2-0.6ms

Of course, this will vary based on hardware specs, response size, and other factors.

Concurrency#

SQLite supports concurrent access, so it is safe to use from a multi-threaded and/or multi-process application. It supports unlimited concurrent reads. Writes, however, are queued and run in serial, so if you need to make large volumes of concurrent requests, you may want to consider a different backend that’s specifically made for that kind of workload, like RedisCache.

Connection Options#

The SQLite backend accepts any keyword arguments for sqlite3.connect():

>>> backend = SQLiteCache('http_cache', timeout=30)
>>> session = CachedSession(backend=backend)

API Reference#

DbCache

alias of requests_cache.backends.sqlite.SQLiteCache

DbDict

alias of requests_cache.backends.sqlite.SQLiteDict

DbPickeDict

alias of requests_cache.backends.sqlite.SQLitePickleDict

SQLiteCache

SQLite cache backend.

SQLiteDict

A dictionary-like interface for SQLite

SQLitePickleDict

Same as SQLiteDict, but serializes values before saving

requests_cache.backends.sqlite.DbCache#

alias of requests_cache.backends.sqlite.SQLiteCache

requests_cache.backends.sqlite.DbDict#

alias of requests_cache.backends.sqlite.SQLiteDict

requests_cache.backends.sqlite.DbPickeDict#

alias of requests_cache.backends.sqlite.SQLitePickleDict

class requests_cache.backends.sqlite.SQLiteCache(db_path='http_cache', **kwargs)[source]#

Bases: requests_cache.backends.base.BaseCache

SQLite cache backend.

Parameters
  • db_path (Union[Path, str]) – Database file path

  • use_cache_dir – Store datebase in a user cache directory (e.g., ~/.cache/http_cache.sqlite)

  • use_temp – Store database in a temp directory (e.g., /tmp/http_cache.sqlite)

  • use_memory – Store database in memory instead of in a file

  • fast_save – Significantly increases cache write performance, but with the possibility of data loss. See pragma: synchronous for details.

  • kwargs – Additional keyword arguments for sqlite3.connect()

bulk_delete(keys)[source]#

Remove multiple responses and their associated redirects from the cache, with additional cleanup

clear()[source]#

Clear the cache. If this fails due to a corrupted cache or other I/O error, this will attempt to delete the cache file and re-initialize.

contains(key=None, request=None, url=None)#

Check if the specified request is cached

Parameters
create_key(request=None, **kwargs)#

Create a normalized cache key from a request object

Parameters

request (Union[Request, PreparedRequest, CachedRequest, None]) –

Return type

str

property db_path#
Return type

Union[Path, str]

delete(*keys, expired=False, invalid=False, requests=None, urls=None)#

Remove responses from the cache according one or more conditions.

Parameters
  • keys (str) – Remove responses with these cache keys

  • expired (bool) – Remove all expired responses

  • invalid (bool) – Remove all invalid responses (that can’t be deserialized with current settings)

  • requests (Optional[Iterable[Union[Request, PreparedRequest, CachedRequest]]]) – Remove matching responses, according to current request matching settings

  • urls (Optional[Iterable[str]]) – Remove matching GET requests for the specified URL(s)

delete_url(url, method='GET', **kwargs)#
Parameters
  • url (str) –

  • method (str) –

delete_urls(urls, method='GET', **kwargs)#
Parameters
filter(valid=True, expired=True, invalid=False)#

Get responses from the cache, with optional filters

Parameters
  • valid (bool) – Include valid and unexpired responses; set to False to get only expired/invalid/old responses

  • expired (bool) – Include expired responses

  • invalid (bool) – Include invalid responses (as an empty CachedResponse)

Return type

Iterator[CachedResponse]

get_response(key, default=None)#

Retrieve a response from the cache, if it exists

Parameters
  • key (str) – Cache key for the response

  • default – Value to return if key is not in the cache

Return type

Optional[CachedResponse]

has_key(key)#
Parameters

key (str) –

Return type

bool

has_url(url, method='GET', **kwargs)#
Parameters
  • url (str) –

  • method (str) –

Return type

bool

keys(check_expiry=False)#
Parameters

check_expiry (bool) –

Return type

Iterator[str]

remove_expired_responses(*args, **kwargs)[source]#
reset_expiration(expire_after=None)#

Set a new expiration value on existing cache items

Parameters

expire_after (Union[None, int, float, str, datetime, timedelta]) – New expiration value, relative to the current time

response_count(check_expiry=False)#
Parameters

check_expiry (bool) –

Return type

int

save_response(response, cache_key=None, expires=None)#

Save a response to the cache

Parameters
update(other)#

Update this cache with the contents of another cache

Parameters

other (BaseCache) –

property urls#

Get all URLs currently in the cache (excluding redirects)

Return type

Iterator[str]

values(check_expiry=False)#
Parameters

check_expiry (bool) –

Return type

Iterator[CachedResponse]

class requests_cache.backends.sqlite.SQLiteDict(db_path, table_name='http_cache', fast_save=False, use_cache_dir=False, use_memory=False, use_temp=False, **kwargs)[source]#

Bases: requests_cache.backends.base.BaseStorage

A dictionary-like interface for SQLite

Parameters
  • use_cache_dir (bool) –

  • use_memory (bool) –

  • use_temp (bool) –

bulk_commit()[source]#

Context manager used to speed up insertion of a large number of records

Example

>>> d1 = SQLiteDict('test')
>>> with d1.bulk_commit():
...     for i in range(1000):
...         d1[i] = i * 2
bulk_delete(keys=None, values=None)[source]#

Delete multiple keys from the cache, without raising errors for any missing keys. Also supports deleting by value.

clear() None.  Remove all items from D.[source]#
close()[source]#

Close any active connections

connection(commit=False)[source]#

Get a thread-local database connection

Return type

Iterator[Connection]

get(k[, d]) D[k] if k in D, else d.  d defaults to None.#
init_db()[source]#

Initialize the database, if it hasn’t already been

items() a set-like object providing a view on D's items#
keys() a set-like object providing a view on D's keys#
pop(k[, d]) v, remove specified key and return the corresponding value.#

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() (k, v), remove and return some (key, value) pair#

as a 2-tuple; but raise KeyError if D is empty.

property serializer#
setdefault(k[, d]) D.get(k,d), also set D[k]=d if k not in D#
update([E, ]**F) None.  Update D from mapping/iterable E and F.#

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

vacuum()[source]#
values() an object providing a view on D's values#
class requests_cache.backends.sqlite.SQLitePickleDict(db_path, table_name='http_cache', fast_save=False, use_cache_dir=False, use_memory=False, use_temp=False, **kwargs)[source]#

Bases: requests_cache.backends.sqlite.SQLiteDict

Same as SQLiteDict, but serializes values before saving

Parameters
  • use_cache_dir (bool) –

  • use_memory (bool) –

  • use_temp (bool) –

bulk_commit()#

Context manager used to speed up insertion of a large number of records

Example

>>> d1 = SQLiteDict('test')
>>> with d1.bulk_commit():
...     for i in range(1000):
...         d1[i] = i * 2
bulk_delete(keys=None, values=None)#

Delete multiple keys from the cache, without raising errors for any missing keys. Also supports deleting by value.

clear() None.  Remove all items from D.#
close()#

Close any active connections

connection(commit=False)#

Get a thread-local database connection

Return type

Iterator[Connection]

get(k[, d]) D[k] if k in D, else d.  d defaults to None.#
init_db()#

Initialize the database, if it hasn’t already been

items() a set-like object providing a view on D's items#
keys() a set-like object providing a view on D's keys#
pop(k[, d]) v, remove specified key and return the corresponding value.#

If key is not found, d is returned if given, otherwise KeyError is raised.

popitem() (k, v), remove and return some (key, value) pair#

as a 2-tuple; but raise KeyError if D is empty.

property serializer#
setdefault(k[, d]) D.get(k,d), also set D[k]=d if k not in D#
update([E, ]**F) None.  Update D from mapping/iterable E and F.#

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k, v in F.items(): D[k] = v

vacuum()#
values() an object providing a view on D's values#
requests_cache.backends.sqlite.get_cache_path(db_path, use_cache_dir=False, use_temp=False)[source]#

Get a resolved cache path

Parameters
Return type

Path

requests_cache.backends.sqlite.sqlite_template(timeout=5.0, detect_types=0, isolation_level=None, check_same_thread=True, factory=None, cached_statements=100, uri=False)[source]#

Template function to get an accurate signature for the builtin sqlite3.connect()

Parameters