Response caching for LLM API calls.
Caches responses based on message content and model to reduce API costs and improve response times for repeated queries.
Configuration
config :elixir_llm, :cache,
enabled: true,
ttl_ms: 3_600_000, # 1 hour
max_entries: 1000Example
# Automatic caching with chat
{:ok, response, chat} =
ElixirLLM.new()
|> ElixirLLM.with_cache()
|> ElixirLLM.ask("What is 2+2?")
# Direct cache operations
cache_key = ElixirLLM.Cache.key(chat)
{:ok, response} = ElixirLLM.Cache.get(cache_key)
:ok = ElixirLLM.Cache.put(cache_key, response)Cache Key Generation
Cache keys are generated from:
- Model ID
- Message history (content only, not metadata)
- Temperature setting
- Tool definitions (if any)
Summary
Functions
Returns a specification to start this module under a supervisor.
Clears all cached entries.
Deletes a cached entry.
Fetches a cached value or computes and caches it.
Gets a cached response.
Generates a cache key for the given chat.
Caches a response.
Returns the number of cached entries.
Starts the cache.
Types
@type cache_key() :: binary()
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec clear() :: :ok
Clears all cached entries.
@spec delete(cache_key()) :: :ok
Deletes a cached entry.
Fetches a cached value or computes and caches it.
Example
ElixirLLM.Cache.fetch(cache_key, fn ->
provider.chat(chat)
end)
@spec get(cache_key()) :: {:ok, ElixirLLM.Response.t()} | :miss
Gets a cached response.
Returns {:ok, response} if found and not expired, :miss otherwise.
@spec key(ElixirLLM.Chat.t()) :: cache_key()
Generates a cache key for the given chat.
@spec put(cache_key(), ElixirLLM.Response.t()) :: :ok
Caches a response.
@spec size() :: non_neg_integer()
Returns the number of cached entries.
Starts the cache.