ElixirLLM.Cache (ElixirLLM v0.3.0)

Copy Markdown View Source

Response caching for LLM API calls.

Caches responses based on message content and model to reduce API costs and improve response times for repeated queries.

Configuration

config :elixir_llm, :cache,
  enabled: true,
  ttl_ms: 3_600_000,  # 1 hour
  max_entries: 1000

Example

# Automatic caching with chat
{:ok, response, chat} =
  ElixirLLM.new()
  |> ElixirLLM.with_cache()
  |> ElixirLLM.ask("What is 2+2?")

# Direct cache operations
cache_key = ElixirLLM.Cache.key(chat)
{:ok, response} = ElixirLLM.Cache.get(cache_key)
:ok = ElixirLLM.Cache.put(cache_key, response)

Cache Key Generation

Cache keys are generated from:

  • Model ID
  • Message history (content only, not metadata)
  • Temperature setting
  • Tool definitions (if any)

Summary

Functions

Returns a specification to start this module under a supervisor.

Clears all cached entries.

Deletes a cached entry.

Fetches a cached value or computes and caches it.

Gets a cached response.

Generates a cache key for the given chat.

Caches a response.

Returns the number of cached entries.

Starts the cache.

Types

cache_key()

@type cache_key() :: binary()

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

clear()

@spec clear() :: :ok

Clears all cached entries.

delete(key)

@spec delete(cache_key()) :: :ok

Deletes a cached entry.

fetch(key, fun)

@spec fetch(cache_key(), (-> {:ok, term()} | {:error, term()})) ::
  {:ok, term()} | {:error, term()}

Fetches a cached value or computes and caches it.

Example

ElixirLLM.Cache.fetch(cache_key, fn ->
  provider.chat(chat)
end)

get(key)

@spec get(cache_key()) :: {:ok, ElixirLLM.Response.t()} | :miss

Gets a cached response.

Returns {:ok, response} if found and not expired, :miss otherwise.

key(chat)

@spec key(ElixirLLM.Chat.t()) :: cache_key()

Generates a cache key for the given chat.

put(key, response)

@spec put(cache_key(), ElixirLLM.Response.t()) :: :ok

Caches a response.

size()

@spec size() :: non_neg_integer()

Returns the number of cached entries.

start_link(opts \\ [])

Starts the cache.