ElixirLLM.RateLimiter (ElixirLLM v0.3.0)

Copy Markdown View Source

Token bucket rate limiter for LLM API calls.

Provides client-side rate limiting to prevent hitting provider rate limits. Uses ETS for efficient concurrent access.

Configuration

config :elixir_llm, :rate_limiter,
  enabled: true,
  openai: [requests_per_minute: 60],
  anthropic: [requests_per_minute: 50],
  openrouter: [requests_per_minute: 100],
  ollama: [requests_per_minute: 1000]

Example

# Wait for a token before making a request
:ok = ElixirLLM.RateLimiter.acquire(:openai)

# Non-blocking check
case ElixirLLM.RateLimiter.try_acquire(:openai) do
  :ok -> # proceed
  {:error, :rate_limited} -> # handle rate limiting
end

# Check remaining tokens
tokens = ElixirLLM.RateLimiter.available(:openai)

Summary

Functions

Acquires a token for the given provider, blocking if necessary.

Returns the number of available tokens for a provider.

Returns a specification to start this module under a supervisor.

Resets the rate limiter for a provider.

Resets all rate limiters.

Starts the rate limiter.

Attempts to acquire a token without blocking.

Types

provider()

@type provider() :: atom()

Functions

acquire(provider, timeout \\ 5000)

@spec acquire(provider(), timeout()) :: :ok

Acquires a token for the given provider, blocking if necessary.

Returns :ok when a token is available.

available(provider)

@spec available(provider()) :: non_neg_integer()

Returns the number of available tokens for a provider.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

reset(provider)

@spec reset(provider()) :: :ok

Resets the rate limiter for a provider.

reset_all()

@spec reset_all() :: :ok

Resets all rate limiters.

start_link(opts \\ [])

Starts the rate limiter.

try_acquire(provider)

@spec try_acquire(provider()) :: :ok | {:error, :rate_limited}

Attempts to acquire a token without blocking.

Returns :ok if successful, {:error, :rate_limited} otherwise.