Author: Mihai Criveti Version: 0.1.0

Enforces rate limits per user, tenant, and tool across tool_pre_invoke and prompt_pre_fetch hooks. Supports pluggable counting algorithms (fixed window, sliding window, token bucket), an in-process memory backend (single-instance), and a Redis backend (shared across all gateway instances).

If any configured dimension is exceeded, the plugin returns a violation with HTTP 429. All requests include X-RateLimit-* headers. The most restrictive active dimension is surfaced (e.g. if both user and tenant limits are active, the one closest to exhaustion is reported).

- name: RateLimiterPlugin
  kind: plugins.rate_limiter.rate_limiter.RateLimiterPlugin
  version: "0.1.0"
  author: Mihai Criveti
  hooks:
    - prompt_pre_fetch
    - tool_pre_invoke
  mode: enforce          # enforce | permissive | disabled
  config:
    by_user: "30/m"      # per-user limit across all tools
    by_tenant: "300/m"   # shared limit across all users in a tenant
    by_tool:             # per-tool overrides (applied on top of by_user)
      search: "10/m"
      summarise: "5/m"

    # Algorithm — choose one (default: fixed_window)
    algorithm: "fixed_window"    # fixed_window | sliding_window | token_bucket

    # Backend — choose one
    backend: "memory"    # default: single-process, resets on restart
    # backend: "redis"   # shared across all gateway instances

    # Redis options (required when backend: redis)
    redis_url: "redis://redis:6379/0"
    redis_key_prefix: "rl"
    redis_fallback: true  # fall back to memory if Redis is unavailable

Rate string format: "<count>/<unit>" where unit is s/sec/second, m/min/minute, or h/hr/hour. Malformed strings raise ValueError at startup.

Omitting a dimension (e.g. no by_tenant) means that dimension is unlimited — no counter is tracked for it.

Every request (allowed or blocked) includes:

Three counting algorithms are available, selected via the algorithm config field.

Counts requests in a fixed time slot (e.g. "minute 14:03"). Resets at the slot boundary. Simple and fast. The 2× burst at a boundary (N requests at the end of slot T, N requests at the start of T+1) is a known trade-off; use by_user with headroom if this matters.

Stores a timestamp for every request in the current window. At each check, expired timestamps are discarded and the remaining count is compared against the limit. Prevents boundary bursts entirely. Memory usage grows with request volume — roughly one float per request per active key.

Each identity (user, tenant, tool) has a bucket that holds up to count tokens. Tokens refill at a steady rate of count/window. A request consumes one token. Bursts up to the bucket capacity are allowed; sustained rate above count/window is rejected. Useful for APIs where short spikes are acceptable but sustained overload is not.

Redis support: token_bucket with backend: redis is fully supported. The plugin stores {tokens, last_refill} in a Redis hash per key and uses an atomic Lua script to refill and consume tokens in a single round-trip — the same pattern as the other two algorithms. This means token_bucket enforces a true cluster-wide limit in multi-instance deployments.

• Counters are stored in a process-local dict (_store)
• An asyncio.Lock serialises all counter reads and writes — safe under concurrent asyncio tasks
• A background sweep task evicts expired windows every 0.5s — for fixed_window and token_bucket, expired entries are removed promptly; for sliding_window, keys with fully stale timestamps are evicted by the sweep
• Limitation: state is not shared across processes or hosts. In a multi-instance deployment (e.g. 3 gateway instances behind nginx), each instance tracks its own counter — the effective limit is N × configured_limit

• fixed_window: atomic Lua INCR+EXPIRE — one Redis round-trip per check, no race condition
• sliding_window: atomic Lua ZADD+ZREMRANGEBYSCORE+ZCARD+EXPIRE — one round-trip, no race condition
• token_bucket: atomic Lua script — reads {tokens, last_refill} hash, refills proportionally, consumes 1 token, writes back — one round-trip, no race condition
• All gateway instances share the same counter — the configured limit is the true cluster-wide limit
• Requires redis_url to be set
• If redis_fallback: true (default) and Redis is unavailable, the plugin falls back to the in-process MemoryBackend automatically — requests are never blocked due to Redis downtime
• If redis_fallback: false and Redis is unavailable, the exception is caught and the request is allowed through (fail-open)

Multi-instance deployment (important): The memory backend is local to a single gateway instance — rate limit counters are not shared across replicas. For multi-instance deployments (e.g., behind nginx or on OpenShift with multiple gateway pods), always use backend: redis to ensure rate limits are enforced correctly across all instances. The default production configuration (plugins/config.yaml) already sets backend: redis.

config:
  by_user: "60/m"
  by_tenant: "600/m"

config:
  backend: "redis"
  redis_url: "redis://redis:6379/0"
  redis_fallback: true
  by_user: "30/m"
  by_tenant: "3000/m"
  by_tool:
    search: "10/m"

config:
  algorithm: "sliding_window"
  by_user: "30/m"
  by_tenant: "300/m"

config:
  algorithm: "token_bucket"
  by_user: "30/m"   # bucket holds 30 tokens, refills at 30/min

config:
  algorithm: "token_bucket"
  backend: "redis"
  redis_url: "redis://redis:6379/0"
  redis_fallback: true
  by_user: "30/m"

mode: permissive
config:
  by_user: "60/m"

In permissive mode the plugin records violations and emits X-RateLimit-* headers but does not block requests. Useful for baselining traffic before switching to enforce.