RAG Engine

A fork-and-deploy DIY application for loading external data with Render Workflows, storing canonical documents in Postgres/pgvector, embedding through one private service, and exposing citation-backed search through MCP.

source adapter → orchestrated sync → Postgres/pgvector
                                          ↑       ↓
                              private embeddings  MCP → agent

• json loads six credential-free example documents.
• gong is a reference for pagination, calls, transcripts, speakers, CRM projections, and retry behavior.

Both produce the canonical Document → ContentUnit → Chunk model.

Requirements: Docker, Python 3.12+, and Render CLI 2.28+ for Workflow testing.

To exercise the Workflow locally, set RENDER_USE_LOCAL_DEV=true (commented in .env.example), run python -m app.workflows.main, and dispatch a sync with python scripts/start_load_workflow.py. Roughly a third of pytest requires a live database, so start Docker Compose and run migrations first.

cp .env.example .env
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements-dev.txt
docker compose up -d postgres

docker build -f embedding-service/Dockerfile -t rag-embeddings .
docker run --rm -d --name rag-embeddings \
  -p 10000:10000 \
  -e PORT=10000 \
  -e EMBEDDING_API_KEY=dev-embedding-key \
  rag-embeddings

python scripts/migrate.py
python scripts/load.py \
  --from 2026-09-01T00:00:00Z \
  --to 2026-10-01T00:00:00Z \
  --update-checkpoint
python scripts/doctor.py
pytest -q

The date window is inert for the default adapter: rag-engine.yaml sets ignore_window: true on the JSON source, so discover skips both bounds. Pass --skip-embeddings to scripts/load.py for a database-only smoke test that does not need the embedding container.

render.yaml creates rag-db, one private rag-embeddings service, rag-mcp, and rag-scheduler. Render Workflows are created separately as rag-pipeline:

Build command: pip install -r requirements.txt
Start command: python -m app.workflows.main

Give the Workflow DATABASE_URL, EMBEDDING_SERVICE_URL, EMBEDDING_API_KEY, and credentials for enabled adapters. The scheduler only needs its Render API key and dispatches rag-pipeline/load_source.

render blueprints validate ./render.yaml

There is no render blueprint launch command — blueprints validate is the only subcommand. Launch the Blueprint from the Dashboard with New → Blueprint against the connected repository.

Two Blueprint settings affect the runbook: rag-mcp runs python scripts/migrate.py as its preDeployCommand, so migrations run on every MCP deploy and a profile mismatch fails the deploy; and rag-embeddings has autoDeployTrigger: off, so it must be deployed manually.

Satisfy the SourceAdapter protocol in app/contracts.py — it is structural typing, so do not subclass it. Register the constructor in app/registry.py, which is keyed by adapter kind, then add a manifest entry in rag-engine.yaml, which is keyed by adapter id. Add projection migrations only when generic metadata and entities are insufficient; they belong under app/adapters/<kind>/migrations/, resolved by kind rather than id.

app/adapters/json/ is the contract reference and app/adapters/gong/ shows production complexity: authenticated pagination, rate limiting, entities, projection tables, and optional MCP projections.

• Configuration, MCP, and diagnostics
• Storage, sync correctness, and deletion
• Model-change maintenance