Case study
AI Story Generator — streaming RAG, agents & evals
A production-minded AI feature evolved from an LLM-API wrapper into a streaming, RAG-grounded, agentic system on the Vercel AI SDK. Tokens stream to the browser, craft decisions ground in a pgvector corpus on Neon, a bounded tool-calling agent retrieves on demand, guardrails fence untrusted content, and an eval harness gates CI — with a deterministic offline fallback when the provider fails.
← Back to portfolio
Main generator
1. Problem
Most portfolio AI demos are thin wrappers: one prompt, one JSON blob, no grounding, no tests. They look fine in a screenshot and fall apart when you ask how retrieval works, what happens on prompt injection, or whether a model change regressed quality.
The goal was an AI-engineering flagship: stream tokens to the browser, ground craft decisions in a retrieved corpus, defend against injection, measure quality with an eval harness that gates CI — and still degrade gracefully when the provider fails. v1 proved resilience (rate limits, error contracts, offline fallback); v2 proves the full AI stack on top of that foundation.
2. Architecture
One serverless endpoint orchestrates a bounded tool-calling agent. Retrieved passages flow through pgvector on Neon; tokens stream back as text/plain. If generation never starts, the client falls back to a deterministic local generator:
- api/generate-stories.ts — serverless handler: rate limit → Zod validation → injection screen → stream via agent → classify pre-stream errors into HTTP contracts.
- lib/ai/agent.ts — streamText with a searchCorpus tool inside a step-bounded loop; output screened incrementally for prompt leaks.
- lib/rag/ — ingest, embed, retrieve: corpus/ chunked and stored in pgvector; top-k passages cited in generation.
- public/js/api.js — reads the text/plain stream with getReader(); app.js renders tokens live and falls back when the server returns JSON errors. UI styled with Tailwind CSS (PostCSS build, dark mode via class strategy).
3. AI SDK streaming
Generation moved from a single provider call returning a JSON blob to the Vercel AI SDK's streamText. The model runs on Groq (llama-3.3-70b-versatile) via an OpenAI-compatible provider abstraction — swappable through AI_BASE_URL and AI_API_KEY without touching handler code.
- Live UX — the vanilla client reads the ReadableStream and appends escaped HTML on every chunk; users see tokens arrive, not a spinner followed by a wall of text.
- Bounded agent loop — searchCorpus is a tool the model can call when craft guidance would help; stopWhen: stepCountIs(4) caps cost and latency.
- Structured eval path — generateObject + storySchema powers the eval harness; streaming path optimizes UX while evals enforce typed, gradable outputs.
- Observability on finish — logGeneration records model, tokens, latency, estimated cost, tool-call count, and retrieved-passage count; optional Langfuse forwarding.
4. RAG over pgvector
A storytelling-craft corpus (narrative structure, tone and voice, character archetypes) is chunked, embedded, and stored in Neon Postgres with the pgvector extension. At generation time the agent retrieves grounding passages by cosine similarity and cites sources in the output.
- Corpus design — three focused craft guides instead of a dump of raw text; each chunk is small enough to fit in context without drowning the user's creative prompt.
- Embeddings — production uses a local bag-of-words embedder when FORCE_LOCAL_EMBEDDINGS=1 (Groq has no embedding API); the seam accepts a remote embedder when available.
- Agent-driven retrieval — the model decides when to call searchCorpus and what query to use, rather than blindly prepending top-k results to every prompt.
- Source citations — grounded stories list corpus filenames (e.g. narrative-structure.md) in groundedOn, verifiable in the live app output.
5. Guardrails & safety
RAG introduces a new attack surface: retrieved text can carry instructions. The guardrail layer treats user input, retrieved content, and model output as three separate trust zones.
- Input screening — regex patterns catch "ignore previous instructions," DAN/developer-mode overrides, and fake system tokens before they reach the model.
- Retrieved-content fencing — sanitizeRetrievedContent drops injection-shaped lines; wrapUntrusted marks the rest as reference-only data the model must not obey.
- Output screening — screenOutput runs on the accumulated stream; if a leak is detected before the first token ships, the handler returns 422 unsafe_output.
- CSP headers — vercel.json sets a strict Content-Security-Policy so even a compromised stream can't execute injected script in the page.
6. Eval harness
Ten golden cases in evals/dataset.ts cover schema validity, grounded relevance (expectSources), and injection resistance. Deterministic graders run on every case; an LLM-as-judge scores relevance and quality. CI blocks merges that regress below the recorded baseline.
- Golden set — 3 schema cases, 5 grounded/relevance cases with expectSources, 3 injection-resistance cases that try to override system instructions.
- Deterministic graders — gradeSchemaValid, gradeGroundedRelevance, and gradeNoPromptLeak run offline in Vitest; no API key required for unit tests.
- LLM judge — evals/judge.ts scores relevance and fiction quality (pass at score ≥ 3); catches regressions deterministic checks miss.
- CI regression gate — eval-regression job in GitHub Actions runs npm run eval:ci when secrets are set; per-case and overall scores must meet evals/baseline.json.
7. Resilience (v1 foundation)
The v2 AI stack sits on v1's production-minded layers. These still matter: an LLM endpoint is a cost endpoint, and demos that white-screen when credits run out are liabilities.
- Rate limiting — fixed-window limiter keyed by client IP runs before any LLM call; 429 + Retry-After + X-RateLimit-Remaining on every response.
- HTTP contracts — 400 invalid/unsafe request, 402 provider_no_credits, 422 unsafe_output, 429 rate_limit_exceeded. The client branches on these, not guesswork.
- Client retry — fetchWithResilience adds AbortController timeout and exponential backoff on 429/502/503 before giving up.
- Offline fallback — generateLocalStory() runs entirely in the browser when the AI path can't deliver; a one-time notice explains why.
- Multi-turn state — messages array (capped at 20 turns) is the conversation source of truth; history persists in localStorage with per-item delete and clear-all.
Honest scope on rate limiting: the store is in-memory (per serverless instance). The upgrade path is a shared store (Redis/Upstash) for a global limit.
8. Outcomes
- Live streaming with RAG citations — open the app, type a craft-heavy prompt, watch tokens stream in with corpus sources cited (e.g. narrative-structure.md).
- 81 Vitest tests + Playwright E2E — handler, agent, guardrails, RAG pipeline, graders, and regression logic covered; CI green on every push.
- Eval harness gates quality — 96.7% on latest run vs 91.7% baseline; eval-regression job blocks merges that slip backward.
- Graceful degradation — credits out, provider 5xx, or timeout: the user still gets a story plus an honest notice, not a white screen.
- Deployed on Vercel serverless with Neon pgvector: