The Kit

Frameworks — the glue layer

LangChain The default name in LLM plumbing since 2022 — chains, retrieval, tool calling, integrations with practically everything, in Python and JavaScript. Its breadth is the pitch and the complaint: engineers' standing criticism is abstraction depth — debugging through five layers of wrappers to find one API call — and the API churn of its early years burned real goodwill, though the 0.x→1.x stabilization improved matters ⚑ unverified. The commercial arm is real: LangSmith (observability, below) and LangGraph Platform are the paid products; the framework is the funnel. Reach for it when: you want maximum integration coverage and a huge community answer-base; prototyping speed matters more than a minimal stack. Skip it when: your pipeline is simple enough to be three direct API calls — it often is.

LlamaIndex The retrieval specialist. Where LangChain sprawls across everything, LlamaIndex went deep on the data side of RAG: ingestion, parsing, indexing, query strategies. LlamaParse — its hosted document parser for the PDF horror-show (tables, multi-column layouts, scans) — is the standout paid piece, addressing the genuinely hard, genuinely boring problem at the front of every enterprise RAG pipeline. LlamaCloud is the managed platform around it. Reach for it when: the hard part of your problem is documents-in — complex parsing, structured retrieval over messy enterprise files. Skip it when: your corpus is clean text; you won't need most of it.

Haystack The grown-up in the room. Open-source (from deepset, a German company predating the ChatGPT wave — NLP pipelines since 2020), explicit pipeline architecture, fewer magic abstractions, a reputation for production stability rather than feature velocity. deepset's managed platform is the commercial layer. Reach for it when: you're building search/RAG for production and want boring, inspectable plumbing. Trade-off: smaller ecosystem and community than the two above; less of everything, including less to debug.

DSPy The contrarian: a research-born framework (Stanford) that treats prompts as parameters to be optimized rather than strings to be hand-crafted — declare the task signature, let the optimizer find the prompts/examples against your metric. Intellectually the most interesting item on this shelf and the clearest bet on "prompt engineering shouldn't be manual." Reach for it when: you have a measurable metric and hand-tuning has plateaued. Trade-off: you need evals first (it optimizes against them), and the mental model is a step away from everything else here.

Vercel AI SDK / provider SDKs (the no-framework option) Worth an honest entry: a growing share of production teams use no orchestration framework — direct provider SDKs, or a thin typed layer like Vercel's AI SDK for the frontend-streaming case. The provider APIs themselves absorbed the common patterns (tool calling, structured output, caching). Reach for it when: you value debuggability and your pipeline fits in your head. The strongest argument against every other entry in this subsection is that this one keeps working.

Vector databases — the storage shelf

Fuller comparison in the mini-listicle below and [asset: C26 Vector-DB comparison].

Pinecone The managed pioneer — the name that defined the category. Fully serverless, zero-ops, usage-priced; you never see an index server. The engineering is real and the DX is the smoothest in the category. Trade-offs: closed-source and cloud-only (no self-host, real lock-in), and costs that scale smoothly until they don't — the classic serverless bill curve. The category's incumbent-feature squeeze (Postgres et al.) pressures it hardest, because convenience is its whole pitch.

Weaviate Open-source vector database with batteries included: hybrid search built in, optional built-in vectorization modules (it can call the embedding model for you), GraphQL and REST APIs, managed cloud available. Feels most like a database product of the OSS trio. Trade-offs: more moving parts and memory appetite than minimal alternatives; the built-in-vectorization convenience blurs a boundary (embedding choice) you may want explicit.

Qdrant The performance-and-filtering pick. Rust-built, open-source, consistently strong on the benchmarks it publishes ⚑ unverified, with the best-regarded filtered-search story (payload filtering integrated into the HNSW traversal rather than bolted on) and honest quantization options for RAM savings. Managed cloud available. Trade-offs: leaner ecosystem than Weaviate's; fewer convenience layers — it assumes you know what you're doing. (House note: it's what this site's own stack runs on the EVO box — disclosed where relevant.)

Chroma The prototyping default. Open-source, pip-install, runs in-process with your Python — the SQLite of vector stores. Every RAG tutorial on the internet starts here, deliberately: zero setup between you and first retrieval. Trade-offs: the single-node comfort zone; its newer distributed/cloud story is young ⚑ unverified. Start here, expect to graduate.

pgvector The extension that deflated a category. Free, open-source vector search inside Postgres — your embeddings live in the same database as your application data, with real SQL joins, real transactions, one backup story, zero new infrastructure. HNSW indexing since 0.5 made it genuinely fast ⚑ unverified. For the large majority of workloads under a few million vectors, this is the boring, correct answer. Trade-offs: you tune it like Postgres (because it is Postgres); very large scale and advanced quantization are where the specialists still earn their keep.

Milvus The billion-scale option. Open-source, cloud-native architecture with separated storage/compute, GPU index support, built for the scale where everything else on this shelf gets uncomfortable. Zilliz Cloud is the managed offering from its creators. Trade-offs: operational complexity is the tax — it's a distributed system and acts like one. Choosing it for a ten-thousand-vector app is renting a container ship for a kayak trip.

The incumbents (the shelf-crasher entry) Elasticsearch/OpenSearch, Redis, MongoDB Atlas, and every major cloud's native offering all added vector search. None is the best vector database; all are the best already-in-your-stack option, and "good enough where my data already lives" is the strongest force in this category's economics. If you run Elasticsearch today, benchmark it before adding a new database to your pager rotation.

Agent frameworks — the loop shelf

LangGraph LangChain's second act and arguably its redemption: agents as explicit graphs — nodes, edges, state, checkpoints — rather than free-running loops. The design directly answers Guide 2's failure modes: persistence, replay, human-in-the-loop interrupts as first-class citizens. Has become the default serious choice in the LangChain ecosystem and beyond. Paired commercially with LangGraph Platform for deployment. Trade-off: graph-thinking is a real learning curve; simple agents don't need it.

CrewAI The multi-agent framework with the friendliest mental model: agents as a crew — roles, goals, tasks, delegation. Enormously popular as an on-ramp precisely because the metaphor does the teaching. Its enterprise platform is the commercial layer. Honest caveats: the role-play metaphor flatters the underlying reality (it's still prompts in a loop), and multi-agent crews multiply Guide 2's compounding- error problem — more agents is more places to fail, and the research record on multi-agent-vs-good-single- agent is far more mixed than the demos ⚑ unverified.

AutoGen Microsoft's research-grade multi-agent framework — agents as conversing peers, the framework that put multi-agent patterns on the map. Powerful, flexible, and visibly research-born; its API history is a sequence of significant rearchitectures ⚑ unverified. Reach for it when: you're exploring agent patterns and live in the Microsoft/Azure gravity well. Watch: Microsoft's own consolidation of its several agent frameworks.

The provider SDKs (OpenAI Agents SDK, Claude Agent SDK) The frameworks' existential question, in product form: both major labs now ship first-party agent scaffolding — loops, tools, handoffs, sessions — thin, opinionated, and maintained by the people who train the models ⚑ unverified. They lack third-party frameworks' breadth and cross-provider neutrality; they also lack the abstraction tax. The steady migration of capability from framework to provider is this shelf's defining trend — bet on frameworks for what providers won't build (cross-provider portability, deep customization), not for what they haven't built yet.

Pydantic AI The typed minimalist: agents built on Python's Pydantic validation — typed inputs, typed outputs, dependency injection, none of the metaphors. Reach for it when: you want agent structure with library-not-framework weight and you already trust Pydantic (you do; everything else on this shelf uses it under the hood).

Evals and observability — the measurement shelf

LangSmith LangChain's commercial centerpiece: tracing (every step of every chain/agent run, inspectable), eval datasets and runners, prompt versioning, production monitoring. If you're in the LangChain ecosystem it's the path of least resistance and genuinely good at the tracing half. Trade-offs: proprietary SaaS; deepest value assumes LangChain-shaped workloads, though it works standalone.

Braintrust The eval-first platform, built by ex-Figma/Impira infrastructure people and aimed squarely at the discipline in Guide 6: datasets, scorers, experiments, CI integration — evals as regression suite, plus a prompt playground and logging. Strong engineering reputation among teams that treat evals as software rather than vibes. Trade-off: SaaS pricing at scale; you bring the discipline, it brings the harness.

Ragas Open-source and RAG-specific: the standard library for scoring retrieval pipelines on their own axes — faithfulness (is the answer grounded in the retrieved context?), answer relevance, context precision/recall. Uses LLM-as-judge under the hood, so Guide 6's calibration caveat applies in full. Reach for it when: you built a RAG system and cannot currently answer "is the retrieval bad or the generation?" — it exists for exactly that question.

Langfuse The open-source observability alternative: tracing, evals, prompt management, cost tracking — self-hostable, framework-agnostic, with a managed cloud. The default pick when data can't leave your infrastructure, and the community darling of the self-host crowd. (House note: it's the observability layer on this site's own EVO stack — we eat this particular dog food.) Trade-off: you operate it; the managed SaaS competitors' polish is real.

Arize Phoenix Open-source LLM tracing and eval from Arize (ML observability incumbents), notable for being built natively on OpenTelemetry — the emerging answer to instrumentation lock-in: trace once in an open format, point the traces at whichever backend wins. That bet — observability as open plumbing, not platform — is the one to watch on this whole shelf.

promptfoo The pragmatist's eval tool: config-file-driven, CLI-first, red-teaming included — point it at prompts × models × assertions and get a comparison matrix in CI. Open-source. Reach for it when: you want eval coverage this week without adopting a platform.

Weights & Biases Weave / the ML-platform entrants The classical ML-ops incumbents (W&B, and cloud-platform equivalents) all extended into LLM tracing and evals. Strongest when your organization already lives there; the LLM-native tools above are typically sharper on this layer's specific problems.

Mini-listicle — Best vector DB for RAG (mid-2026 cut)

The honest version, by situation rather than by winner. Benchmarks vary by workload; run your own on your own data before committing — every vendor's benchmark chart has the vendor winning. Affiliate links present; verdicts unaffected. [asset: C26 Vector-DB comparison]

1. Best default for most teams — pgvector. If your app has a Postgres and your corpus is under a few million vectors, this is the answer: zero new infrastructure, real SQL, your vectors next to your data. The unfashionable pick and the most frequently correct one.
2. Best zero-ops managed — Pinecone. When nobody on the team should ever think about an index server, the serverless pioneer still has the smoothest ride. Price the growth curve before you marry it.
3. Best open-source all-rounder — Qdrant. Fast, memory-honest, and the strongest filtered-search story — which matters, because real RAG queries always carry filters ("this tenant, this date range"). Self-host or managed.
4. Best batteries-included — Weaviate. Hybrid search out of the box and optional built-in vectorization: the shortest path from documents to working hybrid retrieval if you want the database to do more of the thinking.
5. Best for prototypes and learning — Chroma. pip install, five lines, retrieval working. Start here to learn the mechanics; graduate when you meet production.
6. Best at genuine scale — Milvus. Hundreds of millions to billions of vectors is a different sport with different equipment. If you're actually there, you already know; if you're not sure you're there, you're not.
7. Best already-installed — your existing search stack. Elasticsearch/OpenSearch/Redis with vector support won't win the benchmarks, but "no new database" wins architecture reviews for a reason. Benchmark it first; adopt a specialist only when it loses.

The meta-verdict, consistent with the Dossier: the vector database is rarely why a RAG system is good or bad. Chunking, hybrid retrieval, reranking, and evals move quality far more than the storage engine does. Spend accordingly.