Orchestration

Dossier

What this layer is

A language model, on its own, is a very expensive answering machine with amnesia. It knows the internet as of a training cutoff — frozen — and nothing about your documents, your database, your calendar, or the email that arrived while you were reading this sentence. It cannot look anything up. It cannot press a button. It cannot remember what you told it yesterday. Every conversation starts from zero, and every answer comes from weights that stopped learning months ago.

Orchestration is the layer that fixes all of that without touching the weights. It is the plumbing built around the model: retrieval systems that hand it the right documents at the right moment, tool interfaces that let it call real software, agent loops that let it take more than one step, memory systems that carry context across sessions, vector databases that make meaning searchable, evaluation harnesses that measure whether any of it actually works, and the prompt and context engineering that holds the whole assembly together.

The layer below this one (Foundation) produces intelligence. This layer produces products. The distinction matters more than almost anyone selling an AI product wants you to know.

What it does — the seven sub-systems

Retrieval (RAG). Retrieval-augmented generation connects a frozen model to live data. Documents get chopped into chunks, converted to embeddings (coordinates in a space of meaning), and stored in a vector database. At question time the system finds the chunks whose meaning sits nearest the question and pastes them into the model's context with an instruction: answer from these. The model's weights never change. You are not teaching it; you are briefing it, freshly, on every question. RAG is the load-bearing pattern of the entire layer — the mechanism behind almost every "chat with your documents," "ask your data," and enterprise AI search product on the market.

Agents. An agent is a model in a loop with permission to act. Instead of one prompt and one answer, the system runs: model proposes an action, software executes it, the result goes back into the context, model proposes the next action — repeat until done or until something breaks. The loop is what turns "write me a SQL query" into "find out why revenue dipped in March," a task that requires querying, reading the result, querying again, and deciding when the answer is good enough. Everything currently marketed as "agentic AI" is some version of this loop, dressed up to varying degrees.

Tools and function calling. The mechanism that lets a model touch the world. The developer describes available functions — name, parameters, what each does — and the model, instead of answering in prose, emits a structured request: call this function with these arguments. The surrounding software actually executes the call and feeds the result back. The model never runs anything itself; it writes extremely well-informed requests. This is the hinge on which the whole agent pattern swings, and the reason a chatbot can suddenly check inventory, book a meeting, or file a ticket.

Memory. Models are stateless; products need to not be. Memory systems store what happened — past conversations, user preferences, learned facts — and retrieve the relevant slice into context when it matters. Under the hood, most "memory" is RAG pointed at the conversation history: store what was said, embed it, retrieve what's relevant later. The hard part is not storage; it's deciding what deserves to be remembered, what should be summarized, and what should be allowed to fade.

Vector databases. The storage engines of the layer. Their one job: given a point in embedding space, find the nearest neighbors — fast, across millions or billions of vectors, with filters ("only documents this user can see, from the last 90 days"). A whole product category grew up around this single operation, then partially deflated when incumbents (Postgres, Elasticsearch, MongoDB, Redis) added vector search as a feature. The war between "vector-native database" and "vector search as a checkbox" is one of the layer's defining economic stories.

Evals. The measurement problem. Traditional software has tests: given input X, assert output Y. Model outputs are probabilistic prose — there is no single right answer to assert against. Evals are the discipline of measuring quality anyway: golden datasets, scoring rubrics, human review, and the slightly vertiginous practice of using one model to grade another (LLM-as-judge). Teams that ship reliable AI products are, almost without exception, teams that got serious about evals. It is the least glamorous work in the layer and the most predictive of who survives contact with production.

Prompting and context engineering. The craft of deciding what the model sees. Not the "10 magic prompts" of LinkedIn folklore — the engineering discipline of assembling a context window: system instructions, retrieved documents, tool definitions, conversation history, examples, all fitted into a finite budget where everything competes with everything else for the model's attention. The industry's own vocabulary shifted from "prompt engineering" to "context engineering" as it became clear the hard problem isn't the wording of the instruction; it's the curation of everything around it.

Why this layer exists

Because the alternative is retraining, and retraining is ruinous. Folding new knowledge into a model's weights costs somewhere between a fine-tuning run and a full pretraining run — and it still can't keep up with data that changes hourly. Orchestration exists because the economics of the Foundation layer forced it into existence: the model is expensive to change and cheap to brief, so the industry built an entire engineering discipline around briefing it well.

There is a second, quieter reason. The big labs' models converged in capability faster than anyone's product roadmap expected. When every competitor can call roughly the same intelligence over an API, the differentiation moves to what you wire around it: your data, your retrieval quality, your tool integrations, your evals. Orchestration is where AI products actually compete.

Where it breaks

This layer fails constantly, and the failure modes are structural, not bugs.

Retrieval quality is the ceiling. A RAG system can only be as good as what it retrieves, and retrieval fails quietly. Nearest-in-meaning is not most-relevant: the database returns what's similar to the question, which is not always what answers it. Chunking severs thoughts in half. And the worst failure is invisible by design: when the answer isn't in the corpus, the model doesn't say so — it answers anyway, fluently, from frozen memory. The system looks like it's working right up until someone checks.

Agent reliability compounds — downward. Chain steps together and the error rates multiply. A step that succeeds 95% of the time gives you roughly a 60% success rate over ten steps, and long-horizon agents run far more than ten ⚑ unverified. Agents also fail sideways: they loop, they burn tokens retrying the same broken call, they declare victory on tasks they have not completed. The gap between an agent demo (one curated happy path) and an agent product (every path, adversarial users, ambiguous instructions) is the largest demo-to-production gap in the industry, and it is where a large share of "agentic AI" revenue projections quietly go to die.

Evals are hard, so most teams skip them. You cannot unit-test prose. Golden datasets go stale. Human review doesn't scale. LLM-as-judge is genuinely useful and genuinely circular — models grading models, with documented biases toward verbosity, confidence, and answers that resemble their own. The practical result: a large fraction of deployed AI features ship with no systematic quality measurement at all, and their owners discover regressions the way their users do.

Context rot. The context window is the model's working memory, and it degrades as it fills. Long-context models advertise million-token windows; in practice, recall of any specific fact degrades as the window fills, and material buried in the middle is recalled worst — the "lost in the middle" effect, documented repeatedly since 2023 ⚑ unverified. Agents make this worse by accumulating context as they run: every tool result, every intermediate step, piling up until the signal drowns. Managing the context budget — what to keep, what to summarize, what to evict — is a permanent tax on every system in this layer.

The stack itself churns. The orchestration ecosystem iterates faster than any layer below it. Frameworks deprecate APIs in months; the "standard stack" of eighteen months ago is legacy today. Teams have rewritten the same RAG pipeline three times on three abstractions. This is the layer where betting on the wrong tool costs you a rewrite, not a config change — a fact worth holding onto when reading any vendor's category- defining blog post, including the ones linked from this page.

The economics — the honest part

Here is the thing most AI product marketing is structured to obscure: most "AI products" are mostly this layer. The model is an API call — often to the same three or four providers everyone else calls. What the product company actually built is retrieval over your data, tool integrations, an agent loop, evals, and a user interface. The intelligence is rented; the plumbing is the product.

That has consequences worth spelling out.

The margins live here, and so does the fragility. An application company's gross margin is largely determined by this layer: how efficiently it retrieves (fewer tokens in context = cheaper calls), how well it caches, how rarely its agents loop. But the layer is also where a platform shift hurts most — when a model provider ships memory, retrieval, or agent capabilities natively, whole categories of orchestration startups compress overnight. It has already happened repeatedly: context windows grew and "context stuffing" startups died; providers shipped built-in file search and a tier of RAG-as-a-service products lost their reason to exist ⚑ unverified.

The valuations are ahead of the durability. Venture capital priced parts of this layer as if orchestration were as defensible as the model layer. The vector database category alone raised on the order of hundreds of millions of dollars across Pinecone, Weaviate, Qdrant, Chroma, and Milvus/Zilliz ⚑ unverified before Postgres added a free extension that satisfies a large share of real workloads. The pattern repeats across the layer: the capability is real, the demand is real, and the moat is frequently an 18-month head start on an open-source alternative.

And yet the layer is where the real work is. None of the above means orchestration is froth. It means the value is in the practice, not the tooling: teams that master retrieval quality, agent reliability, and evals ship AI that works, on any framework or none. The tools in The Kit below are genuinely useful. Just hold both thoughts: this layer is where AI becomes product, and it is also where the bubble's revenue projections are least examined. When someone tells you their moat is their agent framework, the correct follow-up is: what happens when that's a model feature?

Anchored links: → Layer III · FOUNDATION — Embeddings (what a vector actually is) · → Layer V · APPLICATION — what ships on top of this plumbing · → BUBBLE WATCH — where orchestration-layer valuations meet the meter.

Concepts & Guides

Guide 1 — How RAG actually works, and where it breaks

The model doesn't know your documents. RAG — retrieval-augmented generation — is the plumbing that hands it the right pages at the right moment, so it answers from your world instead of its memory.

The whole trick, in the order it happens:

1. Chunk. Documents get split into passages, typically a few hundred tokens each. Chunk too big and the useful sentence drowns in noise; too small and the thought is severed mid-clause. Chunking is unglamorous and it decides everything downstream.
2. Embed. Each chunk is converted to an embedding — a vector of numbers acting as coordinates in a space of meaning. Passages that mean similar things land near each other even when they share no words. "Cancel my plan" and "how do I end my subscription" become neighbors.
3. Index. The vectors go into a vector database, whose one job is fast nearest-neighbor search across millions of chunks.
4. Retrieve. At question time, the question is embedded the same way, and the database returns the chunks whose meaning sits closest.
5. Augment. Those chunks are pasted into the model's context alongside the question, with an instruction: answer using these.
6. Generate. The model answers — grounded in retrieved text, not frozen memory.

No retraining. The weights never change. You're handing the model a cheat sheet, freshly assembled per question.

Where it breaks — the half most explainers skip:

• Similarity is not relevance. Embedding search returns what's about the same topic, which is not always what answers the question. A passage can be near the query in meaning-space and still be the wrong page. Production systems almost always add a second stage — a reranker, a model that reads the candidates and reorders them by actual relevance — because raw vector similarity alone isn't good enough.
• Embeddings miss exact matches. Semantic search is weak at part numbers, error codes, names, and jargon the embedding model never learned. Serious systems run hybrid search: vector similarity plus old-fashioned keyword matching (BM25), fused. The twenty-year-old algorithm is still on the team because it still wins on exact terms.
• Missing answers don't come back empty. If the answer isn't in the corpus, the model answers anyway — confidently, from training memory. Unless you engineer the refusal ("if the context doesn't contain the answer, say so"), you have built a fluent confabulation machine with citations. This is the RAG failure that ships to production most often.
• More retrieval is not better retrieval. Stuffing ten chunks in when two would do buries the signal — see context rot, Guide 7.
• Nobody measures it. Retrieval quality is separately measurable (did the right chunks come back?) from generation quality (was the answer right?), and most teams measure neither. When a RAG system is bad, the first question is always: is the retrieval bad, or the generation? Teams that can't answer that question can't fix their system.

The one-sentence version: RAG doesn't make the model smarter; it makes it better-briefed — and a well-briefed model beats a brilliant one talking about things it half-remembers.

Guide 2 — The agent loop

Strip away the marketing and every "AI agent" is the same four-beat loop:

1. Plan / decide. The model reads the goal plus everything that's happened so far, and emits its next move — either an answer, or a request to use a tool.
2. Act. The surrounding software executes the tool call — runs the query, fetches the page, edits the file. The model executes nothing itself.
3. Observe. The result — data, error message, file contents — is appended to the context.
4. Repeat. The model reads the updated context and decides again. The loop ends when the model declares the task done, or a guardrail stops it: a step limit, a cost ceiling, a human checkpoint.

That's the entire architecture. A coding agent, a research agent, a customer-service agent — same loop, different tools. The intelligence is rented from the model; the reliability is built in the loop: what tools it gets, what its context accumulates, where the stop conditions sit, and when a human gets asked.

Where it breaks:

• Errors compound. Each step's failure probability multiplies across the run. Long chains need per-step reliability that current models don't consistently deliver — which is why the agents that actually work in production (coding agents are the flagship case) operate in domains with a built-in verifier: the code compiles or it doesn't, the tests pass or they don't. Where reality pushes back cheaply, agents thrive. Where success is a judgment call, they wander.
• Loops and stalls. Agents retry the same failing call, reformulate the same failing plan, and burn tokens doing it. Every production agent framework exists partly to impose stop conditions the model won't impose on itself.
• Premature victory. Models are trained to be helpful, and "helpfully" declaring a task complete is a known failure mode. The fix is external verification, not asking the model if it's sure.
• Context accumulation. Every observation lands in the context window. A fifty-step run drags along a swamp of stale tool output, and the model's attention degrades exactly when the task gets long enough to matter. Serious agent systems summarize, prune, and checkpoint — Guide 7's discipline, applied on a timer.
• Security is unsolved in the general case. An agent that reads external content and holds tool permissions can be steered by that content — prompt injection. A poisoned web page telling the agent to exfiltrate what it knows is not hypothetical; it is the standing reason agent deployments gate the dangerous tools behind human approval. Treat every "fully autonomous" claim accordingly.

Guide 3 — Tool calling / function calling

How does a language model — a machine that only emits text — check a database? It doesn't. It asks.

The mechanism, precisely:

1. The developer sends the model a list of tool definitions: each tool's name, parameters, types, and a plain-language description of what it does. (These definitions consume context tokens like everything else — a fat toolbox is a real cost.)
2. The model, mid-conversation, decides a tool would help — and instead of prose, emits a structured tool call: the tool's name and arguments as JSON. Modern models are specifically trained for this; the provider guarantees the output parses. ⚑ unverified
3. The application — not the model — validates and executes the call, and returns the result as a new message.
4. The model reads the result and continues: answers, or calls another tool.

The model is a planner writing extremely well-informed requests; the application is the hands. This split is the security model, such as it is: the model can ask for anything, but only executes what you wired up.

Where it breaks:

• The description is the interface. The model chooses tools by reading their descriptions. Vague description, wrong tool. Overlapping descriptions, coin-flip. Writing tool definitions is API-design-for- a-reader-with-no-source-code-access, and most teams discover this only after watching an agent call search_orders when it needed get_order.
• Too many tools degrade choice. Hand a model dozens of tools and selection accuracy drops while the definitions eat the context budget. Production systems curate per-task toolsets or route to sub-agents with narrow toolboxes.
• Arguments can be well-formed and wrong. Valid JSON, plausible values, incorrect meaning — the wrong date range, the wrong customer ID. Schema validation catches malformed; nothing but downstream checks catches mistaken.
• MCP and the standardization wave. The Model Context Protocol — an open standard for connecting tools to models, originated by Anthropic in late 2024 and since adopted broadly across the industry ⚑ unverified — turned tool integration from N×M custom wiring into a plug standard. It also inherits every security problem above and adds supply-chain flavor: a malicious or compromised tool server sits inside your agent's trust boundary. Standardized plumbing, standardized attack surface.

Guide 4 — Vector search and chunking

Two decisions dominate RAG quality, and neither is the model: how you cut the documents, and how you search the pieces.

Chunking. The corpus must be split before it's embedded, and every split is a bet about what "one idea" looks like. Fixed-size chunks (say, 500 tokens with overlap) are simple and blind — they cut mid-sentence, mid-table, mid-thought. Structure-aware chunking follows the document's own seams: headings, paragraphs, code blocks. Semantic chunking embeds sentences first and cuts where the meaning shifts. The advanced move is enrichment: storing each chunk with a model-generated line of context about where it came from ("contextual retrieval" — prepending document-level context to each chunk before embedding, which Anthropic reported cuts retrieval failure rates substantially ⚑ unverified). The empirical, slightly deflating truth: chunking choices move retrieval quality more than embedding-model choices do, and there is no universal best — only best-for-your-corpus, found by measuring.

The search itself. At scale, exact nearest-neighbor search (compare the query to every vector) is too slow, so every vector database runs approximate nearest-neighbor (ANN) search — almost always a graph index called HNSW (hierarchical navigable small world), or a partitioning scheme like IVF. "Approximate" is a real word: the index trades a little recall for a lot of speed, and that trade is tunable. A system returning 95% of the true nearest neighbors at 10x the speed is usually the right trade — but it means vector search can silently miss the best chunk, by design.

Hybrid, always. Dense vectors carry meaning; sparse keyword search (BM25) carries exact terms. Dense search finds "how do I terminate my contract" when the document says "cancellation policy"; keyword search finds "error E-4402" when the embedding model has never seen it. Production retrieval fuses both (reciprocal rank fusion is the standard trick), then typically hands the top candidates to a reranker for a final relevance-ordered cut. Retrieval is a pipeline, not a lookup.

Where it breaks: embedding models are frozen too (new jargon, new product names — invisible until re-embedding); metadata filters interact badly with ANN indexes (filter-then-search vs search-then-filter is a genuine engineering problem each database solves differently); and re-embedding an entire corpus on every embedding-model upgrade is a real, recurring cost that nobody puts on the architecture diagram.

Guide 5 — Memory

Every model API call is stateless: the model remembers nothing you didn't put in the context window, this call, right now. Everything that feels like memory in an AI product is engineered outside the model.

The layers of the illusion, from crude to clever:

• Conversation history — the transcript so far, re-sent with every call. This is why long chats get slow and expensive: the model re-reads the whole conversation every turn. Past a threshold, systems summarize older turns and carry the summary instead — lossy compression, chosen deliberately.
• Retrieved memory — RAG pointed at the past. Store conversation snippets and extracted facts in a vector store; at each turn, retrieve what's relevant to now and inject it. Feels like recall; is search.
• Structured memory — an explicit, curated store of facts ("prefers Python," "works at Acme, ships Thursdays") that the system writes to deliberately — often via the model itself deciding, tool-call style, this is worth remembering. This is roughly how consumer "memory" features in the big assistants work ⚑ unverified, and it introduces the interesting failure: the model curates its own memory, so it can remember wrong.
• Agentic memory files — coding agents reading and writing plain files of notes and instructions that persist across sessions. Memory as a document the agent maintains. Crude, inspectable, surprisingly effective.

Where it breaks: relevance failure (the retrieval misses the one thing you told it, or dredges up last month's stale preference); stale facts (memories don't expire when reality changes — you changed jobs, the memory didn't); compounding errors (a wrong memory, written once, is retrieved and reinforced forever); and privacy (a memory store is a dossier — what's collected, how long it lives, and whether deletion actually deletes are product decisions wearing an engineering costume).

Guide 6 — Evals, and why they're hard

Ask ten engineers what finally made their AI feature reliable and nine will say the same unglamorous word: evals. Ask why most teams don't have any, and you get this guide.

Traditional software is testable because it's deterministic: input X must produce output Y. A model's output is probabilistic prose. There is no Y. "Summarize this contract" has ten thousand acceptable answers and no oracle that recognizes them. So the discipline had to be rebuilt from scratch:

• Golden datasets — a curated set of real inputs with reference answers or grading criteria, run on every change. The AI equivalent of a regression suite, except every entry costs human judgment to create and goes stale as the product and its users drift.
• Programmatic checks — the honest cheap layer: did the output parse, did it cite a real document, did it stay under length, did the code compile. Catches the embarrassing failures; says nothing about quality.
• Human review — the gold standard, unscalable by definition. Used to calibrate everything else.
• LLM-as-judge — the load-bearing compromise: a model grades the outputs against a rubric. Scales beautifully. Also documented to prefer verbose answers, confident answers, answers resembling its own style, and — in pairwise comparisons — whichever answer comes first ⚑ unverified. Usable, but only after you've validated the judge against human judgment on a sample. An uncalibrated judge is a random-number generator with gravitas.

Why it's genuinely hard, not just neglected: the criteria themselves are fuzzy (what makes a summary "good" is a product decision nobody wrote down); the target moves (every model upgrade, prompt tweak, and retrieval change shifts behavior — your gains on the golden set may be losses on what users actually type); and offline metrics correlate imperfectly with online outcomes, which is why mature teams close the loop with production monitoring: sample real traffic, grade it continuously, alert on drift. Evals are not a phase; they're a subscription.

The honest summary: in this layer, evals are the difference between engineering and vibes. A team that cannot measure quality cannot improve it — they can only change things and feel optimistic. Most of the observability tools in The Kit below exist because this problem is real, unsolved in general, and worth paying for in particular.

Guide 7 — Prompt and context engineering

The context window is everything the model can see when it answers: instructions, documents, tool definitions, history, examples. Assembling it well used to be called prompt engineering; the field renamed itself context engineering when it noticed the wording of the instruction was the easy part.

The budget. Every token in context costs money (per-token pricing), latency (long prefills are slow), and — critically — attention. The window is a fixed budget in which the system prompt, the retrieved chunks, the tool definitions, and the conversation so far all compete. Context engineering is deciding, per call, what earns its place.

Context rot. Advertised context windows are enormous — millions of tokens ⚑ unverified. Effective use of them is not. Documented since 2023: models recall information at the start and end of a long context better than the middle ("lost in the middle"), and per-fact recall degrades as the window fills — the marketing number is the size of the room, not the model's attention span inside it. Agents suffer worst, because the loop accumulates context: every tool result, every intermediate step, stacking up until the one line that matters is buried under forty steps of scaffolding.

The working craft, as practiced rather than as blogged:

• Position deliberately. Instructions and the most important material go where recall is best — start and end. The middle is for what you can afford to have skimmed.
• Curate harder than you write. Cutting a stale document out of context typically beats any rewording of the instructions. The best prompt engineers spend most of their time deleting.
• Summarize on a schedule. Long agent runs compact their own history — dropping raw tool output, keeping conclusions — and checkpoint state outside the window (files, scratchpads) rather than dragging it along.
• Cache the static prefix. Providers price repeated context cheaply if the prefix is identical (prompt caching, at large discounts ⚑ unverified), which quietly dictates architecture: stable content first, volatile content last — an economic constraint masquerading as a style guide.
• Examples beat adjectives. Three worked examples of the output you want outperform a paragraph of quality adjectives. "Be concise and professional" is a wish; a sample is a spec.

Where it breaks: prompts are load-bearing and unversioned in too many shops (the production incident where nobody knows what changed, because the prompt lives in a string constant); instructions and retrieved content contest each other (which is also the prompt-injection surface — the model has no hard boundary between "what you must do" and "what you happen to be reading"); and everything is model-specific — a tuned prompt is an optimization against one model's quirks, and the next upgrade re-rolls the dice. Which is, one final time, why the evals guide sits directly above this one.

Anchored links: → Layer III · FOUNDATION — Embeddings · Context windows and attention · → Layer V · APPLICATION — Guardrails · Cost per token · → this layer's Kit, below, for the tools that implement all seven guides.

The Kit

The real tools, vendors, and models of the Orchestration layer — what each one is, when to reach for it, and the honest caveats. This is the layer’s directory slice: 12 entries are flagged affiliate-eligible, and none carry a live link yet — mentions stay plain until partner programs exist, and links will activate by data change, not re-edit.

Winner Matrix

News & Video

What this layer's feed covers

Tag to ORCHESTRATION when the story is about the wiring, not the model or the app:

• Framework and tooling releases — LangChain/LangGraph, LlamaIndex, Haystack, DSPy, CrewAI, AutoGen, provider agent SDKs: major versions, breaking changes, pivots, funding, shutdowns.
• Vector DB category news — releases, benchmarks (label vendor-published as such), pricing changes, raises, and the ongoing incumbent squeeze (Postgres/Elastic/Redis/Mongo vector features).
• Protocol and standards — MCP ecosystem developments, tool-calling API changes across providers, agent-interoperability standards efforts.
• Agent reliability and security — prompt-injection research and incidents, agent benchmark results (SWE-bench-class, web-agent benchmarks), long-horizon reliability studies. Incidents are BUBBLE WATCH crossover candidates when they puncture a revenue narrative.
• RAG and retrieval research — retrieval techniques, context-rot/long-context studies, reranking and embedding developments as they affect pipelines (pure model-training news → Foundation).
• Evals and observability — new eval methods, LLM-as-judge bias findings, observability tooling and the OpenTelemetry standardization thread.
• Layer economics — orchestration-startup funding/acquisitions/shutdowns, provider features that absorb startup categories (each of these is also a BUBBLE WATCH signal — file both).

Boundary rules for the classifier: model capabilities and training → Foundation; end-user product launches → Application; a product story whose substance is its retrieval/agent architecture → Orchestration. When a story is a valuation story wearing a tooling costume, cross-file to Bubble Watch with the clinical labels (PULSE/DRIFT/GAP/NOISE/AUDIT).

Source roster (feed inputs for this layer)

Engineering blogs (first-party, high signal, know their incentives): LangChain blog · LlamaIndex blog · Pinecone blog (their learning-center explainers are category-best even where the product pitch follows) · Qdrant, Weaviate, Chroma blogs · Anthropic engineering blog (MCP, agent-building posts) · OpenAI developer/engineering posts · deepset/Haystack blog.

Practitioners (the honest middle, highest weight): Simon Willison's weblog (the single best running record of this layer's reality, prompt-injection especially) · Eugene Yan · Hamel Husain (evals) · Chip Huyen · Jason Liu (RAG consulting-trench reports) · Lilian Weng (agent/memory surveys) · Sebastian Raschka (when he crosses into this layer).

Aggregators and research: Hacker News (the layer's de-facto letters page) · arXiv cs.CL/cs.AI filtered to RAG/agents/evals keywords · Latent Space (podcast + newsletter — the AI-engineer scene's trade paper) · The Pragmatic Engineer (when it covers AI tooling adoption).

Category trackers: vendor changelogs for everything in The Kit (release-note RSS where offered) · GitHub release feeds for the OSS entries (LangChain, LlamaIndex, Haystack, DSPy, Qdrant, Chroma, Milvus, pgvector, Langfuse, Phoenix, promptfoo, Ragas) — the Hub radar already watches repos; this layer's slice filters to the Kit list.

Video (curated embeds, tagged to this layer)

Seed the shelf with durable explainers over news-cycle clips; every embed gets a one-line editorial caption saying what it's good for. Candidates to source and verify at build time ⚑ unverified:

• The agent loop, from someone who builds them — an AI Engineer Summit / AIE World's Fair talk on agent architecture in production (the conference is this layer's home venue; pick the current year's standout).
• Karpathy on LLM-as-OS / "LLM OS" — the frame that makes orchestration legible to a general audience: the model as CPU, context as RAM, tools as peripherals.
• A real RAG walkthrough with failure modes — not a vendor quickstart; a talk or long-form tutorial that covers chunking, hybrid, reranking, and eval (candidates from practitioner channels; verify against Guide 1 before embedding).
• Prompt injection, demonstrated — Simon Willison's conference talk(s) on prompt injection; the clearest articulation of the layer's standing security problem.
• Evals in practice — a Hamel Husain / practitioner talk on building eval suites (the "your AI product needs evals" genre; pick the one with real artifacts on screen).
• MCP explained by its authors — an Anthropic talk or official deep-dive on the protocol, paired editorially with an independent critique for balance.

Explainer-to-guide linkage: each video embed sits next to the matching guide (Guide 1 ↔ RAG walkthrough, Guide 2 ↔ agent talk, Guide 3 ↔ MCP, Guide 6 ↔ evals talk, Guide 7 ↔ Karpathy) so the feed feeds the encyclopedia rather than floating free.

Affiliate note for this section

News and video stay clean — no affiliate links inside feed items or editorial captions. Sponsored slots, if sold, are labeled SPONSORED, visually distinct, and never adjacent to a negative story about a competitor of the sponsor. The Kit and the listicle above are this page's affiliate surface; the feed is not.