Some developers say it is essential. Others say you can build AI applications without it.

So what exactly is LangChain, why was it created, and when should you actually use it?

Let's start with the problem it was designed to solve.

The Problem: Building an AI Application

Imagine you're building an AI chatbot using a model such as GPT or Claude.

At first, it is simple:

But real AI applications quickly become more complicated.

What if your chatbot needs to:

• answer questions from your company documents?
• search a database?
• remember previous conversations?
• call an external API?
• use different AI models for different tasks?
• use tools to complete actions?

Soon, your application contains a lot of custom code just to connect all these pieces together.

This is the problem LangChain was built to help solve.

LangChain is an open-source framework for building applications and agents powered by large language models. It provides reusable building blocks for connecting models, prompts, tools, data, and application logic rather than requiring you to build all of that infrastructure yourself.

A simple way to think about LangChain is:

The language model is the brain. LangChain is the framework that helps connect the brain to everything else.

Let's look at some of its most important concepts.

1. Prompt Templates

Normally, you might write a prompt directly inside your code.

For example:

Give me the top programming languages to learn in 2026.

But in a real application, part of that prompt might need to change.

Instead, you could create something like:

Give me the top programming languages to learn in {year}.

Now {year} is a variable.

Your application can replace it with:

2026
2027
2028
...

This is the idea behind a prompt template: create a reusable prompt and insert dynamic values when the application runs.

Instead of manually rebuilding your prompt every time, you keep the structure and change only the information that needs to be different.

That makes prompts easier to reuse and maintain.

2. Chains

As the name suggests, a chain is a sequence of steps connected together.

Imagine a user asks your application a question.

The application might:

User Question
      ↓
Format Prompt
      ↓
Send to LLM
      ↓
Format Response
      ↓
Return Answer

Instead of treating each step as completely separate code, you can connect them into a workflow.

The useful idea here is modularity.

If you later want to replace the model, change the prompt, or add another processing step, you can change one part without rebuilding the entire application.

You will still hear the word chain used to describe this style of LLM workflow. In current LangChain, however, the framework is increasingly centered around composable model, tool, prompt, middleware, and agent building blocks; older chain classes such as LLMChain are now part of the legacy langchain-classic package.

The underlying idea remains simple:

Connect several AI-processing steps into one workflow.

3. Document Retrieval

Large language models do not automatically know what is inside your private company documents.

Imagine you ask:

"What is our company's refund policy?"

The answer might exist in a PDF or internal documentation, but the model does not automatically have access to it.

So the application needs to find the relevant information first.

A simple retrieval workflow looks like this:

User asks a question
        ↓
Search documents
        ↓
Find relevant information
        ↓
Send question + information to LLM
        ↓
Generate answer

This approach is commonly called Retrieval-Augmented Generation, or RAG.

LangChain provides components for building retrieval systems, including document loaders, text splitters, retrievers, and integrations with vector stores. Document loaders bring information from external sources into a standard format, while text splitters can divide large documents into smaller chunks that are easier to retrieve and fit into a model's context window.

This allows an AI application to answer questions using information from sources such as:

• PDFs
• company policies
• product documentation
• knowledge bases
• internal documents

So instead of relying only on what the model learned during training, you can provide it with the information it needs at the time of the question.

4. Memory

Now imagine this conversation:

You: What is Docker?

The AI gives you an explanation.

A few seconds later, you ask:

You: How is it different from Kubernetes?

What does "it" mean?

A human understands that you are still talking about Docker.

An AI application also needs access to the previous conversation to understand that context.

This is where memory comes in.

LangChain supports short-term memory that allows an agent to retain information from earlier interactions within the same conversation or thread.

Conceptually:

User: What is Docker?
        ↓
AI answers
        ↓
Conversation remembered
        ↓
User: How is it different from Kubernetes?
        ↓
AI understands "it" = Docker

Without the previous context, the second question is much harder to interpret correctly.

Memory helps make AI applications feel like ongoing conversations instead of a series of unrelated prompts.

5. Agents

The final major concept is agents.

A normal workflow often has steps that you define in advance.

For example:

Step 1 → Step 2 → Step 3

An agent is different because it can decide what to do next.

Imagine asking:

"Check today's weather and send me an email if it's going to rain."

The AI might need to:

Understand request
        ↓
Call Weather Tool
        ↓
Check Forecast
        ↓
Is rain expected?
     ↙       ↘
   Yes       No
    ↓         ↓
Send Email   Finish

You did not have to tell the agent:

First call the weather API, then inspect the result, then call the email service.

You gave it a goal.

The agent decides which tools it needs and can continue using tools until the task is complete. This model-and-tool loop is how LangChain currently describes its agent architecture.

That makes agents useful for applications where the exact sequence of actions is not always known ahead of time.

Should You Always Use LangChain?

No.

If your entire application does this:

Prompt → LLM → Answer

then using the OpenAI, Anthropic, or another model provider's SDK directly may be simpler.

You do not need a framework simply because you are using an LLM.

LangChain becomes more useful as you start needing things such as:

• reusable prompts
• document retrieval
• conversation memory
• multiple models
• tool calling
• agents
• more complex AI workflows

LangChain's current focus is to provide a relatively easy starting point for building LLM applications and agents while still allowing developers to customize the model, tools, prompts, middleware, and surrounding behavior.

Quick Recap

The main LangChain concepts are straightforward:

And underneath all of this, the language model is still the brain.

LangChain provides the framework around that model that helps connect it to prompts, information, tools, and application logic.

So, should you learn LangChain?

If you only want to send a prompt to an LLM and display the answer, probably not yet.

But if you want to build AI applications that work with documents, memory, tools, retrieval, or agents, understanding LangChain—and the concepts behind it—is well worth your time.

They are two distinct approaches to artificial intelligence.

Most of us are already familiar with generative AI through chatbots, image generators, coding assistants, and similar tools.

The simplest difference is:

Generative AI creates content in response to a prompt. Agentic AI pursues a goal through a series of actions.

Let's look at how each works.

What Is Generative AI?

Generative AI systems are fundamentally reactive.

They wait for you to give them something to work with, usually a prompt. Once prompted, their job is to generate content based on patterns learned during training.

That content might be:

• text
• images
• code
• audio
• video

A language model, for example, learns statistical relationships between words from very large datasets. When you give it a prompt, it predicts a useful response based on those learned patterns.

But its work usually ends at generation.

Human prompt
     ↓
Generative AI
     ↓
Generated content
     ↓
Human reviews

It does not normally continue taking actions unless you give it another instruction.

A Simple Generative AI Example

Imagine a YouTube creator preparing a new video.

They might use generative AI to:

• review a script
• suggest thumbnail ideas
• write a description
• generate background music

But there is still a human directing the process.

The AI suggests ideas. The creator reviews them, chooses what works, and asks for changes where necessary.

The AI generates possibilities, but the human curates them.

That is how many people use generative AI today.

What Is Agentic AI?

Agentic AI works differently.

Instead of generating something and stopping, an AI agent can pursue a goal through multiple actions.

It may start with a user request, but then it can decide what to do next, take an action, inspect the result, and continue.

A simple agent cycle looks like this:

Perceive
   ↓
Decide
   ↓
Act
   ↓
Evaluate
   ↓
Repeat

The cycle can continue with minimal human intervention until the goal is reached or the agent needs input.

This makes agentic AI useful for tasks involving multiple steps, decisions, and ongoing management.

A Simple Agentic AI Example

Imagine you want to buy a particular laptop.

With generative AI, you might ask:

"Which laptop should I buy for video editing?"

The AI can generate recommendations, but you still need to search stores, compare prices, check availability, and make the purchase yourself.

An AI shopping agent could go further.

You might give it a goal:

"Buy this laptop for me for less than $1,500."

The agent could then:

Search stores
     ↓
Compare options
     ↓
Check availability
     ↓
Monitor prices
     ↓
Choose the best option
     ↓
Request approval if needed
     ↓
Purchase
     ↓
Track delivery

Instead of simply telling you what to do, the agent takes actions toward completing the task.

That is the key difference:

Generative AI produces an output. Agentic AI works toward an outcome.

A Common Foundation

Generative AI and agentic AI often share the same underlying technology.

For text-based systems, that foundation is usually a large language model, or LLM.

LLMs power many generative AI applications such as chatbots, but they can also provide reasoning capabilities inside AI agents.

With a chatbot, the interaction might look like:

Prompt → LLM → Response

An agent can use the same type of model as part of a larger process:

Goal
 ↓
Decide what to do
 ↓
Use a tool
 ↓
Inspect the result
 ↓
Decide what to do next
 ↓
Continue until complete

The agent may also have access to tools such as:

• web search
• databases
• APIs
• calendars
• email
• code execution

This allows the AI to do more than generate content. It can interact with its environment and take action.

How Do AI Agents Handle Complex Tasks?

One useful capability of an AI agent is breaking a larger goal into smaller steps.

Imagine asking an agent to organize a conference.

It first needs to understand things such as:

• number of attendees
• location
• dates
• budget
• venue requirements

It can then work through the problem step by step:

Understand requirements
        ↓
Research venues
        ↓
Compare suitable venues
        ↓
Check availability
        ↓
Compare prices
        ↓
Request approval
        ↓
Continue planning

If a venue is unavailable, the agent does not necessarily have to stop.

It can evaluate the result, choose another option, and continue.

This ability to plan, act, evaluate, and repeat is central to agentic AI.

Generative AI is still important here because the LLM can help the agent understand information, generate possibilities, and decide what to do next.

Generative AI vs Agentic AI

A useful way to remember the difference is:

Generative AI asks: "What should I create?"

Agentic AI asks: "What should I do next to achieve this goal?"

Does Agentic AI Replace Generative AI?

No.

In many cases, generative AI becomes part of the agentic system.

An agent might use generative AI to:

• understand a request
• summarize information
• generate possible solutions
• write messages
• evaluate results

It can then combine those capabilities with tools that let it take action.

          AI Agent
             ↓
   ┌─────────┴─────────┐
   ↓                   ↓
Generative AI        Tools
   ↓                   ↓
Generate / Reason      Act
   └─────────┬─────────┘
             ↓
          Outcome

Generative AI provides part of the intelligence.

Agentic AI turns that intelligence into a sequence of actions.

The Future: AI as a Collaborator

The most powerful AI systems will probably not be purely generative or purely agentic.

They are likely to combine both.

Imagine an AI assistant helping launch a new product.

Generative AI might:

• brainstorm product names
• write marketing copy
• create images
• draft emails

Agentic capabilities might:

• research competitors
• update project tasks
• schedule meetings
• send approved emails
• monitor results

Sometimes the system needs to generate possibilities.

At other times, it needs to choose an action and carry it out.

Together, these capabilities make AI less like a tool that only responds when prompted and more like a collaborator that can create, plan, act, and adapt.

Generative vs Agentic AI: The Simple Difference

If you remember only one thing, remember this:

Generative AI creates. Agentic AI acts.

Generative AI responds to prompts by generating content such as text, images, code, or audio.

Agentic AI takes a goal and works through a series of actions to achieve it, evaluating results and deciding what to do next along the way.

And increasingly, the two will work together.

Generative AI creates and explores possibilities. Agentic AI turns those possibilities into action.

So what are the differences, and why would you use one over the other?

A good place to start is by understanding what each framework does.

What Is LangChain?

At its core, LangChain is a way of building LLM-powered applications by executing a sequence of functions in a chain.

Imagine we want to build an application that performs three actions:

Retrieve → Summarize → Answer

First, the application retrieves some information. Then it summarizes that information. Finally, it uses the summary to answer a user's question.

LangChain provides components that we can combine to build each part of this workflow.

Retrieve

For the retrieve step, we might use a LangChain component called a document loader.

A document loader fetches and loads content from a data source.

If some of those documents are large, we might also use a text splitter to break the content into smaller, more manageable chunks.

Summarize

Next, we need to summarize the information we retrieved.

For this, we can use a chain to orchestrate the summarization process.

The chain might contain a prompt that tells the model what we want it to do, along with an LLM that performs the summarization.

Answer

Finally, we need to answer the user's question.

We can use another chain that combines memory or conversation context, another prompt, and an LLM to generate the final response.

The LLM used for answering does not even have to be the same model used for summarization.

Our application might look like this:

RETRIEVE              SUMMARIZE                 ANSWER
   ↓                      ↓                        ↓
Document Loader          Chain                    Chain
   ↓                      ↓                        ↓
Text Splitter           Prompt               Memory / Context
                          ↓                        ↓
                         LLM                     Prompt
                          ↓                        ↓
                       Summary                    LLM
                                                   ↓
                                                 Answer

The important idea is that LangChain has a modular architecture.

Different actions can use different components, while chains help orchestrate how those components work together.

What Is LangGraph?

LangGraph is designed for building stateful applications and agents that have more complex workflows.

Instead of representing the application mainly as a sequence of actions, LangGraph lets us represent the workflow as a graph.

A graph has three important concepts:

Nodes, edges, and state.

Imagine we're building a task-management assistant.

The assistant needs to:

• process user input
• add tasks
• complete tasks
• summarize tasks

In LangGraph, each of these actions can become a node.

The connections between those nodes are called edges.

Edges determine where the application should go next.

For example, if a user says:

"Add buy groceries to my task list."

The Process Input node can understand the request and route it to the Add Task node.

If the user asks:

"What tasks do I still have?"

The request can instead be routed to the Summarize Tasks node.

Unlike a simple sequence, the workflow can move between different nodes depending on what is happening.

State

Another important part of LangGraph is state.

State contains information that needs to be available across the workflow.

For our task-management assistant, the state might contain:

Tasks:
- Buy groceries
- Finish report
- Call dentist

The Add Task node can add something to the state.

The Complete Task node can update it.

The Summarize Tasks node can read the same state and use an LLM to generate an overview of the current tasks.

Because the nodes can access and update the shared state, the application can maintain context as the user interacts with it.

This is what makes LangGraph useful for more flexible, stateful workflows.

LangChain vs LangGraph

Now that we understand the basic idea behind both frameworks, the difference becomes easier to see.

LangChain is useful when an application can be built by combining components into a sequence of actions. LangGraph uses a graph structure instead, which means the application can branch, loop, or return to previous parts of the workflow — useful when the next step depends on the current situation rather than being known in advance.

State Management

LangChain can pass information between different parts of an application and maintain conversation context.

LangGraph makes state a central part of the architecture.

Nodes can read and update the state as the application moves through the graph.

This makes LangGraph especially useful when an application needs to maintain context across many actions or interactions.

When Should You Use LangChain?

LangChain is a good choice when you want to connect LLMs with other components and build an application quickly.

For example, you might be building an application that:

• retrieves documents
• summarizes information
• answers questions
• calls tools
• interacts with APIs
• uses an LLM to perform several related tasks

LangChain gives you high-level components that you can combine instead of building everything yourself.

For beginners, it is usually the easier place to start.

When Should You Use LangGraph?

LangGraph becomes useful when your application needs a more flexible workflow.

You might need:

• multiple possible paths
• loops
• shared state
• long-running workflows
• human approval
• different actions depending on previous results

Imagine a research assistant that searches for information and then decides whether it has enough information to answer.

Research
   ↓
Evaluate
   ↓
Enough information?
 ↙              ↘
No              Yes
↓                ↓
Research Again  Answer

The application is not simply moving forward through a fixed sequence.

It can make a decision, return to an earlier step, and continue from there.

That is where LangGraph becomes particularly useful.

Can You Use LangChain and LangGraph Together?

Yes.

LangChain and LangGraph are part of the same ecosystem and can be used together.

In fact, modern LangChain agents are built on top of LangGraph under the hood. LangChain provides the higher-level agent abstraction, while LangGraph handles the underlying orchestration and state.

You can also use LangChain components such as models and tools directly inside a LangGraph workflow.

A simple way to think about the relationship is:

LangChain
Components for building LLM applications
        ↓
LangGraph
Control over complex workflows and state

You might start with LangChain and only use LangGraph directly when your application's workflow becomes more complex.

LangChain or LangGraph: Which Should You Choose?

If you're just getting started, LangChain is usually the easier place to begin.

It gives you reusable components for connecting models, prompts, data, tools, and other parts of an LLM application.

Use LangGraph when you need more control over how those components interact and how the application moves between different states.

The simplest way to remember the difference is:

LangChain helps you build LLM-powered applications by chaining components together.

LangGraph helps you build flexible, stateful workflows using nodes, edges, and state.

They solve related problems, and for more advanced applications, you may end up using both.

We reply immediately, begin coding before requirements are clear, and rush decisions to create the appearance of progress. But moving quickly without understanding the work often creates mistakes, rework, and delays.

The phrase "Slow is smooth, and smooth is fast" offers a better approach: begin deliberately, create a reliable process, and let speed emerge from consistency.

Where the phrase comes from

The exact origin of the phrase is unclear, although it is commonly associated with military marksmanship and tactical training.

In marksmanship, moving too quickly can reduce control and accuracy. A person first learns each movement slowly and correctly. Through repetition, those movements become smooth. Once they are smooth, they can be performed quickly without sacrificing precision.

The idea is similar to familiar sayings such as "measure twice, cut once" and "haste makes waste."

What it means

The phrase describes three stages:

Slow means deliberate. Take enough time to understand the task, identify risks, and perform it correctly.

Smooth means repeatable. Remove unnecessary steps, confusion, and interruptions until the process becomes consistent.

Fast means effective. Once the process is reliable, work moves quickly because less time is lost to mistakes and rework.

The goal is not to remain slow. The goal is to build enough control that speed becomes sustainable.

How it works in practice

Consider someone learning a piece of music.

Trying to play it at full speed immediately usually reinforces mistakes. A better approach is to practise slowly, isolate difficult sections, and repeat them until the movements become natural.

Only then does the musician increase the tempo.

The same pattern applies to most skilled work:

1. Understand the process.
2. Perform it carefully.
3. Remove unnecessary friction.
4. Repeat until it becomes consistent.
5. Increase speed without losing quality.

What this looked like for me

I experienced this principle firsthand while working as an engineering lead.

When I joined the team, there were few established rituals or shared processes. Meetings were arranged as needed, requirements were often unclear, and coding sometimes began before the team fully understood what needed to be built or why.

Questions and clarifications emerged during implementation. Assumptions changed, work was interrupted, and parts of the solution had to be redone.

Together, we began creating a more deliberate way of working. We introduced regular team rituals, clearer tickets, definitions of ready and done, quality gates, UI and UX reviews, and structured sprint planning.

We spent more time understanding the problem, building context, and preparing the work before development began.

At first, this felt slower.

We had to learn how to plan together, write useful tickets, share knowledge, and commit to a realistic amount of work. After several sprints, however, the process became smoother.

We could complete more than 80% of our planned sprint work while preparing upcoming features in parallel. The team understood the what and why early enough to take ownership of the how and when.

This resulted in more predictable delivery, fewer rework loops, and post-launch defects below 5%.

What was especially interesting was that the structure did not make the team rigid. It made us more resilient.

When priorities shifted or designs changed during a sprint, we had enough shared context to assess the impact, reorganise the work, and adapt without losing control of the entire sprint.

Because the foundations were clear, change became easier to absorb. We could distinguish between a necessary adjustment and a disruptive interruption, make informed trade-offs, and continue moving forward.

A team does not become fast by reacting more quickly to every request. It becomes fast by creating enough clarity, knowledge, and trust to deliver consistently and adapt without repeatedly correcting its own work.

Applying it to software development

Software teams often create false speed by starting quickly while ignoring clarity, quality, and reliability.

Features enter development rapidly but are later delayed by unclear requirements, rework, defects, and risky releases. Applying "slow is smooth, and smooth is fast" to software development means optimising for flow, feedback, and completed value rather than visible activity.

1. Understand before building

Building the wrong solution efficiently is still waste.

Before coding begins, the team should understand the user problem, intended outcome, constraints, and definition of success. A short discovery session, prototype, or design discussion may feel slower, but it can prevent weeks of unnecessary work.

The goal is not to define every detail in advance. It is to give the team enough context to make good decisions during implementation and take ownership of the how and when.

2. Work in small, reliable increments

Small changes are easier to understand, review, test, and release.

Clear acceptance criteria, focused pull requests, automated tests, and quality gates require an initial investment, but they reduce rework and make future changes safer.

A large amount of unfinished code may look productive, but it creates no value until it is integrated and available to users.

Slow down once to build confidence into the process. Move faster every time the process is repeated.

3. Improve the system, not just the pace

When delivery is slow, the answer is not always to ask people to work faster.

Teams should examine where work waits, where information becomes unclear, what repeatedly creates rework, and what can be simplified or automated.

Reliable pipelines, smaller releases, clear team rituals, and regular retrospectives make delivery more predictable. They also help teams adapt when priorities or designs change without losing control of the sprint.

The better question is not:

"Who needs to move faster?"

It is:

"What prevents the work from moving smoothly?"

What it does not mean

The principle is not an excuse for bureaucracy, perfectionism, or endless planning.

Slowing down is useful only when it creates clarity, reduces risk, builds knowledge, or improves the process. Once the work is understood and repeatable, it should move quickly.

Good process should not prevent change. It should make the team more capable of responding to change without falling into chaos.

Final thoughts

Real speed is not frantic activity.

It is the ability to move from intention to outcome with minimal confusion, waste, and rework.

In my experience, slowing down to build shared context, clear processes, and team ownership did not reduce our speed. It gave us the foundation to deliver consistently, maintain quality, and remain resilient when priorities changed.

In the beginning, we slowed down to build the system.

Once the system became smooth, the team became fast.

The challenge is deciding where to start.

Instead of asking:

Which AI framework is best?

Ask:

What kind of system am I trying to build?

Do you need a predictable workflow, one agent that can choose its next action, or several agents with specialized responsibilities?

Start with the simplest architecture that can solve the problem. Add autonomy only when it is useful.

What is an agentic AI framework?

A basic AI application sends a prompt to a model and returns the response.

An agentic system can do more. It may:

• Choose and use tools.
• Search documents or databases.
• Call APIs.
• Plan several steps.
• Review its results.
• Ask for human approval.
• Delegate work to another agent.

Agentic AI frameworks provide reusable components for managing these actions, including workflows, tools, memory, state, handoffs, guardrails, and tracing.

These frameworks overlap, and real applications often combine more than one. For example, a project might use one framework for orchestration and another for retrieval.

1. Start with a predictable workflow

A workflow follows a process controlled mainly by application code. The application defines the overall flow and possible transitions, making the system easier to understand, test, and control.

For example:

Receive support request
        ↓
Classify the issue
        ↓
Search the knowledge base
        ↓
Generate a response
        ↓
Create a ticket

AI may perform individual steps, but the application controls the overall sequence.

Use a workflow when:

• The steps are mostly known in advance.
• Business rules must be followed.
• Actions need validation or approval.
• Predictability matters more than autonomy.

LangChain is a practical option for applications where models, tools, and other components need to be connected across a defined sequence of steps.

LlamaIndex is especially useful when the workflow depends heavily on documents, private data, indexing, and retrieval.

For more complex workflows involving branching, persistent state, loops, durable execution, or human approval, LangGraph provides more explicit orchestration and control.

A useful rule

Use normal code or a workflow when you already know what should happen next.

Several steps do not automatically require an autonomous agent.

2. Use an autonomous agent when the path is unclear

A workflow works well when you already know the sequence of steps.

An autonomous agent is useful when the system needs to decide what should happen next.

Instead of defining every step in advance, you give the agent a goal, instructions, and access to tools. The agent can choose which tool to use, review the result, and decide whether another action is needed.

For example, a research agent might:

Understand the question
        ↓
Choose a search tool
        ↓
Review the results
        ↓
Search again if needed
        ↓
Prepare the answer

The exact path may change depending on what the agent discovers.

Use an autonomous agent when:

• The path to the answer cannot be fully predefined.
• The task requires planning or iteration.
• The system needs to choose between several tools.
• The agent needs to adapt based on intermediate results.

The developer still defines the agent's instructions, tools, permissions, and limits.

LangChain provides higher-level abstractions for building agents that can use models, tools, structured outputs, and external services.

OpenAI Agents SDK supports agents, tools, sessions, guardrails, tracing, and handoffs.

LlamaIndex is also relevant when an agent needs to search, retrieve, and reason over documents or organizational data.

For many projects, one agent with several tools is enough.

3. Add role-based agents only when specialization helps

A multi-agent system divides work among agents with different responsibilities.

Unlike a single autonomous agent that manages the entire task, each agent can focus on a particular role, set of tools, or area of expertise.

For example, a content system might include:

• A research agent that gathers relevant information.
• A writing agent that turns the research into a draft.
• A review agent that checks accuracy and quality.
• An editing agent that prepares the final version.

The agents work toward the same goal, but each operates within a more clearly defined responsibility.

There are two common coordination patterns.

Manager pattern

A central agent controls the process and calls specialists when needed.

Manager
 ├── Research specialist
 ├── Writing specialist
 └── Review specialist

Handoff pattern

One agent transfers responsibility to another specialist.

Triage agent
        ↓
Billing agent or technical agent

Use multiple agents when they need:

• Different tools or permissions.
• Different specialist instructions.
• Separate context.
• Independent review.
• Parallel execution.

Do not create another agent simply because the process contains another step.

Each additional agent adds model calls, latency, cost, communication, and more opportunities for failure.

CrewAI is a natural option for systems built around agents with defined roles and tasks.

OpenAI Agents SDK supports both manager-style orchestration and handoffs between specialist agents.

Microsoft Agent Framework is worth evaluating for Microsoft-focused projects that need multi-agent orchestration, workflows, state management, and enterprise integrations.

AutoGen remains relevant for existing projects and conversational multi-agent systems.

4. Prototype first, then prepare for production orchestration

Prototyping and production are development stages, not separate agent architectures.

A workflow, single agent, or multi-agent system can begin as a prototype.

Rapid prototyping

Langflow and Flowise provide visual interfaces for connecting models, prompts, tools, data sources, and agents.

They are useful when:

• You need to validate an idea quickly.
• Requirements are still changing.
• You want to compare models or prompts.
• You need an internal demonstration.

The purpose of a prototype is to learn what works before investing in a full production system.

Moving into production

A working demo is not automatically ready for real users.

Production systems may need:

• State and failure recovery.
• Authentication and permissions.
• Human approval and guardrails.
• Logging, tracing, and evaluation.
• Security, privacy, and cost controls.

Production readiness comes from engineering these controls around the system, not simply choosing a framework described as production-ready.

Quick agentic AI framework selection guide

These are starting points, not strict categories.

A simple framework decision process

These are starting points, not strict categories.

You do not need to begin with a fully autonomous or multi-agent system.

Start with the simplest architecture that can solve the problem, then add complexity only when you have a reason to. A workflow may be enough when the path is known. Add an agent when the system needs to decide what happens next. Add multiple agents only when specialization provides a clear benefit.

Choose a narrow first project, such as:

• Searching internal documents.
• Classifying support requests.
• Creating reports from structured data.
• Routing questions to different tools.
• Researching and summarizing a topic.

Build the simplest version first. Observe where it falls short, then add tools, state, autonomy, or additional agents only when those capabilities solve a real problem.

The framework should follow the system design, not the other way around.

Use normal code when the process is predictable. Use a controlled workflow when the sequence is mostly known but requires state, branching, retries, or approval. Use one agent when choosing the next action requires judgment. Add multiple agents only when genuine specialization, separate context, different permissions, or independent work makes them useful.

There is no prize for building the most agentic system.

The goal is to build the simplest system that can solve the problem reliably.

It reads your project rules, finds the right files, runs the correct tests, checks documentation and returns a proposed solution. It may even ask another agent to review the security implications.

From the outside, this can look like one intelligent system doing everything.

Under the hood, however, several different components are working together.

To understand how modern AI agents operate, there are five important terms you need to know:

1. AGENTS.md
2. Agent Skills
3. Model Context Protocol, or MCP
4. Agent2Agent Protocol, or A2A
5. Subagents

You do not need to be a developer to understand these concepts. Knowing them will help you use agents more effectively, evaluate AI products and diagnose why an agent succeeds—or fails—at a task.

The five terms in 30 seconds

Together, these five concepts explain much of what turns a language model into an agent capable of completing real work.

What makes an AI system an agent?

A large language model provides the language generation and reasoning at the centre of an AI system.

On its own, however, the model is mainly a conversational partner.

An agent adds other layers around the model, including:

• Instructions that shape its behaviour
• Access to tools and data
• Specialist workflows
• Rules and permissions
• Ways to delegate work

Think of the model as a highly capable new employee.

Intelligence helps, but the employee still needs to know how the organisation works, which systems they may use, what procedures they should follow and when to ask a specialist for help.

That is where these five terms come in.

1. AGENTS.md: A README written for AI agents

The simplest way to understand AGENTS.md is this:

AGENTS.md is like a README file written specifically for an AI coding agent.

A traditional README explains a software project to human developers. AGENTS.md gives compatible coding agents the instructions they need to work inside that project.

An AGENTS.md file might explain:

• How to install the project
• Which command runs the tests
• Which coding conventions to follow
• Which files should not be changed
• How pull requests should be formatted
• What checks must happen before committing code

For example, the file might contain this instruction:

Run pnpm test before committing any changes.

When the agent reaches the commit stage, it knows that running the test suite is part of the expected process.

You do not need to repeat the instruction in every conversation. It lives inside the project.

Why AGENTS.md matters

Without project-specific instructions, an agent has to guess how the codebase works.

It might run the wrong command, ignore a team convention, modify generated files or miss an important validation step.

AGENTS.md turns unwritten team knowledge into guidance an agent can follow consistently.

Can a project have multiple AGENTS.md files?

Yes.

A project can have a general AGENTS.md file at its root and more specific files inside individual folders.

For example:

• The root file could define company-wide coding standards.
• A frontend folder could contain accessibility requirements.
• A backend folder could specify database testing rules.

Instructions closer to the code being changed can provide more specific guidance.

What about CLAUDE.md?

Not every coding agent uses the filename AGENTS.md.

Claude Code uses CLAUDE.md files for persistent project instructions. These files can describe architecture, coding standards, workflows and other information Claude should consider when working in the project.

The filename is different, but the basic idea is similar:

• AGENTS.md is an open instruction format for compatible coding agents.
• CLAUDE.md provides persistent instructions specifically for Claude Code.

Both answer the same question:

What should this agent know whenever it works in this project?

AGENTS.md in one sentence

AGENTS.md tells an AI coding agent how to work inside a particular project.

2. Agent Skills: Specialist knowledge invoked when needed

AGENTS.md contains information the agent may need whenever it works in a project.

But some knowledge is useful only for a particular task.

Imagine an agent that knows how to:

• Build a PowerPoint presentation
• Review a legal agreement
• Analyse a spreadsheet
• Prepare a research report
• Convert a document into a particular format

Loading all those instructions every time the agent starts would waste valuable context.

The agent does not need presentation templates and slide-design rules while fixing a software bug.

That is the problem Agent Skills are designed to solve.

What is an Agent Skill?

An Agent Skill is a folder containing instructions and resources that teach an agent how to perform a specialised task.

At the centre of the folder is usually a file called SKILL.md.

The folder can also contain:

• Scripts
• Templates
• Reference documents
• Examples
• Images and other assets

A presentation skill, for example, might include:

• Instructions for structuring a slide deck
• A branded PowerPoint template
• Typography and formatting rules
• Examples of effective slides
• Scripts for creating charts
• A checklist for reviewing the final presentation

How does an agent know when to invoke a skill?

A skill includes a description explaining what it does and when it should be used.

A description might effectively say:

Use this skill when the user asks to create, revise or review a PowerPoint presentation.

The agent initially sees the skill's name and description. When the request is relevant, it can load the complete instructions and any supporting resources it needs.

When the request is unrelated, the skill stays out of the way.

This approach is often called progressive disclosure. The agent loads detailed knowledge only when the task calls for it, rather than carrying every available instruction in its context window.

Some platforms can select a relevant skill automatically, while others also let users invoke a skill directly.

Why Agent Skills matter

Skills turn repeated prompting into reusable capabilities.

Without a skill, you may have to explain the same process every time:

• Which template to use
• How to structure the output
• Which standards to follow
• Which files to consult
• How to check the final result

With a well-designed skill, those instructions can be packaged once and reused.

AGENTS.md versus Agent Skills

This is the key distinction:

AGENTS.md explains how to work in a project. An Agent Skill explains how to perform a task.

For example:

• AGENTS.md might explain how to test your company's application.
• A presentation skill might explain how to create your company's slide decks.

One provides persistent project context. The other is a specialist capability invoked when relevant.

Agent Skills in one sentence

An Agent Skill is a reusable package of instructions and resources that an agent loads for a matching task.

3. MCP: How agents connect to tools and data

An agent may understand your request and know how to complete the task, but it still needs access to the outside world.

Suppose you ask an agent to:

• Find a document in Notion
• Check your calendar
• Search a GitHub repository
• Query a company database
• Look up a customer in your CRM
• Create a payment link

The information and functionality required for those tasks do not live inside the language model.

The agent needs a way to connect to external systems.

That brings us to MCP, or the Model Context Protocol.

What is MCP?

MCP is an open standard for connecting AI applications to tools, data sources and workflows.

Without a shared standard, developers may need to build a separate integration for every combination of agent and external tool.

MCP provides a more consistent interface.

A tool or data source can be exposed through an MCP server. The AI application communicates with that server, and the server handles the underlying connection to the service, API or database.

For example:

1. You ask an agent to find a project plan in Notion.
2. The agent uses an available MCP connection.
3. The MCP server communicates with Notion.
4. The relevant information is returned.
5. The agent uses it to complete your request.

MCP is often compared to a universal adapter for AI applications.

MCP does not mean unlimited access

An MCP connection should not automatically give an agent permission to do everything.

A well-designed setup still needs:

• Authentication
• Permission controls
• Approval steps for sensitive actions
• Activity logging
• Limits on the data and tools the agent can access

The important question is not only whether an agent supports MCP.

You should also ask:

Which systems can it access, what actions can it take and what controls are in place?

MCP in one sentence

MCP gives AI agents a standard way to connect to external tools, data and workflows.

4. A2A: How agents communicate with other agents

MCP helps an agent communicate with tools and data.

But what happens when an agent needs to work with another agent?

That is the purpose of A2A, the Agent2Agent Protocol.

A2A is an open standard designed to help independent AI agents discover one another, communicate and delegate work.

An example of A2A

Imagine a company has several specialist agents:

• A procurement agent negotiates with suppliers.
• A finance agent approves spending.
• A legal agent reviews contracts.
• A research agent investigates vendors.

The procurement agent may negotiate a contract and then need approval from finance.

Without a common protocol, the two agents might require a custom integration—or a human may need to move the information manually.

With A2A, the procurement agent can discover the finance agent's capabilities and hand over the relevant work.

What is an Agent Card?

A central concept in A2A is the Agent Card.

An Agent Card is a structured description of an agent. It can include:

• The agent's identity
• Its capabilities
• The skills or services it offers
• How to communicate with it
• Its authentication requirements

Think of it as a digital business card.

Another agent can inspect the card, decide whether the agent is suitable for a task and determine how to contact it.

MCP versus A2A

The distinction is simple:

• MCP connects agents to tools and data.
• A2A connects independent agents to one another.

They are complementary.

An agent might use MCP to retrieve information from a company database and A2A to ask a finance agent to approve a proposed expense.

A2A in one sentence

A2A provides a shared way for independent agents to discover one another, communicate and delegate work.

5. Subagents: A specialist team inside the system

Sometimes one agent is not enough.

The task may be too large for one context window, require several kinds of expertise or contain many independent pieces that could be completed in parallel.

A subagent is a specialised child agent given a focused part of a larger assignment.

How subagents work

Imagine a main agent is asked to review a software project containing thousands of files.

Instead of analysing everything in one overloaded context, it might delegate separate tasks:

• One subagent reviews authentication.
• Another checks database queries.
• Another examines automated tests.
• Another looks for security issues.
• Another summarises the documentation.

Each subagent completes its assignment and returns the relevant findings to the main agent.

The main agent then combines those findings into a final result.

This is similar to a manager dividing a project among specialist team members.

Why separate context matters

A model's context window is the amount of information it can actively consider during a task.

Filling it with thousands of files, logs or search results can push important details aside.

A subagent can process that material in a separate context and return only the useful conclusion.

For example:

Review these 500 files and return a summary of the authentication architecture and its main risks.

The coordinating agent receives the summary rather than all 500 files.

Subagents may also work in parallel when tasks are independent, although the system still needs to manage costs, permissions and inconsistent results.

Are subagents an open standard?

Not in the same way as MCP or A2A.

"Subagent" describes a common architectural pattern. Different platforms implement child agents, specialist agents and manager-agent workflows differently.

A subagent may also have its own instructions, tools and permissions rather than receiving everything available to its parent.

Subagents in one sentence

Subagents allow a coordinating agent to divide complex work among focused child agents.

How all five terms work together

Imagine asking an AI agent to prepare a product-launch presentation.

Here is how the five components could fit together:

1. AGENTS.md or CLAUDE.md gives the agent your project rules, preferred terminology and company context.
2. A presentation skill teaches it how to structure and design the slide deck.
3. MCP allows it to retrieve product notes from Notion and performance data from your analytics system.
4. A2A lets it request budget information from a finance agent.
5. Subagents research competitors, customer feedback and market trends in parallel.

The final presentation may look like the output of one agent, but several layers of instructions, tools and delegated work helped produce it.

How these terms can help you use AI agents better

You do not need to build an agent to benefit from understanding its architecture.

When an agent produces a poor result, ask where the failure happened.

• Did it lack project instructions? It may need a clearer AGENTS.md, CLAUDE.md or equivalent instruction file.

• Did it lack specialist knowledge? The workflow may need an Agent Skill.

• Could it not access the required information? It may need an MCP connection or another approved integration.

• Was another specialist system required? An A2A-style handoff may be useful.

• Was the task too large or unfocused? It may need to be divided among subagents.

This is more useful than simply concluding that the model is not capable enough.

Sometimes the model is the problem. Often, the missing piece is the instructions, tools or architecture surrounding it.

Frequently asked questions

What is the difference between an AI agent and a chatbot?

A chatbot primarily responds to messages. An AI agent is designed to pursue a goal through multiple steps and may use tools, data, instructions and other agents to complete the task.

Is CLAUDE.md the same as AGENTS.md?

They serve a similar purpose, but they are not the same format. CLAUDE.md provides persistent instructions for Claude Code, while AGENTS.md is an open instruction format supported by compatible coding agents.

Are Agent Skills always active?

No. An agent can initially see only a skill's name and description, then load the complete instructions when the task is relevant. Some platforms also let users invoke a skill directly.

What is the difference between MCP and A2A?

MCP helps agents connect to external tools and data. A2A helps independent agents communicate and delegate work to one another.

Final thoughts

The vocabulary around AI agents can sound complicated, but the five central ideas are straightforward:

• AGENTS.md tells an agent how to work in a project.
• Agent Skills provide specialist knowledge when needed.
• MCP connects agents to tools and data.
• A2A enables communication between independent agents.
• Subagents divide complex work into focused assignments.

Understanding these terms helps you look beyond the language model itself.

You can begin asking better questions about the instructions, skills, integrations and delegation systems that determine what an AI agent can actually accomplish.

Terms such as agents, RAG, and MCP now appear regularly in product discussions, technical documentation, and company roadmaps. But knowing the definitions is only the beginning.

For product engineers, the more important question is:

How do these concepts affect the products we design and build?

This guide explains seven important AI terms through a practical product-engineering lens.

1. AI Agents

An AI agent is a system that works toward a goal by choosing actions, using tools, observing the results, and deciding what to do next.

A chatbot typically responds to one request. An agent may operate as a loop:

Goal → Plan → Act → Observe → Adjust

Consider an AI assistant investigating an increase in customer cancellations. It might:

• Query product analytics
• Search recent support tickets
• Compare customer segments
• Identify common complaints
• Produce a report with recommendations

The user defines the outcome, but the system determines some of the steps.

Why agents matter

Agents are useful when the path to an answer cannot be completely defined in advance.

However, autonomy also creates more opportunities for errors. If a normal workflow already solves the problem reliably, adding an agent may only increase cost, latency, and risk.

A useful rule is:

Use deterministic software when the steps are known. Use an agent when deciding the steps is part of the problem.

2. Reasoning Models

A reasoning model is designed for tasks involving planning, analysis, or multiple dependent steps.

These models are particularly useful for:

• Debugging complex software issues
• Comparing conflicting requirements
• Reviewing evidence
• Planning multi-step operations
• Solving technical or mathematical problems

They may spend more computation on a problem before returning an answer.

Why reasoning models matter

Not every task requires the most capable model.

A fast, inexpensive model may be enough for classification, extraction, or simple summarisation. A reasoning model may be better for difficult analysis.

Product teams should route requests based on complexity rather than sending every task to the largest model available.

The best model is not always the most powerful one. It is the model that provides the required quality at an acceptable cost and speed.

3. Embeddings and Vector Databases

An embedding is a numerical representation of content such as text, images, or audio.

Content with a similar meaning tends to receive a similar representation. This allows a system to find related information even when the wording is different.

For example:

• "I forgot my password."
• "I cannot access my account."

A traditional keyword search may treat these as different queries. An embedding-based search can recognise their semantic similarity.

A vector database stores and searches these numerical representations efficiently.

Why vector databases matter

Vector search supports features such as:

• Semantic search
• Recommendations
• Similar-document retrieval
• Duplicate detection
• Knowledge assistants

However, similarity does not guarantee correctness.

The most similar result could still be outdated, incomplete, or inappropriate for the user. Retrieval quality also depends on document structure, metadata, permissions, filtering, and evaluation.

The vector database is infrastructure. Relevance is a product-quality problem.

4. Retrieval-Augmented Generation

Retrieval-augmented generation, usually called RAG, gives a language model access to information outside its original training data.

A basic RAG system works like this:

Question → Retrieve relevant information → Add it to the model's context → Generate an answer

Imagine an employee asking:

"What is our parental-leave policy?"

A RAG system could retrieve the relevant section from the latest employee handbook and use it to create an answer with a citation.

This is particularly useful when information is:

• Private
• Specialised
• Frequently updated
• Specific to an organisation

Why RAG matters

RAG does not automatically make an AI system accurate.

A production system still needs reliable document ingestion, search, permissions, citations, freshness controls, and testing.

When a RAG answer is wrong, ask two separate questions:

1. Did the system retrieve the correct information?
2. Did the model interpret that information correctly?

This distinction helps teams fix the real problem instead of repeatedly adjusting prompts.

5. Model Context Protocol

The Model Context Protocol, or MCP, provides a standard way for AI applications to connect to external data and tools.

Through these connections, an AI application might:

• Search internal documentation
• Read a file
• Query a database
• Create a calendar event
• Update an issue tracker
• Perform a controlled business action

Without a shared standard, developers may need to build a separate integration for every combination of AI application and external service.

Why MCP matters

MCP can make integrations more reusable, but it does not remove the need for security.

Every tool still requires:

• Authentication
• Authorisation
• Input validation
• Clear permissions
• Audit logs
• Error handling
• Human approval for sensitive actions

A model that can recommend an action is useful. A model that can perform irreversible actions without clear boundaries is dangerous.

AI tools should be designed like production APIs: narrow, predictable, observable, and permission-aware.

6. Mixture of Experts

A mixture-of-experts, or MoE, model contains multiple neural sub-networks known as experts.

A routing mechanism activates only some of these experts for each input. This allows a model to have a large overall capacity without using every parameter for every request.

The term expert can be misleading. These components do not necessarily correspond to understandable roles such as "coding expert" or "finance expert." Their specialisation is learned during training.

Why mixture of experts matters

Most product engineers will not design MoE architectures themselves.

Instead, they will experience the effects through:

• Model quality
• Inference cost
• Latency
• Throughput
• Deployment requirements

This is one reason parameter count alone is not enough to judge a model.

What matters is how well the model performs on your actual use case.

7. AGI and ASI

Artificial general intelligence, or AGI, generally refers to a hypothetical AI system with broad capabilities across many cognitive tasks.

Artificial superintelligence, or ASI, describes a theoretical system whose abilities would substantially exceed human capabilities across many domains.

Neither term has a universally accepted definition or test.

They are useful concepts for discussing the future of AI, but they are not practical product requirements.

Why AGI and ASI matter

Product teams should build around demonstrated capabilities, not predictions about future intelligence.

Instead of asking:

"Is this model approaching AGI?"

Ask:

• Which tasks can it complete reliably?
• Under what conditions does it fail?
• How much autonomy should it have?
• Which actions require approval?
• How will we evaluate its behaviour?

These questions lead to better engineering decisions today.

How these concepts fit together

These seven terms describe different layers of an AI product.

Reasoning models and MoE relate to the model and how it processes tasks.

Embeddings, vector databases, and RAG help the system find and use relevant knowledge.

MCP connects the application to external tools and services.

Agents coordinate actions toward a goal.

AGI and ASI describe possible future levels of general capability.

A simplified system might look like this:

User request → Select an appropriate model → Retrieve relevant knowledge → Access external tools → Perform an action → Validate the result → Ask for human approval when necessary

The model is only one part of the product.

Permissions, evaluations, observability, user experience, fallback behaviour, and cost controls are what turn an impressive prototype into a dependable system.

What I would prioritise as a product engineer

For teams building AI products today, I would prioritise three concepts:

RAG

Because useful AI products often need access to private, specialised, or current information.

MCP and tool design

Because AI becomes more valuable when it can interact safely with real systems.

Agents

Because they introduce both significant opportunities and significant risks through greater autonomy.

Reasoning models support all three, while MoE, AGI, and ASI are important concepts to understand but are less likely to determine an application-level architecture decision.

Final thoughts

The strongest AI products will not necessarily use the largest model or the most autonomous agent.

They will use the right level of intelligence for the problem, provide relevant context, offer carefully designed tools, and place clear boundaries around what the system can do.

As a product and AI engineer, this is the area I find most interesting: translating emerging AI capabilities into products that are useful, measurable, secure, and reliable.

Because the real engineering challenge is not making AI look impressive in a demo.

It is making AI work responsibly in production.

Which model is the most capable? Should it be large or small? Open or proprietary?

These questions matter, but a model alone is not an AI application.

A useful AI system also needs somewhere to run, access to relevant information, a way to coordinate its work, and an interface through which people can use it.

Together, these parts form the AI stack.

A simple version has five layers:

Users experience the stack from the top down. To understand how it is built, it helps to start at the bottom.

Throughout this article, we will use a customer support assistant as an example. A customer asks:

"Where is my order, and can I still change the delivery address?"

1. Infrastructure

Infrastructure is the technical foundation on which the AI system runs.

It includes processors, memory, storage, networking, servers, and the software needed to operate the model and supporting services.

AI models often run on GPUs because GPUs can perform many calculations in parallel. Smaller models may run on a laptop or phone, while larger models usually require cloud or data-centre infrastructure.

There are three common deployment options:

• Local: on a laptop, phone, or workstation
• Cloud: on rented infrastructure that can scale with demand
• On premises: on hardware owned and operated by the organisation

Infrastructure affects which models can run, how quickly they respond, how much the system costs, and where the data is processed.

For our support assistant, this layer runs the model, databases, search systems, and application.

2. Models

The model is the part of the system that understands, generates, or analyses information.

A language model can interpret a customer's question and write a clear response. Other models may specialise in images, audio, classification, search, embeddings, coding, or tool use.

Models differ in several ways.

Some are open and can be hosted by the organisation using them. Others are proprietary and accessed through an external provider.

They also vary in size. Larger models often have broader capabilities, but they need more computing power. Smaller models can be faster, cheaper, and better suited to focused tasks.

Platforms such as Hugging Face provide catalogues of open and specialised models for many different use cases.

The best model is not always the largest one. The right choice depends on the task and the required balance between quality, speed, cost, privacy, and hardware needs.

In our example, the model understands that the customer is asking two questions:

1. Where is the order?
2. Can the delivery address still be changed?

But the model does not know the actual order status or company policy. That information comes from the data layer.

3. Data

A model's built-in knowledge is not always enough.

It may be outdated, based mainly on public information, or missing the private data needed for a particular task.

The data layer adds that missing context. It has four main parts.

Data sources

These are the places where information comes from.

For a support assistant, they might include:

• Customer accounts
• Order and delivery systems
• Product documentation
• Company policies
• Support tickets
• Internal databases and APIs

Data pipelines

Raw data is rarely ready for use immediately.

Pipelines collect, clean, transform, and update it. They may split documents into smaller sections, remove duplicates, extract metadata, apply permissions, or convert different formats into a consistent structure.

Embeddings and indexes

The system needs a way to search information quickly.

One method is to create embeddings, which are numerical representations of meaning. An embedding model creates them, and a vector database or search index stores them.

This allows the system to find related information even when the wording is different.

A customer may ask:

"Can I change where my parcel is going?"

while the company policy uses the phrase:

"Delivery address amendment."

Semantic search can recognise that these mean roughly the same thing.

Retrieval and RAG

Retrieval is the process of finding the information needed for a specific request.

Retrieval-augmented generation, or RAG, gives that information to the model before it produces an answer.

RAG does not permanently teach the model new facts. It supplies the right context for the current request.

The purpose of the data layer is to make the right information available, in the right form, at the right moment.

4. Orchestration

Simple AI tasks may require only one prompt and one response. More complex tasks need several steps.

The orchestration layer coordinates those steps. It connects the model with data, tools, and business rules.

It can be understood as three stages.

Planning

The system decides how to approach the request.

In our example, it recognises that it must check the order status and determine whether an address change is still possible.

Execution

The system performs the required actions.

It may retrieve the customer's order, call the delivery system, search the company policy, and ask the model to prepare a response.

These actions are often called tool calling or function calling.

Review

The system checks whether the result is complete and supported by the available information.

It may ask whether both questions were answered, whether the response matches the policy, or whether the case should be handed to a human agent.

If something is missing, it can repeat an earlier step.

Some systems follow a fixed workflow. Others allow the model to make limited decisions about what to do next. This is where terms such as agents and MCP fit into the stack.

In simple terms:

Orchestration turns one user request into a coordinated process of planning, execution, and review.

5. Application

The application is the part of the system that people interact with.

It turns the models, data, and orchestration underneath into something useful for a real task.

The application layer has two main parts: interfaces and integrations.

Interfaces

The interface defines how users provide input and receive the result.

The most familiar example is text in and text out, but an AI application may also work with images, audio, documents, tables, forms, or numerical data.

For our support assistant, the interface may be a chat window that shows the order status, explains whether the address can be changed, and offers the next step.

A good interface should also let users inspect and improve the result through:

• Follow-up questions
• Citations or links
• Editing and revision
• Confirmation before an action
• Handover to a human

Integrations

Integrations connect the AI application with other tools and systems.

Information can flow into the AI system from customer records, delivery systems, and previous support conversations.

The AI application can also send results back, for example by updating a support ticket, opening an address-change request, or transferring the conversation to a human agent.

The interface helps people communicate with the AI system. Integrations connect it to the rest of their work.

How the layers work together

For our customer support example:

Each layer has a different role:

Infrastructure runs the system. Models provide AI capabilities. Data supplies relevant knowledge. Orchestration coordinates the work. Applications make the system useful.

Why the AI stack matters

Choices at every layer affect the final result.

A larger model may improve quality but increase cost. Better data may be more valuable than a more powerful model. Good orchestration can make a complex task more reliable. A clear application helps users understand and act on the result.

These choices influence four important outcomes:

Quality, speed, cost, and safety.

That is why building an AI product involves much more than selecting a model and writing a prompt.

Final summary

The AI stack can be remembered as five layers:

Infrastructure → Models → Data → Orchestration → Application

The infrastructure runs the model.

The model provides the capability.

The data adds relevant knowledge.

The orchestration coordinates the task.

The application connects the system to the user.

This is not a complete engineering blueprint, but it is a useful map for understanding how the main parts of an AI system fit together.

Calling sendEmail() after returning an HTTP response may look asynchronous, but it is not a durable background-job system. If the process crashes, the deployment restarts, or the machine disappears, that work may vanish. A real task queue records the work somewhere durable enough for another process to pick it up, retry it, and report what happened.

This guide explains the architecture and failure modes that matter in production, then builds a practical Node.js and TypeScript example with BullMQ and Redis.

Background jobs, task queues, and asynchronous code

A background job is work performed outside the request-response path. A task queue is the infrastructure that coordinates that work between producers and workers.

Those ideas are related, but they are not identical to ordinary asynchronous JavaScript:

• await, promises, and non-blocking I/O let one Node.js process use its time efficiently.
• setImmediate() or an unawaited promise schedules work only inside the current process.
• A task queue stores a description of the work so a worker can execute it independently, potentially on another machine and after the producer has exited.

A typical system has three parts:

1. Producer: the web application, API, or service that creates a job.
2. Queue: the broker or data store that holds pending jobs and coordinates delivery.
3. Worker: a separate process that reserves a job, performs the work, and reports success or failure.

The queue creates a boundary between accepting work and executing it. That boundary improves responsiveness and allows the web tier and worker tier to scale independently, but it also introduces distributed-systems problems: duplicate delivery, partial failure, retries, ordering, stale data, and operational backlogs.

What belongs in a background job?

Good candidates are operations that are slow, failure-prone, resource-intensive, or unnecessary for the immediate response:

• Sending email, push notifications, and webhooks
• Resizing images or transcoding video
• Importing large files and generating reports
• Synchronizing data with external services
• Rebuilding search indexes or caches
• Running scheduled cleanup and maintenance
• Processing analytics events

Keep work synchronous when the user needs its result before the request can be considered successful. Validation, authorization, and the database write that creates the requested resource usually belong in the request path. Payment authorization often does too when checkout cannot finish without knowing whether the payment was accepted. Follow-up work such as receipts, fulfillment notifications, and reconciliation can be queued.

A useful rule is:

Put the minimum required state change in the request path. Queue the independent consequences.

The life cycle of a job

Although queue products use different terminology, most jobs follow the same life cycle:

1. The producer serializes a small payload and enqueues it.
2. A worker reserves or receives the job.
3. The queue hides or locks the job while it is being processed.
4. The worker performs the operation.
5. On success, the worker acknowledges completion.
6. On failure, the job is retried, delayed, or retained for investigation.

The critical detail is that most practical systems provide at-least-once execution, not exactly-once execution. A worker can finish the external side effect and crash before acknowledging the job. The queue then makes the job available again, and another worker may execute it a second time.

That is why idempotency is not an optional optimization. It is part of the correctness model.

Do not assume strict FIFO ordering

A queue may store jobs in insertion order, but that does not mean jobs will finish in that order. Multiple workers, different execution times, priorities, retries, delayed jobs, and redelivery can all change the order observed by the application.

When ordering is a business requirement, define its scope precisely. You may need one active consumer, partitioning by an entity such as accountId, or sequence checks in the database. Global ordering usually reduces throughput and availability, so avoid requiring it unless the domain truly needs it.

Durability is configured, not automatic

A queue does not magically make work impossible to lose. Reliability depends on broker configuration, persistence, replication, producer acknowledgements, retention, and the way the application crosses the database-to-queue boundary. A durable queue can still lose a logical task if the database commit succeeds but enqueueing fails afterward. We will address that with the transactional outbox pattern.

A practical Node.js and TypeScript implementation

The following example uses BullMQ with Redis. Install bullmq and ioredis, run Redis, and provide a REDIS_URL environment variable.

The example queues a welcome email after registration. The job contains identifiers rather than a complete user object, which keeps the payload small and prevents stale or sensitive data from being copied into Redis.

Define the queue and job contract

// jobs/email-queue.ts
import { Queue } from 'bullmq'
import IORedis from 'ioredis'

export type EmailJobName = 'send-welcome-email'

export interface EmailJobData {
  schemaVersion: 1
  userId: string
  correlationId: string
}

const redisUrl = process.env.REDIS_URL

if (!redisUrl) {
  throw new Error('REDIS_URL is required')
}

// Producers should fail reasonably quickly when Redis is unavailable.
const producerConnection = new IORedis(redisUrl, {
  maxRetriesPerRequest: 1,
  enableOfflineQueue: false,
})

export const emailQueue = new Queue<EmailJobData, void, EmailJobName>(
  'email',
  {
    connection: producerConnection,
    defaultJobOptions: {
      attempts: 5,
      backoff: {
        type: 'exponential',
        delay: 1_000,
      },
      removeOnComplete: {
        age: 60 * 60,
        count: 1_000,
      },
      removeOnFail: {
        age: 7 * 24 * 60 * 60,
        count: 5_000,
      },
      stackTraceLimit: 20,
    },
  },
)

The producer and worker use different Redis connection behavior. An HTTP request should not wait forever when Redis is unavailable, while a long-running worker should normally keep reconnecting.

Enqueue a job

// jobs/enqueue-welcome-email.ts
import { randomUUID } from 'node:crypto'
import { emailQueue, type EmailJobData } from './email-queue.js'

export async function enqueueWelcomeEmail(
  userId: string,
  correlationId = randomUUID(),
): Promise<string> {
  const data: EmailJobData = {
    schemaVersion: 1,
    userId,
    correlationId,
  }

  const job = await emailQueue.add('send-welcome-email', data, {
    // BullMQ job IDs cannot contain a colon.
    // This suppresses duplicate enqueues while this job ID is retained.
    jobId: `welcome-email-${userId}`,
  })

  if (!job.id) {
    throw new Error('Queue returned a job without an ID')
  }

  return job.id
}

A custom job ID is useful for deduplication, but it is not a complete idempotency strategy. Once an old job is removed, the same ID can be added again. More importantly, a job that has already started can still be redelivered after a crash. The side effect itself must remain safe to repeat.

Process the job in a worker

// workers/email-worker.ts
import { UnrecoverableError, Worker, type Job } from 'bullmq'
import IORedis from 'ioredis'
import { type EmailJobData, type EmailJobName } from '../jobs/email-queue.js'
import { emailProvider } from '../services/email-provider.js'
import { userRepository } from '../repositories/user-repository.js'

const redisUrl = process.env.REDIS_URL

if (!redisUrl) {
  throw new Error('REDIS_URL is required')
}

const workerConnection = new IORedis(redisUrl, {
  maxRetriesPerRequest: null,
})

async function processEmailJob(
  job: Job<EmailJobData, void, EmailJobName>,
): Promise<void> {
  if (job.data.schemaVersion !== 1) {
    throw new UnrecoverableError(
      `Unsupported job schema: ${job.data.schemaVersion}`,
    )
  }

  const user = await userRepository.findById(job.data.userId)

  if (!user) {
    throw new UnrecoverableError(`User ${job.data.userId} no longer exists`)
  }

  await job.log(
    JSON.stringify({
      event: 'sending_welcome_email',
      userId: user.id,
      correlationId: job.data.correlationId,
      attempt: job.attemptsMade + 1,
    }),
  )

  try {
    await emailProvider.sendWelcomeEmail({
      to: user.email,
      // The provider should treat repeated requests with this key as one send.
      idempotencyKey: `welcome-email-${user.id}`,
      signal: AbortSignal.timeout(10_000),
    })
  } catch (error) {
    if (emailProvider.isPermanentError(error)) {
      throw new UnrecoverableError(
        error instanceof Error ? error.message : 'Permanent email error',
      )
    }

    // Throwing a normal Error lets BullMQ apply the retry policy.
    throw error instanceof Error ? error : new Error(String(error))
  }
}

export const emailWorker = new Worker<EmailJobData, void, EmailJobName>(
  'email',
  processEmailJob,
  {
    connection: workerConnection,
    concurrency: 20,
    limiter: {
      max: 50,
      duration: 1_000,
    },
  },
)

emailWorker.on('completed', (job) => {
  console.info('email_job_completed', { jobId: job.id })
})

emailWorker.on('failed', (job, error) => {
  console.error('email_job_failed', {
    jobId: job?.id,
    attemptsMade: job?.attemptsMade,
    error: error.message,
  })
})

emailWorker.on('error', (error) => {
  console.error('email_worker_error', { error })
})

async function shutdown(signal: NodeJS.Signals): Promise<void> {
  console.info('email_worker_shutting_down', { signal })
  await emailWorker.close()
  await workerConnection.quit()
}

for (const signal of ['SIGTERM', 'SIGINT'] as const) {
  process.once(signal, () => {
    void shutdown(signal)
      .then(() => process.exit(0))
      .catch((error: unknown) => {
        console.error('email_worker_shutdown_failed', { error })
        process.exit(1)
      })
  })
}

The worker has a timeout, explicit permanent-error handling, bounded retries, rate limiting, structured context, and graceful shutdown. Those details are not decoration; they determine how the system behaves during deployments and dependency failures.

Retry only failures that may succeed later

Retries are useful only when the next attempt has a meaningful chance of succeeding.

Use exponential backoff so a failing dependency is not hammered continuously. Add random jitter when your queue library or custom retry strategy supports it; otherwise thousands of jobs can wake at the same instant and create a thundering herd.

Retries must be bounded. After the final attempt, retain the failed job or route it to a dead-letter mechanism so an operator can inspect, correct, and redrive it. Infinite retries turn permanent defects into permanent load.

Idempotency, deduplication, and exactly-once myths

These terms are often mixed together:

• Deduplication prevents some duplicate jobs from entering or waiting in the queue.
• Idempotency makes repeated execution produce the same intended final state.
• Exactly-once delivery is not something an application should assume across a queue, database, and external API.

The unsafe pattern is "check, perform the side effect, then record success." Two workers can pass the check concurrently, or the process can crash after the side effect and before the success record is written.

For external operations such as charging a card or sending an email, the strongest practical solution is usually a stable idempotency key understood by the external provider:

// services/capture-payment.ts
export async function captureOrderPayment(orderId: string): Promise<void> {
  const order = await orderRepository.getForPayment(orderId)

  await paymentProvider.capture({
    paymentMethodId: order.paymentMethodId,
    amountInCents: order.amountInCents,
    idempotencyKey: `capture-order-${order.id}`,
  })

  await orderRepository.markPaid(order.id)
}

If the worker crashes after the provider accepts the charge, the retry uses the same key and retrieves or reuses the same provider-side operation instead of charging again.

When the dependency does not support idempotency keys, prefer naturally idempotent operations such as an upsert to a known value. Otherwise, design explicit reconciliation and accept that duplicate side effects may be possible. A database uniqueness constraint can protect database state, but it cannot atomically cover an unrelated external API call.

Also, background jobs commonly exist specifically to perform side effects. "Avoid side effects" is therefore the wrong rule. The better rule is: make side effects explicit, bounded, observable, and safe to repeat.

Close the database-to-queue gap with a transactional outbox

Consider this sequence:

1. Insert a user into PostgreSQL.
2. Enqueue the welcome-email job in Redis.

If the database commit succeeds and Redis is unavailable, the account exists but the job is missing. Reversing the order creates the opposite problem: the worker may run before the database transaction commits or after it rolls back.

A transactional outbox solves this dual-write problem by writing the business record and an outbox event in the same database transaction. A separate dispatcher publishes committed outbox events to the queue.

// services/register-user.ts
import { randomUUID } from 'node:crypto'
import { pool } from '../database.js'
import type { EmailJobData } from '../jobs/email-queue.js'

export async function registerUser(email: string): Promise<string> {
  const client = await pool.connect()

  try {
    await client.query('BEGIN')

    const userId = randomUUID()
    await client.query('INSERT INTO users (id, email) VALUES ($1, $2)', [
      userId,
      email,
    ])

    const eventId = randomUUID()
    const payload: EmailJobData = {
      schemaVersion: 1,
      userId,
      correlationId: eventId,
    }

    await client.query(
      `INSERT INTO outbox_events (id, topic, payload)
       VALUES ($1, $2, $3::jsonb)`,
      [eventId, 'send-welcome-email', JSON.stringify(payload)],
    )

    await client.query('COMMIT')
    return userId
  } catch (error) {
    await client.query('ROLLBACK')
    throw error
  } finally {
    client.release()
  }
}

A compact dispatcher can claim an unpublished row, enqueue it with the outbox ID as the queue job ID, and mark it published:

// workers/outbox-dispatcher.ts
import type { PoolClient } from 'pg'
import { pool } from '../database.js'
import {
  emailQueue,
  type EmailJobData,
  type EmailJobName,
} from '../jobs/email-queue.js'

interface OutboxEvent {
  id: string
  topic: EmailJobName
  payload: EmailJobData
}

export async function publishOneOutboxEvent(): Promise<boolean> {
  const client: PoolClient = await pool.connect()

  try {
    await client.query('BEGIN')

    const result = await client.query<OutboxEvent>(
      `SELECT id, topic, payload
       FROM outbox_events
       WHERE published_at IS NULL
       ORDER BY created_at
       FOR UPDATE SKIP LOCKED
       LIMIT 1`,
    )

    const event = result.rows[0]
    if (!event) {
      await client.query('COMMIT')
      return false
    }

    await emailQueue.add(event.topic, event.payload, {
      jobId: event.id,
    })

    await client.query(
      `UPDATE outbox_events
       SET published_at = NOW()
       WHERE id = $1`,
      [event.id],
    )

    await client.query('COMMIT')
    return true
  } catch (error) {
    await client.query('ROLLBACK')
    throw error
  } finally {
    client.release()
  }
}

A production dispatcher normally claims batches with a lease instead of holding a database transaction open during a network call, but the correctness principle is the same. A crash can still cause the event to be published more than once, so the worker remains idempotent. The outbox gives reliable eventual publication, not magical exactly-once execution.

Design job payloads as durable contracts

A queued payload may live longer than the deployment that created it. Treat it like a versioned API contract.

Keep payloads small and include:

• A schema version
• Stable resource identifiers
• A correlation or trace ID
• Only the data required to locate or perform the work

Avoid embedding large objects, secrets, access tokens, or unnecessary personal data. Do not enqueue a path such as /tmp/upload-123 when another worker or machine cannot access that filesystem. Store the file in shared object storage and pass its object key.

Decide how new workers handle old payload versions before deploying a breaking change. Common strategies are backward-compatible readers, explicit migration, or temporary workers for both versions.

Scheduled jobs, workflows, and batching

Scheduled and recurring jobs

A scheduler should create jobs; workers should execute them. This preserves the same retry, monitoring, and isolation model as one-off work.

With current BullMQ job schedulers, a recurring job can be upserted by a stable scheduler ID:

// jobs/schedule-cleanup.ts
import { Queue } from 'bullmq'
import IORedis from 'ioredis'

const connection = new IORedis(process.env.REDIS_URL!, {
  maxRetriesPerRequest: 1,
})

const maintenanceQueue = new Queue('maintenance', { connection })

await maintenanceQueue.upsertJobScheduler(
  'daily-expired-session-cleanup',
  { pattern: '0 0 3 * * *' },
  {
    name: 'delete-expired-sessions',
    data: { schemaVersion: 1 },
    opts: {
      attempts: 3,
      backoff: { type: 'exponential', delay: 5_000 },
    },
  },
)

For recurring work, define the timezone, daylight-saving behavior, missed-run policy, and overlap policy. If one run takes longer than its interval, decide whether the next run should wait, be skipped, or execute concurrently. The job itself should still be idempotent because scheduler failover or operator action can create duplicates.

Workflows and chains

Split a workflow when each step needs different retries, resource limits, or visibility. For example:

upload received → virus scan → image variants → CDN publish → database update

Do not replace a durable workflow with a chain of in-process promises when the process may restart. Use queue-native parent-child flows or a workflow engine when the sequence has long waits, compensation, human approval, or complex state transitions.

Batching

Batching can reduce API calls and database round trips, but an in-memory buffer inside one worker is not durable and does not coordinate across replicas. Prefer broker-supported bulk enqueue, database-backed batches, or a dedicated batch-aggregation job. Define how partial failures are retried so successful items are not repeated unnecessarily.

Scaling workers correctly in Node.js

For I/O-heavy jobs, increasing BullMQ worker concurrency lets one Node.js process make progress on other jobs while it waits on databases and network calls. Measure dependency capacity before increasing it: concurrency that overwhelms a database or vendor API reduces throughput rather than improving it.

CPU-heavy jobs are different. Image codecs, document parsing, compression, and large calculations can block the Node.js event loop. That delays queue lock renewal and may cause the job to be classified as stalled and processed again. Use separate worker processes, BullMQ sandboxed processors, or a worker_threads pool for CPU-bound JavaScript. Scale those workers according to available CPU cores rather than setting a large asynchronous concurrency value.

Separate queues are often useful for different workload classes:

• Fast, latency-sensitive jobs
• Slow or CPU-heavy jobs
• Jobs calling a rate-limited dependency
• Critical jobs with stricter operational handling

This prevents a flood of slow work from starving password-reset emails or other urgent tasks. Prefer separate queues over many priority levels when the workloads need different scaling, retry, or alerting policies.

Backpressure and rate limiting

A queue absorbs bursts; it does not create infinite downstream capacity. If jobs arrive faster than workers can safely process them, queue age grows until users experience stale results or retention limits are reached.

Apply backpressure at several layers:

• Limit worker concurrency to the capacity of dependencies.
• Rate-limit calls to external services.
• Reject, defer, or coalesce low-value work when the backlog is unhealthy.
• Deduplicate rapid updates when only the latest state matters.
• Scale using queue age and throughput, not CPU alone.

Queue depth is useful, but the age of the oldest ready job is often a better user-impact signal. Ten thousand one-millisecond jobs may be healthy; fifty jobs waiting for an hour are not.

Observability and operations

A production queue needs the same operational discipline as a database or API.

Track at least:

• Jobs enqueued, started, completed, failed, and retried
• Queue depth and age of the oldest job
• End-to-end latency from enqueue to completion
• Processing duration percentiles by job type
• Exhausted jobs and dead-letter or failed-set size
• Stalled jobs and lock-renewal problems
• Worker concurrency, utilization, memory, and event-loop lag

Every log entry should include the queue, job name, job ID, attempt number, and correlation ID. Do not log complete payloads when they contain personal data or secrets.

Alert on symptoms that require action: sustained queue age, a rising retry ratio, jobs exhausting attempts, repeated stalls, no active workers, or a producer enqueue failure rate above normal. A dashboard without alerts simply documents an outage after the fact.

Operational tooling should support pausing a queue, inspecting a job, retrying or redriving failed work, and safely deleting obsolete jobs. Keep enough completed and failed history for debugging, but configure retention so Redis or broker storage cannot grow without limit.

Choosing a queue technology

There is no universally best queue. Choose based on delivery model, operational constraints, ecosystem, and the surrounding architecture.

Use a task queue when one piece of work should be handled by one logical worker. Use an event stream or pub/sub system when an event should be observed independently by multiple consumers. Some systems use both: an event is published, and each subscriber creates its own task for controlled execution.

Production checklist

Before launching a background-job system, verify that:

• The request path contains only work required for the immediate response.
• The producer handles enqueue failures explicitly.
• Database changes and required jobs use an outbox or another reliable publication mechanism.
• Job payloads are small, versioned, and free of unnecessary sensitive data.
• Workers are safe under duplicate execution.
• External side effects use stable idempotency keys when available.
• Retries are limited, classified, and delayed with backoff.
• Permanent failures are retained and alert an operator.
• Timeouts and cancellation are implemented for external calls.
• Worker concurrency matches the workload and dependency capacity.
• CPU-heavy work runs outside the main Node.js event loop.
• Queue age, retries, failures, and stalls are monitored.
• Deployments shut workers down gracefully.
• Operators can pause, inspect, retry, and redrive jobs.
• Retention and Redis or broker durability settings are tested, not assumed.

Conclusion

Background jobs improve responsiveness and scalability by moving independent work out of the request path. The queue itself, however, is only the beginning.

Reliable systems assume that jobs can be delivered more than once, that ordering can change, and that every boundary can fail. They use small versioned payloads, bounded retries, idempotent side effects, transactional outboxes, separate worker processes, graceful shutdown, and metrics based on user-visible delay.

The most important design principle is simple:

A job is not reliable because it was placed on a queue. It is reliable because the entire path—from database commit to enqueue, execution, side effect, acknowledgement, retry, and recovery—was designed for failure.

One of the biggest sources of confusion is that many AI tools appear to overlap, yet they operate at completely different levels of the technology stack. A developer building an AI application might use Claude as the language model, OpenRouter to access it, LangGraph to orchestrate agents, n8n to automate business workflows, Cursor to write the code, and Pinecone to store long-term knowledge—all within the same project.

Rather than thinking of AI as a collection of competing products, it's more useful to think of it as an ecosystem of complementary platforms.

This guide breaks down the major layers of today's AI ecosystem, explains what each category does, highlights the leading platforms, and clarifies when you would choose one over another.

The AI Technology Stack

A simplified view of how the layers stack on top of each other:

                       Applications
        (ChatGPT, Harvey, Sierra, Copilot, Notion AI)
                              |
             Agents + Workflow Automation
       (LangGraph, Agents SDK, n8n, Retool, Zapier)
                              |
            Tools, Protocols & Integrations
         (MCP: tools/data · A2A: agent-to-agent)
                              |
                Models + AI Platforms
   (Claude, GPT, Gemini · Bedrock, Vertex AI, OpenRouter)
                              |
              Infrastructure & Compute
     (NVIDIA, TPU, AWS Trainium, CoreWeave, Lambda)

   ────────────────────────────────────────────
   Supporting layers (not optional add-ons):
   Knowledge & Data · Security & Governance ·
   Evaluation & Observability
   ────────────────────────────────────────────

In practice, these layers are not a strict hierarchy — the stack is not perfectly vertical, and some products span multiple layers at once. n8n and Copilot Studio can build end-user applications, not just internal workflows. Microsoft, Google, OpenAI, and Databricks are examples of companies operating across models, infrastructure, platforms, and applications simultaneously. The diagram shows the typical flow of dependency, not a strict boundary between categories.

One distinction is worth flagging early, since it's probably the most common point of confusion in the market today: workflows follow predefined paths, while agents dynamically decide which tools, steps, or information sources are needed to reach a goal. LangGraph and n8n can look similar from a distance, but one is deciding as it goes and the other is executing a script.

At a high level, the modern AI ecosystem can be viewed as eleven interconnected layers:

Contents

1. Foundation Models and Cognitive Engines
2. AI Infrastructure and Compute
3. AI Platforms and Model Providers
4. Protocols and Standards
5. Agent Frameworks and Orchestration
6. Low-Code Workflow Automation and Application Builders
7. AI-Native Developer Tools
8. Knowledge and Retrieval Platforms
9. AI Evaluation, Monitoring, and Observability
10. AI Security and Governance
11. AI Applications and Vertical Platforms

1. Foundation Models and Cognitive Engines

Foundation models are the "brains" of modern AI systems. They perform reasoning, generate code, answer questions, summarize documents, create images, and increasingly perform complex planning tasks.

These models can be accessed directly through consumer applications or via APIs that developers integrate into their own software. In production environments, organizations increasingly use multiple models rather than a single provider, routing different tasks to different models based on cost, latency, capability, privacy, and reliability requirements.

Frontier models continue to improve rapidly, with capabilities, pricing, and performance changing frequently. Rather than focusing on individual model names, it is more useful to understand the strategic differences between providers: reasoning ability, coding performance, context length, cost, openness, multimodal capability, and enterprise integration.

Beyond language models, foundation-model techniques are expanding into image, video, and audio generation. Platforms such as Midjourney, Runway, and ElevenLabs represent this broader multimodal AI ecosystem, alongside the multimodal capabilities increasingly built into models like GPT and Gemini.

This section covers three distinct categories that often get lumped together: model providers who train the underlying models, model ecosystems and platforms that host and distribute them, and AI search applications that are built on top of models but compete on interface rather than model quality.

Model providers

Claude (Anthropic)

Claude has become one of the strongest models for software engineering and analytical work. It is particularly known for:

• Exceptional programming capabilities
• Strong logical reasoning
• Very large context windows
• High-quality long-form writing
• Excellent document analysis

Best suited for: software development, technical documentation, research, and enterprise knowledge work.

ChatGPT (OpenAI)

ChatGPT remains the most versatile all-round AI platform, combining leading language models with an extensive ecosystem.

Its strengths include:

• Advanced Data Analysis
• Image generation
• Voice interaction
• Web browsing
• Custom GPTs
• Enterprise integrations

Best suited for: general productivity, business users, developers, and enterprise deployments.

Google Gemini

Gemini integrates deeply with Google Workspace and Google's cloud ecosystem.

It differentiates itself through:

• Native multimodal capabilities
• Extremely large context windows
• Integration with Gmail, Docs, Drive, and Workspace
• Strong performance across text, code, images, and video

Best suited for: organizations invested in Google Cloud or Google Workspace.

DeepSeek

DeepSeek significantly disrupted the market by demonstrating that open-weight models could approach frontier-level reasoning at substantially lower cost.

Its strengths include:

• Competitive coding performance
• Excellent mathematical reasoning
• Very low inference cost
• Open-weight availability

Best suited for: cost-sensitive deployments and organizations wanting greater control over model hosting.

Meta's Llama family remains one of the most important open-weight ecosystems, particularly for organizations that want to fine-tune models, deploy privately, or avoid dependence on a single commercial provider. Together, DeepSeek and Llama have made "open-weight" a strategic category in its own right, not just a budget alternative to frontier APIs.

Model ecosystems and platforms

Hugging Face

Hugging Face has become the central marketplace, hosting platform, and collaboration hub for open AI models. It provides:

• Model hosting
• Dataset repositories
• Open-source model discovery
• Inference services
• Fine-tuning tools
• Enterprise AI infrastructure

Best suited for: researchers, developers, and organizations building with open models.

AI search applications

Perplexity

Perplexity occupies a different category from the general-purpose chat assistants above: it combines large language models with real-time web search, citations, and answer synthesis. Its differentiation is not building the biggest model, but creating a research-oriented AI interface optimized for finding and summarizing current information.

Best suited for: research, knowledge discovery, competitive analysis, and fact-finding.

Other notable foundation model providers

The frontier model landscape continues to expand, with major contributions from:

• xAI (Grok)
• Mistral AI
• Cohere
• Alibaba (Qwen)

Each brings different strengths in reasoning, multilingual capabilities, enterprise deployment, or open-weight accessibility.

Open-weight vs. proprietary models

One of the biggest strategic decisions a company makes is which side of this line to build on. OpenAI, Anthropic, and Google optimize for frontier capability, managed services, and enterprise reliability. Open-weight ecosystems such as Llama, DeepSeek, and Mistral prioritize flexibility, customization, and private deployment. Most enterprise stacks end up using both: a proprietary frontier model for the highest-stakes reasoning, and open-weight models for cost-sensitive or privacy-sensitive workloads.

2. AI Infrastructure and Compute

Every AI system depends on specialized computing infrastructure: GPUs, networking, storage, and cloud platforms that train and run models at scale. This layer is largely invisible to end users but is one of the most strategically important parts of the AI ecosystem — it determines how much AI costs, how fast it runs, and who can afford to build at the frontier.

Leading platforms include:

• NVIDIA — dominant provider of AI acceleration hardware
• Google TPU — specialized AI chips powering Google's AI infrastructure
• AWS Trainium and Inferentia — custom AI chips for cloud workloads
• AMD Instinct — GPU alternative gaining traction for AI training and inference
• CoreWeave — specialized GPU cloud provider
• Lambda — AI infrastructure provider for developers and enterprises

Best suited for: organizations training models, operating large AI workloads, or requiring dedicated AI infrastructure.

3. AI Platforms and Model Providers

Many organizations no longer integrate directly with a single language model. Instead, they use AI platforms that provide access to multiple models through a unified interface.

These platforms solve operational challenges such as:

• Model routing
• Cost optimization
• Automatic failover
• Enterprise governance
• Usage monitoring
• API standardization

Leading platforms include:

• OpenRouter
• AWS Bedrock
• Azure AI Foundry
• Google Vertex AI
• Together AI
• Fireworks AI
• Groq
• Replicate

OpenRouter has become popular among developers because it provides a unified API layer across many commercial and open-weight models, making experimentation and model switching easier.

Microsoft Copilot ecosystem

Microsoft is a central player at this layer rather than at the model layer itself. Its advantage is not primarily building foundation models — it's embedding AI into existing enterprise workflows people already use every day. That shows up as Microsoft 365 Copilot, GitHub Copilot, Copilot Studio, Azure AI Foundry, and Dynamics 365 Copilot, which together package OpenAI's models (and others) into governed, enterprise-ready deployments at massive scale.

Best suited for: enterprises already invested in Microsoft 365, Azure, or Dynamics.

Google Cloud AI ecosystem

Google occupies a similar strategic position through Gemini Enterprise, Vertex AI, Google Workspace integrations, and Agentspace. Its advantage comes from combining frontier models with Google's search, data infrastructure, and productivity ecosystem — a different path to the same goal of packaging AI into tools people already use.

Best suited for: enterprises already invested in Google Cloud or Google Workspace.

Why use these platforms?

Instead of rewriting your application every time a better model appears, these platforms allow you to switch between providers with minimal effort. This abstraction layer is becoming increasingly important because enterprises increasingly want model flexibility rather than dependency on a single AI provider.

4. Protocols and Standards

As the AI stack matures, open protocols are emerging as the glue between layers — much like HTTP standardized the web.

Model Context Protocol (MCP)

MCP is an open standard originally introduced by Anthropic that has gained broad adoption across AI applications, developer tools, and agent frameworks, connecting AI applications to external tools, data sources, and workflows through a universal interface.

Instead of building a custom integration for every database, API, or SaaS product, developers expose capabilities through MCP servers that any compatible client can use. MCP has become one of the leading approaches for standardizing AI-to-tool communication.

Best suited for: tool connectivity, data access, and reducing N×M integration work across models and services.

Agent-to-Agent (A2A) Protocol

A2A, originally introduced by Google, defines how autonomous agents discover and communicate with each other across different frameworks and vendors.

Where MCP connects an agent to tools and data, A2A connects agents to other agents — enabling multi-vendor orchestration without locking into a single platform's ecosystem.

Best suited for: enterprise deployments where agents built with different frameworks need to collaborate.

5. Agent Frameworks and Orchestration

While foundation models generate responses, agent frameworks enable AI systems to execute complex workflows, call external tools, remember context, collaborate with other agents, and complete multi-step tasks.

This is where AI moves from answering questions to performing work. Agent frameworks do not create autonomous employees. They provide controlled systems for combining models, tools, memory, and workflows to complete specific tasks.

Google ADK

Google's Agent Development Kit is an enterprise-focused execution framework designed for production AI systems.

It emphasizes:

• Reliable execution
• Multi-language support
• Agent-to-Agent communication
• Secure tool execution
• Enterprise governance

Best for organizations building large-scale production agents.

LangGraph

LangGraph has become one of the leading frameworks for building stateful production AI systems. Part of the broader LangChain ecosystem, it pairs naturally with LangSmith for tracing and evaluation.

Its strengths include:

• Deterministic workflows
• Long-running agents
• Complex state management
• Human approval steps
• Production orchestration with checkpointing and crash recovery

Best for sophisticated enterprise AI workflows where reliability matters more than speed of prototyping.

Other major frameworks

The ecosystem has consolidated around a smaller set of production-ready options:

• OpenAI Agents SDK — lightweight agent orchestration with tool use, handoffs, and structured workflows
• CrewAI — rapid multi-agent prototyping with role-based agents
• Microsoft Agent Framework — the successor to AutoGen and Semantic Kernel, focused on Azure and .NET enterprise teams
• LlamaIndex — RAG-native pipelines for search and question-answering
• Mastra — TypeScript-native framework for production agents
• Vercel AI SDK — streaming, tool calling, and agent patterns for React and Next.js applications

AutoGen is now in maintenance mode — new Microsoft-centric agent work flows through Agent Framework instead. For web-native teams, Mastra and the Vercel AI SDK have become the default starting points alongside LangGraph for Python-heavy backends.

6. Low-Code Workflow Automation and Application Builders

Not every AI solution requires a team of software engineers.

Many organizations simply need to automate repetitive work, connect existing software, or build lightweight internal tools. These platforms dramatically reduce development effort.

Workflow Automation

These tools connect existing business systems using visual workflows.

Leading platforms include:

• n8n
• Make
• Zapier
• Lindy

Zapier remains one of the most widely adopted business automation platforms, particularly among non-technical teams, and has increasingly added AI-driven workflow creation. Microsoft Power Automate remains a major enterprise automation platform, particularly among organizations already using Microsoft 365 and Dynamics.

Typical use cases: CRM automation, AI-powered email processing, document workflows, ticket routing, customer support automation.

Internal Application Builders

These platforms allow teams to rapidly build dashboards, portals, and internal software.

Popular options include:

• Retool
• ToolJet
• Appsmith

Instead of building applications from scratch, developers assemble interfaces using reusable components connected to databases and APIs.

Enterprise AI Portals

Some organizations require secure environments where employees can interact with AI while keeping sensitive company data protected.

Examples include:

• Langdock
• Glean
• Writer
• Microsoft Copilot Studio — enterprise platform for creating custom AI agents connected to business data, workflows, and Microsoft 365 services

These platforms emphasize governance, compliance, identity management, and enterprise search.

7. AI-Native Developer Tools

Traditional code editors are rapidly evolving into AI-powered development environments.

Rather than acting as simple autocomplete tools, they increasingly understand entire repositories and collaborate with developers throughout the software lifecycle.

Many highly productive developers increasingly combine multiple AI coding tools, using each where it performs best. A common stack is Cursor or Copilot for daily editing, paired with Claude Code for complex refactors, debugging, and agentic terminal work — each tool covers a different part of the workflow.

Cursor

Cursor helped popularize the AI-native IDE category.

Its strengths include:

• Repository-wide awareness
• AI-assisted refactoring
• Intelligent debugging
• Codebase understanding
• Integrated chat and multi-file Composer mode

Best suited for: daily feature work, fast iteration, and developers who live inside an IDE.

Claude Code

Claude Code brings AI directly into the terminal. It can modify repositories, execute commands, run tests, fix compilation errors, and manage Git workflows without requiring a graphical interface.

Best suited for: large-scale refactors, autonomous debugging cycles, and complex multi-step engineering tasks.

Other notable developer tools

• GitHub Copilot — the largest enterprise AI coding ecosystem, deeply integrated into GitHub, VS Code, and Microsoft's developer platform
• GitHub Copilot Workspace — extends Copilot from autocomplete into planning and multi-step implementation of an entire task
• Windsurf — AI-native IDE with strong agentic editing
• Sourcegraph Cody — codebase search and context-aware assistance at scale
• Google Jules — asynchronous coding agent for background task execution

Open-source alternatives

A growing set of teams prefer open-source, model-flexible coding agents over closed platforms:

• Aider — terminal-based pair programming with any supported model
• OpenCode — terminal-first AI coding agent focused on local development, model flexibility, and developer control
• Continue — open-source, model-flexible assistant for VS Code and JetBrains

These tools differ primarily in how deeply they integrate with existing developer workflows — and increasingly interoperate through MCP.

8. Knowledge and Retrieval Platforms

Large language models do not inherently remember your company's documents, customer records, or proprietary knowledge.

Knowledge platforms solve this problem by storing information in vector databases optimized for semantic search. Modern enterprise knowledge systems increasingly combine vector search, keyword retrieval, metadata filtering, and knowledge graphs rather than relying on embeddings alone.

Leading technologies include:

• Pinecone
• Weaviate
• Qdrant
• Milvus

These platforms power Retrieval-Augmented Generation (RAG), allowing AI systems to retrieve relevant company knowledge before generating responses.

They are a foundational component of most enterprise AI applications. Many teams also use managed RAG from cloud providers — such as Amazon Bedrock Knowledge Bases or Vertex AI Search — rather than assembling vector infrastructure themselves.

Beyond vector search, many enterprise systems are adopting GraphRAG approaches that combine document retrieval with structured relationships between people, processes, products, and business entities. Examples include Microsoft GraphRAG, Neo4j, and Amazon Neptune.

Enterprise knowledge search

Above the developer-facing vector databases sits a second, employee-facing category: enterprise knowledge platforms that provide secure AI search across a company's own systems — email, chat, wikis, tickets, and file storage. Examples include Glean, Microsoft Copilot Search, Google Agentspace, and Elastic AI Search. These platforms focus less on developers building applications and more on giving employees direct, governed access to organizational knowledge.

Data infrastructure

While vector databases power semantic retrieval, enterprise AI increasingly depends on broader data infrastructure — warehouses, lakehouses, streaming platforms, and governance systems that provide reliable access to business information. A production AI system rarely connects only to documents; it increasingly connects to operational databases, customer records, analytics warehouses, and event streams as well. Leading platforms in this layer include Snowflake, Databricks, Google BigQuery, Microsoft Fabric, and Confluent.

9. AI Evaluation, Monitoring, and Observability

Building an AI system is only the beginning.

Production deployments require continuous monitoring to answer questions such as:

• Is the model producing accurate responses?
• Has quality degraded?
• Which prompts perform best?
• Which model is most cost-effective?
• Are hallucinations increasing?

Leading observability platforms include:

• Langfuse
• LangSmith
• Weights & Biases
• Arize AI

This layer is becoming increasingly important as organizations move AI into production — especially alongside agent frameworks like LangGraph, where every state transition can be traced and evaluated.

10. AI Security and Governance

As AI moves into production, organizations need controls around privacy, security, compliance, and responsible usage. This layer manages data protection, model access policies, prompt security, AI risk management, and regulatory compliance — the concerns that turn a working prototype into something a security team will actually approve.

Leading platforms include:

• Lakera — prompt injection and LLM security testing
• Protect AI — AI/ML supply chain and model security
• HiddenLayer — AI detection and response for model attacks
• Microsoft Purview (AI capabilities) — data protection and compliance for AI usage
• Palo Alto Networks (AI Security) — network and application-layer AI security

A related category focuses specifically on AI governance — model inventory, policy enforcement, and regulatory compliance rather than attack prevention. Examples include IBM watsonx.governance, OneTrust AI Governance, and Credo AI.

Best suited for: enterprises deploying AI at scale, particularly in regulated industries.

11. AI Applications and Vertical Platforms

At the top of the stack sit the products end users actually touch. These split into two distinct groups.

General AI applications are built on foundation models but packaged for a broad audience rather than a single industry:

• ChatGPT
• Perplexity
• Notion AI
• GitHub Copilot

Vertical AI applications are companies that don't build general-purpose AI models at all — instead, they combine domain expertise, proprietary workflows, and AI models to solve highly specific business problems within individual industries:

The line between the two groups isn't always sharp — GitHub Copilot is general-purpose for developers but industry-specific in practice — but the distinction still helps: general AI applications compete on breadth, vertical AI applications compete on depth.

Putting It All Together

Understanding where a product sits within the AI ecosystem makes it much easier to choose the right tool.

For example:

Claude vs. ChatGPT: Both are foundation models, but Claude is often preferred for long-form reasoning and software engineering, while ChatGPT offers a broader productivity ecosystem with integrated tools.

Cursor vs. Claude Code: Both assist developers, but Cursor is an AI-native IDE for daily editing, whereas Claude Code is a terminal-first agent for complex autonomous work. Most teams use both.

LangGraph vs. n8n: Both automate workflows, but LangGraph is a programming framework for building AI agents, while n8n is a visual automation platform for business processes.

MCP vs. an agent framework: MCP is a connectivity protocol for tools and data; LangGraph is an execution engine for agent logic. You typically need both.

OpenRouter vs. AWS Bedrock: Both provide access to multiple models, but OpenRouter is optimized for developer flexibility and easy model switching, while Bedrock focuses on enterprise governance, security, and AWS integration.

Retool vs. Langdock: Retool is designed for building internal applications, while Langdock provides a secure enterprise interface for interacting with multiple AI models and organizational knowledge.

Although these products may seem to compete, they often work together within the same architecture.

Choosing the Right AI Stack

A practical decision framework:

If you are an individual user

Start with ChatGPT, Claude, or Gemini, and choose based on workflow preference.

If you are a developer

A typical stack: Claude / GPT / Gemini / Llama for the model layer, Cursor / Claude Code for the application layer, LangGraph / Agents SDK for agent orchestration, Pinecone / Weaviate / GraphRAG for the knowledge layer, AWS Bedrock / Azure AI Foundry / Vertex AI for the platform layer, and LangSmith / Langfuse for observability.

If you are a business team

A typical stack: ChatGPT Enterprise / Gemini Enterprise / Microsoft Copilot for daily productivity, n8n / Zapier / Make for automation, Retool for internal tools, and enterprise AI search tools for governed knowledge access.

If you are building a startup

A typical stack: a foundation model API, an agent framework, a vector database, an observability layer, and a cloud AI platform for deployment and scaling — plus an evaluation and security layer once the product reaches production, which many startups underestimate until something goes wrong.

Example: Building an AI Customer Support Agent

The layers above rarely show up one at a time — a real system stacks several of them together. A production customer support agent might combine:

The important insight is that no single product creates the entire solution — the system is the product.

Final Thoughts

The AI ecosystem is no longer defined by a handful of chatbots. It has matured into a layered platform landscape where foundation models, infrastructure providers, AI clouds, open protocols, agent frameworks, workflow builders, developer tools, retrieval systems, observability platforms, security and governance tools, and vertical applications each play distinct roles.

Understanding these layers helps cut through marketing noise and makes it easier to evaluate new products as they emerge. Rather than asking, "Which AI tool is best?", a more useful question is:

Which layer of the AI stack am I trying to solve?

Once you can answer that, choosing the right platform becomes significantly easier.

The long-term trend is convergence. Model providers are moving upward into applications, application companies are moving downward into infrastructure, and cloud providers are building complete AI platforms that span every layer. The winning companies will likely not own every layer — but they will understand how to combine the layers effectively.

The companies that succeed with AI will rarely be those that simply adopt the most powerful model. They will be the ones that design the right system: combining models, data, workflows, infrastructure, governance, and human expertise to create measurable business value.

I've noticed the same pattern in nearly every fast-growing startup I've worked with or interviewed at.

Early on, low-code tools — Retool, Make, Zapier, n8n, Airtable, and tools like them — are a superpower. Operations teams can build dashboards, automate workflows, and move far faster than engineering ever could.

As the company scales, the same systems start quietly becoming a liability — not because the tools are bad, but because they're now responsible for business logic they were never designed to own.

I ran into this firsthand while building internal systems for a startup in the private markets space. Since then, I've heard the same story from multiple other growth-stage startups building out their operations layer this way.

The trap

Low-code tools are optimized for speed of delivery, not for the guarantees that become important once real money and real operational scale are involved — things like idempotency, retries, rate limits, and transactional integrity.

So the failure mode is almost always the same set of symptoms. Records get created twice because a webhook fired twice and nothing was there to deduplicate it. A third-party API rate-limits a workflow mid-run, and the automation just stops, with no alert, no retry, and half a business process complete. Two workflows touch the same record at the same time and one silently overwrites the other. Nobody notices for weeks, because there's no error — just quietly wrong data sitting in a system that ops and finance no longer fully trust.

That last part is the real cost. It's not a downtime incident. It's the slow erosion of confidence in your own data. Once someone on the ops or finance team has been burned by a silent duplicate or a missing record, they stop trusting the automation and start manually double-checking everything it does — which defeats the entire reason you adopted the tool.

Why this happens

The problem isn't that low-code platforms are unreliable. They're optimized for orchestration, not ownership.

Triage, not rewrite

This is usually where a senior engineer gets pulled in.

Most teams think the answer is a rewrite.

It usually isn't.

The real job is triage.

The moment a workflow becomes the canonical place where business rules live, you've unintentionally turned an automation platform into your application backend. That's a very different job.

From there, the work is clear: go through the tangle of low-code workflows and sort them into two buckets.

Keep in low-code:

• Notify people
• Sync systems
• Format data for dashboards
• Trigger downstream workflows
• Connect third-party tools

Move behind an API:

• Decide whether a payment is valid
• Reconcile transactions
• Apply approval rules
• Update customer state
• Make compliance decisions
• Own source-of-truth records

The principle underneath all of this is simple:

Low-code should orchestrate. Your backend should own state.

Get that split right, and you build a thin, well-defined backend service that owns the business logic and exposes clean endpoints — the low-code tools become clients of that service instead of the system of record. Ops keeps their speed on the workflow layer. Engineering guards the parts of the system where a silent bug becomes a real business problem.

Get it wrong — either by rewriting everything, or by leaving business-critical logic in a visual workflow builder because "it's not that complicated" — and you either burn a quarter of engineering time on a rewrite nobody asked for, or keep shipping the same class of silent bugs at higher and higher volume.

Why this matters for engineering leadership

Engineering leadership at a growth-stage startup isn't about replacing every imperfect system with a perfect one. It's about deciding where guarantees matter and where speed matters — and putting each tool in the role it was designed for.

That's a systems design skill, but it's also a judgment call about the business — you have to understand what's actually at risk (money, compliance, customer trust) versus what's just operational convenience. The engineers and leaders who get this right tend to be the ones who've actually lived through the low-code-to-backend transition once, not the ones who read about it.

If you're an operations leader at a growth-stage startup right now running your business on low-code tools, this isn't a warning to abandon them. It's a prompt to ask one question honestly: which of these workflows would actually hurt if it silently duplicated a record tomorrow? If the answer is money, compliance, or customer trust, you've probably found the boundary between orchestration and your backend.

Everything so far assumes messages always arrive and subscribers always succeed — and, just as importantly, that everything lives in one process's memory. Real systems cross that boundary: publishers and subscribers run on different machines, connected by a network that drops packets, and processes that crash and restart. That single step — leaving one memory space — is what creates a whole layer of concerns the in-memory version never had to face: persistence (a message must survive a crash, so it's written to durable storage before it counts as published), acknowledgements (a broker can't know a subscriber finished unless the subscriber explicitly says so), replication (the broker itself can't be a single point of failure), and network failures (any message, or any acknowledgement, can simply not arrive). None of these need solving here in depth — but every concept below exists because of them. These four — delivery guarantees, idempotency and retries, dead-letter queues, and backpressure — are also the vocabulary of nearly every serious conversation about event-driven systems, whether that's a design review or a systems interview.

Delivery guarantees

There are three worth knowing: at-most-once (a message is delivered zero or one times — it can be lost, never duplicated), at-least-once (delivered one or more times — never silently lost, but sometimes processed twice), and exactly-once (delivered precisely once — the one that's genuinely hard to guarantee across a network, and not something retries alone get you to). Most systems, including Kafka and SQS by default, settle for at-least-once: the broker redelivers until it receives an acknowledgement, and a lost acknowledgement looks identical to a lost message — so duplicates aren't a rare accident, they're the mechanism working as designed.

Idempotency

At-least-once delivery pushes the duplicate problem onto the consumer, which means idempotency isn't an optimization — it's the other half of the delivery guarantee. A consumer is idempotent when processing the same message a second time produces the same outcome as the first: a safe no-op, not a duplicate charge or a duplicate row. The standard mechanism is a unique ID per logical event and an atomic "claim" against a ledger of processed IDs (a database unique constraint doing the check-and-record in one step) — checked before the real work runs, so a redelivered message gets recognized and skipped no matter which consumer instance it lands on.

Fault tolerance

When a subscriber fails, retrying immediately is tempting and usually wrong — if the failure was caused by an overloaded downstream dependency, hammering it with instant retries just adds to the overload. The standard fix is exponential backoff with jitter: each retry waits roughly twice as long as the last, with some randomness added so a batch of consumers that failed at the same moment don't all retry in lockstep and recreate the exact spike they're recovering from.

Error handling

A message can't retry forever. After some capped number of attempts, it should stop competing for the consumer's attention and move to a dead-letter queue (DLQ) instead — a place to inspect what failed and why, rather than a place where messages just quietly vanish. A DLQ doesn't fix anything by itself; its value is observability (a growing DLQ is a leading indicator of a bug worth alerting on) and recoverability (once the underlying issue is fixed, those messages can be replayed back through the system deliberately).

Flow control

A queue doesn't create capacity — it just delays the moment you run out of it. If subscribers can sustainably handle less than what publishers are producing, an unbounded queue just grows until something else breaks. Backpressure means giving each queue a real limit and deciding, deliberately, what happens once it's hit: block the publisher until there's room (no data loss, but a slow subscriber now stalls everyone upstream of it), drop the newest or oldest message (bounded memory, but silent, potentially costly data loss), or reject the new message with an error (no silent loss, but publishing is no longer fire-and-forget — something upstream now has to handle the rejection). For anything where losing or duplicating an event has a real cost, rejecting loudly is usually the safer default than failing silently.

Put together, these four are what separate a pub/sub implementation that works in a demo from one that survives contact with real traffic: it degrades predictably under failure, instead of losing data, duplicating it, or falling over.

A single channel doesn't scale past a toy example — every subscriber sees every message. The fix is a routing layer: publishers emit to a named topic, and subscribers express interest in specific topics (or patterns of topics) instead of receiving everything indiscriminately. Think of it as three roles: the publisher emits an event with a topic attached, a routing layer — the exchange, in message-broker terms — decides which subscriber lists match that topic, and each matching subscriber has its own filtered queue that only ever receives what it asked for.

Rather than bolting routing onto the naive synchronous loop from Part 1 — which would reintroduce every bug Part 2 just fixed — this combines topic matching with the per-subscriber queues and workers we already built. The exchange decides who a message goes to; each matched subscriber's own queue and worker still decide when it gets processed.

// topic-pubsub.ts — topic routing combined with Part 2's async workers
type Subscriber = (payload: unknown) => void;

interface TopicWorker {
  subscriber: Subscriber;
  queue: unknown[];
  processing: boolean;
}

class TopicPubSub {
  // Maps a topic pattern (e.g. "order.*") to that pattern's workers.
  // This Map is the exchange: it decides who a message reaches.
  private topics = new Map<string, TopicWorker[]>();

  subscribe(pattern: string, subscriber: Subscriber): void {
    const workers = this.topics.get(pattern) ?? [];
    workers.push({ subscriber, queue: [], processing: false });
    this.topics.set(pattern, workers);
  }

  unsubscribe(pattern: string, subscriber: Subscriber): void {
    const workers = this.topics.get(pattern);
    if (!workers) return;
    const i = workers.findIndex((w) => w.subscriber === subscriber);
    if (i !== -1) workers.splice(i, 1);
    if (workers.length === 0) this.topics.delete(pattern);
  }

  // A pattern segment of "*" matches exactly one topic segment — "order.*"
  // matches "order.created" and "order.failed", but not "order.created.retry"
  // (different segment count) and not "order" alone.
  private matches(pattern: string, topic: string): boolean {
    const patternParts = pattern.split(".");
    const topicParts = topic.split(".");
    if (patternParts.length !== topicParts.length) return false;
    return patternParts.every((p, i) => p === "*" || p === topicParts[i]);
  }

  publish(topic: string, payload: unknown): void {
    for (const [pattern, workers] of [...this.topics]) {
      if (this.matches(pattern, topic)) {
        // Snapshot: protects the loop from mid-dispatch mutations
        for (const worker of [...workers]) {
          worker.queue.push(payload);
          if (!worker.processing) this.processQueue(worker);
        }
      }
    }
  }

  private async processQueue(worker: TopicWorker): Promise<void> {
    worker.processing = true;
    while (worker.queue.length > 0) {
      const message = worker.queue.shift();
      try {
        await Promise.resolve(worker.subscriber(message));
      } catch (err) {
        console.error("Subscriber failed:", err);
      }
    }
    worker.processing = false;
  }
}

A subscriber to order.* catches both order.created and order.failed without needing to subscribe to each individually — and without receiving user.registered at all. And because delivery still goes through per-subscriber queues, everything Part 2 established carries over: a throwing subscriber is isolated to itself, a slow one only backs up its own queue, and per-subscriber ordering holds.

[Publisher] ── "order.created" ──> [Exchange]
                                       │  matches patterns
                          ┌────────────┼────────────────┐
                    "order.*"    "order.created"    "user.*"
                       ✓               ✓               ✗
                       │               │
                  [queue+worker]  [queue+worker]    (not delivered)

This publisher → exchange → filtered-queue shape is RabbitMQ's topic exchange almost literally — its routing keys use the same dot-segmented patterns, with * matching one segment. Kafka takes a coarser approach (whole topics, with consumers subscribing per topic), and SNS filters with message attributes — but the role split is the same everywhere: something in the middle decides who gets what, so publishers and subscribers never have to know about each other.

Edge case: orphaned topics. unsubscribe deletes a pattern's entry once its worker list is empty, so the routing table doesn't accumulate dead patterns forever in a long-running process.

Edge case: messages matching no subscribers. Publishing to a topic nobody's listening for is a silent no-op — it doesn't throw, but it also doesn't tell you the data went nowhere. In production systems this is worth catching deliberately: an unrouted-message log or metric (publishes per topic, publishes with zero matches), so "nothing is consuming this event" is something you find out from a dashboard rather than from a missing side effect weeks later.

Edge case: mid-dispatch mutation. publish() iterates over snapshots ([...this.topics], [...workers]) rather than the live collections. If a subscriber's callback calls unsubscribe() while a dispatch is in flight, mutating the array mid-loop would shift indices under the iterator and silently skip sibling subscribers. Snapshotting makes the in-flight dispatch immune to that: the unsubscribe still takes effect for all future publishes, it just can't corrupt the current one.

Next up: how do you make this setup production-ready — resilient to failures, and tolerant of the things that inevitably go wrong once real traffic hits it? Continue to Part 4: Production Resilience and Guarantees.

The fix for both edge cases from Part 1 is the same: stop calling subscribers directly from inside publish(). Enqueue the message instead, return immediately, and let each subscriber process on its own schedule. Each subscriber gets its own queue and its own dedicated worker that drains it.

// pubsub.ts — non-blocking publish, per-subscriber failure isolation
type Subscriber = (message: unknown) => void;

interface SubscriberWorker {
  subscriber: Subscriber;
  queue: unknown[];
  processing: boolean;
}

class PubSub {
  private subscribers: SubscriberWorker[] = [];

  subscribe(subscriber: Subscriber): void {
    this.subscribers.push({ subscriber, queue: [], processing: false });
  }

  publish(message: unknown): void {
    for (const worker of this.subscribers) {
      worker.queue.push(message);
      if (!worker.processing) this.processQueue(worker);
    }
  }

  private async processQueue(worker: SubscriberWorker): Promise<void> {
    worker.processing = true;
    while (worker.queue.length > 0) {
      const message = worker.queue.shift();
      try {
        await Promise.resolve(worker.subscriber(message));
      } catch (err) {
        console.error("Subscriber failed:", err);
      }
    }
    worker.processing = false;
  }
}

publish() now just pushes and returns — it never waits on a subscriber, so a slow one only ever blocks itself. And each subscriber's try/catch means a thrown error is isolated to that subscriber; it no longer takes any other subscriber down with it.

                  ┌──> [queue: ▓▓▓░░] ──> [worker A] ──> handler A
[publish()] ──────┼──> [queue: ▓░░░░] ──> [worker B] ──> handler B
 returns          └──> [queue: ▓▓▓▓▓] ──> [worker C] ──> handler C (slow —
 immediately                                              only its own queue grows)

This queue-per-consumer shape is exactly what managed systems give you at infrastructure scale: an SQS queue per consuming service, a BullMQ queue per job type, a RabbitMQ queue bound per consumer — same isolation idea, with the queue living outside your process instead of in an array.

Ordering: giving each subscriber its own worker isn't just about isolation — it's also what keeps messages in order. The while loop only dequeues the next message after await-ing the current one, so message n can't start before message n-1 finishes, no matter how fast either one actually is. Publish three messages with wildly different processing times — the slowest first, the fastest second — and they still come out in the order they were published, because the queue never starts message two before message one is done.

Edge case: memory limits. Nothing bounds worker.queue here. A subscriber that's chronically slower than the rate messages arrive just accumulates an ever-growing array, with no signal back to the publisher that anything's wrong.

Edge case: a worker crashing mid-job. If the process dies while a message is being processed — after the real work happened, but before anything recorded that it happened — that message's fate becomes ambiguous. Was it done? Partially done? On restart, is it retried and processed twice, or is it just gone?

Next up: every subscriber still sees every message, whether it's relevant to it or not. How do you route messages to the subscribers that actually care, instead of broadcasting everything to everyone? Continue to Part 3: Message Routing with Topics.

Pub/Sub (short for Publisher-Subscriber) is a messaging pattern that lets components communicate without knowing about each other directly: publishers emit events, subscribers react to them, and neither side holds a reference to the other. It's one of the core communication patterns underneath event-driven architecture (EDA) — when services react to order.created or user.registered events instead of calling each other directly, some form of pub/sub is what's carrying those events. That decoupling is what makes it worth reaching for: a component can react to something happening elsewhere without being wired into the code that caused it.

This four-part series builds the pattern up from the simplest possible version to the production concerns that show up once real traffic and real failures enter the picture. The implementations are intentionally simplified, in-memory, and single-process — the point is to understand the mechanics from the inside. Production systems like Kafka, RabbitMQ, or AWS SNS/SQS solve the same problems plus a layer of distributed-systems concerns (persistence, replication, network failures) that an in-memory version never has to face; where the concepts map onto those tools, the series points it out.

At its core, pub/sub decouples two roles: publishers, which emit events without knowing who's listening, and subscribers, which react to events without knowing who published them. Here's the simplest possible version — one array of subscriber functions, and a loop that calls each one when something is published.

// pubsub-basic.ts
export type Subscriber = (message: unknown) => void;

class BasicPubSub {
  private subscribers: Subscriber[] = [];

  subscribe(subscriber: Subscriber): void {
    this.subscribers.push(subscriber);
  }

  unsubscribe(subscriber: Subscriber): void {
    const i = this.subscribers.indexOf(subscriber);
    if (i !== -1) this.subscribers.splice(i, 1);
  }

  publish(message: unknown): void {
    for (const subscriber of this.subscribers) {
      subscriber(message);
    }
  }
}

A concrete use case makes this tangible: a user.registered event fires once, and several independent subscribers react to it — one sends a welcome email, another starts an onboarding checklist, a third logs it for analytics.

                          ┌──> [Email subscriber]      sends welcome email
[Publisher] ──publish──── ┼──> [Onboarding subscriber] starts checklist
 (signup code)            └──> [Analytics subscriber]  logs the event

const ps = new BasicPubSub();
ps.subscribe((user) => sendWelcomeEmail(user));
ps.subscribe((user) => startOnboardingChecklist(user));
ps.subscribe((user) => logSignupEvent(user));

ps.publish({ id: "u_123", email: "new-user@example.com" });

None of these subscribers know about each other, and the signup code that published the event doesn't know they exist either — that's decoupling in practice, not just in theory.

Edge case: what happens if a subscriber crashes? Badly, as written. for...of doesn't catch anything, so if the welcome-email subscriber throws, the onboarding and analytics subscribers never run at all — one bad subscriber takes the rest of that publish down with it.

Edge case: what happens if the publisher outpaces the subscriber? In this version, it can't. publish() is synchronous and blocking — it doesn't return until every subscriber has finished — so the caller is automatically throttled to the slowest subscriber's speed, whether it wants to be or not. That's not backpressure by design, it's just blocking, and it's also the real problem here: a slow welcome-email provider now determines how fast the signup flow itself can run.

Next up: what happens if one subscriber is slow, or throws, and you don't want it to affect anyone else — or block the publisher? Continue to Part 2: Asynchronous Workers.