Phase 6 — Distributed Systems for AI

AI systems are distributed systems with a language model in them. When one agent run fans out into tool calls, queues, workers, and databases across several machines, the hard problems are no longer about prompts — they are the classic distributed-systems problems: what happens when a message is delivered twice, when a worker dies mid-task, when two nodes disagree, when a queue backs up, or when a downstream service gets slow.

This phase builds that toolkit. It is the difference between an agent that works in a notebook and an agent platform that survives a bad day.

What you will be able to do

By the end of this phase you should be able to:

• Reason about scalability, availability, reliability, fault tolerance, consistency, and the CAP trade-off.
• Choose between horizontal and vertical scaling, and between stateless and stateful services.
• Design traffic flow with load balancers, reverse proxies, API gateways, and service discovery.
• Work with message queues, Kafka, RabbitMQ, SQS, and Redis for asynchronous work.
• Explain event-driven architecture, pub/sub, consumer groups, and event sourcing.
• Reason about partitioning, ordering, and the delivery guarantees: at-most-once, at-least-once, exactly-once.
• Make operations idempotent, use distributed locks and leader election safely, and retry with backoff and jitter.
• Protect systems with timeouts, dead-letter queues, circuit breakers, bulkheads, rate limiting, and backpressure.
• Cache, replicate, shard, and partition data, and apply the saga pattern, transactional outbox, and CQRS.
• Run distributed agents: worker pools, scheduling, distributed state, and reliable long-running workflows.

The shape of a distributed AI system

flowchart LR
    U["Users / API"] --> G["API gateway"]
    G --> S["Stateless API workers"]
    S --> Q["Queue / Kafka"]
    Q --> W["Agent worker pool"]
    W --> M["Model + tools"]
    W --> DB["Postgres"]
    W --> C["Redis cache"]
    W --> DLQ["Dead-letter queue"]
    S -.-> SD["Service discovery"]
    W -.-> O["Observability"]
    DB -.-> R["Replicas / shards"]

Every box is a place where partial failure is normal, and every arrow is a place where a message can be lost, duplicated, delayed, or reordered. The engineering is making those outcomes boring.

Topic order

1. Distributed systems fundamentals — scalability, availability, reliability, fault tolerance.
2. Consistency and the CAP theorem — what you trade away.
3. Scaling services — horizontal vs vertical, stateless vs stateful.
4. Load balancing, proxies, and discovery — getting traffic to the right place.
5. Message queues and producer-consumer — asynchronous work.
6. Kafka — the distributed log.
7. Redis for distributed systems — locks, counters, queues, and caching.
8. RabbitMQ and SQS — brokered and cloud queues.
9. Event-driven architecture and pub/sub — events, consumer groups, and event sourcing.
10. Partitioning and ordering — scale and sequence.
11. Delivery guarantees — at-most-once, at-least-once, exactly-once.
12. Idempotency — the price of at-least-once.
13. Distributed locks and leader election — coordination.
14. Retries, backoff, jitter, and timeouts — failing without making it worse.
15. Dead-letter queues — where bad messages go.
16. Circuit breakers and bulkheads — containing failure.
17. Rate limiting and backpressure — protecting the system.
18. Caching and distributed caching — speed without stale lies.
19. Replication, sharding, and database partitioning — scaling data.
20. Saga pattern and transactional outbox — consistency without distributed transactions.
21. CQRS — separating reads and writes.
22. Workflow engines and distributed task execution — durable orchestration.
23. Agent worker pools and scheduling — running many agents.
24. Distributed state management — where the truth lives.
25. Long-running workflow reliability — surviving for hours or days.

How to study this phase. Ask two questions of every pattern: what happens when this component fails halfway? and what happens when this message is delivered twice? Nearly every idea here is an answer to one of those two.