Stanford CS329Z: Engineering AI Agents (Course Guide)
A guide to Stanford CS329Z: Engineering AI Agents, written in my own words from the course website as it stood on 6 November 2026. The slides, assignments and course text belong to the CS329Z teaching team; follow the links for the originals. The schedule is tentative and may change during the quarter.
CS329Z is a 3-unit Stanford course for Autumn 2026, taught by Diyi Yang, Michael Ryan and John Yang. It is about building software around language models: from a single prompted call to pipelines, then to compound systems that combine models with retrievers, tools and optimizers, and finally to agents that choose their own next step.
Students first write retrieval, tool calling and the agent loop by hand, then see how frameworks such as DSPy and LangGraph package the same patterns. The second half of the schedule moves from how agents are built to the data they need, how to evaluate them and how to keep them safe. A team project that has to be evaluated properly is the largest part of the grade.
At a glance
- When and where: Mondays and Wednesdays, 1:30–2:50 p.m. (Pacific), Skilling Auditorium, from 23 September to the demo day on 9 December 2026.
- Prerequisites: one of CS224N, CS224U, CS224V or CS336, or equivalent background in NLP.
- Grading basis: letter grade or Credit/No Credit.
- Teaching assistants: Owen Queen, Anusheh Chaudry, Shreyas Sharma and Houjun Liu.
- Official pages: schedule and readings, logistics, project.
Schedule
The slides are public PDFs. “Not yet” means they had not been posted by 6 November.
| Week | Date | Session | Slides |
|---|---|---|---|
| 1 | Wed 23 Sep | Introduction: What Are Agentic Systems? | |
| 2 | Mon 28 Sep | LLMs for Builders | |
| 2 | Wed 30 Sep | Retrieval-Augmented Generation (RAG) | |
| 3 | Mon 5 Oct | Tool Use & Function Calling | |
| 3 | Wed 7 Oct | Frameworks & Orchestration (code) | |
| 4 | Mon 12 Oct | Agent Design Patterns & Scaffolds | |
| 4 | Wed 14 Oct | Agent Memory Architectures | |
| 5 | Mon 19 Oct | Multi-Agent Systems | |
| 5 | Wed 21 Oct | Optimization | |
| 6 | Mon 26 Oct | Guest lecture: Terminal-Bench (Ryan Marten, Alex Shaw) | |
| 6 | Wed 28 Oct | What Data Do Agents Need? | |
| 7 | Mon 2 Nov | Data Selection & Quality | |
| 7 | Wed 4 Nov | Evaluation Fundamentals & Benchmark Design | |
| 8 | Mon 9 Nov | Fireside chat with Andrew Ng | Not yet |
| 8 | Wed 11 Nov | LLM-as-Judge & Evaluation Infrastructure | Not yet |
| 9 | Mon 16 Nov | Guest lecture: SWE-bench (Carlos Jimenez, Alexander Wettig) | Not yet |
| 9 | Wed 18 Nov | Agent Safety & Guardrails | Not yet |
| 10 | 23 and 25 Nov | No class (Thanksgiving) | |
| 11 | Mon 30 Nov | Coding & Software Agents | Not yet |
| 11 | Wed 2 Dec | Proactive Agents | Not yet |
| Finals | Wed 9 Dec | Final project demo day, 3:30–6:30 p.m. |
What each unit covers
The sessions are grouped into units. Each summary below is mine, followed by two or three of the assigned readings, all free to read. The course schedule lists the full set, including optional readings.
Foundations and landscape (23–28 Sep). What makes a system agentic rather than a fixed workflow, the parts of an agent (model, reasoning, memory, tools, environment), and the model internals that matter when you build on one: decoding, attention and context length, post-training, and context engineering.
- Zaharia et al., The Shift from Models to Compound AI Systems, BAIR Blog, 2024
- Kapoor et al., AI Agents That Matter, TMLR, 2025
- Rajasekaran et al., Effective Context Engineering for AI Agents, Anthropic, 2025
Building blocks (30 Sep–5 Oct). The two ways an agent reaches beyond what the model memorized: retrieval (indexing, sparse and dense retrievers, reranking) and tools called through function calling or the Model Context Protocol, plus how each is benchmarked and how it fails.
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS, 2020
- Schick et al., Toolformer: Language Models Can Teach Themselves to Use Tools, NeurIPS, 2023
- Model Context Protocol specification, revision 2025-06-18
Frameworks and agent design (7–12 Oct). What frameworks such as DSPy and LangGraph take over from hand-written code, a tour of open-source agent harnesses, and the standard patterns: fixed workflows versus agents, ReAct-style reasoning and acting, reflection, and ways to keep a long context under control.
- Khattab et al., DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, ICLR, 2024
- Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, ICLR, 2023
- Schluntz and Zhang, Building Effective AI Agents, Anthropic, 2024
Memory and multi-agent systems (14–19 Oct). What an agent should remember and where to keep it, when several agents beat one, how agents coordinate, and the usual reasons multi-agent systems break.
- Sumers et al., Cognitive Architectures for Language Agents, TMLR, 2024
- Packer et al., MemGPT: Towards LLMs as Operating Systems, arXiv, 2023
- Cemri et al., Why Do Multi-Agent LLM Systems Fail?, NeurIPS Datasets & Benchmarks, 2025
Optimization (21 Oct). Improving a system by tuning its prompts and examples or by training the model’s weights, and how to choose between the two.
- Agrawal et al., GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, ICLR, 2026
- Opsahl-Ong et al., Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs, EMNLP, 2024
Data for agentic systems (28 Oct–2 Nov). Where the data for optimizing and evaluating an agent comes from (logged traces, demonstrations, feedback, synthetic data), and how to pick the examples worth keeping.
- Shankar, Data Flywheels for LLM Applications, 2024
- Yang et al., SWE-smith: Scaling Data for Software Engineering Agents, NeurIPS Datasets & Benchmarks, 2025
- Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation with Human Preferences, UIST, 2024
Evaluation for agentic systems (4–11 Nov). Designing a benchmark, choosing between code-based graders, LLM judges and human raters, and measuring agents whose results change from run to run.
- Zhu et al., Establishing Best Practices for Building Rigorous Agentic Benchmarks, 2025
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS, 2023
- Grace et al., Demystifying Evals for AI Agents, Anthropic, 2026
Safety (18 Nov). Prompt injection, privacy risks that come with tool access, sandboxing and permissions, and where to keep a person in the loop.
- Shao et al., PrivacyLens: Evaluating Privacy Norm Awareness of LMs in Action, NeurIPS Datasets & Benchmarks, 2024
- OpenAI, Understanding Prompt Injections: A Frontier Security Challenge, 2025
Coding and proactive agents (30 Nov–2 Dec). How coding agents such as SWE-agent, OpenHands and Claude Code are put together, and agents that act before they are asked.
- Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS, 2024
- Wang et al., OpenHands: An Open Platform for AI Software Developers, ICLR, 2025
- Shaikh et al., Creating General User Models from Computer Use, UIST, 2025
Guest sessions. Two benchmark teams talk about their work: Terminal-Bench on 26 October (paper, site) and SWE-bench on 16 November (paper). Andrew Ng joins Diyi Yang for a fireside chat on 9 November.
Coursework and grading
There are two homeworks, each followed by a short closed-book quiz in which students explain their design choices:
- HW1, Build an Agentic Harness (weeks 3–6, starter code). Build a company’s internal assistant with no agent framework, only chat-completion calls and your own code. It starts as pipelines that search a corporate email archive and grows into a full harness with tools, a terminal, memory and human approval.
- HW2, Evaluate an Agent (weeks 6–9, starter code). Given a ready-made agent, build an evaluation suite for it: code-based graders, at least one LLM judge, benchmark tasks and an error analysis.
Each student also records a 10–15 minute video on a recent agent paper of their choice, due 13 November (handout). Peer review runs through the quarter in four rounds: paper videos, midpoint demos, HW2 evaluations and final demos, with two classmates’ work reviewed each time.
The project is done in teams of one to three people (the staff recommend three), each with a mentor from the teaching team. Teams build and evaluate an agentic system of their choice that touches at least one of the course topics. The project page is clear that a system shown working once will not score well: a team has to say what tasks the agent should handle, where its test data comes from and how success is measured, then compare against baselines and analyse the failures. The milestones are:
- A 1–2 page proposal in week 3.
- A recorded midpoint demo and a 3–4 page midway report in week 7.
- An 8-page final report in ICLR format and a live demo on 9 December.
The midpoint demo and the final report each come with a GitHub repository whose README lets someone else run the system, and every written milestone includes a statement of how AI tools were used.
| Component | Weight |
|---|---|
| Project (proposal 5, midway report 5, midpoint demo 5, final report 15, final demo 10) | 40% |
| Homework (HW1 and HW2, 7.5 each) | 15% |
| Quizzes after each homework (7.5 each) | 15% |
| Paper video | 10% |
| Peer review (four rounds, 5 each) | 20% |
On AI tools, the course expects students to use them. AI-written code is allowed in the homework and the project as long as the student can validate and explain all of it, which the staff check in oral exams. The project reports must be written by the students, though AI can help with brainstorming and feedback.
Following along from outside Stanford
Most of the material is public: the slides posted so far, the lecture 5 demo code, the starter code for both homeworks and almost all of the readings. The lecture recordings are on Canvas and discussion happens on Ed, both limited to people taking the course.
Stanford CS329Z: Engineering AI Agents (Course Guide)
