[Fall 2025] CS590-06: Building Intelligent Agents with Frontier Models

Instructor: Shuyan Zhou
Time: Tuesday and Thursday 1:25pm - 2:40pm
Location: LSRC A155
Office hour: Thursday 2:40pm - 3:10pm at LSRC D329
Contact: We will use Slack for most communications. You will be added to Slack after the first lecture. If you join the class late, email us and we’ll add you. Once you’re on Slack, we prefer Slack messages over emails for all logistical questions. We also encourage students to use Slack for paper discussion and project collaboration. We also post important announcement on Canvas.
Feedback form: here

Overview

This graduate-seminar course will explore foundations and recent progress on the design and development of AI agents powered by large language models (LLMs). The course will be a mix of lectures and student-led presentations. You will have the opportunity to dive deeply into topics via in-class presentations, discussions and a course project. A central theme will be the transformation of language models from passive generators into autonomous agents—systems capable of pursuing goals, reasoning over time, and interacting with complex, dynamic environments. Follow your curiosity, push the boundaries of current research, and have fun!

Syllabus

Date	Day	Topic
8/26/25	Tuesday	[L] Intro [slides]
8/28/25	Thursday	[L] Pretraining and Scaling Law [slides]
9/2/25	Tuesday	[L] SFT & RLHF & RLVR [slides]
9/4/25	Thursday	[L] Benchmarking and Evaluation [slides]
9/9/25	Tuesday	Tool use agent Group 1: τ-bench: A Benchmark for Tool–Agent–User Interaction in Real-World Domains MUA-RL: Multi-Turn User-Interacting Agent Reinforcement Learning For Agentic Tool Use Group 2: PAL: Program-Aided Language Models Toolformer: Language Models Can Teach Themselves to Use Tools TORL: Scaling Tool-Integrated RL Recommended Readings: AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents Gorilla: Large Language Model Connected with Massive APIs DocPrompting: Generating Code by Retrieving the Docs ReAct: Synergizing Reasoning and Acting in Language Models
9/11/25	Thursday	Deep research Group 1: BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents WebGPT: Browser-Assisted Question-Answering with Human Feedback Group 2: GAIA: A Benchmark for General AI Assistants WebSailor: Navigating Super-human Reasoning for Web Agent Recommended Readings: Search-R1: Efficiently Searching for Reasoning Chains
9/16/25	Tuesday	Computer Use Agent Group 1: OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments WebArena: A Realistic Web Environment for Building Autonomous Agents TheAgentCompany: Benchmarking LLM Agents on Consequential Real-World Tasks Group 2: UI-TARS: Pioneering Automated GUI Interaction with Native Agents UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning Recommended Readings: CoAct-1: Computer-using Agents with Coding as Actions OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories Beyond Browsing: API-Based Web Agents
9/18/25	Thursday	Coding/SWE agent Group 1: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? LiveCodeBench: Holistic Evaluation of Real-World Coding Agents and Frameworks Group 2: Agentless: Demystifying LLM-based Software Engineering Agents OpenHands: An Open Platform for AI Software Developers as Generalist Agents Recommended Readings: Building effective agents, Anthropic Blog Post SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
9/23/25	Tuesday	Data for agents Group 1: InSTA: Towards Internet-Scale Training For Agents Group 2: SWE-Gym: Training Software Engineering Agents and Verifiers with SWE-Gym Recommended Readings: OpenCUA: An Open Framework for Computer Use Agents
9/25/25	Thursday	Agent memory Group 1: Cognitive Architectures for Language Agents Group 2: Voyager: An Open-Ended Embodied Agent with Large Language Models Agent Workflow Memory Recommended Readings: MemGPT: Towards LLMs as Operating Systems MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
9/30/25	Tuesday	Scaling interactions for agents Group 1: Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interaction Group 2: Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters Stream of Search (SoS): Learning to Search in Language Recommended Readings: Large Language Monkeys: Scaling Inference Compute with Repeated Sampling Tree of Thoughts: Deliberate Problem Solving with Large Language Models
10/2/25	Thursday	Invited Talk by Guohao Li (Eigent AI) Related Work CAMEL
10/7/25	Tuesday	Project lighting talk
10/9/25	Thursday	Fall break
10/14/25	Tuesday	Invited Talk: Demystifying Long Chain-of-Thought Reasoning in LLMs by Xiang Yue (CMU) Related Work Demystifying Long Chain-of-Thought Reasoning in LLMs Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? LIMO: Less is More for Reasoning LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters! Learning to Reason without External Rewards Reinforcement Learning for Reasoning in Large Language Models with One Training Example The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity Spurious Rewards: Rethinking Training Signals in RLVR Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
10/16/25	Thursday	World model Mastering Diverse Domains through World Models Recommended Readings: Critiques of World Models A Path Towards Autonomous Machine Intelligence
10/21/25	Tuesday	Self improvement AlphaEvolve: A coding agent for scientific and algorithmic discovery
10/23/25	Thursday	Multi-agent Why Do Multi-Agent LLM Systems Fail? Recommended Readings: Don't sleep on single-agent systems Don't build multi-agents Multi-agent research system
10/28/25	Tuesday	LLM as simulators Generative Agents: Interactive Simulacra of Human Behavior Recommended Readings: Towards Measuring the Representation of Subjective Global Opinions in Language Models User simulators bridge RL with real-world interaction
10/30/25	Thursday	Invited Talk: TBD
11/4/25	Tuesday	Embodied agents SayCan: Grounding Language in Robotic Affordances OpenVLA: An Open-Source Vision-Language-Action Model RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
11/6/25	Thursday	How can AIs help scientific research? Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers Recommended Readings: Galactica: A Large Language Model for Science
11/11/25	Tuesday	Societal impact of AI agents Future of Work with AI Agents: Auditing Automation and Augmentation Potential across the U.S. Workforce SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? Recommended Readings: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
11/13/25	Thursday	Safety and ethical considerations
11/18/25	Tuesday	Project presentation
11/20/25	Thursday	Project presentation
11/25/25	Tuesday	Project presentation + Report feedback

Deadlines

Weekly

• Main presenters: Submit slides in the dedicated Slack channel 24 hours before class.

Presentation

• 8/26: Form out in Slack to submit presentation preferences
• 8/28: Assignment out, open for negotiation, swap (feel free to use Slack DMs)
• 9/2: Final assignment out

Project

• 8/28-9/16 Team formation and topic preference
• 10/7 in-class project lighting talk
• 11/17 (Mon): Submit the project presentation slides (if you are presenting on 11/18)
• 11/19 (Wed): Submit the project presentation slides (if you are presenting on 11/20 and 11/25)
• 12/11: Final report

Resources

Compute To be announced

Example projects To be announced in Slack

Acknowledgement

The class format, guidelines and materials are inspired by the following classes:

• CS 294-288: Data-Centric Large Language Models by Sewon Min at UCB
• ME 555-06 Robotics Manipulation by Xianyi Cheng at Duke
• 11-711: Advanced NLP by Graham Neubig and Xiang Yue at CMU