Modern language models' in-context learning and instruction-following abilities make it possible to adapt agent behavior without updating model weights. This thesis studies online self-improvement, where an agent encounters each task only once and must accumulate reusable experience across a sequence of tasks, reflecting deployments in which actions have persistent consequences and retries may be impossible.

First, we ask whether the prompt optimizer itself can learn across episodes. Building on Agentic Context Engineering (ACE), we evaluate two variants with persistent optimizer-side state, including one bootstrapped through offline retries. AppWorld's related tasks and recurring interfaces favor agent-level experience reuse, and neither variant reliably improves over its corresponding ACE-style baseline. In a smaller-model experiment, the stateful variant also produces a substantially larger playbook and performs worse, suggesting a possible cost from uncurated prompt growth.

We next test whether ACE's agent-level adaptation benefit persists on Toolathlon, a harder and more heterogeneous tool-use benchmark, and find no consistent improvement over non-adapting agents. Because the experiment does not identify why, we develop an automated, evidence-grounded workflow and audit 74 frequently failed tasks using benchmark internals and concrete trajectories. All 74 selected reports contain benchmark-related issues, while 35 also contain supported agent issues; this failure-enriched subset does not estimate Toolathlon's overall defect rate. Instead, the audit shows that a failed score may not identify what an adapting agent should learn. Together, the studies identify recurring experience, usable adaptation state, and reliable failure attribution as potential bottlenecks for online self-improvement.