Recent advances in Large Language Models and Vision-Language Models have enabled agents that interact with graphical user interfaces and web environments on behalf of users. Although these agents show promising capabilities, they remain unreliable in complex interactive settings. A key reason is that existing evaluations often emphasize high-level outcomes, such as task success or next-action accuracy, without diagnosing the underlying capabilities that cause agents to succeed or fail. As a result, important failure modes in perception, grounding, temporal reasoning, and planning can remain hidden behind aggregate metrics.

This thesis argues that building reliable GUI and web agents requires fine-grained diagnostic evaluation of core agentic capabilities. We study this problem through two complementary directions. First, we examine planning in web agents by evaluating whether Vision-Language Models understand the temporal structure of web interaction and can assess whether a sequence of actions achieves a given goal. Our analysis shows that current models struggle with fundamental planning prerequisites, including ordering webpage states, reasoning about state changes, and selecting effective plans. Second, we investigate grounding in GUI agents by testing whether models consistently identify the same UI element across diverse natural-language descriptions. Focusing on desktop environments, we introduce a sensitivity-based evaluation that reveals substantial variation in model predictions when the same target is referred to from different perspectives.

Together, these studies demonstrate that models can perform well under standard benchmarks while still lacking crucial fine-grained capabilities needed for reliable interaction. By decomposing agent evaluation into targeted diagnostic tests, this thesis provides a more precise understanding of current limitations in GUI and web agents. The proposed evaluations and benchmarks offer tools for identifying failure modes, guiding model development, and advancing toward more robust and trustworthy interactive AI systems.