Anticipating the near-future actions of multiple people is a core capability for embodied systems that must plan, coordinate, and assist in shared, human-in-the-loop environments. Yet most action-anticipation research targets a single agent, overlooking the inter-agent dependencies---cooperation, coordination, and competition---that shape how groups behave. Naively extending single-agent models to multiple people, by processing each agent independently or fusing their predictions only at a late stage, fails to capture these relationships and yields limited gains.

This thesis presents InteractFormer, a unified model for multi-agent future prediction that treats inter-agent interaction as a first-class signal. Rather than reasoning over coarse, pre-extracted representations, InteractFormer operates directly on fine-grained visual tokens: an agent visual cross-attention module lets each agent attend to the others' observations at every time step, a lightweight bounding-box cross-attention injects explicit spatial configuration, and a temporal cross-attention module captures how these interactions evolve over time. The resulting design jointly predicts the futures of all agents in a manner that is spatially grounded, temporally coherent, and largely task-agnostic.

We evaluate InteractFormer on two complementary benchmarks: LEMMA, which features egocentric household collaboration, and SportsHHI, which features dense, high-frequency interactions in team sports. Across both domains, our method consistently improves over strong single- and multi-agent baselines on action anticipation and interaction understanding, with the largest gains in genuinely multi-agent scenarios. Attention visualizations and a controlled shuffled-agent ablation further confirm that the model exploits true inter-agent dependencies rather than incidental visual context, offering both accuracy and interpretability for collaborative behavior anticipation.