Contact-rich manipulation requires reasoning about the complex dynamics at the interface between interacting objects. These interactions generate structure-borne signals that encode critical information about contact, slip, and material behavior, but leveraging such information in robotics remains challenging due to limitations in existing sensing and modeling approaches. Acoustic sensing provides a direct way to capture these signals, offering new opportunities for understanding physical interaction. This thesis investigates how acoustic sensing can be used to model and represent contact in increasingly complex settings.

The first part of this thesis utilizes acoustic sensing as a binary signal for object slip during non-prehensile transport with a tray. By using piezoelectric microphones to capture vibration patterns associated with slip, we learn a motion-conditioned friction model that enables an optimization-based motion planner to adapt inertial constraints online. This approach minimizes object slip, demonstrating that acoustic signals can augment analytical models by capturing real-world effects that are difficult to model explicitly.

The second part focuses on learning structured representations of slip for in-hand manipulation from acoustic signals. We develop a multi-channel acoustic sensing system embedded in a parallel-jaw gripper, with a learned model that estimates continuous slip direction and magnitude in real time from spatially distributed piezoelectric microphone signals. This formulation enables closed-loop control and highlights the importance of spatially structured sensing in resolving ambiguities in contact-rich interaction.

The final part extends this representation learning approach to human-object interaction, where perception becomes more challenging due to high degrees of freedom, occlusion, and distributed contact. We introduce a visuo-acoustic system that integrates wearable active acoustic sensing with vision to estimate contact on hand meshes. By learning a cross-modal representation that fuses acoustic and visual features, the system improves the accuracy and robustness of contact estimation, particularly when visual observations are incomplete or ambiguous. This demonstrates how acoustic sensing complements existing perception modalities in more complex settings.

Together, these contributions demonstrate that acoustic sensing provides a practical foundation for understanding contact-rich physical interaction. Using structure-borne vibrations, this work enables more robust modeling and representation of contact, leading to improved perception and control in robotic manipulation.