Task-oriented grasping requires robots to reason not only about the geometry of objects but also about the function and semantics of their parts in context. While large language models (LLMs) offer a powerful source of commonsense knowledge, they are not grounded in physical geometry. This thesis explores how symbolic object representations can bridge this gap, enabling LLMs to guide grasp selection in a zero-shot setting. We first propose a method that decomposes objects into convex parts and represents them as symbolic graphs, allowing the LLM to assign semantic roles and select task-relevant grasps using only an object name and task description. Building on this, we introduce a 3D-aware approach that encodes object structure as semantic Constructive Solid Geometry (CSG) trees, which are optimized to match real-world observations. These symbolic representations enable the LLM to reason about part affordances directly in 2D or 3D space. Through extensive real-world experiments, we demonstrate that grounding language-based reasoning in structured geometry yields robust, interpretable, and generalizable task-oriented grasping, outperforming prior baselines across varied objects, viewpoints, and occlusions.