Inductive biases have proven effective and often essential in the design of performant deep
learning systems. This thesis presents two contributions that target distinct facets of how
inductive biases can improve modern deep models. In Chapter 1, we focus on controlling
the diversity–likelihood trade-off at inference-time of generative models. In Chapter 2, we
study the architectural inductive bias for 3D perception for multi-view transformer models.
Chapter 1 presents Temporal Score Rescaling (TSR), a mechanism to steer the
sampling diversity of denoising diffusion and flow matching models, allowing users to sample
from a sharper or broader distribution than the training distribution. We build on the
observation that these models leverage (learned) score functions of noisy data distributions
for sampling, and show that rescaling these allows one to effectively control a ‘local’ sampling
temperature. Notably, this approach does not require any finetuning or alterations to training
strategy, and can be applied to any off-the-shelf model. We validate our framework across five
disparate tasks — image generation, pose estimation, depth prediction, robot manipulation,
and protein design. Across these tasks, our approach allows sampling from sharper (or
flatter) distributions and yields consistent performance gains.
Chapter 2 presents RayRoPE, where we study positional encodings for multi-view trans-
formers that process tokens from a set of posed input images, and seeks a mechanism that
encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity,
and can adapt to the geometry of the underlying 3D scene. We find that prior encoding
schemes for multi-view attention do not meet these desiderata. RayRoPE represents patch
positions based on associated rays and computes query-frame projective coordinates to en-
sure SE(3) invariance. To adapt to scene geometry, RayRoPE predicts a per-token depth
to obtain its position along the corresponding ray, while also modeling uncertainty and ana-
lytically computing the expected positional encoding. We validate our method on the tasks
of novel-view synthesis and stereo depth estimation. While remaining efficient, RayRoPE
consistently improves over alternate position encoding schemes.
Together, these two contributions illustrate complementary roles of inductive bias in
modern deep learning: temperature sampling at inference time and positional encoding for
geometric structure. Both demonstrate that carefully chosen inductive biases can meaning-
fully improve model behavior without altering the underlying training pipeline.