Abstract:
The appearance of a scene arises from the interaction between surface reflectance (often approximated as albedo) and incident illumination (shading), and decomposing an image into these photometric intrinsics is a long-standing challenge in computer vision, with applications from relighting and retexturing to scene understanding. The problem is fundamentally ill-posed---a dark pixel may be dark paint under bright light or bright paint in shadow---and progress is further limited by the scarcity of ground-truth reflectance and shading for real-world scenes, which confines learning-based methods to synthetic data or sparse annotations that generalize poorly to natural environments.
This thesis introduces a physics-based approach to intrinsic image decomposition that replaces statistical priors with an additional physical measurement: a single thermal image. We leverage the principle of energy conservation---light not reflected from an opaque surface is absorbed, converted to heat, and observed by a thermal camera---so that visible and thermal images capture complementary halves of the same energy budget. From this principle we derive a theory relating the ordinalities (relative magnitudes) of visible and thermal image intensities at pairs of scene points to the ordinalities of their underlying albedo and shading. Applied to neighboring pixels, the theory classifies image edges as albedo- or shading-dominant; applied to arbitrary non-local point pairs, it yields dense ordinal constraints. These constraints form supervisory signals that optimize a randomly initialized neural network parameterization of albedo and shading, requiring no learned priors from training data.
Expert-annotation studies across diverse materials and natural scenes show high consistency with the estimated ordinalities from our theory. Quantitative evaluations on scenes with known reflectance and shading under natural and artificial lighting, together with qualitative comparisons on complex indoor and outdoor scenes, demonstrate superior performance over both physics-based and recent learning-based methods, and indicate that thermal imaging offers a potential route to curating real-world supervision for intrinsic image decomposition.
This thesis introduces a physics-based approach to intrinsic image decomposition that replaces statistical priors with an additional physical measurement: a single thermal image. We leverage the principle of energy conservation---light not reflected from an opaque surface is absorbed, converted to heat, and observed by a thermal camera---so that visible and thermal images capture complementary halves of the same energy budget. From this principle we derive a theory relating the ordinalities (relative magnitudes) of visible and thermal image intensities at pairs of scene points to the ordinalities of their underlying albedo and shading. Applied to neighboring pixels, the theory classifies image edges as albedo- or shading-dominant; applied to arbitrary non-local point pairs, it yields dense ordinal constraints. These constraints form supervisory signals that optimize a randomly initialized neural network parameterization of albedo and shading, requiring no learned priors from training data.
Expert-annotation studies across diverse materials and natural scenes show high consistency with the estimated ordinalities from our theory. Quantitative evaluations on scenes with known reflectance and shading under natural and artificial lighting, together with qualitative comparisons on complex indoor and outdoor scenes, demonstrate superior performance over both physics-based and recent learning-based methods, and indicate that thermal imaging offers a potential route to curating real-world supervision for intrinsic image decomposition.