Supervised learning for high-level vision tasks has advanced significantly over the last decade. One of the primary driving forces for these improvements has been the availability of vast amounts of labeled data. However, annotating data is an expensive and time-consuming process. For example, densely segmenting a natural scene image takes approximately 30 minutes. This mode of supervised learning becomes a hurdle as we generalize to new tasks. In this thesis, we explore a few techniques to improve learning by using alternate modes of supervision. First, we explore how richly annotated ground view segmentation benchmarks may be used to improve the performance of aerial semantic segmentation. Next, we explore how image retrieval may be performed by learning universal representations that generalize well to new tasks. Lastly, we propose to model user behavior as implicit supervision for discovering the latent factors of variation in data to improve image retrieval. Our research suggests that, apart from improving learning algorithms and collecting more data, it is possible to learn better representations from alternative modes of supervision.