Traditional methods for off-road terrain estimation use two approaches. The first approach estimates terrain as it is traversed, while the second approach estimates terrain before it is traversed. The first approach accurately measures the terrain because it directly captures the effects of the terrain on the vehicle. However this approach cannot predict the terrain ahead of the vehicle. On the other hand, the second approach is able to predict the terrain ahead of the vehicle, but since it relies on indirect terrain features, this approach cannot accurately measure the effects of the terrain on the vehicle. This thesis proposes to combine the best of both approaches by developing two metrics for classifying roughness, an inertial measurement unit based roughness metric and a point to plane distance roughness metric. Images auto-labeled with these roughness metrics are then trained with deep learning network designed for semantic segmentation. The result is a system which was trained without the subjectivity of a human, and with the ability to predict the vehicle's response to the future terrain at a higher degree of accuracy than traditional systems.