We propose an automatic method to measure depression severity from body movement dynamics in participants undergoing treatment for depression. Participants in a clinical trial for treatment of depression were interviewed on up to four occasions at 7-week intervals with the clinician-administered Hamilton Rating Scale for Depression. Body movement was tracked using OpenPose from full-body video recordings of the interviews. Gram matrices formulation was used for body shape and trajectory representations from each video interview. Kinematic features were extracted and encoded for video based representation using Gaussian Mixture Models (GMM) and Fisher vector encoding. A multi-class SVM was used to classify the encoded body movement dynamics into three levels of depression severity: severe, mild, and remission. Accuracy was high for severe depression (68.57%) followed by mild depression (56%), and then remission (37.93%). The obtained results suggest that automatic detection of depression severity from body movement is feasible.