The ability to estimate the motion of objects is crucial for robotic agents to act and interact in their environments. This is particularly challenging for non-rigid objects, as they exhibit complex dynamics with large degrees of freedom. This work presents a methodology to learn deformation models of 3D elastic objects based on graph networks. Our approach exploits physical reasoning by introducing soft constraints into the loss function during training. We evaluate the proposed method in a simulation framework developed for robotic manipulation tasks. Results show that such a physics-informed model is able to predict more accurate object motions with less data than a purely data-driven based model.