Building 3D models is an important step to perform more autonomous and dexterous manipulation on deformable objects. Current techniques for modeling deformable objects are inflexible and suffer from discrepancies with actual behaviors when assumptions on their material are not properly fulfilled, or when objects can only be observed from limited viewpoints. Deformable object models require close integration between computer vision, deep learning and robotics. In this work, a framework for modeling 3D deformable objects from multi-view images is presented to support robotic manipulation.