Abstract: We study the problem of reconstructing the template-aligned mesh for human body estimation from unstructured point cloud data. Recent studies of the shape matching problem using DNN methodologies have shown state-of-the-art results with generic point-wise architectures, but in so doing exploit much weaker human shape and surface priors in the inference than previous methods with explicit shape surface models. Since they are bound to improve the performance even more, we investigate the impact of adding back such constraints by proposing a new dedicated human template matching process with a point-based deep-autoencoder architecture, where surface consistency of surface points is enforced and parameterized with a specialized Gaussian Process layer, and whose global consistency and generalization abilities are enforced with adversarial training. The choice of these elements is grounded in a detailed review of failure cases in standard datasets SURREAL and FAUST. We validate and evaluate the impact of these components on this data with measured improvement over state of the art DNN methods, which also show through a leap in the visual quality of the results.