Abstract: Deep learning methods, especially multi-task learning with CNNs, have achieved good results in many fields of computer vision. Semantic segmentation and shape detection of lumbar vertebrae, sacrum, and femoral heads from clinical X-ray images are important and challenging tasks. In this paper, we propose a multi-task deep neural network, MBNet. It is developed based on our new multi-path convolutional neural network, BiLuNet, for semantic segmentation on X-ray images. Our MBNet has two branches, one is for semantic segmentation of lumbar vertebrae, sacrum, and femoral heads. It shares the main features with the second branch to learn and classify by supervised learning. The output of the second branch is to predict the inspected values for lumbar vertebra inspection. These networks are capable of performing the two tasks with very limited training data. We collect our dataset and annotated it by doctors for model training and performance evaluation. Compared to the state-of-the-art methods, our BiLuNet model provides better mIoUs with the same training data. The experimental results have demonstrated the feasibility of our MBNet for semantic segmentation of lumbar vertebrae, as well as the parameter prediction for the doctors to perform clinical diagnosis of low back pains.Code is available at https://github.com/LuanTran07/BiLUnet-Lumbar-Spine.