Abstract: Facial makeup transfer aims to render a non-makeup face image in an arbitrary given makeup one while preserving face identity. The most advanced method separates makeup style information from face images to realize makeup transfer. However, makeup style includes several semantic clear local styles which are still entangled together. In this paper, we propose a novel unified adversarial disentangling network to further decompose face images into four independent components, i.e., personal identity, lips makeup style, eyes makeup style and face makeup style. Owing to the disentangled makeup representation, our method can not only flexible control the degree of local makeup styles, but also can transfer local makeup styles from different images into the final result, which any other approaches fail to handle. For makeup removal, different from other methods which regard makeup removal as the reverse process of makeup transfer, we integrate the makeup transfer with the makeup removal into one uniform framework and obtain multiple makeup removal results. Extensive experiments have demonstrated that our approach can produce visually pleasant and accurate makeup transfer results compared to the state-of-the-art methods.