Abstract: Facial action units (AUs) recognition is a multi-label clas- sification problem, where regular spatial and temporal patterns exist in AU labels due to facial anatomy and human’s behavior habits. Ex- ploiting AU correlation is beneficial for obtaining robust AU detector or reducing the dependency of a large amount of AU-labeled samples. Several related works have been done to apply AU correlation to model’s objective function or the extracted features. However, this may not be optimal as all the AUs still share the same backbone network, requir- ing to update the model as a whole. In this work, we present a novel AU Relation Embedded deep model (RE-Net) for AU detection that applies the AU correlation to the model’s parameter space. Specifically, we format the multi-label AU detection problem as a domain adaptation task and propose a model that contains both shared and AU specific pa- rameters, where the shared parameters are used by all the AUs, and the AU specific parameters are owned by individual AU. The AU relationship based regularization is applied to the AU specific parameters. Extensive experiments on three public benchmarks demonstrate that our method outperforms the previous work and achieves state-of-the-art performance on both AU detection task and AU intensity estimation task.