Abstract: Computer aided fine-grained classification of bone marrow cells is a significant task because manual morphological examination is time-consuming and highly dependent on the expert knowledge. Limited methods are proposed for the fine-grained classification of bone marrow cells. This can be partially attributed to challenges of insufficient data, high intra-class and low inter-class variances.In this work, we design a novel framework Attention-based Suppression and Attention-based Enhancement Net (ASAE-Net) to better distinguish different classes. Concretely, inspired by recent advances of weakly supervised learning, we develop an Attention-based Suppression and Attention-based Enhancement (ASAE) layer to capture subtle differences between cells. In ASAE layer, two parallel modules with no training parameters improve the discrimination in two different ways. Furthermore, we propose a Gradient-boosting Maximum-Minimum Cross Entropy (GMMCE) loss to reduce the confusion between subclasses. In order to decrease the intra-class variance, we adjust the hue in a simple way. In addition, we adopt a balanced sampler aiming to alleviate the issue of the data imbalance.Extensive experiments prove the effectiveness of our method. Our approach achieves favorable performance against other methods on our dataset.