Abstract: Wi-Fi-based gesture recognition technology has broad application prospects in smart medical, smart home, industrial production, game interaction, etc. However, in practical applications, the accuracy of Wi-Fi-based gesture recognition will be significantly reduced when a gesture recognition model trained on a dataset from original users (source domain) has been applied to a new user (target domain). In order to solve this problem, a domain adaptive gesture recognition scheme for Wi-Fi is proposed. First, a new lightweight convolutional neural network is proposed for pre-training the source domain. Then, a new domain adaptive network is designed for unsupervised transfer learning which uses the correlation alignment loss to align the second-order statistics of the source and target domain depth features as well as use center loss to improve the discriminability of features to realize aggregate within classes and scattered between classes. Experiments demonstrate that the scheme proposed in this paper works well for recognizing the hand gesture actions of new users. When the user changed, the scheme proposed in this paper improves the action recognition average accurate rate from 62.7% to 90.2%, which can significantly enhance the robustness of Wi-Fi gesture recognition across users.