Data Acquisition for Domain Adaptation of Closed-Box Models

Machine learning (ML) marketplace provides customers with various ML solutions to accelerate their business. Models in the ML market are often available as closed boxes, but they may suffer from distribution shifts in new domains. Prior techniques cannot address this problem, because they are either impractical to use or against the property of closed-box models. Instead, we propose to acquire extra data to construct a "padding" model to help the original closed box with its classification weaknesses in the target domain. Our solution consists of a "weakness detector" to discover the deficiency of the original closed-box model and the Augmented Ensemble approach to combine the source and the padding model for better performance in the target domain and further diversifying the ML marketplace. Extensive experiments on several popular benchmark datasets confirm the superiority and robustness of our proposed framework over baseline approaches.