Tuning Big Data Systems Via Deep Learning

Abstract

Modern database systems, including IBM Db2 have numerous parameters, “knobs,” that require precise configuration to achieve optimal workload performance. Even for experts, manually “tuning” these knobs is a challenging process. We present Db2une, an automatic query-aware tuning system that leverages deep learning to maximize performance while minimizing resource usage. Db2une uses a specialized transformer-based query-embedding pipeline and graph neural networks to feed as input to a stability-oriented deep reinforcement learning model. In Db2une, we introduce a multi-phased, database meta-data driven training approach—which incorporates cost estimates, interpolation of these costs, and database statistics—to efficiently discover optimal tuning configurations without the need to execute queries. Thus, our model scales to large workloads where executing queries repeatedly would be prohibitively expensive. Through experimental evaluation, we demonstrate Db2une’s efficiency and effectiveness over a variety of workloads. We show that Db2une provides recommendations surpassing those of other state-of-the-art systems and IBM experts.