Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMs
Richard Cornelius SuwandiAbstract
Context-Aware Kernel Evolution (CAKE) enhances Bayesian optimization by using large language models to adaptively generate and refine Gaussian process kernels, outperforming traditional methods across various tasks.
The efficiency of Bayesian optimization (BO) relies heavily on the choice of the Gaussian process (GP) kernel, which plays a central role in balancing exploration and exploitation under limited evaluation budgets. Traditional BO methods often rely on fixed or heuristic kernel selection strategies, which can result in slow convergence or suboptimal solutions when the chosen kernel is poorly suited to the underlying objective function. To address this limitation, we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO with large language models (LLMs). Concretely, CAKE leverages LLMs as the crossover and mutation operators to adaptively generate and refine GP kernels based on the observed data throughout the optimization process. To maximize the power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to select the most effective kernel through balancing the model fit measured by the Bayesian information criterion (BIC) with the expected improvement at each iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO method consistently outperforms established baselines across a range of real-world tasks, including hyperparameter optimization, controller tuning, and photonic chip design. Our code is publicly available at https://github.com/cake4bo/cake.
Community
✨ Bayesian optimization (BO) relies heavily on the choice of Gaussian process (GP) kernel, a critical component that encodes structural assumptions like smoothness or periodicity. Fixed kernels, commonly used in practice, often mismatch the true underlying function, leading to slow convergence and poor sample efficiency, especially when evaluations are costly and data is scarce.
🍰 We propose Context-Aware Kernel Evolution (CAKE), a novel framework that leverages large language models (LLMs) as intelligent genetic operators to adaptively evolve GP kernels during optimization. Starting from a population of base kernels, CAKE iteratively scores, mutates, and recombines kernels using LLM-driven operations, guided by a fitness function that balances model fit and complexity. Over time, this yields a context-aware set of kernels that continuously adapt as more observations are collected.
🛠️ Beyond BO, CAKE offers a general-purpose approach to adaptive kernel design, applicable to SVMs, kernel PCA, metric learning, and more — anywhere kernels encode assumptions that should evolve with context.
💻 Code is available at: https://github.com/richardcsuwandi/cake
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- BOOST: Bayesian Optimization with Optimal Kernel and Acquisition Function Selection Technique (2025)
- ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization (2025)
- Learn to Explore: Meta NAS via Bayesian Optimization Guided Graph Generation (2025)
- Enhancing Trust-Region Bayesian Optimization via Newton Methods (2025)
- Scalable Neural Network-based Blackbox Optimization (2025)
- NOSTRA: A noise-resilient and sparse data framework for trust region based multi objective Bayesian optimization (2025)
- ROOT: Rethinking Offline Optimization as Distributional Translation via Probabilistic Bridge (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper