[Optuna v4.5] Gaussian Process-Based Sampler (GPSampler) Can Now Perform Constrained Multi-Objective Optimization
Introduction
This article introduces the constrained multi-objective optimization feature of GPSampler released in Optuna v4.5.
Gaussian process-based Bayesian optimization (GPBO) has gained popularity in various domains, including materials science and machine learning. The initial release of GPSampler aimed to provide a simple and user-friendly implementation of GPBO. While its component customization, such as kernels and acquisition functions, is less flexible than that of BoTorch and Ax (maintained by Meta Platforms, Inc.), GPSampler runs faster with fewer dependencies.
At the same time, the first version of GPSampler supported only limited problem setups, motivating further development. Optuna v4.2 extended GPSampler to handle inequality constraints, as announced in this article. Optuna v4.4 then supported the multi-objective optimization. With the latest release, Optuna v4.5 now supports constrained multi-objective optimization.
The new feature is useful in many domains, including materials science, machine learning, and drug discovery, where multiple objectives must be balanced under realistic physical or experimental constraints. Our constrained multi-objective GPBO in Optuna v4.5 directly offers this capability, with the potential to accelerate scientific progress across a wide range of domains.
In this article, we explain its usage, provide a brief technical overview, and present benchmarking results against samplers available in Optuna.
How to Use GPSampler in Optuna
GPSampler requires additional dependencies on top of optuna:
$ pip install optuna==4.5.0
# scipy and torch are also necessary for GPSampler.
$ pip install scipy
$ pip install torch --index-url https://download.pytorch.org/whl/cpuOnce the installation is done, you can easily use GPSampler by passing it to your study:
import optuna
def objective(trial: optuna.Trial) -> tuple[float, float]:
x = trial.suggest_float("x", -5.0, 5.0)
y = trial.suggest_float("y", -5.0, 5.0)
c = x**2 + y**2 - 4.0
trial.set_user_attr("c", c)
return x**2 + y**2, (x - 2) ** 2 + (y - 2) ** 2
def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float]:
c = trial.user_attrs["c"]
return (c,)
sampler = optuna.samplers.GPSampler(constraints_func=constraints)
study = optuna.create_study(sampler=sampler, directions=["minimize"] * 2)
study.optimize(objective, n_trials=100)Multi-Objective Optimization with GPSampler
The acquisition function of our implementation is the logarithm of the Expected Hypervolume Improvement (log EHVI), which essentially relies on Eq. (1) of Daulton et al. [1] and Algorithm 2 of Lacour et al [6]. Technical details are available in the following PRs:
- #6039: Implement the box decomposition algorithm to speed up the hypervolume improvement calculations.
- #6052: Implement the quasi Monte-Carlo-based log EHVI.
- #6069: Add support for multi-objective optimization in
GPSampler.
As discussed in the PR, the box decomposition algorithm improves the time complexity of log EHVI, resulting in a 6–800 times speedup over the naive approach and making our implementation practical up to four-objective problems. Note that although our implementation mostly aligns with the BoTorch implementation, some details rely on different logic. For further information, please refer to this PR.
Constrained Multi-Objective Optimization with GPSampler
On top of the multi-objective optimization described above, we extend GPSampler to handle inequality constraints following the approaches of J. Gardner et al. [3] and M. Gelbart et al. [4]. In this setting, GP models are employed for both the objective functions and the constraint functions, with correlations between them being ignored. See the following PRs for more details:
- #6198: Implement the constrained log EHVI.
- #6224: Add support for constrained multi-objective optimization in
GPSampler.
In our implementation, the feasibility of a candidate point is evaluated using the Probability of Improvement (PI). The overall acquisition function is constructed as the product of the EHVI and the feasibility probabilities derived from PI.
Performance Benchmarking
We conduct benchmarking to see the effectiveness of constrained multi-objective optimization by GPSampler. Specifically, we compare this new feature against: (i) multi-objective GPSampler without constraint handling (hereafter referred to as unconstrained GPSampler) to assess the benefit of incorporating constraints; and (ii) TPESampler and NSGAIISampler in Optuna, that support constrained multi-objective optimization.
Benchmarking Setup
Benchmarking Problem. We use the C2-DTLZ2 problem [5], newly added to OptunaHub for constrained multi-objective optimization benchmarking. See the OptunaHub documentation and the corresponding PR for more details, and refer to the original papers [2, 6] for the detailed definitions of the objective and constraint functions. We set the number of objectives to two and the number of variables to three.
Implementation Details. We run each optimization for 300 trials and repeat it five times with different random seeds to evaluate the standard error. Since the C2-DTLZ2 problem is deterministic, i.e., no noise is included in the objective functions, we use deterministic_objective=True for GPSampler. See this PR for the implementation we used.
Machine Specifications. The benchmarks are conducted on a computer running Arch Linux with Intel® Core™ i9–14900HX processor (24 cores, 32 threads, up to 5.8GHz) and Python 3.11.0.
Results and Discussion
Analysis of Obtained Pareto Fronts and the Effects of Constraint Consideration
Figure 1 illustrates the obtained Pareto fronts to highlight the distribution and quality of the solutions.
GPSampler (shown in (a)) effectively reduces wasted evaluations (colored gray) in infeasible regions compared to unconstrained GPSampler (shown in (b)) and TPESampler (shown in (c)), while NSGAIISampler (shown in (d)) remains far from convergence after 300 trials.Effectiveness of Constraint Handling in GPSampler
The C2-DTLZ2 problem directly introduces infeasible regions on the Pareto front to evaluate the algorithm’s ability to handle disconnected Pareto fronts [5]. As a result, while it is possible to obtain Pareto solutions without considering the constraints, such attempts often include wasted evaluations in infeasible regions, making constrained optimization particularly important.
Comparing the constrained GPSampler (Figure 1(a)) with the unconstrained GPSampler (Figure 1(b)), the former yields fewer infeasible observations (colored gray) between feasible Pareto solutions (colored blue), showing that the constrained one successfully captures the constraints and reduces wasted evaluations.
Comparison with Other Samplers
Compared to TPESampler (Figure 1(c)), GPSampler (Figure 1(a)) also generates fewer wasted evaluations in infeasible regions. At the same time, the Pareto front close to where one of the objective values approaches zero is more thoroughly covered by TPESampler. This is because, owing to the nature of GP-based acquisition functions, the optimization tends to oversample near the boundaries, which reduces variance in those regions and makes further exploration of the adjacent Pareto front more difficult.
Furthermore, as shown in Figure 1(d), NSGAIISampler has not yet converged after 300 trials, with few optimal solutions achieved, highlighting the superior performance of GPSampler.
Analysis of Feasible Hypervolume History
Figure 2 shows the feasible hypervolume history, enabling us to assess how quickly and effectively each sampler improves performance under the given constraints.
GPSampler (blue) achieves faster convergence to higher hypervolume values compared to TPESampler (pink) and NSGAIISampler (yellow), demonstrating the effectiveness of the new feature in reducing evaluation cost.As shown in Figure 2, the hypervolume history of GPSampler reaches the final performance of TPESampler with fewer than 50 trials and surpasses NSGAIISampler much earlier, demonstrating its superior sample efficiency. The rapid convergence to the best hypervolume is particularly important for computationally expensive objectives, since reducing function evaluations directly leads to substantial overall time savings.
Conclusion
We extended GPSampler to support constrained multi-objective optimization in Optuna v4.5. Benchmarks on the C2-DTLZ2 problem [5] confirmed that the new feature reduces wasted evaluations in infeasible regions and achieves faster convergence in feasible hypervolume compared to TPESampler and NSGAIISampler. This new functionality marked a major step forward for Optuna, making it a strong candidate for black-box optimization of computationally expensive objectives. Please give it a try!
References
[1] Daulton, S., Balandat, M., and Bakshy, E. Differentiable expected hypervolume improvement for parallel multi-objective Bayesian optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 9851–9864, 2020.
[2] Deb, K., Thiele, L., Laumanns, M., and Zitzler, E. Scalable test problems for evolutionary multi-objective optimization. In Evolutionary Multi-objective Optimization, pp. 105–145. Springer-Verlag, 2005.
[3] Gardner, J., Kusner, M., Zhixiang, W., Weinberger, K., and Cunningham, J. Bayesian Optimization with Inequality Constraints. In Proceedings of the 31st International Conference on Machine Learning, volume 32, pp. 937–945. PMLR, 2014.
[4] Gelbart, M. A., Snoek, J., and Adams, R. P. In Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, pp. 250–259. AUAI Press, 2014.
[5] Jain, H., and Deb, K. An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point Based Nondominated Sorting Approach, Part II: Handling Constraints and Extending to an Adaptive Approach. In IEEE Transactions on Evolutionary Computation, 18(4):602–622, 2014.
[6] Lacour, R., Klamroth, K., and Fonseca, CM. A box decomposition algorithm to compute the hypervolume indicator. Computers & Operations Research, volume 79, pp. 347–360, 2017.
