ENH: Add constrained optimization

kernc · 2020-11-17T13:03:23Z

Fixes #355
Fixes #700
Fixes #770
Fixes #108
Fixes #977
Probably fixes #730
Probably fixes #371
Probably fixes #457
Closes #836

A proposal for constrained optimization. An alternative to #836 that you might find less invasive and, hopefully, far better designed!

This allows, e.g., to only seek 2D solutions within a unit circle:

def constraint(params):
    x, y = params
    return x**2 + y**2 <= 1

# or

@skopt.util.use_named_args(dimensions)
def constraint(x, y):
    return x**2 + y**2 <= 1

...

Optimizer(..., space_constraint=constraint)

The issues with:

def objective(params):
    if invalid(params):
        return np.finfo(float).max  # i.e. np.inf
    return ...

are:

"if I set n_calls of the optimizer to 20, I want that to be 20 valid calls"
potentially heavily misguiding underlying estimators relative to objective values.

~~Will tidy-up/tests after gotten~~ 🆗 Apparently, this is the number one missing feature here, so feedback strongly appreciated!

schmoelder · 2020-12-11T10:12:57Z

I have a question regarding the upcoming implementation of constrained optimization.

If I understood correctly, it will simply check a callable whether a new individual is within the constraints and keep on drafting until that function returns 'True'. Is that correct?

Or will it also include a way of efficiently sampling higher dimensional polytopes (which might be necessary depending on the problem at hand).

Anyway, looking forward to the release. :)

kernc · 2020-12-11T11:48:01Z

simply check a callable whether a new individual is within the constraints and keep on drafting until that function returns 'True'. Is that correct?

With this implementation, as well as with the alternative abomination in #836, that's correct. That implementation also features a terminate condition. 🤔 Added in 1e4110b.

efficiently sampling higher dimensional polytopes

How would one go about that given an arbitrary constraint fun 8000 ction?

schmoelder · 2020-12-14T17:16:08Z

I'm no expert on this topic either, but some colleagues of mine have published something which might be interesting: https://academic.oup.com/bioinformatics/advance-article-abstract/doi/10.1093/bioinformatics/btaa872/5921168

This approach should work for linearly constrained problems. Not sure if it's applicable for arbitrarily/nonlinear constrained functions, though.

psmgeelen · 2021-04-07T18:10:37Z

Maybe this is a silly question, but when can we expect the merge?

DanyBIM · 2021-07-19T13:15:37Z

Hello !
Thank you for this update, this is definitely what I was looking for !
Just a comment since this code did not work immediately for me. It seems that the copy function of Optimizer object does not copy the constraint !

@DanyBIM

Thanks @DanyBIM

00sapo · 2021-10-01T09:24:54Z

Hello, will this branch be merged sooner or later?

00sapo · 2021-10-01T09:55:58Z

To me, this PR is not working with "grid" mode:

import numpy as np
import skopt
from skopt.space import Real
import matplotlib.pyplot as plt

space = [
    Real(1, 10, name='a'),
    Real(1, 10, name='b'),
]


@skopt.utils.use_named_args(space)
def objective(**hyperparams):
    print("--------------------")
    print("Testing hyperparams:")
    print(hyperparams)
    return np.random.rand()


@skopt.utils.use_named_args(space)
def constraint(**hyperparams):
    return hyperparams['b'] < hyperparams['a']


res = skopt.dummy_minimize(
    objective,
    dimensions=space,
    space_constraint=constraint,
    initial_point_generator='grid',
    n_calls=100)

x_iter = np.array(res['x_iters'])

plt.scatter(x_iter[:, 0], x_iter[:, 1])
plt.show()

I got:

--------------------
Testing hyperparams:
{'a': 8.363636363636363, 'b': 9.181818181818182}
Traceback (most recent call last):
  File "/home/sapo/Develop/test_skopt/test.py", line 25, in <module>
    res = skopt.dummy_minimize(
  File "/home/sapo/Develop/test_skopt/.venv/lib/python3.9/site-packages/skopt/optimizer/dummy.py", line 11
9, in dummy_minimize
    return base_minimize(func, dimensions, base_estimator="dummy",
  File "/home/sapo/Develop/test_skopt/.venv/lib/python3.9/site-packages/skopt/optimizer/base.py", line 304
, in base_minimize
    result = optimizer.tell(next_x, next_y)
  File "/home/sapo/Develop/test_skopt/.venv/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", lin
e 496, in tell
    check_x_in_space(x, self.space)
  File "/home/sapo/Develop/test_skopt/.venv/lib/python3.9/site-packages/skopt/utils.py", line 195, in chec
k_x_in_space
    raise ValueError("Point (%s) is not within the bounds of"
ValueError: Point ([8.363636363636363, 9.181818181818182]) is not within the bounds of the space ([(1, 10)
, (1, 10)]).

qup20 · 2022-10-09T14:31:23Z

Hello，When will this PR be merged into the master branch？
Thank you！

purbeshmitra · 2023-07-06T01:26:45Z

Is there any chance that this will be added to the master branch?

SaeednHT · 2023-08-08T16:23:18Z

To me, this PR is not working with "grid" mode:

import numpy as np
import skopt
from skopt.space import Real
import matplotlib.pyplot as plt

space = [
    Real(1, 10, name='a'),
    Real(1, 10, name='b'),
]


@skopt.utils.use_named_args(space)
def objective(**hyperparams):
    print("--------------------")
    print("Testing hyperparams:")
    print(hyperparams)
    return np.random.rand()


@skopt.utils.use_named_args(space)
def constraint(**hyperparams):
    return hyperparams['b'] < hyperparams['a']


res = skopt.dummy_minimize(
    objective,
    dimensions=space,
    space_constraint=constraint,
    initial_point_generator='grid',
    n_calls=100)

x_iter = np.array(res['x_iters'])

plt.scatter(x_iter[:, 0], x_iter[:, 1])
plt.show()

I got:

--------------------
Testing hyperparams:
{'a': 8.363636363636363, 'b': 9.181818181818182}
Traceback (most recent call last):
  File "/home/sapo/Develop/test_skopt/test.py", line 25, in <module>
    res = skopt.dummy_minimize(
  File "/home/sapo/Develop/test_skopt/.venv/lib/python3.9/site-packages/skopt/optimizer/dummy.py", line 11
9, in dummy_minimize
    return base_minimize(func, dimensions, base_estimator="dummy",
  File "/home/sapo/Develop/test_skopt/.venv/lib/python3.9/site-packages/skopt/optimizer/base.py", line 304
, in base_minimize
    result = optimizer.tell(next_x, next_y)
  File "/home/sapo/Develop/test_skopt/.venv/lib/python3.9/site-packages/skopt/optimizer/optimizer.py", lin
e 496, in tell
    check_x_in_space(x, self.space)
  File "/home/sapo/Develop/test_skopt/.venv/lib/python3.9/site-packages/skopt/utils.py", line 195, in chec
k_x_in_space
    raise ValueError("Point (%s) is not within the bounds of"
ValueError: Point ([8.363636363636363, 9.181818181818182]) is not within the bounds of the space ([(1, 10)
, (1, 10)]).

You can use initial_point_generator="grid_modified" with constrained skopt which is available in the following link:
https://github.com/SaeednHT/scikit-optimize-constrained

AdamCoxson · 2023-12-19T18:27:41Z

@SaeednHT

Brilliant. skopt_modcn was exactly what I needed to constrain my neural network configurations :) Thank you!

I had to install:

pip install -i https://test.pypi.org/simple/ skopt-modcn==0.0.1
conda install -c conda-forge pydoe

For anyone who wants to see how I used it:

bounds = [                               # A quick set for debug
  Categorical([50,80,100,120,140,150,180,200], name = 'h1'),
  Categorical([10, 20, 50,80,100,120,140,150, 180, 200], name = 'h2'),
  Categorical([0,50,80,100,120,140,150,180,200], name = 'h3'),
  Categorical([0,50,80,100,120,140,150,180,200], name = 'h4'),
  Categorical([0,50,80,100,120,140,150,180,200], name = 'h5'),
  Integer(5, 20,  name='epochs'), 
 Categorical([50,100,150,200,300,400], name = 'batch'),
  Real(0.00001,0.02,"log-uniform",  name='lr')]

@use_named_args(bounds)
def neural_net_bayesian_eval(h1, h2, h3, h4, h5, epochs, batch, lr):
  #Some neural network training 
    return error
        
 @use_named_args(bounds)
 def layer_constraint(h1, h2, h3, h4, h5, epochs, batch, lr):
      # neuron number must always decrease for increasing layer number
      neurons=[h1,h2,h3,h4,h5]
      valid=True
      for i in range(len(neurons)-1):
          if neurons[i] < neurons[i+1]: valid=False
      return valid

gp_output = gp_minimize(neural_net_bayesian_eval,                      
                     dimensions=bounds,             
                     space_constraint=layer_constraint, # valid neuron config
                     acq_func="gp_hedge",                 
                     n_calls = num_evaluations,     
                     n_initial_points = random_vals,                      
                     random_state=1234,
                     n_jobs=4,
                     verbose=True )

SaeednHT · 2023-12-19T19:31:20Z

@SaeednHT

Brilliant. skopt_modcn was exactly what I needed to constrain my neural network configurations :) Thank you!

I had to install:

pip install -i https://test.pypi.org/simple/ skopt-modcn==0.0.1
conda install -c conda-forge pydoe

For anyone who wants to see how I used it:

bounds = [                               # A quick set for debug
  Categorical([50,80,100,120,140,150,180,200], name = 'h1'),
  Categorical([10, 20, 50,80,100,120,140,150, 180, 200], name = 'h2'),
  Categorical([0,50,80,100,120,140,150,180,200], name = 'h3'),
  Categorical([0,50,80,100,120,140,150,180,200], name = 'h4'),
  Categorical([0,50,80,100,120,140,150,180,200], name = 'h5'),
  Integer(5, 20,  name='epochs'), 
 Categorical([50,100,150,200,300,400], name = 'batch'),
  Real(0.00001,0.02,"log-uniform",  name='lr')]

@use_named_args(bounds)
def neural_net_bayesian_eval(h1, h2, h3, h4, h5, epochs, batch, lr):
  #Some neural network training 
    return error
        
 @use_named_args(bounds)
 def layer_constraint(h1, h2, h3, h4, h5, epochs, batch, lr):
      # neuron number must always decrease for increasing layer number
      neurons=[h1,h2,h3,h4,h5]
      valid=True
      for i in range(len(neurons)-1):
          if neurons[i] < neurons[i+1]: valid=False
      return valid

gp_output = gp_minimize(neural_net_bayesian_eval,                      
                     dimensions=bounds,             
                     space_constraint=layer_constraint, # valid neuron config
                     acq_func="gp_hedge",                 
                     n_calls = num_evaluations,     
                     n_initial_points = random_vals,                      
                     random_state=1234,
                     n_jobs=4,
                     verbose=True )

I am glad to know this could help. Thank you for providing an example.

ENH: Add space constraint

ae3978e

kernc changed the title ~~ENH: Add space constraint~~ ENH: Add constrained optimization Nov 17, 2020

kernc added 6 commits November 18, 2020 23:11

Add API/docstrings to *_minimize functions

4da808a

Validate user input

11632c9

Copy over existing constraint function from passed Space object

97dc603

Apply constraint function in Space.__contains__

e02b5d7

Add tests (test_space, test_common)

cbee3a5

Set acq_optimizer to 'sampling' when space is constrained

08e8888

kernc marked this pull request as ready for review November 18, 2020 22:50

kernc mentioned this pull request Nov 19, 2020

Model-based optimization with scikit-optimize kernc/backtesting.py#154

Merged

kernc mentioned this pull request Dec 23, 2020

Variable parameter bound based on other parameter #977

Open

Raise RuntimeError on unsatisfiable constraint

1e4110b

kernc mentioned this pull request Jan 22, 2021

New maintainers #847

Open

Carry over space.contraint in Optimizer.copy()

dfe87b0

Thanks @DanyBIM

QuentinSoubeyran mentioned this pull request Oct 1, 2021

[Request] Make new releases compatible with modern scikit-learn #1071

Open

11 tasks

taebow mentioned this pull request Aug 6, 2022

Hyperopt search space constraints freqtrade/freqtrade#7189

Closed

xmatthias mentioned this pull request Sep 5, 2022

Proposed way to keep the RSI symmetric freqtrade/freqtrade#7345

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ENH: Add constrained optimization #971

ENH: Add constrained optimization #971

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ENH: Add constrained optimization #971

Are you sure you want to change the base?

ENH: Add constrained optimization #971

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!