Variable Normalization

Most optimization algorithms struggle when optimization variables spread over multiple orders of magnitude. This is important because the magnitude or range of the parameters can impact the optimization process, and the relative importance of each parameter can be difficult to determine without normalization. Normalizing parameters makes the optimization process more efficient and helps ensure that the results are more accurate and reliable. Additionally, normalization can prevent the optimization process from becoming biased towards certain parameters, which could lead to suboptimal or inefficient solutions.

CADET-Process provides several transformation methods which can help to soften these challenges.

In the following x will refer to the value exposed to the optimizer, whereas variable refers to the actual parameter value.

Linear Normalization

The linear normalization maps the variable space from the lower and upper bound to a range between \(0\) and \(1\) by applying the following transformation:

\[ x^\prime = \frac{x - x_{lb}}{x_{ub} - x_{lb}} \]

Show code cell content

Hide code cell content

from CADETProcess.optimization import OptimizationProblem
optimization_problem = OptimizationProblem('normalization_demo')

/home/docs/checkouts/readthedocs.org/user_builds/cadet-process/conda/latest/lib/python3.13/site-packages/ax/modelbridge/best_model_selector.py:45: FutureWarning: functools.partial will be a method descriptor in future Python versions; wrap it in enum.member() if you want to preserve the old behavior
  MEAN: Callable[[ARRAYLIKE], np.ndarray] = partial(np.mean)
/home/docs/checkouts/readthedocs.org/user_builds/cadet-process/conda/latest/lib/python3.13/site-packages/ax/modelbridge/best_model_selector.py:46: FutureWarning: functools.partial will be a method descriptor in future Python versions; wrap it in enum.member() if you want to preserve the old behavior
  MIN: Callable[[ARRAYLIKE], np.ndarray] = partial(np.min)
/home/docs/checkouts/readthedocs.org/user_builds/cadet-process/conda/latest/lib/python3.13/site-packages/ax/modelbridge/best_model_selector.py:47: FutureWarning: functools.partial will be a method descriptor in future Python versions; wrap it in enum.member() if you want to preserve the old behavior
  MAX: Callable[[ARRAYLIKE], np.ndarray] = partial(np.max)

optimization_problem.add_variable('var_norm_lin', lb=-100, ub=100, transform='linear')

OptimizationVariable(name=var_norm_lin, evaluation_objects=[], parameter_path=None, lb=-100, ub=100)

Log Normalization

The log normalization maps the variable space from the lower and upper bound to a range between \(0\) and \(1\) by applying the following transformation:

\[ x^\prime = \frac{log \left( \frac{x}{x_{lb}} \right) }{log \left( \frac{x_{ub} }{x_{lb}} \right) } \]

optimization_problem.add_variable('var_norm_log', lb=-100, ub=100, transform='log')

OptimizationVariable(name=var_norm_log, evaluation_objects=[], parameter_path=None, lb=-100, ub=100)

Auto Transform

This transform will automatically switch between a linear and a log transform if the ratio of upper and lower bounds is larger than some value (\(1000\) by default).

optimization_problem.add_variable('var_norm_auto', lb=-100, ub=100, transform='auto')

OptimizationVariable(name=var_norm_auto, evaluation_objects=[], parameter_path=None, lb=-100, ub=100)