Setting up model parameters

You can manually set the matching function y^=f(x) (called a predictive or forecast function) that would produce predictions of the response y^​based on known features, but this is often difficult to do due to the diversity of objects and complex dependencies between features and response. Therefore, in supervised machine learning, the correspondence between features and response is sought in a certain class of functions.fw(x), parameterized by a vector of parameters $w$, which are selected based on a training sample consisting of N objects:

(x1,y1),(x2,y2),...(xN,yN).

The model parameters will also be called model weights .

For example, a class of functions might be the set of all constant predictions:

or consist of all linear functions of the features:

There are also more complex families of functions, which will be discussed in the following chapters.

To select the best one from a family of functions (which is equivalent to selecting a specific vector of parameters w^), it is necessary to numerically formalize which forecasts we will consider good and which bad. For this, a loss function is defined, L(y, y^), depending on the true value of the response$y$and predictedy^The higher the value of the loss function, the worse the forecast is considered.

Basic loss functions in a regression problem

Let’s consider the main loss functions in a regression problem:

A smooth combination of both loss functions, called the Huber function (smooth L1 loss), is also widely used:

{0.5⋅x2,β⋅( ∣ x ∣–0.5⋅β ) ,​if  ∣ x ∣<β ,if  ∣ x ∣≥β ,​​

In the latter caseβ>0– a hyperparameter that specifies the region of a quadratic dependence that extends linearly beyond this region, preserving the continuity of the function itself and the continuity of its derivative.

Being smooth, this function is convenient for optimization, but at the same time is as robust to atypical observations as the error module.

Example of losses for classification

For a classification problem, the simplest loss function is the error indicator, calculated using the formula$I$, where I {condition} is the indicator function. I {condition} returns 1 if the condition is met and 0 otherwise. Looking ahead, we should note that in practice, this function cannot be used for tuning models, since it is not differentiable. Therefore, other loss functions are used for classification, which will be discussed below.

Theoretical and empirical risk

To configure the model parameters, traditionally, one wants to minimize the expected losses on new objects coming from some probability distribution, called theoretical risk:

In practice, this value is uncalculable due to the fact that we do not have information about the theoretical distribution of objects.p(x,y) but we only know a limited training sample(x1,y1),(x2,y2),...(xN,yN). Therefore, in practice, the parameters are found by minimizing the empirical risk L(w), which is a sample estimate of the theoretical risk based on the training sample:

Parameter estimationw^is defined as the empirical risk minimizer:

WhereX∈R^N×D– the “object-feature” matrix, also called the feature matrix. The rows of this matrix correspond to D-dimensional feature vectors for each of N objects of the training sample, andY∈R^N– the vector of responses that we want to learn to predict based on the features.

A model with tuned parameters is also called a forecasting algorithm.

Model parameters should not be confused with hyperparameters , which are not configured based on the training sample, but are specified by the user or selected based on a separate sample, called the validation sample.

Explore More IT Terms