Supervised learning

Features, responses, and training set

Machine learning works with so-called objects. In the spam classification task, the objects are emails, and in the travel time prediction task, the objects are the starting and ending points of the route and environmental information that influences the travel time.

Each object is described by a pair(x,y), Where

• x– input information that we know about the object;
• y– the output information that we want to predict for the object x.

In this case, for ease of processing, the input information is in most cases encoded by some vector of fixed length x∈R^D, where each element of this vector is called a feature, and the entire vector$x$– a feature vector. The feature number will henceforth be denoted by a superscript:x=[x1,x2,...xD].

Output information y is called the response or target variable.

In the most typical situation, we know a labeled sample of N objects:

(x1,y1),(x2,y2),...(xN,yN)

This problem is called a supervised learning problem because the model can use the correct “teacher” labeling for a set of N objects for customizing their parameters. Teacher-based tagging is obtained either through manual tagging by subject matter experts or by logging input and output data (which we want to predict based on future inputs) in advance. For example, we can log which emails a user has voluntarily marked as spam, so that we can predict their preferences in the future.

Types of supervised learning problems

Depending on the type of response, supervised learning problems are divided into the following categories:

• Regression: The response is a number y∈R. You will become familiar with various regression models of this type in later articles of the tutorial.
  • Examples: predicting travel time along a route; the focal length of a camera to sharpen faces; or the price of a stock the next day.
• Vector regression: the response is immediately a vector of real answers y∈RM.  The vector regression problem can be solved using individual regression algorithms. But it’s more effective to build a forecast for all M predictions immediately, which can be achieved, for example, with the help of neural networks, which are the subject of the second part of the tutorial – ” deep learning “.
  • Examples: we forecast the future price of not just one stock, but several at once; when forecasting the weather, we simultaneously predict temperature, humidity, pressure, and wind speed.
• Ranking: the response takes on real relevance valuesy∈R; however, in actual use, it is not the absolute response values ​​that are important, but the relative ones, because the final result is an ordering of objects by degree of relevance.
  • Examples: in an information search, rank relevant documents or products in a store based on a user’s search query.
• Classification: the response takes one of C discrete values, y ∈ {1, 2,…, C}. A special case of classification is binary classification, when there are only two classes. In this case, one of them is called positive, and the other is negative, andy∈{+1,- 1}. Classifiers are described in general terms in a separate article of the tutorial. In subsequent chapters, you will also become familiar with the main classification models.
  • Examples of binary classification include determining whether a person is sick or healthy based on test results; determining whether an email is spam or a useful message; predicting whether a client with certain characteristics will repay a loan to a bank.
  • Examples of multi-class classification: identifying a person from a photo, diagnosing a disease based on symptoms; classifying news by topic (sports/politics/economics/technology/culture).
• Labeling: Similar to classification, but an object can belong to several classes or none.
  • Examples: automatic hashtags for images; assignment of text to thematic categories that may be present in it simultaneously.

An example of a regression problem

Below is an example of a regression problem in which a real response on the Y axis must be predicted based on a one-dimensional feature on the X axis. The training sample is indicated by orange dots, for which the dependence must be reconstructed: y^ = f (x) to be able to predict the target value y for any new object x. As we can see, this can be done in various ways with varying forecast errors.

Example of a classification problem

Below is an example of a classification problem in which each object is described by two featuresx1x1Andx2x2, denoted by the X and Y axes. Each object in the training set is represented by a point on the graph. The target value for prediction is one of three classes, each of which is shown in a different color. Using these points, it is necessary to reconstruct the general pattern of correlation between any point (x1,x2) and ( x1,x2)one of the classes.

Special formulations of the supervised learning problem

If, in addition to the training objects, the feature descriptions are known in advance N + 1​,… xN + M​ .For test objects for which a future prediction must be made, this task is called transductive learning. Additional knowledge about the test objects allows for a more precise model tuning specifically for these objects.

There are situations when not only the target variables are known for the training sample objects, but also explanations (privileged information) of why the response is exactly as it is. Consider, for example, a medical classification task in which a diagnosis must be made for patients with given characteristics (such as gender, age, history of doctor visits, general condition, current complaints). In this task, the labeling (the doctor’s final diagnosis) may contain additional explanations.$z$, characterizing the doctor’s comments and why they made a particular diagnosis. In this case, the training set consists of triplets (input features, response, explanation):

(x1,y1,z1),(x2,y2,z2),...(xN,yN,zN)

It is necessary to build a model that, as before, will predict the response based on the input featuresy^=f(x) already for new objects, but for more precise tuning, the model can also use explanations z$z$, available in the training set. Model tuning in this case is called learning using privileged information.

Explore More IT Terms