Support Vector Regression

In this post I cover the fundamentals of Support Vector Regression.
A high level summary is that an SVR model is regularised regression using the epsilon-insensistive loss function. The SVR objective can then be reformulated to use kernels.

Main references:

• PRML by Christoper Bishop - Section 7

Epsilon-Inensistive Loss Function

In standard linear regression we use the squared error loss, which can be motivated by making the probabilistic assumption that the outcome variables are normally distributed conditionally given the predictors.

A way to view support vector regression (SVR) is by introducing the $\mathit{ϵ}$-insensistive loss function.
This is defined below:

$$\begin{array}{r}{E}_{ϵ}(y-g(\mathbf{x};\mathbf{w}))=\{\begin{array}{ll}0,& |g(\mathbf{x};\mathbf{w})-y|<ϵ\\ |g(\mathbf{x};\mathbf{w})-y|-ϵ,& \text{otherwise}\end{array}\end{array}$$

In words, this loss function only punishes incorrect predictions when the discrepancy between the actual value and the predicted value is larger than some choice of threshold $ϵ$.

$g(\mathbf{x};\mathbf{w})$ represents the affine function $g(\mathbf{x};\mathbf{w}):=⟨{\mathbf{w}}^{\prime },\mathbf{x}⟩+w_0$, that is used to construct our decision rule.

Regularised Regression

The objective function for SVR can be defined as regularised regression using the $\mathit{ϵ}$-insensistive loss function, where the penalty term uses the squared $L_2$ norm.

The aim is to minimise this objective, with respect to the vector $\mathbf{w}:=({\mathbf{w}}^{\prime },w_0)$

$$\sum _{i=1}^n{E}_{ϵ}(y_i-g({\mathbf{x}}_i;\mathbf{w}))+\lambda ||{\mathbf{w}}^{\prime }|{|}^2$$

Introducing Slack Variables

Minimising the objective function given above, can be rewritten as a constrained optimisation problem by introducing slack variables $\xi$, $\hat{\xi }$.

This is given by:

$$\begin{array}{r}\underset{\mathit{\xi },\hat{\mathit{\xi }},\mathbf{w}}{\mathrm{argmin}}\sum _{i=1}^n({\xi }_i+\widehat{{\xi }_i})+\lambda ||{\mathbf{w}}^{\prime }|{|}^2\\ \text{subject to:}\\ y_i-g({\mathbf{x}}_i;\mathbf{w})\le ϵ+{\xi }_i\\ g({\mathbf{x}}_i;\mathbf{w})-y_i\le ϵ+{\hat{\xi }}_i\\ {\xi }_i,{\hat{\xi }}_i\ge 0\end{array}$$

Often a hyperparameter $C$ is added in front of the $\sum _{i=1}^n({\xi }_i+\widehat{{\xi }_i})$ term, called the inverse regularization parameter, smaller values of $C$ put less weight on the minimisation of the errors.

Including the hyperparameter $C$ and the setting the regularization term to $\frac{1}{2}$, gives the standard formulation of the SVR objective function:

SVR Standard Formulation:

$$\begin{array}{r}\underset{\mathit{\xi },\hat{\mathit{\xi }},\mathbf{w}}{min}C\sum _{i=1}^n({\xi }_i+\widehat{{\xi }_i})+\frac{1}{2}||{\mathbf{w}}^{\prime }|{|}^2\\ \text{subject to:}\\ \text{(1)}& y_i-g({\mathbf{x}}_i;\mathbf{w})\le ϵ+{\xi }_{i}g({\mathbf{x}}_i;\mathbf{w})-y_i\le ϵ+{\hat{\xi }}_i\\ {\xi }_i,{\hat{\xi }}_i\ge 0\end{array}$$

Dual Formulation

In the SVM post, I gave the dual form of the soft margin classifier and then showed that this allows us to implement the Kernel trick. The Kernel trick is a way of applying a transformation to our features, but avoiding its direct computation. Therefore allowing us to transform to very high dimensional spaces, even infinite dimensional spaces.

We can carry out the analogous process for SVR.

The primal problem is given above (SVR Standard Formulation), the dual formulation of SVR is then given by:

$$\begin{array}{rl}\underset{\mathbf{a},\hat{\mathbf{a}}}{min}& \frac{1}{2}(\mathbf{a}-\hat{\mathbf{a}}{)}^{T}K(\mathbf{a}-\hat{\mathbf{a}})+ϵ\sum _{i=1}^N(a_i+\widehat{a_i})+\sum _{i=1}^N{y}_i(a_i-\widehat{a_i})\\ & \text{subject to:}\\ & 0\le a_i,{\hat{a}}_i\le C\text{for}i=1,\dots ,N\\ & \sum _{i=1}^N(a_i-{\hat{a}}_i)=0\end{array}$$

Where the matrix $K$ is constructed from the kernel function: $K_{ij}:=k({\mathbf{x}}_i,{\mathbf{x}}_j):=\varphi ({\mathbf{x}}_i{)}^T\varphi ({\mathbf{x}}_j)$

Proof

The primal problem is given as:

$$\begin{array}{r}\underset{\mathit{\xi },\hat{\mathit{\xi }},{\mathbf{w}}^{\prime }}{min}C\sum _{i=1}^n({\xi }_i+\widehat{{\xi }_i})+\frac{1}{2}||{\mathbf{w}}^{\prime }|{|}^2\\ \text{s.t}{y}_i-g({\mathbf{x}}_i;\mathbf{w})\le ϵ+{\xi }_i\\ g({\mathbf{x}}_i;\mathbf{w})-y_i\le ϵ+{\hat{\xi }}_i\\ {\xi }_i,{\hat{\xi }}_i\ge 0\end{array}$$

This primal problem is a convex (quadratic) problem, hence we will have strong duality if Slater’s condition holds. Slater’s condition requires that there exists a set of feasible points such that the inequality constraints are strictly satisfied. In fact, there is a weaker condition that just requires existence of a feasible point, if all the inequality constraints are affine.

Slater’s condition holds for the primal problem, as for any choice of $ϵ$ we can choose large enough values of ${\xi }_i$ and ${\hat{\xi }}_i$ such that the inequality constraints are satisfied for each point. Therefore, the optimal solution value of the primal objective will be equal to the dual optimal solution.

Introducing the Lagrange multipliers $a_i⩾0$, $a_i⩾0$, ${\mu }_i⩾0$, and ${\hat{\mu }}_i⩾0$, the Lagrangian for the primal is given by:

$$\begin{array}{rl}& L(\mathit{\xi },\hat{\mathit{\xi }},\mathbf{w},\mathbf{a},\hat{\mathbf{a}},\mathit{\mu },\hat{\mathit{\mu }}):=\\ & C\sum _{i=1}^N({\xi }_i+\widehat{{\xi }_i})+\frac{1}{2}||{\mathbf{w}}^{\prime }|{|}^2-\sum _{i=1}^N({\mu }_i{\xi }_i+{\hat{\mu }}_i\widehat{{\xi }_i})-\sum _{i=1}^N{a}_i(ϵ+{\xi }_i+g({\mathbf{x}}_i;\mathbf{w})-y_i)-\sum _{i=1}^N{\hat{a}}_i(ϵ+{\hat{\xi }}_i+y_i-g({\mathbf{x}}_i;\mathbf{w}))\end{array}$$

We now let the linear model take the form $g(\mathbf{x};\mathbf{w})=⟨{\mathbf{w}}^{\prime },\varphi (\mathbf{x})⟩+w_0$, where we have included a feature transform. Now to find the minimum with respect to $(\mathit{\xi },\hat{\mathit{\xi }},\mathbf{w})$ for constant Lagrange multipliers, we set the derivatives of the Lagrangian with respect to $\mathbf{w}:=({\mathbf{w}}^{\prime },w_0)$, ${\xi }_i$, and ${\hat{\xi }}_i$ to zero:

$$\begin{array}{}\text{(2)}& & \frac{\partial L}{\partial {\mathbf{w}}^{\prime }}=0\Rightarrow {\mathbf{w}}^{\prime }=\sum _{i=1}^N(a_i-{\hat{a}}_i)\varphi ({\mathbf{x}}_i)\text{(3)}& & \frac{\partial L}{\partial w_0}=0\Rightarrow \sum _{i=1}^N(a_i-{\hat{a}}_i)=0\text{(4)}& & \frac{\partial L}{\partial {\xi }_i}=0\Rightarrow a_i+{\mu }_i=C\text{(5)}& & \frac{\partial L}{\partial {\hat{\xi }}_i}=0\Rightarrow {\hat{a}}_i+{\hat{\mu }}_i=C\end{array}$$

Using these equations and substituting into the Lagrangian, we get that dual problem is given by maximising the Lagrange dual function below.

$$\begin{array}{r}\tilde{L}(\mathbf{a},\hat{\mathbf{a}})=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N(a_i-{\hat{a}}_i)(a_j-{\hat{a}}_j)k({\mathbf{x}}_i,{\mathbf{x}}_j)-ϵ\sum _{i=1}^N(a_i+{\hat{a}}_i)+\sum _{i=1}^N(a_i-{\hat{a}}_i)y_i\end{array}$$

Where $k(\mathbf{x},{\mathbf{x}}^{\prime })=\varphi (\mathbf{x}{)}^T\varphi ({\mathbf{x}}^{\prime })$

We have the constraints on the Lagrange multipliers that $a_i⩾0$, $a_i⩾0$, ${\mu }_i⩾0$, and ${\hat{\mu }}_i⩾0$. The Lagrange multipliers ${\mu }_i⩾0$, and ${\hat{\mu }}_i⩾0$ do not appear in the dual objective, but from equations $\text{(4)}$, $\text{(5)}$, we have the following constraints (often called box constraints):

$$\begin{array}{}\text{(6)}& 0⩽a_i⩽C\text{(7)}& 0⩽{\hat{a}}_i⩽C\end{array}$$

---

Also note that substituting the equation ${\mathbf{w}}^{\prime }=\sum _{n=1}^N(a_i-{\hat{a}}_i)\varphi ({\mathbf{x}}_i)$ $\text{(2)}$ into $g(\mathbf{x};\mathbf{w})=⟨{\mathbf{w}}^{\prime },\varphi (\mathbf{x})⟩+w_0$, we get that the decision function can be rewritten in terms of the kernel function:

$$\begin{array}{}\text{(8)}& g(\mathbf{x})=\sum _{i=1}^N(a_i-{\hat{a}}_i)k(\mathbf{x},{\mathbf{x}}_i)+w_0\end{array}$$

Interpreting the KKT Conditions

If we have primal and dual optimal points they must satisfy the KKT conditions. The KKT conditions often allow us to interpret hor

The KKT conditions for the SVR problem are:

$$\begin{array}{r}{a}_i(y_i-g({\mathbf{x}}_i;\mathbf{w})-ϵ-{\xi }_i)=0\\ {\hat{a}}_i(g({\mathbf{x}}_i;\mathbf{w})-y_i-ϵ-{\hat{\xi }}_i)=0\\ {\mu }_i\cdot {\xi }_i=0\\ {\hat{\mu }}_i\cdot {\hat{\xi }}_i=0\end{array}$$

For the first two equations, if $a_i$ is non-zero this implies that the value of the corresponding prediction function $g({\mathbf{x}}_i;\mathbf{w})$ either lies on the borders of the $ϵ$ tube or above.
The reason being, is that if $a_i$ is non-zero this implies $y_i-g({\mathbf{x}}_i;\mathbf{w})-ϵ-{\xi }_i=0$. Then there are two scenarios where this holds, either $y_i-g({\mathbf{x}}_i;\mathbf{w})=ϵ$ which implies ${\xi }_i=0$, or if $y_i-g({\mathbf{x}}_i;\mathbf{w})>ϵ$ then ${\xi }_i=y_i-g({\mathbf{x}}_i;\mathbf{w})-ϵ$.

Another observation, is that if we add the equations $y_i-g({\mathbf{x}}_i;\mathbf{w})-ϵ-{\xi }_i=0$ and $g({\mathbf{x}}_i;\mathbf{w})-y_i-ϵ-{\hat{\xi }}_i=0$. We get that the two equations are incompatible, therefore it is not possible to have $a_i$ and $\widehat{a_i}$ as non-zero simultaenously. Note that from the prediction function $\text{(8)}$, only data points where either $a_i$ or ${\hat{a}}_i$ is nonzero contributes to the prediction. As mentioned previously these are data points that lie on the border or outside of the $ϵ$-tube, known as the support vectors.

For the last two equations, we can replace them with $(C-a_i)\xi =0$ and $(C-{\hat{a}}_i)\hat{\xi }=0$ respectively, using equations $\text{(4)},\text{(5)}$.

Solving in Practice

The dual problem is a quadratic convex problem, and there exist many libraries to solve these type of problems. For SVMs and SVRs there is the LIBSVM library written mainly in C/C++, this document describes the algorithm used for $ϵ$-SVR. LIBSVM is implemented in other open source languages, for example the R package e1071 is an interface for LIBSVM.

Pros and Cons

• The objective function has hyperparameters $ϵ$ and $C$ which requires tuning in order to choose one. Additionaly if we implement Kernels this will introduce further hyperparameters, e.g $\sigma$ for the RBF-kernel.