Ridge and Kernel Ridge Regression

The Big Picture

Ridge Regression and Kernel Ridge Regression are both used for regression, which means they predict a number.

Examples:

• Predict house price.
• Predict temperature.
• Predict received signal strength.
• Predict localization error in meters.
• Predict channel gain.

The important question is:

Given some input features, what numerical value should the model predict?

Ridge Regression starts from Linear Regression and makes it more stable. Kernel Ridge Regression starts from Ridge Regression and makes it able to learn curved, non-linear patterns.

Start from Linear Regression

Linear Regression assumes the output can be predicted by adding weighted input features.

For one feature:

For many features:

Where:

For example, in house-price prediction:

Training means finding the best weights so the predictions are close to the true values.

How Linear Regression Learns

The model makes predictions, compares them with the true values, and tries to reduce the error.

For ordinary Linear Regression, the most common objective is:

In words:

make the squared prediction errors as small as possible

This is called ordinary least squares.

The Problem with Ordinary Linear Regression

Linear Regression can become unstable when:

• The dataset is small.
• The dataset contains noise.
• Some features are strongly correlated.
• There are too many features compared with the number of samples.
• The model tries too hard to fit every training point.

When this happens, the model may learn very large weights.

Large weights often mean:

small change in input -> large change in prediction

That can make the model sensitive, unstable, and poor on new data.

What Ridge Regression Adds

Ridge Regression adds a penalty for large weights.

Ordinary Linear Regression:

Ridge Regression:

A common Ridge objective is:

Where:

The penalty Σⱼβⱼ² is called an L2 penalty.

Simple Ridge Intuition

Imagine two models:

Both might fit the training data, but Model B is risky because the weights are huge. Ridge prefers a model like A if both models make similar prediction errors.

So Ridge says:

fit the data, but do not let the weights become unnecessarily large

What Alpha Does

The alpha value controls how much Ridge cares about keeping weights small.

If alpha is too small, Ridge may still overfit. If alpha is too large, the model may become too simple.

Ridge Compared with Other Regressions

Ridge is still a linear model. It makes Linear Regression more stable, but it does not automatically learn complex curves unless the features already describe those curves.

Why Ridge Is Useful

Ridge is useful because it improves generalization.

Generalization means:

performing well on new data, not only on the training data

Ridge is a good first choice when:

• You need a strong baseline.
• You want interpretable coefficients.
• Your features are correlated.
• You have many features.
• You suspect ordinary Linear Regression is overfitting.

But What If the Pattern Is Not Linear?

Linear and Ridge Regression both try to fit a linear relationship in the original feature space.

But many real patterns are not linear.

Example:

x increases -> y first increases, then decreases

A straight line cannot model that well.

One solution is to create new features manually:

original feature: x
new features: x, x^2, x^3

Then a linear model can fit a curve:

This works, but manually creating all possible useful features can become difficult.

This is where kernels help.

What Is a Kernel?

A kernel is a function that measures similarity between two data points.

Simple idea:

kernel(point A, point B) = how similar A and B are

If two points are very similar, the kernel value is high. If they are very different, the kernel value is low.

For example, an RBF kernel behaves like this:

nearby points   -> high similarity
faraway points  -> low similarity

So instead of only asking:

what are the feature values?

a kernel method asks:

how similar is this new point to the training points?

Why Do We Use Kernels?

We use kernels because they allow simple algorithms to learn non-linear patterns.

The powerful idea is:

non-linear pattern in original space
        can become
linear pattern in a richer feature space

But we do not want to manually build that richer feature space. A kernel lets the model behave as if it is using many extra transformed features without explicitly creating them.

This is called the kernel trick.

Kernel Trick in Plain Language

Suppose the original data has only two features:

A richer feature space might include:

Manually calculating all of these can be expensive.

A kernel gives the model the effect of this richer comparison directly:

compare points in a richer space without explicitly building that space

That is why kernels are useful.

Common Kernels

Where:

How Kernel Ridge Regression Works

Kernel Ridge Regression combines two ideas:

Ridge Regression = control model complexity using regularization
Kernel method    = compare points through a similarity function

So Kernel Ridge Regression does this:

1. Measure similarity between training points using a kernel.
2. Learn how much each training point should influence predictions.
3. Regularize the solution so it does not overfit.
4. Predict a new value from similarities to the training data.

The prediction has this form:

Where:

The training solution can be written compactly as:

In simple words:

How the Kernel Helps

Suppose the true pattern is curved.

Plain Ridge tries to fit:

Kernel Ridge can fit:

The kernel helps by letting the model say:

points near this training example should have similar predictions
points far away should have less influence

This makes KRR useful for smooth non-linear regression problems.

Kernel Ridge Regression vs. KNN Regression

Kernel Ridge Regression and K-Nearest Neighbors can feel similar because both use the training samples during prediction. A new point is compared with stored training points, and nearby or similar points strongly influence the output.

KNN regression predicts by averaging the target values of the nearest neighbors:

Kernel Ridge Regression predicts using kernel similarities and learned influence weights:

They can perform similarly when:

• The target function is smooth.
• Nearby inputs usually have nearby output values.
• Features are scaled properly.
• The RBF kernel bandwidth and the KNN value of k create a similar neighborhood size.

For an RBF kernel:

Nearby points have high similarity, and faraway points have low similarity. This is close in spirit to KNN, where nearby points dominate and faraway points are ignored.

The difference is in how the influence is computed:

So the short intuition is:

KNN: nearby examples vote by their target values.
KRR: similar examples influence the prediction through a smooth, regularized kernel model.

Using training samples in either prediction formula is not data leakage. Leakage happens only if validation or test samples are included during fitting, scaling, tuning, or kernel-matrix construction before evaluation.

Ridge vs. Kernel Ridge Regression

Kernel Ridge vs. Support Vector Regression

Kernel Ridge Regression and Support Vector Regression can both use kernels, but they learn in different ways.

KRR can be easier to understand first because it is basically:

Ridge Regression + kernel similarity

Which One Should I Use?

Start simple. Try Ridge first. If residual plots show curved patterns, then try Kernel Ridge.

Practical Checklist

• Scale features before using Ridge or KRR.
• Tune alpha with cross-validation.
• For RBF KRR, tune both alpha and gamma.
• Use ordinary Ridge as a baseline before KRR.
• Use KRR carefully on large datasets because kernel matrices can become expensive.
• Check residual plots to see whether the model is missing non-linear structure.

Python: Ridge Regression

from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = fetch_california_housing(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = make_pipeline(
    StandardScaler(),
    Ridge(alpha=1.0)
)

model.fit(X_train, y_train)
pred = model.predict(X_test)

print("MAE:", round(mean_absolute_error(y_test, pred), 3))
print("R2:", round(r2_score(y_test, pred), 3))

Python: Tune Ridge Alpha

from sklearn.linear_model import RidgeCV

ridge_cv = make_pipeline(
    StandardScaler(),
    RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0, 100.0])
)

ridge_cv.fit(X_train, y_train)
pred = ridge_cv.predict(X_test)

print("MAE:", round(mean_absolute_error(y_test, pred), 3))
print("R2:", round(r2_score(y_test, pred), 3))

Python: Kernel Ridge Regression

from sklearn.kernel_ridge import KernelRidge
from sklearn.model_selection import GridSearchCV

krr = make_pipeline(
    StandardScaler(),
    GridSearchCV(
        KernelRidge(kernel="rbf"),
        param_grid={
            "alpha": [0.01, 0.1, 1.0, 10.0],
            "gamma": [0.01, 0.1, 1.0],
        },
        scoring="neg_mean_absolute_error",
        cv=5,
    )
)

krr.fit(X_train, y_train)
pred = krr.predict(X_test)

print("MAE:", round(mean_absolute_error(y_test, pred), 3))
print("R2:", round(r2_score(y_test, pred), 3))

Takeaway

Ridge Regression is Linear Regression with a stabilizing penalty. It is still linear, but it is often more reliable than ordinary Linear Regression when data is noisy or features are correlated.

Kernel Ridge Regression adds kernels, which let the model compare points by similarity and learn smooth non-linear patterns. The easiest way to remember it is:

Ridge Regression controls overfitting.
Kernels add non-linear similarity.
Kernel Ridge Regression combines both.

References and Further Reading

• A. E. Hoerl and R. W. Kennard, "Ridge Regression: Biased Estimation for Nonorthogonal Problems", Technometrics, vol. 12, no. 1, pp. 55-67, 1970.
• B. Scholkopf and A. J. Smola, Learning with Kernels, MIT Press, 2002.
• Scikit-learn documentation, "Ridge".
• Scikit-learn documentation, "KernelRidge".