Linear Regression vs Logistic Regression
Linear regression predicts a continuous number; logistic regression predicts the probability of a class. Despite its name, logistic regression is a classification algorithm: it passes the same weighted sum of features through a sigmoid function, squashing the output into a probability between 0 and 1, and draws a decision boundary where that probability crosses a threshold. Linear regression outputs the weighted sum directly, unbounded in both directions, which is exactly what you want for prices and temperatures and exactly wrong for probabilities.
The two models share more machinery than their names suggest. Both compute a linear score from the features, both produce a linear decision structure, a fitted line or plane for linear regression and a straight-line boundary for logistic regression, and both are trained by minimizing a loss. The losses differ for good reason: linear regression minimizes squared error, which has a closed-form least-squares solution, while logistic regression minimizes log loss, also called cross-entropy, which has no closed form and is fit iteratively with gradient-based methods.
Why not just run linear regression on 0 and 1 labels and threshold the output? Because squared error is the wrong objective for classification: predictions escape the 0 to 1 range, and correctly classified but extreme points pull the line toward themselves, shifting the boundary and degrading accuracy. Log loss penalizes confident wrong answers steeply and leaves the boundary where the probabilities say it belongs. The shared linear core is also why both models are among the most interpretable in machine learning, one coefficient per feature.
Side by side
Task
Regression; predicts a continuous value such as a price, temperature, or demand level.
Classification; predicts the probability that an example belongs to a class, then thresholds it into a label.
Output range
Unbounded; the raw weighted sum can be any real number, positive or negative.
Bounded between 0 and 1; the sigmoid squashes the same weighted sum into a valid probability.
Core equation
y equals the weighted sum of features plus an intercept, read off directly as the prediction.
The probability equals the sigmoid of that same weighted sum, so the linear score becomes log-odds instead of the output itself.
Loss function
Mean squared error; heavily penalizes large residuals and yields the classic least-squares fit.
Log loss, also called cross-entropy; steeply penalizes confident wrong probability estimates.
How it is fit
Closed-form ordinary least squares solves it in one step, or gradient descent for very large data.
No closed form exists; it is fit iteratively with gradient descent or Newton-style solvers, though the loss is convex so the optimum is unique.
Geometric picture
A best-fit line or plane through the data cloud, minimizing vertical squared distances.
An S-shaped probability surface over the features, whose 0.5 contour is a straight decision boundary.
Interpreting coefficients
Each coefficient is the change in the predicted value per unit change in that feature, holding others fixed.
Each coefficient is the change in log-odds per unit change in the feature; exponentiating gives an odds ratio.
Key assumptions
A roughly linear relationship between features and target, with independent errors of constant variance for valid inference.
A linear relationship between features and the log-odds of the class, and independent observations.
When it fails
Strongly nonlinear relationships, heavy outliers that squared error chases, and extrapolation far beyond the training range.
Classes that are not linearly separable in the given features, and perfectly separable data, where coefficients diverge without regularization.
Typical use cases
Price and demand forecasting, trend estimation, and quantifying how much each factor moves a continuous outcome.
Spam detection, churn and default prediction, medical risk scores, and any yes-or-no decision needing calibrated probabilities.
| Dimension | Linear Regression | Logistic Regression |
|---|---|---|
| Task | Regression; predicts a continuous value such as a price, temperature, or demand level. | Classification; predicts the probability that an example belongs to a class, then thresholds it into a label. |
| Output range | Unbounded; the raw weighted sum can be any real number, positive or negative. | Bounded between 0 and 1; the sigmoid squashes the same weighted sum into a valid probability. |
| Core equation | y equals the weighted sum of features plus an intercept, read off directly as the prediction. | The probability equals the sigmoid of that same weighted sum, so the linear score becomes log-odds instead of the output itself. |
| Loss function | Mean squared error; heavily penalizes large residuals and yields the classic least-squares fit. | Log loss, also called cross-entropy; steeply penalizes confident wrong probability estimates. |
| How it is fit | Closed-form ordinary least squares solves it in one step, or gradient descent for very large data. | No closed form exists; it is fit iteratively with gradient descent or Newton-style solvers, though the loss is convex so the optimum is unique. |
| Geometric picture | A best-fit line or plane through the data cloud, minimizing vertical squared distances. | An S-shaped probability surface over the features, whose 0.5 contour is a straight decision boundary. |
| Interpreting coefficients | Each coefficient is the change in the predicted value per unit change in that feature, holding others fixed. | Each coefficient is the change in log-odds per unit change in the feature; exponentiating gives an odds ratio. |
| Key assumptions | A roughly linear relationship between features and target, with independent errors of constant variance for valid inference. | A linear relationship between features and the log-odds of the class, and independent observations. |
| When it fails | Strongly nonlinear relationships, heavy outliers that squared error chases, and extrapolation far beyond the training range. | Classes that are not linearly separable in the given features, and perfectly separable data, where coefficients diverge without regularization. |
| Typical use cases | Price and demand forecasting, trend estimation, and quantifying how much each factor moves a continuous outcome. | Spam detection, churn and default prediction, medical risk scores, and any yes-or-no decision needing calibrated probabilities. |
When to use Linear Regression
- Your target is a continuous quantity, a price, a duration, a temperature, and you need a numeric prediction.
- You need to quantify effect sizes plainly: how many units the outcome moves per unit of each feature.
- You want a fast, closed-form baseline before trying anything nonlinear, to see how far a straight line gets you.
- The relationship looks roughly linear in a scatter plot, or becomes linear after a log or polynomial transform.
- You need classical statistical inference, confidence intervals and hypothesis tests on coefficients, from a well-understood model.
When to use Logistic Regression
- Your target is a category, spam or not, churn or stay, and you need class probabilities rather than raw scores.
- Calibrated probabilities matter downstream, for example ranking customers by risk or setting a decision threshold from costs.
- You want an interpretable classifier whose coefficients translate into odds ratios stakeholders can audit.
- You need a strong, fast classification baseline that resists overfitting on small or wide datasets, especially with L1 or L2 regularization.
- The classes are approximately linearly separable in your features, or become so after sensible feature engineering.
The bottom line
The choice is made for you by the target variable: continuous outcome means linear regression, categorical outcome means logistic regression, and running linear regression on 0/1 labels is the one tempting shortcut you should refuse, since squared error distorts the decision boundary and produces out-of-range probabilities. The real decision is whether a linear model is enough at all. Fit the appropriate one first, because both are fast, stable, and interpretable, and treat its performance as the bar any nonlinear model must clearly beat. If a tuned random forest or gradient boosting model only edges out logistic regression by a hair, the simpler model's interpretability usually wins the tiebreak.