Logistic Regression

The Mathematical Foundation of the Sigmoid

To understand Logistic Regression, we must first address why simple linear regression fails at classification. Linear regression predicts continuous values across the entire real number line, whereas classification requires bounded outputs representing probabilities between zero and one. We solve this by applying the logistic function, also known as the sigmoid function, which maps any real-valued number into the (0, 1) range. The formula is 1 / (1 + exp(-z)). By wrapping our linear combination of input features, z = w*x + b, inside this function, we transform unbounded linear predictions into a smooth 'S' curve. This probabilistic interpretation allows us to treat the output as the likelihood of a positive class, providing a robust mathematical basis for decision-making. We use this function because it is differentiable, which is essential for optimization techniques that adjust model weights to minimize errors efficiently.

import numpy as np

def sigmoid(z):
    # The sigmoid function forces outputs into the range (0, 1)
    return 1 / (1 + np.exp(-z))

# Example usage with linear combination z
z = np.array([-10, 0, 10])
print(sigmoid(z))  # Returns array close to [0, 0.5, 1]

The Log Loss Cost Function

Optimization in classification requires a cost function that penalizes predictions based on how far they deviate from the ground truth. We cannot use Mean Squared Error here because the sigmoid function creates a non-convex surface, leading to many local minima. Instead, we use Log Loss, or Binary Cross-Entropy. The logic is rooted in Maximum Likelihood Estimation: we want to maximize the probability of the observed labels given our parameters. When the actual label is 1, the cost is -log(predicted_probability); when the label is 0, it is -log(1 - predicted_probability). This mechanism heavily penalizes 'confident but wrong' predictions, forcing the model to adjust weights aggressively when it misclassifies an example. By minimizing this loss, we ensure the model converges on the parameters that best explain the training data, ultimately creating a reliable decision boundary for future unseen samples.

def log_loss(y_true, y_pred):
    # Penalize predictions heavily if they are wrong
    # Clip values to avoid log(0) errors
    y_pred = np.clip(y_pred, 1e-15, 1 - 1e-15)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Example with perfect and imperfect predictions
print(log_loss(np.array([1, 0]), np.array([0.9, 0.1]))) # Low loss

Gradient Descent for Optimization

Once the loss function is defined, we must find the optimal weights that minimize it. Because the log loss is a convex function, gradient descent is guaranteed to find the global minimum if the learning rate is chosen appropriately. We calculate the gradient of the cost function with respect to each weight, which essentially tells us the direction of steepest ascent. We then move in the opposite direction by a step size defined by our learning rate. The update rule involves taking the difference between the prediction and the actual target, then multiplying it by the feature input. This process is iterative; we perform these updates over multiple passes, known as epochs, across the entire dataset. Through this refinement, the linear boundary slowly rotates and translates until it provides the best possible separation between the two classes based on the available training inputs.

def update_weights(X, y, weights, lr):
    # Calculate predictions
    predictions = sigmoid(np.dot(X, weights))
    # Gradient is (X^T * (pred - y)) / n
    gradient = np.dot(X.T, (predictions - y)) / len(y)
    # Move against the gradient
    return weights - lr * gradient

# Weights represent the slope/intercept for features
weights = np.zeros(3) # e.g., bias + 2 features

Implementing a Classifier

Combining the sigmoid function, the log loss calculation, and the gradient descent loop results in a functional logistic regression classifier. We structure our data into a matrix where rows are samples and columns are features, typically adding a column of ones to handle the intercept term. During the training phase, we initialize weights randomly or to zero, then iterate through the update rule for a set number of cycles. After training, the 'decision' is made by checking if the probability is greater than a specific threshold, typically 0.5. This thresholding converts a continuous probability into a discrete label. Because the model relies on a linear combination of features, it is crucial that the input data is relatively well-scaled. If features have vastly different ranges, the gradient descent process may become unstable or converge extremely slowly due to the nature of the weight updates.

class LogisticRegression:
    def fit(self, X, y, iterations=1000, lr=0.1):
        self.w = np.zeros(X.shape[1])
        for _ in range(iterations):
            self.w = update_weights(X, y, self.w, lr)
    
    def predict(self, X):
        return (sigmoid(np.dot(X, self.w)) >= 0.5).astype(int)

Practical Limitations and Interpretation

While powerful, Logistic Regression assumes a linear relationship between input features and the log-odds of the outcome. If the decision boundary is non-linear—for instance, if the classes are nested like circles—this model will perform poorly without manual feature engineering. You can overcome this by creating interaction terms or polynomial features to capture non-linear patterns. Furthermore, the model is susceptible to outliers, as they can disproportionately influence the weights during the gradient calculation. Despite these constraints, the primary advantage is interpretability: the weight coefficients indicate exactly how much each feature influences the outcome probability. By examining these weights, you can understand the relative importance of different features in the decision-making process. This transparency makes Logistic Regression an ideal candidate for regulated industries where model explainability is a legal or operational requirement for stakeholders.

# Visualize feature importance by inspecting learned weights
model = LogisticRegression()
model.fit(X_train, y_train)
for i, val in enumerate(model.w):
    print(f"Feature {i} impact: {val}")
# Higher absolute value indicates higher influence on prediction

Interview questions

All Machine Learning interview questions →

Check yourself

Take the full Machine Learning quiz →