Machine learning interviews test your understanding of core concepts — bias/variance, evaluation metrics, model selection — rather than any one framework.
Supervised learning trains a model on labeled data, where each input comes with a known output, so the model learns to map inputs to those outputs—think predicting house prices or classifying emails as spam. Unsupervised learning works with unlabeled data and tries to find structure on its own, like clustering customers into segments or reducing dimensionality to visualize patterns. Reinforcement learning is different from both: an agent interacts with an environment, takes actions, and learns from rewards or penalties over time rather than from a fixed dataset. The key distinction is the feedback signal—supervised learning gets explicit correct answers, unsupervised learning gets none, and reinforcement learning gets delayed, evaluative feedback in the form of reward. A good example that blends the first two is semi-supervised learning, which uses a small amount of labeled data alongside a much larger unlabeled set.
Overfitting happens when a model learns the noise and idiosyncrasies of the training data rather than the underlying pattern, so it performs well on training data but poorly on unseen data. Underfitting is the opposite problem: the model is too simple to capture the real pattern, so it performs poorly on both training and test data. You detect overfitting by watching the gap between training and validation loss—if training loss keeps dropping while validation loss rises or plateaus, that's a red flag. You detect underfitting when both training and validation performance are poor and don't improve much even with more data. To fix overfitting you can add regularization, gather more data, simplify the model, or use dropout and early stopping, while fixing underfitting usually means increasing model capacity, adding more relevant features, or training longer.
Bias is the error introduced by approximating a real-world problem with a simplified model, and high bias usually shows up as underfitting because the model is too rigid to capture the true relationship. Variance is the error introduced by the model's sensitivity to small fluctuations in the training data, and high variance usually shows up as overfitting because the model changes drastically depending on which training samples it happens to see. The tradeoff is that reducing bias, for example by making a model more flexible, tends to increase variance, and vice versa, so you can't minimize both indefinitely with the same lever. The goal is to find the sweet spot that minimizes total generalization error, which decomposes into bias squared, variance, and irreducible noise. In practice you navigate this tradeoff using techniques like regularization, ensembling, and cross-validation to tune model complexity.
We split data into training, validation, and test sets so we can get an honest estimate of how a model will perform on data it hasn't seen. The training set fits the model's parameters, the validation set is used to tune hyperparameters and make model selection decisions, and the test set is held out until the very end to give an unbiased final performance estimate. If you tune decisions using the test set, you leak information and your performance estimate becomes overly optimistic. Cross-validation, most commonly k-fold, addresses the problem of limited data by splitting the training data into k folds, training on k-1 of them and validating on the remaining fold, then rotating through all folds and averaging the results. This gives a more robust estimate of model performance than a single train-validation split, especially when data is scarce, though it costs more compute since you're effectively training k models instead of one.
Both L1 and L2 regularization add a penalty term to the loss function to discourage overly complex models and reduce overfitting. L1, or Lasso, adds the sum of the absolute values of the weights, and because of the geometry of that penalty it tends to push some weights all the way to zero, effectively performing feature selection. L2, or Ridge, adds the sum of the squared weights, which shrinks weights toward zero smoothly but rarely to exactly zero, so it keeps all features but reduces their influence. L1 is useful when you suspect many features are irrelevant and you want a sparse, interpretable model, while L2 is useful when most features contribute somewhat and you just want to control their magnitude. You can also combine both in what's called elastic net when you want a balance of sparsity and smooth shrinkage.
A confusion matrix breaks predictions down into true positives, true negatives, false positives, and false negatives, giving you a full picture of where a classifier is right and wrong rather than a single aggregate number. Precision is the fraction of predicted positives that are actually positive, which matters when false positives are costly, like flagging a legitimate transaction as fraud. Recall is the fraction of actual positives the model successfully catches, which matters when false negatives are costly, like missing a cancer diagnosis. F1 score is the harmonic mean of precision and recall, giving you a single number that balances both when neither can be ignored. Accuracy becomes misleading on imbalanced datasets—if only 1% of transactions are fraudulent, a model that always predicts 'not fraud' gets 99% accuracy while being completely useless, so precision, recall, and F1 are usually far more informative in that setting.
Gradient descent is an optimization algorithm that minimizes a loss function by iteratively updating model parameters in the direction opposite to the gradient of the loss with respect to those parameters. The learning rate controls the size of each update step—too large and the algorithm can overshoot the minimum or diverge entirely, too small and training becomes painfully slow or gets stuck on a shallow plateau. Variants like stochastic gradient descent update parameters using a single sample or small batch at a time rather than the full dataset, which speeds up training and adds useful noise that can help escape poor local minima. Adaptive methods like Adam adjust the effective learning rate per parameter based on historical gradients, which often makes training more stable and less sensitive to the initial learning rate choice. In practice, people commonly use learning rate schedules or warmup strategies to get the benefits of a larger rate early in training and finer convergence later on.
Feature engineering is the process of creating, transforming, or selecting input variables so a model can learn the underlying pattern more effectively, and it often matters more to final performance than the choice of algorithm itself. Examples include creating interaction terms, extracting date parts like day-of-week from a timestamp, encoding categorical variables, or aggregating transaction history into summary statistics. Feature scaling, like standardization or min-max normalization, matters because many algorithms are sensitive to input scale—gradient descent converges faster and more reliably when features are on similar scales, and distance-based methods like k-NN or k-means would otherwise let large-magnitude features dominate the distance calculation. Tree-based models like random forests and gradient boosting are generally scale-invariant since they split on thresholds rather than compute distances or gradients directly on raw values, so scaling matters less there. Getting feature engineering and scaling right is often where the biggest performance gains come from in practice, more so than hyperparameter tuning.
A decision tree splits data recursively based on feature thresholds that best separate the target variable, and while it's easy to interpret, a single deep tree tends to overfit and has high variance. Random forests address that by building many decision trees, each trained on a bootstrapped sample of the data and considering a random subset of features at each split, then averaging their predictions to reduce variance without much increase in bias. Gradient boosting takes a different approach: it builds trees sequentially, where each new tree is trained to correct the residual errors of the ensemble so far, which reduces bias and often achieves higher accuracy than random forests but is more prone to overfitting and more sensitive to hyperparameters. Random forests train trees independently and in parallel, so they're generally faster to train and more forgiving of default settings, while boosting methods like XGBoost or LightGBM usually need more careful tuning of learning rate and tree depth but often win on structured, tabular data problems. Both are ensemble methods, but the core difference comes down to random forests reducing variance through parallel averaging while boosting reduces bias through sequential error correction.
Classification predicts a discrete label or category, like whether an email is spam or which digit a handwritten image represents, while regression predicts a continuous numeric value, like a house price or tomorrow's temperature. The choice of loss function and evaluation metrics differs accordingly—classification typically uses cross-entropy loss and metrics like accuracy, precision, and F1, while regression typically uses mean squared error or mean absolute error and metrics like RMSE or R-squared. Some algorithms can be adapted for either task, like decision trees or neural networks, just by changing the output layer and loss function. There's also a middle ground, ordinal regression, for predicting ordered categories like star ratings, which doesn't fit neatly into either bucket. Ultimately the distinction comes down to the nature of the target variable, not the algorithm itself.
The ROC curve plots the true positive rate against the false positive rate at every possible classification threshold, showing the tradeoff between catching positives and generating false alarms as you move the decision threshold. AUC, the area under that curve, summarizes it into a single number between 0 and 1 representing the probability the model ranks a randomly chosen positive example higher than a randomly chosen negative one. An AUC of 0.5 means the model is no better than random guessing, while an AUC of 1.0 means perfect separation between classes. ROC-AUC is threshold-independent, which makes it useful for comparing models before you've settled on an operating threshold, but it can be overly optimistic on heavily imbalanced datasets because the false positive rate stays low even with many false positives when negatives vastly outnumber positives. In those imbalanced cases, precision-recall AUC is often a more informative metric since it focuses directly on the minority class.
Imbalanced datasets, where one class vastly outnumbers another, are a problem because models trained naively tend to just predict the majority class and still achieve high accuracy while being useless for the minority class you usually care about. One approach is resampling: oversampling the minority class, for example with SMOTE which generates synthetic minority examples, or undersampling the majority class to balance the training distribution. Another approach is adjusting the algorithm itself, such as using class weights in the loss function so misclassifying the minority class is penalized more heavily. Choosing the right evaluation metric matters just as much as fixing the data—precision, recall, F1, and precision-recall AUC give a much clearer picture than raw accuracy on imbalanced problems. Ensemble techniques and threshold tuning after training, rather than using the default 0.5 cutoff, are also commonly used to improve minority class performance.
The curse of dimensionality refers to the ways data behaves counterintuitively and becomes sparse as the number of features grows. As dimensions increase, the volume of the feature space grows exponentially, so the fixed amount of training data you have covers an ever-shrinking fraction of that space, making it harder for models to generalize. Distance-based measures also become less meaningful in high dimensions because the ratio between the nearest and farthest points tends to shrink, which hurts algorithms like k-NN or k-means that rely on distance to make decisions. This is part of why high-dimensional datasets are prone to overfitting unless you have proportionally more data or apply dimensionality reduction. Common ways to fight the curse include feature selection, regularization, and techniques like PCA that project data onto a lower-dimensional space while preserving as much variance as possible.
Bagging, short for bootstrap aggregating, trains multiple models independently and in parallel on different bootstrapped samples of the training data, then combines their predictions by averaging or voting, which primarily reduces variance. Boosting trains models sequentially, where each new model focuses on the examples the previous ones got wrong, gradually reducing bias by turning a collection of weak learners into a strong one. Random forest is the classic example of bagging, while AdaBoost, gradient boosting, and XGBoost are classic examples of boosting. Because boosting builds models sequentially and keeps chasing residual errors, it's more prone to overfitting on noisy data and more sensitive to hyperparameters than bagging. Both are ensemble strategies, but bagging is about stabilizing high-variance models through averaging, while boosting is about incrementally improving a high-bias model through sequential correction.
A loss function quantifies how far a model's predictions are from the actual target values for a single training example, and training works by adjusting model parameters to minimize the average loss across the dataset, often called the cost function. For regression tasks, common choices are mean squared error, which penalizes large errors heavily due to squaring, and mean absolute error, which is more robust to outliers since it penalizes errors linearly. For classification tasks, cross-entropy loss, also called log loss, is standard because it heavily penalizes confident wrong predictions and pairs naturally with probabilistic outputs from a softmax or sigmoid layer. Hinge loss is another classification option, used in support vector machines, which focuses on maximizing the margin between classes rather than calibrating probabilities. The choice of loss function shapes what the model actually optimizes for, so it needs to match the problem—for example, using MAE instead of MSE when you want a model that's less sensitive to outliers in the target variable.
Discriminative models learn the boundary between classes directly by modeling the conditional probability of a label given the input, P(y given x), with logistic regression, SVMs, and most neural network classifiers as examples. Generative models instead learn the joint distribution P(x, y), effectively modeling how the data itself is generated for each class, with naive Bayes, Gaussian mixture models, and models like GANs or diffusion models as examples. Because generative models learn the full data distribution, they can be used to generate new synthetic samples, which discriminative models generally cannot do out of the box. Discriminative models tend to perform better on pure classification tasks when you have enough labeled data, since they don't spend modeling capacity on aspects of the input distribution that don't help distinguish classes. Generative models can be advantageous with limited labeled data or missing features, since they model the full joint distribution rather than just the decision boundary.
Principal component analysis is a dimensionality reduction technique that projects data onto a new set of axes, called principal components, ordered by how much variance in the original data they capture. The first principal component captures the most variance, the second captures the most remaining variance while staying orthogonal to the first, and so on, so you can often represent most of the meaningful structure in the data using far fewer dimensions than you started with. It's useful for visualizing high-dimensional data in two or three dimensions, speeding up training by reducing feature count, and mitigating the curse of dimensionality or multicollinearity between correlated features. One tradeoff is that the resulting components are linear combinations of the original features, so they lose the direct interpretability raw features had. PCA is also sensitive to feature scale, so it's standard practice to standardize features before applying it.