ML model evaluation
interview questions, 2027

Training a model is increasingly automated; deciding whether it is good enough, for whom and at what threshold is not. That is why model evaluation is one of the clearest dividing lines between junior and senior data scientists and ML engineers.

The roles that test it pay well. On AmbitionBox in October 2026, data scientists in India typically earned ₹15.2–16.8 lakh and ML engineers ₹12.6–13.9 lakh, with data scientists at 6–9 years reaching ₹20.5–22.6 lakh.⁷⁸

Typical data scientist pay in India (AmbitionBox, Oct 2026)

₹15.2–16.8 lakh

Data scientists at 6–9 years

₹20.5–22.6 lakh

Typical ML engineer pay in India

₹12.6–13.9 lakh

Value of a 93%-accurate model on 7% default data that predicts "no default"

0

Google's Machine Learning Crash Course has clear free modules on classification metrics and thresholds.³

"Fraud is 0.5% of transactions. How do you evaluate the model?"

Precision, recall and PR-AUC, with a threshold chosen from the cost of investigating false alarms versus missing fraud — and report expected rupees saved, not just scores.

"Precision went up and recall went down after a change. Good or bad?"

Depends on costs and capacity. If the review team can only handle 200 alerts a day, higher precision may be better; if missed fraud is very costly, recall matters more.

"How would you evaluate a demand forecast?"

Compare MAE or weighted MAPE against a naive or seasonal-naive baseline on rolling time-based folds, and check errors by product and region, especially for high-volume items.

"Your model scores 0.95 AUC offline and does worse live. Why?"

Leakage, training–serving skew, distribution shift, or offline metric not matching the business outcome. Investigate features at prediction time and compare distributions.

"What does calibration mean and when does it matter?"

Predicted probabilities matching observed rates; it matters when probabilities feed pricing, credit limits or expected-value decisions.

"How do you check a model is fair?"

Define protected groups and the relevant fairness criterion, measure metrics by group, investigate gaps, and document trade-offs and mitigations — following model documentation practices such as model cards.⁴

"How do you compare two models properly?"

Same data splits, cross-validated scores with variability, statistical comparison, error analysis by segment, and finally an online test.

"How would you evaluate an LLM-based support bot?"

A representative test set of real queries, rubric-based grading (human or model with spot checks), groundedness against sources, resolution rate in a controlled rollout, plus safety, latency and cost.

Table: How model evaluation expectations have changed (editorial summary)

NIST's AI Risk Management Framework treats measurement — including validity, reliability, fairness and robustness — as a core function of managing AI risk,⁶ which is why evaluation now appears in governance conversations as well as technical ones.

• One honest evaluation report — metric choice justified by costs, baseline compared, slices shown.
• A threshold analysis — precision/recall at several thresholds with expected business value.
• A calibration check with a reliability plot.
• A short model card — intended use, data, metrics, limits.
• A verified model evaluation score to link.

1. Week 1: Classification metrics — Confusion matrices, precision/recall, thresholds by cost.
2. Week 2: Curves — ROC vs PR curves on balanced and imbalanced data.
3. Week 3: Validation — Grouped and time-series cross-validation; find leakage in two examples.
4. Week 4: Calibration and slices — Reliability curves; segment error analysis; fairness metrics.
5. Week 5: Regression and forecasting — MAE, RMSE, MAPE vs baselines; take the ML Model Evaluation Certification.
6. Week 6: LLM and production evaluation — Build a small rubric-based evaluation; write a model card.

Related tests: Machine Learning Fundamentals, LLM Evaluation and Benchmarking and AI Ethics and Bias.

A model scores 100,000 transactions; 500 are fraud. Each missed fraud costs about ₹8,000; each false alarm costs ₹150 in review time.

Table: Threshold choice by expected cost (illustrative numbers)

The cheapest threshold here is about 0.7, even though 0.5 catches more fraud. Add one more constraint — the review team can only check 3,000 alerts a day — and the answer is confirmed. This is the reasoning interviewers want: metrics translated into business cost, with operational limits.

1. Imbalanced classifier — Train a model on a fraud-style dataset; report accuracy, then precision, recall and PR-AUC, and see how different the story is.
2. Threshold by cost — Assign rupee costs to false positives and negatives and choose the threshold that minimises expected cost.
3. Leakage hunt — Add a leaky feature on purpose, watch validation scores jump, then remove it and use a time-based split.
4. Calibration plot — Plot a reliability curve, apply isotonic calibration and compare Brier scores.
5. Slice report — Break performance down by three segments and write a short model card.

Evaluation is a seniority signal in data science and ML. Typical pay on AmbitionBox, October 2026:

Table: Roles that test model evaluation and their typical pay in India (AmbitionBox, October 2026; indicative)

• Reporting accuracy on imbalanced problems.
• No baseline to compare against.
• Leaky validation for time-dependent data.
• Ignoring slices where the model fails.
• Scores without business translation.

Metric and calibration references are scikit-learn's documentation and Google's Machine Learning Crash Course; fairness and documentation references are Fairlearn and the Model Cards paper; risk context is NIST's AI RMF. Pay figures come from AmbitionBox (read October 2026). Interview content and the history table are editorial summaries. Science of Apps publishes the free certification linked here. Last reviewed October 2026.

• Confusion matrix — a table of true and false positives and negatives.
• Precision / recall — correctness of positive predictions, and coverage of actual positives.
• ROC-AUC / PR-AUC — areas under curves summarising ranking quality across thresholds.
• Calibration — agreement between predicted probabilities and observed frequencies.
• Data leakage — information in training that wouldn't be available at prediction time.
• Model card — a short document describing a model's intended use, data, performance and limits.