Data Science & AI questions

Pandas is Python's core data analysis library built around the DataFrame (table) and Series (column). It handles reading data (CSV/Excel/SQL), cleaning (nulls, types, duplicates), transformation (filtering, groupby, merge/join, pivot) and analysis — basically Excel + SQL powers inside Python, scalable to millions of rows.

import pandas as pd
df = pd.read_csv("sales.csv")
df.groupby("region")["amount"].sum().sort_values(ascending=False)

loc selects by label (index/column names); iloc selects by integer position.

df.loc[5, "sales"]     # row with index label 5, column "sales"
df.iloc[5, 2]          # 6th row, 3rd column by position
df.loc[df["region"]=="West", ["sales","profit"]]  # boolean filtering

df.isnull().sum()                 # count nulls per column
df.dropna()                       # drop rows with any null
df["age"].fillna(df["age"].median())        # fill with median
df["city"].fillna("Unknown")                # fill with constant
df["sales"] = df["sales"].interpolate()     # interpolate time series

Choice depends on meaning: drop when few and random; median/mode-fill for skewed numeric/categorical; interpolate for time series; sometimes a "missing" flag column is itself informative.

pd.merge() — SQL-style join on key columns (how = inner/left/right/outer). df.join() — convenience join on the index. pd.concat() — stacks DataFrames vertically (rows, like UNION ALL) or horizontally.

pd.merge(orders, customers, on="customer_id", how="left")
pd.concat([jan_df, feb_df, mar_df])

df.groupby("region").agg(
    total_sales=("amount", "sum"),
    avg_order  =("amount", "mean"),
    orders     =("order_id", "nunique")
).reset_index()

Split-apply-combine: split rows into groups, apply aggregations, combine into a summary table — the Pandas equivalent of GROUP BY in SQL / a pivot table in Excel.

df.duplicated().sum()                          # how many duplicate rows
df[df.duplicated(subset=["email"], keep=False)]  # view all dup emails
df = df.drop_duplicates(subset=["email"], keep="first")

An anonymous one-line function, mostly with apply() for quick row/column transformations:

df["price_band"] = df["price"].apply(
    lambda x: "High" if x > 1000 else "Low")

For pure element-wise math, vectorized operations (df["a"]*df["b"]) are faster than apply+lambda.

• List [1,2,2] — ordered, mutable, allows duplicates.
• Tuple (1,2) — ordered, immutable; used for fixed records, dict keys.
• Set {1,2} — unordered, unique values; fast membership tests, dedup.
• Dict {"a":1} — key→value mapping; the workhorse for lookups and JSON-like data.

chunks = pd.read_csv("big.csv", chunksize=1_000_000)
total = sum(chunk["amount"].sum() for chunk in chunks)

Options: chunked reading and aggregating per chunk; loading only needed columns (usecols) with efficient dtypes; converting to Parquet; or scaling out with DuckDB/Polars/Dask/Spark for genuinely big data.

Mean = average; median = middle value; mode = most frequent. Median wins with skewed data/outliers: for salaries [30k, 35k, 40k, 45k, 10L], mean ≈ 2.3L (misleading) while median = 40k (representative). That's why house prices and incomes are reported as medians.

Both measure spread around the mean. Variance = average of squared deviations; standard deviation = its square root (same units as the data). Two shops can both average ₹50k daily sales — SD 2k means steady, SD 30k means wild swings. Low SD = consistent, high SD = volatile.

Correlation measures how two variables move together (−1 to +1); causation means one drives the other. Ice-cream sales correlate with drownings — summer causes both. Analysts must say "associated with", check confounders, and only claim causation from controlled experiments (A/B tests).

The probability of seeing results at least this extreme if there were truly no effect (null hypothesis true). p = 0.03 → only a 3% chance this pattern is pure luck → at the common 0.05 threshold we call it statistically significant. It is NOT the probability the hypothesis is true, and significance ≠ business importance.

Question: did the new checkout page increase conversion? H0 (null): no difference. H1: conversion increased. Run an A/B test, compute the test statistic (e.g. two-proportion z-test), get the p-value; if p < 0.05 reject H0 and roll out the new page. Also check sample size/power before trusting the result.

CLT: the distribution of sample means approaches a normal curve as sample size grows (~30+), regardless of the population's shape. It's why we can build confidence intervals and run t-tests on revenue-per-user or delivery times even when the raw data is skewed.

Type I (false positive): rejecting a true null — claiming the campaign worked when it didn't (probability = α, usually 5%). Type II (false negative): missing a real effect (probability = β; power = 1−β). Business framing: Type I wastes money on a fake win; Type II leaves a real win on the table.

Values far from the rest. Detect: IQR rule (outside Q1−1.5·IQR to Q3+1.5·IQR), z-score > 3, or box plots. Treat: investigate first (data-entry error vs genuine event) → fix errors, cap/winsorize, analyze with and without, or use robust metrics (median). Never silently delete — a fraud spike "outlier" may be the whole story.

ML is teaching computers to learn patterns from data and make predictions/decisions without being explicitly programmed with rules — e.g. learning from past transactions which future ones look fraudulent.

• Supervised: learn from labelled data (X → known Y). Predict price, detect spam. Algorithms: linear/logistic regression, decision trees, random forest, XGBoost.
• Unsupervised: find structure in unlabelled data. Customer segmentation (K-Means), anomaly detection, PCA.
• Reinforcement: an agent learns by trial-and-error rewards — game AI, robotics, recommendation tuning.

Both supervised. Regression predicts a continuous number (next month's sales, house price). Classification predicts a category (churn: yes/no, sentiment: positive/neutral/negative). Same data can frame both: predict revenue (regression) vs predict "will spend >₹10k?" (classification).

Overfitting = the model memorizes training data (noise included) and fails on new data — 99% train accuracy, 65% test. Prevent: train/test split & cross-validation, simpler models, regularization (L1/L2), pruning/limiting tree depth, more data, early stopping, dropout (neural nets). Underfitting is the opposite — too simple to capture the pattern.

Bias = error from oversimplifying (underfit); variance = error from oversensitivity to training data (overfit). Simple models: high bias/low variance; complex models: low bias/high variance. The art is the sweet spot in the middle — tuned via validation curves and regularization.

Split data (e.g. 80/20) so the model is evaluated on unseen data. K-fold cross-validation goes further: split into K parts, train K times each holding out one fold, average the scores — a more reliable estimate, especially on small datasets. Golden rule: test data must never leak into training (fit scalers/encoders on train only).

With 99% legit transactions, a model saying "never fraud" is 99% accurate and useless. Precision = of predicted positives, how many were right (cost of false alarms). Recall = of actual positives, how many we caught (cost of missing). F1 = harmonic mean of both. Fraud/cancer screening → prioritize recall; spam filtering → precision.

Linear regression fits a line y = b0 + b1x to predict a number (ad spend → sales). Logistic regression passes that line through a sigmoid to output a 0–1 probability for classification (will the customer churn?). Despite the name, logistic regression is a classification algorithm — and a favourite interview trick question.

A decision tree splits data by the most informative questions ("Income > 50k?") into purer and purer groups — easy to explain, prone to overfitting. A random forest builds hundreds of trees on random data/feature subsets and votes — much more accurate and stable. Follow-up: bagging (forest, parallel) vs boosting (XGBoost, sequential error-fixing).

Unsupervised algorithm grouping data into K clusters: pick K centers → assign points to nearest center → recompute centers → repeat until stable. Use case: customer segmentation on recency/frequency/monetary value revealing "champions", "at-risk", "bargain hunters" for targeted marketing. K is chosen via the elbow method/silhouette score.

Creating better model inputs from raw data — often more impactful than changing algorithms. Examples: date → day-of-week/month/festival-flag; amount → log(amount) for skew; address → distance-from-city-center; transactions → RFM features per customer; categorical → one-hot/target encoding; scaling for distance-based models.

Traditional ML predicts (a number, a class) from patterns; Generative AI creates new content — text, images, code — by learning the underlying distribution of data. LLMs like GPT and Claude generate text token by token, predicting the next most likely token given context, trained on massive corpora.

A Large Language Model is a neural network (transformer architecture) with billions of parameters trained on huge text datasets to predict the next token. A token is a chunk of text (~4 characters / ¾ of a word in English) — models read and generate token by token, and context windows and API pricing are measured in tokens.

Designing inputs to get reliably good LLM outputs. Techniques: be specific with role + task + format ("You are a SQL expert; return only the query"); give few-shot examples; ask for step-by-step reasoning on complex problems; constrain output (JSON schema, word limits); iterate. As an analyst: "Write a SQL query for monthly sales by region from table X with columns A, B, C" beats "help with SQL".

Confident but false outputs — invented statistics, fake citations, wrong formulas. Mitigate: ground the model with RAG (retrieval of real documents), ask for sources, lower temperature for factual tasks, validate critical outputs (run the SQL, check the number), and human review. Analysts should treat LLM output as a smart draft, not truth.

Architecture that fixes an LLM's knowledge limits: user query → retrieve relevant chunks from your documents (via embeddings + vector database) → inject them into the prompt → LLM answers grounded in your data. This is how "chat with your company docs/PDF" products work — no retraining needed, answers stay current and citable.

Numeric vector representations of text/items where similar meanings sit close together — "refund not received" ≈ "money not returned". Uses: semantic search, clustering customer feedback into themes, deduplication, recommendation, powering RAG. They let you do math on meaning.

Balanced answer: AI accelerates the mechanical parts — writing SQL/DAX drafts, cleaning scripts, chart suggestions, summarizing findings — so analysts shift up the value chain: framing the right business questions, validating AI output, data quality, storytelling and decisions. "I use AI as a productivity multiplier, but I verify everything because I own the accuracy."

Nested circles: AI = any technique making machines act intelligently → ML = the subset that learns from data instead of hard-coded rules → Deep Learning = the subset of ML using multi-layer neural networks (images, speech, LLMs). Every LLM is DL, every DL is ML, every ML is AI — not vice versa.