Module 7: Programming for Analysis (Python / R)

Module 7: Programming for Analysis (Python / R)

1. pandas/NumPy Fundamentals — Filtering, Grouping, Merging, Reshaping

NumPy underneath pandas — the one-line architectural point worth knowing cold: a pandas Series is a NumPy array with an index attached. This explains why vectorized operations (df['col'] * 2) are fast — they’re NumPy’s compiled C operations under the hood — and why looping over rows with .iterrows() is a well-known anti-pattern (it abandons vectorization and runs in slow, interpreted Python).

import pandas as pd
import numpy as np

# --- Filtering ---
df[df['amount'] > 100]                                    # boolean mask
df[(df['region'] == 'South') & (df['amount'] > 100)]      # multiple conditions — note: & not 'and'
df.query('amount > 100 and region == "South"')            # query-string syntax, often more readable
df[df['region'].isin(['South', 'West'])]                   # membership filter

# --- Grouping — the pandas equivalent of SQL GROUP BY ---
df.groupby('region')['amount'].sum()
df.groupby(['region', 'category']).agg(
    total_amount=('amount', 'sum'),
    order_count=('order_id', 'count'),
    avg_amount=('amount', 'mean')
).reset_index()

# transform — returns a result aligned back to the ORIGINAL row count (unlike agg, which collapses rows)
df['region_avg'] = df.groupby('region')['amount'].transform('mean')
df['pct_of_region_total'] = df['amount'] / df.groupby('region')['amount'].transform('sum')

# --- Merging — pandas equivalent of SQL JOIN ---
pd.merge(orders, customers, on='customer_id', how='left')       # mirrors LEFT JOIN exactly
pd.merge(orders, customers, on='customer_id', how='inner', indicator=True)  
# indicator=True adds a '_merge' column showing match source — invaluable for debugging join coverage

# --- Reshaping ---
df.pivot_table(index='region', columns='month', values='amount', aggfunc='sum')   # wide format — like Excel PivotTable
df.melt(id_vars='customer_id', var_name='month', value_name='amount')             # wide → long (tidy format)

The agg vs. transform distinction is a frequently tested interview point: agg collapses each group into a single row (shape changes); transform returns a value for every original row, broadcasting the group result back — essential for things like “amount as % of region total,” which can’t be done with a plain groupby alone.

Talking point on .iterrows(): “If I ever see .iterrows() in a pandas pipeline, that’s usually a sign the operation should be vectorized instead — NumPy/pandas vectorized operations run as compiled loops under the hood, while .iterrows() falls back to slow, row-by-row Python object creation. On a dataset with a few hundred thousand rows, that difference can be the gap between a query that runs in milliseconds and one that takes minutes.”

Merge validation — a real production habit worth naming: “After any merge, I always check the row count before and after — an unexpected row count increase usually means a many-to-many relationship I didn’t account for, the exact join-cardinality bug from the SQL module, just showing up in pandas instead of SQL.” (Direct callback to the SQL joins module — good for showing the concepts transfer across tools.)

2. Exploratory Data Analysis (EDA) Workflow

EDA is the structured version of “get to know the dataset before you trust any conclusion from it” — and a senior-level answer treats it as a repeatable workflow, not an ad hoc poke-around.

A clean EDA sequence to present:

  1. Shape and structure — df.shape, df.info(), df.dtypes — row/column counts, types, memory footprint.
  2. Missingness scan — df.isna().sum() / .mean() — connects directly to the missing-data module; do this before any analysis to know what you’re working with.
  3. Univariate exploration — df.describe() for numeric columns (mean, std, quartiles, min/max — directly surfaces potential outliers via min/max vs. quartiles); df['col'].value_counts() for categoricals.
  4. Distribution shape — histograms, box plots — is it skewed? Bimodal? This directly informs which descriptive stat (mean vs. median) is trustworthy, from the Statistics module.
  5. Bivariate exploration — correlation matrix (df.corr()) for numeric relationships, grouped aggregates/cross-tabs for categorical relationships.
  6. Outlier and anomaly scan — apply the IQR/z-score methods from the cleaning module.
  7. Hypothesis formation — EDA’s actual purpose isn’t pretty charts, it’s generating specific, testable questions to carry into the “Analyze” stage of the lifecycle.

Talking point on EDA’s real purpose (this is the point that separates a senior framing from a junior one): “EDA isn’t a separate deliverable — it’s risk reduction before you commit to an analysis approach. If I skip EDA and jump straight to a hypothesis test, I might run a t-test on a heavily skewed distribution where the normality assumption doesn’t hold, or build a correlation analysis on a variable that’s secretly bimodal because it’s actually two different populations merged together. The 10-15 minutes of EDA is what tells you which tools from the statistics module are even valid to use.” This explicitly ties EDA back to the assumption-checking theme from the Stats module — strong synthesis move.

# A compact EDA pattern worth showing fluency in
df.describe(include='all').T          # transpose for readability on wide datasets
df.select_dtypes('number').corr()      # numeric correlation matrix
pd.crosstab(df['region'], df['churned'], normalize='index')   # % churn by region

3. Basic Visualization — matplotlib/seaborn

matplotlib is the low-level foundation (full control, more verbose); seaborn is built on top of it (statistical defaults, less code, better-looking output out of the box). The practical answer for “which one”: seaborn for fast statistical exploration, drop down to matplotlib when you need fine-grained control over a specific chart for a stakeholder deliverable.

import matplotlib.pyplot as plt
import seaborn as sns

# Distribution — connects to "is it skewed" from EDA
sns.histplot(df['amount'], kde=True)

# Relationship between two numeric variables — connects to correlation analysis
sns.scatterplot(data=df, x='ad_spend', y='revenue', hue='region')

# Categorical comparison — connects to groupby aggregates
sns.barplot(data=df, x='region', y='amount', estimator='mean')

# Outlier visualization — direct visual companion to the IQR method
sns.boxplot(data=df, x='region', y='amount')

# Correlation heatmap — visual version of df.corr()
sns.heatmap(df.select_dtypes('number').corr(), annot=True, cmap='coolwarm')

Chart selection logic — this is a direct callback to the orientation module’s “Visualize” stage, and worth restating concretely:

Question being asked Right chart
How is this distributed? Histogram, box plot
Trend over time? Line chart
Comparing categories? Bar chart
Relationship between two numeric variables? Scatter plot
Composition / parts of a whole? Stacked bar (avoid pie charts beyond 3-4 categories)
Many variable correlations at once? Heatmap

Talking point: “Exploratory visualization and presentation visualization have different goals — in EDA, I want fast, ugly, disposable charts that help me understand the data; matplotlib defaults are fine for that. When something graduates to a stakeholder deliverable, that’s a different design pass entirely — closer to the dashboard/communication principles than to the exploration tooling.”

4. Jupyter Notebook Best Practices for Reproducible Analysis

This is the section that tests engineering discipline applied to analysis — exactly the kind of thing that resonates with your architect background, since notebooks have well-known failure modes that mirror bad coding practices.

The core problem with notebooks: execution order can silently diverge from cell display order. You can run cells out of sequence, delete a cell that defined a variable still being used below, or re-run an earlier cell after later ones — and the notebook looks coherent top-to-bottom while the actual execution state is corrupted. This is the single most-cited notebook failure mode and a strong interview talking point.

Practical disciplines:

  • Restart and Run All before trusting any result — the only reliable way to confirm the notebook actually works top-to-bottom in execution order, not just “however I happened to run cells while exploring.”
  • Cell organization — imports/config at the top, then data loading, then EDA, then analysis, then visualization — a clear, linear narrative, not a scratch pad.
  • Avoid global mutable state surprises — re-running a cell that does df = df[df['amount'] > 0] a second time silently re-filters an already-filtered df. Either use clearly distinct variable names per transformation stage, or rebuild from the original source at the top of a logical block.
  • Extract reusable logic into .py modules — once a function is used more than once or needs to be tested, it belongs in an importable script, not copy-pasted across notebook cells — this is the same “don’t repeat yourself, make it testable” principle from software engineering, just often ignored in notebook culture.
  • Parameterize, don’t hardcode — file paths, date ranges, thresholds as named variables/constants near the top, not scattered as magic literals through the analysis — supports rerunning the same notebook for a different time window or dataset.
  • Pin dependencies — requirements.txt / environment files, since “works on my machine” notebook bugs from silent version drift in pandas/numpy/etc. are common and hard to diagnose later.
  • Strip output before committing to version control (or use a tool like nbstripout) — diffs on notebooks with embedded output/images are nearly unreadable in git otherwise.

Talking point that connects directly to your professional background: “A lot of the ‘notebooks aren’t real engineering’ criticism is really about execution-order discipline, not the tool itself — the same hygiene that prevents bugs in any codebase (clear state, no hidden mutation, version-controlled dependencies) applies directly to notebooks. ‘Restart and Run All passes cleanly’ is the notebook equivalent of a green CI build — if it doesn’t pass that test, the analysis isn’t actually reproducible yet, regardless of how convincing it looked while I was building it interactively.”

5. (Optional) Intro to scikit-learn for Analysts Moving Toward Light Predictive Work

Framed deliberately as the boundary-crossing point into Data Science — ties directly back to the DA-vs-DS overlap discussion from the orientation module.

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.metrics import mean_absolute_error, accuracy_score, roc_auc_score

X = df[['ad_spend', 'site_visits', 'discount_pct']]
y = df['revenue']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
mean_absolute_error(y_test, predictions)

The essential conceptual guardrails an analyst needs before going further down this path (this is more important than the syntax, and is exactly what an interviewer probes for to check you know your own boundary):

  • Train/test split exists to prevent overfitting evaluation — evaluating a model on the same data it was trained on always looks better than it actually performs on new data; this is the predictive-modeling analog of why you don’t p-hack by testing your hypothesis on the same data that generated it.
  • A regression coefficient is not a causal effect — direct callback to the correlation-vs-causation module; a linear regression coefficient describes association controlling for the other included variables, not proof that changing X causes a change in Y, unless the data comes from a true experiment.
  • Know when to hand off to a Data Scientist: feature engineering at scale, hyperparameter tuning, model monitoring/drift, and production deployment are genuinely a different skill set and role — an analyst using LinearRegression() for a quick directional read on a business question is appropriate; building and maintaining a production scoring model is DS/MLE territory.

Talking point that closes the loop on the whole track’s organizing theme: “This is exactly the DA/DS boundary from the very first module — using scikit-learn for a quick regression to answer ‘does ad spend predict revenue, controlling for site visits’ is well within an analyst’s scope. The line I’d draw is: if the output is a one-time directional insight for a stakeholder, that’s analyst work; if the output is a model that needs to run continuously, get retrained, and have monitored prediction quality in production, that’s a handoff to a Data Scientist or ML Engineer — even if the code looks almost identical at the prototype stage.”

This effectively completes the practical analyst toolkit arc. Remaining open items from earlier in the conversation: Data Visualization & Dashboard Design (still pending as its own dedicated module — chart anti-patterns, dashboard design principles, tool-specific Tableau/Power BI depth) and the Word doc compilation of everything built so far (Orientation, SQL, Excel, Statistics, A/B Testing, Data Cleaning, pandas/EDA — that’s seven substantial modules now sitting inline).