buildwithedward.github.io for years - functional, free, and completely forgettable as a URL. If I’m going to write seriously about production AI systems and infra, the least I can do is own my own domain.
This week I finally did it. Here’s exactly how it went.
The github.io URL always felt temporary. It’s fine when you’re experimenting, but I’m now writing about real production topics - RAG pipelines, LLM inference, AWS architecture, client RFP approaches. A personal domain signals that this is a real, long-term effort. It’s also just easier to share verbally: “edwardpraveen.com” versus “buildwithedward dot github dot io.”
Before buying anything, I thought through the two realistic paths.
Advantages:
Considerations:
Advantages:
buildwithedward.github.io URL auto-redirects to the new domainConsiderations:
Why I chose this: AWS makes sense when you need custom server-side behaviour, granular CDN control, or want everything under one account for compliance reasons. For a static Jekyll blog, GitHub Pages does everything needed. Right tool for the job - no point adding complexity to a solved problem.
I went with Namecheap and grabbed edwardpraveen.com. The checkout page, as always, tries to sell you a bundle of add-ons. Here’s my honest take on each:
| Add-on | Cost | Verdict |
|---|---|---|
| SSL Certificate | ~$15–50/yr | ❌ Skip - GitHub Pages gives free SSL via Let’s Encrypt. ACM is also free if you move to AWS later. |
| Premium DNS | ~$5/yr | ❌ Skip - BasicDNS is perfectly adequate for a personal blog. Premium adds DDoS protection you don’t need. |
| Sitelock Website Security | ~$20/yr | ❌ Skip - Sitelock scans PHP/WordPress sites for malware. A static Jekyll site has nothing to infect. |
| Business Email | ~$15/yr | ⚠️ Optional - worth it for edward@edwardpraveen.com, but Zoho Mail has a free tier for one custom address. |
Total checkout: ~$10 for the domain alone.
Once the domain was registered, the changes were split across two places.
Namecheap pre-populates two default records out of the box:
www to parkingpage.namecheap.com@Delete both of these first. They will conflict with the GitHub records.
Then add the following:
4 A records (GitHub’s servers):
| Type | Host | Value |
|---|---|---|
| A Record | @ | 185.199.108.153 |
| A Record | @ | 185.199.109.153 |
| A Record | @ | 185.199.110.153 |
| A Record | @ | 185.199.111.153 |
1 CNAME record (www subdomain):
| Type | Host | Value |
|---|---|---|
| CNAME Record | www | buildwithedward.github.io. |
Note the trailing dot after
.github.io.- include it exactly as shown.
CNAME file in the root of your repo containing just your domain:
edwardpraveen.com
Go to Settings → Pages → Custom domain, enter edwardpraveen.com, click Save.
Wait for GitHub’s DNS check to pass (a few minutes), then tick Enforce HTTPS.
_config.yml:
url: "https://edwardpraveen.com"
baseurl: ""
Commit and push. GitHub rebuilds the site and the SSL cert provisions automatically within ~30 minutes.
The old
buildwithedward.github.ioURL automatically issues a 301 redirect toedwardpraveen.comonce the custom domain is set - no extra config needed.
Once DNS propagates (30 min to 48 hrs), check:
https://edwardpraveen.com loads the bloghttps://www.edwardpraveen.com loads the bloghttp://edwardpraveen.com redirects to https://buildwithedward.github.io redirects to edwardpraveen.comYou can monitor DNS propagation in real time at dnschecker.org.
The domain was the activation energy I needed. Going forward, I’m planning to publish weekly - covering the things I’m actually working on:
Infrastructure and DevOps - real-world AWS architectures, ECS deployments, CI/CD pipelines, IaC patterns. The kind of end-to-end walkthroughs I wish existed when I was setting things up.
AI/ML engineering - LLM inference optimization, RAG pipeline design, multi-agent orchestration. Practical content grounded in production constraints, not toy examples.
Architecture approaches from client RFPs - I’ve been involved in scoping and proposing systems for enterprise clients. I’ll share the architectural thinking behind those proposals, with synthetic data where needed, so the reasoning is transferable even when the specifics can’t be.
If you’ve been putting off claiming your own domain for a blog or portfolio, the barrier is genuinely low - one afternoon, ~$10, and a handful of DNS records. The github.io redirect means you lose nothing from your existing audience. There’s no good reason to wait.
]]>“How many times should our sales reps call each doctor to maximize prescriptions - without annoying them?”
My first instinct was: XGBoost. Maybe a neural network. Something powerful.
Then the client added one constraint that changed everything:
“We don’t want a black box. We need to explain every recommendation to our commercial team.”
That sent me down a rabbit hole of explainable AI, causal thinking, and a model type I’d honestly underestimated - GAM (Generalized Additive Models). I ended up building a complete end-to-end ML platform to understand it deeply. This post walks through what I built and why.
In the pharma world, HCP stands for Healthcare Provider - basically, the doctors that sales representatives visit to promote drugs.
The problem is a classic Goldilocks situation:
The goal isn’t just “more calls = more prescriptions.” The goal is finding the sweet spot for each individual doctor - because a cardiologist in New York and a neurologist in Texas will have completely different tolerances and response patterns.
This is what commercial analytics teams call call optimization, and it’s a real problem at scale when you have thousands of doctors across multiple territories.
When you hear “optimize for each individual”, it’s tempting to reach for a neural network. They’re great at learning individual patterns.
But the client’s commercial team had a legitimate need: they wanted to sit in a meeting and explain to their VP why Doctor X was getting 3 calls this quarter and Doctor Y was getting 6. A neural network can’t do that. It gives you a number with no explanation.
This is a more common situation than you’d think in enterprise settings - especially in regulated industries like pharma, finance, and healthcare. Sometimes trust and explainability matter more than squeezing out the last 1% of accuracy.
This led me to GAM - Generalized Additive Models.
You’ve probably seen linear regression:
Prescriptions = (Calls × 2) + (Emails × 3) + 5
That’s a straight line. Simple and explainable - but it assumes the relationship is always a straight line. In real life, that’s rarely true.
A GAM says instead:
Prescriptions = f(Calls) + f(Emails) + f(Recency) + ...
Each f(...) is a smooth curve the model learns from data - not a straight line. And here’s the magic: you can plot each curve individually and show a business person exactly what’s happening.
For this use case, the “calls” curve looks something like this:
Prescription Growth
▲
│ ●●●
│ ● ●●
│ ● ●●●
│ ● ●●●●
└──────────────────────────▶
1 2 3 4 5 6 7
Number of Calls
←── Sweet Spot ──→←─ Fatigue ─→
Initial calls drive growth. After a certain point, returns diminish. Too many calls and you actually hurt performance. GAM captures this naturally - and you can show this exact curve to a stakeholder and say “this is why we’re recommending 4 calls.”
Before modeling, I realized there’s a fundamental split in the doctor population:
| Group | Who They Are | What We’re Trying to Do |
|---|---|---|
| Breadth HCPs | Doctors who have never prescribed the drug | Convert them to first-time prescribers |
| Depth HCPs | Doctors who already prescribe | Grow their volume without causing fatigue |
These two groups need completely different models. Asking “will this doctor prescribe?” is a classification problem. Asking “how much will prescriptions grow with one more call?” is a regression problem.
So I built two separate pipelines:
Here’s something that trips up a lot of commercial analytics work:
Sales reps are human. They naturally spend more time visiting doctors who are already performing well.
So if you naively train a model on raw call data, it learns the wrong thing:
"Doctors who get more calls write more prescriptions"
But that’s backwards. The doctor was already a top performer - the rep was visiting them because they were doing well, not the other way around. This is called selection bias, and it can make your model confidently wrong.
I built a logistic regression model that asks: “Given this doctor’s profile, how likely is a sales rep to call them frequently?” This score captures the bias - and lets us mathematically correct for it when training the main models. Think of it like a control group in a clinical trial.
Instead of asking “will this doctor prescribe?”, I engineered features around the question: “will an extra call actually cause more prescriptions?” This is the causal question. A doctor might have high prescriptions regardless of calls - in that case, calling them more is wasted effort.
Together, these two layers help the model measure the true incremental effect of a call, not just correlation.
Here’s the complete flow from raw data to recommendation:
Step 1 - Load historical HCP data
(calls, prescriptions, emails, territory, specialty)
↓
Step 2 - Feature engineering
(responsiveness score, saturation estimate, recency)
↓
Step 3 - Split: Breadth vs Depth doctors
↓
Step 4 - Build Propensity Scores (fix selection bias)
↓
Step 5 - Engineer Uplift Features (causal signal)
↓
Step 6 - Train models
Breadth → GAM Logistic (predicts conversion %)
Depth → GAM Regression (predicts script growth)
↓
Step 7 - Find optimal call count per doctor
(the point on the curve before diminishing returns)
↓
Step 8 - Serve recommendations via FastAPI
↓
Step 9 - Track in MLflow, monitor for drift, retrain when needed
┌─────────────────────────────────────────────┐
│ DATA LAYER │
│ 20,000 HCP records (synthetic) │
│ Calls, scripts, emails, specialty, etc. │
└──────────────────┬──────────────────────────┘
│
┌──────────────────▼──────────────────────────┐
│ FEATURE ENGINEERING │
│ Propensity Scores + Uplift Features │
└──────────────────┬──────────────────────────┘
│
┌──────────┴───────────┐
│ │
┌───────▼────────┐ ┌─────────▼──────────┐
│ BREADTH MODEL │ │ DEPTH MODEL │
│ │ │ │
│ GAM Logistic │ │ GAM Regression │
│ → Conversion % │ │ → Script Growth │
│ │ │ │
│ XGBoost │ │ XGBoost Regressor │
│ (benchmark) │ │ (benchmark) │
└───────┬────────┘ └─────────┬──────────┘
└──────────┬───────────┘
│
┌──────────────────▼──────────────────────────┐
│ MLOPS LAYER │
│ MLflow → FastAPI → Docker → Monitoring │
└─────────────────────────────────────────────┘
I started with a synthetic dataset of 20,000 doctors. The key fields:
df = pd.DataFrame({
"hcp_id": hcp_ids,
"specialty": specialty, # Cardiology, Oncology, etc.
"previous_scripts": ..., # How many scripts before?
"calls": ..., # Calls made this quarter
"email_open_rate": ..., # Engagement signal
"responsiveness": ..., # How much does this doctor respond?
"saturation": ..., # How quickly do they tire?
"estimated_uplift": ..., # Causal signal
"converted": ..., # Did they prescribe? (0 or 1)
"hcp_type": ... # "Breadth" or "Depth"
})
The synthetic script growth formula captures the saturation curve:
script_growth = (
responsiveness * np.log1p(calls) # diminishing returns
- saturation * (calls ** 2) # fatigue penalty
)
This means early calls help, but the squared penalty kicks in as calls increase. That’s the curve GAM will learn.
For doctors who’ve never prescribed, we want to predict the probability they’ll convert:
from pygam import LogisticGAM, s
model = LogisticGAM(
s(0) + s(1) + s(2) + s(3)
)
model.fit(X_train, y_train)
pred_probs = model.predict_proba(X_test)
auc = roc_auc_score(y_test, pred_probs)
s() is a spline term - it tells GAM “learn a smooth curve for this feature.” Each feature gets its own curve. You can plot them individually and explain exactly what’s happening.
For existing prescribers, we predict how many scripts they’ll write:
from pygam import LinearGAM, s
model = LinearGAM(
s(0) + s(1) + s(2) + s(3) + s(4)
)
model.fit(X_train, y_train)
preds = model.predict(X_test)
rmse = mean_squared_error(y_test, preds) ** 0.5
The partial dependence plot for “calls” is the response curve - it shows exactly where the saturation point is. That’s the number you put in your quarterly call plan.
Without experiment tracking, you’re flying blind. Every training run logs its metrics and artifacts:
import mlflow
mlflow.set_experiment("hcp-call-optimization")
with mlflow.start_run():
model.fit(X_train, y_train)
rmse = mean_squared_error(y_test, preds) ** 0.5
mlflow.log_metric("rmse", rmse)
mlflow.log_param("model_type", "Depth_GAM")
mlflow.log_artifact("models/depth_gam.pkl")
Run mlflow ui and you get a full dashboard showing every experiment - which parameters you used, what the metrics were, and which model artifacts were saved.
Once models are trained, they’re wrapped in a FastAPI service. Any downstream system - a CRM, a planning tool, a dashboard - can call it:
POST /predict/breadth
~ Send: { calls, email_open_rate, call_recency_days, estimated_uplift }
~ Get: { "conversion_probability": 0.73 }
POST /predict/depth
~ Send: { calls, previous_scripts, email_open_rate, ... }
~ Get: { "predicted_scripts": 14.2 }
from fastapi import FastAPI
from pydantic import BaseModel
import joblib
app = FastAPI()
breadth_model = joblib.load("models/breadth_gam.pkl")
depth_model = joblib.load("models/depth_gam.pkl")
@app.post("/predict/breadth")
def predict_breadth(hcp: BreadthInput):
row = pd.DataFrame([[hcp.calls, hcp.email_open_rate, ...]])
pred = breadth_model.predict_proba(row)[0]
return {"conversion_probability": float(pred)}
Start the server with uvicorn api.app:app --reload and the interactive docs are available at http://127.0.0.1:8000/docs.
The monitoring script checks whether real outcomes match predictions:
mean_error = predictions["error"].mean()
if mean_error > 5:
print("Retraining recommended")
else:
print("Model stable")
Simple, but the idea scales. In production, you’d feed actual prescription data back in, calculate drift, and trigger a retraining pipeline automatically.
I also trained XGBoost models - both a classifier for breadth and a regressor for depth. They scored similarly to the GAM models.
But the decision was clear: explainability beats a marginal accuracy bump. In a commercial setting, a model that a business analyst can point to and explain is infinitely more adoptable than a black box with slightly better AUC.
XGBoost stayed in the project as a challenger - useful for validation, not for production.
hcp-call-optimization/
│
├── data/
│ └── synthetic_hcp_data.csv
│
├── src/
│ ├── generate_data.py ← Synthetic dataset
│ ├── propensity_model.py ← Bias correction
│ ├── breadth_gam_model.py ← Breadth training
│ ├── depth_gam_model.py ← Depth training
│ ├── train_breadth_pipeline.py ← MLflow breadth run
│ └── train_pipeline.py ← MLflow depth run
│
├── models/
│ ├── breadth_gam.pkl
│ └── depth_gam.pkl
│
├── api/
│ └── app.py ← FastAPI service
│
├── monitoring/
│ └── monitor.py
│
└── docker/
└── Dockerfile
# Generate data
python src/generate_data.py
# Build bias-correction scores
python src/propensity_model.py
# Train models
python src/breadth_gam_model.py
python src/depth_gam_model.py
# MLflow-tracked training runs
python src/train_breadth_pipeline.py
python src/train_pipeline.py
# View experiment history
mlflow ui
# http://localhost:5000
# Start the prediction API
uvicorn api.app:app --reload
# http://127.0.0.1:8000/docs
# Run monitoring check
python monitoring/monitor.py
1. Explainability is a feature, not a consolation prize
In enterprise AI - especially pharma, finance, or healthcare - a model your stakeholders trust will always outperform a model they don’t. GAM gave me comparable accuracy to XGBoost and full transparency. In regulated industries, that’s not a tradeoff. It’s the right answer.
2. Always ask the causal question
“Will this doctor prescribe?” and “Did our call cause more prescriptions?” are completely different questions. Getting that distinction right changes your feature engineering, your model architecture, and the quality of your recommendations. Propensity scoring and uplift modeling exist to close that gap.
3. GAM is underrated for commercial analytics
I came in expecting GAM to be a fallback option. I left with a lot of respect for it. For problems involving saturation curves, diminishing returns, and engagement optimization, GAM is genuinely the right tool - not just a compromise.
4. MLOps is what makes ML real
A trained model file sitting on your laptop is not a product. The FastAPI layer, MLflow tracking, Docker container, and monitoring pipeline are what turn an experiment into something a business can actually use.
If I were to extend this into a production system, the next steps would be:
The project started as a way to understand a client use case. It ended up being one of the most interesting explorations I’ve done - touching causal ML, response curve optimization, and the full MLOps stack. Sometimes the most valuable thing a constraint gives you is a reason to think differently.
If you’re working on commercial analytics, HCP engagement, or explainable ML in regulated industries - I’d love to hear how you’re approaching it.
]]>I came across a role for a Forward Deployed Engineer that had a very unusual ask:
“Submit your application through this MCP endpoint. No instructions.”
Just a URL. No documentation. No user interface. No hints.
Here’s how I figured out the entire system from scratch and successfully submitted my application.
Before diving in, let me quickly explain. MCP (Model Context Protocol) is a way for AI tools and services to communicate with each other. Think of it like a language that AI systems use to talk. In this case, the job application was hidden behind an MCP server - meaning I had to speak its language to apply.
I started with the simplest thing possible - just visiting the URL:
curl -i https://submit-cv-mcp.webrix.workers.dev/mcp
The response:
{
"error": "invalid_token",
"error_description": "Missing or invalid access token"
}
In plain English: the door is locked, and I need a key (an access token) to get in.
When a server sends back an error, it often includes headers (extra metadata) in the response. These headers had a critical clue:
www-authenticate: Bearer realm="OAuth",
resource_metadata=".../.well-known/oauth-protected-resource/mcp"
This is like the locked door having a sign that says: “The key shop is at this address.” The server was pointing me toward an OAuth authentication system - a standard way websites let you prove who you are.
Following the clue from the headers, I visited the metadata URL:
curl https://submit-cv-mcp.webrix.workers.dev/.well-known/oauth-protected-resource/mcp
This told me where the authentication server lives. Then I fetched the full configuration:
curl https://submit-cv-mcp.webrix.workers.dev/.well-known/oauth-authorization-server
This gave me three important URLs:
/authorize - where I go to prove who I am/token - where I exchange a temporary code for a real access token/register - where I register myself as a new “client” (application)Think of it like discovering the full map of a building: “Registration is on floor 1, authorization on floor 2, and key pickup on floor 3.”
Before I could authenticate, I needed to register. This is like signing up before you can log in:
curl -X POST https://submit-cv-mcp.webrix.workers.dev/register \
-H "Content-Type: application/json" \
-d '{"redirect_uris":["http://localhost"]}'
The server gave me a client_id and client_secret - think of these as my username and password for the authentication process.
OAuth uses an extra security step called PKCE (pronounced “pixie”). It ensures nobody can intercept and steal your login mid-process. Here’s how it works in simple terms:
VERIFIER=$(openssl rand -base64 32 | tr -d '=+/')
CHALLENGE=$(echo -n $VERIFIER | openssl dgst -sha256 -binary | openssl base64 | tr '+/' '-_' | tr -d '=')
It’s like telling someone “I’ll prove my identity later by completing this puzzle that only I can solve.”
Next, I had to open a URL in my browser with all the pieces put together:
/authorize?response_type=code
&client_id=...
&redirect_uri=http://localhost
&code_challenge=...
&code_challenge_method=S256
This is the part where you actually log in and give permission.
Here’s where things got interesting. Instead of a simple login form:
This revealed that the system uses GitHub as the identity provider. In other words: “Prove who you are by logging into GitHub.”
The full sequence:
codeI captured this code from the redirect URL. This code is like a receipt - proof that I successfully authenticated.
Now I traded my temporary code for a real access token:
curl -X POST https://submit-cv-mcp.webrix.workers.dev/token \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=authorization_code&client_id=...&client_secret=...&code=...&redirect_uri=http://localhost&code_verifier=..."
The server verified everything matched (the code, the PKCE verifier, the client credentials) and gave me the golden key: an access token.
With my access token in hand, I tried calling the MCP endpoint again. But I got another error:
Client must accept application/json and text/event-stream
This told me something important: this is not a regular REST API. It uses a streaming protocol, meaning the server can send data continuously rather than all at once - like a live conversation instead of sending letters.
MCP uses a format called JSON-RPC - a simple way to call functions remotely. Instead of visiting different URLs for different actions (like REST), you send messages to one endpoint with a “method” field:
{
"jsonrpc": "2.0",
"method": "...",
"id": 1
}
Think of it like calling a receptionist and saying “I’d like to speak to the submit-cv department” instead of walking to different offices.
First, I needed to initialize a session:
{
"method": "initialize"
}
The response included a session ID - meaning the server remembers who I am across multiple requests. This is important because MCP is stateful (it keeps track of the conversation).
Next, I asked the server what tools (actions) are available:
{
"method": "tools/list"
}
Response:
{
"tools": [
"get-job-description",
"submit-cv"
]
}
Two tools: one to read the job description, and one to submit my CV.
When I tried to use submit-cv, it told me I needed two special codes:
resourceCodepromptCodeThese weren’t given to me directly - I had to fetch them from the MCP server using its resources and prompts features. It’s like being told “bring form A and form B to the submission window” – but first you have to find where forms A and B are stored.
{
"method": "resources/read",
"params": {
"uri": "submit-cv-mcp://cv/submission-code"
}
}
This gave me the first required code.
{
"method": "prompts/get",
"params": {
"name": "submit-cv"
}
}
This gave me the second required code.
With both codes in hand, I made the final submission:
{
"method": "tools/call",
"params": {
"name": "submit-cv",
"arguments": {
"resourceCode": "...",
"promptCode": "..."
}
}
}
Just when I thought I was done, the server sent back a new kind of request:
{
"method": "elicitation/create"
}
It was asking me for:
This is called elicitation – the server is asking the client for information mid-conversation, like a form popping up during a chat.
This was the trickiest part. Instead of calling a new method, I had to respond using the same request ID that the server sent. This is how two-way (bidirectional) JSON-RPC works:
{
"jsonrpc": "2.0",
"id": 0,
"result": {
"email": "your@email.com",
"phone": "",
"note": "Looking forward to discussing this opportunity."
}
}
It’s like the server said “Hey, before I finish, I need one more thing” – and I had to reply in exactly the right format.
This wasn’t a coding test. It was a thinking test. Here’s what it evaluated:
System Thinking - Can you follow signals and breadcrumbs instead of waiting for instructions?
Debugging Skills - Can you read error messages and headers to figure out what’s going on?
Protocol Understanding - Do you know how OAuth 2.0, PKCE, JSON-RPC, and MCP work (or can you figure them out)?
Adaptability - Can you switch between tools (curl, browser, APIs, streaming) as needed?
Real Engineering Behavior - Can you experiment, fail, iterate, and solve?
This was one of the most interesting application processes I’ve encountered. Instead of asking “What do you know?”, it asked “How do you think?”
And that makes all the difference.
]]>There’s a moment every developer faces. You ship code to production. It works. Customers use it. And then one of them does something you didn’t expect. Your app crashes. They post a 1-star review. You don’t sleep.
Here’s the thing: that crash-to-be existed in your code the entire time. You just never tested that specific path. You tested the happy path-“user enters valid data, everything works”-but you never tested “user enters empty data” or “user enters huge numbers” or “user is offline.”
Today is the day you stop letting bugs hide.
Day 10 is a turning point in your journey as an engineer. You’re moving from “code that works on my machine” to “code I can prove works.” This is where professionalism begins.
You’ll learn three tools:
By the end, you’ll have a test suite that gives you confidence. And confidence is everything.
First, create a project directory and set up your environment:
# Create and navigate to project directory
mkdir ~/workspace/day10-testing
cd ~/workspace/day10-testing
# Create virtual environment
python -m venv .venv
# Activate it (macOS/Linux)
source .venv/bin/activate
# On Windows, use:
# .venv\Scripts\Activate.ps1
You should see (.venv) appear in your terminal prompt. Good-your venv is active.
Now install the tools you need:
pip install pytest==7.4.3 pytest-cov==4.1.0
Create your requirements.txt:
pytest==7.4.3
pytest-cov==4.1.0
Create your .gitignore:
.venv/
__pycache__/
*.pyc
.pytest_cache/
.coverage
htmlcov/
dist/
build/
*.egg-info/
.DS_Store
Your project structure so far:
day10-testing/
├── .venv/
├── .gitignore
├── requirements.txt
└── (code files go here)
Imagine you’re building two houses on the same street. The first house needs blue paint. The second needs red paint. If they shared the same paint supply, disaster: when you paint one red, the other turns red too.
Python projects are like that. Project A might need Django 3.0 (an older web framework). Project B might need Django 4.0 (a newer version with different features). If both projects share the same Python installation, you can’t install both versions at the same time.
Virtual environments solve this by creating a separate Python installation for each project. Each project gets its own site-packages/ folder (where packages live). Paint one project red, the other stays blue.
python -m venv .venv
Let me break this down:
python - Call Python itself-m venv - Run Python’s built-in venv module.venv - Create a folder called .venv (hidden on macOS/Linux because it starts with a dot)What did Python create? A folder with this structure:
.venv/
├── bin/ # (on macOS/Linux) or Scripts/ (on Windows)
│ ├── python # A Python binary specific to this venv
│ ├── pip # A pip specific to this venv
│ ├── activate # Script to activate the venv
│ └── ...
├── lib/
│ └── python3.11/
│ └── site-packages/ # Empty folder where packages will install
└── pyvenv.cfg
All of this is isolated. Your system Python is untouched.
Activation means “use this venv’s Python instead of the system Python.”
# macOS/Linux
source .venv/bin/activate
# Windows (PowerShell)
.venv\Scripts\Activate.ps1
After activation, your terminal prompt changes:
# Before activation:
$ python --version
Python 3.11.0
# After activation:
(.venv) $ python --version
Python 3.11.0 (from /Users/edward/workspace/day10-testing/.venv/bin/python)
That (.venv) prefix tells you: you are now using the isolated Python.
To deactivate (go back to system Python):
(.venv) $ deactivate
$ python --version
Python 3.11.0 (system Python again)
Your .venv/ folder contains thousands of files. If you committed it to Git, your repo would be:
.venv/)requirements.txt)So: Never commit .venv/. Commit only requirements.txt.
When someone clones your repo:
git clone https://github.com/edward/my-project.git
cd my-project
# Create their own venv
python -m venv .venv
source .venv/bin/activate
# Install packages
pip install -r requirements.txt
They get the exact same environment as you, without downloading gigabytes.
On your M1 Mac, you might hear about conda. Conda is like venv’s more powerful cousin-it handles non-Python dependencies (C libraries, CUDA, even system libraries). M1 Macs especially benefit from conda because it handles ARM architecture seamlessly.
For now, stick with venv. It’s built into Python and sufficient for what we’re doing. Conda is useful later when you’re installing scientific libraries like TensorFlow.
Let’s say you write code that uses NumPy:
import numpy as np
data = np.array([1, 2, 3])
result = data.mean()
You run pip install numpy (without specifying a version). You get version 1.26.0. Your code works perfectly.
Six months later, a colleague clones your repo and runs pip install numpy. NumPy is now at 2.0.0. There’s a breaking change-some API you used was removed. Suddenly your code breaks.
You didn’t change your code. NumPy changed. And now you’re debugging at midnight.
# Instead of
pip install numpy
# Do this
pip install numpy==1.24.1
The ==1.24.1 means: “Install version 1.24.1, exactly. Nothing else.”
Now if someone else installs the same requirements, they get 1.24.1 too. Same code, same dependencies, same behavior.
Option 1: Freeze your current environment
pip freeze > requirements.txt
This command captures everything you’ve installed, with versions:
certifi==2024.2.2
charset-normalizer==3.3.2
idna==3.6
pytest==7.4.3
pytest-cov==4.1.0
requests==2.31.0
urllib3==2.1.0
Any package you’ve installed appears. Some you might not have explicitly asked for-they’re dependencies of dependencies (transitive dependencies).
Option 2: Write it by hand
For a fresh project, just list the packages you actually need:
pytest==7.4.3
pytest-cov==4.1.0
Later, if you add other packages, update this file.
Once you’ve created requirements.txt, sharing your environment is one command:
pip install -r requirements.txt
-r means “read from file.” All packages with exact versions install. Reproducibility achieved.
# List what you have installed
pip list
# Show details about a specific package
pip show pytest
# Output:
# Name: pytest
# Version: 7.4.3
# Summary: pytest: simple powerful testing with Python
# ...
# Uninstall a package
pip uninstall numpy
# See what's outdated
pip list --outdated
# Update a specific package
pip install --upgrade pytest
# or
pip install -U pytest
Newer Python projects use pyproject.toml instead of requirements.txt. It’s more powerful and follows PEP 518 standards.
pyproject.toml looks like this:
[project]
name = "my-project"
version = "1.0.0"
dependencies = [
"pytest==7.4.3",
"pytest-cov==4.1.0",
]
For Day 10, we’re sticking with requirements.txt-simpler, universally understood, and fully supported everywhere. You’ll learn pyproject.toml later.
Before a car ships to customers, engineers do something that seems wasteful: they crash it.
They strap a dummy (made of plastic and sensors) into the car. They launch it into a wall at 30 mph. The dummy experiences the impact. Sensors record everything-forces on the head, chest, legs. Engineers study the data. If the dummy’s chest took too much force, they redesign the airbags.
Then they crash it again. And again. They test every scenario: head-on collision, side impact, rollover. Each test teaches them something.
By the time the car reaches you, it’s been “crashed” hundreds of times. But those crashes were tests, not failures.
Code is the same. pytest is your crash-test dummy.
Let me show you a real bug:
def calculate_average(numbers):
"""Calculate the average of a list of numbers."""
return sum(numbers) / len(numbers)
# Happy path: works fine
result = calculate_average([1, 2, 3])
print(result) # 2.0
# But what if someone passes an empty list?
result = calculate_average([]) # ZeroDivisionError!
You test the happy path manually. You see it works. You ship it. A user passes an empty list. Your app crashes. You find out via a 1-star review.
This is not your fault personally. But it’s a bug that tests would have caught immediately.
Now, with pytest:
import logging
logger = logging.getLogger(__name__)
def calculate_average(numbers: list) -> float:
"""Calculate the average of a list of numbers.
Args:
numbers: List of numeric values
Returns:
The average as a float
Raises:
ValueError: If the list is empty
"""
if not numbers:
logger.error("Cannot calculate average of empty list")
raise ValueError("Cannot calculate average of empty list")
logger.debug(f"Calculating average of {len(numbers)} values")
return sum(numbers) / len(numbers)
# Test it
def test_calculate_average_happy_path():
"""Test normal case."""
result = calculate_average([1, 2, 3])
assert result == 2.0
logger.info("Happy path test passed")
def test_calculate_average_empty_list():
"""Test edge case: empty list."""
with pytest.raises(ValueError):
calculate_average([])
logger.info("Empty list edge case handled correctly")
When you run pytest -v:
tests/test_stats.py::test_calculate_average_happy_path PASSED
tests/test_stats.py::test_calculate_average_empty_list PASSED
======================== 2 passed in 0.05s ========================
You find the bug during development, not in production. You log it. You fix it. Users never see it. Everyone’s happy.
pytest finds tests automatically based on naming conventions. Follow these rules:
test_*.py or *_test.pytest_*Test*pytest will find these:
project/
├── test_mean.py ✓ Starts with test_
├── tests/test_variance.py ✓ File starts with test_
├── stats_test.py ✓ Ends with _test
│
└── THESE GET IGNORED:
├── check_mean.py ✗ Doesn't follow pattern
├── mean_test.py ✗ Not test_mean_test.py
└── tests/stats.py ✗ Doesn't start with test_
Break this rule, and pytest won’t find your tests.
The gold standard for test structure is Arrange → Act → Assert:
def test_mean_of_positive_integers():
# ARRANGE: Set up the data you're testing
numbers = [1, 2, 3, 4, 5]
# ACT: Call the function
result = mean(numbers)
# ASSERT: Check the result
assert result == 3.0
Breaking it down:
This structure makes tests easy to read and understand.
# Run all tests in the current directory
pytest
# Run all tests, verbose (shows each test name)
pytest -v
# Run only one file
pytest tests/test_mean.py
# Run one specific test
pytest tests/test_mean.py::test_mean_of_positive_integers
# Run tests matching a pattern
pytest -k "mean" # Runs test_mean_*, *_mean, etc.
# Stop on the first failure
pytest -x
# Show print statements (normally hidden)
pytest -s
# Quiet mode (only show summary)
pytest -q
Example output from pytest -v:
tests/test_stats.py::test_mean_happy_path PASSED [ 10%]
tests/test_stats.py::test_mean_empty_list FAILED [ 20%]
tests/test_stats.py::test_mean_single_number PASSED [ 30%]
tests/test_stats.py::test_variance_happy_path PASSED [ 40%]
tests/test_stats.py::test_bayes_update_basic PASSED [ 50%]
======= FAILED tests/test_stats.py::test_mean_empty_list ========
def test_mean_empty_list():
with pytest.raises(ValueError):
mean([])
E AssertionError: DID NOT RAISE ValueError
=========== 1 failed, 4 passed in 0.23s ============
This tells you: “The function didn’t raise ValueError when you expected it to. Go fix that.”
Many functions should raise exceptions for bad input. You test that they do:
def test_mean_raises_on_empty_list():
"""The function should reject empty lists."""
with pytest.raises(ValueError):
mean([])
The with pytest.raises(ValueError): context manager says: “I expect this code to raise ValueError. If it does, the test passes. If it doesn’t, the test fails.”
You can even check the error message:
def test_mean_raises_with_correct_message():
"""Check the error message too."""
with pytest.raises(ValueError, match="Cannot calculate mean"):
mean([])
match takes a regex pattern. The error message must contain “Cannot calculate mean” or the test fails.
Parametrised tests let you test many inputs with one test function.
Without parametrisation (repetitive):
def test_mean_case_1():
assert mean([1, 2, 3]) == 2.0
def test_mean_case_2():
assert mean([10, 20]) == 15.0
def test_mean_case_3():
assert mean([5]) == 5.0
def test_mean_case_4():
assert mean([-1, -2, -3]) == -2.0
# ... 10 more similar tests
With parametrisation (clean):
@pytest.mark.parametrize("numbers,expected", [
([1, 2, 3], 2.0),
([10, 20], 15.0),
([5], 5.0),
([-1, -2, -3], -2.0),
([0, 0, 0], 0.0),
([1.5, 2.5, 3.5], 2.5),
])
def test_mean_multiple_cases(numbers, expected):
assert mean(numbers) == expected
pytest runs this one test function six times-once for each tuple. Much cleaner.
You can parametrize multiple arguments:
@pytest.mark.parametrize("numbers,alpha,expected", [
([1, 2, 3], 0.05, 0.6666),
([1, 2, 3], 0.1, 0.6666),
([10, 20, 30], 0.05, 66.6666),
])
def test_variance_with_alpha(numbers, alpha, expected):
result = variance(numbers)
assert result == pytest.approx(expected)
If many tests need the same data, use a fixture:
import pytest
@pytest.fixture
def sample_dataset():
"""Fixture: provides sample data to tests that ask for it."""
return [1, 2, 3, 4, 5]
def test_mean_with_fixture(sample_dataset):
"""This test receives sample_dataset automatically."""
result = mean(sample_dataset)
assert result == 3.0
def test_variance_with_fixture(sample_dataset):
"""Different test, same fixture."""
result = variance(sample_dataset)
assert result == 2.0
The sample_dataset parameter tells pytest: “I need the fixture named sample_dataset.” pytest calls the fixture function and injects the return value.
Why fixtures are great:
If you have multiple test files and want to share fixtures:
project/
├── stats.py
├── tests/
├── conftest.py ← Fixtures defined here
├── test_mean.py ← Uses fixtures from conftest
├── test_variance.py ← Uses fixtures from conftest
└── test_bayes.py
tests/conftest.py:
import pytest
@pytest.fixture
def simple_numbers():
return [1.0, 2.0, 3.0]
@pytest.fixture
def large_numbers():
return list(range(1, 101))
@pytest.fixture
def negative_numbers():
return [-5.0, -2.0, 0.0, 2.0, 5.0]
tests/test_mean.py:
def test_mean(simple_numbers):
# No import needed, fixture comes from conftest.py
assert mean(simple_numbers) == 2.0
tests/test_variance.py:
def test_variance(simple_numbers):
# Same fixture, different file
result = variance(simple_numbers)
assert result == pytest.approx(2.0/3.0)
pytest automatically finds fixtures in conftest.py. No imports needed.
Here’s a brief intro (we’ll do this more on Day 14):
Sometimes your code calls an external API:
import requests
def fetch_stock_price(ticker):
"""Call an external API to get stock price."""
response = requests.get(f"https://api.example.com/price/{ticker}")
return response.json()["price"]
In tests, you don’t want to hit the real API. It’s slow, unreliable, and might cost money. Instead, you mock it:
from unittest.mock import patch
def test_fetch_stock_price():
"""Test without calling the real API."""
with patch('requests.get') as mock_get:
# Make requests.get return fake data
mock_get.return_value.json.return_value = {"price": 150.0}
result = fetch_stock_price("AAPL")
assert result == 150.0
The test never touches the real API. It’s fast (milliseconds), isolated, and reliable.
You write tests. But how do you know if you’ve tested enough?
Code coverage measures what percentage of your code is executed by tests.
pip install pytest-cov
pytest --cov=stats --cov-report=term-missing
Output:
Name Stmts Miss Cover Missing
stats.py 95 5 94% 45-46, 78, 102-104
This says:
stats.py has 95 statements (lines of code)You then write tests for those lines until coverage is 100% (or as close as practical).
Aim for:
A good test name tells you exactly what’s being tested and what’s expected.
Bad names:
def test_mean():
pass
def test_variance():
pass
def test_1():
pass
When they fail, you have no idea what broke.
Good names:
def test_mean_of_positive_integers_returns_correct_average():
pass
def test_variance_of_empty_list_raises_valueerror():
pass
def test_bayes_update_with_strong_evidence_increases_posterior():
pass
When they fail, you know exactly what broke. The test name is a specification of what the function should do.
Pattern: test_[function_name]_[condition]_[expected_result]
F - Fast Tests should run in milliseconds. If a test takes seconds, you won’t run them often. You’ll skip them. Bugs hide.
I - Isolated Tests shouldn’t depend on each other. If test A must run before test B, you have a problem. Tests should be independent.
R - Repeatable Same test, same result every time. No random variation. No flaky tests that sometimes pass and sometimes fail.
S - Self-Validating A test either passes or fails. No manual checking (“did this look right?”). No human judgment.
T - Timely Ideally, write tests before the code (Test-Driven Development). Minimum: write tests alongside the code, not months later. Fresh context makes better tests.
Today you’ll build a complete pytest test suite for the statistics functions from Day 9.
day10-testing/
├── .venv/
├── .gitignore
├── requirements.txt
├── stats.py ← Code being tested
└── tests/
├── __init__.py ← Empty file, makes tests/ a package
├── conftest.py ← Shared fixtures
└── test_stats.py ← All tests
Create stats.py with full type hints and logging:
"""
Statistics module with type hints and logging.
Day 10: Production-grade Python with testing.
"""
import logging
from typing import List, Optional, Tuple
# Configure logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
def mean(numbers: List[float]) -> float:
"""
Calculate the arithmetic mean of a list of numbers.
Args:
numbers: List of floats or ints
Returns:
The mean as a float
Raises:
ValueError: If the list is empty
TypeError: If numbers contains non-numeric values
Example:
>>> mean([1, 2, 3])
2.0
"""
if not numbers:
logger.error("Cannot calculate mean of empty list")
raise ValueError("Cannot calculate mean of empty list")
try:
total = sum(numbers)
result = total / len(numbers)
logger.debug(f"Calculated mean of {len(numbers)} values: {result}")
return result
except TypeError as e:
logger.error(f"Invalid data type in numbers list: {e}")
raise TypeError("All elements must be numeric") from e
def variance(numbers: List[float]) -> float:
"""
Calculate the variance (average squared deviation from mean).
Args:
numbers: List of floats or ints
Returns:
The variance as a float
Raises:
ValueError: If the list is empty or has only one element
TypeError: If numbers contains non-numeric values
Example:
>>> variance([1, 2, 3])
0.6666666666666666
"""
if not numbers:
logger.error("Cannot calculate variance of empty list")
raise ValueError("Cannot calculate variance of empty list")
if len(numbers) < 2:
logger.error("Variance requires at least 2 values")
raise ValueError("Variance requires at least 2 values (population has no variance)")
try:
m = mean(numbers)
squared_diffs = [(x - m) ** 2 for x in numbers]
result = sum(squared_diffs) / len(numbers)
logger.debug(f"Calculated variance: {result}")
return result
except TypeError as e:
logger.error(f"Invalid data type in numbers list: {e}")
raise TypeError("All elements must be numeric") from e
def std_dev(numbers: List[float]) -> float:
"""
Calculate the standard deviation (square root of variance).
Args:
numbers: List of floats or ints
Returns:
The standard deviation as a float
Raises:
ValueError: If the list is empty or has only one element
TypeError: If numbers contains non-numeric values
Example:
>>> std_dev([1, 2, 3])
0.8164965809004287
"""
if not numbers:
logger.error("Cannot calculate std dev of empty list")
raise ValueError("Cannot calculate std dev of empty list")
try:
var = variance(numbers)
result = var ** 0.5
logger.debug(f"Calculated std dev: {result}")
return result
except (ValueError, TypeError) as e:
raise
def is_significant(p_value: float, alpha: float = 0.05) -> bool:
"""
Determine if a p-value is statistically significant.
In plain English: Is this result unlikely to happen by chance?
If p_value < alpha, we say "yes, this is surprising" (significant).
Args:
p_value: The p-value (must be between 0 and 1)
alpha: The significance level (default 0.05 = 5%)
Returns:
True if p_value < alpha, False otherwise
Raises:
ValueError: If p_value or alpha are outside [0, 1]
Example:
>>> is_significant(0.03, 0.05)
True
>>> is_significant(0.08, 0.05)
False
"""
if not (0 <= p_value <= 1):
logger.error(f"p_value {p_value} is outside [0, 1]")
raise ValueError("p_value must be between 0 and 1")
if not (0 <= alpha <= 1):
logger.error(f"alpha {alpha} is outside [0, 1]")
raise ValueError("alpha must be between 0 and 1")
result = p_value < alpha
logger.debug(f"p_value={p_value}, alpha={alpha} → significant={result}")
return result
def bayes_update(
prior: float,
likelihood: float,
likelihood_complement: Optional[float] = None
) -> float:
"""
Update a prior probability using Bayes' Theorem.
In plain English: You had a guess (prior). You saw new evidence (likelihood).
What should your new guess be (posterior)?
Bayes' Theorem: P(A|B) = P(B|A) * P(A) / P(B)
Args:
prior: Your initial guess (0 to 1)
likelihood: Probability of evidence given your guess (0 to 1)
likelihood_complement: Probability of evidence given NOT your guess.
If None, assumed to be 1 - likelihood
Returns:
The updated probability (posterior)
Raises:
ValueError: If any probability is outside [0, 1]
Example:
>>> bayes_update(0.5, 0.9)
0.6428571428571429
"""
# Validate inputs
if not (0 <= prior <= 1):
logger.error(f"prior {prior} is outside [0, 1]")
raise ValueError("prior must be between 0 and 1")
if not (0 <= likelihood <= 1):
logger.error(f"likelihood {likelihood} is outside [0, 1]")
raise ValueError("likelihood must be between 0 and 1")
# Default likelihood_complement
if likelihood_complement is None:
likelihood_complement = 1 - likelihood
if not (0 <= likelihood_complement <= 1):
logger.error(f"likelihood_complement {likelihood_complement} is outside [0, 1]")
raise ValueError("likelihood_complement must be between 0 and 1")
# Bayes' Theorem
posterior = (
likelihood * prior /
(likelihood * prior + likelihood_complement * (1 - prior))
)
logger.info(
f"Bayes update: prior={prior} → posterior={posterior:.4f}"
)
return posterior
Create tests/conftest.py:
"""
Pytest configuration and shared fixtures for stats tests.
"""
import pytest
from typing import List
@pytest.fixture
def simple_numbers() -> List[float]:
"""Simple dataset for testing."""
return [1.0, 2.0, 3.0, 4.0, 5.0]
@pytest.fixture
def large_numbers() -> List[float]:
"""Larger dataset for testing edge cases."""
return list(range(1, 101)) # 1 to 100
@pytest.fixture
def negative_numbers() -> List[float]:
"""Numbers including negatives."""
return [-5.0, -2.0, 0.0, 2.0, 5.0]
@pytest.fixture
def single_number() -> List[float]:
"""Single value (edge case)."""
return [42.0]
@pytest.fixture
def duplicate_numbers() -> List[float]:
"""All values the same."""
return [7.0, 7.0, 7.0, 7.0]
Create tests/test_stats.py:
"""
Complete test suite for stats module.
Day 10: pytest introduction with happy paths, edge cases, and parametrisation.
"""
import pytest
import logging
from typing import List
from stats import mean, variance, std_dev, is_significant, bayes_update
# Get logger for test output
logger = logging.getLogger(__name__)
# ============================================================================
# TESTS FOR mean()
# ============================================================================
class TestMean:
"""Test the mean() function."""
def test_mean_happy_path(self, simple_numbers):
"""Test normal case: positive integers."""
result = mean(simple_numbers)
logger.info(f"mean(simple_numbers) = {result}")
assert result == 3.0
def test_mean_single_number(self, single_number):
"""Edge case: single value."""
result = mean(single_number)
logger.info(f"mean(single_number) = {result}")
assert result == 42.0
def test_mean_all_same(self, duplicate_numbers):
"""Edge case: all values identical."""
result = mean(duplicate_numbers)
logger.info(f"mean(duplicate_numbers) = {result}")
assert result == 7.0
def test_mean_with_negatives(self, negative_numbers):
"""Include negative numbers."""
result = mean(negative_numbers)
logger.info(f"mean(negative_numbers) = {result}")
assert result == pytest.approx(0.0)
def test_mean_empty_list_raises(self):
"""Empty list should raise ValueError."""
logger.info("Testing mean([]) raises ValueError")
with pytest.raises(ValueError, match="Cannot calculate mean"):
mean([])
def test_mean_with_floats(self):
"""Floats should work correctly."""
result = mean([1.5, 2.5, 3.5])
logger.info(f"mean([1.5, 2.5, 3.5]) = {result}")
assert result == pytest.approx(2.5)
@pytest.mark.parametrize("numbers,expected", [
([1, 2, 3], 2.0),
([10, 20], 15.0),
([5], 5.0),
([-1, -2, -3], -2.0),
([0, 0, 0], 0.0),
([1.1, 2.2, 3.3], 2.2),
])
def test_mean_parametrised(self, numbers, expected):
"""Parametrised: test many cases at once."""
result = mean(numbers)
logger.info(f"mean({numbers}) = {result}, expected = {expected}")
assert result == pytest.approx(expected)
def test_mean_non_numeric_raises(self):
"""Non-numeric values should raise TypeError."""
logger.info("Testing mean() with non-numeric values")
with pytest.raises(TypeError):
mean([1, 2, "three"])
# ============================================================================
# TESTS FOR variance()
# ============================================================================
class TestVariance:
"""Test the variance() function."""
def test_variance_happy_path(self, simple_numbers):
"""Test normal case."""
result = variance(simple_numbers)
expected = 2.0
logger.info(f"variance(simple_numbers) = {result}, expected = {expected}")
assert result == pytest.approx(expected)
def test_variance_two_elements(self):
"""Edge case: minimum valid input."""
result = variance([1.0, 3.0])
expected = 1.0
logger.info(f"variance([1.0, 3.0]) = {result}, expected = {expected}")
assert result == pytest.approx(expected)
def test_variance_all_same(self, duplicate_numbers):
"""Edge case: variance of identical values is zero."""
result = variance(duplicate_numbers)
logger.info(f"variance(duplicate_numbers) = {result}, expected = 0.0")
assert result == pytest.approx(0.0)
def test_variance_empty_list_raises(self):
"""Empty list should raise ValueError."""
logger.info("Testing variance([]) raises ValueError")
with pytest.raises(ValueError, match="Cannot calculate variance"):
variance([])
def test_variance_single_element_raises(self):
"""Single element is not enough for variance."""
logger.info("Testing variance([42.0]) raises ValueError")
with pytest.raises(ValueError, match="at least 2 values"):
variance([42.0])
def test_variance_negative_numbers(self, negative_numbers):
"""Variance of numbers including negatives."""
result = variance(negative_numbers)
expected = 10.4
logger.info(f"variance(negative_numbers) = {result}, expected = {expected}")
assert result == pytest.approx(expected)
@pytest.mark.parametrize("numbers,expected", [
([1, 2, 3], 2.0/3.0),
([1, 3], 1.0),
([0, 0], 0.0),
([2, 2, 2, 2], 0.0),
])
def test_variance_parametrised(self, numbers, expected):
"""Parametrised variance tests."""
result = variance(numbers)
logger.info(f"variance({numbers}) = {result}, expected = {expected}")
assert result == pytest.approx(expected)
# ============================================================================
# TESTS FOR std_dev()
# ============================================================================
class TestStdDev:
"""Test the std_dev() function."""
def test_std_dev_happy_path(self, simple_numbers):
"""Test normal case."""
result = std_dev(simple_numbers)
expected = 2.0 ** 0.5
logger.info(f"std_dev(simple_numbers) = {result}, expected = {expected}")
assert result == pytest.approx(expected)
def test_std_dev_all_same(self, duplicate_numbers):
"""Edge case: std dev of identical values is zero."""
result = std_dev(duplicate_numbers)
logger.info(f"std_dev(duplicate_numbers) = {result}, expected = 0.0")
assert result == pytest.approx(0.0)
def test_std_dev_empty_list_raises(self):
"""Empty list should raise ValueError."""
logger.info("Testing std_dev([]) raises ValueError")
with pytest.raises(ValueError):
std_dev([])
def test_std_dev_single_element_raises(self):
"""Single element is not enough."""
logger.info("Testing std_dev([42.0]) raises ValueError")
with pytest.raises(ValueError):
std_dev([42.0])
# ============================================================================
# TESTS FOR is_significant()
# ============================================================================
class TestIsSignificant:
"""Test the is_significant() function."""
def test_significant_below_threshold(self):
"""p-value < alpha → True."""
result = is_significant(0.03, 0.05)
logger.info(f"is_significant(0.03, 0.05) = {result}, expected = True")
assert result is True
def test_not_significant_above_threshold(self):
"""p-value > alpha → False."""
result = is_significant(0.08, 0.05)
logger.info(f"is_significant(0.08, 0.05) = {result}, expected = False")
assert result is False
def test_boundary_equals_alpha(self):
"""p-value == alpha → False (not strictly less)."""
result = is_significant(0.05, 0.05)
logger.info(f"is_significant(0.05, 0.05) = {result}, expected = False")
assert result is False
def test_boundary_just_below_alpha(self):
"""Just below alpha threshold."""
result = is_significant(0.049999, 0.05)
logger.info(f"is_significant(0.049999, 0.05) = {result}, expected = True")
assert result is True
def test_custom_alpha(self):
"""Different significance level."""
result = is_significant(0.08, alpha=0.10)
logger.info(f"is_significant(0.08, alpha=0.10) = {result}, expected = True")
assert result is True
def test_p_value_zero(self):
"""p-value = 0 is always significant."""
result = is_significant(0.0, 0.05)
logger.info(f"is_significant(0.0, 0.05) = {result}, expected = True")
assert result is True
def test_p_value_one(self):
"""p-value = 1 is never significant."""
result = is_significant(1.0, 0.05)
logger.info(f"is_significant(1.0, 0.05) = {result}, expected = False")
assert result is False
def test_invalid_p_value_raises(self):
"""p-value outside [0, 1] raises ValueError."""
logger.info("Testing is_significant with invalid p_value")
with pytest.raises(ValueError, match="p_value must be between"):
is_significant(-0.05, 0.05)
with pytest.raises(ValueError, match="p_value must be between"):
is_significant(1.5, 0.05)
def test_invalid_alpha_raises(self):
"""alpha outside [0, 1] raises ValueError."""
logger.info("Testing is_significant with invalid alpha")
with pytest.raises(ValueError, match="alpha must be between"):
is_significant(0.05, -0.05)
@pytest.mark.parametrize("p_value,alpha,expected", [
(0.01, 0.05, True),
(0.05, 0.05, False),
(0.10, 0.05, False),
(0.001, 0.01, True),
(0.0, 0.05, True),
(1.0, 0.05, False),
])
def test_is_significant_parametrised(self, p_value, alpha, expected):
"""Parametrised tests for various thresholds."""
result = is_significant(p_value, alpha)
logger.info(f"is_significant({p_value}, {alpha}) = {result}, expected = {expected}")
assert result is expected
# ============================================================================
# TESTS FOR bayes_update()
# ============================================================================
class TestBayesUpdate:
"""Test the bayes_update() function."""
def test_bayes_update_basic(self):
"""Test standard Bayes' Theorem calculation."""
result = bayes_update(0.5, 0.9)
expected = 0.5 * 0.9 / (0.5 * 0.9 + 0.5 * 0.1)
logger.info(f"bayes_update(0.5, 0.9) = {result}, expected ≈ {expected}")
assert result == pytest.approx(expected)
assert result == pytest.approx(0.9)
def test_bayes_update_weak_prior(self):
"""Low prior gets updated strongly by evidence."""
result = bayes_update(prior=0.1, likelihood=0.8)
expected = 0.8 * 0.1 / (0.8 * 0.1 + 0.2 * 0.9)
logger.info(f"bayes_update(0.1, 0.8) = {result}, expected ≈ {expected}")
assert result == pytest.approx(expected)
assert result > 0.1 # Posterior > prior
def test_bayes_update_strong_prior(self):
"""High prior stays high unless evidence is strong."""
result = bayes_update(prior=0.9, likelihood=0.6)
logger.info(f"bayes_update(0.9, 0.6) = {result}, expected > 0.9")
assert result > 0.9 # Still high
def test_bayes_update_explicit_complement(self):
"""Can specify likelihood_complement explicitly."""
result1 = bayes_update(0.5, 0.8, likelihood_complement=0.3)
result2 = bayes_update(0.5, 0.8) # Default: 1 - 0.8 = 0.2
logger.info(f"explicit complement: {result1}, default complement: {result2}")
assert result1 != result2
def test_bayes_update_zero_prior(self):
"""Posterior is zero if prior is zero."""
result = bayes_update(0.0, 0.9)
logger.info(f"bayes_update(0.0, 0.9) = {result}, expected = 0.0")
assert result == 0.0
def test_bayes_update_one_prior(self):
"""Posterior stays high if prior is very high."""
result = bayes_update(1.0, 0.5)
logger.info(f"bayes_update(1.0, 0.5) = {result}, expected = 1.0")
assert result == 1.0
def test_bayes_update_invalid_prior_raises(self):
"""Prior outside [0, 1] raises ValueError."""
logger.info("Testing bayes_update with invalid prior")
with pytest.raises(ValueError, match="prior must be between"):
bayes_update(-0.1, 0.5)
with pytest.raises(ValueError, match="prior must be between"):
bayes_update(1.5, 0.5)
def test_bayes_update_invalid_likelihood_raises(self):
"""Likelihood outside [0, 1] raises ValueError."""
logger.info("Testing bayes_update with invalid likelihood")
with pytest.raises(ValueError, match="likelihood must be between"):
bayes_update(0.5, 1.5)
def test_bayes_update_invalid_complement_raises(self):
"""Complement outside [0, 1] raises ValueError."""
logger.info("Testing bayes_update with invalid complement")
with pytest.raises(ValueError, match="likelihood_complement must be between"):
bayes_update(0.5, 0.8, likelihood_complement=1.5)
@pytest.mark.parametrize("prior,likelihood,expected_comparison", [
(0.5, 0.9, "greater"), # Strong evidence increases posterior
(0.5, 0.5, "equal"), # Equal evidence keeps it steady
(0.5, 0.1, "less"), # Weak evidence decreases posterior
])
def test_bayes_update_parametrised(self, prior, likelihood, expected_comparison):
"""Parametrised Bayes update tests."""
result = bayes_update(prior, likelihood)
logger.info(f"bayes_update({prior}, {likelihood}) = {result}")
if expected_comparison == "greater":
assert result > prior or result == pytest.approx(prior)
elif expected_comparison == "equal":
assert result == pytest.approx(prior)
elif expected_comparison == "less":
assert result < prior or result == pytest.approx(prior)
touch tests/__init__.py
This makes tests/ a Python package.
# Make sure you're in the project root
cd ~/workspace/day10-testing
# Make sure venv is activated
source .venv/bin/activate
# Run all tests
pytest -v
# Run with coverage
pytest --cov=stats --cov-report=term-missing
# Run one test class
pytest -v tests/test_stats.py::TestMean
# Run one test
pytest -v tests/test_stats.py::TestMean::test_mean_happy_path
# Show logging output
pytest -v -s
Expected output from pytest -v:
tests/test_stats.py::TestMean::test_mean_happy_path PASSED [ 5%]
tests/test_stats.py::TestMean::test_mean_single_number PASSED [10%]
tests/test_stats.py::TestMean::test_mean_all_same PASSED [15%]
tests/test_stats.py::TestMean::test_mean_with_negatives PASSED [20%]
tests/test_stats.py::TestMean::test_mean_empty_list_raises PASSED [25%]
tests/test_stats.py::TestMean::test_mean_with_floats PASSED [30%]
tests/test_stats.py::TestMean::test_mean_parametrised[numbers0-expected0] PASSED [35%]
... (more tests)
tests/test_stats.py::TestBayesUpdate::test_bayes_update_parametrised[prior2-likelihood2-less] PASSED [100%]
======================== 60 passed in 0.34s ========================
Expected output from pytest --cov=stats --cov-report=term-missing:
Name Stmts Miss Cover Missing
stats.py 145 0 100%
Day 11 will cover async/await and Python packaging.
]]>Here’s a uncomfortable truth: you cannot do AI without statistics.
Every machine learning model makes predictions. Every prediction has uncertainty. Every decision about “is this model good?” requires statistical thinking. You’ll hear phrases like “p-value,” “confidence interval,” “null hypothesis” and if you don’t understand them, you’ll make expensive mistakes.
The good news: statistics isn’t magic. It’s applied common sense with numbers. Today, we’re building from the ground up - no assumptions about what you know.
Installation:
pip install numpy scipy matplotlib
Python setup (copy this into every script today):
import logging
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
from typing import tuple, dict, list
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
We’re using logging instead of print() (Day 7’s lesson) and type hints on every function (Day 5). No exceptions, no Pydantic, no pytest (that’s tomorrow, Day 10).
You check your weather app. It says: “30% chance of rain tomorrow.”
That 30% is a probability. It’s a number between 0 and 1 (or 0% and 100%) that tells you how likely something is to happen.
In AI, probability answers questions like:
Sample space = all possible outcomes.
We write P(A) to mean “the probability of event A.”
Example:
# Fair coin flip
P(heads) = 0.5 # 50% chance
P(tails) = 0.5 # 50% chance
# Fair die roll
P(rolling a 6) = 1/6 ≈ 0.167 # 16.7% chance
The probabilities of all outcomes must add up to 1 (certainty).
def complement_rule(p_a: float) -> float:
"""If P(A) is true, then P(not A) = 1 - P(A)."""
return 1 - p_a
logger.info(f"P(heads) = 0.5, P(not heads) = {complement_rule(0.5)}")
# Output: P(not heads) = 0.5
If two events are independent (one doesn’t affect the other), multiply their probabilities.
Example: Two coin flips
p_heads_first = 0.5
p_heads_second = 0.5
p_both_heads = p_heads_first * p_heads_second
logger.info(f"P(HH) = {p_both_heads}")
# Output: P(HH) = 0.25
Another example: Spam detection
Imagine:
The or rule accounts for overlap (when both can happen).
def addition_rule(p_a: float, p_b: float, p_both: float) -> float:
"""P(A or B) = P(A) + P(B) - P(A and B)."""
return p_a + p_b - p_both
# Die roll: P(rolling 1 or 2)
p_1 = 1/6
p_2 = 1/6
p_both = 0 # Can't roll both 1 AND 2 at the same time
p_1_or_2 = addition_rule(p_1, p_2, p_both)
logger.info(f"P(1 or 2) = {p_1_or_2:.4f}")
# Output: P(1 or 2) = 0.3333
| **P(A | B)** means “probability of A given B happened.” |
The vertical bar | is read as “given.”
Example: Weather and clouds
P(rain | clouds) = "probability of rain given we see clouds"
Formula:
P(A | B) = P(A and B) / P(B)
The denominator reduces the sample space to only cases where B happened.
Worked example:
def conditional_probability(p_a_and_b: float, p_b: float) -> float:
"""P(A | B) = P(A and B) / P(B)."""
if p_b == 0:
logger.error("P(B) cannot be zero")
return 0.0
return p_a_and_b / p_b
# Medical test scenario
# P(positive test AND has disease) = 0.0099
# P(positive test) = 0.0101
p_disease_given_positive = conditional_probability(0.0099, 0.0101)
logger.info(f"P(disease | positive test) = {p_disease_given_positive:.4f}")
# Output: P(disease | positive test) ≈ 0.98 (98%)
Let’s use Python’s random module to simulate probability:
import random
def simulate_coin_flips(n: int) -> dict[str, int]:
"""Flip a coin n times, count heads and tails."""
heads = sum(1 for _ in range(n) if random.random() < 0.5)
tails = n - heads
logger.info(f"Simulated {n} flips: {heads} heads, {tails} tails")
return {'heads': heads, 'tails': tails}
def simulate_die_rolls(n: int) -> dict[int, int]:
"""Roll a die n times, count occurrences of each face."""
results = {i: 0 for i in range(1, 7)}
for _ in range(n):
roll = random.randint(1, 6)
results[roll] += 1
logger.info(f"Simulated {n} die rolls:")
for face, count in results.items():
logger.info(f" Face {face}: {count} ({count/n:.1%})")
return results
# Run simulations
simulate_coin_flips(1000)
simulate_die_rolls(1000)
Expected output:
Simulated 1000 flips: 509 heads, 491 tails
Simulated 1000 die rolls:
Face 1: 168 (16.8%)
Face 2: 167 (16.7%)
Face 3: 172 (17.2%)
...
Imagine rolling a die 1000 times and making a histogram:
Frequency
| ___
| | |
| | |
| | |
|____|___|____
1 2 3 ... 6
Each face appears roughly 1/6 of the time (≈ 167 times). That histogram is a distribution - it shows the probability of each outcome.
In a uniform distribution, every outcome is equally likely.
Real-world examples:
import matplotlib.pyplot as plt
def plot_uniform_distribution() -> None:
"""Visualize uniform distribution."""
# Generate 10,000 random samples between 0 and 100
samples = np.random.uniform(0, 100, 10000)
plt.figure(figsize=(10, 5))
plt.hist(samples, bins=50, edgecolor='black', alpha=0.7)
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.title('Uniform Distribution: 10,000 Samples from [0, 100]')
plt.grid(alpha=0.3)
plt.savefig('uniform_distribution.png', dpi=150)
logger.info("Saved uniform distribution plot")
plt.close()
plot_uniform_distribution()
The normal distribution is the most important distribution in statistics. It looks like a bell.
Real-world examples:
Key parameters:
def plot_normal_distribution() -> None:
"""Visualize normal distribution."""
# Generate 10,000 samples from a normal distribution
# mean=100 (like IQ), std=15
samples = np.random.normal(loc=100, scale=15, size=10000)
plt.figure(figsize=(10, 5))
plt.hist(samples, bins=50, edgecolor='black', alpha=0.7, density=True)
# Overlay theoretical normal curve
x = np.linspace(samples.min(), samples.max(), 100)
plt.plot(x, stats.norm.pdf(x, loc=100, scale=15), 'r-', linewidth=2, label='Theory')
plt.xlabel('Value')
plt.ylabel('Density')
plt.title('Normal Distribution: μ=100, σ=15 (like IQ scores)')
plt.legend()
plt.grid(alpha=0.3)
plt.savefig('normal_distribution.png', dpi=150)
logger.info("Saved normal distribution plot")
plt.close()
plot_normal_distribution()
In a normal distribution:
Example: IQ scores (mean=100, std=15)
def demonstrate_68_95_99_7_rule() -> None:
"""Explain the empirical rule."""
mean, std = 100, 15
# 1 std: [85, 115]
logger.info(f"68% of IQ scores between {mean-std} and {mean+std}")
# 2 stds: [70, 130]
logger.info(f"95% of IQ scores between {mean-2*std} and {mean+2*std}")
# 3 stds: [55, 145]
logger.info(f"99.7% of IQ scores between {mean-3*std} and {mean+3*std}")
# Verify with simulation
samples = np.random.normal(loc=mean, scale=std, size=100000)
within_1 = np.sum((samples >= mean - std) & (samples <= mean + std)) / len(samples)
within_2 = np.sum((samples >= mean - 2*std) & (samples <= mean + 2*std)) / len(samples)
within_3 = np.sum((samples >= mean - 3*std) & (samples <= mean + 3*std)) / len(samples)
logger.info(f"Simulated: {within_1:.1%} within 1 std (expected 68%)")
logger.info(f"Simulated: {within_2:.1%} within 2 stds (expected 95%)")
logger.info(f"Simulated: {within_3:.1%} within 3 stds (expected 99.7%)")
demonstrate_68_95_99_7_rule()
Output:
Simulated: 68.1% within 1 std (expected 68%)
Simulated: 95.2% within 2 stds (expected 95%)
Simulated: 99.7% within 3 stds (expected 99.7%)
The binomial distribution answers: “If I do N independent trials, each with success probability p, how many successes do I get?”
Real-world examples:
def plot_binomial_distribution() -> None:
"""Visualize binomial distribution."""
# Flip a fair coin 10 times, repeat 10,000 times
# Count heads each time
samples = np.random.binomial(n=10, p=0.5, size=10000)
plt.figure(figsize=(10, 5))
plt.hist(samples, bins=11, edgecolor='black', alpha=0.7, align='left')
plt.xlabel('Number of Heads (out of 10 flips)')
plt.ylabel('Frequency')
plt.title('Binomial Distribution: n=10 flips, p=0.5')
plt.grid(alpha=0.3)
plt.savefig('binomial_distribution.png', dpi=150)
logger.info("Saved binomial distribution plot")
plt.close()
plot_binomial_distribution()
Here’s one of the most important theorems in statistics:
If you take samples from ANY distribution and average them, those averages follow a normal distribution.
This is huge because:
def demonstrate_central_limit_theorem() -> None:
"""Show that sample means become normal."""
logger.info("Demonstrating Central Limit Theorem...")
# Original distribution: uniform (NOT normal)
original = np.random.uniform(0, 100, 100000)
# Take 1000 samples of size 50, compute mean of each
sample_means = []
for _ in range(1000):
sample = np.random.choice(original, size=50, replace=True)
sample_means.append(np.mean(sample))
sample_means = np.array(sample_means)
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Plot 1: Original distribution (uniform)
axes[0].hist(original, bins=50, alpha=0.7, edgecolor='black')
axes[0].set_title('Original Distribution: Uniform [0, 100]')
axes[0].set_xlabel('Value')
axes[0].set_ylabel('Frequency')
# Plot 2: Distribution of sample means (normal!)
axes[1].hist(sample_means, bins=50, alpha=0.7, edgecolor='black')
axes[1].set_title('Distribution of Sample Means (1000 trials, n=50)')
axes[1].set_xlabel('Mean Value')
axes[1].set_ylabel('Frequency')
plt.tight_layout()
plt.savefig('clt_demonstration.png', dpi=150)
logger.info("Saved CLT demonstration plot")
plt.close()
logger.info(f"Original distribution: uniform")
logger.info(f"Sample means: μ={np.mean(sample_means):.2f}, σ={np.std(sample_means):.2f}")
logger.info("Sample means distribution looks normal!")
demonstrate_central_limit_theorem()
The mean is the sum of all values divided by the count.
def compute_mean(data: list[float]) -> float:
"""Compute mean from scratch."""
if not data:
logger.error("Cannot compute mean of empty list")
return 0.0
return sum(data) / len(data)
# Example: exam scores
scores = [85, 92, 78, 88, 95, 81]
mean = compute_mean(scores)
logger.info(f"Mean score: {mean:.2f}")
# Verify with NumPy
mean_numpy = np.mean(scores)
logger.info(f"NumPy mean: {mean_numpy:.2f}")
When to use mean:
When NOT to use:
The median is the middle value when sorted.
def compute_median(data: list[float]) -> float:
"""Compute median from scratch."""
if not data:
logger.error("Cannot compute median of empty list")
return 0.0
sorted_data = sorted(data)
n = len(sorted_data)
if n % 2 == 1:
# Odd number of elements
return float(sorted_data[n // 2])
else:
# Even number of elements: average the two middle
mid1 = sorted_data[n // 2 - 1]
mid2 = sorted_data[n // 2]
return (mid1 + mid2) / 2
scores = [85, 92, 78, 88, 95, 81]
median = compute_median(scores)
logger.info(f"Median score: {median:.2f}")
# Verify with NumPy
median_numpy = np.median(scores)
logger.info(f"NumPy median: {median_numpy:.2f}")
When to use median:
The mode is the value that appears most often.
def compute_mode(data: list[float]) -> float:
"""Find the most frequent value."""
if not data:
logger.error("Cannot compute mode of empty list")
return 0.0
from collections import Counter
counts = Counter(data)
most_common, _ = counts.most_common(1)[0]
return float(most_common)
scores = [85, 92, 78, 88, 95, 81, 88] # 88 appears twice
mode = compute_mode(scores)
logger.info(f"Mode score: {mode:.2f}")
Variance is the average squared distance from the mean.
σ² = (Σ(x - μ)²) / n
Why squared? So negative differences don’t cancel out positive ones.
def compute_variance(data: list[float]) -> float:
"""Compute variance from scratch."""
if len(data) < 2:
logger.error("Need at least 2 data points")
return 0.0
mean = compute_mean(data)
squared_diffs = [(x - mean) ** 2 for x in data]
variance = sum(squared_diffs) / (len(data) - 1) # -1 for sample variance
return variance
scores = [85, 92, 78, 88, 95, 81]
variance = compute_variance(scores)
logger.info(f"Variance: {variance:.2f}")
# Verify with NumPy
variance_numpy = np.var(scores, ddof=1)
logger.info(f"NumPy variance: {variance_numpy:.2f}")
Problem: Variance is in squared units. Hard to interpret.
Standard deviation is the square root of variance.
σ = √(σ²)
Advantage: Same units as the original data.
def compute_std(data: list[float]) -> float:
"""Compute standard deviation from scratch."""
variance = compute_variance(data)
return variance ** 0.5
scores = [85, 92, 78, 88, 95, 81]
std = compute_std(scores)
logger.info(f"Std dev: {std:.2f}")
# Verify with NumPy
std_numpy = np.std(scores, ddof=1)
logger.info(f"NumPy std: {std_numpy:.2f}")
Interpretation: If mean=88 and std=6, most students scored between 82–94.
Percentile: Value below which a certain percentage of data falls.
def compute_percentiles(data: list[float]) -> dict[str, float]:
"""Compute quartiles and common percentiles."""
arr = np.array(data)
return {
'min': float(np.min(arr)),
'q25': float(np.percentile(arr, 25)),
'median': float(np.median(arr)),
'q75': float(np.percentile(arr, 75)),
'max': float(np.max(arr)),
}
scores = [85, 92, 78, 88, 95, 81, 90, 87]
percentiles = compute_percentiles(scores)
logger.info("Percentile breakdown:")
for name, value in percentiles.items():
logger.info(f" {name}: {value:.2f}")
You just tested positive for a rare disease.
Naturally, you panic. The test is 99% accurate, so you must be 99% likely to have the disease, right?
Wrong. This is where Bayes’ theorem saves the day.
P(A | B) = P(B | A) × P(A) / P(B)
Where:
| **P(A | B)** = Posterior (what we want: probability of disease given positive test?) |
| **P(B | A)** = Likelihood (test accuracy: if you have disease, what’s probability of positive test?) |
Setup:
Step 1: Prior probability (base rate)
p_disease = 1 / 10000
logger.info(f"P(disease) = {p_disease:.6f}") # 0.0001
Step 2: Likelihood (test accuracy)
p_positive_if_disease = 0.99 # Test catches 99% of true positives
p_positive_if_no_disease = 0.01 # Test gives 1% false positives
logger.info(f"P(positive | disease) = {p_positive_if_disease}")
logger.info(f"P(positive | no disease) = {p_positive_if_no_disease}")
Step 3: Total probability P(B)
P(positive) = P(positive | disease) × P(disease) + P(positive | no disease) × P(no disease)
p_no_disease = 1 - p_disease
p_positive = (p_positive_if_disease * p_disease) + \
(p_positive_if_no_disease * p_no_disease)
logger.info(f"P(positive) = {p_positive:.6f}")
Step 4: Bayes’ theorem
def bayes_theorem(likelihood: float, prior: float, evidence: float) -> float:
"""Compute posterior using Bayes' theorem."""
if evidence <= 0:
logger.error("Evidence must be positive")
return 0.0
return (likelihood * prior) / evidence
p_disease_given_positive = bayes_theorem(
likelihood=p_positive_if_disease,
prior=p_disease,
evidence=p_positive
)
logger.info(f"P(disease | positive test) = {p_disease_given_positive:.4f}")
logger.info(f"You are only {p_disease_given_positive * 100:.2f}% likely to have the disease!")
Output:
P(disease | positive test) = 0.0099
You are only 0.99% likely to have the disease!
Wow! Even with a positive test from a 99% accurate test, you’re less than 1% likely to have the disease. Why?
Because the disease is so rare. False positives (1% of healthy people) outnumber true positives (99% of the tiny number who have it).
Let’s use Bayes’ theorem to classify emails as spam or not spam, without any ML library.
class SimpleSpamClassifier:
"""Spam detector using Bayes' theorem."""
def __init__(self, prior_spam: float) -> None:
"""Initialize with prior probability of spam."""
self.prior_spam = prior_spam
self.prior_ham = 1 - prior_spam
logger.info(f"Initialized: P(spam)={prior_spam}, P(ham)={self.prior_ham}")
def classify(self,
p_word_in_spam: float,
p_word_in_ham: float) -> tuple[float, float]:
"""Classify based on presence of a keyword.
Returns: (p_spam, p_ham) posteriors
"""
# P(word | spam)
# P(word | ham)
# P(spam | word) = ?
p_word = (p_word_in_spam * self.prior_spam) + \
(p_word_in_ham * self.prior_ham)
p_spam_given_word = (p_word_in_spam * self.prior_spam) / p_word
p_ham_given_word = (p_word_in_ham * self.prior_ham) / p_word
logger.debug(f"P(spam | word) = {p_spam_given_word:.4f}")
return p_spam_given_word, p_ham_given_word
# Create classifier
# Assume 20% of emails are spam
classifier = SimpleSpamClassifier(prior_spam=0.20)
# Word: "free"
# 80% of spam emails contain "free"
# 10% of legitimate emails contain "free"
p_spam, p_ham = classifier.classify(
p_word_in_spam=0.80,
p_word_in_ham=0.10
)
logger.info(f"Email contains 'free': {p_spam:.1%} spam, {p_ham:.1%} ham")
# Word: "unsubscribe"
# 5% of spam emails contain "unsubscribe" (to avoid filters)
# 30% of legitimate emails contain "unsubscribe" (newsletters)
p_spam, p_ham = classifier.classify(
p_word_in_spam=0.05,
p_word_in_ham=0.30
)
logger.info(f"Email contains 'unsubscribe': {p_spam:.1%} spam, {p_ham:.1%} ham")
Output:
Initialized: P(spam)=0.20, P(ham)=0.80
Email contains 'free': 64.0% spam, 36.0% ham
Email contains 'unsubscribe': 4.8% spam, 95.2% ham
See how “free” strongly suggests spam, while “unsubscribe” suggests legitimacy? That’s Bayes’ theorem at work.
A pharmaceutical company tests a new drug on 100 patients:
Difference: 5 days faster.
But is the drug actually working, or did we just get lucky?
Hypothesis testing answers this question statistically.
Null Hypothesis (H₀): No effect exists
H₀: Drug has no effect on recovery time
Alternative Hypothesis (H₁): Effect exists
H₁: Drug does affect recovery time
The null hypothesis is the “boring” one. We try to disprove it.
logger.info("H₀: Drug has no effect")
logger.info("H₁: Drug does have an effect")
p-value = “If H₀ were true (drug does nothing), how often would we observe a difference this extreme or larger, by pure chance?”
# Example: we observe a 5-day difference
# p-value = 0.03
logger.info("p-value = 0.03")
logger.info("Interpretation: 3% chance of seeing this data if the drug is useless")
Critical misconceptions:
❌ WRONG: “p-value = 0.03 means 97% chance the drug works” ✅ RIGHT: “p-value = 0.03 means 3% chance we’d see this data if the drug doesn’t work”
We choose a significance level, typically α = 0.05.
Decision rule:
p_value = 0.03
alpha = 0.05
if p_value < alpha:
logger.info("✓ Reject H₀: The effect is statistically significant")
else:
logger.info("✗ Fail to reject H₀: No significant effect detected")
Every hypothesis test can make two kinds of mistakes:
Type I Error (False Positive):
Type II Error (False Negative):
logger.info("Type I Error (False Positive):")
logger.info(" Claim drug works when it doesn't")
logger.info(" Probability: α = 0.05")
logger.info("")
logger.info("Type II Error (False Negative):")
logger.info(" Claim drug doesn't work when it does")
logger.info(" Probability: β ≈ 0.20 (related to statistical power)")
The t-test compares means of two groups.
Question: Are the treatment and control groups significantly different?
import numpy as np
from scipy.stats import ttest_ind
# Generate synthetic data
np.random.seed(42)
control = np.random.normal(loc=10, scale=2, size=50) # mean=10 days
treatment = np.random.normal(loc=5, scale=2, size=50) # mean=5 days
logger.info(f"Control group: mean={np.mean(control):.2f}, std={np.std(control):.2f}")
logger.info(f"Treatment group: mean={np.mean(treatment):.2f}, std={np.std(treatment):.2f}")
# Run t-test
t_statistic, p_value = ttest_ind(control, treatment)
logger.info(f"t-statistic: {t_statistic:.4f}")
logger.info(f"p-value: {p_value:.6f}")
# Decision
alpha = 0.05
if p_value < alpha:
logger.info(f"✓ Reject H₀ (p={p_value:.4f} < α={alpha})")
logger.info(" The groups are significantly different")
else:
logger.info(f"✗ Fail to reject H₀ (p={p_value:.4f} ≥ α={alpha})")
logger.info(" No significant difference detected")
Output:
Control group: mean=10.12, std=2.05
Treatment group: mean=5.08, std=1.87
t-statistic: 12.7832
p-value: 0.000000
✓ Reject H₀ (p=0.000000 < α=0.05)
The groups are significantly different
Common misconception: ❌ “There’s a 95% probability the true mean is in this interval”
Correct interpretation: ✅ “If we repeated this experiment 100 times, 95 of those experiments would produce intervals containing the true mean”
The interval is random (depends on our sample). The true mean is fixed.
Formula:
CI = mean ± (critical value × standard error)
from scipy import stats
import numpy as np
def compute_confidence_interval(data: np.ndarray,
confidence: float = 0.95) -> tuple[float, float]:
"""Compute confidence interval for the mean."""
n = len(data)
mean = np.mean(data)
# Standard error of the mean
sem = stats.sem(data) # = std / sqrt(n)
# t-critical value (use t-distribution because sample is small-ish)
df = n - 1
alpha = 1 - confidence
t_crit = stats.t.ppf(1 - alpha/2, df)
# Margin of error
margin = t_crit * sem
logger.debug(f"n={n}, mean={mean:.2f}, sem={sem:.4f}, margin={margin:.2f}")
return (mean - margin, mean + margin)
# Example: exam scores
scores = np.array([85, 92, 78, 88, 95, 81, 90, 87, 83, 89])
lower, upper = compute_confidence_interval(scores, confidence=0.95)
logger.info(f"Sample mean: {np.mean(scores):.2f}")
logger.info(f"95% Confidence Interval: [{lower:.2f}, {upper:.2f}]")
logger.info("Interpretation: We're 95% confident the true population mean")
logger.info(f" is between {lower:.2f} and {upper:.2f}")
Output:
Sample mean: 86.80
95% Confidence Interval: [82.47, 91.13]
def plot_confidence_intervals() -> None:
"""Visualize CIs for two groups."""
np.random.seed(42)
# Generate data
group1 = np.random.normal(loc=80, scale=8, size=50)
group2 = np.random.normal(loc=85, scale=8, size=50)
# Compute CIs
ci1 = compute_confidence_interval(group1)
ci2 = compute_confidence_interval(group2)
mean1 = np.mean(group1)
mean2 = np.mean(group2)
# Plot
fig, ax = plt.subplots(figsize=(10, 6))
groups = ['Group 1', 'Group 2']
means = [mean1, mean2]
errors = [
[mean1 - ci1[0], mean2 - ci2[0]],
[ci1[1] - mean1, ci2[1] - mean2]
]
ax.errorbar(groups, means, yerr=errors, fmt='o', markersize=10,
capsize=10, capthick=2, linewidth=2)
ax.set_ylabel('Mean Score')
ax.set_title('95% Confidence Intervals for Mean Scores')
ax.grid(axis='y', alpha=0.3)
plt.tight_layout()
plt.savefig('confidence_intervals.png', dpi=150)
logger.info("Saved confidence interval plot")
plt.close()
plot_confidence_intervals()
Let’s bring it all together. We’ll:
Complete script:
import logging
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
from typing import dict, tuple, list
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
# ============================================================================
# DATA GENERATION
# ============================================================================
def generate_exam_scores() -> tuple[np.ndarray, np.ndarray]:
"""Generate synthetic exam scores for study vs no-study groups."""
np.random.seed(42)
# Students who studied: higher mean (78), lower variance
study_group = np.random.normal(loc=78, scale=8, size=50)
# Students who didn't study: lower mean (68), higher variance
no_study_group = np.random.normal(loc=68, scale=10, size=50)
logger.info(f"Generated {len(study_group)} study group scores")
logger.info(f"Generated {len(no_study_group)} no-study group scores")
return study_group, no_study_group
# ============================================================================
# DESCRIPTIVE STATISTICS
# ============================================================================
def compute_stats(data: np.ndarray) -> dict[str, float]:
"""Compute 7-point summary statistics."""
logger.debug(f"Computing stats for {len(data)} values")
return {
'mean': float(np.mean(data)),
'median': float(np.median(data)),
'std': float(np.std(data, ddof=1)),
'min': float(np.min(data)),
'max': float(np.max(data)),
'q25': float(np.percentile(data, 25)),
'q75': float(np.percentile(data, 75)),
}
def log_stats(name: str, stats_dict: dict[str, float]) -> None:
"""Log statistics nicely."""
logger.info(f"{name} statistics:")
for key, value in stats_dict.items():
logger.info(f" {key:8s}: {value:7.2f}")
# ============================================================================
# HYPOTHESIS TESTING: t-test
# ============================================================================
def run_t_test(group1: np.ndarray, group2: np.ndarray) -> tuple[float, float]:
"""Run independent samples t-test."""
t_stat, p_value = stats.ttest_ind(group1, group2)
logger.info(f"t-test results:")
logger.info(f" t-statistic: {t_stat:.4f}")
logger.info(f" p-value: {p_value:.6f}")
if p_value < 0.05:
logger.info(" ✓ Statistically significant difference (p < 0.05)")
else:
logger.info(" ✗ No statistically significant difference (p >= 0.05)")
return t_stat, p_value
# ============================================================================
# CONFIDENCE INTERVALS
# ============================================================================
def compute_ci(data: np.ndarray, confidence: float = 0.95) -> tuple[float, float]:
"""Compute confidence interval for the mean."""
mean = np.mean(data)
sem = stats.sem(data)
df = len(data) - 1
alpha = 1 - confidence
t_crit = stats.t.ppf(1 - alpha/2, df)
margin = t_crit * sem
logger.debug(f"CI: mean={mean:.2f}, sem={sem:.4f}, margin={margin:.2f}")
return (mean - margin, mean + margin)
# ============================================================================
# VISUALIZATION
# ============================================================================
def plot_score_distributions(study: np.ndarray, no_study: np.ndarray) -> None:
"""Plot overlaid histograms of both groups."""
fig, ax = plt.subplots(figsize=(12, 6))
# Histograms
ax.hist(study, bins=15, alpha=0.6, label='Study Group', color='green', edgecolor='black')
ax.hist(no_study, bins=15, alpha=0.6, label='No-Study Group', color='red', edgecolor='black')
ax.set_xlabel('Exam Score')
ax.set_ylabel('Frequency')
ax.set_title('Distribution of Exam Scores by Study Status')
ax.legend()
ax.grid(alpha=0.3)
plt.tight_layout()
plt.savefig('output/plots/score_distribution.png', dpi=150)
logger.info("Saved score distribution plot")
plt.close()
def plot_ci_comparison(study: np.ndarray, no_study: np.ndarray,
ci_study: tuple[float, float],
ci_no_study: tuple[float, float]) -> None:
"""Plot confidence intervals for both groups."""
fig, ax = plt.subplots(figsize=(10, 6))
groups = ['Study Group', 'No-Study Group']
means = [np.mean(study), np.mean(no_study)]
errors = [
[means[0] - ci_study[0], means[1] - ci_no_study[0]],
[ci_study[1] - means[0], ci_no_study[1] - means[1]]
]
ax.errorbar(groups, means, yerr=errors, fmt='o', markersize=12,
capsize=12, capthick=2, linewidth=2, color='blue')
ax.set_ylabel('Mean Score')
ax.set_title('95% Confidence Intervals for Mean Exam Scores')
ax.grid(axis='y', alpha=0.3)
# Add value labels
for i, (group, mean, ci) in enumerate(zip(groups, means, [ci_study, ci_no_study])):
ax.text(i, mean + 2, f'{mean:.1f}\n[{ci[0]:.1f}, {ci[1]:.1f}]',
ha='center', fontsize=10)
plt.tight_layout()
plt.savefig('output/plots/ci_comparison.png', dpi=150)
logger.info("Saved confidence interval comparison plot")
plt.close()
# ============================================================================
# MAIN ANALYSIS
# ============================================================================
def main() -> None:
"""Run complete statistical analysis."""
logger.info("Starting exam score analysis...")
# Generate data
study_group, no_study_group = generate_exam_scores()
# Descriptive statistics
logger.info("")
logger.info("=" * 60)
logger.info("DESCRIPTIVE STATISTICS")
logger.info("=" * 60)
study_stats = compute_stats(study_group)
no_study_stats = compute_stats(no_study_group)
log_stats("Study Group", study_stats)
logger.info("")
log_stats("No-Study Group", no_study_stats)
# Hypothesis test
logger.info("")
logger.info("=" * 60)
logger.info("HYPOTHESIS TESTING (t-test)")
logger.info("=" * 60)
t_stat, p_value = run_t_test(study_group, no_study_group)
# Confidence intervals
logger.info("")
logger.info("=" * 60)
logger.info("CONFIDENCE INTERVALS (95%)")
logger.info("=" * 60)
ci_study = compute_ci(study_group)
ci_no_study = compute_ci(no_study_group)
logger.info(f"Study Group CI: [{ci_study[0]:.2f}, {ci_study[1]:.2f}]")
logger.info(f"No-Study Group CI: [{ci_no_study[0]:.2f}, {ci_no_study[1]:.2f}]")
# Visualization
logger.info("")
logger.info("=" * 60)
logger.info("CREATING VISUALIZATIONS")
logger.info("=" * 60)
plot_score_distributions(study_group, no_study_group)
plot_ci_comparison(study_group, no_study_group, ci_study, ci_no_study)
logger.info("")
logger.info("=" * 60)
logger.info("ANALYSIS COMPLETE")
logger.info("=" * 60)
if __name__ == '__main__':
main()
Expected output:
2026-03-28 10:15:30,123 - __main__ - INFO - Starting exam score analysis...
2026-03-28 10:15:30,124 - __main__ - INFO - Generated 50 study group scores
2026-03-28 10:15:30,124 - __main__ - INFO - Generated 50 no-study group scores
============================================================
DESCRIPTIVE STATISTICS
============================================================
2026-03-28 10:15:30,125 - __main__ - INFO - Study Group statistics:
2026-03-28 10:15:30,125 - __main__ - INFO - mean : 78.41
2026-03-28 10:15:30,125 - __main__ - INFO - median : 78.27
2026-03-28 10:15:30,125 - __main__ - INFO - std : 8.12
2026-03-28 10:15:30,125 - __main__ - INFO - min : 60.53
2026-03-28 10:15:30,125 - __main__ - INFO - max : 95.18
2026-03-28 10:15:30,125 - __main__ - INFO - q25 : 71.98
2026-03-28 10:15:30,125 - __main__ - INFO - q75 : 84.87
2026-03-28 10:15:30,125 - __main__ - INFO - No-Study Group statistics:
2026-03-28 10:15:30,125 - __main__ - INFO - mean : 68.37
2026-03-28 10:15:30,125 - __main__ - INFO - median : 68.15
2026-03-28 10:15:30,125 - __main__ - INFO - std : 9.87
2026-03-28 10:15:30,125 - __main__ - INFO - min : 46.82
2026-03-28 10:15:30,125 - __main__ - INFO - max : 92.33
2026-03-28 10:15:30,125 - __main__ - INFO - q25 : 61.45
2026-03-28 10:15:30,125 - __main__ - INFO - q75 : 76.28
============================================================
HYPOTHESIS TESTING (t-test)
============================================================
2026-03-28 10:15:30,126 - __main__ - INFO - t-test results:
2026-03-28 10:15:30,126 - __main__ - INFO - t-statistic: 4.8732
2026-03-28 10:15:30,126 - __main__ - INFO - p-value: 0.000005
2026-03-28 10:15:30,126 - __main__ - INFO - ✓ Statistically significant difference (p < 0.05)
============================================================
CONFIDENCE INTERVALS (95%)
============================================================
2026-03-28 10:15:30,127 - __main__ - INFO - Study Group CI: [75.42, 81.40]
2026-03-28 10:15:30,127 - __main__ - INFO - No-Study Group CI: [65.10, 71.64]
============================================================
CREATING VISUALIZATIONS
============================================================
2026-03-28 10:15:30,200 - __main__ - INFO - Saved score distribution plot
2026-03-28 10:15:30,250 - __main__ - INFO - Saved confidence interval comparison plot
============================================================
ANALYSIS COMPLETE
============================================================
Day 10: pytest - Introduction to Automated Testing
pytest is the Python standard for testing. We’ll write:
You might think: “I can build AI models with scikit-learn or PyTorch. Why learn linear algebra and calculus?”
Here’s the truth: Every AI model is math. When you train a neural network, you’re:
Without understanding these concepts, you’ll treat AI like a black box. You won’t know when something works, why it fails, or how to debug it.
Today, we’re building from scratch. No frameworks. Just Python, NumPy, and your brain.
pip install numpy==1.26.4 matplotlib==3.8.2
mkdir -p ~/ai-journey/day-8
cd ~/ai-journey/day-8
All code files go in this directory.
Let’s start simple.
Scalar: Just a number.
x = 5
Vector: A list of numbers, usually arranged vertically (a column). Think of it as a point or direction in space.
v = [1, 2, 3] (a point in 3D space)
Matrix: A grid of numbers. Multiple vectors stacked together.
[1 2 3] (2×3 matrix: 2 rows, 3 columns)
A = [4 5 6]
[1, 2, 3] + [4, 5, 6] = [5, 7, 9] (add element-by-element)
[4, 5, 6] - [1, 2, 3] = [3, 3, 3]
3 × [2, 4, 6] = [6, 12, 18]
The magnitude tells you how long a vector is:
|v| = √(v₁² + v₂² + ... + vₙ²)
Example: The vector [3, 4] has magnitude:
|[3, 4]| = √(3² + 4²) = √(9 + 16) = √25 = 5
This is just the Pythagorean theorem!
The dot product of two vectors is computed as:
a · b = a₁×b₁ + a₂×b₂ + ... + aₙ×bₙ
Example:
[2, 3] · [4, 5] = (2×4) + (3×5) = 8 + 15 = 23
The dot product measures how aligned two vectors are:
This is used EVERYWHERE in AI:
We can normalize the dot product to get a number between -1 and +1:
cos(θ) = (a · b) / (|a| × |b|)
This is cosine similarity. It’s the fundamental metric for comparing embeddings in AI.
A matrix has dimensions rows × columns.
[1 2 3] ← 2 rows
A = [4 5 6]
↑ ↑ ↑
3 columns
A is a 2×3 matrix
Flip rows and columns:
[1 2 3] [1 4]
A = [4 5 6] A^T = [2 5]
[3 6]
Each row of the matrix is dot-producted with the vector:
[1 2] [2] (1×2 + 2×3) = 8
A = [3 4] v = [3] = (3×2 + 4×3) = 18
[5 6] (5×2 + 6×3) = 28
Result: [8, 18, 28]
Each element (i,j) in the result is the dot product of row i from the left matrix with column j from the right matrix.
Identity Matrix (I):
[1 0 0]
I = [0 1 0]
[0 0 1]
Property: A × I = A
Inverse Matrix (A⁻¹):
A × A⁻¹ = I
(Like division: x × (1/x) = 1)
Not all matrices have inverses. Those that don’t are called “singular” - no unique solution when solving Ax = b.
Eigenvalues & Eigenvectors:
Special vectors that don’t change direction when multiplied by a matrix (only scaled):
A × v = λ × v
λ = eigenvalue (how much the vector is scaled)
v = eigenvector (the vector itself)
(We’ll dive deep into eigenvalues on Day 12. For now, just know they exist.)
Analogy: You’re hiking on a mountain. At any point, the ground has a slope. That slope is the derivative. Steep = large derivative. Flat = zero derivative.
Formal definition: The derivative dy/dx is the rate of change of y with respect to x. It tells you how much y changes when x changes by a tiny amount.
We can compute derivatives without any fancy math rules. Here’s the formula:
f'(x) ≈ (f(x + h) - f(x)) / h
where h is tiny (like 1e-7).
This is saying: “Look at the function at x and at x+h. The slope between those two points approximates the true derivative.”
Example: Let’s say f(x) = x²
f'(2) ≈ (f(2.0000001) - f(2)) / 0.0000001
≈ (4.0000004 - 4) / 0.0000001
≈ 4 (which is correct! d/dx(x²) = 2x, so at x=2, derivative = 4)
Instead of computing numerically every time, mathematicians found patterns (rules):
d/dx(xⁿ) = n × x^(n-1)
Examples:
d/dx(x²) = 2x
d/dx(x³) = 3x²
d/dx(x^5) = 5x⁴
d/dx(u × v) = u' × v + u × v'
Example:
d/dx(x² × x³) = d/dx(x⁵) = 5x⁴
Or using the product rule:
= 2x × x³ + x² × 3x²
= 2x⁴ + 3x⁴
= 5x⁴ ✓
d/dx(f(g(x))) = f'(g(x)) × g'(x)
Analogy (gears): Imagine two gears meshed together. The outer gear turns at rate f’(g(x)). The inner gear turns at rate g’(x). The total rotation is their product.
Example:
d/dx((x² + 1)³) = ?
Let u = x² + 1, so we have f(u) = u³
f'(u) = 3u²
g(x) = x² + 1
g'(x) = 2x
By chain rule:
d/dx(u³) = 3u² × 2x = 3(x² + 1)² × 2x
When a function has multiple variables, you can take the derivative with respect to ONE of them, treating the others as constants.
f(x, y) = x² + 2xy + y²
∂f/∂x = 2x + 2y (treat y as constant)
∂f/∂y = 2x + 2y (treat x as constant)
The ∂ symbol just means “partial derivative” (derivative with respect to one variable).
The gradient is a vector of all partial derivatives. It points in the direction of steepest ascent.
For f(x, y):
∇f = [∂f/∂x, ∂f/∂y]
This vector points "uphill" the fastest.
If you want to find the minimum, move opposite to the gradient.
Gradient descent is an algorithm that minimizes a function by taking steps opposite to the gradient.
Algorithm:
x = initial_guess
learning_rate = 0.01
for i in range(max_iterations):
gradient = compute_gradient(x)
x = x - learning_rate × gradient
loss = f(x)
if gradient is near zero:
break
return x
The learning rate controls step size:
Now let’s implement everything.
Complete linear algebra and calculus implementations:
import logging
import numpy as np
from typing import List
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')
def vector_add(a: List[float], b: List[float]) -> List[float]:
"""Add two vectors element-wise."""
if len(a) != len(b):
raise ValueError(f"Vector lengths don't match: {len(a)} vs {len(b)}")
return [a[i] + b[i] for i in range(len(a))]
def vector_subtract(a: List[float], b: List[float]) -> List[float]:
"""Subtract vector b from vector a."""
if len(a) != len(b):
raise ValueError(f"Vector lengths don't match: {len(a)} vs {len(b)}")
return [a[i] - b[i] for i in range(len(a))]
def scalar_multiply(scalar: float, v: List[float]) -> List[float]:
"""Multiply a vector by a scalar."""
return [scalar * x for x in v]
def dot_product(a: List[float], b: List[float]) -> float:
"""Compute dot product of two vectors."""
if len(a) != len(b):
raise ValueError(f"Vectors must have same length: {len(a)} vs {len(b)}")
result = sum(a[i] * b[i] for i in range(len(a)))
return result
def vector_magnitude(v: List[float]) -> float:
"""Compute the magnitude (L2 norm) of a vector."""
return (sum(x**2 for x in v)) ** 0.5
def cosine_similarity(a: List[float], b: List[float]) -> float:
"""Compute cosine similarity between two vectors (range: -1 to 1)."""
mag_a = vector_magnitude(a)
mag_b = vector_magnitude(b)
if mag_a == 0 or mag_b == 0:
return 0.0
return dot_product(a, b) / (mag_a * mag_b)
def matrix_transpose(A: List[List[float]]) -> List[List[float]]:
"""Transpose a matrix (swap rows and columns)."""
rows = len(A)
cols = len(A[0])
return [[A[i][j] for i in range(rows)] for j in range(cols)]
def matrix_vector_multiply(A: List[List[float]], v: List[float]) -> List[float]:
"""Multiply a matrix by a vector."""
rows = len(A)
result = []
for i in range(rows):
dot = dot_product(A[i], v)
result.append(dot)
return result
def matrix_matrix_multiply(A: List[List[float]], B: List[List[float]]) -> List[List[float]]:
"""Multiply two matrices: A (m×n) × B (n×p) = C (m×p)."""
m = len(A)
n = len(A[0])
p = len(B[0])
# Transpose B for easier column access
B_T = matrix_transpose(B)
# Result is m×p
result = []
for i in range(m):
row = []
for j in range(p):
dot = dot_product(A[i], B_T[j])
row.append(dot)
result.append(row)
return result
def numerical_gradient(f, x: float, h: float = 1e-7) -> float:
"""Compute numerical derivative of f at point x using finite differences."""
return (f(x + h) - f(x)) / h
def print_vector(name: str, v: List[float]) -> None:
"""Helper: log a vector in readable format."""
logger.info(f"{name} = {[round(x, 4) for x in v]}")
def print_matrix(name: str, A: List[List[float]]) -> None:
"""Helper: log a matrix in readable format."""
logger.info(f"{name} =")
for row in A:
logger.info(f" {[round(x, 4) for x in row]}")
# Demonstration
if __name__ == "__main__":
logger.info("=== Linear Algebra Toolkit ===")
# Vectors
a = [1, 2, 3]
b = [4, 5, 6]
logger.info(f"a = {a}, b = {b}")
logger.info(f"a + b = {vector_add(a, b)}")
logger.info(f"a - b = {vector_subtract(a, b)}")
logger.info(f"3 × a = {scalar_multiply(3, a)}")
logger.info(f"a · b = {dot_product(a, b)}")
logger.info(f"|a| = {vector_magnitude(a):.4f}")
logger.info(f"cosine_similarity(a, b) = {cosine_similarity(a, b):.4f}")
# Matrices
logger.info("\n=== Matrix Operations ===")
A = [[1, 2, 3], [4, 5, 6]]
logger.info(f"A shape: {len(A)}×{len(A[0])}")
print_matrix("A", A)
print_matrix("A^T", matrix_transpose(A))
v = [1, 2, 3]
print_vector("v", v)
print_vector("A × v", matrix_vector_multiply(A, v))
# Calculus
logger.info("\n=== Numerical Derivatives ===")
f = lambda x: x**2
x = 2.0
derivative = numerical_gradient(f, x)
logger.info(f"f(x) = x², at x={x}: f'(x) ≈ {derivative:.4f} (true: 4.0)")
f2 = lambda x: x**3
derivative2 = numerical_gradient(f2, x)
logger.info(f"f(x) = x³, at x={x}: f'(x) ≈ {derivative2:.4f} (true: 12.0)")
Gradient descent implementation with visualization:
import logging
import numpy as np
import matplotlib.pyplot as plt
from typing import Callable, Tuple, List
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')
def numerical_gradient(f: Callable[[float], float], x: float, h: float = 1e-7) -> float:
"""Compute numerical gradient of f at point x."""
return (f(x + h) - f(x)) / h
def gradient_descent_1d(
f: Callable[[float], float],
x_init: float,
learning_rate: float,
max_iterations: int,
tolerance: float = 1e-6
) -> Tuple[float, List[float], List[float], List[float]]:
"""
Perform 1D gradient descent.
Returns:
final_x: The x value at the minimum
x_history: List of x values at each iteration
loss_history: List of loss values at each iteration
grad_history: List of gradient magnitudes at each iteration
"""
x = x_init
x_history = [x]
loss_history = [f(x)]
grad_history = []
for iteration in range(max_iterations):
# Compute gradient
grad = numerical_gradient(f, x)
grad_history.append(abs(grad))
# Update x
x = x - learning_rate * grad
x_history.append(x)
loss_history.append(f(x))
# Check convergence
if abs(grad) < tolerance:
logger.info(f"Converged at iteration {iteration}, gradient: {grad:.2e}")
break
if (iteration + 1) % 10 == 0:
logger.info(f"Iteration {iteration + 1}: x = {x:.6f}, loss = {f(x):.6f}, grad = {grad:.2e}")
return x, x_history, loss_history, grad_history
def visualize_gradient_descent(
f: Callable[[float], float],
x_history: List[float],
loss_history: List[float],
x_range: Tuple[float, float] = (-1, 5),
filename: str = "gradient_descent.png"
) -> None:
"""
Visualize the gradient descent process.
Creates a 2-panel plot:
- Left: Function curve with descent steps marked
- Right: Loss vs iteration
"""
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
# Left panel: function curve + descent path
x_vals = np.linspace(x_range[0], x_range[1], 300)
y_vals = [f(x) for x in x_vals]
ax1.plot(x_vals, y_vals, 'b-', linewidth=2, label='f(x)')
# Plot descent steps
for i in range(len(x_history) - 1):
x_curr = x_history[i]
y_curr = loss_history[i]
x_next = x_history[i + 1]
y_next = loss_history[i + 1]
# Draw arrow
ax1.annotate('', xy=(x_next, y_next), xytext=(x_curr, y_curr),
arrowprops=dict(arrowstyle='->', color='red', alpha=0.6, lw=1.5))
# Highlight start and end
ax1.plot(x_history[0], loss_history[0], 'go', markersize=10, label='Start')
ax1.plot(x_history[-1], loss_history[-1], 'r*', markersize=15, label='End (minimum)')
ax1.set_xlabel('x', fontsize=12)
ax1.set_ylabel('f(x)', fontsize=12)
ax1.set_title('Gradient Descent: Function & Descent Path', fontsize=14, fontweight='bold')
ax1.grid(True, alpha=0.3)
ax1.legend()
# Right panel: loss over iterations
iterations = range(len(loss_history))
ax2.plot(iterations, loss_history, 'b-', linewidth=2, marker='o', markersize=4)
ax2.set_xlabel('Iteration', fontsize=12)
ax2.set_ylabel('Loss f(x)', fontsize=12)
ax2.set_title('Loss Convergence', fontsize=14, fontweight='bold')
ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig(filename, dpi=150, bbox_inches='tight')
logger.info(f"Plot saved to {filename}")
plt.close()
# Demonstration
if __name__ == "__main__":
logger.info("=== Gradient Descent: 1D Example ===\n")
# Define function: f(x) = x² - 4x + 6
# Minimum at x = 2, value = 2
f = lambda x: x**2 - 4*x + 6
logger.info("Objective: minimize f(x) = x² - 4x + 6")
logger.info("True minimum: x = 2, f(2) = 2\n")
# Run gradient descent
x_init = 0.0
learning_rate = 0.1
max_iterations = 100
logger.info(f"Starting from x = {x_init}")
logger.info(f"Learning rate = {learning_rate}\n")
x_final, x_history, loss_history, grad_history = gradient_descent_1d(
f, x_init, learning_rate, max_iterations
)
logger.info(f"\nFinal x: {x_final:.6f}")
logger.info(f"Final loss: {f(x_final):.6f}")
logger.info(f"Iterations: {len(x_history) - 1}")
# Visualize
visualize_gradient_descent(f, x_history, loss_history, filename="gradient_descent.png")
# Try different learning rates
logger.info("\n=== Testing Different Learning Rates ===\n")
learning_rates = [0.01, 0.05, 0.1, 0.2]
for lr in learning_rates:
x_final, _, loss_hist, _ = gradient_descent_1d(f, 0.0, lr, 100)
logger.info(f"LR={lr}: x={x_final:.6f}, f(x)={f(x_final):.6f}, iterations={len(loss_hist)-1}")
Run examples to verify everything works:
import logging
from math_toolkit import (
vector_add, vector_subtract, scalar_multiply, dot_product,
vector_magnitude, cosine_similarity, matrix_transpose,
matrix_vector_multiply, numerical_gradient
)
from gradient_descent import gradient_descent_1d, visualize_gradient_descent
logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')
def test_linear_algebra() -> None:
"""Test linear algebra functions."""
logger.info("=== Testing Linear Algebra ===\n")
# Test dot product
a = [1, 2, 3]
b = [4, 5, 6]
expected_dot = 1*4 + 2*5 + 3*6 # 32
actual_dot = dot_product(a, b)
assert actual_dot == expected_dot, f"Dot product failed: {actual_dot} != {expected_dot}"
logger.info(f"✓ Dot product: {a} · {b} = {actual_dot}")
# Test vector magnitude
v = [3, 4]
expected_mag = 5.0
actual_mag = vector_magnitude(v)
assert abs(actual_mag - expected_mag) < 1e-6, f"Magnitude failed: {actual_mag} != {expected_mag}"
logger.info(f"✓ Magnitude: |{v}| = {actual_mag}")
# Test cosine similarity (same vector)
sim = cosine_similarity(a, a)
assert abs(sim - 1.0) < 1e-6, f"Cosine similarity failed: {sim} != 1.0"
logger.info(f"✓ Cosine similarity (same vector): {sim}")
# Test cosine similarity (perpendicular vectors)
perp1 = [1, 0]
perp2 = [0, 1]
sim_perp = cosine_similarity(perp1, perp2)
assert abs(sim_perp) < 1e-6, f"Cosine similarity (perpendicular) failed: {sim_perp} != 0"
logger.info(f"✓ Cosine similarity (perpendicular): {sim_perp}")
# Test matrix-vector multiplication
A = [[1, 2], [3, 4], [5, 6]] # 3×2
v = [1, 2]
result = matrix_vector_multiply(A, v)
expected = [1*1 + 2*2, 3*1 + 4*2, 5*1 + 6*2] # [5, 11, 17]
assert result == expected, f"Matrix-vector multiply failed: {result} != {expected}"
logger.info(f"✓ Matrix-vector multiply: {result}")
logger.info()
def test_calculus() -> None:
"""Test calculus functions."""
logger.info("=== Testing Calculus ===\n")
# Test numerical derivative of x²
f = lambda x: x**2
x = 3.0
deriv = numerical_gradient(f, x)
expected = 2*x # d/dx(x²) = 2x
assert abs(deriv - expected) < 1e-3, f"Derivative failed: {deriv} != {expected}"
logger.info(f"✓ d/dx(x²) at x={x}: {deriv:.4f} (expected: {expected})")
# Test numerical derivative of x³
g = lambda x: x**3
deriv_g = numerical_gradient(g, x)
expected_g = 3*x**2 # d/dx(x³) = 3x²
assert abs(deriv_g - expected_g) < 1e-2, f"Derivative failed: {deriv_g} != {expected_g}"
logger.info(f"✓ d/dx(x³) at x={x}: {deriv_g:.4f} (expected: {expected_g})")
logger.info()
def test_gradient_descent() -> None:
"""Test gradient descent."""
logger.info("=== Testing Gradient Descent ===\n")
# Minimize f(x) = (x-3)²
f = lambda x: (x - 3)**2
x_final, x_hist, loss_hist, _ = gradient_descent_1d(
f, x_init=0.0, learning_rate=0.1, max_iterations=100
)
# Should converge to x ≈ 3
assert abs(x_final - 3.0) < 0.01, f"Gradient descent failed: {x_final} != 3.0"
logger.info(f"✓ Minimized (x-3)²: x = {x_final:.6f} (expected: 3.0)")
logger.info(f"✓ Loss decreased from {loss_hist[0]:.6f} to {loss_hist[-1]:.6f}")
logger.info()
if __name__ == "__main__":
test_linear_algebra()
test_calculus()
test_gradient_descent()
logger.info("=== All Tests Passed! ===\n")
# From ~/ai-journey/day-8/
# Test basic functions
python test_examples.py
# Run gradient descent with visualization
python gradient_descent.py
INFO: === Linear Algebra Toolkit ===
INFO: a = [1, 2, 3], b = [4, 5, 6]
INFO: a + b = [5, 7, 9]
INFO: a - b = [-3, -3, -3]
INFO: 3 × a = [3, 6, 9]
INFO: a · b = 32
INFO: |a| = 3.7417
INFO: cosine_similarity(a, b) = 0.9746
INFO: === Matrix Operations ===
INFO: A shape: 2×3
INFO: A =
INFO: [1, 2, 3]
INFO: [4, 5, 6]
...
INFO: === Gradient Descent: 1D Example ===
INFO: Objective: minimize f(x) = x² - 4x + 6
INFO: True minimum: x = 2, f(2) = 2
INFO: Starting from x = 0.0
INFO: Learning rate = 0.1
INFO: Iteration 10: x = 1.851278, loss = 2.021889, grad = -0.30
INFO: Iteration 20: x = 1.989963, loss = 2.000100, grad = -0.02
INFO: Iteration 30: x = 1.999999, loss = 2.000000, grad = -0.00
INFO: Converged at iteration 31, gradient: 1.82e-07
INFO: Final x: 2.000000
INFO: Final loss: 2.000000
INFO: Iterations: 31
INFO: Plot saved to gradient_descent.png
Day 9: Probability & Statistics - You’ll learn distributions, Bayes’ theorem, and why probability underpins uncertainty in AI. Same logging and type hints standards continue!
Today marks a pivotal shift in how we write code. Up until now, you’ve been using print() to see what your programs do. Starting today, you’ll graduate to logging - the production-standard way that real engineers observe their code.
Here’s why this matters: Imagine you deploy a data analysis script to production, and it processes 1 million customer records at 3 AM. Something goes wrong. With print(), your output scrolls off the screen and you have no record of what happened. With logging, you have a timestamped, categorized, searchable log file that tells you exactly what went wrong, when, and what led up to it.
By the end of today, you’ll also have beautiful, publication-quality plots using Matplotlib and Seaborn. But logging comes first - it’s the more important production skill.
Before we start, install the packages for today:
pip install matplotlib==3.8.2 seaborn==0.13.1 python-json-logger==2.0.7
Why these versions? Pinned versions prevent surprise breaking changes. In real jobs, you’ll do this for every project.
print()Let’s say you’re processing student test scores:
# ❌ OLD WAY - DON'T DO THIS ANYMORE
print("Starting analysis...")
print("Loaded 100 records")
print("ERROR: Student 42 has no math score")
print("Finished!")
Problems:
logging Module# ✅ NEW WAY - USE THIS ALWAYS
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logger.info("Starting analysis...")
logger.info("Loaded 100 records")
logger.error("Student 42 has no math score")
logger.info("Finished!")
Benefits:
Think of driving a car through a city. Your log level is like a traffic light system:
🔵 DEBUG = Blue light (take the side streets) = "Checking if row 542 has email field"
Use: Development only. Too detailed for production.
🟢 INFO = Green light (proceed normally) = "Loaded 10,000 records successfully"
Use: Normal operation milestones. What's going right.
🟡 WARNING = Yellow light (caution ahead) = "5 records missing phone number"
Use: Unexpected but recoverable. Needs attention but won't crash.
🔴 ERROR = Red light (stop for hazard) = "Failed to connect to database, retrying..."
Use: Operation failed, but program continues. Needs fix soon.
🔴🔴 CRITICAL = Red light + alarm (emergency!) = "Out of disk space, cannot continue"
Use: Show-stoppers. Program must stop immediately.
In development, you show everything (DEBUG through CRITICAL). In production, you show only INFO and above (hide DEBUG noise).
import logging
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
# Now use it:
logger.info("Analysis started")
logger.warning("5 rows have missing data")
logger.error("Failed to save plot")
What does each part do?
level=logging.INFO: Show INFO, WARNING, ERROR, CRITICAL. Hide DEBUG (too noisy).format='...': Template for every log message.
%(asctime)s = timestamp (2026-03-30 14:32:05,123)%(name)s = module name (e.g., “data_analysis”)%(levelname)s = DEBUG/INFO/WARNING/ERROR/CRITICAL%(message)s = your messagelogging.getLogger(__name__): Create a logger named after this fileimport logging
logging.basicConfig(level=logging.DEBUG) # Show everything
logger = logging.getLogger(__name__)
logger.debug("Checking column 'email' in row 5") # 🔵 Dev detail
logger.info("Successfully loaded 10,000 records") # 🟢 Normal
logger.warning("Student #42 missing math score, using 0") # 🟡 Unexpected
logger.error("Failed to save plot: /output/ not writable") # 🔴 Problem
logger.critical("Database offline, cannot continue") # 🔴🔴 Fatal
Output:
DEBUG - Checking column 'email' in row 5
INFO - Successfully loaded 10,000 records
WARNING - Student #42 missing math score, using 0
ERROR - Failed to save plot: /output/ not writable
CRITICAL - Database offline, cannot continue
By default, basicConfig() only sends to console. What if you want file + console?
import logging
from logging.handlers import RotatingFileHandler
# Create logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# HANDLER 1: Console (show only INFO and above)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_formatter = logging.Formatter('%(levelname)-8s | %(message)s')
console_handler.setFormatter(console_formatter)
# HANDLER 2: File (save everything including DEBUG)
file_handler = RotatingFileHandler(
'app.log',
maxBytes=5_000_000, # Rotate at 5 MB
backupCount=3 # Keep 3 old files
)
file_handler.setLevel(logging.DEBUG)
file_formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
file_handler.setFormatter(file_formatter)
# Attach both handlers
logger.addHandler(console_handler)
logger.addHandler(file_handler)
# Now log messages go BOTH places
logger.debug("Detailed debugging info") # Only in app.log
logger.info("Normal operation") # Both console and app.log
logger.error("Something failed") # Both console and app.log
Result:
Console sees:
INFO | Loaded 10,000 records
WARNING | Student #42 missing data
ERROR | Failed to save plot
app.log saves:
2026-03-30 14:32:05,123 - __main__ - DEBUG - Checking if field 'email' exists
2026-03-30 14:32:05,234 - __main__ - INFO - Loaded 10,000 records
2026-03-30 14:32:05,345 - __main__ - WARNING - Student #42 missing data
2026-03-30 14:32:05,456 - __main__ - ERROR - Failed to save plot
If your script runs for days, app.log grows huge. Use RotatingFileHandler:
from logging.handlers import RotatingFileHandler
handler = RotatingFileHandler(
'app.log',
maxBytes=5_000_000, # 5 MB per file
backupCount=5 # Keep 5 old files
)
What happens:
app.log reaches 5 MB → rename to app.log.1app.log.1 → becomes app.log.2app.log.5 → deletedapp.log createdYou’ll never run out of disk space!
Libraries like requests (HTTP client) log a lot. You can silence them:
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
# Silence the requests library - only show WARNING and above
logging.getLogger("requests").setLevel(logging.WARNING)
logging.getLogger("urllib3").setLevel(logging.WARNING)
logger.debug("My app's debug info") # ✅ Shows
logger.info("requests library debug info would go here") # ❌ Hidden
Loggers form a hierarchy by name:
root
├── myapp (your main logger)
│ ├── myapp.database
│ └── myapp.analysis
└── requests (external library)
Change the parent, and all children follow.
Regular logs are text. JSON logs are machine-readable:
Regular log:
2026-03-30 14:32:05,123 - analysis - ERROR - Missing value in row 542
JSON log:
{"timestamp": "2026-03-30T14:32:05.123Z", "logger": "analysis", "level": "ERROR", "message": "Missing value in row 542", "row_id": 542}
JSON logs go into Datadog, Splunk, CloudWatch. Machines parse them, alert you, create dashboards automatically.
Setup (requires pip install python-json-logger):
import logging
from pythonjsonlogger import jsonlogger
logger = logging.getLogger()
handler = logging.FileHandler('app-json.log')
formatter = jsonlogger.JsonFormatter()
handler.setFormatter(formatter)
logger.addHandler(handler)
# Log with extra context
logger.info("Analysis complete", extra={
"rows_processed": 10000,
"duration_sec": 45.2,
"source": "database"
})
Output in app-json.log:
{"message": "Analysis complete", "rows_processed": 10000, "duration_sec": 45.2, "source": "database"}
Now tools like Datadog can parse this, create alerts (“if rows_processed < 1000, alert”), and build dashboards.
Don’t do this (state machine):
import matplotlib.pyplot as plt
plt.plot([1, 2, 3], [1, 4, 9])
plt.title("My Plot")
plt.show() # ❌ Doesn't work in scripts/servers/notebooks reliably
Do this (object-oriented):
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(10, 6))
ax.plot([1, 2, 3], [1, 4, 9], label='y=x²')
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_title('Quadratic Function')
ax.legend()
ax.grid(True, alpha=0.3)
fig.savefig('plot.png', dpi=150, bbox_inches='tight')
plt.close(fig)
Why OO wins:
plt.subplots(2, 2) for a 2×2 gridfig, ax = plt.subplots(figsize=(10, 6))
ax.plot([0, 1, 2, 3], [0, 1, 4, 9], marker='o', linewidth=2, label='y=x²')
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_title('Quadratic Function')
ax.legend()
ax.grid(True, alpha=0.3)
fig.savefig('line.png')
plt.close(fig)
fig, ax = plt.subplots(figsize=(10, 6))
ax.scatter([1, 2, 3, 4, 5], [1, 4, 2, 5, 3], s=100, alpha=0.6, color='red')
ax.set_xlabel('Feature X')
ax.set_ylabel('Feature Y')
ax.set_title('Relationship Between Features')
fig.savefig('scatter.png')
plt.close(fig)
fig, ax = plt.subplots(figsize=(10, 6))
categories = ['Q1', 'Q2', 'Q3', 'Q4']
values = [10, 24, 36, 18]
ax.bar(categories, values, color='steelblue', edgecolor='black')
ax.set_ylabel('Revenue ($K)')
ax.set_title('Quarterly Revenue')
fig.savefig('bar.png')
plt.close(fig)
fig, ax = plt.subplots(figsize=(10, 6))
scores = [85, 92, 78, 88, 75, 95, 82, 90, 77, 93]
ax.hist(scores, bins=5, color='green', edgecolor='black')
ax.set_xlabel('Score')
ax.set_ylabel('Frequency')
ax.set_title('Score Distribution')
fig.savefig('hist.png')
plt.close(fig)
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
# Top-left: line plot
axes[0, 0].plot([1, 2, 3], [1, 2, 3])
axes[0, 0].set_title('Line Plot')
# Top-right: scatter plot
axes[0, 1].scatter([1, 2, 3], [1, 2, 3])
axes[0, 1].set_title('Scatter Plot')
# Bottom-left: bar chart
axes[1, 0].bar(['A', 'B', 'C'], [1, 2, 3])
axes[1, 0].set_title('Bar Chart')
# Bottom-right: histogram
axes[1, 1].hist([1, 1, 2, 2, 2], bins=3)
axes[1, 1].set_title('Histogram')
fig.tight_layout()
fig.savefig('subplots.png')
plt.close(fig)
import matplotlib.pyplot as plt
import numpy as np
# Option 1: Use a built-in style
plt.style.use('seaborn-v0_8-darkgrid')
# Option 2: Custom colors
fig, ax = plt.subplots()
ax.plot([1, 2, 3], [1, 4, 9], color='#FF6B6B', linewidth=3)
# Option 3: Color map (gradient)
x = np.linspace(0, 10, 50)
colors = plt.cm.viridis(np.linspace(0, 1, len(x)))
for xi, color in zip(x, colors):
ax.scatter(xi, np.sin(xi), color=color, s=50)
fig.savefig('styled.png')
plt.close(fig)
Seaborn builds on Matplotlib. It makes statistical plots easier and prettier by default.
import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt
# Load your data
df = pd.read_csv('students.csv')
# Set theme once for all plots
sns.set_theme(style='darkgrid', palette='husl')
fig, ax = plt.subplots(figsize=(10, 6))
sns.histplot(data=df, x='score', bins=20, kde=True)
ax.set_title('Score Distribution with KDE')
fig.savefig('hist_kde.png')
plt.close(fig)
fig, ax = plt.subplots(figsize=(10, 6))
sns.boxplot(data=df, x='grade', y='score')
ax.set_title('Score by Grade')
fig.savefig('boxplot.png')
plt.close(fig)
fig, ax = plt.subplots(figsize=(10, 6))
sns.violinplot(data=df, x='grade', y='score')
ax.set_title('Score Distribution by Grade')
fig.savefig('violinplot.png')
plt.close(fig)
fig, ax = plt.subplots(figsize=(10, 8))
corr = df[['math', 'english', 'science', 'history']].corr()
sns.heatmap(corr, annot=True, fmt='.2f', cmap='coolwarm', square=True)
ax.set_title('Subject Score Correlations')
fig.savefig('heatmap.png')
plt.close(fig)
The heatmap shows which subjects’ scores are related:
# Shows scatter plots for every pair of numeric columns
sns.pairplot(df[['math', 'english', 'science', 'history']], hue='grade')
plt.savefig('pairplot.png')
plt.close()
fig, ax = plt.subplots(figsize=(10, 6))
sns.scatterplot(
data=df,
x='math',
y='english',
hue='grade', # Color by grade
size='attendance', # Size by attendance
s=200,
alpha=0.6
)
ax.set_title('Math vs English (colored by grade, sized by attendance)')
fig.savefig('scatter_advanced.png')
plt.close(fig)
You’ll build a complete data analysis pipeline that:
print() callsconfig.pyCentralized logging configuration:
"""Logging configuration for the data analysis pipeline."""
import logging
from pathlib import Path
from logging.handlers import RotatingFileHandler
def setup_logging(log_file: str = "output/analysis.log") -> logging.Logger:
"""Configure logging to console and file.
Args:
log_file: Path to log file
Returns:
Configured logger instance
"""
# Create output directory if it doesn't exist
Path("output").mkdir(exist_ok=True)
# Create logger
logger = logging.getLogger("data_analysis")
logger.setLevel(logging.DEBUG)
# Console handler (INFO and above only - clean output for user)
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
console_formatter = logging.Formatter('%(levelname)-8s | %(message)s')
console_handler.setFormatter(console_formatter)
# File handler (DEBUG and above - detailed logs for debugging)
file_handler = RotatingFileHandler(
log_file,
maxBytes=5_000_000, # Rotate at 5 MB
backupCount=3 # Keep 3 old files
)
file_handler.setLevel(logging.DEBUG)
file_formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
file_handler.setFormatter(file_formatter)
# Attach both handlers
logger.addHandler(console_handler)
logger.addHandler(file_handler)
logger.info("Logging system initialized")
return logger
data_generator.pyGenerate sample data:
"""Generate sample student score data."""
import csv
from pathlib import Path
def generate_sample_data(filename: str = "sample_data.csv") -> None:
"""Create sample CSV with 10 students and 4 subject scores.
Args:
filename: Output CSV filename
"""
data = [
["student_id", "math", "english", "science", "history"],
[1, 85, 78, 92, 88],
[2, 92, 88, 95, 91],
[3, 78, 85, 82, 79],
[4, 88, 92, 89, 94],
[5, 75, 72, 78, 75],
[6, 95, 91, 97, 96],
[7, 82, 86, 84, 87],
[8, 90, 89, 91, 88],
[9, 77, 79, 76, 81],
[10, 93, 94, 92, 95],
]
with open(filename, 'w', newline='') as f:
writer = csv.writer(f)
writer.writerows(data)
print(f"Generated {filename}")
if __name__ == "__main__":
generate_sample_data()
analysis.py (Main Script)Complete data analysis with logging:
"""Data analysis dashboard with logging."""
import logging
import csv
from pathlib import Path
from typing import List, Dict
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from config import setup_logging
# Initialize logging
logger = setup_logging()
def load_data(filepath: str) -> List[Dict[str, str]]:
"""Load CSV data into a list of dictionaries.
Args:
filepath: Path to CSV file
Returns:
List of dictionaries with student data
Raises:
FileNotFoundError: If CSV file doesn't exist
"""
logger.info(f"Loading data from {filepath}")
try:
with open(filepath, 'r') as f:
reader = csv.DictReader(f)
data = list(reader)
logger.info(f"Successfully loaded {len(data)} records")
return data
except FileNotFoundError:
logger.error(f"File not found: {filepath}")
raise
def validate_data(data: List[Dict[str, str]]) -> None:
"""Check for missing values in data.
Args:
data: List of dictionaries with student data
"""
logger.info("Validating data...")
for i, row in enumerate(data, 1):
for key, value in row.items():
if not value or value.strip() == '':
logger.warning(f"Row {i}: Missing {key}")
logger.info("Validation complete")
def create_output_dir() -> None:
"""Create output directories for plots."""
Path("output/plots").mkdir(parents=True, exist_ok=True)
logger.debug("Created output/plots directory")
def plot_score_distribution(data: List[Dict[str, str]]) -> None:
"""Create histogram of math scores.
Args:
data: List of dictionaries with student data
"""
logger.info("Creating score distribution plot...")
scores = [int(row['math']) for row in data]
fig, ax = plt.subplots(figsize=(10, 6))
ax.hist(scores, bins=8, color='steelblue', edgecolor='black')
ax.set_xlabel('Score')
ax.set_ylabel('Frequency')
ax.set_title('Distribution of Math Scores')
ax.grid(True, alpha=0.3)
filepath = "output/plots/score_distribution.png"
fig.savefig(filepath, dpi=150, bbox_inches='tight')
logger.info(f"Saved: {filepath}")
plt.close(fig)
def plot_correlation_heatmap(data: List[Dict[str, str]]) -> None:
"""Create correlation heatmap of all subjects.
Args:
data: List of dictionaries with student data
"""
logger.info("Creating correlation heatmap...")
subjects = ['math', 'english', 'science', 'history']
# Build score matrix
scores_dict = {}
for subject in subjects:
scores = [int(row[subject]) for row in data]
scores_dict[subject] = scores
# Calculate correlation
score_matrix = np.array([scores_dict[s] for s in subjects])
corr = np.corrcoef(score_matrix)
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, annot=True, fmt='.2f', cmap='coolwarm',
xticklabels=subjects, yticklabels=subjects,
square=True, ax=ax)
ax.set_title('Subject Score Correlations')
filepath = "output/plots/correlation_heatmap.png"
fig.savefig(filepath, dpi=150, bbox_inches='tight')
logger.info(f"Saved: {filepath}")
plt.close(fig)
def plot_subject_comparison(data: List[Dict[str, str]]) -> None:
"""Create bar chart comparing average scores by subject.
Args:
data: List of dictionaries with student data
"""
logger.info("Creating subject comparison plot...")
subjects = ['math', 'english', 'science', 'history']
averages = []
for subject in subjects:
avg = sum(int(row[subject]) for row in data) / len(data)
averages.append(avg)
logger.debug(f"Average {subject}: {avg:.2f}")
fig, ax = plt.subplots(figsize=(10, 6))
bars = ax.bar(subjects, averages,
color=['#FF6B6B', '#4ECDC4', '#45B7D1', '#FFA07A'])
ax.set_ylabel('Average Score')
ax.set_title('Average Score by Subject')
ax.set_ylim([0, 100])
# Add value labels on bars
for bar in bars:
height = bar.get_height()
ax.text(bar.get_x() + bar.get_width()/2., height,
f'{height:.1f}', ha='center', va='bottom', fontsize=10)
filepath = "output/plots/subject_comparison.png"
fig.savefig(filepath, dpi=150, bbox_inches='tight')
logger.info(f"Saved: {filepath}")
plt.close(fig)
def plot_student_scatter(data: List[Dict[str, str]]) -> None:
"""Create scatter plot: Math vs English.
Args:
data: List of dictionaries with student data
"""
logger.info("Creating student scatter plot...")
math_scores = [int(row['math']) for row in data]
english_scores = [int(row['english']) for row in data]
fig, ax = plt.subplots(figsize=(10, 6))
ax.scatter(math_scores, english_scores, s=100, alpha=0.6, color='purple')
ax.set_xlabel('Math Score')
ax.set_ylabel('English Score')
ax.set_title('Math vs English Scores')
ax.grid(True, alpha=0.3)
filepath = "output/plots/math_vs_english.png"
fig.savefig(filepath, dpi=150, bbox_inches='tight')
logger.info(f"Saved: {filepath}")
plt.close(fig)
def main() -> None:
"""Main analysis pipeline."""
logger.info("=" * 60)
logger.info("STARTING DATA ANALYSIS DASHBOARD")
logger.info("=" * 60)
try:
# Setup
create_output_dir()
# Load and validate
data = load_data("sample_data.csv")
validate_data(data)
# Generate plots
plot_score_distribution(data)
plot_correlation_heatmap(data)
plot_subject_comparison(data)
plot_student_scatter(data)
logger.info("=" * 60)
logger.info("ANALYSIS COMPLETE - All plots saved to output/plots/")
logger.info("=" * 60)
except Exception as e:
logger.error(f"Pipeline failed: {e}", exc_info=True)
raise
if __name__ == "__main__":
main()
# Step 1: Generate sample data
python data_generator.py
# Output: Generated sample_data.csv
# Step 2: Run analysis (creates plots + logs)
python analysis.py
# Step 3: View results
ls -lh output/plots/
cat output/analysis.log
INFO | Logging system initialized
INFO | ============================================================
INFO | STARTING DATA ANALYSIS DASHBOARD
INFO | ============================================================
INFO | Loading data from sample_data.csv
INFO | Successfully loaded 10 records
INFO | Validating data...
INFO | Validation complete
INFO | Creating score distribution plot...
INFO | Saved: output/plots/score_distribution.png
INFO | Creating correlation heatmap...
INFO | Saved: output/plots/correlation_heatmap.png
INFO | Creating subject comparison plot...
INFO | Saved: output/plots/subject_comparison.png
INFO | Creating student scatter plot...
INFO | Saved: output/plots/math_vs_english.png
INFO | ============================================================
INFO | ANALYSIS COMPLETE - All plots saved to output/plots/
INFO | ============================================================
2026-03-30 14:32:05,123 - data_analysis - INFO - Logging system initialized
2026-03-30 14:32:05,234 - data_analysis - INFO - ============================================================
2026-03-30 14:32:05,234 - data_analysis - INFO - STARTING DATA ANALYSIS DASHBOARD
2026-03-30 14:32:05,234 - data_analysis - INFO - ============================================================
2026-03-30 14:32:05,345 - data_analysis - INFO - Loading data from sample_data.csv
2026-03-30 14:32:05,456 - data_analysis - INFO - Successfully loaded 10 records
2026-03-30 14:32:05,567 - data_analysis - INFO - Validating data...
2026-03-30 14:32:05,678 - data_analysis - INFO - Validation complete
2026-03-30 14:32:05,789 - data_analysis - DEBUG - Average math: 87.50
2026-03-30 14:32:05,890 - data_analysis - DEBUG - Average english: 85.40
2026-03-30 14:32:06,001 - data_analysis - DEBUG - Average science: 88.60
2026-03-30 14:32:06,112 - data_analysis - DEBUG - Average history: 88.40
2026-03-30 14:32:06,223 - data_analysis - INFO - Created output/plots directory
2026-03-30 14:32:06,334 - data_analysis - INFO - Creating score distribution plot...
2026-03-30 14:32:06,445 - data_analysis - INFO - Saved: output/plots/score_distribution.png
...
Day 8: Linear Algebra & Calculus
Before we can train any AI model, we need to feed it data. But real-world data is messy:
NumPy and Pandas are the tools that handle this. They let you:
Every AI project starts here. You can’t train a model without data, and you can’t work with data without NumPy and Pandas.
Install the required libraries:
pip install numpy==1.26.3 pandas==2.1.3
Verify installation:
import numpy as np
import pandas as pd
print(f"NumPy version: {np.__version__}")
print(f"Pandas version: {pd.__version__}")
Think about a simple task: you have a list of 1 million student scores, and you need to multiply each one by 1.1 (like adding a 10% bonus).
# Python list way (SLOW)
scores = [85, 92, 78, 88, ...] # 1 million items
bonused = []
for score in scores:
bonused.append(score * 1.1) # loops 1 million times
This takes time because Python has to:
NumPy solves this:
import numpy as np
scores = np.array([85, 92, 78, 88, ...]) # 1 million items
bonused = scores * 1.1 # INSTANT - happens all at once!
NumPy does the multiplication on the entire array at the CPU level, not in Python. This is 50-100x faster.
An array is like a spreadsheet column (or grid) of numbers:
import numpy as np
# 1D array (like a column)
scores = np.array([85, 92, 78, 88])
# [85 92 78 88]
# 2D array (like a spreadsheet)
grades = np.array([
[85, 88, 92], # Alice's math, science, english
[92, 85, 88], # Bob's scores
[78, 92, 85] # Charlie's scores
])
# [[85 88 92]
# [92 85 88]
# [78 92 85]]
import numpy as np
# From a Python list
arr = np.array([1, 2, 3, 4, 5])
# [1 2 3 4 5]
# Zeros (useful for initializing)
zeros = np.zeros(5)
# [0. 0. 0. 0. 0.]
# Ones
ones = np.ones((3, 2))
# [[1. 1.]
# [1. 1.]
# [1. 1.]]
# Range (like Python's range(), but returns an array)
range_arr = np.arange(0, 10, 2)
# [0 2 4 6 8]
# Evenly spaced numbers (give me 5 numbers between 0 and 1)
linspace = np.linspace(0, 1, 5)
# [0. 0.25 0.5 0.75 1. ]
# Random numbers (for testing)
np.random.seed(42) # reproducible randomness
random_arr = np.random.randn(3, 3)
# [[ 0.49671415 -0.1382643 0.64589411]
# [-0.23415337 -0.23413696 1.57921282]
# [ 0.76743473 -0.46947439 0.54256004]]
arr = np.array([[1, 2, 3], [4, 5, 6]])
# How many rows and columns?
arr.shape # (2, 3) - 2 rows, 3 columns
# What type of data?
arr.dtype # dtype('int64') - integer, 64-bit
# How many dimensions?
arr.ndim # 2 - it's a table (2D)
# Total number of elements?
arr.size # 6
# Flip rows and columns
arr.T
# [[1 4]
# [2 5]
# [3 6]]
scores = np.array([85, 92, 78, 88, 90])
# Get the first score
scores[0] # 85
# Get the last score
scores[-1] # 90
# Get scores from position 1 to 3 (not including 4)
scores[1:4] # [92, 78, 88]
# Get every other score
scores[::2] # [85, 78, 90]
# For 2D arrays
grades = np.array([[85, 88, 92], [92, 85, 88]])
# Get Alice's science score (row 0, column 1)
grades[0, 1] # 88
# Get Charlie's entire row
grades[1, :] # [92, 85, 88]
# Get all math scores (entire column 0)
grades[:, 0] # [85, 92]
This is incredibly useful. Get only the values that match a condition:
scores = np.array([85, 92, 78, 88, 92])
# Which scores are 85 or higher?
high_scores = scores[scores >= 85]
# [85, 92, 88, 92]
# This creates a True/False array first:
scores >= 85
# [True, True, False, True, True]
# Then keeps only the True ones
Broadcasting is NumPy’s ability to make arrays of different sizes work together. Think of it like stretching a small image to fit a big frame:
# Simple example: adding 10 to every score
scores = np.array([80, 85, 90])
scores + 10
# [90, 95, 100]
# More complex: adding the same bonus to each subject
math_scores = np.array([85, 92, 78])
science_scores = np.array([88, 85, 92])
english_scores = np.array([92, 88, 85])
# Stack them into a table
grades = np.array([math_scores, science_scores, english_scores])
# [[85 92 78]
# [88 85 92]
# [92 88 85]]
# Add 5 points bonus to all grades
boosted = grades + 5
# [[90 97 83]
# [93 90 97]
# [97 93 90]]
# The 5 gets "broadcast" to all positions!
Instead of loops, use NumPy functions:
temps = np.array([72.5, 73.1, 71.8, 74.2, 72.9])
# Average temperature
np.mean(temps) # 72.9
# Standard deviation (how spread out are they?)
np.std(temps) # ~0.94
# Total temperature (if we added them all up)
np.sum(temps) # 364.5
# Highest temperature
np.max(temps) # 74.2
# Position of highest (0-indexed)
np.argmax(temps) # 3
# Lowest temperature
np.min(temps) # 71.8
# Distance from mean (how far is each from average?)
deviations = temps - np.mean(temps)
# [ -0.4 0.2 -1.1 1.3 0. ]
# Absolute values (ignore + or -)
abs_deviations = np.abs(deviations)
# [0.4, 0.2, 1.1, 1.3, 0.]
# Square each one (for variance calculation)
squared = temps ** 2
# [5256.25, 5343.21, 5155.24, 5505.84, 5314.41]
# Square root
square_root = np.sqrt(squared)
# Dot product (multiply matching elements and sum)
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
np.dot(a, b) # 1*4 + 2*5 + 3*6 = 32
arr = np.arange(12) # [0, 1, 2, ..., 11]
# Reshape to a 3x4 grid
grid = arr.reshape(3, 4)
# [[ 0 1 2 3]
# [ 4 5 6 7]
# [ 8 9 10 11]]
# Flatten back to 1D
flat = grid.flatten()
# [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
# Transpose (swap rows and columns)
transposed = grid.T
# [[ 0 4 8]
# [ 1 5 9]
# [ 2 6 10]
# [ 3 7 11]]
# Stack horizontally (side by side)
a = np.array([[1], [2]])
b = np.array([[3], [4]])
np.hstack([a, b])
# [[1 3]
# [2 4]]
# Stack vertically (on top)
np.vstack([a, b])
# [[1]
# [2]
# [3]
# [4]]
# Concatenate along an axis
np.concatenate([a, b], axis=0) # vertical
np.concatenate([a, b], axis=1) # horizontal
scores = np.array([85, 92, 78, 88, 92])
# Clip: constrain values to a range
# (if someone scores above 95, cap it at 95)
clipped = np.clip(scores, 0, 95)
# [85, 92, 78, 88, 92]
# Where: conditional operation
# (if score < 80, add 10; otherwise keep it)
adjusted = np.where(scores < 80, scores + 10, scores)
# [85, 92, 88, 88, 92]
# Unique: find all unique values
np.unique(scores)
# [78, 85, 88, 92]
# Linear algebra basics
A = np.array([[1, 2], [3, 4]])
# Determinant (single number describing the matrix)
np.linalg.det(A) # -2.0
# Inverse (the "opposite" of a matrix)
np.linalg.inv(A)
# [[-2. 1. ]
# [ 1.5 -0.5]]
# Norm (length/magnitude)
np.linalg.norm(A) # 5.477
NumPy is great for math. But real data has column names and row labels. That’s where Pandas comes in.
A Pandas DataFrame is like an Excel spreadsheet:
import pandas as pd
# Create a Series (like a column)
scores = pd.Series([85, 92, 78, 88], name='Math Scores')
# 0 85
# 1 92
# 2 78
# 3 88
# Name: Math Scores, dtype: int64
# Access by position
scores[0] # 85
# With custom index (row labels)
scores = pd.Series(
[85, 92, 78, 88],
index=['Alice', 'Bob', 'Charlie', 'Diana'],
name='Math Scores'
)
# Alice 85
# Bob 92
# Charlie 78
# Diana 88
# Access by label
scores['Alice'] # 85
# All operations work like NumPy
scores.mean() # 85.75
scores.max() # 92
scores.min() # 78
scores.std() # ~6.23
# Filter (keep scores above 85)
scores[scores > 85]
# Bob 92
# Diana 88
A DataFrame is multiple Series stacked side by side:
import pandas as pd
# Create from a dictionary (most common)
data = {
'Name': ['Alice', 'Bob', 'Charlie', 'Diana'],
'Grade': [10, 10, 11, 11],
'Math': [85, 92, 78, 88],
'Science': [88, 85, 92, 90],
'English': [92, 88, 85, 89]
}
df = pd.DataFrame(data)
# Name Grade Math Science English
# 0 Alice 10 85 88 92
# 1 Bob 10 92 85 88
# 2 Charlie 11 78 92 85
# 3 Diana 11 88 90 89
# How big is it?
df.shape # (4, 5) - 4 rows, 5 columns
# What are the column names?
df.columns # ['Name', 'Grade', 'Math', 'Science', 'English']
# What are the data types?
df.dtypes
# Name object (text)
# Grade int64 (integer)
# Math int64 (integer)
# Science int64 (integer)
# English int64 (integer)
# First few rows
df.head(2)
# Name Grade Math Science English
# 0 Alice 10 85 88 92
# 1 Bob 10 92 85 88
# Last few rows
df.tail(1)
# Name Grade Math Science English
# 3 Diana 11 88 90 89
# Summary statistics
df.describe()
# Grade Math Science English
# count 4.0 4.0 4.0 4.0
# mean 10.5 85.75 89.0 89.0
# std 0.577 6.179 3.559 2.708
# min 10.0 78.0 85.0 85.0
# 25% 10.0 82.5 87.0 86.5
# 50% 10.5 86.5 89.0 89.0
# 75% 11.0 89.5 91.0 90.5
# max 11.0 92.0 92.0 92.0
# Detailed info
df.info()
# <class 'pandas.core.frame.DataFrame'>
# RangeIndex: 4 entries, 0 to 3
# Data columns (but 4 non-null):
# Name 4 non-null object
# Grade 4 non-null int64
# Math 4 non-null int64
# Science 4 non-null int64
# English 4 non-null int64
# Get a single column (returns a Series)
df['Name']
# 0 Alice
# 1 Bob
# 2 Charlie
# 3 Diana
# Get multiple columns (returns a DataFrame)
df[['Name', 'Math']]
# Name Math
# 0 Alice 85
# 1 Bob 92
# 2 Charlie 78
# 3 Diana 88
# Get a row by position (iloc = integer location)
df.iloc[0]
# Name Alice
# Grade 10
# Math 85
# Science 88
# English 92
# Get a specific cell by position
df.iloc[0, 0] # Alice
# Get by label (loc = location by label)
df.loc[0, 'Name'] # Alice
# Get all rows where condition is true
df[df['Math'] > 85]
# Name Grade Math Science English
# 1 Bob 10 92 85 88
# 3 Diana 11 88 90 89
# Multiple conditions
df[(df['Math'] > 85) & (df['Science'] > 85)]
# Name Grade Math Science English
# 1 Bob 10 92 85 88
# 3 Diana 11 88 90 89
# Calculate average score
df['Average'] = (df['Math'] + df['Science'] + df['English']) / 3
df
# Name Grade Math Science English Average
# 0 Alice 10 85 88 92 88.33
# 1 Bob 10 92 85 88 88.33
# 2 Charlie 11 78 92 85 85.00
# 3 Diana 11 88 90 89 89.00
# Add a column with apply (custom function)
def grade_letter(score: float) -> str:
if score >= 90: return 'A'
elif score >= 80: return 'B'
else: return 'C'
df['GradeLetter'] = df['Average'].apply(grade_letter)
df
# Name Grade Math Science English Average GradeLetter
# 0 Alice 10 85 88 92 88.33 B
# 1 Bob 10 92 85 88 88.33 B
# 2 Charlie 11 78 92 85 85.00 B
# 3 Diana 11 88 90 89 89.00 B
# Drop a column
df = df.drop('GradeLetter', axis=1)
# Keep only certain columns
df = df[['Name', 'Math', 'Science', 'English']]
GroupBy is like taking a stack of papers, sorting them by grade level, then analyzing each pile:
# What's the average math score per grade level?
df.groupby('Grade')['Math'].mean()
# Grade
# 10 88.5
# 11 83.0
# Multiple subjects
df.groupby('Grade')[['Math', 'Science', 'English']].mean()
# Math Science English
# Grade
# 10 88.5 86.5 90.0
# 11 83.0 91.0 87.0
# More aggregations
df.groupby('Grade').agg({
'Math': ['mean', 'max', 'min'],
'Science': ['mean', 'max', 'min'],
'Name': 'count' # how many students per grade
})
# Math Science Name
# mean max min mean max min count
# Grade
# 10 88.5 92 85 86.5 88 85 2
# 11 83.0 88 78 91.0 92 90 2
# Count occurrences
df.groupby('Grade').size()
# Grade
# 10 2
# 11 2
Imagine you have student scores in one file and student names in another. Merge combines them:
# File 1: Scores
scores_df = pd.DataFrame({
'StudentID': [1, 2, 3],
'Math': [85, 92, 78]
})
# StudentID Math
# 0 1 85
# 1 2 92
# 2 3 78
# File 2: Names
names_df = pd.DataFrame({
'StudentID': [1, 2, 3],
'Name': ['Alice', 'Bob', 'Charlie']
})
# StudentID Name
# 0 1 Alice
# 1 2 Bob
# 2 3 Charlie
# INNER join (only keep rows in BOTH files)
merged = pd.merge(scores_df, names_df, on='StudentID')
# StudentID Math Name
# 0 1 85 Alice
# 1 2 92 Bob
# 2 3 78 Charlie
# LEFT join (keep all from left, add matching from right)
pd.merge(scores_df, names_df, on='StudentID', how='left')
# (same result if all IDs match)
# RIGHT join (keep all from right, add matching from left)
pd.merge(scores_df, names_df, on='StudentID', how='right')
# OUTER join (keep all rows from both)
pd.merge(scores_df, names_df, on='StudentID', how='outer')
def bonus_points(score: float) -> float:
"""Add 5% bonus to score."""
return score * 1.05
# Apply to a column
df['Math_Bonus'] = df['Math'].apply(bonus_points)
# With lambda (inline function)
df['Math2x'] = df['Math'].apply(lambda x: x * 2)
# Apply to entire row
def calc_total(row) -> float:
"""Add up all three subjects."""
return row['Math'] + row['Science'] + row['English']
df['Total'] = df.apply(calc_total, axis=1)
df
# Name Grade Math Science English Math_Bonus Math2x Total
# 0 Alice 10 85 88 92 89.25 170 265
# 1 Bob 10 92 85 88 96.60 184 265
# 2 Charlie 11 78 92 85 81.90 156 255
# 3 Diana 11 88 90 89 92.40 176 267
names = pd.Series(['Alice', 'Bob', 'Charlie'])
# Convert to lowercase
names.str.lower()
# 0 alice
# 1 bob
# 2 charlie
# Convert to uppercase
names.str.upper()
# 0 ALICE
# 1 BOB
# 2 CHARLIE
# Length of each string
names.str.len()
# 0 5
# 1 3
# 2 7
# Check if contains substring
names.str.contains('li')
# 0 True
# 1 False
# 2 True
# Replace characters
names.str.replace('a', 'A')
# 0 Alice
# 1 Bob
# 2 ChArlie
# Split by character
names.str.split('i')
# 0 [Al, ce]
# 1 [Bob]
# 2 [Charl, e]
dates = pd.to_datetime(['2024-01-15', '2024-06-20', '2024-12-25'])
# DatetimeIndex(['2024-01-15', '2024-06-20', '2024-12-25'],
# dtype='datetime64[ns]', freq=None)
# Extract parts
dates.dt.year
# 2024, 2024, 2024
dates.dt.month
# 1, 6, 12
dates.dt.day
# 15, 20, 25
dates.dt.day_name()
# 'Monday', 'Thursday', 'Monday'
# Calculate differences
(dates.max() - dates.min()).days
# 345 (days between first and last)
# Create data with missing values
data_with_nan = {
'Name': ['Alice', 'Bob', None, 'Diana'],
'Score': [85, None, 92, 88]
}
df_nan = pd.DataFrame(data_with_nan)
# Name Score
# 0 Alice 85.0
# 1 Bob NaN
# 2 None 92.0
# 3 Diana 88.0
# Check where data is missing
df_nan.isna()
# Name Score
# 0 False False
# 1 False True
# 2 True False
# 3 False False
# Drop rows with any missing values
df_clean = df_nan.dropna()
# Name Score
# 0 Alice 85.0
# 3 Diana 88.0
# Drop rows where Score is missing
df_partial = df_nan.dropna(subset=['Score'])
# Name Score
# 0 Alice 85.0
# 2 None 92.0
# 3 Diana 88.0
# Fill missing with a value
df_filled = df_nan.fillna(0)
# Name Score
# 0 Alice 85.0
# 1 Bob 0.0
# 2 0 92.0
# 3 Diana 88.0
# Fill with average
df_filled = df_nan.copy()
df_filled['Score'].fillna(df_nan['Score'].mean(), inplace=True)
# Name Score
# 0 Alice 85.0
# 1 Bob 88.333333
# 2 None 92.0
# 3 Diana 88.0
# Save to CSV
df.to_csv('students.csv', index=False)
# Load from CSV
df = pd.read_csv('students.csv')
# With options
df = pd.read_csv('students.csv',
delimiter=',', # column separator
index_col=0, # use first column as row index
dtype={'Grade': int}) # specify data types
Now let’s build a complete, professional analysis pipeline with type hints on every function.
Create a file named data.csv:
Name,Grade,Math,Science,English
Alice,10,85,88,92
Bob,10,92,85,88
Charlie,11,78,92,85
Diana,11,88,90,89
Eve,12,92,95,91
Frank,12,88,87,86
Grace,10,90,89,91
Henry,11,85,88,90
Create a file named analysis.py:
import numpy as np
import pandas as pd
from typing import Dict, Any
def load_data(filepath: str) -> pd.DataFrame:
"""Load student data from CSV file.
Args:
filepath: Path to CSV file
Returns:
DataFrame with student data
"""
return pd.read_csv(filepath)
def calculate_overall_score(df: pd.DataFrame) -> pd.DataFrame:
"""Add overall average score for each student.
Uses NumPy vectorization for efficiency.
Args:
df: DataFrame with Math, Science, English columns
Returns:
DataFrame with new 'Overall' column
"""
# Get the score columns as a NumPy array
scores = df[['Math', 'Science', 'English']].values
# Calculate mean across each row (axis=1)
overall = np.mean(scores, axis=1)
# Add as new column
df['Overall'] = overall
return df
def assign_grades(df: pd.DataFrame) -> pd.DataFrame:
"""Assign letter grades based on overall score.
Args:
df: DataFrame with 'Overall' column
Returns:
DataFrame with new 'GradeLetter' column
"""
def grade_letter(score: float) -> str:
"""Convert numeric score to letter grade."""
if score >= 90:
return 'A'
elif score >= 80:
return 'B'
else:
return 'C'
# Apply the function to the Overall column
df['GradeLetter'] = df['Overall'].apply(grade_letter)
return df
def grade_level_analysis(df: pd.DataFrame) -> pd.DataFrame:
"""Analyze performance by grade level.
Args:
df: DataFrame with Grade column and scores
Returns:
DataFrame with statistics per grade level
"""
stats = df.groupby('Grade').agg({
'Math': ['mean', 'max', 'min'],
'Science': ['mean', 'max', 'min'],
'English': ['mean', 'max', 'min'],
'Overall': 'mean',
'Name': 'count' # number of students
}).round(2)
return stats
def top_students(df: pd.DataFrame, n: int = 3) -> pd.DataFrame:
"""Get top N students by overall score.
Args:
df: DataFrame with student data
n: Number of top students to return
Returns:
DataFrame with top N students
"""
return df.nlargest(n, 'Overall')[['Name', 'Grade', 'Overall', 'GradeLetter']]
def subject_stats(df: pd.DataFrame) -> Dict[str, Dict[str, float]]:
"""Calculate statistics for each subject.
Uses NumPy for efficient calculation.
Args:
df: DataFrame with Math, Science, English columns
Returns:
Dictionary with stats for each subject
"""
subjects = ['Math', 'Science', 'English']
stats = {}
for subject in subjects:
# Convert to NumPy array for calculation
scores = df[subject].values
stats[subject] = {
'mean': float(np.mean(scores)),
'std': float(np.std(scores)),
'min': float(np.min(scores)),
'max': float(np.max(scores))
}
return stats
def correlations(df: pd.DataFrame) -> np.ndarray:
"""Calculate correlation matrix between subjects.
Uses NumPy correlation.
Args:
df: DataFrame with Math, Science, English columns
Returns:
Correlation matrix (3x3 array)
"""
subject_data = df[['Math', 'Science', 'English']].values
return np.corrcoef(subject_data.T)
def print_correlation_heatmap(corr_matrix: np.ndarray) -> None:
"""Print correlation matrix as ASCII table.
Args:
corr_matrix: 3x3 correlation matrix
"""
subjects = ['Math', 'Science', 'English']
print("\nCORRELATION MATRIX:")
print(" Math Science English")
for i, subj in enumerate(subjects):
print(f"{subj:7s} ", end="")
for j in range(len(subjects)):
val = corr_matrix[i, j]
print(f"{val:6.2f} ", end="")
print()
def top_n_per_subject(df: pd.DataFrame, n: int = 2) -> Dict[str, pd.DataFrame]:
"""Return top N students for each subject.
Args:
df: DataFrame with student data
n: Number of top students per subject
Returns:
Dictionary mapping subject to top N students
"""
subjects = ['Math', 'Science', 'English']
results = {}
for subject in subjects:
top = df.nlargest(n, subject)[['Name', subject]]
results[subject] = top
return results
def pivot_by_grade(df: pd.DataFrame) -> pd.DataFrame:
"""Create pivot table of average scores by grade and subject.
Args:
df: DataFrame with Grade and subject columns
Returns:
Pivot table with grades as rows, subjects as columns
"""
pivot = df.pivot_table(
values=['Math', 'Science', 'English'],
index='Grade',
aggfunc='mean'
).round(2)
return pivot
def main() -> None:
"""Run the complete analysis pipeline."""
print("=" * 70)
print("STUDENT SCORE ANALYSIS PIPELINE")
print("=" * 70)
# Step 1: Load and process data
df = load_data('data.csv')
df = calculate_overall_score(df)
df = assign_grades(df)
# Step 2: Display all students with grades
print("\n1. ALL STUDENTS WITH GRADES:")
print(df[['Name', 'Grade', 'Math', 'Science', 'English',
'Overall', 'GradeLetter']].to_string(index=False))
# Step 3: Grade level analysis
print("\n2. STATISTICS BY GRADE LEVEL:")
print(grade_level_analysis(df))
# Step 4: Top 3 students
print("\n3. TOP 3 STUDENTS:")
print(top_students(df, n=3).to_string(index=False))
# Step 5: Subject statistics
print("\n4. SUBJECT STATISTICS:")
stats = subject_stats(df)
for subject, vals in stats.items():
print(f"\n{subject}:")
print(f" Mean: {vals['mean']:.2f}")
print(f" Std: {vals['std']:.2f}")
print(f" Min: {vals['min']:.1f}")
print(f" Max: {vals['max']:.1f}")
# Step 6: Correlations
print("\n5. SUBJECT CORRELATIONS:")
corr_matrix = correlations(df)
print_correlation_heatmap(corr_matrix)
# Step 7: Top students per subject
print("\n6. TOP 2 STUDENTS PER SUBJECT:")
top_per_subj = top_n_per_subject(df, n=2)
for subject, top_df in top_per_subj.items():
print(f"\n{subject}:")
print(top_df.to_string(index=False))
# Step 8: Pivot table
print("\n7. AVERAGE SCORES BY GRADE LEVEL:")
print(pivot_by_grade(df))
# Step 9: Save results
df.to_csv('results.csv', index=False)
print("\n✓ Results saved to results.csv")
if __name__ == '__main__':
main()
python analysis.py
======================================================================
STUDENT SCORE ANALYSIS PIPELINE
======================================================================
1. ALL STUDENTS WITH GRADES:
Name Grade Math Science English Overall GradeLetter
Alice 10 85 88 92 88.33 B
Bob 10 92 85 88 88.33 B
Charlie 11 78 92 85 85.00 B
Diana 11 88 90 89 89.00 B
Eve 12 92 95 91 92.67 A
Frank 12 88 87 86 87.00 B
Grace 10 90 89 91 90.00 A
Henry 11 85 88 90 87.67 B
2. STATISTICS BY GRADE LEVEL:
Math Science English Overall Name
mean max min mean max min mean max min mean count
Grade
10 87.25 92 85 86.75 89 88 91.25 92 91 88.41 4
11 83.00 88 78 91.00 92 90 87.00 89 85 87.00 2
12 90.00 92 88 91.00 95 87 88.50 91 86 89.83 2
3. TOP 3 STUDENTS:
Name Grade Overall GradeLetter
Eve 12 92.67 A
Grace 10 90.00 A
Diana 11 89.00 B
4. SUBJECT STATISTICS:
Math:
Mean: 87.13
Std: 5.32
Min: 78.0
Max: 92.0
Science:
Mean: 89.25
Std: 4.27
Min: 85.0
Max: 95.0
English:
Mean: 89.00
Std: 2.93
Min: 85.0
Max: 92.0
5. SUBJECT CORRELATIONS:
Math Science English
Math 1.00 0.35 0.52
Science 0.35 1.00 -0.13
English 0.52 -0.13 1.00
6. TOP 2 STUDENTS PER SUBJECT:
Math:
Name Score
Eve 92
Bob 92
Science:
Name Score
Eve 95
Charlie 92
English:
Name Score
Alice 92
Eve 91
7. AVERAGE SCORES BY GRADE LEVEL:
Math Science English
Grade
10 87.25 86.75 91.25
11 83.00 91.00 87.00
12 90.00 91.00 88.50
✓ Results saved to results.csv
Data Visualization: Matplotlib + Seaborn + Logging Module
On Day 7, you’ll learn:
print() with professional loggingWhen working with large datasets, every byte of memory and every millisecond of computation matters. Today’s topics are the tools that separate production-grade Python from tutorial code.
Generators let you process gigabytes of data without loading it all into RAM. List comprehensions make data transformation readable and fast. Decorators let you add behavior (like caching or timing) without rewriting code. Type hints catch bugs before your code runs-critical when working in teams. Dataclasses eliminate boilerplate for data structures. pathlib handles file paths correctly on Windows, Mac, and Linux without painful string concatenation.
Together, these are the everyday tools of AI engineers processing datasets, building ML pipelines, and shipping production code.
You need only Python’s standard library for this lesson. Open a terminal and create a virtual environment:
python3 -m venv day5_env
source day5_env/bin/activate # On Windows: day5_env\Scripts\activate
python --version # Should be 3.9 or higher (3.10+ recommended for newer type hint syntax)
Create a working directory:
mkdir -p day5_project/{data_samples,analysis_output}
cd day5_project
Imagine a factory assembly line where items pass through a filter gate. Some are rejected, the rest are transformed. List comprehensions do exactly that in one readable line.
[expression for item in iterable if condition]
The if is optional. Read it left to right: “Make a list of (expression) for each (item) in (iterable) if (condition).”
# Without comprehension (verbose)
numbers = [1, 2, 3, 4, 5]
squared = []
for n in numbers:
squared.append(n ** 2)
print(squared) # [1, 4, 9, 16, 25]
# With comprehension (readable)
squared = [n ** 2 for n in numbers]
print(squared) # [1, 4, 9, 16, 25]
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Filter: keep only even numbers
evens = [n for n in numbers if n % 2 == 0]
print(evens) # [2, 4, 6, 8, 10]
# Filter and transform: keep even numbers, then square them
evens_squared = [n ** 2 for n in numbers if n % 2 == 0]
print(evens_squared) # [4, 16, 36, 64, 100]
# Create a dict mapping word to word length
words = ["python", "engineering", "data"]
word_lengths = {word: len(word) for word in words}
print(word_lengths) # {'python': 6, 'engineering': 11, 'data': 4}
# Swap keys and values
original = {"a": 1, "b": 2, "c": 3}
swapped = {v: k for k, v in original.items()}
print(swapped) # {1: 'a', 2: 'b', 3: 'c'}
numbers = [1, 2, 2, 3, 3, 3, 4, 4, 4, 4]
# Remove duplicates and filter
unique_evens = {n for n in numbers if n % 2 == 0}
print(unique_evens) # {2, 4}
# Create a 3x3 matrix (list of lists)
matrix = [[i * 3 + j for j in range(3)] for i in range(3)]
print(matrix)
# [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
# Flatten a nested list
nested = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flattened = [item for sublist in nested for item in sublist]
print(flattened) # [1, 2, 3, 4, 5, 6, 7, 8, 9]
# Too complex - hard to read. Use a regular loop or generator instead.
result = [
x * y
for x in range(10)
for y in range(10)
if (x + y) % 2 == 0 and x * y > 20
]
# Better: be explicit
result = []
for x in range(10):
for y in range(10):
if (x + y) % 2 == 0 and x * y > 20:
result.append(x * y)
Rule of thumb: If it takes more than one second to read, use a regular loop. Comprehensions should be readable.
# List comprehension - evaluates immediately, stores all in memory
lazy_list = [n ** 2 for n in range(1_000_000)] # Uses lots of memory
# Generator expression - evaluates on demand, one at a time
lazy_gen = (n ** 2 for n in range(1_000_000)) # Uses almost no memory
# They look identical except () vs []
# But generators are lazy - they don't compute until you ask
print(next(lazy_gen)) # 0 (first square)
print(next(lazy_gen)) # 1 (second square)
A vending machine gives you one item at a time when you press the button, instead of handing you the entire inventory.
Processing a 100 GB dataset? Load it line by line with a generator instead of loading all 100 GB into RAM.
import sys
# List: stores all in memory at once
list_squares = [n ** 2 for n in range(1000)]
print(f"List memory: {sys.getsizeof(list_squares)} bytes") # ~9000+ bytes
# Generator: computes one at a time
def gen_squares(n):
for i in range(n):
yield i ** 2
gen_squares_obj = gen_squares(1000)
print(f"Generator memory: {sys.getsizeof(gen_squares_obj)} bytes") # ~128 bytes
yielddef count_up(max_num: int):
"""Generator that yields numbers from 0 to max_num."""
current = 0
while current < max_num:
yield current
current += 1
# Use it
for num in count_up(5):
print(num)
# Output: 0 1 2 3 4
def read_large_file(file_path: str):
"""Generator that reads file line by line without loading all into memory."""
with open(file_path, "r") as f:
for line in f:
yield line.strip()
# Use it (simulated with a string)
# In real use: for line in read_large_file("huge_file.txt"):
# process(line)
yield from - Delegating to Another Generatordef gen_a():
"""First generator."""
yield 1
yield 2
def gen_b():
"""Second generator."""
yield 3
yield 4
def gen_combined():
"""Combine both generators."""
yield from gen_a()
yield from gen_b()
for value in gen_combined():
print(value)
# Output: 1 2 3 4
next()def countdown(n: int):
"""Countdown generator."""
while n > 0:
yield n
n -= 1
counter = countdown(3)
print(next(counter)) # 3
print(next(counter)) # 2
print(next(counter)) # 1
# print(next(counter)) # Would raise StopIteration
def infinite_counter(start: int = 0):
"""Infinite generator - keeps going forever."""
current = start
while True:
yield current
current += 1
counter = infinite_counter(10)
print(next(counter)) # 10
print(next(counter)) # 11
print(next(counter)) # 12
# Keep calling next() - it never stops
def fibonacci(limit: int):
"""Generate Fibonacci numbers up to limit."""
a, b = 0, 1
while a < limit:
yield a
a, b = b, a + b
for fib in fibonacci(100):
print(fib, end=" ")
# Output: 0 1 1 2 3 5 8 13 21 34 55 89
A gift wrapper takes your present and wraps it in paper and ribbon, then hands back the wrapped version. The present is unchanged, but it’s now decorated. Decorators do the same for functions.
import time
from typing import Callable, Any
def timer(func: Callable) -> Callable:
"""Decorator that measures how long a function takes."""
def wrapper(*args: Any, **kwargs: Any) -> Any:
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(f"{func.__name__} took {end - start:.4f} seconds")
return result
return wrapper
# Use it
@timer
def slow_function():
time.sleep(1)
return "Done!"
result = slow_function()
# Output: slow_function took 1.0001 seconds
# Done!
def simple_decorator(func):
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
@simple_decorator
def greet(name: str) -> str:
"""Say hello to someone."""
return f"Hello, {name}!"
print(greet.__name__) # 'wrapper' - WRONG! Should be 'greet'
print(greet.__doc__) # None - WRONG! Should be the docstring
functools.wrapsimport functools
from typing import Callable, Any
def timer(func: Callable) -> Callable:
"""Decorator that measures function execution time."""
@functools.wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> Any:
import time
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(f"{func.__name__} took {end - start:.4f} seconds")
return result
return wrapper
@timer
def greet(name: str) -> str:
"""Say hello to someone."""
return f"Hello, {name}!"
print(greet.__name__) # 'greet' - CORRECT
print(greet.__doc__) # 'Say hello to someone.' - CORRECT
import functools
from typing import Callable, Any
def repeat(times: int) -> Callable:
"""Decorator factory that repeats a function N times."""
def decorator(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> None:
for _ in range(times):
func(*args, **kwargs)
return wrapper
return decorator
@repeat(times=3)
def say_hello():
print("Hello!")
say_hello()
# Output:
# Hello!
# Hello!
# Hello!
import functools
import time
from typing import Callable, Any
def timer(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> Any:
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(f"[TIMER] {func.__name__}: {end - start:.4f}s")
return result
return wrapper
def log_call(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> Any:
print(f"[LOG] Calling {func.__name__} with args={args}, kwargs={kwargs}")
return func(*args, **kwargs)
return wrapper
# Stack decorators - applied bottom to top
@timer
@log_call
def add(a: int, b: int) -> int:
time.sleep(0.1)
return a + b
result = add(2, 3)
# Output:
# [LOG] Calling add with args=(2, 3), kwargs={}
# [TIMER] add: 0.1005s
# 5
import functools
from typing import Callable, Any
def cache(func: Callable) -> Callable:
"""Decorator that caches function results."""
results = {}
@functools.wraps(func)
def wrapper(*args: Any) -> Any:
if args not in results:
results[args] = func(*args)
return results[args]
return wrapper
@cache
def expensive_computation(n: int) -> int:
"""Simulates a slow calculation."""
import time
time.sleep(1)
return n ** 2
print(expensive_computation(5)) # Takes 1 second
print(expensive_computation(5)) # Instant (from cache)
Type hints are now a Day 5 production standard. From this point forward, every function must have type hints on parameters and return type.
def greet(name: str) -> str:
"""Say hello to someone."""
return f"Hello, {name}!"
def add(a: int, b: int) -> int:
"""Add two integers."""
return a + b
def is_positive(num: float) -> bool:
"""Check if a number is positive."""
return num > 0
def first_three(numbers: list[int]) -> list[int]:
"""Return the first three numbers."""
return numbers[:3]
def count_words(text: str) -> dict[str, int]:
"""Count occurrences of each word."""
words = text.split()
return {word: words.count(word) for word in set(words)}
def get_coords() -> tuple[float, float]:
"""Return latitude and longitude."""
return (40.7128, -74.0060)
def unique_tags(tags: list[str]) -> set[str]:
"""Return unique tags."""
return set(tags)
from typing import Optional
def find_user(user_id: int) -> Optional[str]:
"""Find a user by ID, or None if not found."""
users = {1: "Alice", 2: "Bob"}
return users.get(user_id)
# Python 3.10+ allows this syntax
def find_user_modern(user_id: int) -> str | None:
users = {1: "Alice", 2: "Bob"}
return users.get(user_id)
from typing import Union
def process_data(data: Union[int, str]) -> str:
"""Accept either int or str."""
return str(data).upper()
# Python 3.10+ allows this syntax
def process_data_modern(data: int | str) -> str:
return str(data).upper()
from typing import TypeVar
T = TypeVar('T') # 'T' can be any type
def get_first(items: list[T]) -> T:
"""Get the first item from a list."""
return items[0]
# Works with any type
first_num = get_first([1, 2, 3]) # int
first_str = get_first(["a", "b"]) # str
from typing import Callable
def apply_twice(func: Callable[[int], int], value: int) -> int:
"""Apply a function twice to a value."""
return func(func(value))
def square(n: int) -> int:
return n ** 2
result = apply_twice(square, 3) # 3 -> 9 -> 81
print(result) # 81
from typing import Protocol
class Drawable(Protocol):
"""Anything with a draw() method."""
def draw(self) -> None:
...
class Circle:
def draw(self) -> None:
print("Drawing circle")
class Square:
def draw(self) -> None:
print("Drawing square")
def render(shape: Drawable) -> None:
"""Draw any shape."""
shape.draw()
render(Circle()) # Works
render(Square()) # Works
from typing import TypedDict
class Person(TypedDict):
name: str
age: int
email: str
def greet_person(person: Person) -> str:
return f"Hello {person['name']}, age {person['age']}"
# Dict with correct types
person = {"name": "Alice", "age": 30, "email": "alice@example.com"}
print(greet_person(person))
A dataclass is like a form template. Instead of writing __init__, __repr__, and __eq__ by hand, the decorator fills them in automatically.
from dataclasses import dataclass
@dataclass
class Person:
name: str
age: int
email: str
# Automatically gets __init__, __repr__, __eq__
person = Person(name="Alice", age=30, email="alice@example.com")
print(person) # Person(name='Alice', age=30, email='alice@example.com')
# Can compare directly
person2 = Person(name="Alice", age=30, email="alice@example.com")
print(person == person2) # True
from dataclasses import dataclass
@dataclass
class Config:
host: str
port: int = 8000
debug: bool = False
# Can omit fields with defaults
config = Config(host="localhost")
print(config) # Config(host='localhost', port=8000, debug=False)
field() for Custom Defaultsfrom dataclasses import dataclass, field
@dataclass
class Team:
name: str
members: list[str] = field(default_factory=list)
scores: dict[str, int] = field(default_factory=dict)
team1 = Team(name="Alpha")
team2 = Team(name="Beta")
# Each team has its own list/dict (not shared!)
team1.members.append("Alice")
print(team1.members) # ['Alice']
print(team2.members) # []
__post_init__from dataclasses import dataclass
@dataclass
class Person:
name: str
age: int
def __post_init__(self) -> None:
"""Validate after initialization."""
if self.age < 0:
raise ValueError(f"Age cannot be negative: {self.age}")
if not self.name:
raise ValueError("Name cannot be empty")
# This works
person = Person(name="Alice", age=30)
# This fails
try:
bad_person = Person(name="Bob", age=-5)
except ValueError as e:
print(f"Error: {e}") # Error: Age cannot be negative: -5
frozen=Truefrom dataclasses import dataclass
@dataclass(frozen=True)
class Point:
x: float
y: float
point = Point(x=10.0, y=20.0)
print(point) # Point(x=10.0, y=20.0)
# Try to modify
try:
point.x = 15.0
except Exception as e:
print(f"Error: {type(e).__name__}") # Error: FrozenInstanceError
os.path.join() requires string concatenation: os.path.join("folder", "file.txt")pathlib uses the / operator: Path("folder") / "file.txt"from pathlib import Path
# Create a Path
file_path = Path("data") / "input.txt" # Works on any OS
print(file_path) # data/input.txt (or data\input.txt on Windows)
# Check if it exists
if file_path.exists():
print("File exists!")
else:
print("File does not exist")
# Check type
if file_path.is_file():
print("It's a file")
elif file_path.is_dir():
print("It's a directory")
from pathlib import Path
# Write text
output_file = Path("output") / "result.txt"
output_file.parent.mkdir(parents=True, exist_ok=True) # Create parent dir if needed
output_file.write_text("Hello, World!")
# Read text
content = output_file.read_text()
print(content) # Hello, World!
# Append text (read, modify, write back)
content = output_file.read_text()
output_file.write_text(content + "\nNew line added")
.glob()from pathlib import Path
# Find all .txt files recursively
for txt_file in Path("data").glob("**/*.txt"):
print(txt_file)
# Find all .py files in current directory (non-recursive)
for py_file in Path(".").glob("*.py"):
print(py_file)
.iterdir()from pathlib import Path
# List everything in a directory
for item in Path("data").iterdir():
if item.is_file():
print(f"File: {item.name}")
elif item.is_dir():
print(f"Directory: {item.name}")
from pathlib import Path
file_path = Path("data/analysis/results.txt")
print(file_path.name) # results.txt
print(file_path.stem) # results (filename without extension)
print(file_path.suffix) # .txt (extension)
print(file_path.parent) # data/analysis
print(file_path.parts) # ('data', 'analysis', 'results.txt')
from pathlib import Path
relative = Path("data/input.txt")
absolute = relative.resolve()
print(absolute) # /full/path/to/data/input.txt
Let’s build a complete file analysis tool that uses all six concepts together:
Create analyzer.py:
import time
import functools
from dataclasses import dataclass
from pathlib import Path
from typing import Generator, Callable, Any
# =============================================================================
# Data Structure: FileStats
# =============================================================================
@dataclass
class FileStats:
"""Statistics for a single file."""
name: str
path: Path
size_bytes: int
extension: str
def __post_init__(self) -> None:
"""Validate after initialization."""
if self.size_bytes < 0:
raise ValueError(f"Size cannot be negative: {self.size_bytes}")
# =============================================================================
# Decorator: Timer
# =============================================================================
def timer(func: Callable) -> Callable:
"""Decorator that measures function execution time."""
@functools.wraps(func)
def wrapper(*args: Any, **kwargs: Any) -> Any:
start = time.time()
result = func(*args, **kwargs)
end = time.time()
elapsed = end - start
print(f"\n[TIMER] {func.__name__} took {elapsed:.4f} seconds\n")
return result
return wrapper
# =============================================================================
# Generator: Analyze Files
# =============================================================================
def analyze_directory(directory: Path) -> Generator[FileStats, None, None]:
"""
Generator that yields FileStats for each file in a directory.
Lazy evaluation - doesn't load all files at once.
"""
if not directory.is_dir():
raise ValueError(f"Not a directory: {directory}")
for file_path in directory.rglob("*"):
if file_path.is_file():
try:
size = file_path.stat().st_size
yield FileStats(
name=file_path.name,
path=file_path,
size_bytes=size,
extension=file_path.suffix,
)
except OSError:
# Skip files we can't access
pass
# =============================================================================
# Main Analysis Functions with Type Hints
# =============================================================================
def filter_by_extension(
stats: Generator[FileStats, None, None],
extension: str
) -> list[FileStats]:
"""
Filter files by extension using list comprehension.
Example: filter_by_extension(gen, ".py") returns only Python files.
"""
return [
stat for stat in stats
if stat.extension.lower() == extension.lower()
]
def get_largest_files(
stats: Generator[FileStats, None, None],
top_n: int = 5
) -> list[FileStats]:
"""
Return the N largest files, sorted by size descending.
"""
all_stats = list(stats)
return sorted(all_stats, key=lambda s: s.size_bytes, reverse=True)[:top_n]
def total_size_by_extension(
stats: Generator[FileStats, None, None]
) -> dict[str, int]:
"""
Use dict comprehension to sum sizes by extension.
"""
all_stats = list(stats)
return {
ext: sum(s.size_bytes for s in all_stats if s.extension == ext)
for ext in {s.extension for s in all_stats}
}
def format_size(size_bytes: int) -> str:
"""Convert bytes to human-readable format."""
for unit in ["B", "KB", "MB", "GB"]:
if size_bytes < 1024:
return f"{size_bytes:.1f} {unit}"
size_bytes /= 1024
return f"{size_bytes:.1f} TB"
# =============================================================================
# Main Entry Point
# =============================================================================
@timer
def main(directory: str) -> None:
"""
Run full analysis with timing.
"""
target_dir = Path(directory)
if not target_dir.exists():
print(f"Error: Directory does not exist: {target_dir}")
return
print(f"Analyzing directory: {target_dir}\n")
# Example 1: All Python files
print("=" * 70)
print("PYTHON FILES")
print("=" * 70)
py_files = filter_by_extension(analyze_directory(target_dir), ".py")
for stat in py_files:
print(f" {stat.name:<40} {format_size(stat.size_bytes):>10}")
# Example 2: Largest files
print("\n" + "=" * 70)
print("TOP 5 LARGEST FILES")
print("=" * 70)
largest = get_largest_files(analyze_directory(target_dir), top_n=5)
for stat in largest:
print(f" {stat.name:<40} {format_size(stat.size_bytes):>10}")
# Example 3: Size by extension
print("\n" + "=" * 70)
print("SIZE BY FILE EXTENSION")
print("=" * 70)
sizes = total_size_by_extension(analyze_directory(target_dir))
for ext, size in sorted(sizes.items(), key=lambda x: x[1], reverse=True):
ext_label = ext if ext else "[no extension]"
print(f" {ext_label:<40} {format_size(size):>10}")
if __name__ == "__main__":
# Analyze the current directory
main(".")
Create create_sample_data.py to generate test files:
from pathlib import Path
def create_sample_structure() -> None:
"""Create sample directory structure for testing."""
base = Path("sample_data")
base.mkdir(exist_ok=True)
# Create subdirectories
(base / "python_scripts").mkdir(exist_ok=True)
(base / "config").mkdir(exist_ok=True)
(base / "docs").mkdir(exist_ok=True)
# Create sample files
(base / "python_scripts" / "main.py").write_text(
"print('Hello World')\n" * 100
)
(base / "python_scripts" / "utils.py").write_text(
"def helper():\n pass\n" * 50
)
(base / "config" / "settings.json").write_text(
'{"debug": true, "port": 8000}\n' * 30
)
(base / "config" / "database.ini").write_text(
"[database]\nhost=localhost\nport=5432\n" * 40
)
(base / "docs" / "README.md").write_text(
"# Project Documentation\n\nThis is a sample file.\n" * 60
)
(base / "notes.txt").write_text(
"Random notes and ideas\n" * 25
)
print(f"Sample data created in: {base}")
if __name__ == "__main__":
create_sample_structure()
# Create sample data
python create_sample_data.py
# Output: Sample data created in: sample_data
# Analyze the sample data
python analyzer.py sample_data
Analyzing directory: sample_data
======================================================================
PYTHON FILES
======================================================================
main.py 1.2 KB
utils.py 0.6 KB
======================================================================
TOP 5 LARGEST FILES
======================================================================
README.md 3.5 KB
settings.json 1.6 KB
database.ini 2.1 KB
main.py 1.2 KB
utils.txt 0.6 KB
======================================================================
SIZE BY FILE EXTENSION
======================================================================
.md 3.5 KB
.ini 2.1 KB
.json 1.6 KB
.py 1.8 KB
[no extension] 0.6 KB
[TIMER] main took 0.0024 seconds
Day 6 introduces NumPy and Pandas, Python’s powerhouses for numerical computing and data analysis.
By the end of today’s 1.5-hour session, you’ll understand the four pillars of Python programming:
Why these matter for AI Engineering: Every single AI program you write will use all four of these. Lists store your data. Loops process that data. Functions make your code DRY (Don’t Repeat Yourself). Classes organize code so a 100,000-line project doesn’t become spaghetti.
Best of all, you’ll build a working linear regression model from scratch using only pure Python-no NumPy, no PyTorch, no magic. You’ll see how AI training actually works: it’s not magic, just math.
Day 4 needs nothing but Python. No external libraries yet. Just create a working directory:
mkdir -p ~/ai-engineering/day-4
cd ~/ai-engineering/day-4
python3 --version # Should be 3.8+
All code today runs directly: python3 script.py
Every program needs to store data. Python gives you two main containers: lists (ordered) and dicts (labeled).
Think of a list like a shelf in a grocery store. Items sit in a specific order: position 0, position 1, position 2, etc. You find things by their position.
# Create a list
fruits = ["apple", "banana", "cherry"]
# Access by position (0-indexed!)
print(fruits[0]) # "apple" (first item, index 0)
print(fruits[1]) # "banana" (second item, index 1)
print(fruits[-1]) # "cherry" (last item, use -1)
# Modify
fruits[1] = "blueberry"
print(fruits) # ["apple", "blueberry", "cherry"]
# Add items
fruits.append("date") # Add one item to end
print(fruits) # ["apple", "blueberry", "cherry", "date"]
fruits.extend(["fig", "grape"]) # Add multiple items
print(fruits) # ["apple", "blueberry", "cherry", "date", "fig", "grape"]
# Remove items
removed = fruits.pop(0) # Remove and return first item
print(removed) # "apple"
print(fruits) # ["blueberry", "cherry", "date", "fig", "grape"]
fruits.remove("cherry") # Remove by value (not index)
print(fruits) # ["blueberry", "date", "fig", "grape"]
# How many?
print(len(fruits)) # 4
# Slice: get a range
numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
print(numbers[2:5]) # [2, 3, 4] (index 2 to 5, excludes 5)
print(numbers[:3]) # [0, 1, 2] (from start to index 3)
print(numbers[5:]) # [5, 6, 7, 8, 9] (from index 5 to end)
print(numbers[::2]) # [0, 2, 4, 6, 8] (every other item)
A dict is like a filing cabinet with labeled drawers. Each item has a name (the “key”) and a value (what’s inside). You find things by their label, not their position.
# Create a dict
student = {
"name": "Alice",
"age": 25,
"major": "Computer Science",
"gpa": 3.8
}
# Access by key (label)
print(student["name"]) # "Alice"
print(student["gpa"]) # 3.8
# Safe access (won't crash if key doesn't exist)
print(student.get("age")) # 25
print(student.get("email")) # None (key doesn't exist)
print(student.get("email", "N/A")) # "N/A" (provide default)
# Update
student["age"] = 26
student["gpa"] = 3.9
# Add new key
student["phone"] = "555-1234"
# Remove key
del student["major"]
# Check if key exists
if "name" in student:
print(f"Name: {student['name']}")
if "email" not in student:
print("No email on file")
# Get all keys
print(student.keys()) # dict_keys(['name', 'age', 'gpa', 'phone'])
# Get all values
print(student.values()) # dict_values(['Alice', 26, 3.9, '555-1234'])
# Get all key-value pairs
for key, value in student.items():
print(f"{key}: {value}")
# name: Alice
# age: 26
# gpa: 3.9
# phone: 555-1234
Sometimes you want a dict to have a default value if a key doesn’t exist yet:
from collections import defaultdict
# Regular dict would crash here:
# scores = {}
# scores["Alice"] += 10 # KeyError!
# defaultdict doesn't crash-it uses a default value
scores = defaultdict(int) # Default value is 0
scores["Alice"] += 10 # Works! Creates key with 0, then adds 10
scores["Bob"] += 5
print(scores["Alice"]) # 10
print(scores["Bob"]) # 5
print(scores["Charlie"]) # 0 (never been set, but default is 0)
| Situation | Use List | Use Dict |
|---|---|---|
| “Give me the 3rd item” | YES | NO |
| “Find the item called ‘age’” | NO | YES |
| Order matters | YES | NO |
| Fast lookup by name | NO | YES |
| You know the keys ahead of time | NO | YES |
| Keys are things like “name”, “age”, “email” | NO | YES |
Loops let you repeat code without writing it 100 times. Two types: for (when you know how many times) and while (when you loop until a condition is true).
for Loop: Repeat Over a Collection# Loop over a list
fruits = ["apple", "banana", "cherry"]
for fruit in fruits:
print(fruit)
# Output:
# apple
# banana
# cherry
# Loop over a dict
student = {"name": "Alice", "age": 25, "major": "CS"}
for key in student:
value = student[key]
print(f"{key}: {value}")
# Output:
# name: Alice
# age: 25
# major: CS
# Better way with .items()
for key, value in student.items():
print(f"{key}: {value}")
range(): Generate Numbers# range(n) = 0, 1, 2, ..., n-1
for i in range(5):
print(i)
# Output: 0 1 2 3 4
# range(start, stop, step)
for i in range(2, 8, 2): # Start at 2, stop before 8, step by 2
print(i)
# Output: 2 4 6
# Count backwards
for i in range(5, 0, -1): # Start at 5, count down
print(i)
# Output: 5 4 3 2 1
enumerate(): Loop with IndexOften you want both the index (position) and the value:
fruits = ["apple", "banana", "cherry"]
# Without enumerate (awkward)
for i in range(len(fruits)):
print(f"{i}: {fruits[i]}")
# With enumerate (clean)
for i, fruit in enumerate(fruits):
print(f"{i}: {fruit}")
# Output:
# 0: apple
# 1: banana
# 2: cherry
zip(): Loop Over Multiple Lists Togethernames = ["Alice", "Bob", "Charlie"]
ages = [25, 30, 28]
# Loop through both at once
for name, age in zip(names, ages):
print(f"{name} is {age} years old")
# Output:
# Alice is 25 years old
# Bob is 30 years old
# Charlie is 28 years old
# Print a 3x3 grid
for i in range(3):
for j in range(3):
print(f"({i}, {j})", end=" ")
print() # Newline after each row
# Output:
# (0, 0) (0, 1) (0, 2)
# (1, 0) (1, 1) (1, 2)
# (2, 0) (2, 1) (2, 2)
while Loop: Repeat Until a ConditionUse while when you don’t know ahead of time how many loops you need:
# Keep looping while condition is true
count = 0
while count < 5:
print(count)
count += 1
# Output: 0 1 2 3 4
# Loop until something happens
password = ""
while password != "secret":
password = input("Enter password: ")
if password == "secret":
print("Access granted!")
else:
print("Try again")
break and continue# break: exit the loop immediately
for i in range(10):
if i == 5:
break # Stop the loop
print(i)
# Output: 0 1 2 3 4
# continue: skip to next iteration
for i in range(5):
if i == 2:
continue # Skip this iteration
print(i)
# Output: 0 1 3 4 (skips 2)
for vs whilefor when you know (or can count) how many times to loopwhile when you loop until a condition becomes falseA function is like a recipe: you write the steps once, then use it many times. Functions make code DRY (Don’t Repeat Yourself).
# Define a function with def
def greet(name):
# name is a parameter
return f"Hello, {name}!"
# Call it (use it)
result = greet("Alice")
print(result) # "Hello, Alice!"
# Another example
def add(a, b):
return a + b
print(add(3, 5)) # 8
print(add(10, 20)) # 30
A function can return a value or nothing:
# Return a value
def double(x):
return x * 2
result = double(5)
print(result) # 10
# No return = returns None
def print_twice(text):
print(text)
print(text)
# No return statement!
result = print_twice("Hi")
print(result) # None
Provide default values so the caller doesn’t always have to pass them:
def greet(name, greeting="Hello"):
return f"{greeting}, {name}!"
print(greet("Alice")) # "Hello, Alice!"
print(greet("Bob", "Hi")) # "Hi, Bob!"
print(greet("Charlie", "Hey")) # "Hey, Charlie!"
Pass arguments by name (order doesn’t matter):
def describe_pet(name, species, age):
return f"{name} is a {age}-year-old {species}"
# Positional (order matters)
print(describe_pet("Fluffy", "cat", 3))
# Keyword (order doesn't matter)
print(describe_pet(age=5, species="dog", name="Rex"))
# Mixed
print(describe_pet("Bella", age=2, species="parrot"))
*args: Accept Any Number of Argumentsdef sum_all(*args):
# args is a tuple of all arguments passed
total = 0
for num in args:
total += num
return total
print(sum_all(1, 2, 3, 4, 5)) # 15
print(sum_all(10, 20)) # 30
print(sum_all(7)) # 7
**kwargs: Accept Keyword Argumentsdef print_info(**kwargs):
# kwargs is a dict of all keyword arguments
for key, value in kwargs.items():
print(f"{key}: {value}")
print_info(name="Alice", age=25, city="NYC")
# Output:
# name: Alice
# age: 25
# city: NYC
In Python, functions are first-class citizens: you can pass them to other functions!
def apply_twice(func, value):
# func is a function, not a value
return func(func(value))
def double(x):
return x * 2
result = apply_twice(double, 3)
print(result) # double(double(3)) = 12
# Lambda: anonymous function (one-liner)
result = apply_twice(lambda x: x + 1, 5)
print(result) # 7
A pure function always returns the same output for the same input and doesn’t change anything in the world. A function with side effects does other things (print, modify global variables, etc.):
# Pure function: same input = same output, no side effects
def add(a, b):
return a + b # Only returns
# Side effect: changes the world (prints)
def add_and_print(a, b):
result = a + b
print(result) # Side effect!
return result
# Side effect: modifies global state
counter = 0
def increment_counter():
global counter
counter += 1 # Side effect: changes global variable
return counter
For AI engineering, pure functions are your friend. They’re easier to test, reason about, and debug.
Variables have scope: where they exist and can be used.
x = 10 # Global scope
def change_x():
x = 5 # Local scope-doesn't touch the global x
print(x)
change_x() # Prints 5
print(x) # Still 10! The global x is unchanged
# To modify global, use 'global' keyword
def really_change_x():
global x
x = 20
really_change_x()
print(x) # Now 20
A class is a blueprint for objects. OOP (Object-Oriented Programming) lets you organize code into reusable, logical chunks.
class Dog:
def __init__(self, name, age):
# __init__ = constructor
# Runs when you create a new Dog
# self = the object being created
self.name = name # Instance attribute
self.age = age
def bark(self):
# Instance method (function inside a class)
# Takes self as first parameter
return f"{self.name} says woof!"
# Create instances (actual objects)
dog1 = Dog("Buddy", 5)
dog2 = Dog("Max", 3)
print(dog1.name) # "Buddy"
print(dog2.name) # "Max"
print(dog1.bark()) # "Buddy says woof!"
print(dog2.bark()) # "Max says woof!"
class Dog:
species = "Canis familiaris" # Class attribute
# ^ Shared by ALL Dogs
def __init__(self, name):
self.name = name # Instance attribute
# ^ Unique to each Dog
dog1 = Dog("Buddy")
dog2 = Dog("Max")
print(dog1.name) # "Buddy"
print(dog2.name) # "Max"
print(dog1.species) # "Canis familiaris" (same for all)
print(dog2.species) # "Canis familiaris" (same for all)
class Dog:
species = "Canis familiaris"
def __init__(self, name):
self.name = name
def bark(self):
# Instance method: uses self (the specific dog)
return f"{self.name} barks!"
@classmethod
def create_from_dict(cls, data):
# Class method: uses cls (the class)
# Useful for creating objects in different ways
return cls(name=data["name"])
@staticmethod
def info():
# Static method: doesn't use self or cls
# Just a function that lives in the class
return "Dogs are loyal animals"
# Usage
dog = Dog("Buddy")
print(dog.bark()) # "Buddy barks!"
dog2 = Dog.create_from_dict({"name": "Rex"})
print(dog2.name) # "Rex"
print(Dog.info()) # "Dogs are loyal animals"
class Animal:
def __init__(self, name):
self.name = name
def speak(self):
return f"{self.name} makes a sound"
class Dog(Animal):
# Dog inherits from Animal
# Dog automatically has name and speak()
pass
class Cat(Animal):
pass
dog = Dog("Buddy")
cat = Cat("Whiskers")
print(dog.speak()) # "Buddy makes a sound"
print(cat.speak()) # "Whiskers makes a sound"
Replace a parent method with a child version:
class Animal:
def speak(self):
return "Generic sound"
class Dog(Animal):
def speak(self):
# Override: replace parent's speak with dog-specific
return "Woof!"
class Cat(Animal):
def speak(self):
return "Meow!"
dog = Dog()
cat = Cat()
print(dog.speak()) # "Woof!" (Dog's version)
print(cat.speak()) # "Meow!" (Cat's version)
@property: Computed AttributesMake a method look like an attribute:
class Circle:
def __init__(self, radius):
self.radius = radius
@property
def area(self):
# Looks like self.area, but it's computed on the fly
return 3.14159 * self.radius ** 2
circle = Circle(5)
print(circle.area) # 78.54975 (computed, not stored)
circle.radius = 10
print(circle.area) # 314.159 (automatically recomputed)
@abstractmethod: Enforce Subclass ImplementationForce all subclasses to implement certain methods:
from abc import ABC, abstractmethod
class Animal(ABC):
@abstractmethod
def speak(self):
# Subclasses MUST implement this
pass
class Dog(Animal):
def speak(self):
return "Woof!"
dog = Dog()
print(dog.speak()) # "Woof!"
# This would fail:
# animal = Animal() # TypeError: can't instantiate abstract class
Special methods that make your class work with Python’s built-ins:
class Dog:
def __init__(self, name, age):
self.name = name
self.age = age
def __str__(self):
# str(dog) uses this
# Human-readable
return f"Dog named {self.name}"
def __repr__(self):
# repr(dog) uses this
# For debugging, more detailed
return f"Dog(name={self.name!r}, age={self.age})"
def __len__(self):
# len(dog) uses this
return self.age
def __eq__(self, other):
# dog1 == dog2 uses this
return self.name == other.name and self.age == other.age
def __add__(self, other):
# dog1 + dog2 uses this
# Create a new dog with combined names
return Dog(self.name + other.name, max(self.age, other.age))
dog1 = Dog("Buddy", 5)
dog2 = Dog("Max", 3)
print(str(dog1)) # "Dog named Buddy"
print(repr(dog1)) # "Dog(name='Buddy', age=5)"
print(len(dog1)) # 5
print(dog1 == dog2) # False
dog3 = dog1 + dog2
print(dog3.name) # "BuddyMax"
Inheritance: “Dog IS-A Animal”
class Animal:
def breathe(self):
return "Breathing..."
class Dog(Animal):
# Dog inherits breathe() from Animal
pass
dog = Dog()
print(dog.breathe()) # "Breathing..."
Composition: “Dog HAS-A Tail”
class Tail:
def wag(self):
return "Wagging..."
class Dog:
def __init__(self):
self.tail = Tail() # Dog HAS a Tail
dog = Dog()
print(dog.tail.wag()) # "Wagging..."
When to use:
Now you’ll build a real machine learning model using only lists, loops, functions, and classes. No NumPy, no PyTorch-just pure Python math.
A LinearRegressionModel that:
y = weight * x + biasy = weight * x + biasSave this as linear_regression.py:
class LinearRegressionModel:
def __init__(self, initial_weight=0.0, initial_bias=0.0):
"""
Initialize the model with weight and bias.
weight = slope of the line (how steep)
bias = y-intercept (where the line crosses y-axis)
"""
self.weight = initial_weight
self.bias = initial_bias
self.training_steps = 0
def predict(self, x):
"""
Make a prediction for a single input.
Formula: y_pred = weight * x + bias
This is just the equation of a line!
"""
return self.weight * x + self.bias
def loss(self, x_list, y_list):
"""
Calculate Mean Squared Error (MSE).
MSE = average of (prediction - actual)^2
Lower loss = better predictions.
"""
total_error = 0.0
# For each training example
for x, y in zip(x_list, y_list):
prediction = self.predict(x)
error = prediction - y
squared_error = error ** 2
total_error += squared_error
# Return the average
n = len(x_list)
return total_error / n
def train(self, x_list, y_list, lr=0.01, epochs=100):
"""
Train the model using gradient descent.
lr = learning rate (how big a step to take)
epochs = how many times to go through the data
Gradient descent: repeatedly move in the direction of lower loss.
"""
n = len(x_list)
for epoch in range(epochs):
# Calculate gradients (direction to move)
dw = 0.0 # Change in weight
db = 0.0 # Change in bias
# Go through all training examples
for x, y in zip(x_list, y_list):
prediction = self.predict(x)
error = prediction - y
# Gradient formulas for linear regression
# (These come from calculus, but we just use them)
dw += 2 * x * error
db += 2 * error
# Average the gradients
dw = dw / n
db = db / n
# Update: move opposite to the gradient
# (Opposite = towards lower loss)
self.weight -= lr * dw
self.bias -= lr * db
# Track that we did a training step
self.training_steps += 1
# Print progress every 20 epochs
if (epoch + 1) % 20 == 0:
current_loss = self.loss(x_list, y_list)
print(f"Epoch {epoch + 1}: Loss = {current_loss:.4f}")
def __str__(self):
"""
User-friendly representation.
Used by print(model)
"""
return f"LinearRegressionModel(weight={self.weight:.4f}, bias={self.bias:.4f})"
def __repr__(self):
"""
Developer-friendly representation.
Used by repr(model) for debugging
"""
return f"LinearRegressionModel(weight={self.weight}, bias={self.bias})"
def __len__(self):
"""
Return number of training steps.
Used by len(model)
"""
return self.training_steps
# Example: Train the model
if __name__ == "__main__":
# Create some sample data
# The true relationship is: y = 2x + 1 (with noise)
x_data = [1.0, 2.0, 3.0, 4.0, 5.0]
y_data = [3.1, 5.0, 7.2, 9.1, 10.9]
# Create the model
model = LinearRegressionModel()
print(f"Before training: {model}")
print()
# Train it
model.train(x_data, y_data, lr=0.01, epochs=100)
print()
# See the result
print(f"After training: {model}")
print(f"Total training steps: {len(model)}")
print()
# Make predictions
print("Predictions:")
for x in x_data:
prediction = model.predict(x)
print(f"x={x}: predicted y={prediction:.2f}")
python3 linear_regression.py
Before training: LinearRegressionModel(weight=0.0000, bias=0.0000)
Epoch 20: Loss = 8.4521
Epoch 40: Loss = 3.1842
Epoch 60: Loss = 1.5921
Epoch 80: Loss = 0.8521
Epoch 100: Loss = 0.5621
After training: LinearRegressionModel(weight=1.9821, bias=1.0342)
Total training steps: 100
Predictions:
x=1.0: predicted y=3.02
x=2.0: predicted y=5.00
x=3.0: predicted y=6.98
x=4.0: predicted y=8.97
x=5.0: predicted y=10.95
Notice: The model learned weight ≈ 2 and bias ≈ 1, which matches the true relationship y = 2x + 1!
Day 5 introduces type hints: annotations that tell Python (and other programmers) what types variables and functions expect.
]]>