A few years ago, my partner Karthik and I built a deep learning model for our CS 230 final project at Stanford. The goal was straightforward: could we use cheap, widely available weather data to forecast PM2.5 air pollution in the Bay Area?
The model worked, but what stuck with me more was the process of getting there. We went through four model versions, debugged bias problems, watched performance improve and then plateau, and made a lot of tradeoff decisions without an obvious right answer. Now that I work as a PM, I keep coming back to those lessons when building AI features. Here are five of them.
Air pollution causes 5 million deaths per year globally and costs the world economy an estimated $225 billion annually [1]. The primary metric is PM2.5, fine particulate matter small enough to enter the bloodstream. The problem: the sensors that measure it are expensive and can’t be deployed everywhere.
Our question:
Can a neural network trained on freely available weather data accurately forecast PM2.5, even in locations it has never seen?
We pulled five years of daily data (2015–2019) from CIMIS weather stations and EPA air quality monitors across the Bay Area. We built an LSTM, a type of recurrent neural network designed for sequential data, and iterated through four model versions, tuning architecture and loss functions each time. Our final model reduced RMSE from 6.99 to 3.70, and generalized to Oakland even though the model had never trained on East Bay data.
Generalizing to a location the model had never seen was the result I was most proud of.
Before we trained anything, we needed to answer a simple question: what does “good” look like? Without a baseline, you have no way to know whether your model is actually doing anything useful.
We established our baseline performance using the simplest reasonable approach, then used RMSE as our consistent measure across every model version. Every improvement was relative to something concrete. When version 3 dropped RMSE from 6.06 to 5.38, we knew exactly what that meant.
As a PM, I now treat defining a baseline as a requirement before any new development. Before we discuss solutions, vendors, or timelines, I want to know: what does the system do today, how do we measure it, and what would a simple version achieve? If the team can’t answer those questions, we’re not ready to build.
We didn’t get to RMSE 3.70 in one shot. We ran four model versions, each informed by what the previous one got wrong.
Version 2 had high bias. It tracked the trend but consistently underestimated PM2.5 spikes. Adding more layers helped more than adding neurons, which version 3 confirmed and version 4 held up at scale.
What I learned is that decisions like switching loss functions, adding layers, and adjusting dropout weren’t just engineering calls. They had product consequences too. Deeper networks are more accurate but slower to train and harder to explain. MSE loss penalizes large errors more than MAE, which matters when the dangerous edge cases are precisely the big spikes.
I see the same kinds of tradeoffs when picking between model vendors, choosing evaluation criteria, or scoping the first release of an AI feature. The questions are the same: what are we optimizing for, what are we giving up, and are we choosing that on purpose?