On the astonishing power of random sampling

Studying utility management with any kind of scientific rigor is hard; the sheer number, complexity, and opacity of water and sewer systems make it effectively impossible to get the organizational and financial data necessary to analyze all of them at once. So instead, we look at a small set of utilities and try to infer something about the universe of utilities based on what we see in the smaller set. That’s fine, of course. Most real-world research involves analyzing samples: seeing the entire universe of anything is really hard, so science almost always relies on samples.

But because sampling is easy, an awful lot of research on utility policy, management, and finance is built on highly biased data. Sample bias is a particularly common affliction in studies of pricing and affordability. That’s a problem, because biased data give a biased view of the world and can lead us to the wrong conclusions.

This post - the first in a series on sample bias - focuses on pricing, illuminates the dangers of bad sampling, and illustrates the power of randomization.

Thing is, there are good reasons to think that prices at these easy-to-observe utilities are systematically different from utilities in general. Price correlates negatively with utility size, for example, and utilities with egregiously high prices or poor financial conditions probably won’t respond to survey questionnaires. The result is that water/sewer rate surveys likely underestimate average prices.

Randomization to the rescue!

Random sampling solves the problem of sample bias by giving every utility—large or small, expensive or cheap—an equal chance of being included in a survey. Instead of relying on voluntary responses or targeting only big systems, randomization ensures that the sample reflects the full diversity of utilities. This approach produces more a representative dataset, helping policymakers and researchers understand the true landscape of water and sewer pricing.

Unfortunately, simple random sampling doesn’t work well for water and sewer utilities because the system sizes are extremely uneven. There are nearly 50,000 community water systems and around 15,000 sewer systems nationwide, but most serve very few people, while a small number serve millions. Ownership is also highly skewed: a large majority of Americans get water/sewer service from a local government, but small systems are disproportionately private. If we randomly select utilities, we’ll mostly get tiny, disproportionately private systems, which skews the data and makes it hard to study larger ones or understand public-private differences. Stratified sampling solves this problem by dividing utilities into groups—like public/private, or small/medium/large—and then randomly selecting from each group. Stratified sampling yields a more balanced and useful picture of the entire sector.

With those data we can calculate the true average water utility price in Wisconsin, which was $41.78 per month for a residential customer who used 6,000 gallons. Comparing this true average to averages calculated with different sampling methods is an eye-opening exercise.

Large utilities only. Let’s start with the most common approach: just the largest utilities that serve populations of 50,000 or more. Getting rates data for these big systems is easy: larger utilities typically have good websites, often with full financial statements alongside up-to-date rates information. In 2024, the twelve Wisconsin water utilities serving populations of at least 50,000 charged an average of $34.91—16.5% lower than the true average.

Largest 200 utilities. What if we expand the sample to the largest 200 utilities? In Wisconsin that would mean every utility serving a population of 2,700 or more—a lot more work than just the big boys if you have to get the data yourself. In 2024, the average monthly bill at 6,000 gallons was $37.08 for these largest 200 Wisconsin utilities, which is closer to the true average, but still underestimated by 11.3%.

Simple random sample of 100. Now let’s look at a simple random sample. If we randomly draw 100 utilities, the 2024 average monthly bill at 6,000 gallons turns out to be $42.45—a little more than the true average, but much closer than we get with a larger but much more biased sample with an error of just +1.6%.

Stratified sample of 100. Next, consider a stratified sample of 100 utilities. We take all 572 utilities and assign each one to a stratum depending on the size of population it serves: 1,000 or fewer; 1,001-5,000; 5,001-10,000; and 10,001-25,000, and greater than 25,000. We then randomly draw 20 utilities from each stratum and calculate the average from the resulting sample of 100 utilities. Based on the raw data from that stratified sample, the 2024 average monthly bill at 6,000 gallons is $44.64, which means we’ve now overshot the mark by +6.8%.

That’s because stratification intentionally introduces bias into the sample, since some utilities (larger ones) were more likely than others (smaller ones) to be selected. Fortunately, since we know these biases, we can correct for them.

Weighted stratified sample of 100. Finally, let’s look at the stratified, randomized sample after adjusting for the chance that each utility was selected into the sample—a process called inverse probability weighting.^† After applying weights, the stratified sample yields an average of $41.56—barely 0.5% away from the true average. As if by magic, stratified, random sampling with inverse provability weights gives a near-perfect approximation of all 572 utilities from a sample of just 100.

Here's a summary of how closely each sampling method comes to reality (nonrandom samples in dark blue, randomized samples in light blue):

Picking the big (i.e., easy) utilities grossly underestimates the true average, randomization mildly overestimates the true average, and weighted, stratified sampling is dang near perfect.

Sweat the sample

Rigorous research on utility policy, management, and finance is hard. When it comes to the human side of the water sector, most of the things we care about can’t be studied in controlled laboratory conditions. Messy, observational data are usually all we have. The temptation to use easily accessible data^** is understandable but dangerous, as it can paint an inaccurate picture that leads to bad decisions.

No dataset is perfect, but when it comes to accurate water and sewer utilities research, you can bank on ratified, randomized sampling FTW. Good sampling takes a bit more care, but offers more accurate, more responsible results when research budgets and timelines are tight. For utility leaders and professionals, understanding sampling is crucial to evaluating any management or financial study before you.

*Sometimes the sample is biased on purpose to generate a desired result.

†That sounds tricky, but the math is straightforward and should be pretty easy for anyone who works with survey data.

**Or worse yet, cherry-picked case studies.