From Finance

Zombie Metrics

This guy measures water affordability as (Avg bill ÷ MHI)<2.0%

Terrible, horrible, no good, very bad measurement, part 4

My criticism of average bill ÷ Median Household Income (MHI) as a measure of household-level water affordability isn’t especially new. Lots of other people have pointed out the problems with this conventional methodology, and I’ve been presenting and publishing these arguments for more than twelve(!) years. But golden numbers are stubborn, and bad habits are hard to break—even when people know better.

The remarkable persistence of a bad idea

Over the years I’ve presented to hundreds of utility professionals and spoken personally with scores of managers, analysts, and rate consultants about the pathologies of %MHI and the virtues of alternative approaches. The reception is universally warm and agreeable, as most water professionals genuinely care about affordability and immediately recognize the fundamental flaws of the conventional approach.

And yet.

Alas, there’s an and yet.

Even well-informed specialists continue to use and promote the tried-and-false conventional methodology. Researchers who recognize that average-bill-as-%MHI is deeply flawed employ it anyway because it’s easy and widely recognized (for example). Managers who know that %MHI is a misleading statistic continue to put it in front of their elected officials because it’s familiar and they feel that they have to use this metric because everyone else does, and because they believe it’s an EPA standard (it isn’t). Advocates, analysts, and rate consultants who I like and respect persist with the conventional approach in their studies, even when they know these metrics are fundamentally flawed (many have told me as much!).

Examples abound. The Alliance for Water Efficiency has a nice tool that’s designed to help water utilities model the financial impacts of various rate structures. Sensibly enough, their model includes an assessment of affordability. Unfortunately, it uses the familiar flawed metric:

C’mon man

UNC’s Environmental Finance Center continues to feature average-bill-as-%MHI as the sole affordability indicator on its rates dashboards. Folks at EFC know about the problems with this metric (they blogged about it here), but continue to display it prominently nonetheless.

Et tu, EFC?

 

Easy metrics die hard, it seems.

Breaking habits

Water and sewer ratemaking is a niche specialty (to put it mildly). That’s good news, because if the community of specialists who analyze and design rates for a living get affordability metrics right, there’s a good chance that the utilities they serve will get affordability right, too.

I’ve developed better ways to measure affordability; others are working on this issue, too. At this stage there’s no consensus over the best metrics (naturally, I think mine are great). But abandoning the flawed measurement convention is an important first step.

Water Affordability’s Golden Numbers

Not actually in the book of Numbers

Terrible, horrible, no good, very bad measurement, part 3

As my last couple of posts explain, the conventional method of measuring household-level water affordability is to divide a utility’s average residential bill by its community’s Median Household Income (MHI). If the resulting percentage is less than 2.0 or 2.5 (4.0 or 4.5 for water and sewer combined), then water is deemed “affordable;” if it’s greater, then water is “unaffordable.”

This post explains the fourth, and in my view most serious, problem with the conventional approach: the arbitrary threshold that it uses to define affordability.

An arbitrary standard

I’ve never—and I mean never—seen a theoretical or empirical rationale for a water/sewer affordability standard as a function of %MHI. Apparently the 2%MHI threshold emerged from the mists of federal regulatory history in a 1970s-era USDA rural grant program. At some point EPA began using the metric as part of its regulatory enforcement program. Despite those strange and ill-fitting origins, the 2.0 (or 2.5) %MHI affordability threshold is now held up as a definitive measure of household-level affordability, apparently for no other reason than convention.

The affordability of anything is rarely a strictly yes/no phenomenon—things are more or less affordable relative to the costs of other things and the resources available to pay for them. But rather than deal with those nuances, analysts simply cite precedent and invoke average-bill-as-%MHI as an EPA “standard” (though as I’ve noted before, it’s not an actual EPA standard, and the agency never intended %MHI to be a measure of household-level affordability).

Golden numbers

The 2% or 4%MHI threshold has become what organizational theorists call a golden number. Golden numbers are standards that have no basis in theory or evidence, but are so widespread that they take on independent importance simply by virtue of their familiarity. The trouble with golden numbers like 2%MHI is that they preempt or short-circuit serious deliberation over values. Managers of a utility that satisfies the %MHI threshold can use the standard as an excuse dismiss affordability concern, even if many of its customers struggle to pay their bills.

Worse yet, a golden number like 2%MHI that has a tangential relationship to an EPA guideline puts a veneer of legitimacy on what is ultimately an arbitrary norm. Performance metrics reflect an organization’s values and help guide management toward decisions consistent with those values. In that regard, the golden number that predominates water rate analysis fails fundamentally.

Communities and their elected leaders should set affordability standards consistent with their values. Average bill <2%MHI distorts the affordability picture and distracts from meaningful consideration of the issue.

Terrible, Horrible, No Good, Very Bad Measurement

This officer is not trying to measure statewide alcohol sales

A feral howl on the conventional method of assessing household water affordability, part 1

Recently a colleague asked me how I first got interested in water and sewer affordability. I was tempted to claim that it was pure concern for the welfare of my fellow human being. That’s certainly the main reason. But I think there’s another factor at work: my near-pathological intolerance of bad measurement in public policy.

This is the first in a series of posts in which I advance a critique of rant about the long-standing, widely-applied standard method of measuring water and sewer utility affordability in the United States. The lengthier, formal version of this discussion is in an article published earlier this year in Journal AWWA.

Next week I’ll be in Las Vegas at the AWWA Annual Conference & Exposition, where I’ll present better metrics, show some initial results of a new national study of affordability, and join Kathryn Sorensen in telling the story of how better measurement is helping to shape policy in Phoenix. But in this post I’m just going to rant.

The conventional wisdom foolishness

The most widely applied method of measuring water/sewer utility affordability in the United States is to calculate the average residential bill for a given utility as a percentage of the community’s Median Household Income (%MHI). Usually this percentage is calculated for an entire utility, but sometimes it is calculated for a neighborhood or a census tract. This percentage is then compared with an affordability standard (usually 2.0 or 2.5 percent). A simple binary declaration follows this standard: if a utility’s average bill as %MHI is less than this standard, then it is deemed “affordable;” if it is greater, then it is “unaffordable.” Sometimes these %MHI standards are applied separately to water and sewer rates, other times they are combined water-plus-sewer costs (thus the standard becomes 4.0 or 4.5 %MHI).

This approach has become the default way to measure affordability in both scholarly research and policy analysis, with no other rationale than that it is convenient (a 4th grader could do it on the back of an envelope) and conventional (everybody else does it that way).

The reasonable origins of a silly idea

The %MHI method originated with the EPA, where the metric was first developed as a gauge of a community’s financial capability for purposes of negotiating regulatory compliance by its utilities. The idea of %MHI as a measure of financial capability traces at least to the EPA’s 1984 Financial Capability Guidebook. The idea of identifying specific %MHI thresholds for financial capability apparently come from EPA’s 1995 guidelines on Water Quality Standards (though these documents don’t offer an empirical or theoretical rationale for 1.0, 2.0, or 2.5%MHI standards).

As a measure of utility-level financial capacity (can a utility handle the costs of regulatory compliance?), average bill as %MHI isn’t crazy—it gives a vaguely reasonable sense of the utility’s demands on a community’s financial resources.

Here’s where it fails…

Unfortunately, at some point over the past 20 years, researchers, policy analysts, managers, and rate consultants started using %MHI as a measure of household-level affordability (can a poor family pay its water/sewer bill?). Despite its widespread use, %MHI is seriously, fundamentally, irredeemably flawed as a metric of household affordability.

The main trouble with using average bill ÷ MHI as a measure of affordability is that it does not measure affordability—at least not at the household level, in the way that most people think about affordability. Although EPA has drawn a great deal of criticism for its affordability metrics, this misapplication of %MHI to household-level analysis really isn’t EPA’s fault; the federal government never intended for their utility-level metric to be used in customer-level analysis. Alas, lots of otherwise smart and well-meaning people have done just that.

Over the next few posts I’ll get into the many reasons why average bill ÷ MHI is a terrible, horrible, no good, very bad metric.