From Affordability

Misoverestimatickated

About that water affordability study
in The Guardian...

Water is a ZILLION PERCENT unaffordable! Also, aliens.

The Guardian recently published a big story on water utility affordability in the United States. The headline was shocking: “Millions of Americans Can't Afford Water, as Bills Rise 80% in a Decade,” and “Analysis of U.S. cities shows emergency on affordability of running water amid COVID-19 pandemic.” The story was based on a study that The Guardian commissioned from Boston-area attorney Roger Colton. Colton’s report—which The Guardian called “the first nationwide research of its kind”* —was the basis for the story’s claims.

The Colton report sets out to do three things:

1) “examine whether the affordability of water is common in the U.S. today”;

2) examine whether “water affordability has changed in recent years;” and

3) “examine the extent to which… reasonable projections of water rate increases will affect water affordability in the near future.”

Put simply, the report aims to measure the level of water affordability in the U.S., past affordability trends, and projected future affordability.

Unfortunately, the Colton report is ​deeply flawed. Not just flawed in some narrow, technical sense; it’s flawed in ways that grossly misrepresent the state of water affordability in the U.S. and point to the wrong approaches to addressing the issue.

I really didn’t want to write this post. It’s heartening when mainstream journalists pay attention to the oft-neglect water sector, especially on matters of affordability, so I hate to be negative about it. ​I'm also painfully aware of Brandolini’s Law​, and ​any time spent counteracting bad research is time not spent on my own research.

​But over the past month I’ve fielded several inquiries about The Guardian article asking whether the Colton report is valid. The short answer is no.

​The long answer requires wading into the tall methodological grass. So grab your weed eater and follow me.

The study

Although The Guardian article’s claims are national in scope, the underlying analysis is limited to twelve cities. The dozen were not selected at random, nor ​are they the twelve largest. Rather, they were selected by Guardian staff, supposedly “to provide a diversity of geographic regions… population sizes… and poverty.” All inferences from the Colton report come from this hand-picked, non-representative sample of systems that collectively serve a little more than 2% of the U.S. population.

But the Colton study’s deepest problems are about measurement, not sampling. Answering the questions that ​the study raises requires four things:

a) prices for water service in the United States

b) household resources available to pay for water

c) projected future prices and resources

d) defining affordability

You can think of those as the numerator, the denominator, the forecast, and the judgment. All of them flawed in ways that compound and confound.

The numerator

A study of water and sewer affordability should start with the price of water and sewer service, right? Strangely, the Colton study didn’t collect any rates data or calculate prices. Instead, Colton got price data from Circle of Blue’s annual reports on water prices in 30 U.S. cities. Circle of Blue calculates prices at three benchmark volumes, representing demand for a family of four at 50 gallons per capita per day (gpcd), 100 gpcd, and 150 gpcd. That works out to roughly 6,000, 12,000, and 18,000 gallons per month. Nationally, residential indoor water demand averages around 50-60 gpcd and has been falling steadily for the past twenty years. This indoor demand represents basic needs for drinking, cooking, cleaning, and sanitation. That’s the main public policy concern for affordability—from a public health standpoint we don’t really care about the affordability of lawn watering or car washing.

But the Colton report used Circle of Blue’s 12,000 gallon monthly price as the basis for its numerator. So ​The Guardian’s startling graphs ​reflect the prices for for very high volumes of water.

Next Colton used the 12,000 gallon monthly price to “derive estimated bills given average household sizes for each Census Tract,” according to a footnote on page 6. How Colton derived the prices is a complete mystery; he never explains the procedure, and the footnote is the only mention of how he calculated prices.** A simple table of monthly prices is nowhere in the 88-page report.†

The denominator

Claims about the affordability of anything implies some relationship between costs and resources. Colton’s sole measure of resource affordability is the Federal Poverty Level (FPL).*** The FPL’s quirky origin story is fascinating—it’s based on 1962 food costs—and the problems with FPL are well-documented in voluminous research. FPL It remains a touchstone in policy discussions nonetheless, mainly because it’s familiar and easily available from the US Census Bureau.

The main problem with FPL for purposes of assessing water and sewer affordability is that it’s insensitive to local costs of living. FPLS also a one-size-fits-all number national number (hence, “federal”): whether you live in pricey San Francisco or cheap Buffalo, the denominator is the same. Home energy, health care, taxes, and especially housing varies wildly across the country, but FPL is insensitive to those costs. Efforts like United Way’s ALICE are aimed at providing a more realistic assessment of cost of living by accounting for that variation.

The projection

Colton’s study projected affordability through 2030, which involved ​forecasting both numerators (price) and denominators (FPL). Once again, the report is vague about how it made those projections. A footnote on p.33 indicates that Colton used inflation rates from a 2017 U.S. Department of Energy’s report, which calculated 2008-2016 increases in water/sewer prices for selected U.S. cities based on average volumetric rate. It’s not clear whether Colton used city-specific projections for his twelve cities, or a 4.71% annual inflation rate, which ​​a footnote ​in a DOE study ​says was the “national average annual price increase for water and sewerage maintenance” from 1996-2016. It’s impossible to know exactly how Colton projected prices because price projections never appear in the report. To project the denominator, Colton uses 2010-2018 increase in FPL and assumes that the same growth will continue through 2030.

​It appears likely that Colton used straightline projection to forecast an already-inflated numerator using with either an eight- or twenty-year retrospective rate of change (it’s not clear which), either nationally or regionally (again not clear). He projected the FPL denominator using a different eight years of national FPL escalation.

The judgment

All of this culminates in a declarations of how many people or what percentage of the population suffers from “unaffordable” water. Those conclusions fuel The Guardian headlines, but require a binary definition of what is affordable. So how did the Colton report define affordability?

“In assessing whether a water bill was ‘affordable,’ the base level of affordability was set at 4% of income,” says the report on (p.8). That 4% threshold is the main definition of affordability in both the Colton report and The Guardian stories. The Colton report goes on to set “Affordable Burdens” for various income ranges.

Where did these affordability burden thresholds come from?

Colton made them up.

No poll, no Blue-Ribbon Committee, no philosophical inquiry on the meaning of affordability, no analysis of economic tradeoffs. They’re just arbitrary numbers. Every claim about the nation’s water affordability crisis and most of the scary graphs in The Guardian articles boil down to these thresholds. They’re based on nothing.

The stakes

Does any of this matter? Shouldn’t we just be happy that mainstream publications like The Guardian and Consumer Reports are paying attention to water affordability? Who cares if the methodological details are a bit dodgy, if the overall point—that water prices are rising faster than poor folks’ incomes—is generally fair?

There are at least three big reasons we should care about assessments that are so egregiously inaccurate. First, this study grossly misstates the scope and nature of the problem. It’s good to draw attention, but bad measurement can lead to the wrong inferences about what’s wrong and how to fix it.

The Colton report’s treatment of Austin, Texas is a great example of what’s so pernicious about poor measurement. The Guardian screams that water bills in the Texas capital increased 154% from 2010-2018, and will be unaffordable for 26% of all Austin residents by 2030. The naïve reader could be forgiven for thinking that Austin Water executives are mustache-twirling villains trying to squeeze money out of poor folks. But remember that Colton’s calculations are based 100 gpcd of demand—far higher than basic indoor needs. Austin Water employs a progressive rate structure that’s actually designed to protect affordability for basic needs and curb inefficiently high water use--and the big price increases start at 11,000 gallons monthly. Here’s the Circle of Blue plot of Austin’s water rates from 2010-2018:

At the more reasonable 50 gpcd, Austin’s monthly water prices increased an average of $2.28 annually from 2010-2018—still an increase, to be sure, but not quite the rolling disaster The Guardian describes. Meanwhile, Colton’s 100 gpcd assumption makes Philadelphia’s regressive, declining block rates look relatively affordable, even as the City of Brotherly Love sticks low-volume customers with higher prices than Austin's. Austin’s  progressive pricing is exactly the kind of thing that we ought to encourage to help affordability! Bad measurement leads to bad inferences about rate design.

Second, the arbitrary affordability thresholds that ​create sensational headlines ​preempt public debate over what affordability really means. Understanding the burdens and economic tradeoffs that low-income households face is critical to tackling the affordability challenge. But what exactly constitutes affordable water ought to be a matter of community values, not an analyst’s arbitrary judgment.

Finally, the ​(literally) incredible claims in The Guardian undermine legitimate efforts to assess and address the water affordability challenge. Studies that emphasize impact over accuracy risk achieving neither. Over time, ​slipshod studies can cause officials and the public to become cynical and dismiss an issue as overblown and ideological (see, for example, political discourse on COVID-19 prevention or climate change).

Water affordability is too important to allow to suffer that fate. Important issues demand responsible research.



*Apparently The Guardian missed this study (2015), this study (2017), this study (2018), this study (2019), and this study (2020). Each was peer-reviewed, nationwide in scope, and included far more utilities than the Colton report.

**The prices of water and sewer service are strangely absent from the Colton report. The actual prices of water/sewer service never appear in the 88-page report.

I’m guessing that Colton did something like multiply 12,000 gallon price to the ratio of average census tract household size and Circle of Blue’s assumed 4-person household. But that’s pure speculation.

40 Degree Day

The congressional COVID cavalry isn’t coming to save the water sector

"Nobody got nothing to say about a 40-degree day."

The ink was barely dry on a $2 trillion coronavirus response law when Congress started working on a second massive coronavirus relief bill. Water infrastructure was initially high on the congressional priority list for the next phase, with rumors of perhaps hundreds of billions of dollars for water and sewer systems. Structured carefully, a massive infusion of federal funds could help end shutoffs that threaten public health, give immediate relief for utilities reeling from lost revenue, help spur economic recovery through investment in badly-needed infrastructure, and maybe even drive fundamental structural reforms to the water sector. There was palpable excitement in the water sector as legislation was taking shape in April and early May. ​I even fielded a few inquiries from policymakers looking for guidance on structuring the bill!

On May 15 the House of Representatives passed the Health and Economic Recovery Omnibus Emergency Solutions (HEROES) bill—an unprecedented $3 trillion package of programs to aid “the economy, public health, state and local governments, individuals, and businesses” in response to the COVID-19 crisis. The bill now sits in the Senate, its final shape and prospects ​uncertain.

The outcome probably won’t matter much for the water sector, however: the HEROES bill passed by the House provides next to nothing for water and sewer systems, and provisions for low-income water bill assistance are structured in a way that will mainly help customers in large utilities with existing assistance programs. That, my friends, is what Stringer Bell would call a 40-degree day.

Aid for utilities?

Back in late March I argued for a formulaic, conditional grant program that would channel $70 billion in federal assistance directly to utilities that agreed to end shutoffs, ensure service to all occupied residences, forgive financial penalties accrued during the pandemic, and restructure prices to maintain affordability. Grants would be based on the poverty rates of utilities’ service areas. The main merits of the conditional grant model are reach and speed: the program would help nearly everyone, minimize administrative costs to utilities, eliminate qualification processes, and get the water flowing fast to stave off a public health emergency. It would also provide an immediate and durable economic stimulus, as utilities could use these funds flexibly to support jobs and capital investment.

Unfortunately, HEROES provisions for water systems are all conditions and no grants. The bill requires water systems that receive HEROES funds to end shutoffs, safely restore service to customers who had been shut off, and forgive penalties (😃).

And what do water and sewer utilities receive in exchange for giving up the main mechanism they have for ensuring timely payment? Limited subsidies for low-income assistance programs, and a boatload of administrative work (😐).

That’s it. That’s all. HEROES includes no categorical grants to water or sewer systems.*

I've never identified so much with a gymnast.

​Wither customer assistance?

​For a few years now, House Democrats have been pushing for a $1.5 billion Low-Income Household Drinking Water and Wastewater Assistance Program (LIWAP?), a LIHEAP-style policy for water bill assistance. I’ve argued before that a LIHEAP-style program isn’t an awful idea but has some pretty severe limitations. Congress apparently heeded that warning, as §190703 retains the LIWAP label but doesn’t build a new LIHEAP-style program. Instead, it would channel funds through state and Tribal governments to utilities to support water and sewer bill reductions for low-income customers.

Allotments to utilities would be based on federal income and poverty guidelines. In addition to ending shutoffs, restoring service, and forgiving nonpayment penalties, utilities that receive bill assistance funds would be subject to audit. To its credit, the House bill seems to acknowledge administrative burdens on customers associated with such programs: §190703(g) requires utilities to “conduct outreach activities designed to ensure that such households are made aware of the rate assistance” and to notify customers of the assistance that they receive. HEROES allows utilities to spend up to 8% of federal funds on support for administrative processes, but participating utilities would also be subject to mandated state audits.

The main problem with this approach is that it puts significant administrative obligations on utilities with relatively little payoff. It is difficult to see why small or medium-sized water and sewer utilities would opt to participate in a program that carries such onerous requirements for a program that will in most cases benefit a small, politically weak minority of their customers. Why would utilities with little organizational capacity agree to heavy shackles with so few shekels? Unfortunately, water/sewer affordability is, on average, worse in smaller systems. HEROES is unlikely to do much for the nation’s neediest water customers.

San Antonio has a good assistance program, and HEROES could make it better.

​The main beneficiaries of the $1.5 billion HEROES water assistance program will be large systems that already run assistance programs, since their administrative and audit processes are already in place.**

​HEROES funding would allow those large utilities to expand or extend benefits, and maybe boost their administrative capacity.

HEROES act? More like ZEROES act, amirite?

I dunno, man. The bill is more than 1,800 pages and funds everything from unemployment insurance to suicide prevention to wildlife biosurveillance. I sure didn’t read the whole thing, and there’s undoubtedly a lot of good stuff in there. The direct cash benefits at the heart of the bill will surely help lots of people. But the HEROES bill that emerged from the House of Representatives does little for water affordability directly, doesn’t help water/sewer systems generally, and certainly does not provide sufficient leverage to achieve more fundamental reforms to the water sector. The bill does give members of Congress the chance to take a position on water affordability and claim credit for tackling shutoffs. Whatever its fate in the Senate, the HEROES bill has accomplished those political goals.

​Systemic reforms ​to the U.S. water sector remain needed and possible, but will likely have to wait until COVID-19 recedes and a new Congress arrives.



*There’s one exception: HEROES allocates $20 million to the Bureau of Indian Affairs to support Tribal water provision, which really need help with capacity. More on this topic coming soon!

**Friends on Capitol Hill tell me that large utilities lobbied for this program.

Defense Against the Dark Arts

During a public health crisis, getting the research right is paramount

Beware the confundus charm

​It started with a tweet.

A new peer-reviewed Utilities Policy article on water utility ownership, low-income households, and shutoffs? From a pair of professors at major research universities? This was right up my alley!

The paper’s title—Does public ownership of utilities matter for local government water policies?—is intriguing. Water system ownership, regulatory policy, and especially shutoffs are enormously important and notoriously difficult to analyze due to data difficulties. The tweet and top highlight finding were provocative: “Cities and towns with government-owned utilities shutoff customer drinking water less.” These are headline-grabbing claims, sure to draw the attention of water sector leaders and policymakers looking for ways to tackle an ongoing public health crisis. I dove into the paper immediately, excited about what I might find!

Unfortunately, the study is deeply flawed. Few readers—even within the academy—have the appetite to get into the methodological details necessary to understand what the data really show. My deep dive revealed that the article’s authors came to profoundly incorrect conclusions. Last week I emailed them with my concerns. This post is my attempt to clarify what the article obfuscates and to set the record straight.

The claims

From tweet to title to text, Homsy & Warner’s article promises to explore the role that “local government utility ownership play[s] in meeting equity and environmental goals.” The study’s literature review frames its goals as an inquiry on “Publicly owned versus privately owned water supply” (section 2.3), with interest in public/private differences in “equity” (2.1) and “environment: water resource management” (2.2). Specifically, they’re interested in whether publicly owned water systems ​provide more protections against shutoffs for low-income households and greater water conservation. ​The implied comparison is with private water utilities.

The empirical analysis uses a 2015 ICMA survey sent to municipal chief administrative officers. The sample included all municipalities, towns, and counties with populations over 25,000 and a sample of smaller communities of populations between 2,500 and 25,000. The survey yielded 1,897 responses for a 22% response rate.* Respondents were asked whether their governments own water utilities. If the respondent said yes, the community was coded as having a “publicly owned utility” or “Government-owned utility”; if the respondent said no, then it was coded as having a “not publicly owned utility.”

When "Government-owned" does not necessarily mean owned by a government. 

The survey also asked questions about policy, including whether the city or county had taken action to "protect low-income households from water service shutoff.” The answer to this question is Homsy & Warner’s only measure of shutoff protection. They don't actually ​count shutoffs. About 8% responded that their governments provided shutoff protections.

These survey data were analyzed with a series of logistic regressions to predict the likelihood that a community has water shutoff protections as a function of public ownership and a series of other variables. Ownership emerges as a very strong predictor, leading Homsy & Warner to declare that: “if a utility is government-owned, the municipality is about two times more likely to have water shutoff protection policies.”

They conclude that “ownership matters, as communities with publicly owned utilities appear more inclined to protect residents from water service shutoffs.” That would be a big, important finding—if ​the data supported it. The trouble is that the data don’t show anything of the sort.

Where it all goes wrong: misleading measurement

The first hint that something was amiss was that just 55% of Homsey & Warner’s sample were publicly owned utilities, with the authors implying that the other 45% were served by private utilities. That struck me as extremely low, since about 85% of Americans get their drinking water from a local government, with about 15% served by private, investor-owned firms. How could the ICMA sample be so grossly skewed in favor of non-government utilities?**

​The answer is in the way that Homsy & Warner code “public ownership.” Recall that the ICMA survey was sent only to counties and municipalities and that respondents who reported that their governments own water systems were coded as “publicly owned.” What’s missing is special districts.

​​Special districts are local governments with narrow functions and limited powers; there are tens of thousands of such districts across the U.S.. The city and county officials responding to this survey who get their water from special districts would answer "no," and then Homsy & Warner would code the community's water system as "Not Publicly Owned."

Water districts, Potter. Remember the water districts.

Consider, for example, Central Arkansas Water. Headquartered in Little Rock, Central Arkansas Water serves a population of more than 450,000, including the cities of Little Rock, North Little Rock, Alexander, Sherwood, and Wrightsville, along with parts of unincorporated Pulaski County. Survey respondents from Pulaski County and all five of these cities would report that their governments did not own a water system. ​Homsy & Warner would code their water systems as "not publicly owned," even though a government--Central Arkansas Water--supplies their water.

​Around 30% of the local government water systems in the United States are owned and operated by special districts. These special districts are definitely local governments.​ Without accounting for special district utilities, we cannot infer anything about public/private water system ownership from these data.

Sins of inference

But it gets worse. The study’s first highlighted claim is that “cities and towns with government-owned utilities shutoff customer drinking water less.” Thing is, this study doesn’t measure shutoffs. It measures whether a municipal or county government has a policy to protect low-income households from shutoffs. Reliable shutoff data are notoriously hard to find, which is why serious research on the subject is rare. A reader who doesn’t get into the empirical weeds wouldn’t notice that data don’t support this claim.

Apart from problems with coding ownership and counting shutoffs is the question of what this all means for water governance. What should we infer from the fact that a municipality that owns a water system is more likely to have water shutoff protections than a municipality that does not own a water system? Homsy & Warner conclude that “communities with publicly owned utilities appear more inclined to protect residents from water service shutoffs.” That’s a bit like finding that people who own cars are more likely to have jumper cables than people who don’t own cars. ​Would we then infer that car owners care more about their families' transportation than transit riders?

Why it matters

Policymakers, advocates, regulators, and utility managers are looking for answers to a historically tough challenge in the COVID-19 pandemic. Researchers everywhere are working hard to find ways to help protect public health. ​

We certainly need to understand water shutoffs and how to prevent them. But the stakes are too high for policy researchers to play fast and loose with data and inferences. This isn’t an abstract, theoretical squabble over the literary interpretation of Hamlet. Real policies to protect real people in a moment of real crisis are on the line, and our communities need valid findings. The urgency of pandemic only heightens the need for ​rigorous, responsible policy research.



*It does not appear that Homsy & Warner adjusted their estimates to account for sample stratification or non-response bias.

**With just a 22% response rate, some of the issue is probably nonresponse bias. But it’s unlikely that nonresponse accounts for a 30% gap in share of publicly owned utilities.

In this sentence and throughout the article, Homsy & Warner are vague about what publicly owned utilities are being compared with.