During a public health crisis, getting the research right is paramount
It started with a tweet.
A new peer-reviewed Utilities Policy article on water utility ownership, low-income households, and shutoffs? From a pair of professors at major research universities? This was right up my alley!
The paper’s title—Does public ownership of utilities matter for local government water policies?—is intriguing. Water system ownership, regulatory policy, and especially shutoffs are enormously important and notoriously difficult to analyze due to data difficulties. The tweet and top highlight finding were provocative: “Cities and towns with government-owned utilities shutoff customer drinking water less.” These are headline-grabbing claims, sure to draw the attention of water sector leaders and policymakers looking for ways to tackle an ongoing public health crisis. I dove into the paper immediately, excited about what I might find!
Unfortunately, the study is deeply flawed. Few readers—even within the academy—have the appetite to get into the methodological details necessary to understand what the data really show. My deep dive revealed that the article’s authors came to profoundly incorrect conclusions. Last week I emailed them with my concerns. This post is my attempt to clarify what the article obfuscates and to set the record straight.
From tweet to title to text, Homsy & Warner’s article promises to explore the role that “local government utility ownership play[s] in meeting equity and environmental goals.” The study’s literature review frames its goals as an inquiry on “Publicly owned versus privately owned water supply” (section 2.3), with interest in public/private differences in “equity” (2.1) and “environment: water resource management” (2.2). Specifically, they’re interested in whether publicly owned water systems provide more protections against shutoffs for low-income households and greater water conservation. The implied comparison is with private water utilities.The empirical analysis uses a 2015 ICMA survey sent to municipal chief administrative officers. The sample included all municipalities, towns, and counties with populations over 25,000 and a sample of smaller communities of populations between 2,500 and 25,000. The survey yielded 1,897 responses for a 22% response rate.* Respondents were asked whether their governments own water utilities. If the respondent said yes, the community was coded as having a “publicly owned utility” or “Government-owned utility”; if the respondent said no, then it was coded as having a “not publicly owned utility.”
The survey also asked questions about policy, including whether the city or county had taken action to "protect low-income households from water service shutoff.” The answer to this question is Homsy & Warner’s only measure of shutoff protection. They don't actually count shutoffs. About 8% responded that their governments provided shutoff protections.
These survey data were analyzed with a series of logistic regressions to predict the likelihood that a community has water shutoff protections as a function of public ownership and a series of other variables. Ownership emerges as a very strong predictor, leading Homsy & Warner to declare that: “if a utility is government-owned, the municipality is about two times more likely to have water shutoff protection policies.”
They conclude that “ownership matters, as communities with publicly owned utilities appear more inclined to protect residents from water service shutoffs.” That would be a big, important finding—if the data supported it. The trouble is that the data don’t show anything of the sort.
Where it all goes wrong: misleading measurement
The first hint that something was amiss was that just 55% of Homsey & Warner’s sample were publicly owned utilities, with the authors implying that the other 45% were served by private utilities. That struck me as extremely low, since about 85% of Americans get their drinking water from a local government, with about 15% served by private, investor-owned firms. How could the ICMA sample be so grossly skewed in favor of non-government utilities?**The answer is in the way that Homsy & Warner code “public ownership.” Recall that the ICMA survey was sent only to counties and municipalities and that respondents who reported that their governments own water systems were coded as “publicly owned.” What’s missing is special districts.
Special districts are local governments with narrow functions and limited powers; there are tens of thousands of such districts across the U.S.. The city and county officials responding to this survey who get their water from special districts would answer "no," and then Homsy & Warner would code the community's water system as "Not Publicly Owned."
Consider, for example, Central Arkansas Water. Headquartered in Little Rock, Central Arkansas Water serves a population of more than 450,000, including the cities of Little Rock, North Little Rock, Alexander, Sherwood, and Wrightsville, along with parts of unincorporated Pulaski County. Survey respondents from Pulaski County and all five of these cities would report that their governments did not own a water system. Homsy & Warner would code their water systems as "not publicly owned," even though a government--Central Arkansas Water--supplies their water.
Around 30% of the local government water systems in the United States are owned and operated by special districts. These special districts are definitely local governments. Without accounting for special district utilities, we cannot infer anything about public/private water system ownership from these data.
Sins of inference
But it gets worse. The study’s first highlighted claim is that “cities and towns with government-owned utilities shutoff customer drinking water less.”† Thing is, this study doesn’t measure shutoffs. It measures whether a municipal or county government has a policy to protect low-income households from shutoffs. Reliable shutoff data are notoriously hard to find, which is why serious research on the subject is rare. A reader who doesn’t get into the empirical weeds wouldn’t notice that data don’t support this claim.
Apart from problems with coding ownership and counting shutoffs is the question of what this all means for water governance. What should we infer from the fact that a municipality that owns a water system is more likely to have water shutoff protections than a municipality that does not own a water system? Homsy & Warner conclude that “communities with publicly owned utilities appear more inclined to protect residents from water service shutoffs.” That’s a bit like finding that people who own cars are more likely to have jumper cables than people who don’t own cars. Would we then infer that car owners care more about their families' transportation than transit riders?
Why it matters
Policymakers, advocates, regulators, and utility managers are looking for answers to a historically tough challenge in the COVID-19 pandemic. Researchers everywhere are working hard to find ways to help protect public health.
We certainly need to understand water shutoffs and how to prevent them. But the stakes are too high for policy researchers to play fast and loose with data and inferences. This isn’t an abstract, theoretical squabble over the literary interpretation of Hamlet. Real policies to protect real people in a moment of real crisis are on the line, and our communities need valid findings. The urgency of pandemic only heightens the need for rigorous, responsible policy research.
*It does not appear that Homsy & Warner adjusted their estimates to account for sample stratification or non-response bias.
**With just a 22% response rate, some of the issue is probably nonresponse bias. But it’s unlikely that nonresponse accounts for a 30% gap in share of publicly owned utilities.
†In this sentence and throughout the article, Homsy & Warner are vague about what publicly owned utilities are being compared with.
Lessons from California water conservation, 2019
Tough water times may be back in California. After the Golden State suffered through a historically severe drought from 2012-2017, pleasantly wet weather in 2018-2019 refilled reservoirs and replenished mountain snowpacks. But the state’s drought monitor shows that the past few months have trended drier, and water managers are worried that the state might be slipping back toward another drought.
One fortunate legacy of California’s recent struggles with drought is the California Water Board’s investment in data collection and dissemination. Researchers are learning important lessons about water management from that wealth of data on water consumption and conservation. Complete data for 2019 are now available, and I’ve just had a first look.
Three things emerge from my initial cut at the 2019 data that merit mention against the ominous backdrop of a looming drought.
1. A way of life
California’s overall urban water use remains down significantly from the pre-drought days. The state government established emergency conservation rules during the peak of the drought, cutting statewide urban water consumption by nearly 25%. But Governor Jerry Brown and other leaders also vowed to make water conservation “a way of life” that would extend beyond the emergency.
Californians seem to have taken the goal to heart, at least in aggregate. Though water conservation is not at the zealous peak we saw in 2015-2016, it remains strong:
Falling conservation in 2018 led to some hand-wringing, but conservation rebounded in 2019 to a statewide aggregate 18.1% relative to the state’s official 2013 baseline. That’s pretty remarkable and suggests that much of the reduction in California’s urban water demand is more or less permanent. Absent severe drought or monsoon-like precipitation, California’s overall water conservation will likely remain in the mid-to-high teens for the foreseeable future. That’s a public policy success story.
2. Persistent public-private disparity
One of the most fascinating findings that emerged from my earlier analysis of the California drought with Youlang Zhang and David Switzer was that the state’s private, investor-owned utilities conserved significantly more water than local government utilities during the drought.* We linked that difference in drought response to the institutions that govern water finance. We also found that the public-private conservation gap persisted even after the drought ended, with the greatest disparities during the summer when water demands are at their highest. Back in 2018 your humble blogger forecasted that that “public and private conservation will converge in the spring and diverge again in the summer and autumn.” And indeed, the pattern held: there was essentially no difference between public and private water utility conservation during the winter months, but during the 2018 May-September peak season, California’s investor-owned systems saved more water than their local government counterparts.
Did the pattern persist in 2019? In short, yes:
As you can see, public and private utility conservation move in pretty close parallel. During the non-peak period (January-April and October-December), there is essentially no difference in average conservation by ownership. But a public-private gap emerged again during the May-September peak period. In 2018 the difference was 2.3%; in 2019 the gap was smaller—investor-owned utilities saved an average of 1.5% more water than local governments. To give that percentage some context, think of it this way: if public utilities saved at the same rate as private utilities in 2019, the difference would have been about 15 billion gallons, roughly equivalent to the City of Long Beach’s annual water consumption.
The public-private disparity in summertime conservation now also appears to be a way of life in California. The difference is almost certainly related to pricing and revenue. Decoupling über alles.
3. Indoor outpacing outdoor?Finally, the five years of data now in front of us show how California’s urban water conservation relates to overall seasonal demand. In 2015, when severe outdoor irrigation restrictions were in effect for much of California, 70% of the state’s conservation came from reduced demand in the peak season. Since then, the share of peak season consumption has fallen to about half:
In 2019 non-peak conservation crept up to 52% of overall savings. It’s hard to infer exactly what’s driving this pattern from aggregated monthly data, but the picture suggests that the most persistent water savings seen in California over the past five years has come from indoor efficiency. Average water consumption in 2019 was about 98 gallons per person per day (gpcd), but some cities are far lower. For example, water demand in conservative San Francisco shows hardly any seasonal peak at all, and averages just 42 gpcd—a level approaching the theoretical minimum to sustain developed world conditions.
While there are probably still plenty of savings to be had from indoor efficiency, meeting the challenge of another potential drought year—and driving further sustainable conservation—will require tackling peak demands through pricing and regulation. Get ready for another year of water cops and rate revolts.
© 2020 Manny Teodoro
A California Surprise, Part 3
California’s private utilities continued to out-conserve public utilities even after the state lifted its mandate.
In 2015 the California State Water Resources Control Board (SWRCB) ordered drinking water utilities to reduce water usage by 25% statewide. As I reported in an earlier post, something surprising happened: compared with local governments, the state’s private, investor-owned utilities imposed stricter water use regulations, were nearly twice as likely to comply with the state mandate, and conserved significantly more water overall. My last post traced this disparity to differences in the institutions that govern water rates in California and the conservation incentives that those institutions create.
In June 2016 the state declared an end to the drought emergency and lifted the mandatory conservation rules, and allowed utilities to set their own conservation targets. Somewhat to the state’s chagrin, 84% of utilities–including nearly all of the state’s private utilities–responded by setting their conservation targets at zero (so much for self-regulation!). Still, conservation has continued, with the state’s utilities still using far less water on average than they did before the drought.
But curiously, the public-private conservation disparity has persisted well after the state mandate ended and most conservation targets dropped to zero.
Even after the state lifted the rules in June 2016, private utilities continued to impose stricter irrigation restrictions on average compared with public utilities. Through February 2017, public utilities allowed an average of about one more day per week relative to private utilities. In the spring of 2017 the gap began to narrow. In June 2017 the relationship switched: since then, public utilities have been imposed tougher irrigation restrictions. The cause of that switch isn’t clear, and we haven’t really looked into it.
But the really surprising thing is the persistent disparity in overall conservation. This graph plots average conservation (relative to the same month in 2013) for public and private utilities from June 2016 through the end of calendar 2017:
As you can see, overall conservation declined for public and private alike as drought conditions eased. But the really interesting thing here is the consistent difference between public and private sectors: except for February through May 2017, private utilities conserved about 2-3% more on average than public utilities.
While the difference doesn’t seem like much in percentage terms, in a state as large as California, a few percentage points can be enormous in absolute volumes of water. Had public utilities saved at the same rate as private utilities over that period of time, the difference would have been about 52 billion gallons—more water than San Francisco uses in two years. Regression analysis in my forthcoming article coauthored with Youlang Zhang and David Switzer confirms what the graph implies: after accounting for demographics, population density, climatic conditions, source water, and other variables, we find that California’s private utilities out-conserved public utilities in the post-mandate period by a statistically significant average of 2.7%.
Decoupling is a hell of a drug
This consistent public-private difference lends greater weight to the idea that rate decoupling facilitates water conservation for private utilities, and that political constraints hamper public sector conservation. If 2018 holds to form, public and private conservation will converge in the spring and diverge again in the summer and autumn. Look for an update here in early 2019 when full 2018 data are available.