The water comms test kitchen serves up a low-tech triumph
*chef's kiss*
A couple years ago I had the opportunity to work with Jean Smith at the Northeast Ohio Regional Sewer District (NEORSD) to promote their low-income customer assistance program with a sophisticated, large-scale field experiment. This extraordinary study yielded important, actionable findings—including some real surprises. Our full research paper is now forthcoming in Public Administration Review. This post serves up a succulent summary.
Utility bill assistance in Northeast Ohio
NEORSD is a sewer utility that serves a population of roughly a million in the Cleveland area. NEORSD offers several low-income customer assistance programs (CAPs) in response to growing affordability challenges. Water/sewer CAPs usually suffer from low take-up, with a small minority of eligible households participating.* NEORSD was no different: in 2022, less than 2% of eligible customers were enrolled in the district’s “Rate Reduction” program.
Ohio’s largest sewer system and the world’s greatest Twitter 𝕏 account
Heavy administrative burdens are among the most oft-cited reasons for lackluster CAP participation, starting with simple awareness: most CAPs require customers to sign up, but a customer can’t receive assistance if (s)he doesn’t know that the program exists.
Utilities are trying lots of things to promote CAPs. Over the years, NEORSD has tried robo-calls, social media, radio advertising, in-person resource fairs, and more. Participation lagged nonetheless. NEORSD leaders weren’t sure how much any of these efforts help, or which are the most cost effective. That’s a common theme in water comms : case studies are common and earnest advice abounds, but we actually know very little about what really works and what doesn’t when it comes to CAP promotion.
So Jean and I decided to start with a staple of utility communications, direct mail. In early 2023 NEORSD fielded an experiment that used dozens of postal “treatments” across its customer base to discover whether direct mail works, and, if so, which features are more and less effective in promoting the district’s CAPs.
Cooking up a taste test
That’s effectively the approach that Jean and I took for NEORSD’s direct mail campaign: our experiment was a systematic "taste test" using direct mail as our culinary creations. Instead of sugar and flour, we varied key elements of the mailers:
- Mailing Type: Letters versus postcards.
- Mailing Frequency: Single mailings versus two-wave mailings (an initial mailing followed by a reminder two weeks later).
- Color: Color versus black-and-white.
- Sender: Mailings from NEORSD versus those from a local non-profit partner.
- Discount Framing: Presenting the program's value as a percentage discount ("Save 40%") versus an estimated dollar amount discount ("Save $29 per month").
- Language: English-only versus bilingual (English and Spanish) mailings.
We had reason to think that each of these variables might affect response rates, based on published research and NEORSD’s past experiences. Of course, these things can all vary simultaneously in ways that can contravene or reinforce each other. To tease out the effects of all these variables simultaneously, we used a conjoint design. Conjoint design is a sophisticated method that allows researchers to determine the relative importance of different features by randomly varying them simultaneously and then observing how these variations influence outcomes. This approach allowed us to test various combinations of mailing characteristics to identify which features were most appealing.
For example, here’s a black-and-white postcard in English from NEORSD that frames the CAP as a 40% discount:
click to embiggen
As another example, here’s a colored, bilingual letter from a Cleveland community organization that frames the CAP discount as a $29 dollar average savings:
click to embiggen
Ultimately, we tested a total of 64 unique mailings (recipes?). A thousand single-family customers were randomly assigned to each recipe, resulting in 64,000 participants in this grand "taste test." The other 220,000 NEORSD customers served as a control group. The initial wave of mailings was distributed in January 2023, with a second wave sent two weeks later for select recipes. NEORSD suspended all other promotional efforts while the experiment was in progress to help improve our confidence that the mailers were working (or not).
NEORSD customer service staff logged all contacts in which customers inquired about assistance programs. We linked those inquiries to treatment and control group records, then used statistical analysis to figure out whether our mailings caused increased CAP inquiries and how different mailing characteristics affected the likelihood of CAP inquiries.
Proof of the pudding
So does direct mail work?
Yes, emphatically.
Here are the average the response rates for customers who received nothing (the control group) vs customers who received at least one mailing (the treatment groups):
Treatment N=64,000; Control N=220,233. Thin bars represent 95% confidence intervals.
That’s an average 1.5% response for customers who got at least one mailer compared with 0.8% for those who did not—customers who received mailers were nearly twice as likely to contact NEORSD to inquire about assistance. This result probably understates the impact of direct mail because CAP information almost certainly “spills over.” That is, some folks in the control group probably got information that was targeted at people in the treatment groups; a customer who receives a letter or postcard might share it with friends or neighbors who got different mailings or no mailings at all. That actually makes us more confident that the mailers work. In practical terms, this experiment is best understood as a test of a direct mail campaign.
Key ingredients
Did different mailing characteristics lead to better or worse response rates? Let’s start with the medium (letters vs postcards) and the number of mailings (one or two). Here’s what we found:
Thin bars represent 95% confidence intervals.
Both types of mailings yielded higher response rates compared with the control group, and notably, postcards generated higher response rates than letters. While the difference between letters and postcards is not statistically significant, it’s worth emphasizing because postcards are much cheaper than letters to produce and send. In this experiment, NEORSD spent about 47 cents on production, postage, and mailing service for each postcard, compared with 64 cents for each letter. Repeated mailings resulted in a roughly 0.3% increase in response rate compared to the average for all types of mailings and is clearly better than a single letter. However, the difference between a single postcard and two mailings isn’t statistically significant.
Maybe our most surprising result of all is a “non-effect.” Here how different mailing characteristics affected response rates:
Thin bars represent 95% confidence intervals.
There’s no statistically significant difference between black-and-white postcards and color postcards. Likewise, there’s no meaningful difference between letters from the utility and letters from a community nonprofit, or between dollar discount and percentage discount messages, or between English-only an English-Spanish bilingual mailings.
Conclusions for water comms cooks
What do we take away from all this? First, it’s clear that direct mail works to promote customer bill assistance. Second, the cheapest, simplest mailer of all—the humble black-and-white postcard—proved to be the most effective. That’s great news for utilities with tight communications budgets. Third, the nuances of messaging don’t seem to matter all that much. Utility communicators need not get wrapped around the axle with details of mailer design; once again, simplicity rules the day.
Finally, this study is another useful illustration of a scientific approach to water communications. Don’t just serve up best guesses—take your recipes to the test kitchen and find out what really pleases the public palate!The part where Manny chats with himself
poscards? really?
I mean, yeah. Why not?
bro this is the digital age
serve me up some cyberspace AI algorithmic magic
USPS delivers nearly 3 billion pieces of direct mail advertising every year, generating $700 million in revenue. Direct mail isn’t sexy, but it works.
postcards are the opposite of sexy
Postcards require very little effort by the recipient and are more accessible to many low-income people. Another recent study found that old-school postcards outperformed online communication when promoting low-income assistance.
and plain old black-and-white is better than color?
Yes! Isn't it great when the cheapest technology is also the best technology?
but that's so boring
There’s growing evidence that boring is good when it comes to communications between government agencies and the public. It's called the formality effect. At a moment when we’re all constantly bombarded with flashy, colorful images, an austere black-and-white document seems to signal legitimacy.
whatever, man
y'all got 1.5% response and you’re carrying on like you figured out cold fusion
The typical response rate for mass direct mail is around 1.0%, so our 1.5% is better than average for the industry
so you're saying a postcard is the best way to communicate with customers?
I’m saying direct mail is an effective way to communicate, and black-and-white postcards are especially effective. Direct mail is also cheap, and postcards are especially cheap.
ok fine
so should utilities all use direct mail to promote CAPs?
Depends on the alternatives.
With CAP advertising the idea is to promote the CAP to people who otherwise wouldn’t know about it. The way to think about planning an outreach campaign is cost per respondent: how much does the utility need to spend on outreach to contact each potential program participant? Here’s how it breaks down for NEORSD in this experiment, given its mailing costs and the response rates we got:
With black-and-white postcards, each new contact cost $31.25. Whether that’s a good value or not depends on the cost per respondent for the alternatives. Postcards clearly are more cost-effective than other kinds of mailings, and might be better than TV, radio, or online advertising. It's probably a lot more cost-effective than labor-intensive methods like door hangers or in-person events.
could postcards work for other utility communication needs, too?
There’s every reason to think that postcards can be effective in communicating about conservation, rate increases, water quality, job recruitment, and more. In fact, you could try—
are you seriously talking about more exp—
—experiments to see what kinds of direct mail work for those purposes. Then share the results with the world so that the whole water sector can learn from successes and failures. That’s how we make progress.
how about that bilingual result?
I dunno, man.
Going into this study, I was sure that the bilingual mailers would boost response. About 4% of Cuyahoga County’s population and about 8% of its poor population speaks Spanish.
if i’m reading this right, bilingual messages had a negative effect 
That’s right. And the effect is flirting with statistical significance. It’s notable because it’s so unexpected. Water comms experts often advocate for bilingual messaging, and that makes intuitive sense. So that result really caught us by surprise.
your intuition was wrong then
That’s why we science, dear electronic alter ego.
Intuition tells us the world is flat. It’s critical to subject our ideas to rigorous testing.
could it be just a spurious result, or something weird about cleveland?
Probably not. I was really troubled by that finding, so I read a little more scientific literature. Turns out a couple of other studies** found something eerily similar: bilingual communications seem to evoke negative responses from majority language speakers in some situations. I’m not sure what cognitive processes cause that, but utilities ought to be careful about when, where, and how they deploy bilingual communications. It's not as straightforward as we thought.
Most of all, we need more and better science to guide water comms.
Be honest.
i am being honest!
i read that—
Be honest.
i watched a brad pitt movie
Thank you.
it seems like all the experiments make comms “experts” obsolete, like sabermetrics did to the old-school baseball scouts
should utilities ditch their comms people?
On the contrary, the experiments demonstrate powerfully that communications matter. We need to raise our comms game.
A body of rigorous research is the key to evolving utility comms from a craft to a profession. Science isn’t a threat to water comms—science is the key to maximizing its impact.
how about you maximize me a slice of that cake y'all baked
The cake was a metaphor.
There is no actual cake. Just mail.
it would be better with cake
you should do it with cake next time
Why do I even bother.
*An assistance program with low take-up can be worse for affordability than no program at all.
**See this study and this study, for example.
Loved this, Manny- for the simple way you explain your scientific approach to helping understand utility communications. And the thought of potentially cooking up black and white postcards for recruiting. That’s an interesting idea …
Thanks for this! Fantastic explanations throughout, backed by real data. Clear and funny! I look forward to working with you on another study in California to continue amplifying your research and results.