Plain wrong: Millions of utility customers’ passwords stored in plain text
By Jim Salter
9 - 11 minutes
In September of 2018, an anonymous independent security researcher (who we'll call X) noticed that their power company's website was offering to email—not reset!—lost account passwords to forgetful users. Startled, X fed the online form the utility account number and the last four phone number digits it was asking for. Sure enough, a few minutes later the account password, in plain text, was sitting in X's inbox.
This was frustrating and insecure, and it shouldn't have happened at all in 2018. But this turned out to be a flaw common to websites designed by the Atlanta firm SEDC. After finding SEDC's copyright notices in the footer of the local utility company's website, X began looking for more customer-facing sites designed by SEDC. X found and confirmed SEDC's footer—and the same offer to email plain-text passwords—in more than 80 utility company websites.
Those companies service 15 million or so clients (estimated from GIS data and in some cases from PR brags on the utility sites themselves). But the real number of affected Americans could easily be several times that large: SEDC itself claims that more than 250 utility companies use its software.
Most of the information in this story came to Ars directly from X, who was frustrated with the extreme indifference of the companies involved after several months of continued effort to work with them. I confirmed the technical details of X's claims by examining many of the affected sites myself, and by reviewing X's communications with their own utility company and with SEDC. EFF Senior Information Security Counsel Nate Cardozo (who will famously be leaving to help WhatsApp clean up its privacy issues) and EFF attorney Jamie Williams have been advising X concerning legal and ethical disclosure responsibilities during the entire process—because even today, the threat of legal action may come before a potentially flawed company offers anything resembling thanks or takes the necessary steps towards better security hygiene.
Ars reached out to X's utility company and to Mark Cole (SEDC's General Counsel), but we received no response from either.
X did not attempt to discover any direct exploits into any of the utility firms or into SEDC's own payment site. There's no smoking gun here that says "somebody can just grab all these passwords and run." That said, website compromises are much like entropy: they trend toward the maximum. Once a website is breached, the first thing attackers do is to dump the password database.
This isn't generally necessary to access accounts on the compromised site itself; once you've got root in the infrastructure, odds are pretty good you can already do whatever you'd like in that site. But what makes those user passwords so valuable is their use as pivot points.
Most users, unfortunately, re-use passwords between different websites and Internet-accessible accounts with wild abandon. They may change it up a little bit—add a number on the end here, stick a special character in the middle there—but this doesn't actually add a significant degree of security. Modern penetration tools (like Burp Intruder) automatically "fuzz" passwords as necessary when the attacker attempts to use them to access other, more valuable resources.
These exploitable resources include just about anything from which you can make money; eBay accounts, Amazon accounts, even World of Warcraft accounts are popular targets for an attacker looking to make a quick buck. Even more worryingly, if a user re-used their email password (or some variant thereof) to log into the compromised website, the attacker can leverage access to the user's email account to reset the passwords to just about anything else the user accesses online.
This is romper-room security stuff for 2008, let alone 2018. Arguably, the utility firms that contracted with SEDC for billing software and services are large enough—and responsible for enough customers' data—that they should be aware of and diligent about this kind of problem themselves. In reality, most companies are an awful lot like end users: they don't know, or care, as much about security as the typical Ars Technica reader might like.
In 2019, it's ridiculous that vendors are replying to security researchers via general counsel, not a bug bounty program.
Nate Cardozo, Senior Information Security Counsel, EFF
If you find yourself wondering—"What's so bad about storing passwords as plaintext?"—it comes down to this: passwords are among the most important data assets any organization has. In much the same way, companies get fire insurance with the hope they'll never use it, organizations must have robust hashing regimens to protect passwords in the event there is ever a breach. This point has come up on Ars again and again over the years, and it's explained well by password cracking expert Jeremi Gosney in this story about a 2015 hack against LastPass:
If we could trust computers to keep secrets a secret, then we wouldn’t have to worry about protecting sensitive data at rest. But we can’t, so we do. Password databases can be compromised through a myriad of vectors—up to and including physical theft—and you have to plan for the eventuality that your database will be compromised. How you protect the data in the database is what really matters, and this is precisely why we have password hashing, and this is also why the threat model for password hashing starts with a compromised password database. Think of password hashing as an insurance policy. The stronger the password hashing is, the more time you buy for yourself and your users in the event of a breach: time to identify and contain the breach, time to notify your users, and time for your users to update their passwords.
What about SEDC? No company that bills itself as "cyber security experts" should need the security implications of plain-text password storage explained to them. This software shouldn't have been written this way in the first place, and the response to the disclosure of this vulnerability should have been rapid, courteous, and sincere—not a protracted, hostile back-and-forth with legal counsel.
When X informed SEDC there was a security problem, the corporate response varied from crickets chirping to cold shoulders. Eventually, SEDC's attorney sent X an email that could reasonably be paraphrased as: "there's no problem with what SEDC is doing, stop bothering SEDC, and you're only allowed to talk to me from now on."
In addition, the letter suggested the problem wasn't a serious security issue. As SEDC general counsel Mark Cole put in in an email:
We were especially surprised by your accusation because SEDC and many of our customers undergo annual PCI assessments, annual PCI penetration tests, and quarterly PCI ASV scans, none of which have identified as a PCI DSS vulnerability the practice of which you have complained. After your initial calls to [redacted] Electric Cooperative, we expressly raised your concern with a certified PCI Qualified Security Assessor who confirmed that your issue does not present a PCI violation: the password attached to a non-administrator end user userid simply does not allow access to credit card information in a manner that violates PCI DSS.
Finally, I must request that you cease contacting SEDC employees, customers (other than any utility of which you may be an end-user), and third parties to repeat your erroneous assertions about this matter.
Mark Cole, General Counsel for SEDC
There's a fair bit to chew on here. X believed that by storing customer passwords in plain text, SEDC could be violating PCI-DSS, the industry regulations governing storage of customer billing information. Cole objected strenuously—and very directly—to this idea. I have been personally responsible for preparing for and responding to numerous PCI-DSS audits for client firms over the years, and based on this experience with the regulations, I think it's probable that Mr. Cole is correct. The customer passwords in question appear to be stored "on premises" with the client utility firms, not in SEDC's datacenter. While having access to the password grants a potential attacker the ability to view power bills or to add credit cards to a customer's profile, it probably can't be used to gain the customer's own credit card information.
The utility firm websites are typically hosted on business IP addresses belonging to local fiber providers, such as Comporium. This heavily implies that the sites are hosted on each utility firm's own premises. When a customer clicks the link to actually pay a bill, the local site first links out to a subdomain of sedata.com, which is hosted in a datacenter somewhere in Georgia (the home state of SEDC). This is a common technique for shielding small businesses from the extremely painful restrictions that PCI-DSS applies to data storage; by segregating credit card data to a specialist firm and never collecting that data on your own servers at all, you can just answer "not applies" to the majority of your PCI-DSS audit requirements.
I am only addressing the PCI-DSS question here because Cole appears to be framing this situation solely as a PCI-DSS issue. But let's be clear: storing passwords in plaintext is a hideous security practice regardless of industry regulations, and one that should not be tolerated. It should especially not be tolerated in municipal utilities, where customers have little choice about doing business—it's not like you can "just choose a different power company" if the one servicing your area is playing fast and loose with online security.
Where things stand now
Many of SEDC's client firms have been patched to some degree in the last couple of weeks. Since neither X nor I have attempted to access any account other than X's own, there's no way of knowing for sure how many of the utility firms are still actually emailing passwords to users on request; we can only note that most of them still offer to do so. X's own utility company still appears to offer emailed passwords, but putting X's account ID and the last four digits of X's telephone number into the form now results in an emailed link to a password reset form instead of an actual emailed password.
So is the situation "fixed"? It's unclear. SEDC's counsel—who did not respond to Ars request for an interview—gave as little technical information as possible during the entire 120+ day saga with X. "In 2019, it's ridiculous that vendors are replying to security researchers via general counsel, not a bug bounty program," Cardozo noted. Cole's final correspondence with X is both careful and cagey. It says, in part:
I wanted to let you know SEDC has changed the way our software handles “forgotten password” requests for the payment portal, and we have disclosed the change to all our Customers. We also have disclosed this change and the history of your communications of which we are aware—with SEDC and our employees, with some of our Customers, and with social media generally—in detail to our Board of Directors, which is comprised of a dozen of our Customer-Members. They do not believe any further “disclosure” by SEDC is needed or appropriate.
Given that there has been no PCI violation nor any indication of third party access to anyone’s PII (in fact, the plain-text password at issue does not enable such access), it is unclear what “disclosure” you think should be made, much less under what authority you think such a disclosure would be required.
Mark Cole, General Counsel for SEDC
What Mr. Cole did not say is that "the passwords are now encrypted," let alone that "they are encrypted now, using a strong hash, with cryptographic salt unique to each record."
The presence of antiquated Dot Net Nuke headers in the utility firms' HTML source doesn't inspire much confidence in the security situation, either. The HTML source of modern DNN sites shows copyright dates extending through 2017, where the ones at SEDC's utility firm clients only claim copyright dates through 2013. It's possible that SEDC has backported security fixes for several known vulnerabilities for older versions of Dot Net Nuke—but without frank and complete disclosures, there's no way to know for certain without directly "penetration testing" the sites themselves.
Keep yourself safe
As bad as plain-text password storage is, it only affects you significantly if you re-used that password—or "variations on" that password—in the first place. So don't do that! (And tell your friends and family not to do it.) Your dog's name is a terrible password. It's still terrible if you put an exclamation mark at the end of it and swap the "i" in "Fido" with a 1.
But a thirty-digit, completely random string of ASCII characters is also a terrible password if you use it in more than one place.
For many people, your best bet is generally to use—but not to re-use!—memorable passphrases, and storing them in a password manager such as KeePass if and when you need reminders. But be cautious about becoming dependent on password managers that randomly generate strings of ASCII for passwords and automatically fill in forms by way of browser plugins; while they're convenient, they leave you absolutely crippled when you don't have access to that manager. (Browser plugins also represent a potential vulnerability; a coding error in a plugin might allow an attacker to harvest a hundred passwords at once via hidden iframes in a watering hole attack.)
Where possible, you should also turn on two-factor authentication, such as Google Authenticator or a Yubikey. But be careful about this, too—if your phone or other authentication device gets lost or destroyed, and you haven't carefully stored a one-time-use recovery code in a password database or safe deposit box, you can easily find yourself locked out of your own accounts—possibly permanently. (Resetting 2FA on Linode, Digital Ocean, and other services is the thing I hate the most about replacing my own phone.)
Password storage issues like this one are maddening no matter what—but if you practice good password hygiene, they don't need to be scary.