Troy Hunt’s blog showcases a lot of the different issues with which he is familiar. He is a Microsoft MVP and Pluralsight author whose credentials also include working with Pfizer. His blog posts focus on customer and individual user interfaces and security. Written with an approachable tone, this blog is a great one for the non-technical c-suite reader.
It's another day-late weekly update courtesy of another hectic week. Scott and I were at NDC Sydney doing a bunch of talks and other events and I just simply didn't get time to push this out until sitting at the airport waiting for the plan home.
This week's update is a little different as we did it at SSW's recording setup in front of a live audience. Better video, better audio and some questions asked in the process too. Other than that, it's business as usual: more keyloggers on payment forms, more data breaches and a massive extended validation smack-down.
Lastly, just as I went to publish this post, I noticed SSW had taken down the original video. I've reached out to them to get a new link, but I managed to download and publish the audio earlier on so I'm publishing that for now.
That's it - I'm calling it - extended validation certificates are dead. Sure, you can still buy them (and there are companies out there that would just love to sell them to you!), but their usefulness has now descended from "barely there" to "as good as non-existent". This change has come via a combination of factors including increasing use of mobile devices, removal of the EV visual indicator by browser vendors and as of today, removal from Safari on iOS (it'll also be gone in Mac OS Mojave when it lands next week):
I chose Comodo's website to illustrate this change as I was reminded of the desperation involved in selling EV just last month when they sent around a marketing email with the title "How To Get The Green Address Bar On Your Website". The "alternate truth" of what EV does comes through very early on, starting with this image:
This is indeed what Firefox looks like today, but they entirely neglect to mention anywhere within the marketing email that this is an arbitrary visual indicator chosen at the discretion of the browser vendor. Obviously Apple have already killed it off, but even for many people on Chrome, the Comodo website actually looks very different:
So it turns out that 3 different machines in my workshop today are part of the Chrome experiment to remove the EV indicator from the browser. The usefulness of EV is going, going... pic.twitter.com/5DpTbsfNNO
The email goes on to talk about how EV fights deceptive websites and claims the following:
The verified company name display allows the user to quickly determine the legal entity behind the website, making phishing and deception harder.
In other words, seeing the company name results in higher levels of trust or if we invert that statement, not seeing the company name results in decreased trust, right? The problem is, people simply aren't conditioned to expect to see the company name and there's very simple, effective demonstration of why this is the case:
I did honestly try to get clarity on the source of this work as well, first by tweeting the author of it then, after not receiving a reply, following up with him again and copying @TechSpective for whom he's the editor in chief along with @devopsdotcom (which follows me) who published the survey:
I wish this was made clear in the report itself because Comodo's vested interest is clearly going to introduce bias. It'd be like an oil company commissioning a report that concludes fossil fuels aren't harmful to the environment or a tobacco company stating smoking doesn't lead to adverse health outcomes. If you ever had any doubt about whether DevOps.com actually believes in the "findings", take a look at how much confidence they themselves have in EV certificates and who they chose to go to when acquiring a cert:
This resource is mentioned again throughout the Comodo email but we'll skip that for now. Moving on, they then state that you can "activate the green address bar" simply by purchasing an EV cert:
To activate the green address bar on your website, you just need to purchase and install an Extended Validation (EV) SSL certificate.
Unless you're using the world's most popular browser running on an iOS device:
Same again if you load the site up in Chrome on an Android, the world's most popular operating system:
Even try going to Microsoft Edge on iOS and it's a now predictable result:
These are really, really important images as far as the value proposition of EV goes for two key reasons: Firstly, we're approaching two thirds of all browsing being done on mobile which means that those images above - the ones that don't show EV - are the predominant browsing experience any website owner should be considering. Secondly, as a result, this means that companies cannot tell their customers to expect EV because most of them will never see it. Despite this, Comodo suggests there's value in EV because of the "bigger security display":
The larger security indicator makes it very clear to the user that the website is secure.
You know what makes people think the website is "secure"? When the website says "secure" just as it does next to the URL in the browser right now if you're reading this in Chrome on the desktop! Paradoxically, you only get the "secure" indicator when not using an EV cert and one could quite reasonably argue that this actually creates a greater sense of confidence by literally using the word "secure". And in case you're reading this and thinking "hang on, Chrome doesn't do that anymore", you're completely right:
I wrote the first part of that paragraph before Chrome 69 hit on September 4 and removed both the "Secure" text and the green indicators. That's not just a DV change either, sites with EV now also look rather different:
The point I'm trying to highlight here is both the fact that visual indicators are entirely at the discretion of the client and that they change over time. As such, the title "How To Get The Green Address Bar On Your Website" is now even more incorrect than it was when it was written! In fact, the only piece of the email that even came close to accurately representing EV was the admission that you can't get an EV wildcard cert. But wait! There's a solution and it's easily available just by spending more money, it's called a multi-domain certificate and the default option when looking at Comodo's Enterprise SSL Pro with EV Multi-Domain product will actually save you $5,002.44*:
* Note: You must spend $9,746.75 before the saving is realised
To be clear, this isn't a 4-year certificate either; as the text at the bottom of the image points out, the CA/B Forum guidelines limit certificate validity to 2 years and after that you need to manually go back through the entire verification and issuance process again. But hey, let's not allow that to get in the way of selling 4 year's worth of certs!
And what if you don't renew the cert then? Well, you get a great big pile of this:
Which brings me to the second point: certificate renewal should be automated and that's something that you simply can't do once identity verification is required. DV is easy and indeed automation is a cornerstone of Let's Encrypt which is a really important attribute of it. I recently spent some time with the development team in a major European bank and they were seriously considering ditching EV for precisely this reason. Actually, it was more than that reason alone, it was also the risk presented if they needed to quickly get themselves a new cert (i.e. due to key compromise) as the hurdles you have jump over are so much higher for EV than they are DV. Plus, long-lived certs actually create other risks due to the fact that revocation is broken so iterating quickly (for example, Let's Encrypt certs last for 3 months) is a virtue. Certs lasting for 2 years is not a virtue, unless you're coming from the perspective of being able to cash in on them...
(Paradoxically, the LinkedIn story I linked to above is on TheSSLStore.com which is a certificate reseller. You can probably see where this is going, but rather than suggesting that automation is a key part of the solution to cert renewal, they instead suggest solutions "that scale to Enterprise level" from CAs such as Comodo who, of course, are pushing EV. No mention of Let's Encrypt, but then this is also the company that's been vocally critical of them for issuing certs to phishing sites (that do correctly validate domain ownership) whilst neglecting to mention that Comodo was issuing just as many at that time!)
A lack of wildcard support is one of the big technical reasons EV is avoided (the other reasons are mostly just common-sense ones), and loading up subject alternate names is a barely sufficient alternative. For example, we use a wildcard cert for Report URI so that you can send reports to https://[my company name].report-uri.com and we've got hundreds of those. Comodo will happily support that scale too:
Other than the fact that Scott Helme and I aren't really in a position to shell out $808k, this is also a far cry from what a genuine wildcard cert does as you need to specify all host names at the time of issuance as opposed to being able to dynamically serve them up.
The final point of note on the marketing email is the promise of a warranty:
That actually links straight back to the page with the super pricey multi-domain EV certs and doesn't even attempt explain what the warranty is, which is a bit odd. But it's also consistent because nobody actually knows what the warranty is and if anyone has ever claimed it. Seriously - that's not intended to be a flippant statement, Scott and I genuinely tried to get to the bottom of that earlier this year and we simply couldn't get straight answers. When we did manage to engage in dialogue, I was accused of being in "nerdville":
I asked a very reasonable question Andreas and it’s an important one because certs are being sold with a warranty and I’m trying to understand what that means. Real customers want to know this - what does it cover and are there documented examples of it being used? Do you know?
This was admittedly a very surprising response from someone that holds a position as the CEO at CertCentre because one would imagine that he, of all people, would want to espouse the virtues of cert warranties (assuming there actually are any, of course). If you're paying a company like CertCentre money for a product with a stated set of features, being a "nerd" by asking how those features work seems perfectly reasonable and not something that should result in ridicule from the bloke running the place. Unfortunately, rather than answering the question, Andreas decided it was easier to take the tried and tested ostrich approach:
It's now 52% which is enormously positive for the web in general. But it was this comment about EV which piqued my curiosity:
Despite seeing strong growth in HTTPS across the top 1 million sites, EV certificates have not seen much of that growth at all.
Let's put it in raw numbers: in Feb there were 366,005 sites redirecting from HTTP to HTTPS and 19,802 of them used EV certs so call it 5.41% of all HTTPS sites using EV. Fast forward to August and there were 489,293 sites redirecting to HTTPS with 25,158 serving up EV certs which equates to 5.14%. In other words, the EV market share declined by about 5%. As a proportion of all sites using certificates, EV is far from growing, it's actually going backwards.
(Incidentally, in case you're looking at the 489k figure above and thinking "that's actually less than half of 1M", Scott's scan failed on about 47k websites so they're excluded from the stats.)
As it turns out, many sites are actually removing EV certs. Last month Scott detailed a number of major sites that used to have EV and they spanned everything from Shutterstock to Target to UPS to Visa to the UK police. Around the same time, I noticed that even Twitter had killed their EV cert:
Hey, anyone else notice that Twitter recently ditched their EV certs? I'd love to know why (I mean other than the fact they're completely useless). @Scott_Helme?
Certainly, as of today, EV is not being served up when I connect from Australia so for whatever reason, Twitter don't see it as important enough to show consistently and will switch in and out of EV as you move across the globe. That also says something significant about the effectiveness of EV: if they're willing to constantly add and remove it depending on where you are, do you think people are behaving differently and no longer trusting the site when they don't see EV? No, of course not, but that's the foundation that the mechanics of EV is built on!
I don't just want to focus on Comodo and CertCentre though because disinformation campaigns go well beyond those 2, for example:
Moving past the choice of historic browsers used in the illustration (just how old is that image?!), the piece that tweet links to makes the following claim:
Web security experts recommend adopting EV SSL Certificate for platforms such as E-commerce, Banking, Social Media, Health Care, Governmental and Insurance platforms.
Now I'm not sure who they're referring to in those first few words, but I do know that with the exception of banking, that statement simply doesn't hold water for the remaining industry sectors. It only takes a few minutes to demonstrate how fundamentally wrong this is so let's do it now:
We're on a boat! This week, Scott Helme is back in town so I'm treating him to a rare sight for the Englishman - sunshine ☀️
This is going to be a brief blog post but it's a necessary one because I can't load the data I'm about to publish into Have I Been Pwned (HIBP) without providing more context than what I can in a single short breach description. Here's the story:
Kayo.moe is a free, public, anonymous hosting service. The operator of the service (Kayo) reached out to me earlier this week and advised they'd noticed a collection of files uploaded to the site which appeared to contain personal data from a breach. Let me be crystal clear about one thing early on:
This is not about a data breach of kayo.me - there's absolutely no indication of any sort of security incident involving a vulnerability of that service.
Concerned that the data may indicate a previously unknown breach, Kayo then sent me over a total of 755 files totaling 1.8GB. The vast majority of the files were in a format similar to this:
This is very typical username:password pair used in credential stuffing attacks. These attacks typically take data from multiple breaches then combine them into a single unified list so that they can be used in account takeover attempts on other services. In May last year, I loaded more than 1 billion records from other incidents very similar to this and the real risk it poses to people is that if they've reused their password in multiple places, each of those accounts is now in jeopardy if the username and password appears in one of these lists.
The data also contained a variety of other files; some with logs, some with partial credit card data and some with Spotify details. This doesn't indicate a Spotify breach, however, as I consistently see pastes implying a breach yet every time I've delved into it, it's always come back to account takeover via password reused. In short, this data is a combination of sources intended to be used for malicious purposes.
When I pulled the email addresses out of the file, I found almost 42M unique values. I took a sample set and found about 89% of them were already in HIBP which meant there was a significant amount of data I've never seen before. (Later, after loading the entire data set, that figure went up to 93%.) There was no single pattern for the breaches they appeared in and the only noteworthy thing that stood out was a high hit rate against numeric email address aliases from Facebook also seen in the (most likely fabricated) Badoo incident. Inverting that number and pro-rata'ing to the entire data set, I'd never seen more than 4M of the addresses. So I loaded the data.
There's always questions after data like this is loaded so let me do a very brief Q&A:
Do the filenames indicate the source? No, each file name is obfuscated, I believe as part of the upload process to kayo.me.
Can I provide the password used? No, I've written about why not and it still poses an unacceptable risk to both individuals in the breach and myself.
Had these passwords been seen before? I found a sample set of the data showed that more than 91% of the passwords were already in Pwned Passwords so if you're worried about yours, check there.
Will you load these into Pwned Passwords? Possibly. My hesitation is that there's a large number of files that aren't all in a consistent format so it's a non-trivial exercise. I'm committing to looking at it, but I can't put a timeframe on it.
Doesn't this make the data useless in HIBP? Time and time again, I've asked if I should load incidents like this under the constraints mentioned above and I always get a resounding "yes". If it's not of use to you, ignore it.
What can I actually do about this? These lists take advantage of password reuse so if you're not reusing passwords, you're all good. If you are, get a password manager (I use 1Password).
In short, this is another one of those awareness incidents. I made a commitment to HIBP subscribers to let them know when I see their data so here we are, even if it's not as immediately actionable as a data breach with a clearly identifiable source is. To be honest, if your personal security practices are up to scratch (password manager plus 2FA), this is a bit of a non-event.
Finally, I want to thank Kayo for the support and I'll ask for their input in the comments below if there's any questions related specifically to the kayo.moe service.
The kayo.moe credential stuffing data is now in HIBP and as with previous similar data sets, it's flagged as unverified.
Here's how it normally plays out: It all begins when a company pops up online and makes some sort of ludicrous statement related to their security posture, often as part of a discussion on a public social media platform such as Twitter. Shortly thereafter, the masses descend on said organisation and express their outrage at the stated position. Where it gets interesting (and this is the whole point of the post), is when another group of folks pop up and accuse the outraged group of doing a bit of this:
Shaming. Or chastising, putting them in their place or taking them down a peg or two. Whatever synonym you choose, the underlying criticism is that the outraged group is wrong for expressing their outrage towards the organisation involved, especially if it's ever construed as being targeted towards whichever individual happens to be the mouthpiece of the organisation at the time. Shame, those opposed to it will say, is not the way. I disagree and I want to explain - and demonstrate - precisely why.
Let's start with a few classic examples of the sort of behaviour I'm talking about in terms of those ludicrous statements:
@passy We'd lose our security certificate if we allowed pasting. It could leave us open to a "brute force" attack. Thanks ^Steve
See the theme? Crazy statements made by representatives of the companies involved. The last one from Betfair is a great example and the entire thread is worth a read. What it boiled down to was the account arguing with a journalist (pro tip: avoid arguing being a dick to those in a position to write publicly about you!) that no, you didn't just need a username and birth date to reset the account password. Eventually, it got to the point where Betfair advised that providing this information to someone else would be a breach of their terms. Now, keeping in mind that the username is your email address and that many among us like cake and presents and other birthday celebratory patterns, it's reasonable to say that this was a ludicrous statement. Further, I propose that this is a perfect case where shaming is not only due, but necessary. So I wrote a blog post..
Shortly after that blog post, three things happened and the first was that it got press. The Register wrote about it.Venture Beat wrote about it. Many other discussions were held in the public forum with all concluding the same thing: this process sucked. Secondly, it got fixed. No longer was a mere email address and birthday sufficient to reset the account, you actually had to demonstrate that you controlled the email address! And finally, something else happened that convinced me of the value of shaming in this fashion:
A couple of months later, I delivered the opening keynote at OWASP's AppSec conference in Amsterdam. After the talk, a bunch of people came up to say g'day and many other nice things. And then, after the crowd died down, a bloke came up and handed me his card - "Betfair Security". Ah shit. But the hesitation quickly passed as he proceeded to thank me for the coverage. You see, they knew this process sucked - any reasonable person with half an idea about security did - but the internal security team alone telling management this was not cool wasn't enough to drive change. Negative media coverage, however, is something management actually listens to. Exactly the same scenario played out at a very similar time when I wrote about how you really don't want bank grade security with one of the financial institutions on that list rapidly fixing their shortcomings after that blog post. A little while later at another conference, the same discussion I'd had in Amsterdam played out: "we knew our SSL config was bad, we just couldn't get the leadership support to fix it until we were publicly shamed".
I wanted to set that context because it helps answer questions such as this one:
Why does it often take being named and shamed before they actually do something about these vulnerabilities. Still nice to see they have actually changed the site now.
What public shaming does is appeals to a different set of priorities; if, for example, I was to privately email NatWest about their lack of HTTPS then I'd likely get back a response along the lines of "we take security seriously" and my feedback would go into a queue somewhere. As it was, the feedback I was providing was clearly falling on deaf ears:
I'm sorry you feel this way. I can certainly pass on your concerns and feed this back to the tech team for you Troy? DC
And now we have another perfect example of precisely the sort of response that needs to be shamed so NatWest earned themselves a blog post. How this changed their priorities was to land the negative press on the desk of an executive somewhere who decided this wasn't a good look. As a result, their view on the security of this page is rather different than it was just 9 months ago:
Now I don't know how much of this change was due to my public shaming of their security posture, maybe they were going to get their act together afterward anyway. Who knows. However, what I do know for sure is that I got this DM from someone not long after that post got media attention (reproduced with their permission):
Hi Troy, I just want to say thanks for your blog post on the Natwest HTTPS issue you found that the BBC picked up on. I head up the SEO team at a Media agency for a different bank and was hitting my head against a wall trying to communicate this exact thing to them after they too had a non secure public site separate from their online banking. The quote the BBC must have asked from them prompted the change to happen overnight, something their WebDev team assured me would cost hundreds of thousands of pounds and at least a year to implement! I was hitting my head against the desk for 6 months before that so a virtual handshake of thanks from my behalf! Thanks!
Let me change gear a little and tackle a common complaint about shaming in this fashion and I'll begin with this tweet:
Ok England, look, this sort of stuff was funny for a while and I appreciate the laughs, but it’s starting to get a bit ridiculous. Can one of you please pop down to @santanderukhelp HQ and straighten this mess out? https://t.co/SlMnmFOnVw
Notwithstanding my civic duty as an Aussie to take the piss out of the English, clearly this was a ridiculous statement for Santander to make. Third party password managers are precisely what we need to address the scourge of account takeover attacks driven by sloppy password management on behalf of individuals. Yet somehow, Santander had deliberately designed their system to block the ability to use them. Their customer service rep then echoed this position which subsequently led to the tweet above. That tweet, then led to this one:
Please, just not another witch hunt on some poor clueless Customer Service rep... :(
Andy is concerned that shaming in this fashion targets the individual behind the social media account (JM) rather than the organisation itself. I saw similar sentiments expressed after T-Mobile in Austria defended storing passwords in plain text with this absolute clanger:
@Korni22 What if this doesn't happen because our security is amazingly good? ^Käthe
In each incident, the respective corporate Twitter accounts got a lot of pretty candid feedback. And they deserved it - here's why:
These accounts are, by design, the public face of the respective organisations. Santander literally has the word "help" in the account name and T-Mobile's account indicates that Käthe is a member of the service team. They are absolutely, positively the coal faces of the organisation and it's perfectly reasonable to expect that feedback about their respective businesses should go to them.
Social media accounts are the public face of an organisation. Their specific remit is to engage with the public who’ll likely have something to say about this policy.
This is not to say that the feedback should be rude or abusive; it shouldn't and at least in the discussions I've been involved in, that's extremely rare to see. But to suggest that one shouldn't engage with the individuals controlling the corporate social media account in this fashion is ludicrous - that's exactly who you should be engaging with!
Business as usual there, just another day on the internet. But watch how Medibank then deals with that tweet:
Hi Troy, We just wanted to let you know that we've checked in with our digital team and they've let us know that they are already in the process of resolving this. We'll be deploying the ability to paste in about two weeks. Thanks again for the feedback! ☺️ Kindly, Sam.
And in case you're wondering, yes, I did give them an e-pat on the back for that because they well and truly deserved it! The point is that shaming, when done right, leads to positive change without needing to be offensive or upsetting to the folks controlling the social accounts.
The final catalyst for finishing this blog post (I've been dropping example into it since Xmas!) was a discussion just last week which, once again, highlighted everything said here. As per usual, it starts with a ridiculous statement on security posture:
Our website is secure and security certificates are up to date. Pages where customers enter data are HTTPS. Non HTTPS pages are safe to use despite messages from some browsers (e.g. Chrome) that say they are not.
Also this is a social media account not a first response security account. Yes they are wrong, but as with T mobile and others- are we using a social media mgr to shame an org? Yes we need better awareness. But shame isn’t the way.
'these guys' = some person working a minimum wage customer service job + raising the issue led to the issue being resolved. Calling them 'not bright' when they have to deal with whatever questions get thrown their way despite no real investment in them is not nice.
And just to be clear, stating that "Non HTTPS pages are safe to use despite messages from some browsers" is not a very bright position to take whether you're on minimum wage or you're the CEO. Income doesn't factor when you make public statements as a company representative. Predictably, just as with all the previous example, positive change followed:
That whole incident actually turned out to be much more serious than they originally thought and once again, the issue was brought to the forefront by shaming. I've seen this play out so many times before that frankly, I've little patience for those decrying shaming in this fashion because it might hurt the feelings of the very people charged with receiving feedback from the public. If a company is going to take a position on security either in the way they choose to build their services or by what their representatives state on the public record, they can damn well be held accountable for it:
I’m *absolutely* fed up of social media managers/comms teams taking control and making erroneous statements. If they have the balls to say something that’s demonstrably false and won’t back down when shown proof, be it on their head.
Whether those rejecting shaming of the likes I've shared above agree with the practice or not, they can't argue with the outcome. I'm sure there'll be those that apply motherhood statements such as "the end doesn't justify the means", but that would imply that the means is detrimental in some way which it simply isn't. Keep it polite, use shaming constructively to leverage social pressure and we're all better off for it.
A few little bits and pieces this week ranging from a new web cam (primarily to do Windows Hello auth), teaching my 8-year-old son HTML, progress with Firefox and HIBP, some really ridiculous comments from Namecheap re SSL (or TLS or HTTPS) and a full set of Pwned Passwords as NTLM hashes. I didn't mention it when I recorded, but there's already a bunch of sample code on how to dump your AD hashes and compare them to the Pwned Passwords list in the comments on that blog post.
I love that a service I use every day has taken something I've built and is doing awesome things with it! GitHub has actually downloaded the entire 517M set of passwords rather than hitting the API like many other users, and that's just fine. In fact, I've had a heap of requests for more downloadable data, namely password hashes in NTLM format.
If you're not familiar with NTLM hashes then this probably won't be of much use to you anyway, but if you are and you're working in a Windows environment and are responsible for Active Directory, this may well be kinda handy. Because NTLM hashes aren't salted (do read the two answers there if you're wondering why), providing them in downloadable form means they can easily be used to compare to hashes within an AD environment just as they are. I asked one of the folks who requested this to put together a little script that actually makes them usable and he's subsequently published that on GitHub. I'm sure other people will create other great things as well and if you do, please share them in the comments below.
Home! I got up early today to a balmy 16-degree winter's day as we approach the last week before spring and felt genuinely thankful to be in this location. I've gotta stay home more...
This week, there's no new blog posts due to travel commitments so it's a bit shorter, but there's still the usual array of goings on. I update how the Mozilla testing with HIBP is going, I'm going to update my Ubiquiti network at home and I get a bit cranky about people installing spyware on other people's phones. I've made my thoughts about that perfectly clear in the past too:
I’m a parent with young kids now coming online. I’m also a guy who sees a lot of data breaches and no-way no-how will I ever resort to installing this sort of product on either of my kids’ devices. https://t.co/m2kU1zxcHj
And yeah, precisely the scenario that was in that ZDNet article from earlier this year has now played out yet again courtesy of SpyFone. Just don't, seriously, talk to your kids about the cybers instead. All that and more in this week's update.