SysAdmins have no trouble making big lists of what can go wrong and what we're doing to stave that off a little longer. The tricky problem is pushing large organizations to take a harder look at systemic risks and taking them seriously. I mean, the big companies have to have disaster recovery (DR) plans for compliance reasons; but there are a lot of differences between box-ticking DR plans and comprehensive DR plans. Any company big enough to get past the running out of money is the biggest disaster phase has probably spent some time thinking about what to do if things go wrong. But how do you ..read more

In a Slack I'm on someone asked a series of questions that boil down to: Our company has a Reliability team, but another team is ignoring SLA/SLO obligations. What can SRE do to fix this? I got most of the way through a multi-paragraph answer before noticing my answer was, "This isn't SRE's job, it's management's job." I figured a blog post might help explain this stance better. The genius behind the Site Reliability Engineer concept at Google is they figured out how to make service uptime and reliability matter to business management. The mathematical framework behind SRE is all about quant ..read more

will never happen more than once at a company. I say this knowing that chunks of Germany's civil infrastructure managed to standardize on SuSE desktops, and some may still be using SuSE. Some might view this as proof it can be done, I say that Linux desktops not spreading beyond this example is proof of why it didn't happen. The biggest reason we have the German example is because the decision was top down. Government decision making is different than corporate decision making, which is why we're not going to see the same thing happen, a Linux desktop (actually laptop) mandate from on high ..read more

This is blogging in anger after fighting this over the weekend. Because I'm like that I have a backup cable ISP in case my primary fiber ISP flakes out. I work from home, so the existence of internet is critical to me getting paid, and neither cell phone has good enough service to hotspot reliably. Thus, having two ISPs. It's expensive, but then so would be missing work for a week while I wait for a cable tech to come out to diagnose why their stuff isn't working. The backup ISP hasn't been working well for a while, but the network card pointing to the second cable modem flaked out two weeks a ..read more

20 years ago today I published the first post to this blog. It was a non-sequitur post because I didn't want my first post to be "is this thing on?" or similar. I had to look up what the Exchange worm I mentioned was, and it was probably MyDoom. That was a mass mailing worm, because this was before anti-virus was a routine component of email setups. I started a blog because I needed something to host on this new "web pages from your home directory" feature I was asked to create, and this was the first content on that project! I needed something to look at to prove it worked, and having externa ..read more

I'm seeing more and more folk post, "we got out of IE, we can get out of Chrome," on social media, referencing the nigh-monopoly Chrome has on the browsing experience. Unless you're using Firefox or Safari, you're using Chrome or a Chromium-derived browser. For those of you too young to remember what internet life under Internet Explorer was like, here is a short list of why it was not great: Once Microsoft got the browser-share lock in, it kind of stopped innovating the browser. It conquered the market, so they could pull back investment in it. IE didn't follow standards. But then, Microsoft ..read more

Every time the topic of documentation comes up at work, at multiple workplaces, someone always says a variant of the following: What we really need is markdown in a git repository. We get version control, there is a lot of tooling to make markdown work good in git, it's great And every time I have to grit my teeth and hope I don't cause dental damage. My core complaint is that internal documentation has fundamentally different objectives than open source software documentation repositories, and pretending they're the same problem domain means we'll be re-having the documentation discussion i ..read more

I intended to write a "history of my browser usage" post as part of a longer piece on the Chrome monoculture, but this blog is nearly 20 years old and it turns out I already did a history. https://sysadmin1138.net/mt/blog/2008/09/a-history-of-browsing.shtml where I give a good account of my pre-Mozilla browser usage. The first HTTP browser I used was NCSA Mosaic on a DEC station in college, flirted with Opera a few times, before going Mozilla-land. https://sysadmin1138.net/mt/blog/2009/11/the-firefox-anniversary.shtml Firefox had their fifth anniversary, and I walked through my changes in Moz ..read more

This is a controversial take, but the phrase "it's industry standard" is over-used in technical design discussions of the internal variety. Yes, there are some actual full up standards. Things like RFCs and ISO-standards are actual standards. There are open standards that are widely adopted, like OpenTelemetry and the Cloud Native Computing Foundation suite, but these are not yet industry standards. The phrase "industry standard" implies consensus, agreement, a uniform way of working in a specific area. Have you seen the tech industry? Really seen it? It is utterly vast. The same industry incl ..read more

Some engineers at Google have put forth a proposal called Web-Environment-Integrity that has the open source community up in arms. The leading criticisms of this proposal are "Google wants to make DRM for websites" and "Google wants to ban ad-blockers." These are catchy headlines intended to capture attention, they're also mostly true. For the people who don't want to wade through the discourse, this post is about what WEI does and where it came from. This story begins in the previous decade when Google put forth the "Zero Trust framework" as a way to get rid of the corporate VPN. Zero Trust w ..read more