I have a story about paper cuts when using a little Linux box. One of my sites has an older Raspberry Pi installed in a spot that takes some effort to access. A couple of weeks ago, it freaked out and stopped allowing remote logins. My own simple management stuff was still running and was reporting that something was wrong, but it wasn't nearly enough detail to find out exactly what happened. I had to get a console connected to it in order to find out that it was freaking out about its filesystem because something stupid had apparently happened to the SD card. I don't know exactly why it would ..read more

I've gotten a few bits of feedback asking for my thoughts and/or reactions to the whole "xz backdoor" thing that happened over the past couple of days. Most of my thoughts on the matter apply to autoconf and friends, and they aren't great. I don't have to cross paths with those tools too often these days, but there was a point quite a while back when I was constantly building things from source, and a ./configure --with-this --with-that was a given. It was a small joy when the thing let me reuse the old configure invocation so I didn't have to dig up the specifics again. I got that the whole r ..read more

There was this team which was running a pretty complicated data storage, leader election and "discovery" service. They had something like 3200 machines and had something like 300 different clusters/cells/ensembles/...(*) running across them. This service ran something kind of like etcd, only not that. The way it worked was that a bunch of "participant" machines would start an election process, and then they'd decide who was going to lead them for a while. That leader got to handle all of the write traffic and it did all of the usual raft/paxos-ish spooky coordination stuff amongst the particip ..read more

Oh dear, it's popcorn for breakfast yet again. Another outage in a massive set of web sites. It's been about 10 years, so let's talk about the outage that marks the point where I started feeling useful in that job: Friday, August 1, 2014. That's the one where FB went down and people started calling 911 to complain about it, and someone from the LA County sheriff's office got on Twitter to say "knock it off, we know and it's not an emergency". Right, so, it's been well-documented what happened that day, even on the outside world - SRECon talks, a bunch of references in papers, you name it. It w ..read more

It's that time again, when code written in the past four years shows up in our lives and breaks something. Still, while you're enjoying the clown show of game companies telling people to manually set the clocks on their consoles and people not being able to fill up their cars, keep one thing in mind: Only half of the fun of a leap year happens on February 29th. The rest of it happens in ten months, when a bunch more code finds out that it's somehow day 366, and promptly flips out. Thus, instead of preparing to party, those people get to spend the day finding out why their device is being stupi ..read more

Maybe 15 years ago, I heard that a certain cell phone camera would lose the ability to autofocus for about two weeks, then it would go back to working for another two weeks, and so on. It had something to do with the time (<some unit> since the epoch), the bits in use, and a fun little thing called sign extension. I got some of this from a leaflet that was posted around where I worked at the time. It was posted in areas where the public could see it, so I figure it's fair game. Here's a nice little test program to show what I'm talking about: #include <stdio.h> static unsigned lo ..read more

In the vein of my "flash" story from a few years ago, here's one about "smurf". Back around 1997, there was something new going around in the realm of net abuse: "smurfing" a target. This one involved a nice little trick that let you send out a relatively small amount of traffic and let someone else turn it into a much larger amount of traffic, and then that response would be directed onto your target. This required two bits of cooperation from the environment. First, you had to be able to transmit a packet of some sort with the source address set to your target. Yes, this does mean "spoofing ..read more

One common element of the larger places where I've worked is that they tend to have a directory service of some sort that keeps track of who's an employee and who isn't. You can learn some interesting things by periodically dumping that list and then running comparisons against the previous dump. A certain company had this rolled up into an internal service called "epitaphs" where an entry for a person would appear a day or two after they "disappeared from LDAP" - meaning, they left the company. Then other people who still worked there could add comments like "went back to school", "moved to I ..read more

It's time for me to respond to some recent feedback. As usual, this is a mix of topics and the responses are pretty much off the cuff, so strap in and hold on tight. ... At least one person mentioned the 11 hour WPA3 problem on my Raspberry Pis and asked if I was experiencing clock drift. This is kind of funny to me since I've been picky about keeping clocks synced in my personal and professional lives these past few years. So, no, not really. All of those Pis have chrony installed, and it's doing a great job of keeping their clocks disciplined. I was the crazy person who spent $300 of my own ..read more

As my assortment of dumb little home-grown utility programs grew over the years, I found myself needing to know when a given binary was built. Sometimes things exist outside the realm of a packaging system, and so the binary itself needs to convey that metadata from build time. I've seen some places solve for this by having giant placeholder strings baked into their binaries that they then reach in and "stamp" later, turning the "XXXXXXXX" or whatever into "Built by foo@foo.blah.evilcorp on ...". While that approach mostly worked, it was too spooky for me and I decided to stay away from it. My ..read more