Over on the Fediverse, I mentioned that we were still running a machine in production that was old enough that its BMC required Java for its KVM-over-IP functionality (and I've given up on working with it, rather than try to maintain a Java environment that could actually work with it). Naturally there's a story here. The specific hardware in question is from our second generation ZFS fileservers, which ran OmniOS and which we used between 2014 and somewhere around 2019. Since we're still using the machine today, that means it's more than ten years old, although it spent some of that time pow ..read more

Over on the Fediverse, I wished for better IO limits than cgroup (v2) has today: I wish Linux cgroups (v2 of course) had an option/interface that limited *filesystem* IO that you could do, read and/or write. They have 'block IO' limits but these are often ineffective for an assortment of reasons, including that you're not doing block IO (hi, NFS) or that the underlying filesystem and storage stack doesn't support them (... many). A VFS level limit would be nominally simple and very useful. Cgroup(s) v2 have an IO controller, that has both priorities (which only work in limited circumstances ..read more

We just got through a downtime where we rebooted basically everything in our fleet, including things like firewalls. Doing such a reboot around this time of year is somewhat of a tradition for us, since we have the university's winter break coming up and in the past some of our machines have had problems that seem to have been related to being up for 'too long'. Generally we don't like to reboot our machines, because it's disruptive to our users. We're in an unusual sysadmin environment where people directly log in to or use many of the individual servers, so when one of them goes through a r ..read more

Recently on the Fediverse, I said something related to Unix's pre-V7 situation with buffered IO: [...] (I think the V1 approach is right for an assembly based minimal OS, while the stdio approach kind of wants malloc() and friends.) The V1 approach, as documented in its putc.3 and getw.3 manual pages, is that the caller to the buffered IO routines supplies the data area used for buffering, and the library functions merely initialize it and later use it. How you get the data area is up to you and your program; you might, for example, simply have a static block of memory in your BSS segment ..read more

In a comment on my entry on how common (desktop) motherboards are supporting more M.2 NVMe slots but fewer PCIe cards, jmassey was curious about what PCIe cards we needed and used. This is a good and interesting question, especially since some number of our 'servers' are actually built using desktop motherboards for various reasons (for example, a certain number of the GPU nodes in our SLURM cluster, and some of our older compute servers, which we put together ourselves using early generation AMD Threadrippers and desktop motherboards for them). Today, we have three dominant patterns of PCIe ..read more

Back at the start of 2020, I wondered if common (x86 desktop) motherboards would ever have very many M.2 NVMe drive slots, where by 'very many' I meant four or so, which even back then was a common number of SATA ports for desktop motherboards to provide. At the time I thought the answer was probably no. As I recently discovered from investigating a related issue, I was wrong, and it's now fairly straightforward to find x86 desktop motherboards that have as many as four M.2 NVMe slots (although not all four may be able to run at x4 PCIe lanes, especially if you have things like a GPU). For ex ..read more

I recently read Julia Evans' Why pipes sometimes get "stuck": buffering. Part of the reason is that almost every Unix program does some amount of buffering for what it prints (or writes) to standard output and standard error. For C programs, this buffering is built into the standard library, specifically into stdio, which includes familiar functions like printf(). Stdio is one of the many things that appeared first in Research Unix V7. This might leave you wondering if this sort of IO was buffered in earlier versions of Research Unix and if it was, how it was done. The very earliest version o ..read more

Suppose, not hypothetically, that you're switching from one mirrored set of M.2 NVMe drives to another mirrored set of M.2 NVMe drives, and so would like to have three or four NVMe drives in your desktop at the same time. Sadly, you already have one of your two NVMe drives on a PCIe card, so you'd like to get a single PCIe card that handles two or more NVMe drives. If you look around today, you'll find two sorts of cards for this; ones that are very expensive, and ones that are relatively inexpensive but require that your system supports a feature that is generally called PCIe bifurcation. NV ..read more

Once upon a time, life was relatively simple in the x86 world. Most x86 compatible PCs theoretically had one or two UARTs, which were called COM1 and COM2 by MS-DOS and Windows, ttyS0 and ttyS1 by Linux, 'ttyu0' and 'ttyu1' by FreeBSD, and so on, based on standard x86 IO port addresses for them. Servers had a physical serial port on the back and wired the connector to COM1 (some servers might have two connectors). Then life became more complicated when servers implemented BMCs (Baseboard management controllers) and the IPMI specification added Serial over LAN, to let you talk to your server t ..read more

One of the classical big reason to want union types in Go is so that one can implement the general pattern of an option type, in order to force people to deal explicitly with null values. Except this is not quite true on both sides. The compiler can enforce null value checks before use already, and union and option types by themselves don't fully protect you against null values. Much like people ignore error returns (and the Go compiler allows this), people can skip over that they can't extract an underlying value from their Result value and return a zero value from their 'get a result' funct ..read more