A common misconception about Transformers is to believe that they're a sequence-processing architecture. They're not. They're a set-processing architecture. Transformers are 100% order-agnostic (which was the big innovation compared to RNNs, back in late 2016 -- you compute the full matrix of pairwise token interactions instead of processing one token at a time). The way you add order awareness in a Transformer is at the feature level. You literally add to your token embeddings a position embedding / encoding that corresponds to its place in a sequence. The architecture itself just treats the ..read more

The XOXO 2024 Talks I missed attending the last XOXO in person, but I've been catching up on the videos of the talks over the past few days and they have been absolutely worth spending time with. This year was a single day with ten speakers. Andy Baio explains the intended formula: I usually explain that the conference is about, more than anything, the emotional experience of being an artist or creator on the internet, often covering the dark, difficult, painful challenges that they’ve dealt with, or are still struggling with, as a creator. “Big idea” TED-style talks don’t work well, and we a ..read more

The problem with passkeys is that they're essentially a halfway house to a password manager, but tied to a specific platform in ways that aren't obvious to a user at all, and liable to easily leave them unable to access of their accounts. [...] Chrome on Windows stores your passkeys in Windows Hello, so if you sign up for a service on Windows, and you then want to access it on iPhone, you're going to be stuck (unless you're so forward thinking as to add a second passkey, somehow, from the iPhone will on the Windows computer!). The passkey lives on the wrong device, if you're away from the com ..read more

There's a meme floating around at the moment where you ask ChatGPT the following and it appears to offer deep insight into your personality: From all of our interactions what is one thing that you can tell me about myself that I may not know about myself Don't be fooled into thinking there's anything deep going on here. It's effectively acting like a horoscope, hooking into the poorly understood memory feature that OpenAI first announced in February and rolled out fully in September. How "memory" works ChatGPT's memory feature is another example of an LLM being given access to a tool - a cod ..read more

My Jina Reader tool I wanted to feed the Cloudflare Durable Objects SQLite documentation into Claude, but I was on my iPhone so copying and pasting was inconvenient. Jina offer a Reader API which can turn any URL into LLM-friendly Markdown and it turns out it supports CORS, so I got Claude to build me this tool (source code). Paste in a URL to get the Jina Markdown version, along with an all important "Copy to clipboard" button. Tags: projects, markdown, ai-assisted-programming, jina, claude-3-5-sonnet, claude, generative-ai, ai, llms ..read more

Grant Negotiation and Authorization Protocol (GNAP) RFC 9635 was published a few days ago. GNAP is effectively OAuth 3 - it's a newly standardized design for a protocol for delegating authorization so an application can access data on your behalf. The most interesting difference between GNAP and OAuth 2 is that GNAP no longer requires clients to be registered in advance. With OAuth the client_id and client_secret need to be configured for each application, which means applications need to register with their targets - creating a new application on GitHub or Twitter before implementing the auth ..read more

I Was A Teenage Foot Clan Ninja My name is Danny Pennington, I am 48 years old, and between 1988 in 1995 I was a ninja in the Foot Clan. I enjoyed this TMNT parody a lot. Tags: youtube ..read more

Zero-latency SQLite storage in every Durable Object Kenton Varda introduces the next iteration of Cloudflare's Durable Object platform, which recently upgraded from a key/value store to a full relational system based on SQLite. This is a fascinating piece of distributed system design, advocating for a really interesting way to architect a large scale application. The key idea behind Durable Objects is to colocate application logic with the data it operates on. A Durable Object comprises code that executes on the same physical host as the SQLite database that it uses, resulting in blazingly fas ..read more

An LLM TDD loop Super neat demo by David Winterbottom, who wrapped my LLM and files-to-prompt tools in a short Bash script that can be fed a file full of Python unit tests and an empty implementation file and will then iterate on that file in a loop until the tests pass. Via @codeinthehole Tags: llm, ai-assisted-programming, python, generative-ai, pytest, ai, llms ..read more

PostgreSQL 17: SQL/JSON is here! Hubert Lubaczewski dives into the new JSON features added in PostgreSQL 17, released a few weeks ago on the 26th of September. This is the latest in his long series of similar posts about new PostgreSQL features. The features are based on the new SQL:2023 standard from June 2023. If you want to actually read the specification for SQL:2023 it looks like you have to buy a PDF from ISO for 194 Swiss Francs (currently $226). Here's a handy summary by Peter Eisentraut: SQL:2023 is finished: Here is what's new. There's a lot of neat stuff in here. I'm particularly in ..read more