It happened again, I wrote a new book! One of the places I hang out online is The Audio Programmer Discord, a nice and friendly community for people who are into creating audio software. New folks show up on a regular basis, wanting to learn how to create audio plug-ins such as synthesizers, and ask for book recommendations. The problem is, until now there wasn’t any book I could recommend as a good place to start from. To learn audio programming, you have to piece the information together yourself from all over the place: college textbooks, blog posts, videos, obscure forums, old mailing list ..read more

Back in 2018 I self-published the Core ML Survival Guide, a 500-page ebook with lots of tips and tricks for getting the most out of Apple’s Core ML framework and mlmodel files. It’s the documentation that Apple didn’t write. I’ve kept this book updated for every new version of Core ML, but I do not plan to write new material covering the features introduced with iOS 15 and macOS 12. The most important changes announced at WWDC 2021 are a new package format, mlpackage, and a new model type called ML Programs that will eventually replace the old neural network models. These new features are wel ..read more

If you thought machine learning on mobile is already a bit of a stretch, how about machine learning on microcontrollers? Take the Arduino Uno. This has an 8-bit CPU running at 16 MHz, no real floating point support — it does not even have a division instruction — and only has 2KB of RAM. Is it even possible to do machine learning on these devices? The answer is yes, but you need to be smart about the algorithm you’re using. A simple neural network might work but a full-blown convnet will be way too slow without specialized hardware. My interest is in smart algorithms such as Bonsai or SEFR, th ..read more

One question I get a lot is: I’m using the Vision framework to run an object detection model on iOS but the predicted bounding boxes are not being drawn in the right place. Help! This is a very common problem and it happens because there are several different coordinate systems that you have to translate between. Which means you need to do math! In this blog post I’ll explain what’s going on. Vision and VNRecognizedObjectObservation Even though most of what I’m about to explain applies even if you’re not using the Vision framework, I’m assuming you’re using Vision to make the predictions an ..read more

Resizing feature maps is a common operation in many neural networks, especially those that perform some kind of image segmentation task. One issue I ran into recently while converting a neural network to Core ML, is that the original PyTorch model gave different results for its bilinear upsampling than Core ML, and I wanted to understand why. When converting models between deep learning frameworks, it’s common to find small differences between the original model and the converted one, as layers aren’t necessarily implemented in exactly the same way in every framework. This is not always a ..read more

In this series of blog posts we’re taking a deep dive into the new on-device model personalization features from Core ML 3. I’ll show how to create a customizable image classifier using k-Nearest Neighbors as well as a deep neural network, right from inside an iOS app. This is the last part of a four-part series: Introduction to on-device training Rock, Paper, Scissors (Lizard? Spock?) k-Nearest Neighbors Training a Neural Network Follow along with the source code on GitHub. In this last installment, we’ll finally bring out the big guns: a deep neural network. Training a neural network Ev ..read more

In this series of blog posts we’re taking a deep dive into the new on-device model personalization features from Core ML 3. I’ll show how to create a customizable image classifier using k-Nearest Neighbors as well as a deep neural network, right from inside an iOS app. This is part 3 of a four-part series: Introduction to on-device training Rock, Paper, Scissors (Lizard? Spock?) k-Nearest Neighbors Training a Neural Network [coming soon] Follow along with the source code on GitHub. In this installment, we’ll finally do some training, using the good old k-NN algorithm! k-Nearest Neighbors ..read more