At Slack, we’ve long been conservative technologists. In other words, when we invest in leveraging a new category of infrastructure, we do it rigorously. We’ve done this since we debuted machine learning-powered features in 2016, and we’ve developed a robust process and skilled team in the space. Despite that, over the past year we’ve been blown away by the increase in capability of commercially available large language models (LLMs) — and more importantly, the difference they could make for our users’ biggest pain points. Too much to read? Too hard to find stuff? Not anymore — 90% of users wh ..read more

Most of Slack runs on a monolithic service simply called “The Webapp”. It’s big – hundreds of developers create hundreds of changes every week. Deploying at this scale is a unique challenge. When people talk about continuous deployment, they’re often thinking about deploying to systems as soon as changes are ready. They talk about microservices and 2-pizza teams (~8 people). But what does continuous deployment mean when you’re looking at 150 changes on a normal day? That’s a lot of pizzas… Changes per day.   Continuous deployments are preferable to large, one-off deployments. We want our ..read more

We are heavy users of Amazon Compute Compute Cloud (EC2) at Slack — we run approximately 60,000 EC2 instances across 17 AWS regions while operating hundreds of AWS accounts. A multitude of teams own and manage our various instances. The Instance Metadata Service (IMDS) is an on-instance component that can be used to gain an insight to the instance’s current state. Since it first launched over 10 years ago, AWS customers used this service to gather useful information about their instances. At Slack, IMDS is used heavily for instance provisioning, and also used by tools that need to understand t ..read more

Slack users have more power than ever to automate routine tasks and processes, saving themselves time each day. Workflow Builder, a task automation tool built into Slack, has continued to improve since its launch back in 2019. Along with various new steps and triggers, we built a new sidebar section for all available workflow steps. These steps are now accessible to users without having to open a modal. BeforeAfter The enhancement of the Slack Platform, coupled with smart and significant design changes, makes Workflow Builder a new and powerful experience for users automating tasks in Slack. P ..read more

Slack Connect, AKA shared channels, allows communication between different Slack workspaces, via channels shared by participating organizations. Slack Connect has existed for a few years now, and the sheer volume of channels and external connections has increased significantly since the launch. The increased volume introduced scaling problems, but also highlighted that not all external connections are the same, and that our customers have different relationships with their partners. We needed a system that allowed us to customize each connection, while also allowing admins to easily manage the ..read more

On Thursday, 12 Oct. 2022, the EMEA part of the Datastores team — the team responsible for Slack’s database clusters — was having an onsite day in Amsterdam, the Netherlands. We’re sitting together for the first time after new engineers had joined the team, when suddenly a few of us were paged: There was an increase in the number of failed database queries. We stopped what we were doing and staged-in to solve the problem. After investigating the issue with other teams, we discovered that there was a long-running job (async job), and that it was purging a large amount of database records. This ..read more

Introduction Ever wondered what it’s like to intern as a software engineer at Slack? Picture yourself on the famous Ohana floor—the 61st floor of the Salesforce Tower in San Francisco— it is one of many privileges we had as interns. Not only did our experience with Slack’s Data Engineering team let us step onto the tech frontier, but it also gave us the chance to gain invaluable engineering experience and forge relationships across teams. Most importantly, we got to contribute to Slack’s mission through two impactful projects: The Reliable Data Discovery Tool helped users discover relevant dat ..read more

Cron scripts are responsible for critical Slack functionality. They ensure reminders execute on time, email notifications are sent, and databases are cleaned up, among other things. Over the years, both the number of cron scripts and the amount of data these scripts process have increased. While generally these cron scripts executed as expected, over time the reliability of their execution has occasionally faltered, and maintaining and scaling their execution environment became increasingly burdensome. These issues lead us to design and build a better way to execute cron scripts reliably at sc ..read more

Embarking on a journey  Stepping out of SFO with the familiarity of the fogginess of the city, my story at Slack unfolds once again. As a return intern, I found myself prepped for another exciting summer, and this opportunity encompassed a renewed sense of anticipation — a mix between known pathways and new adventures. Returning to an internship can often feel like slipping back into a familiar routine, much like riding an old bike. However, this time around, the gears feel a bit different. It’s not just a return to the familiar, but an opportunity to build upon the lessons learned during ..read more

Summary In recent years, cellular architectures have become increasingly popular for large online services as a way to increase redundancy and limit the blast radius of site failures. In pursuit of these goals, we have migrated the most critical user-facing services at Slack from a monolithic to a cell-based architecture over the last 1.5 years. In this series of blog posts, we’ll discuss our reasons for embarking on this massive migration, illustrate the design of our cellular topology along with the engineering trade-offs we made along the way, and talk about our strategies for successfully ..read more