Yandex develops and open-sources YaFSDP — a tool for faster LLM training and optimized GPU consumption Last week, we open-sourced the YaFSDP method — a new tool designed to dramatically speed up the training of large language models. In this article, we’ll talk about how you can organize LLM training on a cluster and what issues may arise. We’ll also look at alternative training methods like ZeRO and FSDP and explain how YaFSDP differs from them. Problems with Training on Multiple GPUs What are the challenges of distributed LLM training on a cluster with multiple GPUs? To answer ..read more

Just over six months have passed since the last release of the C++ userver framework. During this time, we’ve accomplished a lot: Significantly optimized the framework’s performance, surpassing our main competitors in high-performance framework benchmarks Made configuration much easier Added install, Docker images, Yandex Cloud image, and DEB packages Added new functionality, including server middleware for HTTP and a YDB driver Changed to a new monthly release schedule and streamlined versioning Top 15 in TechEmpower benchmarks Over the past six months, we’ve optimized ..read more

My name is Sergey Kastryulin, and I’m part of the Yandex Research team. My work focuses on computer vision and generative neural network research. In this article, I’ll tell you about the approaches behind YandexART — an image generation technology that creates images and animations from textual descriptions. YandexART powers the Shedevrum app and is available in Yandex Ads and Yandex Market. I’ll explain how we improved the efficiency of this neural network and evaluated the quality of its outputs. This article will interest experts and those looking to test the API in the cloud and incorpora ..read more

Dynamic tables in YTsaurus have a prominent place in the internal Yandex infrastructure, and can store enormous arrays of data that can be read very fast. That is why many Yandex services use YTsaurus when generating a response to external users. All that sounds great, but we can’t rest on our laurels, so we continue to improve and optimize them in every possible way. It is often new features under the hood that make significant improvements. Today, I want to tell you about such improvements that we added as new features in the latest release. Read this post to find out how XOR filt ..read more

Hello there! My name is Evgeny Alaev, and I’m a frontend developer here at Yandex Cloud. I am excited to announce that our Gravity UI design system and its library of components are now available in open source. The Gravity UI component library helps users create modern, stylish interfaces. The library contains the following components: A collection of basic React components A library constructor for building landing pages Detailed guides on how to use components A Figma library ChartKit, a package for data visualization Yagr, a high-performance chart rendering solution based ..read more

Yandex Go, an integrated service within the broader Yandex ecosystem, poses a data management challenge as tough as you will likely encounter in any industry. It’s a single app that brings multiple essential services together. Users can request rides, rent scooters, and order food, groceries, and other items for delivery. From a data management perspective, this translaвtes to 42 million MAU (monthly active users) to handle and to process. So how do we manage it? I’m Maxim Pchelin. I led the Product team for Yandex Go’s Data Management Platform (DMP). I co-authored this article with Vlad ..read more

Today, we published the source code of Yandex DataLens — a service for data analysis and visualization — on GitHub under the public Apache 2.0 license. From now on, anyone can use the open-source version of DataLens in any infrastructure. My name is Gadzhi Gadzhiev. Together with Pavel Dubinin, we are responsible for the development of DataLens here at Yandex Cloud. Today, we’re going to tell you about what DataLens helps users do, the new opportunities that going open source opens up, and the functionalities you can deploy right away. What is DataLens? DataLens is a BI tool that allows y ..read more

Over a year has passed since we made ? userver open source. We have implemented many features at the request of our new users and seen significant growth in our community since then. Furthermore, we’re getting fewer and fewer questions about how to use userver, meaning that our efforts to improve the documentation have paid off. So now we are no longer in beta and are now officially launching our release! What does this mean for you? Why should you consider incorporating userver into your projects, and what technologies are available to ensure the reliable operation of server applic ..read more

Hello, my name is Maxim Babenko, and I head the distributed computing technologies department at Yandex. Today we’re pleased to announce that we have released the YTsaurus platform as open source. YTsaurus is one of the key infrastructure big data systems developed at Yandex and was previously known as YT. After almost a decade of hard work, we want to share YTsaurus with the world. In this article, we’ll take you through the history of YT’s development, explain why YTsaurus is needed, describe its main features, and outline the areas for which it is best suited. The GitHub reposito ..read more

Since our previous post six months ago, two meetings of the international C++ standardization working group have taken place. During first meeting, the committee focused on refining the features of C++23, which include: static operator[] static constexpr in constexpr functions Safe range-based for Interaction of srd::print with other console outputs Monadic interface for std::expected static_assert(false) and other features On a second meeting, the committee worked on developing new features for C++26, including: std::get and std::tuple_size for aggregates #embed Obtain ..read more