Overview SciML, short for Scientific Machine Learning, encompasses work that merges quantitative sciences with machine learning. It has gained significant traction over the past decade, driven by the widespread availability of specialized hardware (such as GPUs and TPUs) and datasets. Additionally, it has been propelled by the overarching influence of the machine learning wave, now ingrained in the zeitgeist of our times. In this context, we’d like to introduce SimulAI, an open-source toolkit under the Apache 2.0 license. SimulAI is designed to be user-friendly, providing a high-level Python i ..read more

Colossal-LLaMA-2: Low Cost and High-quality Domain-specific LLM Solution Using LLaMA and Colossal-AI The most prominent distinction between LLaMA-1 and LLaMA-2 lies in the incorporation of higher-quality corpora, a pivotal factor contributing to significant performance enhancements in LLaMA-2. This, coupled with its commercial availability, extends the potential for creative applications of large models within the open-source community. Nevertheless, it’s widely recognized that the cost of pre-training large models from scratch is exorbitant, often humorously referred to as a domain accessible ..read more

Struggling with quaternions, rotation vectors, right-hand rules and all these stuffs? Try RoMa: an easy-to-to-use, stable and efficient library to deal with rotations and spatial transformations in PyTorch. Overview of the main features of RoMa A bit of context Spatial transformations are essential in physics, engineering, and computer vision. Scientists have been studying 3D geometry for centuries and produced great mathematical tools to deal with such transformations, especially rigid motions and rotations. Yet, implementing these tools properly is not trivial. Indeed, on ..read more

torchdistill — a modular, configuration-driven framework for reproducible deep learning and knowledge distillation experiments Author: Yoshitomo Matsubara This article summarizes key features and concepts of torchdistill (v1.0.0). Refer to the official documentation for its APIs and research projects. Logo torchdistill is a modular, configuration-driven machine learning open source software (ML OSS) for reproducible deep learning and knowledge distillation experiments. The ML OSS is available as a PyPI package (pip install torchdistill) and offers various state-of-the-art knowledge distillatio ..read more

By Yiftach Beer, Omri Bar Overview Activation checkpointing is a technique used for reducing the memory footprint at the cost of more compute. It utilizes the simple observation that we can avoid saving intermediate tensors necessary for backward computation if we just recompute them on demand instead. Currently there are two implementations of activation checkpointing available in PyTorch, reentrant and non-reentrant. The non-reentrant version was implemented later to address some of the limitations of reentrant checkpoint which are detailed in PyTorch’s official docs. You can pass ..read more

Have you ever felt overwhelmed by the complexities of torch.compile? Diving into its workings can feel like black magic, with bytecode and Python internal details that many users fail to understand, hindering them from understanding and debugging torch.compile. I am excited to introduce depyf, a new tool to pull out all the artifacts of torch.compile, and to decompile all the bytecode into source code so that every user understands it. Note: please install the package by pip install depyf before going on to the example below. Example usage import torch+ @torch.compile(backend="eager ..read more

An overview of AI BioTech company Owkin and how their FL framework Substra is used with PyTorch to enable drug discovery. Owkin uses AI to find the right treatment for every patient. Their aim is to integrate the best of human and artificial intelligence to deliver better drugs and diagnostics at scale. Owkin leverages PyTorch in combination with other technologies to build multimodal models that help researchers better understand complex biology through AI and discover new medical treatments. One of the many innovations at Owkin is the use of federated learning to train more robust and ..read more

Machine Learning models thrive on high-quality, fully-annotated data. The traditional supervised learning approach typically requires data on the scale of millions, or even billions, to train large foundational models. However, obtaining such a vast amount of labeled data is often tedious and labor-intensive. As an alternative, semi-supervised learning (SSL) aims to enhance model generalization with only a fraction of labeled data, complemented by a considerable amount of unlabeled data. This blog introduces USB — the Unified Semi-Supervised Learning Framework and Benchmark, covering multi-mod ..read more

Explore TorchOpt, a PyTorch-based library that revolutionizes differentiable optimization with its unified programming abstraction, high-performance distributed execution runtime, and support for various differentiation modes.” This post is authored by Bo Liu, a Ph.D. student at National University of Singapore in Department of Computer Science. He was one of the members of the MetaOPT Team. The team also includes Jie Ren, Xidong Feng, Xuehai Pan, Luo Mai, Yaodong Yang. Introducing TorchOpt The realm of machine learning (ML) has been transformed by differentiable programming, which facili ..read more

by John Pocock, Shan E Ahmed Raza, and the TIAToolbox team. We are excited to announce that TIAToolbox, an end-to-end library for advanced tissue image analytics, has been added to the PyTorch Ecosystem. PyTorch Ecosystem is a collection of high-quality open source projects that are compatible with PyTorch, one of the most popular deep learning frameworks in the world. TIAToolbox supports reading data from many whole slide image (WSI) file formats, such as SVS, NDPI, MIRAX, DICOM, NGFF (OME-Zarr) and OME-TIFF. The toolbox includes handy utilities for handling masking, annotation, patch e ..read more