Making ML Training Efficient, Algorithmically!

Training neural networks is expensive and inefficient. The amount of computation required to train the largest models is estimated to be growing >5x every year, yet hardware performance per dollar is growing at only a fraction of that rate.  As a result, the cost of training the latest neural networks has ballooned to millions of dollars, leaving these capabilities accessible to only the best resourced institutions. We believe that this inaccessibility is a core reason why we haven’t seen AI / ML positively impact more areas of our life.

Today, MosaicML comes out of stealth to address this gap. We believe that unfettered growth of computing is not a sustainable path towards a future powered by artificial intelligence. Our mission is to reduce the time, cost, energy, and carbon impact of AI / ML training so that better AI models and applications can be developed.

We tackle this problem at the algorithmic and systems level. MosaicML makes machine learning more efficient through composing a mosaic of methods that together accelerate and improve training.

Compute is expensive, and we provide methods that extract the most training out of every cycle by eliminating inefficiencies in the learning process. We also find the most efficient hardware, system architecture, and cloud on which to perform these computations. We provide this as a set of tools to allow practitioners to make rational trade offs between cost, time, and the quality of the resulting models.

Our amazing team of researchers and machine learning systems experts have thoughtfully crafted a foundation for our product offerings. Today, we are sharing with the community our initial methods and analysis tools in the form of an open source package (Composer) and a visualization tool (Explorer):

• Composer is an open-source library of methods for efficient ML training and ways to compose them together into recipes. At launch, we’ve implemented around 20 different efficiency methods curated from the literature and rigorously benchmarked their performance benefits. Along with several strong baselines, we aim to accelerate the path from research to industry with reproducible code and ease-of-use. See our Methodology blog for more information on the science underpinning Composer. Many more methods, models, and datasets will be coming on-line in the coming months!
• Explorer is an interface that establishes a language to talk about efficiency. Choose the best set of methods for your desired tradeoff between quality and cost or training time. Visualize measured trade-offs of cost, time, and quality across thousands of training runs on standard benchmarks.  Filter by method, cloud, and hardware type to reach your optimal operating point.

MosaicML has raised a total of $37 million from an amazing set of investors: Lux Capital, DCVC, Future Ventures, Playground Global, AME, Correlation, E14, and some very special angels. We would like to thank them for their support and belief in our vision.

This is only the beginning. We will be releasing more tools to help the ML research & engineering community with compute efficiency, costs, and environmental impact.  Please stay tuned!

Naveen, Hanlin, Jonathan, and Mike