New in Composer 0.11: FSDP Support, Streaming v0.1 Release, Simplified Checkpointing and Distributed Experience
We are excited to announce the release of Composer 0.11 (release notes)! This release includes several new features, plus improvements to existing capabilities - kudos to the Composer community for your engagement, feedback, and contributions to this release.
The Composer library helps developers train PyTorch neural networks faster, at lower cost, and to higher accuracy. Composer includes:
- 20+ methods for speeding up training networks for computer vision and language modeling.
- An easy-to-use trainer that integrates best practices for efficient training.
- Functional forms of all speedup methods that are easy to integrate into your existing training loop.
- Strong and reproducible training baselines to get you started as quickly as possible.
With this Composer v0.11.0 release we are releasing beta support for PyTorch FSDP. PyTorch FSDP is a strategy for distributed training, similar to PyTorch DDP, that distributes work using data-parallelism only. On top of this, FSDP uses model, gradient, and optimizer sharding to dramatically reduce device memory requirements, and enables users to easily scale and train large models. We recently used PyTorch FSDP to simply and efficiently train LLMs with up to 70B parameters.
Here’s how easy it is to use PyTorch FSDP with Composer:
Streaming v0.1 Release
Streaming datasets supports following features:
- Unified data encoding and decoding interface for data conversion into streaming format
- Prefetch target number of samples instead of greedily downloading the entire dataset.
- Random access of samples, lazily loading shards to reduce data loading time
- Fetch sample using Python indexing to view the sample locally
- Data compression support to enable quick download times and lower cloud egress fees. Supported formats include: brotli, gzip, snappy, zstd, and bz2
- Dataset hashing support to ensures data integrity through cryptographic and non-cryptographic hashing algorithm. Supported algorithms include: SHA2, SHA3, MD5, xxHash, etc.
To get started, install the Streaming PyPi package:
Simplified Checkpointing Interface
Composer supports saving and loading checkpoints locally and remotely to S3 or WandB, as well as auto-resuming with those checkpoints. With this release we’ve greatly simplified configuration of loading and saving checkpoints in Composer.
To save checkpoints to S3, now all you need to do is:
Likewise, to load checkpoints from S3, all you have to do is:
Improved Distributed Experience
For example, say we’re creating a Python script which requires downloading a dataset. To avoid race conditions where different ranks try to write the dataset to the same place, we need to ensure that only rank 0 downloads the dataset first. Previously, users needed to handle the distributed communication directly. Now, we’ve exposed our distributed API so you can leverage all of our helpful functions and contexts.
Thanks for reading! If you'd like to learn more about Composer and to be part of the community you are welcomed to download Composer and try it out for your training tasks. As you try it out, come be a part of our community by engaging with us on Twitter, joining our Slack channel, or just giving us a star on Github.
What’s a Rich Text element?
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
Static and dynamic content editing
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
How to customize formatting for each rich text
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.