PyTorch Weekly May 2nd 2023

KevinLing · May 3, 2023, 2:47pm

Welcome to the 1st issue of Pytorch Weekly, a weekly newsletter covering development in Pytorch AI development platform. You can subscribe the newsletter with pytorchweekly@freelists.org or Lingcc/pytorchweekly (github.com) .

News and articles from around the web and events

Hidet is introduced on PyTorch blog as a deep learning compiler for Efficient Model serving. Triton and Hidet Script both allow tensor program developers can easily handle the tile-based programming model.While, compared to Triton, Hidet Script simplifies tensor programming by handling the fine-grained computation and memory resources (e.g., warps, shared memory) manipulation.
TorchBench is introduced by Yueming Hao and other guys from Meta Platforms, Inc. TorchBench is a novel benchmark suite to study the performance of PyTorch software stack and has been used to identify the GPU performance inefficiencies in PyTorch, and it has also been integrated into the PyTorch continuous integration system.
Towards Data Science published an amazing article: Build your own Transformer from scratch using Pytorch writen by Arjun Sarkar. It teaches the reader to build a transformer model step by step in PyTorch.
The latest PyTorch 2.0 Ask the Engineers Q&A Series brought TorchRL by Vincent Moens and Shashank Prasanna from Meta.
Zachary DeVito contribute to the Pytorch Forum about Fast combined C++/Python/TorchScript/Inductor tracebacks
David Stutz proposed a way for Loading and Saving PyTorch Models Without Knowing the Architecture in Advance
Want to check the differences between PyTorch and Jax? check JAX vs. PyTorch: Differences and Similarities [2023]

Run PyTorch on Multiple GPUs thread was actived again since SM2023 tried to fine tune the GPT-2 model on multiple GPUs. Run model on multiple GPUs are not easy to handle, especially for load balance and parallel optimizations. Fresh guys are always recommanded to go through the Multi-GPU examples tutorials. Thanks to ptrblck
According to Would pytorch for cuda 11.6 work when cuda is actually 12.0, PyTorch binary currently shipped directly with CUDA，CUDNN, and cuBLAS, etc, it uses 11.7 and 11.8 by default. And only when build PyTorch from source, will it use the loca installed CUDA toolkit. You are recommanded to use the install method
Result reproducibility is always a headache for ML training. The thread Different training results on different machines has lasted for more than 2 years disscussing about this. PyTorch doc Reproducibility has also mentioned that Pytorch does not guarante completely reproducible results. The thread added a new difference between Windows and Linux which might cause unproducable result since os.listdir or glob.glob on Windows produce an ordered list by default, however Linux output the random file list. it lead to different result.
JOROZCO proposed a way to convert PyTorch model to ONNX format
How to fix “CUDA error: device-side assert triggered” error? introduced CUDA_LAUNCH_BLOCKING=1 to disable asynchronous kernel launhches.