Deep Learning News #5, Feb 27 2021

Sebastian's books: https://sebastianraschka.com/books/ Links mentioned in this video: Apple's On-Device ML System for Federated Evaluation and Tuning - https://arxiv.org/abs/2102.08503 - https://syncedreview.com/2021/02/19/apple-reveals-design-of-its-on-device-ml-system-for-federated-evaluation-and-tuning/ ========================================== Creating Privacy-Preserving Synthetic Data - https://cloudblogs.microsoft.com/opensource/2021/02/18/create-privacy-preserving-synthetic-data-for-machine-learning-with-smartnoise/ - Whitepaper: https://azure.microsoft.com/en-us/resources/microsoft-smartnoisedifferential-privacy-machine-learning-case-studies/ - Code: https://github.com/opendp/smartnoise-samples ========================================== Humans Are Trying to Take Bias Out of Facial Recognition Programs - https://news.northeastern.edu/2021/02/22/humans-are-trying-to-take-bias-out-of-facial-recognition-programs-its-not-working-yet/ - https://arxiv.org/abs/2102.02320 ========================================== Model Search Platform for Finding Optimal ML Models - https://ai.googleblog.com/2021/02/introducing-model-search-open-source.html - https://pdfs.semanticscholar.org/1bca/d4cdfbc01fbb60a815660d034e561843d67a.pdf ========================================== TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models - https://arxiv.org/abs/2102.07988 ========================================== Gradient Checkpointing - https://github.com/rasbt/deeplearning-models/blob/master/pytorch_ipynb/mechanics/gradient-checkpointing-nin.ipynb - https://github.com/cybertronai/gradient-checkpointing ========================================== Zero Redundancy Optimizer (ZeRO) - https://www.deepspeed.ai/tutorials/zero/ - https://www.deepspeed.ai/tutorials/zero-offload/ - https://github.com/facebookresearch/fairscale/pull/413 ========================================== FairScale - https://github.com/facebookresearch/fairscal