Lecture
Efficient Deep Learning
- Date
- Thursday 22 June 2023
- Time
- Location
-
Snellius
Niels Bohrweg 1
2333 CA Leiden - Room
- 413
Deep learning has dramatically improved the state-of-the-art in object detection, speech recognition, natural language processing, and many other domains. The astonishing success of deep learning is achieved by deep neural networks trained with huge amounts of training examples and massive computing resources. The Efficient Deep Learning (EDL) programme is a large NWO-funded programme that aims to significantly improve the applicability of deep learning, by creating data efficient training methods, and tremendously improving computational efficiency, both for training and inference. This requires a comprehensive approach that combines the domains of machine learning and computer systems.
I will discuss two EDL topics in more detail. First, I will focus on the computer systems side, introducing a novel method that reduces the memory usage when training DNNs on multiple GPUs in parallel. Our approach uses incremental profiling and a recommender algorithm to evenly distribute models over the available resources with respect to per-device peak memory usage. This allows us to train neural networks that are 1.55 times larger on the same hardware.
Next, I will focus on domain adaptation, with use case from radio astronomy. As radio telescopes increase in sensitivity, so do their complexity and data-rates. Automated system health management approaches are becoming increasingly critical to ensure nominal telescope operations. We propose a new machine learning anomaly detection framework for classifying both commonly occurring events in radio telescopes as well as detecting unknown rare events that the system has potentially not yet seen. We do this by combining Self Supervised Learning (SSL) anomaly detection with a supervised classification approach. We evaluate our system on the Low Frequency Array (LOFAR) telescope, demonstrating that our system obtains real-time performance, while being more accurate than other methods.
Rob van Nieuwpoort is the CTO at the Netherlands eScience Center, and professor “efficient computing” at the university of Amsterdam.