Resource-efficient Deep Learning

Download or read book Resource-efficient Deep Learning written by Dongkuan Xu and published by . This book was released on 2022 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: The phenomenal success of deep learning in the past decade has been mostly driven by the construction of increasingly large deep neural network models. These models usually impose an ideal assumption that there are sufficient resources, including large-scale parameters, sufficient data, and massive computation, for the optimization. However, this assumption usually fails in real-world scenarios. For example, computer memory may be limited as in edge devices, large-scale data are difficult to obtain due to expensive costs and privacy constraints, and computational power is constrained as in most university labs. As a result, these resource discrepancy issues have hindered the democratization of deep learning techniques in many AI applications, and the development of efficient deep learning methods that can adapt to different resource constraints is of great importance. In this dissertation, I will present my Ph.D. research concerned with the aforementioned resource discrepancy issues to free AI from the parameter-data-computation hungry beast in three threads. The first thread focuses on data efficiency in deep learning technologies. This thread extends advances in deep learning to scenarios with small, sensitive, or unlabeled data, accelerating the acceptance and adoption of AI in real-world applications. In particular, I study self-supervised learning to remove the dependency on labels, few-shot learning to free model from a large number of samples, and and attentive learning to take full advantage of heterogeneous information sources. The second thread of my work focuses on advances of parameter efficiency in deep learning technologies, which enable us to democratize powerful deep learning models at scale to bridge computer memory divide and improve the adaptability of models in dynamic environments. I study network sparsity, i.e., the technology to prune networks, and network modularity, i.e., the technology to modularize neural networks into multiple modules, each of which is a function with its own parameters. The third thread focuses on computation efficiency of deep learning models, from inference to training, reducing the energy consumption of models, promoting environmental sustainability, and complementing data efficiency. More specifically, I study task-agnostic model compression, the task of generating efficient compressed models without utilizing the downstream task label information, avoiding the repetitive compression process, which saves much training cost.