Degree
Doctor of Philosophy (PhD)
Department
Computer Science
Document Type
Dissertation
Abstract
There is increasing interest in efficiently and effectively learning representations from limited datasets, both in unsupervised, semi-supervised, lowshot learning paradigms. Towards the goal of achieving lowshot learning, this dissertation will contribute on two aspects, knowledge transfer and distillation with the aim of performing downstream classification tasks accurately. Although generic features achieved good results in many visual tasks, fine-tuning is a time, data, and resource-consuming process but is required for pretrained models to be more effective and provide state-of-the-art performance. We explore knowledge transfer from a pretrained model combined with associative memory while allowing automated feature extraction and storing patterns for unsupervised object recognition, helping eliminate backpropagation in a new domain. Due to the sparsity of features, noise has proven to be a great inhibitor in learning efficient representations and degrades classification performance. To combat this, we explore to progressively transfer knowledge in training a classifier generative adversarial network such that the resulting model can even classify noisy data accurately without any preprocessing. Since data labeling is expensive, we further explore knowledge transfer combined with self-paced learning to efficiently and effectively use limited amounts of labeled data with a bigger corpus of unlabeled data to train models. We explore localized domain shifts to extract better domain-invariant features and align pseudo-labels with real class labels in a self-paced fashion, using a novel iterative matching technique based on majority consistency over high-confidence predictions. Finally, we explore knowledge distillation and apply it for discovering high-fidelity route choice models. Existing route choice models do not consider dynamic contextual conditions and are one size fits all. They can only make predictions at an aggregate level and for a fixed set of contextual factors. We distill contextual knowledge obtained from Immersive Virtual Environment that captures subjective or contextual factors in route choice to facilitate predicting route choice with higher fidelity.
Recommended Citation
Liu, Qun, "Exploring Knowledge Transfer and Distillation in Deep Learning" (2020). LSU Doctoral Dissertations. 5426.
https://repository.lsu.edu/gradschool_dissertations/5426
Committee Chair
Mukhopadhyay, Supratik
DOI
10.31390/gradschool_dissertations.5426