Yani A Ioannou
University of Cambridge, Department of Engineering, Graduate Student
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests: Computer Science, Artificial Intelligence, Computer Vision, Machine Learning, Object Recognition (Computer Vision), and 10 moreNeural Networks, Artificial Neural Networks, Image Classification, Deep Learning, Pooling, Low Rank, Low Rank Representation, Deep Neural Networks, Convolutional Neural Networks, and Convolutional Neural Network
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Research Interests:
Differents modes de realisation de l'invention concernent des systemes et des procedes de detection d'urgence et de reponse automatises. Dans certains modes de realisation, des ensembles de donnees respectifs generes en relation... more
Differents modes de realisation de l'invention concernent des systemes et des procedes de detection d'urgence et de reponse automatises. Dans certains modes de realisation, des ensembles de donnees respectifs generes en relation avec des capteurs situes de maniere distincte sont compares, et un protocole de reponse approprie est selectionne sur la base de cette comparaison et en fonction d'au moins l'un des ensembles de donnees.
Research Interests:
Research Interests:
Research Interests:
Research Interests: Computer Science, Artificial Intelligence, Computer Vision, Machine Learning, Object Recognition (Computer Vision), and 9 moreNeural Networks, Artificial Neural Networks, Image Classification, Deep Learning, Pooling, Low Rank Representation, Deep Neural Networks, Convolutional Neural Networks, and Convolutional Neural Network
Recent advancements in self-supervised learning have reduced the gap between supervised and unsupervised representation learning. However, most self-supervised and deep clustering techniques rely heavily on data augmentation, rendering... more
Recent advancements in self-supervised learning have reduced the gap between supervised and unsupervised representation learning. However, most self-supervised and deep clustering techniques rely heavily on data augmentation, rendering them ineffective for many learning tasks where insufficient domain knowledge exists for performing augmentation. We propose a new self-distillation based algorithm for domain-agnostic clustering. Our method builds upon the existing deep clustering frameworks and requires no separate student model. The proposed method outperforms existing domain agnostic (augmentation-free) algorithms on CIFAR-10. We empirically demonstrate that knowledge distillation can improve unsupervised representation learning by extracting richer ‘dark knowledge’ from the model than using predicted labels alone. Preliminary experiments also suggest that self-distillation improves the convergence of DeepCluster-v2.
Research Interests:
Despite having high accuracy, neural nets have been shown to be susceptible to adversarial examples, where a small perturbation to an input can cause it to become mislabeled. We propose metrics for measuring the robustness of a neural net... more
Despite having high accuracy, neural nets have been shown to be susceptible to adversarial examples, where a small perturbation to an input can cause it to become mislabeled. We propose metrics for measuring the robustness of a neural net and devise a novel algorithm for approximating these metrics based on an encoding of robustness as a linear program. We show how our metrics can be used to evaluate the robustness of deep neural nets with experiments on the MNIST and CIFAR-10 datasets. Our algorithm generates more informative estimates of robustness metrics compared to estimates based on existing algorithms. Furthermore, we show how existing approaches to improving robustness "overfit" to adversarial examples generated using a specific algorithm. Finally, we show that our techniques can be used to additionally improve neural net robustness both according to the metrics that we propose, but also according to previously proposed metrics.
Research Interests:
Deep learning has in recent years come to dominate the previously separate fields of research in machine learning, computer vision, natural language understanding and speech recognition. Despite breakthroughs in training deep networks,... more
Deep learning has in recent years come to dominate the previously separate fields of research in machine learning, computer vision, natural language understanding and speech recognition. Despite breakthroughs in training deep networks, there remains a lack of understanding of both the optimization and structure of deep networks. The approach advocated by many researchers in the field has been to train monolithic networks with excess complexity, and strong regularization — an approach that leaves much to desire in efficiency. Instead we propose that carefully designing networks in consideration of our prior knowledge of the task and learned representation can improve the memory and compute efficiency of state-of-the art networks, and even improve generalization — what we propose to denote as structural priors. We present two such novel structural priors for convolutional neural networks, and evaluate them in state-of-the-art image classification CNN architectures. The first of these ...
Research Interests:
This paper investigates the connections between two state of the art classifiers: decision forests (DFs, including decision jungles) and convolutional neural networks (CNNs). Decision forests are computationally efficient thanks to their... more
This paper investigates the connections between two state of the art classifiers: decision forests (DFs, including decision jungles) and convolutional neural networks (CNNs). Decision forests are computationally efficient thanks to their conditional computation property (computation is confined to only a small region of the tree, the nodes along a single branch). CNNs achieve state of the art accuracy, thanks to their representation learning capabilities. We present a systematic analysis of how to fuse conditional computation with representation learning and achieve a continuum of hybrid models with different ratios of accuracy vs. efficiency. We call this new family of hybrid models conditional networks. Conditional networks can be thought of as: i) decision trees augmented with data transformation operators, or ii) CNNs, with block-diagonal sparse weight matrices, and explicit data routing functions. Experimental validation is performed on the common task of image classification o...
Research Interests:
Research Interests:
Research Interests:
Research Interests:
ABSTRACT Potential Well Space Embedding (PWSE) has been shown to be an effective global method to recognize segmented objects in range data. Here Local PWSE is proposed as an extension of PWSE. LPWSE features are generated by iterating... more
ABSTRACT Potential Well Space Embedding (PWSE) has been shown to be an effective global method to recognize segmented objects in range data. Here Local PWSE is proposed as an extension of PWSE. LPWSE features are generated by iterating ICP to the local minima of a multiscale registration model at each point. The locations of the local minima are then used to generate feature vectors, which can be matched against a preprocessed database of such features to determine correspondences between images and models. The method has been implemented and tested on real data, and has been found to be effective at recognizing sparse segmented (self-)occluded range images. A classification accuracy of 92% is achieved with 3750 points, dropping to 78% at 500 points, on 50 randomly sub-sampled sparse views of 5 objects.
Research Interests:
Research Interests:
Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction of the compute/storage for inference, and also have the potential to enable efficient training. However, naively training unstructured sparse NNs from... more
Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction of the compute/storage for inference, and also have the potential to enable efficient training. However, naively training unstructured sparse NNs from random initialization results in significantly worse generalization, with the notable exception of Lottery Tickets (LTs) and Dynamic Sparse Training (DST). In this work, we attempt to answer: (1) why training unstructured sparse networks from random initialization performs poorly and; (2) what makes LTs and DST the exceptions? We show that sparse NNs have poor gradient flow at initialization and propose a modified initialization for unstructured connectivity. Furthermore, we find that DST methods significantly improve gradient flow during training over traditional sparse training methods. Finally, we show that LTs do not improve gradient flow, rather their success lies in re-learning the pruning solution they are derived from - however, this comes ...
Research Interests:
RÉSUMÉLes personnes âgées hospitalisées présentent un haut risque de chute. Le système HELPER est un système de détection des chutes fixé au plafond qui envoie une alerte à un téléphone intelligent lorsqu’une chute est détectée. Cet... more
RÉSUMÉLes personnes âgées hospitalisées présentent un haut risque de chute. Le système HELPER est un système de détection des chutes fixé au plafond qui envoie une alerte à un téléphone intelligent lorsqu’une chute est détectée. Cet article décrit la performance du système HELPER, qui a été testé dans un projet pilote mené dans un centre de santé mentale gériatrique. La précision du système pour la détection des chutes a été comparée aux données de l’hôpital liées à la documentation des chutes. Au terme du projet pilote, le personnel infirmier a été interviewé afin de documenter comment cette technologie était perçue. Dans cette étude, le système HELPER n’a pas permis de détecter une chute qui a été documentée par le personnel, mais en a détecté 4 autres qui n’avaient pas été documentées. Bien que la sensibilité du système soit élevée (0.80), les fausses alarmes qu’il génère diminuent sa valeur prédictive (0.01). Les entrevues avec le personnel infirmier ont permis de recueillir plu...
Research Interests:
Research Interests:
Despite having high accuracy, neural nets have been shown to be susceptible to adversarial examples, where a small perturbation to an input can cause it to become mislabeled. We propose metrics for measuring the robustness of a neural net... more
Despite having high accuracy, neural nets have been shown to be susceptible to adversarial examples, where a small perturbation to an input can cause it to become mislabeled. We propose metrics for measuring the robustness of a neural net and devise a novel algorithm for approximating these metrics based on an encoding of robustness as a linear program. We show how our metrics can be used to evaluate the robustness of deep neural nets with experiments on the MNIST and CIFAR-10 datasets. Our algorithm generates more informative estimates of robustness metrics compared to estimates based on existing algorithms. Furthermore, we show how existing approaches to improving robustness " overfit " to adversarial examples generated using a specific algorithm. Finally, we show that our techniques can be used to additionally improve neural net robustness both according to the metrics that we propose, but also according to previously proposed metrics.
Research Interests:
We propose a new method for training computationally efficient and compact convolutional neural networks (CNNs) using a novel sparse connection structure that resembles a tree root. Our sparse connection structure facilitates a... more
We propose a new method for training computationally efficient and compact convolutional neural networks (CNNs) using a novel sparse connection structure that resembles a tree root. Our sparse connection structure facilitates a significant reduction in computational cost and number of parameters of state-of-the-art deep CNNs without compromising accuracy. We validate our approach by using it to train more efficient variants of state-of-the-art CNN architectures, evaluated on the CIFAR10 and ILSVRC datasets. Our results show similar or higher accuracy than the baseline architectures with much less compute, as measured by CPU and GPU timings. For example, for ResNet 50, our model has 40% fewer parameters, 45% fewer floating point operations, and is 31% (12%) faster on a CPU (GPU). For the deeper ResNet 200 our model has 25% fewer floating point operations and 44% fewer parameters, while maintaining state-of-the-art accuracy. For GoogLeNet, our model has 7% fewer parameters and is 21% (16%) faster on a CPU (GPU).
Research Interests:
Deep Convolutional Neural Networks (CNNs) have recently evinced immense success for various image recognition tasks [10, 26]. However, a question of paramount importance is somewhat unanswered in deep learning research is the selected CNN... more
Deep Convolutional Neural Networks (CNNs) have recently evinced immense success for various image recognition tasks [10, 26]. However, a question of paramount importance is somewhat unanswered in deep learning research is the selected CNN optimal for the dataset in terms of accuracy and model size? In this paper, we intend to answer this question and introduce a novel strategy that alters the architecture of a given CNN for a specified dataset, to potentially enhance the original accuracy while possibly reducing the model size. We use two operations for architecture refinement, viz. stretching and symmetrical splitting. Stretching increases the number of hidden units (nodes) in a given CNN layer, while a symmetrical split of say K between two layers separates the input and output channels into K equal groups, and connects only the corresponding input-output channel groups. Our procedure starts with a pre-trained CNN for a given dataset, and optimally decides the stretch and split factors across the network to refine the architecture. We empirically demonstrate the necessity of the two operations. We evaluate our approach on two natural scenes attributes datasets, SUN Attributes [15] and CAMIT-NSAD [19], with architectures of GoogleNet and VGG-11, that are quite contrasting in their construction. We justify our choice of datasets, and show that they are interestingly distinct from each other, and together pose a challenge to our architectural refinement algorithm. Our results substantiate the usefulness of the proposed method.
Research Interests:
This paper investigates the connections between two state of the art classifiers: decision forests (DFs, including decision jungles) and convolutional neural networks (CNNs). Decision forests are computationally efficient thanks to their... more
This paper investigates the connections between two state of the art classifiers: decision forests (DFs, including decision jungles) and convolutional neural networks (CNNs). Decision forests are computationally efficient thanks to their conditional computation property (computation is confined to only a small region of the tree, the nodes along a single branch). CNNs achieve state of the art accuracy , thanks to their representation learning capabilities. We present a systematic analysis of how to fuse conditional computation with representation learning and achieve a continuum of hybrid models with different ratios of accuracy vs. efficiency. We call this new family of hybrid models conditional networks. Conditional networks can be thought of as: i) decision trees augmented with data transformation operators, or ii) CNNs, with block-diagonal sparse weight matrices, and explicit data routing functions. Experimental validation is performed on the common task of image classification on both the CIFAR and Imagenet datasets. Compared to state of the art CNNs, our hybrid models yield the same accuracy with a fraction of the compute cost and much smaller number of parameters.
Research Interests:
Potential Well Space Embedding (PWSE) has been shown to be an effective global method to recognize segmented objects in range data. Here Local PWSE is proposed as an extension of PWSE. LPWSE features are generated by iterating ICP to the... more
Potential Well Space Embedding (PWSE) has been
shown to be an effective global method to recognize segmented objects in range data. Here Local PWSE is proposed as an extension of PWSE. LPWSE features are generated by iterating ICP to the local minima of a multiscale registration model at each point. The locations of the local minima are then used to generate feature vectors, which can be matched against a preprocessed database of such features to determine correspondences between images and models. The method has been implemented and tested on real data, and has been found to be effective at recognizing sparse segmented (self-)occluded range images. A classification accuracy of 92% is achieved with 3750 points, dropping to 78% at 500 points, on 50 randomly sub-sampled sparse views of 5 objects.
shown to be an effective global method to recognize segmented objects in range data. Here Local PWSE is proposed as an extension of PWSE. LPWSE features are generated by iterating ICP to the local minima of a multiscale registration model at each point. The locations of the local minima are then used to generate feature vectors, which can be matched against a preprocessed database of such features to determine correspondences between images and models. The method has been implemented and tested on real data, and has been found to be effective at recognizing sparse segmented (self-)occluded range images. A classification accuracy of 92% is achieved with 3750 points, dropping to 78% at 500 points, on 50 randomly sub-sampled sparse views of 5 objects.
Research Interests:
A novel multi-scale operator for unorganized 3D point clouds is introduced. The Difference of Normals (DoN) provides a computationally efficient, multi-scale approach to processing large unorganized 3D point clouds. The application of DoN... more
A novel multi-scale operator for unorganized 3D point clouds is introduced. The Difference of Normals (DoN) provides a computationally efficient, multi-scale approach to processing large unorganized 3D point clouds. The application of DoN in the multi-scale filtering of two different real-world outdoor urban LIDAR scene datasets is quantitatively and qualitatively demonstrated. In both datasets the DoN operator is shown to segment large 3D point clouds into scale-salient clusters, such as cars, people, and lamp posts towards applications in semi-automatic annotation, and as a pre-processing step in automatic object recognition. The application of the operator to segmentation is evaluated on a large public dataset of outdoor LIDAR scenes with ground truth annotations.
Research Interests:
We propose a new method for creating computationally efficient convolutional neural networks (CNNs) by using low-rank representations of convolutional filters. Rather than approximating filters in previously-trained networks with more... more
We propose a new method for creating computationally efficient convolutional neural networks (CNNs) by using low-rank representations of convolutional filters. Rather than approximating filters in previously-trained networks with more efficient versions, we learn a set of small basis filters from scratch; during training, the network learns to combine these basis filters into more complex filters that are discriminative for image classification. To train such networks, a novel weight initialization scheme is used. This allows effective initialization of connection weights in convolutional layers composed of groups of differently-shaped filters. We validate our approach by applying it to several existing CNN architectures and training these networks from scratch using the CIFAR, ILSVRC and MIT Places datasets. Our results show similar or higher accuracy than conventional CNNs with much less compute. Applying our method to an improved version of VGG-11 network using global max-pooling, we achieve comparable validation accuracy using 41% less compute and only 24% of the original VGG-11 model parameters; another variant of our method gives a 1 percentage point increase in accuracy over our improved VGG-11 model, giving a top-5 center-crop validation accuracy of 89.7% while reducing computation by 16% relative to the original VGG-11 model. Applying our method to the GoogLeNet architecture for ILSVRC, we achieved comparable accuracy with 26% less compute and 41% fewer model parameters. Applying our method to a near state-of-the-art network for CIFAR, we achieved comparable accuracy with 46% less compute and 55% fewer parameters.
Research Interests:
We train CNNs with composite layers of oriented low-rank filters, of which the network learns the most effective linear combination In effect our networks learn a basis space for filters, based on simpler low-rank filters We propose an... more
We train CNNs with composite layers of oriented low-rank filters, of which the network learns the most effective linear combination In effect our networks learn a basis space for filters, based on simpler low-rank filters We propose an initialization for composite layers of heterogeneous filters, to train such networks from scratch Our models are faster and use less parameters With a small number of full filters, our models also generalize better Previous Work: Separable (Factorized) Convolution Explicitly approximate low-rank factorization of trained CNN's full-rank filter Use sequential conv. layers with filters of differing orientation [3, 2]. O(d × [h × w × c]) → O(d × [h × m] + m[w × c]) (for each effective filter) However, in most CNNs, d ≥ m c, so this isn't much faster All previous methods approximated a pre-trained model! With our initialization, we can train these networks from scratch VGG-11 GMP Separable 88% top-5 accuracy on ILSVRC Composite Layer-Initialization
Research Interests:
Research Interests:
Research Interests:
A thesis submitted to the School of Computing in conformity with the requirements for the degree of Master of Science (Computing) at Queen's University, Kingston, Ontario, Canada.