Colloquium (C)

Natural Language Processing has attracted a lot of interest in recent years and has seen large improvements with the use of contextualized neural language models. However, the progress is limited to specific tasks where large datasets are available and models are often brittle outside of these datasets. Also, current models are usually pretrained on extremely large unlabeled datasets, which limits our understanding of low-resource scenarios and makes their use unfeasible for many in the academia and industry.

With millions of scientific papers coming out every year, researchers are forced to allocate their attention to increasingly narrow areas, creating isolated “research bubbles” that limit knowledge discovery. This fragmentation of science slows down progress and prevents the formation of cross-domain links that catalyze breakthroughs. Toward addressing these large-scale challenges for the future of science, my work develops new paradigms for searching, recommending and discovering scholarly knowledge.

In this talk I will present two deep learning-based algorithms for representation learning.
In the first half of the talk I will present SpectralNet, a  deep learning approach for spectral clustering, which is scalable and allows for straight-forward out of sample extension.
In the second half of the talk I will present a deep learning approach for recovery of a single independent component of interest, given another component as a condition.

Deep neural networks are highly overparameterized models, i.e., they are highly complex with typically many more parameters than the number of training data examples. Such overparameterized models are usually learned to perfectly fit their training data; yet, in sharp contrast to conventional machine learning guidelines, they still generalize extremely well to inputs outside of their training dataset.

An error correcting code should ideally be 1) of large rate, 2) noise tolerant and 3) efficiently decodable. Elementary probabilistic constructions, such as random linear codes, achieve an excellent trade-off between the first two objectives, but unfortunately, decoding them is believed to be algorithmically hard. This motivates us to derandomize these constructions in a way that preserves noise tolerance, while adding structure that can be used for algorithmic purposes.

Deep neural networks (DNNs) provide the leading stimulus-computable model of biological visual object recognition, but their power and flexibility come at a price. Due to their capacity to absorb massive data, distinct DNN models often make very similar predictions when tested on stimuli sampled from their training distribution. To enable continual refinement and improvement of DNNs as scientific hypotheses about biological vision, we must be able to compare alternative models efficiently.

The computational resource growth in natural science motivates the use of machine learning for automated scientific discovery. However, unstructured empirical datasets are often high dimensional, unlabeled, and imbalanced. Therefore, discarding irrelevant (i.e., noisy and information-poor) features is essential for the automated discovery of governing parameters in scientific environments. To address this challenge, I will present Gaussian Stochastic Gates (STG), which rely on a probabilistic relaxation of the L0 norm of the number of selected features.

In this talk I will describe some of my work around Online Learning and Reinforcement Learning.
Online Learning is a classic sub-domain of Machine Learning that has provided endless contributions to fields such as Statistical Learning, Optimization, Decision Making and others.
Unlike Reinforcement Learning which focuses on planning long-term decisions in the face of a non-adversarial environment, Online Learning focuses on making short-term decisions in the face of an adversary - and doing so efficiently.

Consider two samples, each obtained by independent draws from two possibly different distributions over the same finite and large alphabet (features). We would like to test whether the two distributions are identical, or not. We propose a method to perform a two-sample test of this form by taking feature-by-feature p-values based on a binomial allocation model, combining the p-values using Higher Criticism. Performance on real-world data (e.g.

Entity representation learning is an active research field. In the last decade, both the NLP and recommender systems communities introduced many methods for mapping words and items to vectors in a latent space. The vast majority of these methods utilize implicit relations (e.g. co-occurrences of words in text, co-consumption of items by users, etc.) for learning the latent entity vectors.