January 30, 2020

3066 words 15 mins read

Paper Group ANR 313

Static Analysis for Probabilistic Programs. Walling up Backdoors in Intrusion Detection Systems. A Linear Systems Theory of Normalizing Flows. Distinguishing mirror from glass: A ‘big data’ approach to material perception. Iterative Collaborative Filtering for Sparse Noisy Tensor Estimation. Learning Transferable Features for Speech Emotion Recogni …

Static Analysis for Probabilistic Programs


Title	Static Analysis for Probabilistic Programs
Authors	Ryan Bernstein
Abstract	Probabilistic programming is a powerful abstraction for statistical machine learning. Applying static analysis methods to probabilistic programs could serve to optimize the learning process, automatically verify properties of models, and improve the programming interface for users. This field of static analysis for probabilistic programming (SAPP) is young and unorganized, consisting of a constellation of techniques with various goals and limitations. The primary aim of this work is to synthesize the major contributions of the SAPP field within an organizing structure and context. We provide technical background for static analysis and probabilistic programming, suggest a functional taxonomy for probabilistic programming languages, and analyze the applicability of major ideas in the SAPP field. We conclude that, while current static analysis techniques for probabilistic programs have practical limitations, there are a number of future directions with high potential to improve the state of statistical machine learning.
Tasks	Probabilistic Programming
Published	2019-09-10
URL	https://arxiv.org/abs/1909.05076v1
PDF	https://arxiv.org/pdf/1909.05076v1.pdf
PWC	https://paperswithcode.com/paper/static-analysis-for-probabilistic-programs
Repo
Framework

Walling up Backdoors in Intrusion Detection Systems


Title	Walling up Backdoors in Intrusion Detection Systems
Authors	Maximilian Bachl, Alexander Hartl, Joachim Fabini, Tanja Zseby
Abstract	Interest in poisoning attacks and backdoors recently resurfaced for Deep Learning (DL) applications. Several successful defense mechanisms have been recently proposed for Convolutional Neural Networks (CNNs), for example in the context of autonomous driving. We show that visualization approaches can aid in identifying a backdoor independent of the used classifier. Surprisingly, we find that common defense mechanisms fail utterly to remove backdoors in DL for Intrusion Detection Systems (IDSs). Finally, we devise pruning-based approaches to remove backdoors for Decision Trees (DTs) and Random Forests (RFs) and demonstrate their effectiveness for two different network security datasets.
Tasks	Autonomous Driving, Intrusion Detection
Published	2019-09-17
URL	https://arxiv.org/abs/1909.07866v2
PDF	https://arxiv.org/pdf/1909.07866v2.pdf
PWC	https://paperswithcode.com/paper/walling-up-backdoors-in-intrusion-detection
Repo
Framework

A Linear Systems Theory of Normalizing Flows


Title	A Linear Systems Theory of Normalizing Flows
Authors	Reuben Feinman, Nikhil Parthasarathy
Abstract	Normalizing Flows are a promising new class of algorithms for unsupervised learning based on maximum likelihood optimization with change of variables. They offer to learn a factorized component representation for complex nonlinear data and, simultaneously, yield a density function that can evaluate likelihoods and generate samples. Despite these diverse offerings, applications of Normalizing Flows have focused primarily on sampling and likelihoods, with little emphasis placed on feature representation. A lack of theoretical foundation has left many open questions about how to interpret and apply the learned components of the model. We provide a new theoretical perspective of Normalizing Flows using the lens of linear systems theory, showing that optimal flows learn to represent the local covariance at each region of input space. Using this insight, we develop a new algorithm to extract interpretable component representations from the learned model, where components correspond to Cartesian dimensions and are scaled according to their manifold significance. In addition, we highlight a stability concern for the learning algorithm that was previously unaddressed, providing a theoretically-grounded solution to mediate the problem. Experiments with toy manifold learning datasets, as well as the MNIST image dataset, provide convincing support for our theory and tools.
Tasks
Published	2019-07-15
URL	https://arxiv.org/abs/1907.06496v4
PDF	https://arxiv.org/pdf/1907.06496v4.pdf
PWC	https://paperswithcode.com/paper/robust-nonlinear-component-estimation-with
Repo
Framework

Distinguishing mirror from glass: A ‘big data’ approach to material perception


Title	Distinguishing mirror from glass: A ‘big data’ approach to material perception
Authors	Hideki Tamura, Konrad E. Prokott, Roland W. Fleming
Abstract	Visually identifying materials is crucial for many tasks, yet material perception remains poorly understood. Distinguishing mirror from glass is particularly challenging as both materials derive their appearance from their surroundings, yet we rarely experience difficulties telling them apart. Here we took a ‘big data’ approach to uncovering the underlying visual cues and processes, leveraging recent advances in neural network models of vision. We trained thousands of convolutional neural networks on >750,000 simulated mirror and glass objects, and compared their performance with human judgments, as well as alternative classifiers based on ‘hand-engineered’ image features. For randomly chosen images, all classifiers and humans performed with high accuracy, and therefore correlated highly with one another. To tease the models apart, we then painstakingly assembled a diagnostic image set for which humans make highly systematic errors, allowing us to decouple accuracy from human-like performance. A large-scale, systematic search through feedforward neural architectures revealed that relatively shallow networks predicted human judgments better than any other models. However, surprisingly, no network correlated better than 0.6 with humans (below inter-human correlations). Thus, although the model sets new standards for simulating human vision in a challenging material perception task, the results cast doubt on recent claims that such architectures are generally good models of human vision.
Tasks
Published	2019-03-05
URL	http://arxiv.org/abs/1903.01671v1
PDF	http://arxiv.org/pdf/1903.01671v1.pdf
PWC	https://paperswithcode.com/paper/distinguishing-mirror-from-glass-a-big-data
Repo
Framework

Iterative Collaborative Filtering for Sparse Noisy Tensor Estimation


Title	Iterative Collaborative Filtering for Sparse Noisy Tensor Estimation
Authors	Devavrat Shah, Christina Lee Yu
Abstract	Consider the task of tensor estimation, i.e. estimating a low-rank 3-order $n \times n \times n$ tensor from noisy observations of randomly chosen entries in the sparse regime. We introduce a generalization of the collaborative filtering algorithm for sparse tensor estimation and argue that it achieves sample complexity that nearly matches the conjectured computationally efficient lower bound on the sample complexity. Our algorithm uses the matrix obtained from the flattened tensor to compute similarity, and estimates the tensor entries using a nearest neighbor estimator. We prove that the algorithm recovers the tensor with maximum entry-wise error and mean-squared-error (MSE) decaying to $0$ as long as each entry is observed independently with probability $p = \Omega(n^{-3/2 + \kappa})$ for any arbitrarily small $\kappa> 0$. Our analysis sheds insight into the conjectured sample complexity lower bound, showing that it matches the connectivity threshold of the graph used by our algorithm for estimating similarity between coordinates.
Tasks
Published	2019-08-03
URL	https://arxiv.org/abs/1908.01241v2
PDF	https://arxiv.org/pdf/1908.01241v2.pdf
PWC	https://paperswithcode.com/paper/iterative-collaborative-filtering-for-sparse
Repo
Framework

Learning Transferable Features for Speech Emotion Recognition


Title	Learning Transferable Features for Speech Emotion Recognition
Authors	Alison Marczewski, Adriano Veloso, Nívio Ziviani
Abstract	Emotion recognition from speech is one of the key steps towards emotional intelligence in advanced human-machine interaction. Identifying emotions in human speech requires learning features that are robust and discriminative across diverse domains that differ in terms of language, spontaneity of speech, recording conditions, and types of emotions. This corresponds to a learning scenario in which the joint distributions of features and labels may change substantially across domains. In this paper, we propose a deep architecture that jointly exploits a convolutional network for extracting domain-shared features and a long short-term memory network for classifying emotions using domain-specific features. We use transferable features to enable model adaptation from multiple source domains, given the sparseness of speech emotion data and the fact that target domains are short of labeled data. A comprehensive cross-corpora experiment with diverse speech emotion domains reveals that transferable features provide gains ranging from 4.3% to 18.4% in speech emotion recognition. We evaluate several domain adaptation approaches, and we perform an ablation study to understand which source domains add the most to the overall recognition effectiveness for a given target domain.
Tasks	Domain Adaptation, Emotion Recognition, Speech Emotion Recognition
Published	2019-12-23
URL	https://arxiv.org/abs/1912.11547v1
PDF	https://arxiv.org/pdf/1912.11547v1.pdf
PWC	https://paperswithcode.com/paper/learning-transferable-features-for-speech
Repo
Framework

Data Ordering Patterns for Neural Machine Translation: An Empirical Study


Title	Data Ordering Patterns for Neural Machine Translation: An Empirical Study
Authors	Siddhant Garg
Abstract	Recent works show that ordering of the training data affects the model performance for Neural Machine Translation. Several approaches involving dynamic data ordering and data sharding based on curriculum learning have been analysed for the their performance gains and faster convergence. In this work we propose to empirically study several ordering approaches for the training data based on different metrics and evaluate their impact on the model performance. Results from our study show that pre-fixing the ordering of the training data based on perplexity scores from a pre-trained model performs the best and outperforms the default approach of randomly shuffling the training data every epoch.
Tasks	Machine Translation
Published	2019-09-23
URL	https://arxiv.org/abs/1909.10642v1
PDF	https://arxiv.org/pdf/1909.10642v1.pdf
PWC	https://paperswithcode.com/paper/data-ordering-patterns-for-neural-machine
Repo
Framework

Recurrent Neural Network-based Model for Accelerated Trajectory Analysis in AIMD Simulations


Title	Recurrent Neural Network-based Model for Accelerated Trajectory Analysis in AIMD Simulations
Authors	Mohammad Javad Eslamibidgoli, Mehrdad Mokhtari, Michael H. Eikerling
Abstract	The presented work demonstrates the training of recurrent neural networks (RNNs) from distributions of atom coordinates in solid state structures that were obtained using ab initio molecular dynamics (AIMD) simulations. AIMD simulations on solid state structures are treated as a multi-variate time-series problem. By referring interactions between atoms over the simulation time to temporary correlations among them, RNNs find patterns in the multi-variate time-dependent data, which enable forecasting trajectory paths and potential energy profiles. Two types of RNNs, namely gated recurrent unit and long short-term memory networks, are considered. The model is described and compared against a baseline AIMD simulation on an iridium oxide slab. Findings demonstrate that both networks can potentially be harnessed for accelerated statistical sampling in computational materials research.
Tasks	Time Series
Published	2019-09-23
URL	https://arxiv.org/abs/1909.10124v2
PDF	https://arxiv.org/pdf/1909.10124v2.pdf
PWC	https://paperswithcode.com/paper/190910124
Repo
Framework

An Incremental Dimensionality Reduction Method for Visualizing Streaming Multidimensional Data


Title	An Incremental Dimensionality Reduction Method for Visualizing Streaming Multidimensional Data
Authors	Takanori Fujiwara, Jia-Kai Chou, Shilpika, Panpan Xu, Liu Ren, Kwan-Liu Ma
Abstract	Dimensionality reduction (DR) methods are commonly used for analyzing and visualizing multidimensional data. However, when data is a live streaming feed, conventional DR methods cannot be directly used because of their computational complexity and inability to preserve the projected data positions at previous time points. In addition, the problem becomes even more challenging when the dynamic data records have a varying number of dimensions as often found in real-world applications. This paper presents an incremental DR solution. We enhance an existing incremental PCA method in several ways to ensure its usability for visualizing streaming multidimensional data. First, we use geometric transformation and animation methods to help preserve a viewer’s mental map when visualizing the incremental results. Second, to handle data dimension variants, we use an optimization method to estimate the projected data positions, and also convey the resulting uncertainty in the visualization. We demonstrate the effectiveness of our design with two case studies using real-world datasets.
Tasks	Dimensionality Reduction
Published	2019-05-10
URL	https://arxiv.org/abs/1905.04000v3
PDF	https://arxiv.org/pdf/1905.04000v3.pdf
PWC	https://paperswithcode.com/paper/an-incremental-dimensionality-reduction
Repo
Framework

Localizing Catastrophic Forgetting in Neural Networks


Title	Localizing Catastrophic Forgetting in Neural Networks
Authors	Felix Wiewel, Bin Yang
Abstract	Artificial neural networks (ANNs) suffer from catastrophic forgetting when trained on a sequence of tasks. While this phenomenon was studied in the past, there is only very limited recent research on this phenomenon. We propose a method for determining the contribution of individual parameters in an ANN to catastrophic forgetting. The method is used to analyze an ANNs response to three different continual learning scenarios.
Tasks	Continual Learning
Published	2019-06-06
URL	https://arxiv.org/abs/1906.02568v1
PDF	https://arxiv.org/pdf/1906.02568v1.pdf
PWC	https://paperswithcode.com/paper/localizing-catastrophic-forgetting-in-neural
Repo
Framework

Sequential online prediction in the presence of outliers and change points: an instant temporal structure learning approach


Title	Sequential online prediction in the presence of outliers and change points: an instant temporal structure learning approach
Authors	Bin Liu, Yu Qi, Ke-Jia Chen
Abstract	In this paper, we consider sequential online prediction (SOP) for streaming data in the presence of outliers and change points. We propose an INstant TEmporal structure Learning (INTEL) algorithm to address this problem. Our INTEL algorithm is developed based on a full consideration of the duality between online prediction and anomaly detection. We first employ a mixture of weighted Gaussian process models (WGPs) to cover the expected possible temporal structures of the data. Then, based on the rich modeling capacity of this WGP mixture, we develop an efficient technique to instantly learn (capture) the temporal structure of the data that follows a regime shift. This instant learning is achieved only by adjusting one hyper-parameter value of the mixture model. A weighted generalization of the product of experts (POE) model is used for fusing predictions yielded from multiple GP models. An outlier is declared once a real observation seriously deviates from the fused prediction. If a certain number of outliers are consecutively declared, then a change point is declared. Extensive experiments are performed using a diverse of real datasets. Results show that the proposed algorithm is significantly better than benchmark methods for SOP in the presence of outliers and change points.
Tasks	Anomaly Detection
Published	2019-07-15
URL	https://arxiv.org/abs/1907.06377v2
PDF	https://arxiv.org/pdf/1907.06377v2.pdf
PWC	https://paperswithcode.com/paper/sequential-online-prediction-in-the-presence
Repo
Framework

Mixing syntagmatic and paradigmatic information for concept detection


Title	Mixing syntagmatic and paradigmatic information for concept detection
Authors	Louis Chartrand, Mohamed Bouguessa
Abstract	In the last decades, philosophers have begun using empirical data for conceptual analysis, but corpus-based conceptual analysis has so far failed to develop, in part because of the absence of reliable methods to automatically detect concepts in textual data. Previous attempts have shown that topic models can constitute efficient concept detection heuristics, but while they leverage the syntagmatic relations in a corpus, they fail to exploit paradigmatic relations, and thus probably fail to model concepts accurately. In this article, we show that using a topic model that models concepts on a space of word embeddings (Hu and Tsujii, 2016) can lead to significant increases in concept detection performance, as well as enable the target concept to be expressed in more flexible ways using word vectors.
Tasks	Topic Models, Word Embeddings
Published	2019-04-09
URL	https://arxiv.org/abs/1904.04461v2
PDF	https://arxiv.org/pdf/1904.04461v2.pdf
PWC	https://paperswithcode.com/paper/mixing-syntagmatic-and-paradigmatic
Repo
Framework

Rapid Whole-Heart CMR with Single Volume Super-resolution


Title	Rapid Whole-Heart CMR with Single Volume Super-resolution
Authors	Jennifer A. Steeden, Michael Quail, Alexander Gotschy, Andreas Hauptmann, Simon Arridge, Rodney Jones, Vivek Muthurangu
Abstract	Background: Three-dimensional, whole heart, balanced steady state free precession (WH-bSSFP) sequences provide delineation of intra-cardiac and vascular anatomy. However, they have long acquisition times. Here, we propose significant speed ups using a deep learning single volume super resolution reconstruction, to recover high resolution features from rapidly acquired low resolution WH-bSSFP images. Methods: A 3D residual U-Net was trained using synthetic data, created from a library of high-resolution WH-bSSFP images by simulating 0.5 slice resolution and 0.5 phase resolution. The trained network was validated with synthetic test data, as well as prospective low-resolution data. Results: Synthetic low-resolution data had significantly better image quality after super-resolution reconstruction. Qualitative image scores showed super-resolved images had better edge sharpness, fewer residual artefacts and less image distortion than low-resolution images, with similar scores to high-resolution data. Quantitative image scores showed super-resolved images had significantly better edge sharpness than low-resolution or high-resolution images, with significantly better signal-to-noise ratio than high-resolution data. Vessel diameters measurements showed over-estimation in the low-resolution measurements, compared to the high-resolution data. No significant differences and no bias was found in the super-resolution measurements. Conclusion: This paper demonstrates the potential of using a residual U-Net for super-resolution reconstruction of rapidly acquired low-resolution whole heart bSSFP data within a clinical setting. The resulting network can be applied very quickly, making these techniques particularly appealing within busy clinical workflow. Thus, we believe that this technique may help speed up whole heart CMR in clinical practice.
Tasks	Super-Resolution
Published	2019-12-22
URL	https://arxiv.org/abs/1912.10503v1
PDF	https://arxiv.org/pdf/1912.10503v1.pdf
PWC	https://paperswithcode.com/paper/rapid-whole-heart-cmr-with-single-volume
Repo
Framework

On the Capabilities and Limitations of Reasoning for Natural Language Understanding


Title	On the Capabilities and Limitations of Reasoning for Natural Language Understanding
Authors	Daniel Khashabi, Erfan Sadeqi Azer, Tushar Khot, Ashish Sabharwal, Dan Roth
Abstract	Recent systems for natural language understanding are strong at overcoming linguistic variability for lookup style reasoning. Yet, their accuracy drops dramatically as the number of reasoning steps increases. We present the first formal framework to study such empirical observations, addressing the ambiguity, redundancy, incompleteness, and inaccuracy that the use of language introduces when representing a hidden conceptual space. Our formal model uses two interrelated spaces: a conceptual meaning space that is unambiguous and complete but hidden, and a linguistic symbol space that captures a noisy grounding of the meaning space in the symbols or words of a language. We apply this framework to study the connectivity problem in undirected graphs—a core reasoning problem that forms the basis for more complex multi-hop reasoning. We show that it is indeed possible to construct a high-quality algorithm for detecting connectivity in the (latent) meaning graph, based on an observed noisy symbol graph, as long as the noise is below our quantified noise level and only a few hops are needed. On the other hand, we also prove an impossibility result: if a query requires a large number (specifically, logarithmic in the size of the meaning graph) of hops, no reasoning system operating over the symbol graph is likely to recover any useful property of the meaning graph. This highlights a fundamental barrier for a class of reasoning problems and systems, and suggests the need to limit the distance between the two spaces, rather than investing in multi-hop reasoning with “many” hops.
Tasks
Published	2019-01-08
URL	https://arxiv.org/abs/1901.02522v2
PDF	https://arxiv.org/pdf/1901.02522v2.pdf
PWC	https://paperswithcode.com/paper/on-the-capabilities-and-limitations-of
Repo
Framework

NAIS: Neural Architecture and Implementation Search and its Applications in Autonomous Driving


Title	NAIS: Neural Architecture and Implementation Search and its Applications in Autonomous Driving
Authors	Cong Hao, Yao Chen, Xinheng Liu, Atif Sarwari, Daryl Sew, Ashutosh Dhar, Bryan Wu, Dongdong Fu, Jinjun Xiong, Wen-mei Hwu, Junli Gu, Deming Chen
Abstract	The rapidly growing demands for powerful AI algorithms in many application domains have motivated massive investment in both high-quality deep neural network (DNN) models and high-efficiency implementations. In this position paper, we argue that a simultaneous DNN/implementation co-design methodology, named Neural Architecture and Implementation Search (NAIS), deserves more research attention to boost the development productivity and efficiency of both DNN models and implementation optimization. We propose a stylized design methodology that can drastically cut down the search cost while preserving the quality of the end solution.As an illustration, we discuss this DNN/implementation methodology in the context of both FPGAs and GPUs. We take autonomous driving as a key use case as it is one of the most demanding areas for high quality AI algorithms and accelerators. We discuss how such a co-design methodology can impact the autonomous driving industry significantly. We identify several research opportunities in this exciting domain.
Tasks	Autonomous Driving
Published	2019-11-18
URL	https://arxiv.org/abs/1911.07446v1
PDF	https://arxiv.org/pdf/1911.07446v1.pdf
PWC	https://paperswithcode.com/paper/nais-neural-architecture-and-implementation
Repo
Framework