April 1, 2020

3183 words 15 mins read

Paper Group ANR 469

Supervised Deep Similarity Matching. Target-Embedding Autoencoders for Supervised Representation Learning. GPU-Accelerated Mobile Multi-view Style Transfer. Shahryar Origami Optimization (SOO): A Novel Approach for Solving Large-scale Expensive Optimization Problems Efficiently. Learning the Ising Model with Generative Neural Networks. Diffusion-ba …

Supervised Deep Similarity Matching


Title	Supervised Deep Similarity Matching
Authors	Shanshan Qin, Nayantara Mudur, Cengiz Pehlevan
Abstract	We propose a novel biologically-plausible solution to the credit assignment problem, being motivated by observations in the ventral visual pathway and trained deep neural networks. In both, representations of objects in the same category become progressively more similar, while objects belonging to different categories becomes less similar. We use this observation to motivate a layer-specific learning goal in a deep network: each layer aims to learn a representational similarity matrix that interpolates between previous and later layers. We formulate this idea using a supervised deep similarity matching cost function and derive from it deep neural networks with feedforward, lateral and feedback connections, and neurons that exhibit biologically-plausible Hebbian and anti-Hebbian plasticity. Supervised deep similarity matching can be interpreted as an energy-based learning algorithm, but with significant differences from others in how a contrastive function is constructed.
Tasks
Published	2020-02-24
URL	https://arxiv.org/abs/2002.10378v2
PDF	https://arxiv.org/pdf/2002.10378v2.pdf
PWC	https://paperswithcode.com/paper/supervised-deep-similarity-matching
Repo
Framework

Target-Embedding Autoencoders for Supervised Representation Learning


Title	Target-Embedding Autoencoders for Supervised Representation Learning
Authors	Daniel Jarrett, Mihaela van der Schaar
Abstract	Autoencoder-based learning has emerged as a staple for disciplining representations in unsupervised and semi-supervised settings. This paper analyzes a framework for improving generalization in a purely supervised setting, where the target space is high-dimensional. We motivate and formalize the general framework of target-embedding autoencoders (TEA) for supervised prediction, learning intermediate latent representations jointly optimized to be both predictable from features as well as predictive of targets—encoding the prior that variations in targets are driven by a compact set of underlying factors. As our theoretical contribution, we provide a guarantee of generalization for linear TEAs by demonstrating uniform stability, interpreting the benefit of the auxiliary reconstruction task as a form of regularization. As our empirical contribution, we extend validation of this approach beyond existing static classification applications to multivariate sequence forecasting, verifying their advantage on both linear and nonlinear recurrent architectures—thereby underscoring the further generality of this framework beyond feedforward instantiations.
Tasks	Representation Learning
Published	2020-01-23
URL	https://arxiv.org/abs/2001.08345v1
PDF	https://arxiv.org/pdf/2001.08345v1.pdf
PWC	https://paperswithcode.com/paper/target-embedding-autoencoders-for-supervised-1
Repo
Framework

GPU-Accelerated Mobile Multi-view Style Transfer


Title	GPU-Accelerated Mobile Multi-view Style Transfer
Authors	Puneet Kohli, Saravana Gunaseelan, Jason Orozco, Yiwen Hua, Edward Li, Nicolas Dahlquist
Abstract	An estimated 60% of smartphones sold in 2018 were equipped with multiple rear cameras, enabling a wide variety of 3D-enabled applications such as 3D Photos. The success of 3D Photo platforms (Facebook 3D Photo, Holopix, etc) depend on a steady influx of user generated content. These platforms must provide simple image manipulation tools to facilitate content creation, akin to traditional photo platforms. Artistic neural style transfer, propelled by recent advancements in GPU technology, is one such tool for enhancing traditional photos. However, naively extrapolating single-view neural style transfer to the multi-view scenario produces visually inconsistent results and is prohibitively slow on mobile devices. We present a GPU-accelerated multi-view style transfer pipeline which enforces style consistency between views with on-demand performance on mobile platforms. Our pipeline is modular and creates high quality depth and parallax effects from a stereoscopic image pair.
Tasks	Image Inpainting, Learning Representation Of Multi-View Data, Lightfield, Novel View Synthesis, Style Transfer
Published	2020-03-02
URL	https://arxiv.org/abs/2003.00706v1
PDF	https://arxiv.org/pdf/2003.00706v1.pdf
PWC	https://paperswithcode.com/paper/gpu-accelerated-mobile-multi-view-style
Repo
Framework

Shahryar Origami Optimization (SOO): A Novel Approach for Solving Large-scale Expensive Optimization Problems Efficiently


Title	Shahryar Origami Optimization (SOO): A Novel Approach for Solving Large-scale Expensive Optimization Problems Efficiently
Authors	Shahryar Rahnamayan, Seyed Jalaleddin Mousavirad, Azam Asilian Bidgoli
Abstract	Many real-world problems are categorized as large-scale problems, and metaheuristic algorithms as an alternative method to solve large-scale problem; they need the evaluation of many candidate solutions to tackle them prior to their convergence, which is not affordable for practical applications since the most of them are computationally expensive. In other words, these problems are not only large-scale but also computationally expensive, that makes them very difficult to solve. There is no efficient surrogate model to support large-scale expensive global optimization (LSEGO) problems. As a result, the algorithms should address LSEGO problems using a limited computational budget to be applicable in real-world applications. In this paper, we propose a simple novel algorithm called Shahryar Origami Optimization (SOO) algorithm to tackle LSEGO problems with a limited computational budget. Our proposed algorithm benefits from two leading steps, namely, finding the region of interest and then shrinkage of the search space by folding it into the half with exponential speed. One of the main advantages of the proposed algorithm is being free of any control parameters, which makes it far from the intricacies of the tuning process. The proposed algorithm is compared with cooperative co-evolution with delta grouping on 20 benchmark functions with dimension 1000. Also, we conducted some experiments on CEC-2017, D=10, 30, 50, and 100 to investigate the behavior of SOO algorithm in lower dimensions. The results show that SOO is beneficial not only in large-scale problems, but also in low-scale optimization problems.
Tasks
Published	2020-03-07
URL	https://arxiv.org/abs/2003.03676v1
PDF	https://arxiv.org/pdf/2003.03676v1.pdf
PWC	https://paperswithcode.com/paper/shahryar-origami-optimization-soo-a-novel
Repo
Framework

Learning the Ising Model with Generative Neural Networks


Title	Learning the Ising Model with Generative Neural Networks
Authors	Francesco D’Angelo, Lucas Böttcher
Abstract	Recent advances in deep learning and neural networks have led to an increased interest in the application of generative models in statistical and condensed matter physics. In particular, restricted Boltzmann machines (RBMs) and variational autoencoders (VAEs) as specific classes of neural networks have been successfully applied in the context of physical feature extraction and representation learning. Despite these successes, however, there is only limited understanding of their representational properties and limitations. To better understand the representational characteristics of both generative neural networks, we study the ability of single RBMs and VAEs to capture physical features of the Ising model at different temperatures. This approach allows us to quantitatively assess learned representations by comparing sample features with corresponding theoretical predictions. Our results suggest that the considered RBMs and convolutional VAEs are able to capture the temperature dependence of magnetization, energy, and spin-spin correlations. The samples generated by RBMs are more evenly distributed across temperature than those of VAEs. We also find that convolutional layers in VAEs are important to model spin correlations.
Tasks	Representation Learning
Published	2020-01-15
URL	https://arxiv.org/abs/2001.05361v1
PDF	https://arxiv.org/pdf/2001.05361v1.pdf
PWC	https://paperswithcode.com/paper/learning-the-ising-model-with-generative
Repo
Framework

Diffusion-based Deep Active Learning


Title	Diffusion-based Deep Active Learning
Authors	Dan Kushnir, Luca Venturi
Abstract	The remarkable performance of deep neural networks depends on the availability of massive labeled data. To alleviate the load of data annotation, active deep learning aims to select a minimal set of training points to be labelled which yields maximal model accuracy. Most existing approaches implement either an `exploration'-type selection criterion, which aims at exploring the joint distribution of data and labels, or a` refinement’-type criterion which aims at localizing the detected decision boundaries. We propose a versatile and efficient criterion that automatically switches from exploration to refinement when the distribution has been sufficiently mapped. Our criterion relies on a process of diffusing the existing label information over a graph constructed from the hidden representation of the data set as provided by the neural network. This graph representation captures the intrinsic geometry of the approximated labeling function. The diffusion-based criterion is shown to be advantageous as it outperforms existing criteria for deep active learning.
Tasks	Active Learning
Published	2020-03-23
URL	https://arxiv.org/abs/2003.10339v1
PDF	https://arxiv.org/pdf/2003.10339v1.pdf
PWC	https://paperswithcode.com/paper/diffusion-based-deep-active-learning
Repo
Framework

Intelligence, physics and information – the tradeoff between accuracy and simplicity in machine learning


Title	Intelligence, physics and information – the tradeoff between accuracy and simplicity in machine learning
Authors	Tailin Wu
Abstract	How can we enable machines to make sense of the world, and become better at learning? To approach this goal, I believe viewing intelligence in terms of many integral aspects, and also a universal two-term tradeoff between task performance and complexity, provides two feasible perspectives. In this thesis, I address several key questions in some aspects of intelligence, and study the phase transitions in the two-term tradeoff, using strategies and tools from physics and information. Firstly, how can we make the learning models more flexible and efficient, so that agents can learn quickly with fewer examples? Inspired by how physicists model the world, we introduce a paradigm and an AI Physicist agent for simultaneously learning many small specialized models (theories) and the domain they are accurate, which can then be simplified, unified and stored, facilitating few-shot learning in a continual way. Secondly, for representation learning, when can we learn a good representation, and how does learning depend on the structure of the dataset? We approach this question by studying phase transitions when tuning the tradeoff hyperparameter. In the information bottleneck, we theoretically show that these phase transitions are predictable and reveal structure in the relationships between the data, the model, the learned representation and the loss landscape. Thirdly, how can agents discover causality from observations? We address part of this question by introducing an algorithm that combines prediction and minimizing information from the input, for exploratory causal discovery from observational time series. Fourthly, to make models more robust to label noise, we introduce Rank Pruning, a robust algorithm for classification with noisy labels. I believe that building on the work of my thesis we will be one step closer to enable more intelligent machines that can make sense of the world.
Tasks	Causal Discovery, Few-Shot Learning, Representation Learning, Time Series
Published	2020-01-11
URL	https://arxiv.org/abs/2001.03780v2
PDF	https://arxiv.org/pdf/2001.03780v2.pdf
PWC	https://paperswithcode.com/paper/intelligence-physics-and-information-the
Repo
Framework

Expressiveness and machine processability of Knowledge Organization Systems (KOS): An analysis of concepts and relations


Title	Expressiveness and machine processability of Knowledge Organization Systems (KOS): An analysis of concepts and relations
Authors	Manolis Peponakis, Anna Mastora, Sarantos Kapidakis, Martin Doerr
Abstract	This study considers the expressiveness (that is the expressive power or expressivity) of different types of Knowledge Organization Systems (KOS) and discusses its potential to be machine-processable in the context of the Semantic Web. For this purpose, the theoretical foundations of KOS are reviewed based on conceptualizations introduced by the Functional Requirements for Subject Authority Data (FRSAD) and the Simple Knowledge Organization System (SKOS); natural language processing techniques are also implemented. Applying a comparative analysis, the dataset comprises a thesaurus (Eurovoc), a subject headings system (LCSH) and a classification scheme (DDC). These are compared with an ontology (CIDOC-CRM) by focusing on how they define and handle concepts and relations. It was observed that LCSH and DDC focus on the formalism of character strings (nomens) rather than on the modelling of semantics; their definition of what constitutes a concept is quite fuzzy, and they comprise a large number of complex concepts. By contrast, thesauri have a coherent definition of what constitutes a concept, and apply a systematic approach to the modelling of relations. Ontologies explicitly define diverse types of relations, and are by their nature machine-processable. The paper concludes that the potential of both the expressiveness and machine processability of each KOS is extensively regulated by its structural rules. It is harder to represent subject headings and classification schemes as semantic networks with nodes and arcs, while thesauri are more suitable for such a representation. In addition, a paradigm shift is revealed which focuses on the modelling of relations between concepts, rather than the concepts themselves.
Tasks
Published	2020-03-11
URL	https://arxiv.org/abs/2003.05258v1
PDF	https://arxiv.org/pdf/2003.05258v1.pdf
PWC	https://paperswithcode.com/paper/expressiveness-and-machine-processability-of
Repo
Framework

Towards a Fast Steady-State Visual Evoked Potentials (SSVEP) Brain-Computer Interface (BCI)


Title	Towards a Fast Steady-State Visual Evoked Potentials (SSVEP) Brain-Computer Interface (BCI)
Authors	Aung Aung Phyo Wai, Yangsong Zhang, Heng Guo, Ying Chi, Lei Zhang, Xian-Sheng Hua, Seong Whan Lee, Cuntai Guan
Abstract	Steady-state visual evoked potentials (SSVEP) brain-computer interface (BCI) provides reliable responses leading to high accuracy and information throughput. But achieving high accuracy typically requires a relatively long time window of one second or more. Various methods were proposed to improve sub-second response accuracy through subject-specific training and calibration. Substantial performance improvements were achieved with tedious calibration and subject-specific training; resulting in the user’s discomfort. So, we propose a training-free method by combining spatial-filtering and temporal alignment (CSTA) to recognize SSVEP responses in sub-second response time. CSTA exploits linear correlation and non-linear similarity between steady-state responses and stimulus templates with complementary fusion to achieve desirable performance improvements. We evaluated the performance of CSTA in terms of accuracy and Information Transfer Rate (ITR) in comparison with both training-based and training-free methods using two SSVEP data-sets. We observed that CSTA achieves the maximum mean accuracy of 97.43$\pm$2.26 % and 85.71$\pm$13.41 % with four-class and forty-class SSVEP data-sets respectively in sub-second response time in offline analysis. CSTA yields significantly higher mean performance (p<0.001) than the training-free method on both data-sets. Compared with training-based methods, CSTA shows 29.33$\pm$19.65 % higher mean accuracy with statistically significant differences in time window less than 0.5 s. In longer time windows, CSTA exhibits either better or comparable performance though not statistically significantly better than training-based methods. We show that the proposed method brings advantages of subject-independent SSVEP classification without requiring training while enabling high target recognition performance in sub-second response time.
Tasks	Calibration
Published	2020-02-04
URL	https://arxiv.org/abs/2002.01171v1
PDF	https://arxiv.org/pdf/2002.01171v1.pdf
PWC	https://paperswithcode.com/paper/towards-a-fast-steady-state-visual-evoked
Repo
Framework

Convergence Guarantees of Policy Optimization Methods for Markovian Jump Linear Systems


Title	Convergence Guarantees of Policy Optimization Methods for Markovian Jump Linear Systems
Authors	Joao Paulo Jansch-Porto, Bin Hu, Geir Dullerud
Abstract	Recently, policy optimization for control purposes has received renewed attention due to the increasing interest in reinforcement learning. In this paper, we investigate the convergence of policy optimization for quadratic control of Markovian jump linear systems (MJLS). First, we study the optimization landscape of direct policy optimization for MJLS, and, in particular, show that despite the non-convexity of the resultant problem the unique stationary point is the global optimal solution. Next, we prove that the Gauss-Newton method and the natural policy gradient method converge to the optimal state feedback controller for MJLS at a linear rate if initialized at a controller which stabilizes the closed-loop dynamics in the mean square sense. We propose a novel Lyapunov argument to fix a key stability issue in the convergence proof. Finally, we present a numerical example to support our theory. Our work brings new insights for understanding the performance of policy learning methods on controlling unknown MJLS.
Tasks
Published	2020-02-10
URL	https://arxiv.org/abs/2002.04090v1
PDF	https://arxiv.org/pdf/2002.04090v1.pdf
PWC	https://paperswithcode.com/paper/convergence-guarantees-of-policy-optimization
Repo
Framework

Non-reversibly updating a uniform [0,1] value for Metropolis accept/reject decisions


Title	Non-reversibly updating a uniform [0,1] value for Metropolis accept/reject decisions
Authors	Radford M. Neal
Abstract	I show how it can be beneficial to express Metropolis accept/reject decisions in terms of comparison with a uniform [0,1] value, u, and to then update u non-reversibly, as part of the Markov chain state, rather than sampling it independently each iteration. This provides a small improvement for random walk Metropolis and Langevin updates in high dimensions. It produces a larger improvement when using Langevin updates with persistent momentum, giving performance comparable to that of Hamiltonian Monte Carlo (HMC) with long trajectories. This is of significance when some variables are updated by other methods, since if HMC is used, these updates can be done only between trajectories, whereas they can be done more often with Langevin updates. I demonstrate that for a problem with some continuous variables, updated by HMC or Langevin updates, and also discrete variables, updated by Gibbs sampling between updates of the continuous variables, Langevin with persistent momentum and non-reversible updates to u samples nearly a factor of two more efficiently than HMC. Benefits are also seen for a Bayesian neural network model in which hyperparameters are updated by Gibbs sampling.
Tasks
Published	2020-01-31
URL	https://arxiv.org/abs/2001.11950v1
PDF	https://arxiv.org/pdf/2001.11950v1.pdf
PWC	https://paperswithcode.com/paper/non-reversibly-updating-a-uniform-01-value
Repo
Framework

Sampling on Graphs: From Theory to Applications


Title	Sampling on Graphs: From Theory to Applications
Authors	Yuichi Tanaka, Yonina C. Eldar, Antonio Ortega, Gene Cheung
Abstract	The study of sampling signals on graphs, with the goal of building an analog of sampling for standard signals in the time and spatial domains, has attracted considerable attention recently. Beyond adding to the growing theory on graph signal processing (GSP), sampling on graphs has various promising applications. In this article, we review current progress on sampling over graphs focusing on theory and potential applications. Most methodologies used in graph signal sampling are designed to parallel those used in sampling for standard signals, however, sampling theory for graph signals significantly differs from that for Shannon–Nyquist and shift invariant signals. This is due in part to the fact that the definitions of several important properties, such as shift invariance and bandlimitedness, are different in GSP systems. Throughout, we discuss similarities and differences between standard and graph sampling and highlight open problems and challenges.
Tasks
Published	2020-03-09
URL	https://arxiv.org/abs/2003.03957v2
PDF	https://arxiv.org/pdf/2003.03957v2.pdf
PWC	https://paperswithcode.com/paper/sampling-on-graphs-from-theory-to
Repo
Framework

Personalized Taste and Cuisine Preference Modeling via Images


Title	Personalized Taste and Cuisine Preference Modeling via Images
Authors	Nitish Nag, Bindu Rajanna, Ramesh Jain
Abstract	With the exponential growth in the usage of social media to share live updates about life, taking pictures has become an unavoidable phenomenon. Individuals unknowingly create a unique knowledge base with these images. The food images, in particular, are of interest as they contain a plethora of information. From the image metadata and using computer vision tools, we can extract distinct insights for each user to build a personal profile. Using the underlying connection between cuisines and their inherent tastes, we attempt to develop such a profile for an individual based solely on the images of his food. Our study provides insights about an individual’s inclination towards particular cuisines. Interpreting these insights can lead to the development of a more precise recommendation system. Such a system would avoid the generic approach in favor of a personalized recommendation system.
Tasks
Published	2020-02-26
URL	https://arxiv.org/abs/2003.08769v1
PDF	https://arxiv.org/pdf/2003.08769v1.pdf
PWC	https://paperswithcode.com/paper/personalized-taste-and-cuisine-preference
Repo
Framework

Accelerating Block Coordinate Descent for Nonnegative Tensor Factorization


Title	Accelerating Block Coordinate Descent for Nonnegative Tensor Factorization
Authors	Andersen Man Shun Ang, Jeremy E. Cohen, Nicolas Gillis, Le Thi Khanh Hien
Abstract	This paper is concerned with improving the empirical convergence speed of block-coordinate descent algorithms for approximate nonnegative tensor factorization (NTF). We propose an extrapolation strategy in-between block updates, referred to as heuristic extrapolation with restarts (HER). HER significantly accelerates the empirical convergence speed of most existing block-coordinate algorithms for dense NTF, in particular for challenging computational scenarios, while requiring a negligible additional computational budget.
Tasks
Published	2020-01-13
URL	https://arxiv.org/abs/2001.04321v1
PDF	https://arxiv.org/pdf/2001.04321v1.pdf
PWC	https://paperswithcode.com/paper/accelerating-block-coordinate-descent-for
Repo
Framework

Towards Minimal Supervision BERT-based Grammar Error Correction


Title	Towards Minimal Supervision BERT-based Grammar Error Correction
Authors	Yiyuan Li, Antonios Anastasopoulos, Alan W Black
Abstract	Current grammatical error correction (GEC) models typically consider the task as sequence generation, which requires large amounts of annotated data and limit the applications in data-limited settings. We try to incorporate contextual information from pre-trained language model to leverage annotation and benefit multilingual scenarios. Results show strong potential of Bidirectional Encoder Representations from Transformers (BERT) in grammatical error correction task.
Tasks	Grammatical Error Correction, Language Modelling
Published	2020-01-10
URL	https://arxiv.org/abs/2001.03521v1
PDF	https://arxiv.org/pdf/2001.03521v1.pdf
PWC	https://paperswithcode.com/paper/towards-minimal-supervision-bert-based
Repo
Framework