April 3, 2020

3475 words 17 mins read

Paper Group AWR 65

Tensor network approaches for learning non-linear dynamical laws. Error bounds in estimating the out-of-sample prediction error using leave-one-out cross validation in high-dimensions. Learning Generative Models using Denoising Density Estimators. Unsupervised Multi-Class Domain Adaptation: Theory, Algorithms, and Practice. Gossip and Attend: Conte …

Tensor network approaches for learning non-linear dynamical laws


Title	Tensor network approaches for learning non-linear dynamical laws
Authors	A. Goeßmann, M. Götte, I. Roth, R. Sweke, G. Kutyniok, J. Eisert
Abstract	Given observations of a physical system, identifying the underlying non-linear governing equation is a fundamental task, necessary both for gaining understanding and generating deterministic future predictions. Of most practical relevance are automated approaches to theory building that scale efficiently for complex systems with many degrees of freedom. To date, available scalable methods aim at a data-driven interpolation, without exploiting or offering insight into fundamental underlying physical principles, such as locality of interactions. In this work, we show that various physical constraints can be captured via tensor network based parameterizations for the governing equation, which naturally ensures scalability. In addition to providing analytic results motivating the use of such models for realistic physical systems, we demonstrate that efficient rank-adaptive optimization algorithms can be used to learn optimal tensor network models without requiring a~priori knowledge of the exact tensor ranks. As such, we provide a physics-informed approach to recovering structured dynamical laws from data, which adaptively balances the need for expressivity and scalability.
Tasks
Published	2020-02-27
URL	https://arxiv.org/abs/2002.12388v1
PDF	https://arxiv.org/pdf/2002.12388v1.pdf
PWC	https://paperswithcode.com/paper/tensor-network-approaches-for-learning-non
Repo	https://github.com/RoteKekse/systemrecovery
Framework	none

Error bounds in estimating the out-of-sample prediction error using leave-one-out cross validation in high-dimensions


Title	Error bounds in estimating the out-of-sample prediction error using leave-one-out cross validation in high-dimensions
Authors	Kamiar Rahnama Rad, Wenda Zhou, Arian Maleki
Abstract	We study the problem of out-of-sample risk estimation in the high dimensional regime where both the sample size $n$ and number of features $p$ are large, and $n/p$ can be less than one. Extensive empirical evidence confirms the accuracy of leave-one-out cross validation (LO) for out-of-sample risk estimation. Yet, a unifying theoretical evaluation of the accuracy of LO in high-dimensional problems has remained an open problem. This paper aims to fill this gap for penalized regression in the generalized linear family. With minor assumptions about the data generating process, and without any sparsity assumptions on the regression coefficients, our theoretical analysis obtains finite sample upper bounds on the expected squared error of LO in estimating the out-of-sample error. Our bounds show that the error goes to zero as $n,p \rightarrow \infty$, even when the dimension $p$ of the feature vectors is comparable with or greater than the sample size $n$. One technical advantage of the theory is that it can be used to clarify and connect some results from the recent literature on scalable approximate LO.
Tasks
Published	2020-03-03
URL	https://arxiv.org/abs/2003.01770v1
PDF	https://arxiv.org/pdf/2003.01770v1.pdf
PWC	https://paperswithcode.com/paper/error-bounds-in-estimating-the-out-of-sample
Repo	https://github.com/RahnamaRad/LO
Framework	none

Learning Generative Models using Denoising Density Estimators


Title	Learning Generative Models using Denoising Density Estimators
Authors	Siavash A. Bigdeli, Geng Lin, Tiziano Portenier, L. Andrea Dunbar, Matthias Zwicker
Abstract	Learning generative probabilistic models that can estimate the continuous density given a set of samples, and that can sample from that density, is one of the fundamental challenges in unsupervised machine learning. In this paper we introduce a new approach to obtain such models based on what we call denoising density estimators (DDEs). A DDE is a scalar function, parameterized by a neural network, that is efficiently trained to represent a kernel density estimator of the data. Leveraging DDEs, our main contribution is to develop a novel approach to obtain generative models that sample from given densities. We prove that our algorithms to obtain both DDEs and generative models are guaranteed to converge to the correct solutions. Advantages of our approach include that we do not require specific network architectures like in normalizing flows, ordinary differential equation solvers as in continuous normalizing flows, nor do we require adversarial training as in generative adversarial networks (GANs). Finally, we provide experimental results that demonstrate practical applications of our technique.
Tasks	Denoising
Published	2020-01-08
URL	https://arxiv.org/abs/2001.02728v1
PDF	https://arxiv.org/pdf/2001.02728v1.pdf
PWC	https://paperswithcode.com/paper/learning-generative-models-using-denoising-1
Repo	https://github.com/logchan/dde
Framework	pytorch

Unsupervised Multi-Class Domain Adaptation: Theory, Algorithms, and Practice


Title	Unsupervised Multi-Class Domain Adaptation: Theory, Algorithms, and Practice
Authors	Yabin Zhang, Bin Deng, Hui Tang, Lei Zhang, Kui Jia
Abstract	In this paper, we study the formalism of unsupervised multi-class domain adaptation (multi-class UDA), which underlies some recent algorithms whose learning objectives are only motivated empirically. A Multi-Class Scoring Disagreement (MCSD) divergence is presented by aggregating the absolute margin violations in multi-class classification; the proposed MCSD is able to fully characterize the relations between any pair of multi-class scoring hypotheses. By using MCSD as a measure of domain distance, we develop a new domain adaptation bound for multi-class UDA as well as its data-dependent, probably approximately correct bound, which naturally suggest adversarial learning objectives to align conditional feature distributions across the source and target domains. Consequently, an algorithmic framework of Multi-class Domain-adversarial learning Networks (McDalNets) is developed, whose different instantiations via surrogate learning objectives either coincide with or resemble a few of recently popular methods, thus (partially) underscoring their practical effectiveness. Based on our same theory of multi-class UDA, we also introduce a new algorithm of Domain-Symmetric Networks (SymmNets), which is featured by a novel adversarial strategy of domain confusion and discrimination. SymmNets afford simple extensions that work equally well under the problem settings of either closed set, partial, or open set UDA. We conduct careful empirical studies to compare different algorithms of McDalNets and our newly introduced SymmNets. Experiments verify our theoretical analysis and show the efficacy of our proposed SymmNets. We make our implementation codes publicly available.
Tasks	Domain Adaptation
Published	2020-02-20
URL	https://arxiv.org/abs/2002.08681v1
PDF	https://arxiv.org/pdf/2002.08681v1.pdf
PWC	https://paperswithcode.com/paper/unsupervised-multi-class-domain-adaptation
Repo	https://github.com/YBZh/MultiClassDA
Framework	pytorch

Gossip and Attend: Context-Sensitive Graph Representation Learning


Title	Gossip and Attend: Context-Sensitive Graph Representation Learning
Authors	Zekarias T. Kefato, Sarunas Girdzijauskas
Abstract	Graph representation learning (GRL) is a powerful technique for learning low-dimensional vector representation of high-dimensional and often sparse graphs. Most studies explore the structure and metadata associated with the graph using random walks and employ an unsupervised or semi-supervised learning schemes. Learning in these methods is context-free, resulting in only a single representation per node. Recently studies have argued on the adequacy of a single representation and proposed context-sensitive approaches, which are capable of extracting multiple node representations for different contexts. This proved to be highly effective in applications such as link prediction and ranking. However, most of these methods rely on additional textual features that require complex and expensive RNNs or CNNs to capture high-level features or rely on a community detection algorithm to identify multiple contexts of a node. In this study we show that in-order to extract high-quality context-sensitive node representations it is not needed to rely on supplementary node features, nor to employ computationally heavy and complex models. We propose GOAT, a context-sensitive algorithm inspired by gossip communication and a mutual attention mechanism simply over the structure of the graph. We show the efficacy of GOAT using 6 real-world datasets on link prediction and node clustering tasks and compare it against 12 popular and state-of-the-art (SOTA) baselines. GOAT consistently outperforms them and achieves up to 12% and 19% gain over the best performing methods on link prediction and clustering tasks, respectively.
Tasks	Community Detection, Graph Representation Learning, Link Prediction, Representation Learning
Published	2020-03-30
URL	https://arxiv.org/abs/2004.00413v1
PDF	https://arxiv.org/pdf/2004.00413v1.pdf
PWC	https://paperswithcode.com/paper/gossip-and-attend-context-sensitive-graph
Repo	https://github.com/zekarias-tilahun/goat
Framework	none

Towards Lifelong Self-Supervision For Unpaired Image-to-Image Translation


Title	Towards Lifelong Self-Supervision For Unpaired Image-to-Image Translation
Authors	Victor Schmidt, Makesh Narsimhan Sreedhar, Mostafa ElAraby, Irina Rish
Abstract	Unpaired Image-to-Image Translation (I2IT) tasks often suffer from lack of data, a problem which self-supervised learning (SSL) has recently been very popular and successful at tackling. Leveraging auxiliary tasks such as rotation prediction or generative colorization, SSL can produce better and more robust representations in a low data regime. Training such tasks along an I2IT task is however computationally intractable as model size and the number of task grow. On the other hand, learning sequentially could incur catastrophic forgetting of previously learned tasks. To alleviate this, we introduce Lifelong Self-Supervision (LiSS) as a way to pre-train an I2IT model (e.g., CycleGAN) on a set of self-supervised auxiliary tasks. By keeping an exponential moving average of past encoders and distilling the accumulated knowledge, we are able to maintain the network’s validation performance on a number of tasks without any form of replay, parameter isolation or retraining techniques typically used in continual learning. We show that models trained with LiSS perform better on past tasks, while also being more robust than the CycleGAN baseline to color bias and entity entanglement (when two entities are very close).
Tasks	Colorization, Continual Learning, Image-to-Image Translation
Published	2020-03-31
URL	https://arxiv.org/abs/2004.00161v1
PDF	https://arxiv.org/pdf/2004.00161v1.pdf
PWC	https://paperswithcode.com/paper/towards-lifelong-self-supervision-for
Repo	https://github.com/vict0rsch/LiSS
Framework	none

Incremental Unsupervised Domain-Adversarial Training of Neural Networks


Title	Incremental Unsupervised Domain-Adversarial Training of Neural Networks
Authors	Antonio-Javier Gallego, Jorge Calvo-Zaragoza, Robert B. Fisher
Abstract	In the context of supervised statistical learning, it is typically assumed that the training set comes from the same distribution that draws the test samples. When this is not the case, the behavior of the learned model is unpredictable and becomes dependent upon the degree of similarity between the distribution of the training set and the distribution of the test set. One of the research topics that investigates this scenario is referred to as domain adaptation. Deep neural networks brought dramatic advances in pattern recognition and that is why there have been many attempts to provide good domain adaptation algorithms for these models. Here we take a different avenue and approach the problem from an incremental point of view, where the model is adapted to the new domain iteratively. We make use of an existing unsupervised domain-adaptation algorithm to identify the target samples on which there is greater confidence about their true label. The output of the model is analyzed in different ways to determine the candidate samples. The selected set is then added to the source training set by considering the labels provided by the network as ground truth, and the process is repeated until all target samples are labelled. Our results report a clear improvement with respect to the non-incremental case in several datasets, also outperforming other state-of-the-art domain adaptation algorithms.
Tasks	Domain Adaptation, Unsupervised Domain Adaptation
Published	2020-01-13
URL	https://arxiv.org/abs/2001.04129v1
PDF	https://arxiv.org/pdf/2001.04129v1.pdf
PWC	https://paperswithcode.com/paper/incremental-unsupervised-domain-adversarial
Repo	https://github.com/ajgallego/dann
Framework	tf

A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset


Title	A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset
Authors	Jin Liu, Shunping Ji
Abstract	A great deal of research has demonstrated recently that multi-view stereo (MVS) matching can be solved with deep learning methods. However, these efforts were focused on close-range objects and only a very few of the deep learning-based methods were specifically designed for large-scale 3D urban reconstruction due to the lack of multi-view aerial image benchmarks. In this paper, we present a synthetic aerial dataset, called the WHU dataset, we created for MVS tasks, which, to our knowledge, is the first large-scale multi-view aerial dataset. It was generated from a highly accurate 3D digital surface model produced from thousands of real aerial images with precise camera parameters. We also introduce in this paper a novel network, called RED-Net, for wide-range depth inference, which we developed from a recurrent encoder-decoder structure to regularize cost maps across depths and a 2D fully convolutional network as framework. RED-Net’s low memory requirements and high performance make it suitable for large-scale and highly accurate 3D Earth surface reconstruction. Our experiments confirmed that not only did our method exceed the current state-of-the-art MVS methods by more than 50% mean absolute error (MAE) with less memory and computational cost, but its efficiency as well. It outperformed one of the best commercial software programs based on conventional methods, improving their efficiency 16 times over. Moreover, we proved that our RED-Net model pre-trained on the synthetic WHU dataset can be efficiently transferred to very different multi-view aerial image datasets without any fine-tuning. Dataset are available at http://gpcv.whu.edu.cn/data.
Tasks
Published	2020-03-02
URL	https://arxiv.org/abs/2003.00637v3
PDF	https://arxiv.org/pdf/2003.00637v3.pdf
PWC	https://paperswithcode.com/paper/a-novel-recurrent-encoder-decoder-structure
Repo	https://github.com/gpcv-liujin/REDNet
Framework	tf

A Comprehensive Approach to Unsupervised Embedding Learning based on AND Algorithm


Title	A Comprehensive Approach to Unsupervised Embedding Learning based on AND Algorithm
Authors	Sungwon Han, Yizhan Xu, Sungwon Park, Meeyoung Cha, Cheng-Te Li
Abstract	Unsupervised embedding learning aims to extract good representation from data without the need for any manual labels, which has been a critical challenge in many supervised learning tasks. This paper proposes a new unsupervised embedding approach, called Super-AND, which extends the current state-of-the-art model. Super-AND has its unique set of losses that can gather similar samples nearby within a low-density space while keeping invariant features intact against data augmentation. Super-AND outperforms all existing approaches and achieves an accuracy of 89.2% on the image classification task for CIFAR-10. We discuss the practical implications of this method in assisting semi-supervised tasks.
Tasks	Data Augmentation, Image Classification
Published	2020-02-26
URL	https://arxiv.org/abs/2002.12158v1
PDF	https://arxiv.org/pdf/2002.12158v1.pdf
PWC	https://paperswithcode.com/paper/a-comprehensive-approach-to-unsupervised
Repo	https://github.com/super-AND/super-AND
Framework	pytorch

Editable Neural Networks


Title	Editable Neural Networks
Authors	Anton Sinitsin, Vsevolod Plokhotnyuk, Vsevolod Plokhotnyuk, Sergei Popov, Artem Babenko
Abstract	These days deep neural networks are ubiquitously used in a wide range of tasks, from image classification and machine translation to face identification and self-driving cars. In many applications, a single model error can lead to devastating financial, reputational and even life-threatening consequences. Therefore, it is crucially important to correct model mistakes quickly as they appear. In this work, we investigate the problem of neural network editing $-$ how one can efficiently patch a mistake of the model on a particular sample, without influencing the model behavior on other samples. Namely, we propose Editable Training, a model-agnostic training technique that encourages fast editing of the trained model. We empirically demonstrate the effectiveness of this method on large-scale image classification and machine translation tasks.
Tasks	Face Identification, Image Classification, Machine Translation, Self-Driving Cars
Published	2020-04-01
URL	https://arxiv.org/abs/2004.00345v1
PDF	https://arxiv.org/pdf/2004.00345v1.pdf
PWC	https://paperswithcode.com/paper/editable-neural-networks-1
Repo	https://github.com/editable-ICLR2020/editable
Framework	pytorch

Computationally Tractable Riemannian Manifolds for Graph Embeddings


Title	Computationally Tractable Riemannian Manifolds for Graph Embeddings
Authors	Calin Cruceru, Gary Bécigneul, Octavian-Eugen Ganea
Abstract	Representing graphs as sets of node embeddings in certain curved Riemannian manifolds has recently gained momentum in machine learning due to their desirable geometric inductive biases, e.g., hierarchical structures benefit from hyperbolic geometry. However, going beyond embedding spaces of constant sectional curvature, while potentially more representationally powerful, proves to be challenging as one can easily lose the appeal of computationally tractable tools such as geodesic distances or Riemannian gradients. Here, we explore computationally efficient matrix manifolds, showcasing how to learn and optimize graph embeddings in these Riemannian spaces. Empirically, we demonstrate consistent improvements over Euclidean geometry while often outperforming hyperbolic and elliptical embeddings based on various metrics that capture different graph properties. Our results serve as new evidence for the benefits of non-Euclidean embeddings in machine learning pipelines.
Tasks
Published	2020-02-20
URL	https://arxiv.org/abs/2002.08665v1
PDF	https://arxiv.org/pdf/2002.08665v1.pdf
PWC	https://paperswithcode.com/paper/computationally-tractable-riemannian
Repo	https://github.com/dalab/matrix-manifolds
Framework	pytorch

Neural Contours: Learning to Draw Lines from 3D Shapes


Title	Neural Contours: Learning to Draw Lines from 3D Shapes
Authors	Difan Liu, Mohamed Nabail, Aaron Hertzmann, Evangelos Kalogerakis
Abstract	This paper introduces a method for learning to generate line drawings from 3D models. Our architecture incorporates a differentiable module operating on geometric features of the 3D model, and an image-based module operating on view-based shape representations. At test time, geometric and view-based reasoning are combined with the help of a neural module to create a line drawing. The model is trained on a large number of crowdsourced comparisons of line drawings. Experiments demonstrate that our method achieves significant improvements in line drawing over the state-of-the-art when evaluated on standard benchmarks, resulting in drawings that are comparable to those produced by experienced human artists.
Tasks
Published	2020-03-23
URL	https://arxiv.org/abs/2003.10333v2
PDF	https://arxiv.org/pdf/2003.10333v2.pdf
PWC	https://paperswithcode.com/paper/neural-contours-learning-to-draw-lines-from
Repo	https://github.com/DifanLiu/NeuralContours
Framework	pytorch

Virtual to Real adaptation of Pedestrian Detectors for Smart Cities


Title	Virtual to Real adaptation of Pedestrian Detectors for Smart Cities
Authors	Luca Ciampi, Nicola Messina, Fabrizio Falchi, Claudio Gennaro, Giuseppe Amato
Abstract	Pedestrian detection through computer vision is a building block for a multitude of applications in the context of smart cities, such as surveillance of sensitive areas, personal safety, monitoring, and control of pedestrian flow, to mention only a few. Recently, there was an increasing interest in deep learning architectures for performing such a task. One of the critical objectives of these algorithms is to generalize the knowledge gained during the training phase to new scenarios having various characteristics, and a suitably labeled dataset is fundamental to achieve this goal. The main problem is that manually annotating a dataset usually requires a lot of human effort, and it is a time-consuming operation. For this reason, in this work, we introduced ViPeD - Virtual Pedestrian Dataset, a new synthetically generated set of images collected from a realistic 3D video game where the labels can be automatically generated exploiting 2D pedestrian positions extracted from the graphics engine. We used this new synthetic dataset training a state-of-the-art computationally-efficient Convolutional Neural Network (CNN) that is ready to be installed in smart low-power devices, like smart cameras. We addressed the problem of the domain-adaptation from the virtual world to the real one by fine-tuning the CNN using the synthetic data and also exploiting a mixed-batch supervised training approach. Extensive experimentation carried out on different real-world datasets shows very competitive results compared to other methods presented in the literature in which the algorithms are trained using real-world data.
Tasks	Domain Adaptation, Pedestrian Detection
Published	2020-01-09
URL	https://arxiv.org/abs/2001.03032v1
PDF	https://arxiv.org/pdf/2001.03032v1.pdf
PWC	https://paperswithcode.com/paper/virtual-to-real-adaptation-of-pedestrian
Repo	https://github.com/ciampluca/Virtual-to-Real-Pedestrian-Detection
Framework	pytorch

Learning Sparse Classifiers: Continuous and Mixed Integer Optimization Perspectives


Title	Learning Sparse Classifiers: Continuous and Mixed Integer Optimization Perspectives
Authors	Antoine Dedieu, Hussein Hazimeh, Rahul Mazumder
Abstract	We consider a discrete optimization based approach for learning sparse classifiers, where the outcome depends upon a linear combination of a small subset of features. Recent work has shown that mixed integer programming (MIP) can be used to solve (to optimality) $\ell_0$-regularized problems at scales much larger than what was conventionally considered possible in the statistics and machine learning communities. Despite their usefulness, MIP-based approaches are significantly slower compared to relatively mature algorithms based on $\ell_1$-regularization and relatives. We aim to bridge this computational gap by developing new MIP-based algorithms for $\ell_0$-regularized classification. We propose two classes of scalable algorithms: an exact algorithm that can handle $p\approx 50,000$ features in a few minutes, and approximate algorithms that can address instances with $p\approx 10^6$ in times comparable to fast $\ell_1$-based algorithms. Our exact algorithm is based on the novel idea of \textsl{integrality generation}, which solves the original problem (with $p$ binary variables) via a sequence of mixed integer programs that involve a small number of binary variables. Our approximate algorithms are based on coordinate descent and local combinatorial search. In addition, we present new estimation error bounds for a class of $\ell_0$-regularized estimators. Experiments on real and synthetic data demonstrate that our approach leads to models with considerably improved statistical performance (especially, variable selection) when compared to competing toolkits.
Tasks
Published	2020-01-17
URL	https://arxiv.org/abs/2001.06471v1
PDF	https://arxiv.org/pdf/2001.06471v1.pdf
PWC	https://paperswithcode.com/paper/learning-sparse-classifiers-continuous-and
Repo	https://github.com/hazimehh/L0Learn
Framework	none

Dual Adversarial Domain Adaptation


Title	Dual Adversarial Domain Adaptation
Authors	Yuntao Du, Zhiwen Tan, Qian Chen, Xiaowen Zhang, Yirong Yao, Chongjun Wang
Abstract	Unsupervised domain adaptation aims at transferring knowledge from the labeled source domain to the unlabeled target domain. Previous adversarial domain adaptation methods mostly adopt the discriminator with binary or $K$-dimensional output to perform marginal or conditional alignment independently. Recent experiments have shown that when the discriminator is provided with domain information in both domains and label information in the source domain, it is able to preserve the complex multimodal information and high semantic information in both domains. Following this idea, we adopt a discriminator with $2K$-dimensional output to perform both domain-level and class-level alignments simultaneously in a single discriminator. However, a single discriminator can not capture all the useful information across domains and the relationships between the examples and the decision boundary are rarely explored before. Inspired by multi-view learning and latest advances in domain adaptation, besides the adversarial process between the discriminator and the feature extractor, we also design a novel mechanism to make two discriminators pit against each other, so that they can provide diverse information for each other and avoid generating target features outside the support of the source domain. To the best of our knowledge, it is the first time to explore a dual adversarial strategy in domain adaptation. Moreover, we also use the semi-supervised learning regularization to make the representations more discriminative. Comprehensive experiments on two real-world datasets verify that our method outperforms several state-of-the-art domain adaptation methods.
Tasks	Domain Adaptation, MULTI-VIEW LEARNING, Unsupervised Domain Adaptation
Published	2020-01-01
URL	https://arxiv.org/abs/2001.00153v1
PDF	https://arxiv.org/pdf/2001.00153v1.pdf
PWC	https://paperswithcode.com/paper/dual-adversarial-domain-adaptation
Repo	https://github.com/yaoyueduzhen/DADA
Framework	pytorch