April 3, 2020

# Paper Group AWR 65

Tensor network approaches for learning non-linear dynamical laws. Error bounds in estimating the out-of-sample prediction error using leave-one-out cross validation in high-dimensions. Learning Generative Models using Denoising Density Estimators. Unsupervised Multi-Class Domain Adaptation: Theory, Algorithms, and Practice. Gossip and Attend: Conte …

#### Tensor network approaches for learning non-linear dynamical laws

Title Tensor network approaches for learning non-linear dynamical laws
Authors A. Goeßmann, M. Götte, I. Roth, R. Sweke, G. Kutyniok, J. Eisert
Abstract Given observations of a physical system, identifying the underlying non-linear governing equation is a fundamental task, necessary both for gaining understanding and generating deterministic future predictions. Of most practical relevance are automated approaches to theory building that scale efficiently for complex systems with many degrees of freedom. To date, available scalable methods aim at a data-driven interpolation, without exploiting or offering insight into fundamental underlying physical principles, such as locality of interactions. In this work, we show that various physical constraints can be captured via tensor network based parameterizations for the governing equation, which naturally ensures scalability. In addition to providing analytic results motivating the use of such models for realistic physical systems, we demonstrate that efficient rank-adaptive optimization algorithms can be used to learn optimal tensor network models without requiring a~priori knowledge of the exact tensor ranks. As such, we provide a physics-informed approach to recovering structured dynamical laws from data, which adaptively balances the need for expressivity and scalability.
Published 2020-02-27
URL https://arxiv.org/abs/2002.12388v1
PDF https://arxiv.org/pdf/2002.12388v1.pdf
PWC https://paperswithcode.com/paper/tensor-network-approaches-for-learning-non
Repo https://github.com/RoteKekse/systemrecovery
Framework none

#### Error bounds in estimating the out-of-sample prediction error using leave-one-out cross validation in high-dimensions

Title Error bounds in estimating the out-of-sample prediction error using leave-one-out cross validation in high-dimensions
Authors Kamiar Rahnama Rad, Wenda Zhou, Arian Maleki
Abstract We study the problem of out-of-sample risk estimation in the high dimensional regime where both the sample size $n$ and number of features $p$ are large, and $n/p$ can be less than one. Extensive empirical evidence confirms the accuracy of leave-one-out cross validation (LO) for out-of-sample risk estimation. Yet, a unifying theoretical evaluation of the accuracy of LO in high-dimensional problems has remained an open problem. This paper aims to fill this gap for penalized regression in the generalized linear family. With minor assumptions about the data generating process, and without any sparsity assumptions on the regression coefficients, our theoretical analysis obtains finite sample upper bounds on the expected squared error of LO in estimating the out-of-sample error. Our bounds show that the error goes to zero as $n,p \rightarrow \infty$, even when the dimension $p$ of the feature vectors is comparable with or greater than the sample size $n$. One technical advantage of the theory is that it can be used to clarify and connect some results from the recent literature on scalable approximate LO.
Published 2020-03-03
URL https://arxiv.org/abs/2003.01770v1
PDF https://arxiv.org/pdf/2003.01770v1.pdf
PWC https://paperswithcode.com/paper/error-bounds-in-estimating-the-out-of-sample
Framework none

#### Learning Generative Models using Denoising Density Estimators

Title Learning Generative Models using Denoising Density Estimators
Authors Siavash A. Bigdeli, Geng Lin, Tiziano Portenier, L. Andrea Dunbar, Matthias Zwicker
Abstract Learning generative probabilistic models that can estimate the continuous density given a set of samples, and that can sample from that density, is one of the fundamental challenges in unsupervised machine learning. In this paper we introduce a new approach to obtain such models based on what we call denoising density estimators (DDEs). A DDE is a scalar function, parameterized by a neural network, that is efficiently trained to represent a kernel density estimator of the data. Leveraging DDEs, our main contribution is to develop a novel approach to obtain generative models that sample from given densities. We prove that our algorithms to obtain both DDEs and generative models are guaranteed to converge to the correct solutions. Advantages of our approach include that we do not require specific network architectures like in normalizing flows, ordinary differential equation solvers as in continuous normalizing flows, nor do we require adversarial training as in generative adversarial networks (GANs). Finally, we provide experimental results that demonstrate practical applications of our technique.
Published 2020-01-08
URL https://arxiv.org/abs/2001.02728v1
PDF https://arxiv.org/pdf/2001.02728v1.pdf
PWC https://paperswithcode.com/paper/learning-generative-models-using-denoising-1
Repo https://github.com/logchan/dde
Framework pytorch

#### Unsupervised Multi-Class Domain Adaptation: Theory, Algorithms, and Practice

Title Unsupervised Multi-Class Domain Adaptation: Theory, Algorithms, and Practice
Authors Yabin Zhang, Bin Deng, Hui Tang, Lei Zhang, Kui Jia
Abstract In this paper, we study the formalism of unsupervised multi-class domain adaptation (multi-class UDA), which underlies some recent algorithms whose learning objectives are only motivated empirically. A Multi-Class Scoring Disagreement (MCSD) divergence is presented by aggregating the absolute margin violations in multi-class classification; the proposed MCSD is able to fully characterize the relations between any pair of multi-class scoring hypotheses. By using MCSD as a measure of domain distance, we develop a new domain adaptation bound for multi-class UDA as well as its data-dependent, probably approximately correct bound, which naturally suggest adversarial learning objectives to align conditional feature distributions across the source and target domains. Consequently, an algorithmic framework of Multi-class Domain-adversarial learning Networks (McDalNets) is developed, whose different instantiations via surrogate learning objectives either coincide with or resemble a few of recently popular methods, thus (partially) underscoring their practical effectiveness. Based on our same theory of multi-class UDA, we also introduce a new algorithm of Domain-Symmetric Networks (SymmNets), which is featured by a novel adversarial strategy of domain confusion and discrimination. SymmNets afford simple extensions that work equally well under the problem settings of either closed set, partial, or open set UDA. We conduct careful empirical studies to compare different algorithms of McDalNets and our newly introduced SymmNets. Experiments verify our theoretical analysis and show the efficacy of our proposed SymmNets. We make our implementation codes publicly available.
Published 2020-02-20
URL https://arxiv.org/abs/2002.08681v1
PDF https://arxiv.org/pdf/2002.08681v1.pdf
Repo https://github.com/YBZh/MultiClassDA
Framework pytorch

#### Gossip and Attend: Context-Sensitive Graph Representation Learning

Title Gossip and Attend: Context-Sensitive Graph Representation Learning
Authors Zekarias T. Kefato, Sarunas Girdzijauskas
Abstract Graph representation learning (GRL) is a powerful technique for learning low-dimensional vector representation of high-dimensional and often sparse graphs. Most studies explore the structure and metadata associated with the graph using random walks and employ an unsupervised or semi-supervised learning schemes. Learning in these methods is context-free, resulting in only a single representation per node. Recently studies have argued on the adequacy of a single representation and proposed context-sensitive approaches, which are capable of extracting multiple node representations for different contexts. This proved to be highly effective in applications such as link prediction and ranking. However, most of these methods rely on additional textual features that require complex and expensive RNNs or CNNs to capture high-level features or rely on a community detection algorithm to identify multiple contexts of a node. In this study we show that in-order to extract high-quality context-sensitive node representations it is not needed to rely on supplementary node features, nor to employ computationally heavy and complex models. We propose GOAT, a context-sensitive algorithm inspired by gossip communication and a mutual attention mechanism simply over the structure of the graph. We show the efficacy of GOAT using 6 real-world datasets on link prediction and node clustering tasks and compare it against 12 popular and state-of-the-art (SOTA) baselines. GOAT consistently outperforms them and achieves up to 12% and 19% gain over the best performing methods on link prediction and clustering tasks, respectively.
Published 2020-03-30
URL https://arxiv.org/abs/2004.00413v1
PDF https://arxiv.org/pdf/2004.00413v1.pdf
PWC https://paperswithcode.com/paper/gossip-and-attend-context-sensitive-graph
Repo https://github.com/zekarias-tilahun/goat
Framework none

#### Towards Lifelong Self-Supervision For Unpaired Image-to-Image Translation

Title Towards Lifelong Self-Supervision For Unpaired Image-to-Image Translation
Authors Victor Schmidt, Makesh Narsimhan Sreedhar, Mostafa ElAraby, Irina Rish
Abstract Unpaired Image-to-Image Translation (I2IT) tasks often suffer from lack of data, a problem which self-supervised learning (SSL) has recently been very popular and successful at tackling. Leveraging auxiliary tasks such as rotation prediction or generative colorization, SSL can produce better and more robust representations in a low data regime. Training such tasks along an I2IT task is however computationally intractable as model size and the number of task grow. On the other hand, learning sequentially could incur catastrophic forgetting of previously learned tasks. To alleviate this, we introduce Lifelong Self-Supervision (LiSS) as a way to pre-train an I2IT model (e.g., CycleGAN) on a set of self-supervised auxiliary tasks. By keeping an exponential moving average of past encoders and distilling the accumulated knowledge, we are able to maintain the network’s validation performance on a number of tasks without any form of replay, parameter isolation or retraining techniques typically used in continual learning. We show that models trained with LiSS perform better on past tasks, while also being more robust than the CycleGAN baseline to color bias and entity entanglement (when two entities are very close).
Tasks Colorization, Continual Learning, Image-to-Image Translation
Published 2020-03-31
URL https://arxiv.org/abs/2004.00161v1
PDF https://arxiv.org/pdf/2004.00161v1.pdf
PWC https://paperswithcode.com/paper/towards-lifelong-self-supervision-for
Repo https://github.com/vict0rsch/LiSS
Framework none

#### Incremental Unsupervised Domain-Adversarial Training of Neural Networks

Title Incremental Unsupervised Domain-Adversarial Training of Neural Networks
Authors Antonio-Javier Gallego, Jorge Calvo-Zaragoza, Robert B. Fisher
Abstract In the context of supervised statistical learning, it is typically assumed that the training set comes from the same distribution that draws the test samples. When this is not the case, the behavior of the learned model is unpredictable and becomes dependent upon the degree of similarity between the distribution of the training set and the distribution of the test set. One of the research topics that investigates this scenario is referred to as domain adaptation. Deep neural networks brought dramatic advances in pattern recognition and that is why there have been many attempts to provide good domain adaptation algorithms for these models. Here we take a different avenue and approach the problem from an incremental point of view, where the model is adapted to the new domain iteratively. We make use of an existing unsupervised domain-adaptation algorithm to identify the target samples on which there is greater confidence about their true label. The output of the model is analyzed in different ways to determine the candidate samples. The selected set is then added to the source training set by considering the labels provided by the network as ground truth, and the process is repeated until all target samples are labelled. Our results report a clear improvement with respect to the non-incremental case in several datasets, also outperforming other state-of-the-art domain adaptation algorithms.
Published 2020-01-13
URL https://arxiv.org/abs/2001.04129v1
PDF https://arxiv.org/pdf/2001.04129v1.pdf
Repo https://github.com/ajgallego/dann
Framework tf

#### A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset

Title A Novel Recurrent Encoder-Decoder Structure for Large-Scale Multi-view Stereo Reconstruction from An Open Aerial Dataset
Authors Jin Liu, Shunping Ji
Abstract A great deal of research has demonstrated recently that multi-view stereo (MVS) matching can be solved with deep learning methods. However, these efforts were focused on close-range objects and only a very few of the deep learning-based methods were specifically designed for large-scale 3D urban reconstruction due to the lack of multi-view aerial image benchmarks. In this paper, we present a synthetic aerial dataset, called the WHU dataset, we created for MVS tasks, which, to our knowledge, is the first large-scale multi-view aerial dataset. It was generated from a highly accurate 3D digital surface model produced from thousands of real aerial images with precise camera parameters. We also introduce in this paper a novel network, called RED-Net, for wide-range depth inference, which we developed from a recurrent encoder-decoder structure to regularize cost maps across depths and a 2D fully convolutional network as framework. RED-Net’s low memory requirements and high performance make it suitable for large-scale and highly accurate 3D Earth surface reconstruction. Our experiments confirmed that not only did our method exceed the current state-of-the-art MVS methods by more than 50% mean absolute error (MAE) with less memory and computational cost, but its efficiency as well. It outperformed one of the best commercial software programs based on conventional methods, improving their efficiency 16 times over. Moreover, we proved that our RED-Net model pre-trained on the synthetic WHU dataset can be efficiently transferred to very different multi-view aerial image datasets without any fine-tuning. Dataset are available at http://gpcv.whu.edu.cn/data.
Published 2020-03-02
URL https://arxiv.org/abs/2003.00637v3
PDF https://arxiv.org/pdf/2003.00637v3.pdf
PWC https://paperswithcode.com/paper/a-novel-recurrent-encoder-decoder-structure
Repo https://github.com/gpcv-liujin/REDNet
Framework tf

#### A Comprehensive Approach to Unsupervised Embedding Learning based on AND Algorithm

Title A Comprehensive Approach to Unsupervised Embedding Learning based on AND Algorithm
Authors Sungwon Han, Yizhan Xu, Sungwon Park, Meeyoung Cha, Cheng-Te Li
Abstract Unsupervised embedding learning aims to extract good representation from data without the need for any manual labels, which has been a critical challenge in many supervised learning tasks. This paper proposes a new unsupervised embedding approach, called Super-AND, which extends the current state-of-the-art model. Super-AND has its unique set of losses that can gather similar samples nearby within a low-density space while keeping invariant features intact against data augmentation. Super-AND outperforms all existing approaches and achieves an accuracy of 89.2% on the image classification task for CIFAR-10. We discuss the practical implications of this method in assisting semi-supervised tasks.
Published 2020-02-26
URL https://arxiv.org/abs/2002.12158v1
PDF https://arxiv.org/pdf/2002.12158v1.pdf
PWC https://paperswithcode.com/paper/a-comprehensive-approach-to-unsupervised
Repo https://github.com/super-AND/super-AND
Framework pytorch

#### Editable Neural Networks

Title Editable Neural Networks
Authors Anton Sinitsin, Vsevolod Plokhotnyuk, Vsevolod Plokhotnyuk, Sergei Popov, Artem Babenko
Abstract These days deep neural networks are ubiquitously used in a wide range of tasks, from image classification and machine translation to face identification and self-driving cars. In many applications, a single model error can lead to devastating financial, reputational and even life-threatening consequences. Therefore, it is crucially important to correct model mistakes quickly as they appear. In this work, we investigate the problem of neural network editing $-$ how one can efficiently patch a mistake of the model on a particular sample, without influencing the model behavior on other samples. Namely, we propose Editable Training, a model-agnostic training technique that encourages fast editing of the trained model. We empirically demonstrate the effectiveness of this method on large-scale image classification and machine translation tasks.
Tasks Face Identification, Image Classification, Machine Translation, Self-Driving Cars
Published 2020-04-01
URL https://arxiv.org/abs/2004.00345v1
PDF https://arxiv.org/pdf/2004.00345v1.pdf
PWC https://paperswithcode.com/paper/editable-neural-networks-1
Repo https://github.com/editable-ICLR2020/editable
Framework pytorch

#### Computationally Tractable Riemannian Manifolds for Graph Embeddings

Title Computationally Tractable Riemannian Manifolds for Graph Embeddings
Authors Calin Cruceru, Gary Bécigneul, Octavian-Eugen Ganea
Abstract Representing graphs as sets of node embeddings in certain curved Riemannian manifolds has recently gained momentum in machine learning due to their desirable geometric inductive biases, e.g., hierarchical structures benefit from hyperbolic geometry. However, going beyond embedding spaces of constant sectional curvature, while potentially more representationally powerful, proves to be challenging as one can easily lose the appeal of computationally tractable tools such as geodesic distances or Riemannian gradients. Here, we explore computationally efficient matrix manifolds, showcasing how to learn and optimize graph embeddings in these Riemannian spaces. Empirically, we demonstrate consistent improvements over Euclidean geometry while often outperforming hyperbolic and elliptical embeddings based on various metrics that capture different graph properties. Our results serve as new evidence for the benefits of non-Euclidean embeddings in machine learning pipelines.
Published 2020-02-20
URL https://arxiv.org/abs/2002.08665v1
PDF https://arxiv.org/pdf/2002.08665v1.pdf
PWC https://paperswithcode.com/paper/computationally-tractable-riemannian
Repo https://github.com/dalab/matrix-manifolds
Framework pytorch

#### Neural Contours: Learning to Draw Lines from 3D Shapes

Title Neural Contours: Learning to Draw Lines from 3D Shapes
Authors Difan Liu, Mohamed Nabail, Aaron Hertzmann, Evangelos Kalogerakis
Abstract This paper introduces a method for learning to generate line drawings from 3D models. Our architecture incorporates a differentiable module operating on geometric features of the 3D model, and an image-based module operating on view-based shape representations. At test time, geometric and view-based reasoning are combined with the help of a neural module to create a line drawing. The model is trained on a large number of crowdsourced comparisons of line drawings. Experiments demonstrate that our method achieves significant improvements in line drawing over the state-of-the-art when evaluated on standard benchmarks, resulting in drawings that are comparable to those produced by experienced human artists.
Published 2020-03-23
URL https://arxiv.org/abs/2003.10333v2
PDF https://arxiv.org/pdf/2003.10333v2.pdf
PWC https://paperswithcode.com/paper/neural-contours-learning-to-draw-lines-from
Repo https://github.com/DifanLiu/NeuralContours
Framework pytorch

#### Virtual to Real adaptation of Pedestrian Detectors for Smart Cities

Title Virtual to Real adaptation of Pedestrian Detectors for Smart Cities
Authors Luca Ciampi, Nicola Messina, Fabrizio Falchi, Claudio Gennaro, Giuseppe Amato
Abstract Pedestrian detection through computer vision is a building block for a multitude of applications in the context of smart cities, such as surveillance of sensitive areas, personal safety, monitoring, and control of pedestrian flow, to mention only a few. Recently, there was an increasing interest in deep learning architectures for performing such a task. One of the critical objectives of these algorithms is to generalize the knowledge gained during the training phase to new scenarios having various characteristics, and a suitably labeled dataset is fundamental to achieve this goal. The main problem is that manually annotating a dataset usually requires a lot of human effort, and it is a time-consuming operation. For this reason, in this work, we introduced ViPeD - Virtual Pedestrian Dataset, a new synthetically generated set of images collected from a realistic 3D video game where the labels can be automatically generated exploiting 2D pedestrian positions extracted from the graphics engine. We used this new synthetic dataset training a state-of-the-art computationally-efficient Convolutional Neural Network (CNN) that is ready to be installed in smart low-power devices, like smart cameras. We addressed the problem of the domain-adaptation from the virtual world to the real one by fine-tuning the CNN using the synthetic data and also exploiting a mixed-batch supervised training approach. Extensive experimentation carried out on different real-world datasets shows very competitive results compared to other methods presented in the literature in which the algorithms are trained using real-world data.
Published 2020-01-09
URL https://arxiv.org/abs/2001.03032v1
PDF https://arxiv.org/pdf/2001.03032v1.pdf
Repo https://github.com/ciampluca/Virtual-to-Real-Pedestrian-Detection
Framework pytorch

#### Learning Sparse Classifiers: Continuous and Mixed Integer Optimization Perspectives

Title Learning Sparse Classifiers: Continuous and Mixed Integer Optimization Perspectives
Authors Antoine Dedieu, Hussein Hazimeh, Rahul Mazumder
Abstract We consider a discrete optimization based approach for learning sparse classifiers, where the outcome depends upon a linear combination of a small subset of features. Recent work has shown that mixed integer programming (MIP) can be used to solve (to optimality) $\ell_0$-regularized problems at scales much larger than what was conventionally considered possible in the statistics and machine learning communities. Despite their usefulness, MIP-based approaches are significantly slower compared to relatively mature algorithms based on $\ell_1$-regularization and relatives. We aim to bridge this computational gap by developing new MIP-based algorithms for $\ell_0$-regularized classification. We propose two classes of scalable algorithms: an exact algorithm that can handle $p\approx 50,000$ features in a few minutes, and approximate algorithms that can address instances with $p\approx 10^6$ in times comparable to fast $\ell_1$-based algorithms. Our exact algorithm is based on the novel idea of \textsl{integrality generation}, which solves the original problem (with $p$ binary variables) via a sequence of mixed integer programs that involve a small number of binary variables. Our approximate algorithms are based on coordinate descent and local combinatorial search. In addition, we present new estimation error bounds for a class of $\ell_0$-regularized estimators. Experiments on real and synthetic data demonstrate that our approach leads to models with considerably improved statistical performance (especially, variable selection) when compared to competing toolkits.
Published 2020-01-17
URL https://arxiv.org/abs/2001.06471v1
PDF https://arxiv.org/pdf/2001.06471v1.pdf
PWC https://paperswithcode.com/paper/learning-sparse-classifiers-continuous-and
Repo https://github.com/hazimehh/L0Learn
Framework none

Abstract Unsupervised domain adaptation aims at transferring knowledge from the labeled source domain to the unlabeled target domain. Previous adversarial domain adaptation methods mostly adopt the discriminator with binary or $K$-dimensional output to perform marginal or conditional alignment independently. Recent experiments have shown that when the discriminator is provided with domain information in both domains and label information in the source domain, it is able to preserve the complex multimodal information and high semantic information in both domains. Following this idea, we adopt a discriminator with $2K$-dimensional output to perform both domain-level and class-level alignments simultaneously in a single discriminator. However, a single discriminator can not capture all the useful information across domains and the relationships between the examples and the decision boundary are rarely explored before. Inspired by multi-view learning and latest advances in domain adaptation, besides the adversarial process between the discriminator and the feature extractor, we also design a novel mechanism to make two discriminators pit against each other, so that they can provide diverse information for each other and avoid generating target features outside the support of the source domain. To the best of our knowledge, it is the first time to explore a dual adversarial strategy in domain adaptation. Moreover, we also use the semi-supervised learning regularization to make the representations more discriminative. Comprehensive experiments on two real-world datasets verify that our method outperforms several state-of-the-art domain adaptation methods.