February 1, 2020

2848 words 14 mins read

Paper Group AWR 183

High-Quality Self-Supervised Deep Image Denoising. BCN20000: Dermoscopic Lesions in the Wild. Discrete Rotation Equivariance for Point Cloud Recognition. Random Matrix-Improved Estimation of the Wasserstein Distance between two Centered Gaussian Distributions. Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. Covari …

High-Quality Self-Supervised Deep Image Denoising


Title	High-Quality Self-Supervised Deep Image Denoising
Authors	Samuli Laine, Tero Karras, Jaakko Lehtinen, Timo Aila
Abstract	We describe a novel method for training high-quality image denoising models based on unorganized collections of corrupted images. The training does not need access to clean reference images, or explicit pairs of corrupted images, and can thus be applied in situations where such data is unacceptably expensive or impossible to acquire. We build on a recent technique that removes the need for reference data by employing networks with a “blind spot” in the receptive field, and significantly improve two key aspects: image quality and training efficiency. Our result quality is on par with state-of-the-art neural network denoisers in the case of i.i.d. additive Gaussian noise, and not far behind with Poisson and impulse noise. We also successfully handle cases where parameters of the noise model are variable and/or unknown in both training and evaluation data.
Tasks	Denoising, Image Denoising
Published	2019-01-29
URL	https://arxiv.org/abs/1901.10277v3
PDF	https://arxiv.org/pdf/1901.10277v3.pdf
PWC	https://paperswithcode.com/paper/self-supervised-deep-image-denoising
Repo	https://github.com/NVlabs/selfsupervised-denoising
Framework	tf

BCN20000: Dermoscopic Lesions in the Wild


Title	BCN20000: Dermoscopic Lesions in the Wild
Authors	Marc Combalia, Noel C. F. Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C. Halpern, Susana Puig, Josep Malvehy
Abstract	This article summarizes the BCN20000 dataset, composed of 19424 dermoscopic images of skin lesions captured from 2010 to 2016 in the facilities of the Hospital Cl'inic in Barcelona. With this dataset, we aim to study the problem of unconstrained classification of dermoscopic images of skin cancer, including lesions found in hard-to-diagnose locations (nails and mucosa), large lesions which do not fit in the aperture of the dermoscopy device, and hypo-pigmented lesions. The BCN20000 will be provided to the participants of the ISIC Challenge 2019, where they will be asked to train algorithms to classify dermoscopic images of skin cancer automatically.
Tasks
Published	2019-08-06
URL	https://arxiv.org/abs/1908.02288v2
PDF	https://arxiv.org/pdf/1908.02288v2.pdf
PWC	https://paperswithcode.com/paper/bcn20000-dermoscopic-lesions-in-the-wild
Repo	https://github.com/greyhypotheses/dermatology
Framework	none

Discrete Rotation Equivariance for Point Cloud Recognition


Title	Discrete Rotation Equivariance for Point Cloud Recognition
Authors	Jiaxin Li, Yingcai Bi, Gim Hee Lee
Abstract	Despite the recent active research on processing point clouds with deep networks, few attention has been on the sensitivity of the networks to rotations. In this paper, we propose a deep learning architecture that achieves discrete $\mathbf{SO}(2)$/$\mathbf{SO}(3)$ rotation equivariance for point cloud recognition. Specifically, the rotation of an input point cloud with elements of a rotation group is similar to shuffling the feature vectors generated by our approach. The equivariance is easily reduced to invariance by eliminating the permutation with operations such as maximum or average. Our method can be directly applied to any existing point cloud based networks, resulting in significant improvements in their performance for rotated inputs. We show state-of-the-art results in the classification tasks with various datasets under both $\mathbf{SO}(2)$ and $\mathbf{SO}(3)$ rotations. In addition, we further analyze the necessary conditions of applying our approach to PointNet based networks. Source codes at https://github.com/lijx10/rot-equ-net
Tasks
Published	2019-03-31
URL	http://arxiv.org/abs/1904.00319v1
PDF	http://arxiv.org/pdf/1904.00319v1.pdf
PWC	https://paperswithcode.com/paper/discrete-rotation-equivariance-for-point
Repo	https://github.com/lijx10/rot-equ-net
Framework	pytorch

Random Matrix-Improved Estimation of the Wasserstein Distance between two Centered Gaussian Distributions


Title	Random Matrix-Improved Estimation of the Wasserstein Distance between two Centered Gaussian Distributions
Authors	Malik Tiomoko, Romain Couillet
Abstract	This article proposes a method to consistently estimate functionals $\frac1p\sum_{i=1}^pf(\lambda_i(C_1C_2))$ of the eigenvalues of the product of two covariance matrices $C_1,C_2\in\mathbb{R}^{p\times p}$ based on the empirical estimates $\lambda_i(\hat C_1\hat C_2)$ ($\hat C_a=\frac1{n_a}\sum_{i=1}^{n_a} x_i^{(a)}x_i^{(a){{\sf T}}}$), when the size $p$ and number $n_a$ of the (zero mean) samples $x_i^{(a)}$ are similar. As a corollary, a consistent estimate of the Wasserstein distance (related to the case $f(t)=\sqrt{t}$) between centered Gaussian distributions is derived. The new estimate is shown to largely outperform the classical sample covariance-based `plug-in’ estimator. Based on this finding, a practical application to covariance estimation is then devised which demonstrates potentially significant performance gains with respect to state-of-the-art alternatives. \|
Tasks
Published	2019-03-08
URL	http://arxiv.org/abs/1903.03447v1
PDF	http://arxiv.org/pdf/1903.03447v1.pdf
PWC	https://paperswithcode.com/paper/random-matrix-improved-estimation-of-the
Repo	https://github.com/maliktiomoko/RMTWasserstein
Framework	none

Graph Neural Networks Exponentially Lose Expressive Power for Node Classification


Title	Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
Authors	Kenta Oono, Taiji Suzuki
Abstract	Graph Neural Networks (graph NNs) are a promising deep learning approach for analyzing graph-structured data. However, it is known that they do not improve (or sometimes worsen) their predictive performance as we pile up many layers and add non-lineality. To tackle this problem, we investigate the expressive power of graph NNs via their asymptotic behaviors as the layer size tends to infinity. Our strategy is to generalize the forward propagation of a Graph Convolutional Network (GCN), which is a popular graph NN variant, as a specific dynamical system. In the case of a GCN, we show that when its weights satisfy the conditions determined by the spectra of the (augmented) normalized Laplacian, its output exponentially approaches the set of signals that carry information of the connected components and node degrees only for distinguishing nodes. Our theory enables us to relate the expressive power of GCNs with the topological information of the underlying graphs inherent in the graph spectra. To demonstrate this, we characterize the asymptotic behavior of GCNs on the Erd\H{o}s – R'{e}nyi graph. We show that when the Erd\H{o}s – R'{e}nyi graph is sufficiently dense and large, a broad range of GCNs on it suffers from the “information loss” in the limit of infinite layers with high probability. Based on the theory, we provide a principled guideline for weight normalization of graph NNs. We experimentally confirm that the proposed weight scaling enhances the predictive performance of GCNs in real data. Code is available at https://github.com/delta2323/gnn-asymptotics.
Tasks	Node Classification
Published	2019-05-27
URL	https://arxiv.org/abs/1905.10947v3
PDF	https://arxiv.org/pdf/1905.10947v3.pdf
PWC	https://paperswithcode.com/paper/on-asymptotic-behaviors-of-graph-cnns-from
Repo	https://github.com/delta2323/gnn-asymptotics
Framework	none

Covariate-Powered Empirical Bayes Estimation


Title	Covariate-Powered Empirical Bayes Estimation
Authors	Nikolaos Ignatiadis, Stefan Wager
Abstract	We study methods for simultaneous analysis of many noisy experiments in the presence of rich covariate information. The goal of the analyst is to optimally estimate the true effect underlying each experiment. Both the noisy experimental results and the auxiliary covariates are useful for this purpose, but neither data source on its own captures all the information available to the analyst. In this paper, we propose a flexible plug-in empirical Bayes estimator that synthesizes both sources of information and may leverage any black-box predictive model. We show that our approach is within a constant factor of minimax for a simple data-generating model. Furthermore, we establish robust convergence guarantees for our method that hold under considerable generality, and exhibit promising empirical performance on both real and simulated data.
Tasks
Published	2019-06-04
URL	https://arxiv.org/abs/1906.01611v2
PDF	https://arxiv.org/pdf/1906.01611v2.pdf
PWC	https://paperswithcode.com/paper/covariate-powered-empirical-bayes-estimation
Repo	https://github.com/UnofficialJuliaMirrorSnapshots/EBayes.jl-bad9efff-1a8e-41fb-9e7d-5d6f530fb0a3
Framework	none

Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling


Title	Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling
Authors	Ouyu Lan, Xiao Huang, Bill Yuchen Lin, He Jiang, Liyuan Liu, Xiang Ren
Abstract	Sequence labeling is a fundamental framework for various natural language processing problems. Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios. In many cases, ground truth labels are costly and time-consuming to collect or even non-existent, while imperfect ones could be easily accessed or transferred from different domains. In this paper, we propose a novel framework named consensus Network (ConNet) to conduct training with imperfect annotations from multiple sources. It learns the representation for every weak supervision source and dynamically aggregates them by a context-aware attention mechanism. Finally, it leads to a model reflecting the consensus among multiple sources. We evaluate the proposed framework in two practical settings of multisource learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings.
Tasks
Published	2019-10-09
URL	https://arxiv.org/abs/1910.04289v1
PDF	https://arxiv.org/pdf/1910.04289v1.pdf
PWC	https://paperswithcode.com/paper/learning-to-contextually-aggregate-multi
Repo	https://github.com/INK-USC/ConNet
Framework	pytorch

Miss Tools and Mr Fruit: Emergent communication in agents learning about object affordances


Title	Miss Tools and Mr Fruit: Emergent communication in agents learning about object affordances
Authors	Diane Bouchacourt, Marco Baroni
Abstract	Recent research studies communication emergence in communities of deep network agents assigned a joint task, hoping to gain insights on human language evolution. We propose here a new task capturing crucial aspects of the human environment, such as natural object affordances, and of human conversation, such as full symmetry among the participants. By conducting a thorough pragmatic and semantic analysis of the emergent protocol, we show that the agents solve the shared task through genuine bilateral, referential communication. However, the agents develop multiple idiolects, which makes us conclude that full symmetry is not a sufficient condition for a common language to emerge.
Tasks
Published	2019-05-28
URL	https://arxiv.org/abs/1905.11871v1
PDF	https://arxiv.org/pdf/1905.11871v1.pdf
PWC	https://paperswithcode.com/paper/miss-tools-and-mr-fruit-emergent
Repo	https://github.com/facebookresearch/fruit-tools-game
Framework	pytorch

Adaptive Exploration for Unsupervised Person Re-Identification


Title	Adaptive Exploration for Unsupervised Person Re-Identification
Authors	Yuhang Ding, Hehe Fan, Mingliang Xu, Yi Yang
Abstract	Due to domain bias, directly deploying a deep person re-identification (re-ID) model trained on one dataset often achieves considerably poor accuracy on another dataset. In this paper, we propose an Adaptive Exploration (AE) method to address the domain-shift problem for re-ID in an unsupervised manner. Specifically, with supervised training on the source dataset, in the target domain, the re-ID model is inducted to 1) maximize distances between all person images and 2) minimize distances between similar person images. In the first case, by treating each person image as an individual class, a non-parametric classifier with a feature memory is exploited to encourage person images to move away from each other. In the second case, according to a similarity threshold, our method adaptively selects neighborhoods in the feature space for each person image. By treating these similar person images as the same class, the non-parametric classifier forces them to stay closer. However, a problem of adaptive selection is that, when an image has too many neighborhoods, it is more likely to attract other images as its neighborhoods. As a result, a minority of images may select a large number of neighborhoods while a majority of images has only a few neighborhoods. To address this issue, we additionally integrate a balance strategy into the adaptive selection. Extensive experiments on large-scale re-ID datasets demonstrate the effectiveness of our method. Our code has been released at https://github.com/dyh127/Adaptive-Exploration-for-Unsupervised-Person-Re-Identification.
Tasks	Person Re-Identification, Unsupervised Person Re-Identification
Published	2019-07-09
URL	https://arxiv.org/abs/1907.04194v1
PDF	https://arxiv.org/pdf/1907.04194v1.pdf
PWC	https://paperswithcode.com/paper/adaptive-exploration-for-unsupervised-person
Repo	https://github.com/dyh127/Adaptive-Exploration-for-Unsupervised-Person-Re-Identification
Framework	pytorch

Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design


Title	Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
Authors	Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, Pieter Abbeel
Abstract	Flow-based generative models are powerful exact likelihood models with efficient sampling and inference. Despite their computational efficiency, flow-based models generally have much worse density modeling performance compared to state-of-the-art autoregressive models. In this paper, we investigate and improve upon three limiting design choices employed by flow-based models in prior work: the use of uniform noise for dequantization, the use of inexpressive affine flows, and the use of purely convolutional conditioning networks in coupling layers. Based on our findings, we propose Flow++, a new flow-based model that is now the state-of-the-art non-autoregressive model for unconditional density estimation on standard image benchmarks. Our work has begun to close the significant performance gap that has so far existed between autoregressive models and flow-based models. Our implementation is available at https://github.com/aravindsrinivas/flowpp
Tasks	Density Estimation, Image Generation
Published	2019-02-01
URL	https://arxiv.org/abs/1902.00275v2
PDF	https://arxiv.org/pdf/1902.00275v2.pdf
PWC	https://paperswithcode.com/paper/flow-improving-flow-based-generative-models
Repo	https://github.com/aravind0706/flowpp
Framework	tf

Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection


Title	Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection
Authors	Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, Gang Yu
Abstract	This report presents our method which wins the nuScenes3D Detection Challenge [17] held in Workshop on Autonomous Driving(WAD, CVPR 2019). Generally, we utilize sparse 3D convolution to extract rich semantic features, which are then fed into a class-balanced multi-head network to perform 3D object detection. To handle the severe class imbalance problem inherent in the autonomous driving scenarios, we design a class-balanced sampling and augmentation strategy to generate a more balanced data distribution. Furthermore, we propose a balanced group-ing head to boost the performance for the categories withsimilar shapes. Based on the Challenge results, our methodoutperforms the PointPillars [14] baseline by a large mar-gin across all metrics, achieving state-of-the-art detection performance on the nuScenes dataset. Code will be released at CBGS.
Tasks	3D Object Detection, Autonomous Driving, Object Detection
Published	2019-08-26
URL	https://arxiv.org/abs/1908.09492v1
PDF	https://arxiv.org/pdf/1908.09492v1.pdf
PWC	https://paperswithcode.com/paper/class-balanced-grouping-and-sampling-for
Repo	https://github.com/poodarchu/Class-balanced-Grouping-and-Sampling-for-Point-Cloud-3D-Object-Detection
Framework	pytorch

Transformers with convolutional context for ASR


Title	Transformers with convolutional context for ASR
Authors	Abdelrahman Mohamed, Dmytro Okhonko, Luke Zettlemoyer
Abstract	The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition. Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks. In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations. These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts. The proposed system has favorable optimization characteristics where our reported results are produced with fixed learning rate of 1.0 and no warmup steps. The proposed model achieves a competitive 4.7% and 12.9% WER on the Librispeech `test clean'' and` test other’’ subsets when no extra LM text is provided.
Tasks	Machine Translation, Speech Recognition
Published	2019-04-26
URL	https://arxiv.org/abs/1904.11660v2
PDF	https://arxiv.org/pdf/1904.11660v2.pdf
PWC	https://paperswithcode.com/paper/transformers-with-convolutional-context-for
Repo	https://github.com/insop/pytorch-hackathon
Framework	pytorch

Port-Hamiltonian Approach to Neural Network Training


Title	Port-Hamiltonian Approach to Neural Network Training
Authors	Stefano Massaroli, Michael Poli, Federico Califano, Angela Faragasso, Jinkyoo Park, Atsushi Yamashita, Hajime Asama
Abstract	Neural networks are discrete entities: subdivided into discrete layers and parametrized by weights which are iteratively optimized via difference equations. Recent work proposes networks with layer outputs which are no longer quantized but are solutions of an ordinary differential equation (ODE); however, these networks are still optimized via discrete methods (e.g. gradient descent). In this paper, we explore a different direction: namely, we propose a novel framework for learning in which the parameters themselves are solutions of ODEs. By viewing the optimization process as the evolution of a port-Hamiltonian system, we can ensure convergence to a minimum of the objective function. Numerical experiments have been performed to show the validity and effectiveness of the proposed methods.
Tasks	Time Series Forecasting
Published	2019-09-06
URL	https://arxiv.org/abs/1909.02702v1
PDF	https://arxiv.org/pdf/1909.02702v1.pdf
PWC	https://paperswithcode.com/paper/port-hamiltonian-approach-to-neural-network
Repo	https://github.com/Zymrael/PortHamiltonianNN
Framework	pytorch

Scaleable input gradient regularization for adversarial robustness


Title	Scaleable input gradient regularization for adversarial robustness
Authors	Chris Finlay, Adam M Oberman
Abstract	In this work we revisit gradient regularization for adversarial robustness with some new ingredients. First, we derive new per-image theoretical robustness bounds based on local gradient information. These bounds strongly motivate input gradient regularization. Second, we implement a scaleable version of input gradient regularization which avoids double backpropagation: adversarially robust ImageNet models are trained in 33 hours on four consumer grade GPUs. Finally, we show experimentally and through theoretical certification that input gradient regularization is competitive with adversarial training. Moreover we demonstrate that gradient regularization does not lead to gradient obfuscation or gradient masking.
Tasks	Adversarial Attack, Adversarial Defense
Published	2019-05-27
URL	https://arxiv.org/abs/1905.11468v2
PDF	https://arxiv.org/pdf/1905.11468v2.pdf
PWC	https://paperswithcode.com/paper/scaleable-input-gradient-regularization-for
Repo	https://github.com/cfinlay/tulip
Framework	pytorch

Network Pruning via Transformable Architecture Search


Title	Network Pruning via Transformable Architecture Search
Authors	Xuanyi Dong, Yi Yang
Abstract	Network pruning reduces the computation costs of an over-parameterized network without performance damage. Prevailing pruning algorithms pre-define the width and depth of the pruned networks, and then transfer parameters from the unpruned network to pruned networks. To break the structure limitation of the pruned networks, we propose to apply neural architecture search to search directly for a network with flexible channel and layer sizes. The number of the channels/layers is learned by minimizing the loss of the pruned networks. The feature map of the pruned network is an aggregation of K feature map fragments (generated by K networks of different sizes), which are sampled based on the probability distribution.The loss can be back-propagated not only to the network weights, but also to the parameterized distribution to explicitly tune the size of the channels/layers. Specifically, we apply channel-wise interpolation to keep the feature map with different channel sizes aligned in the aggregation procedure. The maximum probability for the size in each distribution serves as the width and depth of the pruned network, whose parameters are learned by knowledge transfer, e.g., knowledge distillation, from the original networks. Experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate the effectiveness of our new perspective of network pruning compared to traditional network pruning algorithms. Various searching and knowledge transfer approaches are conducted to show the effectiveness of the two components. Code is at: https://github.com/D-X-Y/NAS-Projects.
Tasks	Network Pruning, Neural Architecture Search, Transfer Learning
Published	2019-05-23
URL	https://arxiv.org/abs/1905.09717v5
PDF	https://arxiv.org/pdf/1905.09717v5.pdf
PWC	https://paperswithcode.com/paper/network-pruning-via-transformable
Repo	https://github.com/D-X-Y/GDAS
Framework	pytorch