February 2, 2020

3025 words 15 mins read

Paper Group AWR 17

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group. CNN-generated images are surprisingly easy to spot… for now. A Graph Neural Network Approach for Scalable Wireless Power Control. Learning a Lattice Planner Control Set for Autonomous Vehicles. Celeb-DF: A Large-scale Challenging Dataset …

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group


Title	Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
Authors	Mario Lezcano-Casado, David Martínez-Rubio
Abstract	We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.
Tasks
Published	2019-01-24
URL	https://arxiv.org/abs/1901.08428v3
PDF	https://arxiv.org/pdf/1901.08428v3.pdf
PWC	https://paperswithcode.com/paper/cheap-orthogonal-constraints-in-neural
Repo	https://github.com/Lezcano/expRNN
Framework	pytorch

CNN-generated images are surprisingly easy to spot… for now


Title	CNN-generated images are surprisingly easy to spot… for now
Authors	Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, Alexei A. Efros
Abstract	In this work we ask whether it is possible to create a “universal” detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used. To test this, we collect a dataset consisting of fake images generated by 11 different CNN-based image generator models, chosen to span the space of commonly used architectures today (ProGAN, StyleGAN, BigGAN, CycleGAN, StarGAN, GauGAN, DeepFakes, cascaded refinement networks, implicit maximum likelihood estimation, second-order attention super-resolution, seeing-in-the-dark). We demonstrate that, with careful pre- and post-processing and data augmentation, a standard image classifier trained on only one specific CNN generator (ProGAN) is able to generalize surprisingly well to unseen architectures, datasets, and training methods (including the just released StyleGAN2). Our findings suggest the intriguing possibility that today’s CNN-generated images share some common systematic flaws, preventing them from achieving realistic image synthesis.
Tasks	Data Augmentation, Image Generation, Super-Resolution
Published	2019-12-23
URL	https://arxiv.org/abs/1912.11035v1
PDF	https://arxiv.org/pdf/1912.11035v1.pdf
PWC	https://paperswithcode.com/paper/cnn-generated-images-are-surprisingly-easy-to
Repo	https://github.com/PeterWang512/CNNDetection
Framework	pytorch

A Graph Neural Network Approach for Scalable Wireless Power Control


Title	A Graph Neural Network Approach for Scalable Wireless Power Control
Authors	Yifei Shen, Yuanming Shi, Jun Zhang, Khaled B. Letaief
Abstract	Deep neural networks have recently emerged as a disruptive technology to solve NP-hard wireless resource allocation problems in a real-time manner. However, the adopted neural network structures, e.g., multi-layer perceptron (MLP) and convolutional neural network (CNN), are inherited from deep learning for image processing tasks, and thus are not tailored to problems in wireless networks. In particular, the performance of these methods deteriorates dramatically when the wireless network size becomes large. In this paper, we propose to utilize graph neural networks (GNNs) to develop scalable methods for solving the power control problem in $K$-user interference channels. Specifically, a $K$-user interference channel is first modeled as a complete graph, where the quantitative information of wireless channels is incorporated as the features of the graph. We then propose an interference graph convolutional neural network (IGCNet) to learn the optimal power control in an unsupervised manner. It is shown that one-layer IGCNet is a universal approximator to continuous set functions, which well matches the permutation invariance property of interference channels and it is robust to imperfect channel state information (CSI). Extensive simulations will show that the proposed IGCNet outperforms existing methods and achieves significant speedup over the classic algorithm for power control, namely, WMMSE. The code is available on https://github.com/yshenaw/Globecom2019.
Tasks
Published	2019-07-19
URL	https://arxiv.org/abs/1907.08487v1
PDF	https://arxiv.org/pdf/1907.08487v1.pdf
PWC	https://paperswithcode.com/paper/a-graph-neural-network-approach-for-scalable
Repo	https://github.com/zhuwenxing/Deep_Learning_Based_Power_Control
Framework	pytorch

Learning a Lattice Planner Control Set for Autonomous Vehicles


Title	Learning a Lattice Planner Control Set for Autonomous Vehicles
Authors	Ryan De Iaco, Stephen L. Smith, Krzysztof Czarnecki
Abstract	This paper introduces a method to compute a sparse lattice planner control set that is suited to a particular task by learning from a representative dataset of vehicle paths. To do this, we use a scoring measure similar to the Fr'echet distance and propose an algorithm for evaluating a given control set according to the scoring measure. Control actions are then selected from a dense control set according to an objective function that rewards improvements in matching the dataset while also encouraging sparsity. This method is evaluated across several experiments involving real and synthetic datasets, and it is shown to generate smaller control sets when compared to the previous state-of-the-art lattice control set computation technique, with these smaller control sets maintaining a high degree of manoeuvrability in the required task. This results in a planning time speedup of up to 4.31x when using the learned control set over the state-of-the-art computed control set. In addition, we show the learned control sets are better able to capture the driving style of the dataset in terms of path curvature.
Tasks	Autonomous Vehicles
Published	2019-03-05
URL	http://arxiv.org/abs/1903.02044v2
PDF	http://arxiv.org/pdf/1903.02044v2.pdf
PWC	https://paperswithcode.com/paper/learning-a-lattice-planner-control-set-for
Repo	https://github.com/rdeiaco/learning_lattice_planner
Framework	none

Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics


Title	Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics
Authors	Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, Siwei Lyu
Abstract	AI-synthesized face-swapping videos, commonly known as DeepFakes, is an emerging problem threatening the trustworthiness of online information. The need to develop and evaluate DeepFake detection algorithms calls for large-scale datasets. However, current DeepFake datasets suffer from low visual quality and do not resemble DeepFake videos circulated on the Internet. We present a new large-scale challenging DeepFake video dataset, Celeb-DF, which contains 5,639 high-quality DeepFake videos of celebrities generated using improved synthesis process. We conduct a comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF.
Tasks	DeepFake Detection, Face Swapping
Published	2019-09-27
URL	https://arxiv.org/abs/1909.12962v4
PDF	https://arxiv.org/pdf/1909.12962v4.pdf
PWC	https://paperswithcode.com/paper/celeb-df-a-new-dataset-for-deepfake-forensics
Repo	https://github.com/danmohaha/celeb-deepfakeforensics
Framework	none

Query-efficient Meta Attack to Deep Neural Networks


Title	Query-efficient Meta Attack to Deep Neural Networks
Authors	Jiawei Du, Hu Zhang, Joey Tianyi Zhou, Yi Yang, Jiashi Feng
Abstract	Black-box attack methods aim to infer suitable attack patterns to targeted DNN models by only using output feedback of the models and the corresponding input queries. However, due to lack of prior and inefficiency in leveraging the query and feedback information, existing methods are mostly query-intensive for obtaining effective attack patterns. In this work, we propose a meta attack approach that is capable of attacking a targeted model with much fewer queries. Its high queryefficiency stems from effective utilization of meta learning approaches in learning generalizable prior abstraction from the previously observed attack patterns and exploiting such prior to help infer attack patterns from only a few queries and outputs. Extensive experiments on MNIST, CIFAR10 and tiny-Imagenet demonstrate that our meta-attack method can remarkably reduce the number of model queries without sacrificing the attack performance. Besides, the obtained meta attacker is not restricted to a particular model but can be used easily with a fast adaptive ability to attack a variety of models.The code of our work is available at https://github.com/dydjw9/MetaAttack_ICLR2020/.
Tasks	Adversarial Attack, Meta-Learning
Published	2019-06-06
URL	https://arxiv.org/abs/1906.02398v3
PDF	https://arxiv.org/pdf/1906.02398v3.pdf
PWC	https://paperswithcode.com/paper/query-efficient-meta-attack-to-deep-neural
Repo	https://github.com/dydjw9/MetaAttack_ICLR2020
Framework	pytorch

Interpretable and Differentially Private Predictions


Title	Interpretable and Differentially Private Predictions
Authors	Frederik Harder, Matthias Bauer, Mijung Park
Abstract	Interpretable predictions, where it is clear why a machine learning model has made a particular decision, can compromise privacy by revealing the characteristics of individual data points. This raises the central question addressed in this paper: Can models be interpretable without compromising privacy? For complex big data fit by correspondingly rich models, balancing privacy and explainability is particularly challenging, such that this question has remained largely unexplored. In this paper, we propose a family of simple models in the aim of approximating complex models using several locally linear maps per class to provide high classification accuracy, as well as differentially private explanations on the classification. We illustrate the usefulness of our approach on several image benchmark datasets as well as a medical dataset.
Tasks
Published	2019-06-05
URL	https://arxiv.org/abs/1906.02004v3
PDF	https://arxiv.org/pdf/1906.02004v3.pdf
PWC	https://paperswithcode.com/paper/interpretable-and-differentially-private
Repo	https://github.com/frhrdr/dp-llm
Framework	pytorch

Multitask Deep Learning with Spectral Knowledge for Hyperspectral Image Classification


Title	Multitask Deep Learning with Spectral Knowledge for Hyperspectral Image Classification
Authors	Shengjie Liu, Qian Shi
Abstract	In this letter, we propose a multitask deep learning method for classification of multiple hyperspectral data in a single training. Deep learning models have achieved promising results on hyperspectral image classification, but their performance highly rely on sufficient labeled samples, which are scarce on hyperspectral images. However, samples from multiple data sets might be sufficient to train one deep learning model, thereby improving its performance. To do so, we trained an identical feature extractor for all data, and the extracted features were fed into corresponding Softmax classifiers. Spectral knowledge was introduced to ensure that the shared features were similar across domains. Four hyperspectral data sets were used in the experiments. We achieved higher classification accuracies on three data sets (Pavia University, Pavia Center, and Indian Pines) and competitive results on the Salinas Valley data compared with the baseline. Spectral knowledge was useful to prevent the deep network from overfitting when the data shared similar spectral response. The proposed method tested on two deep CNNs successfully shows its ability to utilize samples from multiple data sets and enhance networks’ performance.
Tasks	Hyperspectral Image Classification, Image Classification
Published	2019-05-11
URL	https://arxiv.org/abs/1905.04535v4
PDF	https://arxiv.org/pdf/1905.04535v4.pdf
PWC	https://paperswithcode.com/paper/multitask-deep-learning-with-spectral
Repo	https://github.com/stop68/remote-sensing-image-classification
Framework	tf

GSN: A Graph-Structured Network for Multi-Party Dialogues


Title	GSN: A Graph-Structured Network for Multi-Party Dialogues
Authors	Wenpeng Hu, Zhangming Chan, Bing Liu, Dongyan Zhao, Jinwen Ma, Rui Yan
Abstract	Existing neural models for dialogue response generation assume that utterances are sequentially organized. However, many real-world dialogues involve multiple interlocutors (i.e., multi-party dialogues), where the assumption does not hold as utterances from different interlocutors can occur “in parallel.” This paper generalizes existing sequence-based models to a Graph-Structured neural Network (GSN) for dialogue modeling. The core of GSN is a graph-based encoder that can model the information flow along the graph-structured dialogues (two-party sequential dialogues are a special case). Experimental results show that GSN significantly outperforms existing sequence-based models.
Tasks
Published	2019-05-31
URL	https://arxiv.org/abs/1905.13637v1
PDF	https://arxiv.org/pdf/1905.13637v1.pdf
PWC	https://paperswithcode.com/paper/gsn-a-graph-structured-network-for-multi
Repo	https://github.com/morning-dews/GSN-Dialogues
Framework	tf

Optimizing Geometric Multigrid Methods with Evolutionary Computation


Title	Optimizing Geometric Multigrid Methods with Evolutionary Computation
Authors	Jonas Schmitt, Sebastian Kuckuk, Harald Köstler
Abstract	For many linear and nonlinear systems that arise from the discretization of partial differential equations the construction of an efficient multigrid solver is a challenging task. Here we present a novel approach for the optimization of geometric multigrid methods that is based on evolutionary computation, a generic program optimization technique inspired by the principle of natural evolution. A multigrid solver is represented as a tree of mathematical expressions which we generate based on a tailored grammar. The quality of each solver is evaluated in terms of convergence and compute performance using automated local Fourier analysis (LFA) and roofline performance modeling, respectively. Based on these objectives a multi-objective optimization is performed using strongly typed genetic programming with a non-dominated sorting based selection. To evaluate the model-based prediction and to target concrete applications, scalable implementations of an evolved solver can be automatically generated with the ExaStencils framework. We demonstrate our approach by constructing multigrid solvers for the steady-state heat equation with constant and variable coefficients that consistently perform better than common V- and W-cycles.
Tasks
Published	2019-10-07
URL	https://arxiv.org/abs/1910.02749v2
PDF	https://arxiv.org/pdf/1910.02749v2.pdf
PWC	https://paperswithcode.com/paper/optimizing-geometric-multigrid-methods-with
Repo	https://github.com/jonas-schmitt/evostencils
Framework	none

An Adaptive Remote Stochastic Gradient Method for Training Neural Networks


Title	An Adaptive Remote Stochastic Gradient Method for Training Neural Networks
Authors	Yushu Chen, Hao Jing, Wenlai Zhao, Zhiqiang Liu, Ouyi Li, Liang Qiao, Wei Xue, Haohuan Fu, Guangwen Yang
Abstract	We introduce ARSG, an adaptive first-order algorithm for training neural networks. The method computes the gradients at configurable remote observation points, in order to expedite the convergence by adjusting the step size for directions with different curvatures in the stochastic setting. It also scales the updating increment elementwise by a nonincreasing preconditioner to incorporate the avdantage of adaptive methods. The method is efficient in computation and memory, and is straightforward to implement. We analyze the convergence properties for both convex and nonconvex problems by modeling the training process as a dynamic system, that provides a guideline to select the configurable observation factor without grid search. A data-dependent regret bound is proposed to guarantee the convergence in the convex setting, which can be further improved to $O(log(T))$ for strongly convex functions. Numerical experiments demonstrate that ARSG converges faster than popular adaptive methods, such as ADAM, NADAM, AMSGRAD, and RANGER for the tested problems.
Tasks
Published	2019-05-04
URL	https://arxiv.org/abs/1905.01422v6
PDF	https://arxiv.org/pdf/1905.01422v6.pdf
PWC	https://paperswithcode.com/paper/namsg-an-efficient-method-for-training-neural
Repo	https://github.com/rationalspark/NAMSG
Framework	none

Using generative modelling to produce varied intonation for speech synthesis


Title	Using generative modelling to produce varied intonation for speech synthesis
Authors	Zack Hodari, Oliver Watts, Simon King
Abstract	Unlike human speakers, typical text-to-speech (TTS) systems are unable to produce multiple distinct renditions of a given sentence. This has previously been addressed by adding explicit external control. In contrast, generative models are able to capture a distribution over multiple renditions and thus produce varied renditions using sampling. Typical neural TTS models learn the average of the data because they minimise mean squared error. In the context of prosody, taking the average produces flatter, more boring speech: an “average prosody”. A generative model that can synthesise multiple prosodies will, by design, not model average prosody. We use variational autoencoders (VAEs) which explicitly place the most “average” data close to the mean of the Gaussian prior. We propose that by moving towards the tails of the prior distribution, the model will transition towards generating more idiosyncratic, varied renditions. Focusing here on intonation, we investigate the trade-off between naturalness and intonation variation and find that typical acoustic models can either be natural, or varied, but not both. However, sampling from the tails of the VAE prior produces much more varied intonation than the traditional approaches, whilst maintaining the same level of naturalness.
Tasks	Speech Synthesis
Published	2019-06-10
URL	https://arxiv.org/abs/1906.04233v2
PDF	https://arxiv.org/pdf/1906.04233v2.pdf
PWC	https://paperswithcode.com/paper/using-generative-modelling-to-produce-varied
Repo	https://github.com/ZackHodari/average_prosody
Framework	pytorch

Bayesian Optimisation over Multiple Continuous and Categorical Inputs


Title	Bayesian Optimisation over Multiple Continuous and Categorical Inputs
Authors	Binxin Ru, Ahsan S. Alvi, Vu Nguyen, Michael A. Osborne, Stephen J Roberts
Abstract	Efficient optimisation of black-box problems that comprise both continuous and categorical inputs is important, yet poses significant challenges. We propose a new approach, Continuous and Categorical Bayesian Optimisation (CoCaBO), which combines the strengths of multi-armed bandits and Bayesian optimisation to select values for both categorical and continuous inputs. We model this mixed-type space using a Gaussian Process kernel, designed to allow sharing of information across multiple categorical variables, each with multiple possible values; this allows CoCaBO to leverage all available data efficiently. We extend our method to the batch setting and propose an efficient selection procedure that dynamically balances exploration and exploitation whilst encouraging batch diversity. We demonstrate empirically that our method outperforms existing approaches on both synthetic and real-world optimisation tasks with continuous and categorical inputs.
Tasks	Bayesian Optimisation, Multi-Armed Bandits
Published	2019-06-20
URL	https://arxiv.org/abs/1906.08878v1
PDF	https://arxiv.org/pdf/1906.08878v1.pdf
PWC	https://paperswithcode.com/paper/bayesian-optimisation-over-multiple
Repo	https://github.com/rubinxin/CoCaBO_code
Framework	none

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding


Title	ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
Authors	Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, Haifeng Wang
Abstract	Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.
Tasks	Linguistic Acceptability, Multi-Task Learning, Natural Language Inference, Question Answering, Semantic Textual Similarity, Sentiment Analysis
Published	2019-07-29
URL	https://arxiv.org/abs/1907.12412v2
PDF	https://arxiv.org/pdf/1907.12412v2.pdf
PWC	https://paperswithcode.com/paper/ernie-20-a-continual-pre-training-framework
Repo	https://github.com/PaddlePaddle/ERNIE
Framework	none

Adversarial Examples on Graph Data: Deep Insights into Attack and Defense


Title	Adversarial Examples on Graph Data: Deep Insights into Attack and Defense
Authors	Huijun Wu, Chen Wang, Yuriy Tyshetskiy, Andrew Docherty, Kai Lu, Liming Zhu
Abstract	Graph deep learning models, such as graph convolutional networks (GCN) achieve remarkable performance for tasks on graph data. Similar to other types of deep models, graph deep learning models often suffer from adversarial attacks. However, compared with non-graph data, the discrete features, graph connections and different definitions of imperceptible perturbations bring unique challenges and opportunities for the adversarial attacks and defenses for graph data. In this paper, we propose both attack and defense techniques. For attack, we show that the discreteness problem could easily be resolved by introducing integrated gradients which could accurately reflect the effect of perturbing certain features or edges while still benefiting from the parallel computations. For defense, we observe that the adversarially manipulated graph for the targeted attack differs from normal graphs statistically. Based on this observation, we propose a defense approach which inspects the graph and recovers the potential adversarial perturbations. Our experiments on a number of datasets show the effectiveness of the proposed methods.
Tasks	Adversarial Attack, Adversarial Defense
Published	2019-03-05
URL	https://arxiv.org/abs/1903.01610v3
PDF	https://arxiv.org/pdf/1903.01610v3.pdf
PWC	https://paperswithcode.com/paper/the-vulnerabilities-of-graph-convolutional
Repo	https://github.com/stellargraph/stellargraph
Framework	tf