February 2, 2020

3025 words 15 mins read

Paper Group AWR 17

Paper Group AWR 17

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group. CNN-generated images are surprisingly easy to spot… for now. A Graph Neural Network Approach for Scalable Wireless Power Control. Learning a Lattice Planner Control Set for Autonomous Vehicles. Celeb-DF: A Large-scale Challenging Dataset …

Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group

Title Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
Authors Mario Lezcano-Casado, David Martínez-Rubio
Abstract We introduce a novel approach to perform first-order optimization with orthogonal and unitary constraints. This approach is based on a parametrization stemming from Lie group theory through the exponential map. The parametrization transforms the constrained optimization problem into an unconstrained one over a Euclidean space, for which common first-order optimization methods can be used. The theoretical results presented are general enough to cover the special orthogonal group, the unitary group and, in general, any connected compact Lie group. We discuss how this and other parametrizations can be computed efficiently through an implementation trick, making numerically complex parametrizations usable at a negligible runtime cost in neural networks. In particular, we apply our results to RNNs with orthogonal recurrent weights, yielding a new architecture called expRNN. We demonstrate how our method constitutes a more robust approach to optimization with orthogonal constraints, showing faster, accurate, and more stable convergence in several tasks designed to test RNNs.
Published 2019-01-24
URL https://arxiv.org/abs/1901.08428v3
PDF https://arxiv.org/pdf/1901.08428v3.pdf
PWC https://paperswithcode.com/paper/cheap-orthogonal-constraints-in-neural
Repo https://github.com/Lezcano/expRNN
Framework pytorch

CNN-generated images are surprisingly easy to spot… for now

Title CNN-generated images are surprisingly easy to spot… for now
Authors Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, Alexei A. Efros
Abstract In this work we ask whether it is possible to create a “universal” detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used. To test this, we collect a dataset consisting of fake images generated by 11 different CNN-based image generator models, chosen to span the space of commonly used architectures today (ProGAN, StyleGAN, BigGAN, CycleGAN, StarGAN, GauGAN, DeepFakes, cascaded refinement networks, implicit maximum likelihood estimation, second-order attention super-resolution, seeing-in-the-dark). We demonstrate that, with careful pre- and post-processing and data augmentation, a standard image classifier trained on only one specific CNN generator (ProGAN) is able to generalize surprisingly well to unseen architectures, datasets, and training methods (including the just released StyleGAN2). Our findings suggest the intriguing possibility that today’s CNN-generated images share some common systematic flaws, preventing them from achieving realistic image synthesis.
Tasks Data Augmentation, Image Generation, Super-Resolution
Published 2019-12-23
URL https://arxiv.org/abs/1912.11035v1
PDF https://arxiv.org/pdf/1912.11035v1.pdf
PWC https://paperswithcode.com/paper/cnn-generated-images-are-surprisingly-easy-to
Repo https://github.com/PeterWang512/CNNDetection
Framework pytorch

A Graph Neural Network Approach for Scalable Wireless Power Control

Title A Graph Neural Network Approach for Scalable Wireless Power Control
Authors Yifei Shen, Yuanming Shi, Jun Zhang, Khaled B. Letaief
Abstract Deep neural networks have recently emerged as a disruptive technology to solve NP-hard wireless resource allocation problems in a real-time manner. However, the adopted neural network structures, e.g., multi-layer perceptron (MLP) and convolutional neural network (CNN), are inherited from deep learning for image processing tasks, and thus are not tailored to problems in wireless networks. In particular, the performance of these methods deteriorates dramatically when the wireless network size becomes large. In this paper, we propose to utilize graph neural networks (GNNs) to develop scalable methods for solving the power control problem in $K$-user interference channels. Specifically, a $K$-user interference channel is first modeled as a complete graph, where the quantitative information of wireless channels is incorporated as the features of the graph. We then propose an interference graph convolutional neural network (IGCNet) to learn the optimal power control in an unsupervised manner. It is shown that one-layer IGCNet is a universal approximator to continuous set functions, which well matches the permutation invariance property of interference channels and it is robust to imperfect channel state information (CSI). Extensive simulations will show that the proposed IGCNet outperforms existing methods and achieves significant speedup over the classic algorithm for power control, namely, WMMSE. The code is available on https://github.com/yshenaw/Globecom2019.
Published 2019-07-19
URL https://arxiv.org/abs/1907.08487v1
PDF https://arxiv.org/pdf/1907.08487v1.pdf
PWC https://paperswithcode.com/paper/a-graph-neural-network-approach-for-scalable
Repo https://github.com/zhuwenxing/Deep_Learning_Based_Power_Control
Framework pytorch

Learning a Lattice Planner Control Set for Autonomous Vehicles

Title Learning a Lattice Planner Control Set for Autonomous Vehicles
Authors Ryan De Iaco, Stephen L. Smith, Krzysztof Czarnecki
Abstract This paper introduces a method to compute a sparse lattice planner control set that is suited to a particular task by learning from a representative dataset of vehicle paths. To do this, we use a scoring measure similar to the Fr'echet distance and propose an algorithm for evaluating a given control set according to the scoring measure. Control actions are then selected from a dense control set according to an objective function that rewards improvements in matching the dataset while also encouraging sparsity. This method is evaluated across several experiments involving real and synthetic datasets, and it is shown to generate smaller control sets when compared to the previous state-of-the-art lattice control set computation technique, with these smaller control sets maintaining a high degree of manoeuvrability in the required task. This results in a planning time speedup of up to 4.31x when using the learned control set over the state-of-the-art computed control set. In addition, we show the learned control sets are better able to capture the driving style of the dataset in terms of path curvature.
Tasks Autonomous Vehicles
Published 2019-03-05
URL http://arxiv.org/abs/1903.02044v2
PDF http://arxiv.org/pdf/1903.02044v2.pdf
PWC https://paperswithcode.com/paper/learning-a-lattice-planner-control-set-for
Repo https://github.com/rdeiaco/learning_lattice_planner
Framework none

Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics

Title Celeb-DF: A Large-scale Challenging Dataset for DeepFake Forensics
Authors Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, Siwei Lyu
Abstract AI-synthesized face-swapping videos, commonly known as DeepFakes, is an emerging problem threatening the trustworthiness of online information. The need to develop and evaluate DeepFake detection algorithms calls for large-scale datasets. However, current DeepFake datasets suffer from low visual quality and do not resemble DeepFake videos circulated on the Internet. We present a new large-scale challenging DeepFake video dataset, Celeb-DF, which contains 5,639 high-quality DeepFake videos of celebrities generated using improved synthesis process. We conduct a comprehensive evaluation of DeepFake detection methods and datasets to demonstrate the escalated level of challenges posed by Celeb-DF.
Tasks DeepFake Detection, Face Swapping
Published 2019-09-27
URL https://arxiv.org/abs/1909.12962v4
PDF https://arxiv.org/pdf/1909.12962v4.pdf
PWC https://paperswithcode.com/paper/celeb-df-a-new-dataset-for-deepfake-forensics
Repo https://github.com/danmohaha/celeb-deepfakeforensics
Framework none

Query-efficient Meta Attack to Deep Neural Networks

Title Query-efficient Meta Attack to Deep Neural Networks
Authors Jiawei Du, Hu Zhang, Joey Tianyi Zhou, Yi Yang, Jiashi Feng
Abstract Black-box attack methods aim to infer suitable attack patterns to targeted DNN models by only using output feedback of the models and the corresponding input queries. However, due to lack of prior and inefficiency in leveraging the query and feedback information, existing methods are mostly query-intensive for obtaining effective attack patterns. In this work, we propose a meta attack approach that is capable of attacking a targeted model with much fewer queries. Its high queryefficiency stems from effective utilization of meta learning approaches in learning generalizable prior abstraction from the previously observed attack patterns and exploiting such prior to help infer attack patterns from only a few queries and outputs. Extensive experiments on MNIST, CIFAR10 and tiny-Imagenet demonstrate that our meta-attack method can remarkably reduce the number of model queries without sacrificing the attack performance. Besides, the obtained meta attacker is not restricted to a particular model but can be used easily with a fast adaptive ability to attack a variety of models.The code of our work is available at https://github.com/dydjw9/MetaAttack_ICLR2020/.
Tasks Adversarial Attack, Meta-Learning
Published 2019-06-06
URL https://arxiv.org/abs/1906.02398v3
PDF https://arxiv.org/pdf/1906.02398v3.pdf
PWC https://paperswithcode.com/paper/query-efficient-meta-attack-to-deep-neural
Repo https://github.com/dydjw9/MetaAttack_ICLR2020
Framework pytorch

Interpretable and Differentially Private Predictions

Title Interpretable and Differentially Private Predictions
Authors Frederik Harder, Matthias Bauer, Mijung Park
Abstract Interpretable predictions, where it is clear why a machine learning model has made a particular decision, can compromise privacy by revealing the characteristics of individual data points. This raises the central question addressed in this paper: Can models be interpretable without compromising privacy? For complex big data fit by correspondingly rich models, balancing privacy and explainability is particularly challenging, such that this question has remained largely unexplored. In this paper, we propose a family of simple models in the aim of approximating complex models using several locally linear maps per class to provide high classification accuracy, as well as differentially private explanations on the classification. We illustrate the usefulness of our approach on several image benchmark datasets as well as a medical dataset.
Published 2019-06-05
URL https://arxiv.org/abs/1906.02004v3
PDF https://arxiv.org/pdf/1906.02004v3.pdf
PWC https://paperswithcode.com/paper/interpretable-and-differentially-private
Repo https://github.com/frhrdr/dp-llm
Framework pytorch

Multitask Deep Learning with Spectral Knowledge for Hyperspectral Image Classification

Title Multitask Deep Learning with Spectral Knowledge for Hyperspectral Image Classification
Authors Shengjie Liu, Qian Shi
Abstract In this letter, we propose a multitask deep learning method for classification of multiple hyperspectral data in a single training. Deep learning models have achieved promising results on hyperspectral image classification, but their performance highly rely on sufficient labeled samples, which are scarce on hyperspectral images. However, samples from multiple data sets might be sufficient to train one deep learning model, thereby improving its performance. To do so, we trained an identical feature extractor for all data, and the extracted features were fed into corresponding Softmax classifiers. Spectral knowledge was introduced to ensure that the shared features were similar across domains. Four hyperspectral data sets were used in the experiments. We achieved higher classification accuracies on three data sets (Pavia University, Pavia Center, and Indian Pines) and competitive results on the Salinas Valley data compared with the baseline. Spectral knowledge was useful to prevent the deep network from overfitting when the data shared similar spectral response. The proposed method tested on two deep CNNs successfully shows its ability to utilize samples from multiple data sets and enhance networks’ performance.
Tasks Hyperspectral Image Classification, Image Classification
Published 2019-05-11
URL https://arxiv.org/abs/1905.04535v4
PDF https://arxiv.org/pdf/1905.04535v4.pdf
PWC https://paperswithcode.com/paper/multitask-deep-learning-with-spectral
Repo https://github.com/stop68/remote-sensing-image-classification
Framework tf

GSN: A Graph-Structured Network for Multi-Party Dialogues

Title GSN: A Graph-Structured Network for Multi-Party Dialogues
Authors Wenpeng Hu, Zhangming Chan, Bing Liu, Dongyan Zhao, Jinwen Ma, Rui Yan
Abstract Existing neural models for dialogue response generation assume that utterances are sequentially organized. However, many real-world dialogues involve multiple interlocutors (i.e., multi-party dialogues), where the assumption does not hold as utterances from different interlocutors can occur “in parallel.” This paper generalizes existing sequence-based models to a Graph-Structured neural Network (GSN) for dialogue modeling. The core of GSN is a graph-based encoder that can model the information flow along the graph-structured dialogues (two-party sequential dialogues are a special case). Experimental results show that GSN significantly outperforms existing sequence-based models.
Published 2019-05-31
URL https://arxiv.org/abs/1905.13637v1
PDF https://arxiv.org/pdf/1905.13637v1.pdf
PWC https://paperswithcode.com/paper/gsn-a-graph-structured-network-for-multi
Repo https://github.com/morning-dews/GSN-Dialogues
Framework tf

Optimizing Geometric Multigrid Methods with Evolutionary Computation

Title Optimizing Geometric Multigrid Methods with Evolutionary Computation
Authors Jonas Schmitt, Sebastian Kuckuk, Harald Köstler
Abstract For many linear and nonlinear systems that arise from the discretization of partial differential equations the construction of an efficient multigrid solver is a challenging task. Here we present a novel approach for the optimization of geometric multigrid methods that is based on evolutionary computation, a generic program optimization technique inspired by the principle of natural evolution. A multigrid solver is represented as a tree of mathematical expressions which we generate based on a tailored grammar. The quality of each solver is evaluated in terms of convergence and compute performance using automated local Fourier analysis (LFA) and roofline performance modeling, respectively. Based on these objectives a multi-objective optimization is performed using strongly typed genetic programming with a non-dominated sorting based selection. To evaluate the model-based prediction and to target concrete applications, scalable implementations of an evolved solver can be automatically generated with the ExaStencils framework. We demonstrate our approach by constructing multigrid solvers for the steady-state heat equation with constant and variable coefficients that consistently perform better than common V- and W-cycles.
Published 2019-10-07
URL https://arxiv.org/abs/1910.02749v2
PDF https://arxiv.org/pdf/1910.02749v2.pdf
PWC https://paperswithcode.com/paper/optimizing-geometric-multigrid-methods-with
Repo https://github.com/jonas-schmitt/evostencils
Framework none

An Adaptive Remote Stochastic Gradient Method for Training Neural Networks

Title An Adaptive Remote Stochastic Gradient Method for Training Neural Networks
Authors Yushu Chen, Hao Jing, Wenlai Zhao, Zhiqiang Liu, Ouyi Li, Liang Qiao, Wei Xue, Haohuan Fu, Guangwen Yang
Abstract We introduce ARSG, an adaptive first-order algorithm for training neural networks. The method computes the gradients at configurable remote observation points, in order to expedite the convergence by adjusting the step size for directions with different curvatures in the stochastic setting. It also scales the updating increment elementwise by a nonincreasing preconditioner to incorporate the avdantage of adaptive methods. The method is efficient in computation and memory, and is straightforward to implement. We analyze the convergence properties for both convex and nonconvex problems by modeling the training process as a dynamic system, that provides a guideline to select the configurable observation factor without grid search. A data-dependent regret bound is proposed to guarantee the convergence in the convex setting, which can be further improved to $O(log(T))$ for strongly convex functions. Numerical experiments demonstrate that ARSG converges faster than popular adaptive methods, such as ADAM, NADAM, AMSGRAD, and RANGER for the tested problems.
Published 2019-05-04
URL https://arxiv.org/abs/1905.01422v6
PDF https://arxiv.org/pdf/1905.01422v6.pdf
PWC https://paperswithcode.com/paper/namsg-an-efficient-method-for-training-neural
Repo https://github.com/rationalspark/NAMSG
Framework none

Using generative modelling to produce varied intonation for speech synthesis

Title Using generative modelling to produce varied intonation for speech synthesis
Authors Zack Hodari, Oliver Watts, Simon King
Abstract Unlike human speakers, typical text-to-speech (TTS) systems are unable to produce multiple distinct renditions of a given sentence. This has previously been addressed by adding explicit external control. In contrast, generative models are able to capture a distribution over multiple renditions and thus produce varied renditions using sampling. Typical neural TTS models learn the average of the data because they minimise mean squared error. In the context of prosody, taking the average produces flatter, more boring speech: an “average prosody”. A generative model that can synthesise multiple prosodies will, by design, not model average prosody. We use variational autoencoders (VAEs) which explicitly place the most “average” data close to the mean of the Gaussian prior. We propose that by moving towards the tails of the prior distribution, the model will transition towards generating more idiosyncratic, varied renditions. Focusing here on intonation, we investigate the trade-off between naturalness and intonation variation and find that typical acoustic models can either be natural, or varied, but not both. However, sampling from the tails of the VAE prior produces much more varied intonation than the traditional approaches, whilst maintaining the same level of naturalness.
Tasks Speech Synthesis
Published 2019-06-10
URL https://arxiv.org/abs/1906.04233v2
PDF https://arxiv.org/pdf/1906.04233v2.pdf
PWC https://paperswithcode.com/paper/using-generative-modelling-to-produce-varied
Repo https://github.com/ZackHodari/average_prosody
Framework pytorch

Bayesian Optimisation over Multiple Continuous and Categorical Inputs

Title Bayesian Optimisation over Multiple Continuous and Categorical Inputs
Authors Binxin Ru, Ahsan S. Alvi, Vu Nguyen, Michael A. Osborne, Stephen J Roberts
Abstract Efficient optimisation of black-box problems that comprise both continuous and categorical inputs is important, yet poses significant challenges. We propose a new approach, Continuous and Categorical Bayesian Optimisation (CoCaBO), which combines the strengths of multi-armed bandits and Bayesian optimisation to select values for both categorical and continuous inputs. We model this mixed-type space using a Gaussian Process kernel, designed to allow sharing of information across multiple categorical variables, each with multiple possible values; this allows CoCaBO to leverage all available data efficiently. We extend our method to the batch setting and propose an efficient selection procedure that dynamically balances exploration and exploitation whilst encouraging batch diversity. We demonstrate empirically that our method outperforms existing approaches on both synthetic and real-world optimisation tasks with continuous and categorical inputs.
Tasks Bayesian Optimisation, Multi-Armed Bandits
Published 2019-06-20
URL https://arxiv.org/abs/1906.08878v1
PDF https://arxiv.org/pdf/1906.08878v1.pdf
PWC https://paperswithcode.com/paper/bayesian-optimisation-over-multiple
Repo https://github.com/rubinxin/CoCaBO_code
Framework none

ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

Title ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
Authors Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, Haifeng Wang
Abstract Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.
Tasks Linguistic Acceptability, Multi-Task Learning, Natural Language Inference, Question Answering, Semantic Textual Similarity, Sentiment Analysis
Published 2019-07-29
URL https://arxiv.org/abs/1907.12412v2
PDF https://arxiv.org/pdf/1907.12412v2.pdf
PWC https://paperswithcode.com/paper/ernie-20-a-continual-pre-training-framework
Repo https://github.com/PaddlePaddle/ERNIE
Framework none

Adversarial Examples on Graph Data: Deep Insights into Attack and Defense

Title Adversarial Examples on Graph Data: Deep Insights into Attack and Defense
Authors Huijun Wu, Chen Wang, Yuriy Tyshetskiy, Andrew Docherty, Kai Lu, Liming Zhu
Abstract Graph deep learning models, such as graph convolutional networks (GCN) achieve remarkable performance for tasks on graph data. Similar to other types of deep models, graph deep learning models often suffer from adversarial attacks. However, compared with non-graph data, the discrete features, graph connections and different definitions of imperceptible perturbations bring unique challenges and opportunities for the adversarial attacks and defenses for graph data. In this paper, we propose both attack and defense techniques. For attack, we show that the discreteness problem could easily be resolved by introducing integrated gradients which could accurately reflect the effect of perturbing certain features or edges while still benefiting from the parallel computations. For defense, we observe that the adversarially manipulated graph for the targeted attack differs from normal graphs statistically. Based on this observation, we propose a defense approach which inspects the graph and recovers the potential adversarial perturbations. Our experiments on a number of datasets show the effectiveness of the proposed methods.
Tasks Adversarial Attack, Adversarial Defense
Published 2019-03-05
URL https://arxiv.org/abs/1903.01610v3
PDF https://arxiv.org/pdf/1903.01610v3.pdf
PWC https://paperswithcode.com/paper/the-vulnerabilities-of-graph-convolutional
Repo https://github.com/stellargraph/stellargraph
Framework tf
comments powered by Disqus