January 25, 2020

3226 words 16 mins read

Paper Group ANR 1738

Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks. Task-Assisted Domain Adaptation with Anchor Tasks. Robustness Guarantees for Deep Neural Networks on Videos. Learning Image-Specific Attributes by Hyperbolic Neighborhood Graph Propagation. Leveraging exploration in off-policy algorithms via normalizing flows. The Alg …

Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks


Title	Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks
Authors	Charbel Sakr, Naigang Wang, Chia-Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, Kailash Gopalakrishnan
Abstract	Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence. The absence of any framework to analyze the precision requirements of partial sum accumulations results in conservative design choices. This imposes an upper-bound on the reduction of complexity of multiply-accumulate units. We present a statistical approach to analyze the impact of reduced accumulation precision on deep learning training. Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation. We apply our analysis to three benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet AlexNet. In each case, with accumulation precision set in accordance with our proposed equations, the networks successfully converge to the single precision floating-point baseline. We also show that reducing accumulation precision further degrades the quality of the trained network, proving that our equations produce tight bounds. Overall this analysis enables precise tailoring of computation hardware to the application, yielding area- and power-optimal systems.
Tasks
Published	2019-01-19
URL	http://arxiv.org/abs/1901.06588v1
PDF	http://arxiv.org/pdf/1901.06588v1.pdf
PWC	https://paperswithcode.com/paper/accumulation-bit-width-scaling-for-ultra-low
Repo
Framework

Task-Assisted Domain Adaptation with Anchor Tasks


Title	Task-Assisted Domain Adaptation with Anchor Tasks
Authors	Zhizhong Li, Linjie Luo, Sergey Tulyakov, Qieyun Dai, Derek Hoiem
Abstract	Some tasks, such as surface normals or single-view depth estimation, require per-pixel ground truth that is difficult to obtain on real images but easy to obtain on synthetic. However, models learned on synthetic images often do not generalize well to real images due to the domain shift. Our key idea to improve domain adaptation is to introduce a separate anchor task (such as facial landmarks) whose annotations can be obtained at no cost or are already available on both synthetic and real datasets. To further leverage the implicit relationship between the anchor and main tasks, we apply our \freeze technique that learns the cross-task guidance on the source domain with the final network layers, and use it on the target domain. We evaluate our methods on surface normal estimation on two pairs of datasets (indoor scenes and faces) with two kinds of anchor tasks (semantic segmentation and facial landmarks). We show that blindly applying domain adaptation or training the auxiliary task on only one domain may hurt performance, while using anchor tasks on both domains is better behaved. Our \freeze technique outperforms competing approaches, reaching performance in facial images on par with a recently popular surface normal estimation method using shape from shading domain knowledge.
Tasks	Depth Estimation, Domain Adaptation, Semantic Segmentation
Published	2019-08-16
URL	https://arxiv.org/abs/1908.06079v2
PDF	https://arxiv.org/pdf/1908.06079v2.pdf
PWC	https://paperswithcode.com/paper/anchor-tasks-inexpensive-shared-and-aligned
Repo
Framework

Robustness Guarantees for Deep Neural Networks on Videos


Title	Robustness Guarantees for Deep Neural Networks on Videos
Authors	Min Wu, Marta Kwiatkowska
Abstract	The widespread adoption of deep learning models places demands on their robustness. In this paper, we consider the robustness of deep neural networks on videos, which comprise both the spatial features of individual frames extracted by a convolutional neural network and the temporal dynamics between adjacent frames captured by a recurrent neural network. To measure robustness, we study the maximum safe radius problem, which computes the minimum distance from the optical flow set obtained from a given input to that of an adversarial example in the norm ball. We demonstrate that, under the assumption of Lipschitz continuity, the problem can be approximated using finite optimisation via discretising the optical flow space, and the approximation has provable guarantees. We then show that the finite optimisation problem can be solved by utilising a two-player turn-based game in a cooperative setting, where the first player selects the optical flows and the second player determines the dimensions to be manipulated in the chosen flow. We employ an anytime approach to solve the game, in the sense of approximating the value of the game by monotonically improving its upper and lower bounds. We exploit a gradient-based search algorithm to compute the upper bounds, and the admissible A* algorithm to update the lower bounds. Finally, we evaluate our framework on the UCF101 video dataset.
Tasks	Optical Flow Estimation
Published	2019-06-28
URL	https://arxiv.org/abs/1907.00098v2
PDF	https://arxiv.org/pdf/1907.00098v2.pdf
PWC	https://paperswithcode.com/paper/robustness-guarantees-for-deep-neural
Repo
Framework

Learning Image-Specific Attributes by Hyperbolic Neighborhood Graph Propagation


Title	Learning Image-Specific Attributes by Hyperbolic Neighborhood Graph Propagation
Authors	Xiaofeng Xu, Ivor W. Tsang, Xiaofeng Cao, Ruiheng Zhang, Chuancai Liu
Abstract	As a kind of semantic representation of visual object descriptions, attributes are widely used in various computer vision tasks. In most of existing attribute-based research, class-specific attributes (CSA), which are class-level annotations, are usually adopted due to its low annotation cost for each class instead of each individual image. However, class-specific attributes are usually noisy because of annotation errors and diversity of individual images. Therefore, it is desirable to obtain image-specific attributes (ISA), which are image-level annotations, from the original class-specific attributes. In this paper, we propose to learn image-specific attributes by graph-based attribute propagation. Considering the intrinsic property of hyperbolic geometry that its distance expands exponentially, hyperbolic neighborhood graph (HNG) is constructed to characterize the relationship between samples. Based on HNG, we define neighborhood consistency for each sample to identify inconsistent samples. Subsequently, inconsistent samples are refined based on their neighbors in HNG. Extensive experiments on five benchmark datasets demonstrate the significant superiority of the learned image-specific attributes over the original class-specific attributes in the zero-shot object classification task.
Tasks	Object Classification
Published	2019-05-20
URL	https://arxiv.org/abs/1905.07933v2
PDF	https://arxiv.org/pdf/1905.07933v2.pdf
PWC	https://paperswithcode.com/paper/learning-image-specific-attributes-by
Repo
Framework

Leveraging exploration in off-policy algorithms via normalizing flows


Title	Leveraging exploration in off-policy algorithms via normalizing flows
Authors	Bogdan Mazoure, Thang Doan, Audrey Durand, R Devon Hjelm, Joelle Pineau
Abstract	The ability to discover approximately optimal policies in domains with sparse rewards is crucial to applying reinforcement learning (RL) in many real-world scenarios. Approaches such as neural density models and continuous exploration (e.g., Go-Explore) have been proposed to maintain the high exploration rate necessary to find high performing and generalizable policies. Soft actor-critic(SAC) is another method for improving exploration that aims to combine efficient learning via off-policy updates while maximizing the policy entropy. In this work, we extend SAC to a richer class of probability distributions (e.g., multimodal) through normalizing flows (NF) and show that this significantly improves performance by accelerating the discovery of good policies while using much smaller policy representations. Our approach, which we call SAC-NF, is a simple, efficient,easy-to-implement modification and improvement to SAC on continuous control baselines such as MuJoCo and PyBullet Roboschool domains. Finally, SAC-NF does this while being significantly parameter efficient, using as few as 5.5% the parameters for an equivalent SAC model.
Tasks	Continuous Control
Published	2019-05-16
URL	https://arxiv.org/abs/1905.06893v3
PDF	https://arxiv.org/pdf/1905.06893v3.pdf
PWC	https://paperswithcode.com/paper/leveraging-exploration-in-off-policy
Repo
Framework

The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence


Title	The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence
Authors	Radoslaw Martin Cichy, Gemma Roig, Alex Andonian, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Yalda Mohsenzadeh, Kandan Ramakrishnan, Aude Oliva
Abstract	In the last decade, artificial intelligence (AI) models inspired by the brain have made unprecedented progress in performing real-world perceptual tasks like object classification and speech recognition. Recently, researchers of natural intelligence have begun using those AI models to explore how the brain performs such tasks. These developments suggest that future progress will benefit from increased interaction between disciplines. Here we introduce the Algonauts Project as a structured and quantitative communication channel for interdisciplinary interaction between natural and artificial intelligence researchers. The project’s core is an open challenge with a quantitative benchmark whose goal is to account for brain data through computational models. This project has the potential to provide better models of natural intelligence and to gather findings that advance AI. The 2019 Algonauts Project focuses on benchmarking computational models predicting human brain activity when people look at pictures of objects. The 2019 edition of the Algonauts Project is available online: http://algonauts.csail.mit.edu/.
Tasks	Object Classification, Speech Recognition
Published	2019-05-14
URL	https://arxiv.org/abs/1905.05675v1
PDF	https://arxiv.org/pdf/1905.05675v1.pdf
PWC	https://paperswithcode.com/paper/the-algonauts-project-a-platform-for
Repo
Framework

Optical Transient Object Classification in Wide Field Small Aperture Telescopes with Neural Networks


Title	Optical Transient Object Classification in Wide Field Small Aperture Telescopes with Neural Networks
Authors	Peng Jia, Yifei Zhao, Gang Xue, Dongmei Cai
Abstract	Wide field small aperture telescopes are working horses for fast sky surveying. Transient discovery is one of their main tasks. Classification of candidate transient images between real sources and artifacts with high accuracy is an important step for transient discovery. In this paper, we propose two transient classification methods based on neural networks. The first method uses the convolutional neural network without pooling layers to classify transient images with low sampling rate. The second method assumes transient images as one dimensional signals and is based on recurrent neural networks with long short term memory and leaky ReLu activation function in each detection layer. Testing with real observation data, we find that although these two methods can both achieve more than 94% classification accuracy, they have different classification properties for different targets. Based on this result, we propose to use the ensemble learning method to further increase the classification accuracy to more than 97%.
Tasks	Object Classification
Published	2019-04-29
URL	http://arxiv.org/abs/1904.12987v1
PDF	http://arxiv.org/pdf/1904.12987v1.pdf
PWC	https://paperswithcode.com/paper/optical-transient-object-classification-in
Repo
Framework

Model inference for Ordinary Differential Equations by parametric polynomial kernel regression


Title	Model inference for Ordinary Differential Equations by parametric polynomial kernel regression
Authors	David K. E. Green, Filip Rindler
Abstract	Model inference for dynamical systems aims to estimate the future behaviour of a system from observations. Purely model-free statistical methods, such as Artificial Neural Networks, tend to perform poorly for such tasks. They are therefore not well suited to many questions from applications, for example in Bayesian filtering and reliability estimation. This work introduces a parametric polynomial kernel method that can be used for inferring the future behaviour of Ordinary Differential Equation models, including chaotic dynamical systems, from observations. Using numerical integration techniques, parametric representations of Ordinary Differential Equations can be learnt using Backpropagation and Stochastic Gradient Descent. The polynomial technique presented here is based on a nonparametric method, kernel ridge regression. However, the time complexity of nonparametric kernel ridge regression scales cubically with the number of training data points. Our parametric polynomial method avoids this manifestation of the curse of dimensionality, which becomes particularly relevant when working with large time series data sets. Two numerical demonstrations are presented. First, a simple regression test case is used to illustrate the method and to compare the performance with standard Artificial Neural Network techniques. Second, a more substantial test case is the inference of a chaotic spatio-temporal dynamical system, the Lorenz–Emanuel system, from observations. Our method was able to successfully track the future behaviour of the system over time periods much larger than the training data sampling rate. Finally, some limitations of the method are presented, as well as proposed directions for future work to mitigate these limitations.
Tasks	Time Series
Published	2019-08-06
URL	https://arxiv.org/abs/1908.02105v1
PDF	https://arxiv.org/pdf/1908.02105v1.pdf
PWC	https://paperswithcode.com/paper/model-inference-for-ordinary-differential
Repo
Framework

Interpreting Adversarial Examples with Attributes


Title	Interpreting Adversarial Examples with Attributes
Authors	Sadaf Gulshad, Jan Hendrik Metzen, Arnold Smeulders, Zeynep Akata
Abstract	Deep computer vision systems being vulnerable to imperceptible and carefully crafted noise have raised questions regarding the robustness of their decisions. We take a step back and approach this problem from an orthogonal direction. We propose to enable black-box neural networks to justify their reasoning both for clean and for adversarial examples by leveraging attributes, i.e. visually discriminative properties of objects. We rank attributes based on their class relevance, i.e. how the classification decision changes when the input is visually slightly perturbed, as well as image relevance, i.e. how well the attributes can be localized on both clean and perturbed images. We present comprehensive experiments for attribute prediction, adversarial example generation, adversarially robust learning, and their qualitative and quantitative analysis using predicted attributes on three benchmark datasets.
Tasks
Published	2019-04-17
URL	http://arxiv.org/abs/1904.08279v1
PDF	http://arxiv.org/pdf/1904.08279v1.pdf
PWC	https://paperswithcode.com/paper/interpreting-adversarial-examples-with
Repo
Framework

Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems


Title	Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems
Authors	Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, Dawei Yin
Abstract	Recommender systems play a crucial role in our daily lives. Feed streaming mechanism has been widely used in the recommender system, especially on the mobile Apps. The feed streaming setting provides users the interactive manner of recommendation in never-ending feeds. In such an interactive manner, a good recommender system should pay more attention to user stickiness, which is far beyond classical instant metrics, and typically measured by {\bf long-term user engagement}. Directly optimizing the long-term user engagement is a non-trivial problem, as the learning target is usually not available for conventional supervised learning methods. Though reinforcement learning~(RL) naturally fits the problem of maximizing the long term rewards, applying RL to optimize long-term user engagement is still facing challenges: user behaviors are versatile and difficult to model, which typically consists of both instant feedback~(e.g. clicks, ordering) and delayed feedback~(e.g. dwell time, revisit); in addition, performing effective off-policy learning is still immature, especially when combining bootstrapping and function approximation. To address these issues, in this work, we introduce a reinforcement learning framework — FeedRec to optimize the long-term user engagement. FeedRec includes two components: 1)~a Q-Network which designed in hierarchical LSTM takes charge of modeling complex user behaviors, and 2)~an S-Network, which simulates the environment, assists the Q-Network and voids the instability of convergence in policy learning. Extensive experiments on synthetic data and a real-world large scale data show that FeedRec effectively optimizes the long-term user engagement and outperforms state-of-the-arts.
Tasks	Recommendation Systems
Published	2019-02-13
URL	https://arxiv.org/abs/1902.05570v4
PDF	https://arxiv.org/pdf/1902.05570v4.pdf
PWC	https://paperswithcode.com/paper/reinforcement-learning-to-optimize-long-term
Repo
Framework

UNAS: Differentiable Architecture Search Meets Reinforcement Learning


Title	UNAS: Differentiable Architecture Search Meets Reinforcement Learning
Authors	Arash Vahdat, Arun Mallya, Ming-Yu Liu, Jan Kautz
Abstract	Neural architecture search (NAS) aims to discover network architectures with desired properties such as high accuracy or low latency. Recently, differentiable NAS (DNAS) has demonstrated promising results while maintaining a search cost orders of magnitude lower than reinforcement learning (RL) based NAS. However, DNAS models can only optimize differentiable loss functions in search, and they require an accurate differentiable approximation of non-differentiable criteria. In this work, we present UNAS, a unified framework for NAS, that encapsulates recent DNAS and RL-based approaches under one framework. Our framework brings the best of both worlds, and it enables us to search for architectures with both differentiable and non-differentiable criteria in one unified framework while maintaining a low search cost. Further, we introduce a new objective function for search based on the generalization gap that prevents the selection of architectures prone to overfitting. We present extensive experiments on the CIFAR-10, CIFAR-100 and ImageNet datasets and we perform search in two fundamentally different search spaces. We show that UNAS obtains the state-of-the-art average accuracy on all three datasets when compared to the architectures searched in the DARTS space. Moreover, we show that UNAS can find an efficient and accurate architecture in the ProxylessNAS search space, that outperforms existing MobileNetV2 based architectures.
Tasks	Neural Architecture Search
Published	2019-12-16
URL	https://arxiv.org/abs/1912.07651v1
PDF	https://arxiv.org/pdf/1912.07651v1.pdf
PWC	https://paperswithcode.com/paper/unas-differentiable-architecture-search-meets
Repo
Framework

DAPAS : Denoising Autoencoder to Prevent Adversarial attack in Semantic Segmentation


Title	DAPAS : Denoising Autoencoder to Prevent Adversarial attack in Semantic Segmentation
Authors	Seung Ju Cho, Tae Joon Jun, Byungsoo Oh, Daeyoung Kim
Abstract	Nowadays, Deep learning techniques show dramatic performance on computer vision area, and they even outperform human. But it is also vulnerable to some small perturbation called an adversarial attack. This is a problem combined with the safety of artificial intelligence, which has recently been studied a lot. These attacks have shown that they can fool models of image classification, semantic segmentation, and object detection. We point out this attack can be protected by denoise autoencoder, which is used for denoising the perturbation and restoring the original images. We experiment with various noise distributions and verify the effect of denoise autoencoder against adversarial attack in semantic segmentation.
Tasks	Adversarial Attack, Denoising, Image Classification, Object Detection, Semantic Segmentation
Published	2019-08-14
URL	https://arxiv.org/abs/1908.05195v2
PDF	https://arxiv.org/pdf/1908.05195v2.pdf
PWC	https://paperswithcode.com/paper/dapas-denoising-autoencoder-to-prevent
Repo
Framework

Personalization of Deep Learning


Title	Personalization of Deep Learning
Authors	Johannes Schneider, Michail Vlachos
Abstract	We discuss training techniques, objectives and metrics toward personalization of deep learning models. In machine learning, personalization addresses the goal of a trained model to target a particular individual by optimizing one or more performance metrics, while conforming to certain constraints. To personalize, we investigate three methods of `curriculum learning` and two approaches for data grouping, i.e., augmenting the data of an individual by adding similar data identified with an auto-encoder. We show that both `curriculuum learning'' and` personalized’’ data augmentation lead to improved performance on data of an individual. Mostly, this comes at the cost of reduced performance on a more general, broader dataset.
Tasks	Data Augmentation
Published	2019-09-06
URL	https://arxiv.org/abs/1909.02803v3
PDF	https://arxiv.org/pdf/1909.02803v3.pdf
PWC	https://paperswithcode.com/paper/mass-personalization-of-deep-learning
Repo
Framework


Title	Towards NLP with Deep Learning: Convolutional Neural Networks and Recurrent Neural Networks for Offensive Language Identification in Social Media
Authors	Andrei-Bogdan Puiu, Andrei-Octavian Brabete
Abstract	This short paper presents the design decisions taken and challenges encountered in completing SemEval Task 6, which poses the problem of identifying and categorizing offensive language in tweets. Our proposed solutions explore Deep Learning techniques, Linear Support Vector classification and Random Forests to identify offensive tweets, to classify offenses as targeted or untargeted and eventually to identify the target subject type.
Tasks	Language Identification
Published	2019-03-02
URL	http://arxiv.org/abs/1903.00665v2
PDF	http://arxiv.org/pdf/1903.00665v2.pdf
PWC	https://paperswithcode.com/paper/semeval-2019-task-6-identifying-and
Repo
Framework

Absit invidia verbo: Comparing Deep Learning methods for offensive language


Title	Absit invidia verbo: Comparing Deep Learning methods for offensive language
Authors	Silvia Sapora, Bogdan Lazarescu, Christo Lolov
Abstract	This document describes our approach to building an Offensive Language Classifier. More specifically, the OffensEval 2019 competition required us to build three classifiers with slightly different goals: - Offensive language identification: would classify a tweet as offensive or not. - Automatic categorization of offense types: would recognize if the target of the offense was an individual or not. - Offense target identification: would identify the target of the offense between an individual, group or other. In this report, we will discuss the different architectures, algorithms and pre-processing strategies we tried, together with a detailed description of the designs of our final classifiers and the reasons we choose them over others. We evaluated our classifiers on the official test set provided for the OffenseEval 2019 competition, obtaining a macro-averaged F1-score of 0.7189 for Task A, 0.6708 on Task B and 0.5442 on Task C.
Tasks	Language Identification
Published	2019-03-14
URL	http://arxiv.org/abs/1903.05929v3
PDF	http://arxiv.org/pdf/1903.05929v3.pdf
PWC	https://paperswithcode.com/paper/offenseval-at-semeval-2018-task-6-identifying
Repo
Framework