February 1, 2020

3129 words 15 mins read

Paper Group AWR 353

Efficient Representation Learning Using Random Walks for Dynamic Graphs. SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization. NorNE: Annotating Named Entities for Norwegian. SimVODIS: Simultaneous Visual Odometry, Object Detection, and Instance Segmentation. Learning from Multi …

Efficient Representation Learning Using Random Walks for Dynamic Graphs


Title	Efficient Representation Learning Using Random Walks for Dynamic Graphs
Authors	Hooman Peiro Sajjad, Andrew Docherty, Yuriy Tyshetskiy
Abstract	An important part of many machine learning workflows on graphs is vertex representation learning, i.e., learning a low-dimensional vector representation for each vertex in the graph. Recently, several powerful techniques for unsupervised representation learning have been demonstrated to give the state-of-the-art performance in downstream tasks such as vertex classification and edge prediction. These techniques rely on random walks performed on the graph in order to capture its structural properties. These structural properties are then encoded in the vector representation space. However, most contemporary representation learning methods only apply to static graphs while real-world graphs are often dynamic and change over time. Static representation learning methods are not able to update the vector representations when the graph changes; therefore, they must re-generate the vector representations on an updated static snapshot of the graph regardless of the extent of the change in the graph. In this work, we propose computationally efficient algorithms for vertex representation learning that extend random walk based methods to dynamic graphs. The computation complexity of our algorithms depends upon the extent and rate of changes (the number of edges changed per update) and on the density of the graph. We empirically evaluate our algorithms on real world datasets for downstream machine learning tasks of multi-class and multi-label vertex classification. The results show that our algorithms can achieve competitive results to the state-of-the-art methods while being computationally efficient.
Tasks	Representation Learning, Unsupervised Representation Learning
Published	2019-01-05
URL	http://arxiv.org/abs/1901.01346v2
PDF	http://arxiv.org/pdf/1901.01346v2.pdf
PWC	https://paperswithcode.com/paper/efficient-representation-learning-using
Repo	https://github.com/shps/incremental-representation-learning
Framework	tf

SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization


Title	SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
Authors	Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Tuo Zhao
Abstract	Transfer learning has fundamentally changed the landscape of natural language processing (NLP) research. Many existing state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model. To address the above issue in a more principled manner, we propose a new computational framework for robust and efficient fine-tuning for pre-trained language models. Specifically, our proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the capacity of the model; 2. Bregman proximal point optimization, which is a class of trust-region methods and can prevent knowledge forgetting. Our experiments demonstrate that our proposed method achieves the state-of-the-art performance on multiple NLP benchmarks.
Tasks	Transfer Learning
Published	2019-11-08
URL	https://arxiv.org/abs/1911.03437v1
PDF	https://arxiv.org/pdf/1911.03437v1.pdf
PWC	https://paperswithcode.com/paper/smart-robust-and-efficient-fine-tuning-for
Repo	https://github.com/namisan/mt-dnn
Framework	pytorch

NorNE: Annotating Named Entities for Norwegian


Title	NorNE: Annotating Named Entities for Norwegian
Authors	Fredrik Jørgensen, Tobias Aasmoe, Anne-Stine Ruud Husevåg, Lilja Øvrelid, Erik Velldal
Abstract	This paper presents NorNE, a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokm{\aa}l and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names. We here present details on the annotation effort, guidelines, inter-annotator agreement and an experimental analysis of the corpus using a neural sequence labeling architecture.
Tasks
Published	2019-11-27
URL	https://arxiv.org/abs/1911.12146v2
PDF	https://arxiv.org/pdf/1911.12146v2.pdf
PWC	https://paperswithcode.com/paper/norne-annotating-named-entities-for-norwegian
Repo	https://github.com/ltgoslo/norne
Framework	none

SimVODIS: Simultaneous Visual Odometry, Object Detection, and Instance Segmentation


Title	SimVODIS: Simultaneous Visual Odometry, Object Detection, and Instance Segmentation
Authors	Ue-Hwan Kim, Se-Ho Kim, Jong-Hwan Kim
Abstract	Intelligent agents need to understand the surrounding environment to provide meaningful services to or interact intelligently with humans. The agents should perceive geometric features as well as semantic entities inherent in the environment. Contemporary methods in general provide one type of information regarding the environment at a time, making it difficult to conduct high-level tasks. Moreover, running two types of methods and associating two resultant information requires a lot of computation and complicates the software architecture. To overcome these limitations, we propose a neural architecture that simultaneously performs both geometric and semantic tasks in a single thread: simultaneous visual odometry, object detection, and instance segmentation (SimVODIS). Training SimVODIS requires unlabeled video sequences and the photometric consistency between input image frames generates self-supervision signals. The performance of SimVODIS outperforms or matches the state-of-the-art performance in pose estimation, depth map prediction, object detection, and instance segmentation tasks while completing all the tasks in a single thread. We expect SimVODIS would enhance the autonomy of intelligent agents and let the agents provide effective services to humans.
Tasks	Instance Segmentation, Object Detection, Pose Estimation, Semantic Segmentation, Visual Odometry
Published	2019-11-14
URL	https://arxiv.org/abs/1911.05939v2
PDF	https://arxiv.org/pdf/1911.05939v2.pdf
PWC	https://paperswithcode.com/paper/simvodis-simultaneous-visual-odometry-object
Repo	https://github.com/Uehwan/SimVODIS
Framework	pytorch

Learning from Multiple Cities: A Meta-Learning Approach for Spatial-Temporal Prediction


Title	Learning from Multiple Cities: A Meta-Learning Approach for Spatial-Temporal Prediction
Authors	Huaxiu Yao, Yiding Liu, Ying Wei, Xianfeng Tang, Zhenhui Li
Abstract	Spatial-temporal prediction is a fundamental problem for constructing smart city, which is useful for tasks such as traffic control, taxi dispatching, and environmental policy making. Due to data collection mechanism, it is common to see data collection with unbalanced spatial distributions. For example, some cities may release taxi data for multiple years while others only release a few days of data; some regions may have constant water quality data monitored by sensors whereas some regions only have a small collection of water samples. In this paper, we tackle the problem of spatial-temporal prediction for the cities with only a short period of data collection. We aim to utilize the long-period data from other cities via transfer learning. Different from previous studies that transfer knowledge from one single source city to a target city, we are the first to leverage information from multiple cities to increase the stability of transfer. Specifically, our proposed model is designed as a spatial-temporal network with a meta-learning paradigm. The meta-learning paradigm learns a well-generalized initialization of the spatial-temporal network, which can be effectively adapted to target cities. In addition, a pattern-based spatial-temporal memory is designed to distill long-term temporal information (i.e., periodicity). We conduct extensive experiments on two tasks: traffic (taxi and bike) prediction and water quality prediction. The experiments demonstrate the effectiveness of our proposed model over several competitive baseline models.
Tasks	Meta-Learning, Transfer Learning
Published	2019-01-24
URL	http://arxiv.org/abs/1901.08518v2
PDF	http://arxiv.org/pdf/1901.08518v2.pdf
PWC	https://paperswithcode.com/paper/learning-from-multiple-cities-a-meta-learning
Repo	https://github.com/huaxiuyao/MetaST
Framework	tf

RadioTalk: a large-scale corpus of talk radio transcripts


Title	RadioTalk: a large-scale corpus of talk radio transcripts
Authors	Doug Beeferman, William Brannon, Deb Roy
Abstract	We introduce RadioTalk, a corpus of speech recognition transcripts sampled from talk radio broadcasts in the United States between October of 2018 and March of 2019. The corpus is intended for use by researchers in the fields of natural language processing, conversational analysis, and the social sciences. The corpus encompasses approximately 2.8 billion words of automatically transcribed speech from 284,000 hours of radio, together with metadata about the speech, such as geographical location, speaker turn boundaries, gender, and radio program information. In this paper we summarize why and how we prepared the corpus, give some descriptive statistics on stations, shows and speakers, and carry out a few high-level analyses.
Tasks	Speech Recognition
Published	2019-07-16
URL	https://arxiv.org/abs/1907.07073v1
PDF	https://arxiv.org/pdf/1907.07073v1.pdf
PWC	https://paperswithcode.com/paper/radiotalk-a-large-scale-corpus-of-talk-radio
Repo	https://github.com/social-machines/RadioTalk
Framework	none

Acoustic Model Adaptation from Raw Waveforms with SincNet


Title	Acoustic Model Adaptation from Raw Waveforms with SincNet
Authors	Joachim Fainberg, Ondřej Klejch, Erfan Loweimi, Peter Bell, Steve Renals
Abstract	Raw waveform acoustic modelling has recently gained interest due to neural networks’ ability to learn feature extraction, and the potential for finding better representations for a given scenario than hand-crafted features. SincNet has been proposed to reduce the number of parameters required in raw-waveform modelling, by restricting the filter functions, rather than having to learn every tap of each filter. We study the adaptation of the SincNet filter parameters from adults’ to children’s speech, and show that the parameterisation of the SincNet layer is well suited for adaptation in practice: we can efficiently adapt with a very small number of parameters, producing error rates comparable to techniques using orders of magnitude more parameters.
Tasks	Acoustic Modelling
Published	2019-09-30
URL	https://arxiv.org/abs/1909.13759v1
PDF	https://arxiv.org/pdf/1909.13759v1.pdf
PWC	https://paperswithcode.com/paper/acoustic-model-adaptation-from-raw-waveforms
Repo	https://github.com/jfainberg/sincnet_adapt
Framework	tf

Adaptive Continuous Visual Odometry from RGB-D Images


Title	Adaptive Continuous Visual Odometry from RGB-D Images
Authors	Tzu-Yuan Lin, William Clark, Ryan M. Eustice, Jessy W. Grizzle, Anthony Bloch, Maani Ghaffari
Abstract	In this paper, we extend the recently developed continuous visual odometry framework for RGB-D cameras to an adaptive framework via online hyperparameter learning. We focus on the case of isotropic kernels with a scalar as the length-scale. In practice and as expected, the length-scale has remarkable impacts on the performance of the original framework. Previously it was handled using a fixed set of conditions within the solver to reduce the length-scale as the algorithm reaches a local minimum. We automate this process by a greedy gradient descent step at each iteration to find the next-best length-scale. Furthermore, to handle failure cases in the gradient descent step where the gradient is not well-behaved, such as the absence of structure or texture in the scene, we use a search interval for the length-scale and guide it gradually toward the smaller values. This latter strategy reverts the adaptive framework to the original setup. The experimental evaluations using publicly available RGB-D benchmarks show the proposed adaptive continuous visual odometry outperforms the original framework and the current state-of-the-art. We also make the software for the developed algorithm publicly available.
Tasks	Visual Odometry
Published	2019-10-01
URL	https://arxiv.org/abs/1910.00713v1
PDF	https://arxiv.org/pdf/1910.00713v1.pdf
PWC	https://paperswithcode.com/paper/adaptive-continuous-visual-odometry-from-rgb
Repo	https://github.com/MaaniGhaffari/cvo-rgbd
Framework	none

Visual Odometry Revisited: What Should Be Learnt?


Title	Visual Odometry Revisited: What Should Be Learnt?
Authors	Huangying Zhan, Chamara Saroj Weerasekera, Jiawang Bian, Ian Reid
Abstract	In this work we present a monocular visual odometry (VO) algorithm which leverages geometry-based methods and deep learning. Most existing VO/SLAM systems with superior performance are based on geometry and have to be carefully designed for different application scenarios. Moreover, most monocular systems suffer from scale-drift issue.Some recent deep learning works learn VO in an end-to-end manner but the performance of these deep systems is still not comparable to geometry-based methods. In this work, we revisit the basics of VO and explore the right way for integrating deep learning with epipolar geometry and Perspective-n-Point (PnP) method. Specifically, we train two convolutional neural networks (CNNs) for estimating single-view depths and two-view optical flows as intermediate outputs. With the deep predictions, we design a simple but robust frame-to-frame VO algorithm (DF-VO) which outperforms pure deep learning-based and geometry-based methods. More importantly, our system does not suffer from the scale-drift issue being aided by a scale consistent single-view depth CNN. Extensive experiments on KITTI dataset shows the robustness of our system and a detailed ablation study shows the effect of different factors in our system.
Tasks	Monocular Visual Odometry, Visual Odometry
Published	2019-09-21
URL	https://arxiv.org/abs/1909.09803v4
PDF	https://arxiv.org/pdf/1909.09803v4.pdf
PWC	https://paperswithcode.com/paper/190909803
Repo	https://github.com/Huangying-Zhan/DF-VO
Framework	pytorch

Latent Replay for Real-Time Continual Learning


Title	Latent Replay for Real-Time Continual Learning
Authors	Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, Davide Maltoni
Abstract	Training deep neural networks at the edge on light computational devices, embedded systems and robotic platforms is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only embedded devices enabling remarkable levels of adaptiveness and autonomy. However, a number of practical problems need to be solved: catastrophic forgetting before anything else. In this paper we introduce an original technique named “Latent Replay” where, instead of storing a portion of past data in the input space, we store activations volumes at some intermediate layer. This can significantly reduce the computation and storage required by native rehearsal. To keep the representation stable and the stored activations valid we propose to slow-down learning at all the layers below the latent replay one, leaving the layers above free to learn at full pace. In our experiments we show that Latent Replay, combined with existing continual learning techniques, achieves state-of-the-art performance on complex video benchmarks such as CORe50 NICv2 (with nearly 400 small and highly non-i.i.d. batches) and OpenLORIS. Finally, we demonstrate the feasibility of nearly real-time continual learning on the edge through the deployment of the proposed technique on a smartphone device.
Tasks	Continual Learning
Published	2019-12-02
URL	https://arxiv.org/abs/1912.01100v2
PDF	https://arxiv.org/pdf/1912.01100v2.pdf
PWC	https://paperswithcode.com/paper/latent-replay-for-real-time-continual
Repo	https://github.com/lrzpellegrini/CL-CORe-App
Framework	none

Continual Learning via Neural Pruning


Title	Continual Learning via Neural Pruning
Authors	Siavash Golkar, Michael Kagan, Kyunghyun Cho
Abstract	We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performance of previous tasks. In order to deal with the possible compromise between model sparsity and performance, we formalize and incorporate the concept of graceful forgetting: the idea that it is preferable to suffer a small amount of forgetting in a controlled manner if it helps regain network capacity and prevents uncontrolled loss of performance during the training of future tasks. CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks as well as the number of neurons that are being reused. In particular, we see in experiments that CLNP verifies and automatically takes advantage of the fact that the features of earlier layers are more transferable. We show empirically that CLNP leads to significantly improved results over current weight elasticity based methods.
Tasks	Continual Learning
Published	2019-03-11
URL	http://arxiv.org/abs/1903.04476v1
PDF	http://arxiv.org/pdf/1903.04476v1.pdf
PWC	https://paperswithcode.com/paper/continual-learning-via-neural-pruning
Repo	https://github.com/DrTaDa/AGI_and_intelligence_database
Framework	none

Towards Real-Time Automatic Portrait Matting on Mobile Devices


Title	Towards Real-Time Automatic Portrait Matting on Mobile Devices
Authors	Seokjun Seo, Seungwoo Choi, Martin Kersner, Beomjun Shin, Hyungsuk Yoon, Hyeongmin Byun, Sungjoo Ha
Abstract	We tackle the problem of automatic portrait matting on mobile devices. The proposed model is aimed at attaining real-time inference on mobile devices with minimal degradation of model performance. Our model MMNet, based on multi-branch dilated convolution with linear bottleneck blocks, outperforms the state-of-the-art model and is orders of magnitude faster. The model can be accelerated four times to attain 30 FPS on Xiaomi Mi 5 device with moderate increase in the gradient error. Under the same conditions, our model has an order of magnitude less number of parameters and is faster than Mobile DeepLabv3 while maintaining comparable performance. The accompanied implementation can be found at \url{https://github.com/hyperconnect/MMNet}.
Tasks	Image Matting
Published	2019-04-08
URL	http://arxiv.org/abs/1904.03816v1
PDF	http://arxiv.org/pdf/1904.03816v1.pdf
PWC	https://paperswithcode.com/paper/towards-real-time-automatic-portrait-matting
Repo	https://github.com/hyperconnect/MMNet
Framework	tf

Geometrical Regret Matching


Title	Geometrical Regret Matching
Authors	Sizhong Lan
Abstract	We argue that the existing regret matchings for Nash equilibrium approximation conduct “jumpy” strategy updating when the probabilities of future plays are set to be proportional to positive regret measures. We propose a geometrical regret matching which features “smooth” strategy updating. Our approach is simple, intuitive and natural. The analytical and numerical results show that, continuously and “smoothly” suppressing “unprofitable” pure strategies is sufficient for the game to evolve towards Nash equilibrium, suggesting that in reality the tendency for equilibrium could be pervasive and irresistible. Technically, iterative regret matching gives rise to a sequence of adjusted mixed strategies for our study its approximation to the true equilibrium point. The sequence can be studied in metric space and visualized nicely as a clear path towards an equilibrium point. Our theory has limitations in optimizing the approximation accuracy.
Tasks
Published	2019-08-18
URL	https://arxiv.org/abs/1908.09021v8
PDF	https://arxiv.org/pdf/1908.09021v8.pdf
PWC	https://paperswithcode.com/paper/the-path-to-nash-equilibrium
Repo	https://github.com/lansiz/eqpt
Framework	none

A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots


Title	A Survey on Reproducibility by Evaluating Deep Reinforcement Learning Algorithms on Real-World Robots
Authors	Nicolai A. Lynnerup, Laura Nolling, Rasmus Hasle, John Hallam
Abstract	As reinforcement learning (RL) achieves more success in solving complex tasks, more care is needed to ensure that RL research is reproducible and that algorithms herein can be compared easily and fairly with minimal bias. RL results are, however, notoriously hard to reproduce due to the algorithms’ intrinsic variance, the environments’ stochasticity, and numerous (potentially unreported) hyper-parameters. In this work we investigate the many issues leading to irreproducible research and how to manage those. We further show how to utilise a rigorous and standardised evaluation approach for easing the process of documentation, evaluation and fair comparison of different algorithms, where we emphasise the importance of choosing the right measurement metrics and conducting proper statistics on the results, for unbiased reporting of the results.
Tasks
Published	2019-09-09
URL	https://arxiv.org/abs/1909.03772v2
PDF	https://arxiv.org/pdf/1909.03772v2.pdf
PWC	https://paperswithcode.com/paper/a-survey-on-reproducibility-by-evaluating
Repo	https://github.com/dti-research/SenseActExperiments
Framework	none

Region of Attraction for Power Systems using Gaussian Process and Converse Lyapunov Function – Part I: Theoretical Framework and Off-line Study


Title	Region of Attraction for Power Systems using Gaussian Process and Converse Lyapunov Function – Part I: Theoretical Framework and Off-line Study
Authors	Chao Zhai, Hung D. Nguyen
Abstract	This paper introduces a novel framework to construct the region of attraction (ROA) of a power system centered around a stable equilibrium by using stable state trajectories of system dynamics. Most existing works on estimating ROA rely on analytical Lyapunov functions, which are subject to two limitations: the analytic Lyapunov functions may not be always readily available, and the resulting ROA may be overly conservative. This work overcomes these two limitations by leveraging the converse Lyapunov theorem in control theory to eliminate the need of an analytic Lyapunov function and learning the unknown Lyapunov function with the Gaussian Process (GP) approach. In addition, a Gaussian Process Upper Confidence Bound (GP-UCB) based sampling algorithm is designed to reconcile the trade-off between the exploitation for enlarging the ROA and the exploration for reducing the uncertainty of sampling region. Within the constructed ROA, it is guaranteed in probability that the system state will converge to the stable equilibrium with a confidence level. Numerical simulations are also conducted to validate the assessment approach for the ROA of the single machine infinite bus system and the New England $39$-bus system. Numerical results demonstrate that our approach can significantly enlarge the estimated ROA compared to that of the analytic Lyapunov counterpart.
Tasks
Published	2019-06-09
URL	https://arxiv.org/abs/1906.03590v1
PDF	https://arxiv.org/pdf/1906.03590v1.pdf
PWC	https://paperswithcode.com/paper/region-of-attraction-for-power-systems-using
Repo	https://github.com/Chaocas/ROA-for-Power-Systems
Framework	none