October 21, 2019

3419 words 17 mins read

Paper Group AWR 44

Learning to Gather without Communication. Towards Robust Lung Segmentation in Chest Radiographs with Deep Learning. Extrofitting: Enriching Word Representation and its Vector Space with Semantic Lexicons. Location Dependency in Video Prediction. Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advan …

Learning to Gather without Communication


Title	Learning to Gather without Communication
Authors	El Mahdi El Mhamdi, Rachid Guerraoui, Alexandre Maurer, Vladislav Tempez
Abstract	A standard belief on emerging collective behavior is that it emerges from simple individual rules. Most of the mathematical research on such collective behavior starts from imperative individual rules, like always go to the center. But how could an (optimal) individual rule emerge during a short period within the group lifetime, especially if communication is not available. We argue that such rules can actually emerge in a group in a short span of time via collective (multi-agent) reinforcement learning, i.e learning via rewards and punishments. We consider the gathering problem: several agents (social animals, swarming robots…) must gather around a same position, which is not determined in advance. They must do so without communication on their planned decision, just by looking at the position of other agents. We present the first experimental evidence that a gathering behavior can be learned without communication in a partially observable environment. The learned behavior has the same properties as a self-stabilizing distributed algorithm, as processes can gather from any initial state (and thus tolerate any transient failure). Besides, we show that it is possible to tolerate the brutal loss of up to 90% of agents without significant impact on the behavior.
Tasks	Multi-agent Reinforcement Learning
Published	2018-02-21
URL	http://arxiv.org/abs/1802.07834v1
PDF	http://arxiv.org/pdf/1802.07834v1.pdf
PWC	https://paperswithcode.com/paper/learning-to-gather-without-communication
Repo	https://github.com/LearningToGatherWithoutCommunication/Ring
Framework	none

Towards Robust Lung Segmentation in Chest Radiographs with Deep Learning


Title	Towards Robust Lung Segmentation in Chest Radiographs with Deep Learning
Authors	Jyoti Islam, Yanqing Zhang
Abstract	Automated segmentation of Lungs plays a crucial role in the computer-aided diagnosis of chest X-Ray (CXR) images. Developing an efficient Lung segmentation model is challenging because of difficulties such as the presence of several edges at the rib cage and clavicle, inconsistent lung shape among different individuals, and the appearance of the lung apex. In this paper, we propose a robust model for Lung segmentation in Chest Radiographs. Our model learns to ignore the irrelevant regions in an input Chest Radiograph while highlighting regions useful for lung segmentation. The proposed model is evaluated on two public chest X-Ray datasets (Montgomery County, MD, USA, and Shenzhen No. 3 People’s Hospital in China). The experimental result with a DICE score of 98.6% demonstrates the robustness of our proposed lung segmentation approach.
Tasks
Published	2018-11-30
URL	http://arxiv.org/abs/1811.12638v1
PDF	http://arxiv.org/pdf/1811.12638v1.pdf
PWC	https://paperswithcode.com/paper/towards-robust-lung-segmentation-in-chest
Repo	https://github.com/IlliaOvcharenko/lung-segmentation
Framework	pytorch

Extrofitting: Enriching Word Representation and its Vector Space with Semantic Lexicons


Title	Extrofitting: Enriching Word Representation and its Vector Space with Semantic Lexicons
Authors	Hwiyeol Jo, Stanley Jungkyu Choi
Abstract	We propose post-processing method for enriching not only word representation but also its vector space using semantic lexicons, which we call extrofitting. The method consists of 3 steps as follows: (i) Expanding 1 or more dimension(s) on all the word vectors, filling with their representative value. (ii) Transferring semantic knowledge by averaging each representative values of synonyms and filling them in the expanded dimension(s). These two steps make representations of the synonyms close together. (iii) Projecting the vector space using Linear Discriminant Analysis, which eliminates the expanded dimension(s) with semantic knowledge. When experimenting with GloVe, we find that our method outperforms Faruqui’s retrofitting on some of word similarity task. We also report further analysis on our method in respect to word vector dimensions, vocabulary size as well as other well-known pretrained word vectors (e.g., Word2Vec, Fasttext).
Tasks
Published	2018-04-21
URL	http://arxiv.org/abs/1804.07946v2
PDF	http://arxiv.org/pdf/1804.07946v2.pdf
PWC	https://paperswithcode.com/paper/extrofitting-enriching-word-representation
Repo	https://github.com/HwiyeolJo/Extrofitting
Framework	none

Location Dependency in Video Prediction


Title	Location Dependency in Video Prediction
Authors	Niloofar Azizi, Hafez Farazi, Sven Behnke
Abstract	Deep convolutional neural networks are used to address many computer vision problems, including video prediction. The task of video prediction requires analyzing the video frames, temporally and spatially, and constructing a model of how the environment evolves. Convolutional neural networks are spatially invariant, though, which prevents them from modeling location-dependent patterns. In this work, the authors propose location-biased convolutional layers to overcome this limitation. The effectiveness of location bias is evaluated on two architectures: Video Ladder Network (VLN) and Convolutional redictive Gating Pyramid (Conv-PGP). The results indicate that encoding location-dependent features is crucial for the task of video prediction. Our proposed methods significantly outperform spatially invariant models.
Tasks	Video Prediction
Published	2018-10-11
URL	http://arxiv.org/abs/1810.04937v2
PDF	http://arxiv.org/pdf/1810.04937v2.pdf
PWC	https://paperswithcode.com/paper/location-dependency-in-video-prediction
Repo	https://github.com/AIS-Bonn/LocDepVideoPrediction
Framework	none

Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm


Title	Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
Authors	Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, Kwang-Ting Cheng
Abstract	In this work, we study the 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary. While being efficient, the classification accuracy of the current 1-bit CNNs is much worse compared to their counterpart real-valued CNN models on the large-scale dataset, like ImageNet. To minimize the performance gap between the 1-bit and real-valued CNN models, we propose a novel model, dubbed Bi-Real net, which connects the real activations (after the 1-bit convolution and/or BatchNorm layer, before the sign function) to activations of the consecutive block, through an identity shortcut. Consequently, compared to the standard 1-bit CNN, the representational capability of the Bi-Real net is significantly enhanced and the additional cost on computation is negligible. Moreover, we develop a specific training algorithm including three technical novelties for 1- bit CNNs. Firstly, we derive a tight approximation to the derivative of the non-differentiable sign function with respect to activation. Secondly, we propose a magnitude-aware gradient with respect to the weight for updating the weight parameters. Thirdly, we pre-train the real-valued CNN model with a clip function, rather than the ReLU function, to better initialize the Bi-Real net. Experiments on ImageNet show that the Bi-Real net with the proposed training algorithm achieves 56.4% and 62.2% top-1 accuracy with 18 layers and 34 layers, respectively. Compared to the state-of-the-arts (e.g., XNOR Net), Bi-Real net achieves up to 10% higher top-1 accuracy with more memory saving and lower computational cost. Keywords: binary neural network, 1-bit CNNs, 1-layer-per-block
Tasks
Published	2018-08-01
URL	http://arxiv.org/abs/1808.00278v5
PDF	http://arxiv.org/pdf/1808.00278v5.pdf
PWC	https://paperswithcode.com/paper/bi-real-net-enhancing-the-performance-of-1
Repo	https://github.com/liuzechun/Bi-Real-net
Framework	pytorch

Merging and Evolution: Improving Convolutional Neural Networks for Mobile Applications


Title	Merging and Evolution: Improving Convolutional Neural Networks for Mobile Applications
Authors	Zheng Qin, Zhaoning Zhang, Shiqing Zhang, Hao Yu, Yuxing Peng
Abstract	Compact neural networks are inclined to exploit “sparsely-connected” convolutions such as depthwise convolution and group convolution for employment in mobile applications. Compared with standard “fully-connected” convolutions, these convolutions are more computationally economical. However, “sparsely-connected” convolutions block the inter-group information exchange, which induces severe performance degradation. To address this issue, we present two novel operations named merging and evolution to leverage the inter-group information. Our key idea is encoding the inter-group information with a narrow feature map, then combining the generated features with the original network for better representation. Taking advantage of the proposed operations, we then introduce the Merging-and-Evolution (ME) module, an architectural unit specifically designed for compact networks. Finally, we propose a family of compact neural networks called MENet based on ME modules. Extensive experiments on ILSVRC 2012 dataset and PASCAL VOC 2007 dataset demonstrate that MENet consistently outperforms other state-of-the-art compact networks under different computational budgets. For instance, under the computational budget of 140 MFLOPs, MENet surpasses ShuffleNet by 1% and MobileNet by 1.95% on ILSVRC 2012 top-1 accuracy, while by 2.3% and 4.1% on PASCAL VOC 2007 mAP, respectively.
Tasks
Published	2018-03-24
URL	http://arxiv.org/abs/1803.09127v1
PDF	http://arxiv.org/pdf/1803.09127v1.pdf
PWC	https://paperswithcode.com/paper/merging-and-evolution-improving-convolutional
Repo	https://github.com/osmr/imgclsmob
Framework	mxnet

Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate


Title	Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate
Authors	Christo Kirov, Ryan Cotterell
Abstract	Can advances in NLP help advance cognitive modeling? We examine the role of artificial neural networks, the current state of the art in many common NLP tasks, by returning to a classic case study. In 1986, Rumelhart and McClelland famously introduced a neural architecture that learned to transduce English verb stems to their past tense forms. Shortly thereafter, Pinker & Prince (1988) presented a comprehensive rebuttal of many of Rumelhart and McClelland’s claims. Much of the force of their attack centered on the empirical inadequacy of the Rumelhart and McClelland (1986) model. Today, however, that model is severely outmoded. We show that the Encoder-Decoder network architectures used in modern NLP systems obviate most of Pinker and Prince’s criticisms without requiring any simplication of the past tense mapping problem. We suggest that the empirical performance of modern networks warrants a re-examination of their utility in linguistic and cognitive modeling.
Tasks
Published	2018-07-12
URL	https://arxiv.org/abs/1807.04783v2
PDF	https://arxiv.org/pdf/1807.04783v2.pdf
PWC	https://paperswithcode.com/paper/recurrent-neural-networks-in-linguistic
Repo	https://github.com/LeenaShekhar/NLP-Linguistics-ML-Resources
Framework	tf

Deep Multi-Output Forecasting: Learning to Accurately Predict Blood Glucose Trajectories


Title	Deep Multi-Output Forecasting: Learning to Accurately Predict Blood Glucose Trajectories
Authors	Ian Fox, Lynn Ang, Mamta Jaiswal, Rodica Pop-Busui, Jenna Wiens
Abstract	In many forecasting applications, it is valuable to predict not only the value of a signal at a certain time point in the future, but also the values leading up to that point. This is especially true in clinical applications, where the future state of the patient can be less important than the patient’s overall trajectory. This requires multi-step forecasting, a forecasting variant where one aims to predict multiple values in the future simultaneously. Standard methods to accomplish this can propagate error from prediction to prediction, reducing quality over the long term. In light of these challenges, we propose multi-output deep architectures for multi-step forecasting in which we explicitly model the distribution of future values of the signal over a prediction horizon. We apply these techniques to the challenging and clinically relevant task of blood glucose forecasting. Through a series of experiments on a real-world dataset consisting of 550K blood glucose measurements, we demonstrate the effectiveness of our proposed approaches in capturing the underlying signal dynamics. Compared to existing shallow and deep methods, we find that our proposed approaches improve performance individually and capture complementary information, leading to a large improvement over the baseline when combined (4.87 vs. 5.31 absolute percentage error (APE)). Overall, the results suggest the efficacy of our proposed approach in predicting blood glucose level and multi-step forecasting more generally.
Tasks
Published	2018-06-14
URL	http://arxiv.org/abs/1806.05357v1
PDF	http://arxiv.org/pdf/1806.05357v1.pdf
PWC	https://paperswithcode.com/paper/deep-multi-output-forecasting-learning-to
Repo	https://github.com/igfox/multi-output-glucose-forecasting
Framework	pytorch

Can Network Analysis Techniques help to Predict Design Dependencies? An Initial Study


Title	Can Network Analysis Techniques help to Predict Design Dependencies? An Initial Study
Authors	J. Andrés Díaz-Pace, Antonela Tommasel, Daniela Godoy
Abstract	The degree of dependencies among the modules of a software system is a key attribute to characterize its design structure and its ability to evolve over time. Several design problems are often correlated with undesired dependencies among modules. Being able to anticipate those problems is important for developers, so they can plan early for maintenance and refactoring efforts. However, existing tools are limited to detecting undesired dependencies once they appeared in the system. In this work, we investigate whether module dependencies can be predicted (before they actually appear). Since the module structure can be regarded as a network, i.e, a dependency graph, we leverage on network features to analyze the dynamics of such a structure. In particular, we apply link prediction techniques for this task. We conducted an evaluation on two Java projects across several versions, using link prediction and machine learning techniques, and assessed their performance for identifying new dependencies from a project version to the next one. The results, although preliminary, show that the link prediction approach is feasible for package dependencies. Also, this work opens opportunities for further development of software-specific strategies for dependency prediction.
Tasks	Link Prediction
Published	2018-08-08
URL	http://arxiv.org/abs/1808.02776v1
PDF	http://arxiv.org/pdf/1808.02776v1.pdf
PWC	https://paperswithcode.com/paper/can-network-analysis-techniques-help-to
Repo	https://github.com/tommantonela/icsa-2018
Framework	none

TernaryNet: Faster Deep Model Inference without GPUs for Medical 3D Segmentation using Sparse and Binary Convolutions


Title	TernaryNet: Faster Deep Model Inference without GPUs for Medical 3D Segmentation using Sparse and Binary Convolutions
Authors	Mattias P. Heinrich, Max Blendowski, Ozan Oktay
Abstract	Deep convolutional neural networks (DCNN) are currently ubiquitous in medical imaging. While their versatility and high quality results for common image analysis tasks including segmentation, localisation and prediction is astonishing, the large representational power comes at the cost of highly demanding computational effort. This limits their practical applications for image guided interventions and diagnostic (point-of-care) support using mobile devices without graphics processing units (GPU). We propose a new scheme that approximates both trainable weights and neural activations in deep networks by ternary values and tackles the open question of backpropagation when dealing with non-differentiable functions. Our solution enables the removal of the expensive floating-point matrix multiplications throughout any convolutional neural network and replaces them by energy and time preserving binary operators and population counts. Our approach, which is demonstrated using a fully-convolutional network (FCN) for CT pancreas segmentation leads to more than 10-fold reduced memory requirements and we provide a concept for sub-second inference without GPUs. Our ternary approximation obtains high accuracies (without any post-processing) with a Dice overlap of 71.0% that are statistically equivalent to using networks with high-precision weights and activations. We further demonstrate the significant improvements reached in comparison to binary quantisation and without our proposed ternary hyperbolic tangent continuation. We present a key enabling technique for highly efficient DCNN inference without GPUs that will help to bring the advances of deep learning to practical clinical applications. It has also great promise for improving accuracies in large-scale medical data retrieval.
Tasks	3D Medical Imaging Segmentation, Pancreas Segmentation
Published	2018-01-29
URL	http://arxiv.org/abs/1801.09449v1
PDF	http://arxiv.org/pdf/1801.09449v1.pdf
PWC	https://paperswithcode.com/paper/ternarynet-faster-deep-model-inference
Repo	https://github.com/mattiaspaul/TernaryNet
Framework	pytorch

Improving Automatic Source Code Summarization via Deep Reinforcement Learning


Title	Improving Automatic Source Code Summarization via Deep Reinforcement Learning
Authors	Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, Philip S. Yu
Abstract	Code summarization provides a high level natural language description of the function performed by code, as it can benefit the software maintenance, code categorization and retrieval. To the best of our knowledge, most state-of-the-art approaches follow an encoder-decoder framework which encodes the code into a hidden space and then decode it into natural language space, suffering from two major drawbacks: a) Their encoders only consider the sequential content of code, ignoring the tree structure which is also critical for the task of code summarization, b) Their decoders are typically trained to predict the next word by maximizing the likelihood of next ground-truth word with previous ground-truth word given. However, it is expected to generate the entire sequence from scratch at test time. This discrepancy can cause an \textit{exposure bias} issue, making the learnt decoder suboptimal. In this paper, we incorporate an abstract syntax tree structure as well as sequential content of code snippets into a deep reinforcement learning framework (i.e., actor-critic network). The actor network provides the confidence of predicting the next word according to current state. On the other hand, the critic network evaluates the reward value of all possible extensions of the current state and can provide global guidance for explorations. We employ an advantage reward composed of BLEU metric to train both networks. Comprehensive experiments on a real-world dataset show the effectiveness of our proposed model when compared with some state-of-the-art methods.
Tasks	Code Summarization
Published	2018-11-17
URL	http://arxiv.org/abs/1811.07234v1
PDF	http://arxiv.org/pdf/1811.07234v1.pdf
PWC	https://paperswithcode.com/paper/improving-automatic-source-code-summarization
Repo	https://github.com/mf1832146/tree-transformer
Framework	none

Learning to Navigate for Fine-grained Classification


Title	Learning to Navigate for Fine-grained Classification
Authors	Ze Yang, Tiange Luo, Dong Wang, Zhiqiang Hu, Jun Gao, Liwei Wang
Abstract	Fine-grained classification is challenging due to the difficulty of finding discriminative features. Finding those subtle traits that fully characterize the object is not straightforward. To handle this circumstance, we propose a novel self-supervision mechanism to effectively localize informative regions without the need of bounding-box/part annotations. Our model, termed NTS-Net for Navigator-Teacher-Scrutinizer Network, consists of a Navigator agent, a Teacher agent and a Scrutinizer agent. In consideration of intrinsic consistency between informativeness of the regions and their probability being ground-truth class, we design a novel training paradigm, which enables Navigator to detect most informative regions under the guidance from Teacher. After that, the Scrutinizer scrutinizes the proposed regions from Navigator and makes predictions. Our model can be viewed as a multi-agent cooperation, wherein agents benefit from each other, and make progress together. NTS-Net can be trained end-to-end, while provides accurate fine-grained classification predictions as well as highly informative regions during inference. We achieve state-of-the-art performance in extensive benchmark datasets.
Tasks	Fine-Grained Image Classification
Published	2018-09-02
URL	http://arxiv.org/abs/1809.00287v1
PDF	http://arxiv.org/pdf/1809.00287v1.pdf
PWC	https://paperswithcode.com/paper/learning-to-navigate-for-fine-grained
Repo	https://github.com/osmr/imgclsmob
Framework	mxnet

Sparsely Aggregated Convolutional Networks


Title	Sparsely Aggregated Convolutional Networks
Authors	Ligeng Zhu, Ruizhi Deng, Michael Maire, Zhiwei Deng, Greg Mori, Ping Tan
Abstract	We explore a key architectural aspect of deep convolutional neural networks: the pattern of internal skip connections used to aggregate outputs of earlier layers for consumption by deeper layers. Such aggregation is critical to facilitate training of very deep networks in an end-to-end manner. This is a primary reason for the widespread adoption of residual networks, which aggregate outputs via cumulative summation. While subsequent works investigate alternative aggregation operations (e.g. concatenation), we focus on an orthogonal question: which outputs to aggregate at a particular point in the network. We propose a new internal connection structure which aggregates only a sparse set of previous outputs at any given depth. Our experiments demonstrate this simple design change offers superior performance with fewer parameters and lower computational requirements. Moreover, we show that sparse aggregation allows networks to scale more robustly to 1000+ layers, thereby opening future avenues for training long-running visual processes.
Tasks
Published	2018-01-18
URL	http://arxiv.org/abs/1801.05895v3
PDF	http://arxiv.org/pdf/1801.05895v3.pdf
PWC	https://paperswithcode.com/paper/sparsely-aggregated-convolutional-networks
Repo	https://github.com/Lyken17/SparseNet
Framework	tf

Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation


Title	Texar: A Modularized, Versatile, and Extensible Toolkit for Text Generation
Authors	Zhiting Hu, Haoran Shi, Bowen Tan, Wentao Wang, Zichao Yang, Tiancheng Zhao, Junxian He, Lianhui Qin, Di Wang, Xuezhe Ma, Zhengzhong Liu, Xiaodan Liang, Wangrong Zhu, Devendra Singh Sachan, Eric P. Xing
Abstract	We introduce Texar, an open-source toolkit aiming to support the broad set of text generation tasks that transform any inputs into natural language, such as machine translation, summarization, dialog, content manipulation, and so forth. With the design goals of modularity, versatility, and extensibility in mind, Texar extracts common patterns underlying the diverse tasks and methodologies, creates a library of highly reusable modules, and allows arbitrary model architectures and algorithmic paradigms. In Texar, model architecture, inference, and learning processes are properly decomposed. Modules at a high concept level can be freely assembled and plugged in/swapped out. The toolkit also supports a rich set of large-scale pretrained models. Texar is thus particularly suitable for researchers and practitioners to do fast prototyping and experimentation. The versatile toolkit also fosters technique sharing across different text generation tasks. Texar supports both TensorFlow and PyTorch, and is released under Apache License 2.0 at https://www.texar.io.
Tasks	Machine Translation, Text Generation
Published	2018-09-04
URL	https://arxiv.org/abs/1809.00794v2
PDF	https://arxiv.org/pdf/1809.00794v2.pdf
PWC	https://paperswithcode.com/paper/texar-a-modularized-versatile-and-extensible-1
Repo	https://github.com/asyml/texar
Framework	tf

Rank Minimization for Snapshot Compressive Imaging


Title	Rank Minimization for Snapshot Compressive Imaging
Authors	Yang Liu, Xin Yuan, Jinli Suo, David J. Brady, Qionghai Dai
Abstract	Snapshot compressive imaging (SCI) refers to compressive imaging systems where multiple frames are mapped into a single measurement, with video compressive imaging and hyperspectral compressive imaging as two representative applications. Though exciting results of high-speed videos and hyperspectral images have been demonstrated, the poor reconstruction quality precludes SCI from wide applications.This paper aims to boost the reconstruction quality of SCI via exploiting the high-dimensional structure in the desired signal. We build a joint model to integrate the nonlocal self-similarity of video/hyperspectral frames and the rank minimization approach with the SCI sensing process. Following this, an alternating minimization algorithm is developed to solve this non-convex problem. We further investigate the special structure of the sampling process in SCI to tackle the computational workload and memory issues in SCI reconstruction. Both simulation and real data (captured by four different SCI cameras) results demonstrate that our proposed algorithm leads to significant improvements compared with current state-of-the-art algorithms. We hope our results will encourage the researchers and engineers to pursue further in compressive imaging for real applications.
Tasks
Published	2018-07-20
URL	http://arxiv.org/abs/1807.07837v1
PDF	http://arxiv.org/pdf/1807.07837v1.pdf
PWC	https://paperswithcode.com/paper/rank-minimization-for-snapshot-compressive
Repo	https://github.com/liuyang12/DeSCI
Framework	none