October 21, 2019

3127 words 15 mins read

Paper Group AWR 63

Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution. Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package. Liver Lesion Detection from Weakly-labeled Multi-phase CT Volumes with a Grouped Single Shot MultiBox Detector. Improving Whole Slide Segmentation Through Visual Context - A Systematic St …

Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution


Title	Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution
Authors	Judea Pearl
Abstract	Current machine learning systems operate, almost exclusively, in a statistical, or model-free mode, which entails severe theoretical limits on their power and performance. Such systems cannot reason about interventions and retrospection and, therefore, cannot serve as the basis for strong AI. To achieve human level intelligence, learning machines need the guidance of a model of reality, similar to the ones used in causal inference tasks. To demonstrate the essential role of such models, I will present a summary of seven tasks which are beyond reach of current machine learning systems and which have been accomplished using the tools of causal modeling.
Tasks	Causal Inference
Published	2018-01-11
URL	http://arxiv.org/abs/1801.04016v1
PDF	http://arxiv.org/pdf/1801.04016v1.pdf
PWC	https://paperswithcode.com/paper/theoretical-impediments-to-machine-learning
Repo	https://github.com/CausalityReadingGroup/Readings
Framework	none

Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package


Title	Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package
Authors	Ajay Patel, Alexander Sands, Chris Callison-Burch, Marianna Apidianaki
Abstract	Vector space embedding models like word2vec, GloVe, fastText, and ELMo are extremely popular representations in natural language processing (NLP) applications. We present Magnitude, a fast, lightweight tool for utilizing and processing embeddings. Magnitude is an open source Python package with a compact vector storage file format that allows for efficient manipulation of huge numbers of embeddings. Magnitude performs common operations up to 60 to 6,000 times faster than Gensim. Magnitude introduces several novel features for improved robustness like out-of-vocabulary lookups.
Tasks	Word Embeddings
Published	2018-10-26
URL	http://arxiv.org/abs/1810.11190v1
PDF	http://arxiv.org/pdf/1810.11190v1.pdf
PWC	https://paperswithcode.com/paper/magnitude-a-fast-efficient-universal-vector
Repo	https://github.com/plasticityai/magnitude
Framework	pytorch

Liver Lesion Detection from Weakly-labeled Multi-phase CT Volumes with a Grouped Single Shot MultiBox Detector


Title	Liver Lesion Detection from Weakly-labeled Multi-phase CT Volumes with a Grouped Single Shot MultiBox Detector
Authors	Sang-gil Lee, Jae Seok Bae, Hyunjae Kim, Jung Hoon Kim, Sungroh Yoon
Abstract	We present a focal liver lesion detection model leveraged by custom-designed multi-phase computed tomography (CT) volumes, which reflects real-world clinical lesion detection practice using a Single Shot MultiBox Detector (SSD). We show that grouped convolutions effectively harness richer information of the multi-phase data for the object detection model, while a naive application of SSD suffers from a generalization gap. We trained and evaluated the modified SSD model and recently proposed variants with our CT dataset of 64 subjects by five-fold cross validation. Our model achieved a 53.3% average precision score and ran in under three seconds per volume, outperforming the original model and state-of-the-art variants. Results show that the one-stage object detection model is a practical solution, which runs in near real-time and can learn an unbiased feature representation from a large-volume real-world detection dataset, which requires less tedious and time consuming construction of the weak phase-level bounding box labels.
Tasks	Computed Tomography (CT), Lesion Segmentation, Medical Object Detection, Object Detection
Published	2018-07-02
URL	http://arxiv.org/abs/1807.00436v1
PDF	http://arxiv.org/pdf/1807.00436v1.pdf
PWC	https://paperswithcode.com/paper/liver-lesion-detection-from-weakly-labeled
Repo	https://github.com/L0SG/grouped-ssd-pytorch
Framework	pytorch

Improving Whole Slide Segmentation Through Visual Context - A Systematic Study


Title	Improving Whole Slide Segmentation Through Visual Context - A Systematic Study
Authors	Korsuk Sirinukunwattana, Nasullah Khalid Alham, Clare Verrill, Jens Rittscher
Abstract	While challenging, the dense segmentation of histology images is a necessary first step to assess changes in tissue architecture and cellular morphology. Although specific convolutional neural network architectures have been applied with great success to the problem, few effectively incorporate visual context information from multiple scales. With this paper, we present a systematic comparison of different architectures to assess how including multi-scale information affects segmentation performance. A publicly available breast cancer and a locally collected prostate cancer datasets are being utilised for this study. The results support our hypothesis that visual context and scale play a crucial role in histology image classification problems.
Tasks	Image Classification
Published	2018-06-11
URL	http://arxiv.org/abs/1806.04259v1
PDF	http://arxiv.org/pdf/1806.04259v1.pdf
PWC	https://paperswithcode.com/paper/improving-whole-slide-segmentation-through
Repo	https://github.com/ksirinuk/miccai2018
Framework	pytorch

Less but Better: Generalization Enhancement of Ordinal Embedding via Distributional Margin


Title	Less but Better: Generalization Enhancement of Ordinal Embedding via Distributional Margin
Authors	Ke Ma, Qianqian Xu, Zhiyong Yang, Xiaochun Cao
Abstract	In the absence of prior knowledge, ordinal embedding methods obtain new representation for items in a low-dimensional Euclidean space via a set of quadruple-wise comparisons. These ordinal comparisons often come from human annotators, and sufficient comparisons induce the success of classical approaches. However, collecting a large number of labeled data is known as a hard task, and most of the existing work pay little attention to the generalization ability with insufficient samples. Meanwhile, recent progress in large margin theory discloses that rather than just maximizing the minimum margin, both the margin mean and variance, which characterize the margin distribution, are more crucial to the overall generalization performance. To address the issue of insufficient training samples, we propose a margin distribution learning paradigm for ordinal embedding, entitled Distributional Margin based Ordinal Embedding (\textit{DMOE}). Precisely, we first define the margin for ordinal embedding problem. Secondly, we formulate a concise objective function which avoids maximizing margin mean and minimizing margin variance directly but exhibits the similar effect. Moreover, an Augmented Lagrange Multiplier based algorithm is customized to seek the optimal solution of \textit{DMOE} effectively. Experimental studies on both simulated and real-world datasets are provided to show the effectiveness of the proposed algorithm.
Tasks
Published	2018-12-05
URL	http://arxiv.org/abs/1812.01939v1
PDF	http://arxiv.org/pdf/1812.01939v1.pdf
PWC	https://paperswithcode.com/paper/less-but-better-generalization-enhancement-of
Repo	https://github.com/alphaprime/DMOE
Framework	none

Galaxy morphology prediction using capsule networks


Title	Galaxy morphology prediction using capsule networks
Authors	Reza Katebi, Yadi Zhou, Ryan Chornock, Razvan Bunescu
Abstract	Understanding morphological types of galaxies is a key parameter for studying their formation and evolution. Neural networks that have been used previously for galaxy morphology classification have some disadvantages, such as not being invariant under rotation. In this work, we studied the performance of Capsule Network, a recently introduced neural network architecture that is rotationally invariant and spatially aware, on the task of galaxy morphology classification. We designed two evaluation scenarios based on the answers from the question tree in the Galaxy Zoo project. In the first scenario, we used Capsule Network for regression and predicted probabilities for all of the questions. In the second scenario, we chose the answer to the first morphology question that had the highest user agreement as the class of the object and trained a Capsule Network classifier, where we also reconstructed galaxy images. We achieved promising results in both of these scenarios. Automated approaches such as the one introduced here will greatly decrease the workload of astronomers and will play a critical role in the upcoming large sky surveys.
Tasks
Published	2018-09-22
URL	http://arxiv.org/abs/1809.08377v1
PDF	http://arxiv.org/pdf/1809.08377v1.pdf
PWC	https://paperswithcode.com/paper/galaxy-morphology-prediction-using-capsule
Repo	https://github.com/RezaKatebi/Galaxy-Morphology-CapsNet
Framework	pytorch

ELASTIC: Improving CNNs with Dynamic Scaling Policies


Title	ELASTIC: Improving CNNs with Dynamic Scaling Policies
Authors	Huiyu Wang, Aniruddha Kembhavi, Ali Farhadi, Alan Yuille, Mohammad Rastegari
Abstract	Scale variation has been a challenge from traditional to modern approaches in computer vision. Most solutions to scale issues have a similar theme: a set of intuitive and manually designed policies that are generic and fixed (e.g. SIFT or feature pyramid). We argue that the scaling policy should be learned from data. In this paper, we introduce ELASTIC, a simple, efficient and yet very effective approach to learn a dynamic scale policy from data. We formulate the scaling policy as a non-linear function inside the network’s structure that (a) is learned from data, (b) is instance specific, (c) does not add extra computation, and (d) can be applied on any network architecture. We applied ELASTIC to several state-of-the-art network architectures and showed consistent improvement without extra (sometimes even lower) computation on ImageNet classification, MSCOCO multi-label classification, and PASCAL VOC semantic segmentation. Our results show major improvement for images with scale challenges. Our code is available here: https://github.com/allenai/elastic
Tasks	Multi-Label Classification, Semantic Segmentation
Published	2018-12-13
URL	http://arxiv.org/abs/1812.05262v2
PDF	http://arxiv.org/pdf/1812.05262v2.pdf
PWC	https://paperswithcode.com/paper/elastic-improving-cnns-with-instance-specific
Repo	https://github.com/allenai/elastic
Framework	pytorch

Subword-augmented Embedding for Cloze Reading Comprehension


Title	Subword-augmented Embedding for Cloze Reading Comprehension
Authors	Zhuosheng Zhang, Yafang Huang, Hai Zhao
Abstract	Representation learning is the foundation of machine reading comprehension. In state-of-the-art models, deep learning methods broadly use word and character level representations. However, character is not naturally the minimal linguistic unit. In addition, with a simple concatenation of character and word embedding, previous models actually give suboptimal solution. In this paper, we propose to use subword rather than character for word embedding enhancement. We also empirically explore different augmentation strategies on subword-augmented embedding to enhance the cloze-style reading comprehension model reader. In detail, we present a reader that uses subword-level representation to augment word embedding with a short list to handle rare words effectively. A thorough examination is conducted to evaluate the comprehensive performance and generalization ability of the proposed reader. Experimental results show that the proposed approach helps the reader significantly outperform the state-of-the-art baselines on various public datasets.
Tasks	Machine Reading Comprehension, Reading Comprehension, Representation Learning
Published	2018-06-24
URL	http://arxiv.org/abs/1806.09103v1
PDF	http://arxiv.org/pdf/1806.09103v1.pdf
PWC	https://paperswithcode.com/paper/subword-augmented-embedding-for-cloze-reading
Repo	https://github.com/cooelf/subMrc
Framework	none

Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks


Title	Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
Authors	Adrià Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, Antonio Torralba
Abstract	We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer can be added as a preprocessing block to existing task networks and trained altogether in an end-to-end fashion. The effect of the layer is to efficiently estimate how to sample from the original data in order to boost task performance. For example, for an image classification task in which the original data might range in size up to several megapixels, but where the desired input images to the task network are much smaller, our layer learns how best to sample from the underlying high resolution data in a manner which preserves task-relevant information better than uniform downsampling. This has the effect of creating distorted, caricature-like intermediate images, in which idiosyncratic elements of the image that improve task performance are zoomed and exaggerated. Unlike alternative approaches such as spatial transformer networks, our proposed layer is inspired by image saliency, computed efficiently from uniformly downsampled data, and degrades gracefully to a uniform sampling strategy under uncertainty. We apply our layer to improve existing networks for the tasks of human gaze estimation and fine-grained object classification. Code for our method is available in: http://github.com/recasens/Saliency-Sampler
Tasks	Caricature, Gaze Estimation, Image Classification, Object Classification
Published	2018-09-10
URL	http://arxiv.org/abs/1809.03355v1
PDF	http://arxiv.org/pdf/1809.03355v1.pdf
PWC	https://paperswithcode.com/paper/learning-to-zoom-a-saliency-based-sampling
Repo	https://github.com/recasens/Saliency-Sampler
Framework	pytorch

SuperPCA: A Superpixelwise PCA Approach for Unsupervised Feature Extraction of Hyperspectral Imagery


Title	SuperPCA: A Superpixelwise PCA Approach for Unsupervised Feature Extraction of Hyperspectral Imagery
Authors	Junjun Jiang, Jiayi Ma, Chen Chen, Zhongyuan Wang, Zhihua Cai, Lizhe Wang
Abstract	As an unsupervised dimensionality reduction method, principal component analysis (PCA) has been widely considered as an efficient and effective preprocessing step for hyperspectral image (HSI) processing and analysis tasks. It takes each band as a whole and globally extracts the most representative bands. However, different homogeneous regions correspond to different objects, whose spectral features are diverse. It is obviously inappropriate to carry out dimensionality reduction through a unified projection for an entire HSI. In this paper, a simple but very effective superpixelwise PCA approach, called SuperPCA, is proposed to learn the intrinsic low-dimensional features of HSIs. In contrast to classical PCA models, SuperPCA has four main properties. (1) Unlike the traditional PCA method based on a whole image, SuperPCA takes into account the diversity in different homogeneous regions, that is, different regions should have different projections. (2) Most of the conventional feature extraction models cannot directly use the spatial information of HSIs, while SuperPCA is able to incorporate the spatial context information into the unsupervised dimensionality reduction by superpixel segmentation. (3) Since the regions obtained by superpixel segmentation have homogeneity, SuperPCA can extract potential low-dimensional features even under noise. (4) Although SuperPCA is an unsupervised method, it can achieve competitive performance when compared with supervised approaches. The resulting features are discriminative, compact, and noise resistant, leading to improved HSI classification performance. Experiments on three public datasets demonstrate that the SuperPCA model significantly outperforms the conventional PCA based dimensionality reduction baselines for HSI classification. The Matlab source code is available at https://github.com/junjun-jiang/SuperPCA
Tasks	Dimensionality Reduction
Published	2018-06-26
URL	http://arxiv.org/abs/1806.09807v2
PDF	http://arxiv.org/pdf/1806.09807v2.pdf
PWC	https://paperswithcode.com/paper/superpca-a-superpixelwise-pca-approach-for
Repo	https://github.com/junjun-jiang/SuperPCA
Framework	none

Hamiltonian Variational Auto-Encoder


Title	Hamiltonian Variational Auto-Encoder
Authors	Anthony L. Caterini, Arnaud Doucet, Dino Sejdinovic
Abstract	Variational Auto-Encoders (VAEs) have become very popular techniques to perform inference and learning in latent variable models as they allow us to leverage the rich representational power of neural networks to obtain flexible approximations of the posterior of latent variables as well as tight evidence lower bounds (ELBOs). Combined with stochastic variational inference, this provides a methodology scaling to large datasets. However, for this methodology to be practically efficient, it is necessary to obtain low-variance unbiased estimators of the ELBO and its gradients with respect to the parameters of interest. While the use of Markov chain Monte Carlo (MCMC) techniques such as Hamiltonian Monte Carlo (HMC) has been previously suggested to achieve this [23, 26], the proposed methods require specifying reverse kernels which have a large impact on performance. Additionally, the resulting unbiased estimator of the ELBO for most MCMC kernels is typically not amenable to the reparameterization trick. We show here how to optimally select reverse kernels in this setting and, by building upon Hamiltonian Importance Sampling (HIS) [17], we obtain a scheme that provides low-variance unbiased estimators of the ELBO and its gradients using the reparameterization trick. This allows us to develop a Hamiltonian Variational Auto-Encoder (HVAE). This method can be reinterpreted as a target-informed normalizing flow [20] which, within our context, only requires a few evaluations of the gradient of the sampled likelihood and trivial Jacobian calculations at each iteration.
Tasks	Latent Variable Models
Published	2018-05-29
URL	http://arxiv.org/abs/1805.11328v2
PDF	http://arxiv.org/pdf/1805.11328v2.pdf
PWC	https://paperswithcode.com/paper/hamiltonian-variational-auto-encoder
Repo	https://github.com/anthonycaterini/hvae-nips
Framework	tf

A Fourier View of REINFORCE


Title	A Fourier View of REINFORCE
Authors	Adeel Pervez
Abstract	We show a connection between the Fourier spectrum of Boolean functions and the REINFORCE gradient estimator for binary latent variable models. We show that REINFORCE estimates (up to a factor) the degree-1 Fourier coefficients of a Boolean function. Using this connection we offer a new perspective on variance reduction in gradient estimation for latent variable models: namely, that variance reduction involves eliminating or reducing Fourier coefficients that do not have degree 1. We then use this connection to develop low-variance unbiased gradient estimators for binary latent variable models such as sigmoid belief networks. The estimator is based upon properties of the noise operator from Boolean Fourier theory and involves a sample-dependent baseline added to the REINFORCE estimator in a way that keeps the estimator unbiased. The baseline can be plugged into existing gradient estimators for further variance reduction.
Tasks	Latent Variable Models
Published	2018-08-12
URL	http://arxiv.org/abs/1808.03953v1
PDF	http://arxiv.org/pdf/1808.03953v1.pdf
PWC	https://paperswithcode.com/paper/a-fourier-view-of-reinforce
Repo	https://github.com/alpz/fourier-REINFORCE
Framework	tf

Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition


Title	Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition
Authors	Jianshu Zhang, Jun Du, Lirong Dai
Abstract	Handwritten mathematical expression recognition is a challenging problem due to the complicated two-dimensional structures, ambiguous handwriting input and variant scales of handwritten math symbols. To settle this problem, we utilize the attention based encoder-decoder model that recognizes mathematical expression images from two-dimensional layouts to one-dimensional LaTeX strings. We improve the encoder by employing densely connected convolutional networks as they can strengthen feature extraction and facilitate gradient propagation especially on a small training set. We also present a novel multi-scale attention model which is employed to deal with the recognition of math symbols in different scales and save the fine-grained details that will be dropped by pooling operations. Validated on the CROHME competition task, the proposed method significantly outperforms the state-of-the-art methods with an expression recognition accuracy of 52.8% on CROHME 2014 and 50.1% on CROHME 2016, by only using the official training dataset.
Tasks
Published	2018-01-05
URL	http://arxiv.org/abs/1801.03530v2
PDF	http://arxiv.org/pdf/1801.03530v2.pdf
PWC	https://paperswithcode.com/paper/multi-scale-attention-with-dense-encoder-for
Repo	https://github.com/JianshuZhang/WAP
Framework	none

Attention-Based Guided Structured Sparsity of Deep Neural Networks


Title	Attention-Based Guided Structured Sparsity of Deep Neural Networks
Authors	Amirsina Torfi, Rouzbeh A. Shirvani, Sobhan Soleymani, Nasser M. Nasrabadi
Abstract	Network pruning is aimed at imposing sparsity in a neural network architecture by increasing the portion of zero-valued weights for reducing its size regarding energy-efficiency consideration and increasing evaluation speed. In most of the conducted research efforts, the sparsity is enforced for network pruning without any attention to the internal network characteristics such as unbalanced outputs of the neurons or more specifically the distribution of the weights and outputs of the neurons. That may cause severe accuracy drop due to uncontrolled sparsity. In this work, we propose an attention mechanism that simultaneously controls the sparsity intensity and supervised network pruning by keeping important information bottlenecks of the network to be active. On CIFAR-10, the proposed method outperforms the best baseline method by 6% and reduced the accuracy drop by 2.6x at the same level of sparsity.
Tasks	Network Pruning
Published	2018-02-13
URL	http://arxiv.org/abs/1802.09902v4
PDF	http://arxiv.org/pdf/1802.09902v4.pdf
PWC	https://paperswithcode.com/paper/attention-based-guided-structured-sparsity-of
Repo	https://github.com/astorfi/attention-guided-sparsity
Framework	tf

Small steps and giant leaps: Minimal Newton solvers for Deep Learning


Title	Small steps and giant leaps: Minimal Newton solvers for Deep Learning
Authors	João F. Henriques, Sebastien Ehrhardt, Samuel Albanie, Andrea Vedaldi
Abstract	We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires two additional forward-mode automatic differentiation operations per iteration, which has a computational cost comparable to two standard forward passes and is easy to implement. Our method addresses long-standing issues with current second-order solvers, which invert an approximate Hessian matrix every iteration exactly or by conjugate-gradient methods, a procedure that is both costly and sensitive to noise. Instead, we propose to keep a single estimate of the gradient projected by the inverse Hessian matrix, and update it once per iteration. This estimate has the same size and is similar to the momentum variable that is commonly used in SGD. No estimate of the Hessian is maintained. We first validate our method, called CurveBall, on small problems with known closed-form solutions (noisy Rosenbrock function and degenerate 2-layer linear networks), where current deep learning solvers seem to struggle. We then train several large models on CIFAR and ImageNet, including ResNet and VGG-f networks, where we demonstrate faster convergence with no hyperparameter tuning. Code is available.
Tasks
Published	2018-05-21
URL	http://arxiv.org/abs/1805.08095v1
PDF	http://arxiv.org/pdf/1805.08095v1.pdf
PWC	https://paperswithcode.com/paper/small-steps-and-giant-leaps-minimal-newton
Repo	https://github.com/jotaf98/curveball
Framework	none