Paper Group ANR 1698
Character Eyes: Seeing Language through Character-Level Taggers. Lightweight Markerless Monocular Face Capture with 3D Spatial Priors. A Deep Look into Neural Ranking Models for Information Retrieval. A Novel Generalized Artificial Neural Network for Mining Two-Class Datasets. Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparamet …
Character Eyes: Seeing Language through Character-Level Taggers
| Title | Character Eyes: Seeing Language through Character-Level Taggers |
| Authors | Yuval Pinter, Marc Marone, Jacob Eisenstein |
| Abstract | Character-level models have been used extensively in recent years in NLP tasks as both supplements and replacements for closed-vocabulary token-level word representations. In one popular architecture, character-level LSTMs are used to feed token representations into a sequence tagger predicting token-level annotations such as part-of-speech (POS) tags. In this work, we examine the behavior of POS taggers across languages from the perspective of individual hidden units within the character LSTM. We aggregate the behavior of these units into language-level metrics which quantify the challenges that taggers face on languages with different morphological properties, and identify links between synthesis and affixation preference and emergent behavior of the hidden tagger layer. In a comparative experiment, we show how modifying the balance between forward and backward hidden units affects model arrangement and performance in these types of languages. |
| Tasks | |
| Published | 2019-03-12 |
| URL | http://arxiv.org/abs/1903.05041v1 |
| http://arxiv.org/pdf/1903.05041v1.pdf | |
| PWC | https://paperswithcode.com/paper/character-eyes-seeing-language-through |
| Repo | |
| Framework | |
Lightweight Markerless Monocular Face Capture with 3D Spatial Priors
| Title | Lightweight Markerless Monocular Face Capture with 3D Spatial Priors |
| Authors | Shridhar Ravikumar |
| Abstract | We present a simple lightweight markerless facial performance capture framework using just a monocular video input that combines Active Appearance Models for feature tracking and prior constraints on 3D shapes into an integrated objective function. 2D monocular inputs inherently lack information along the depth axis and can lead to physically implausible solutions. In order to address this loss of information, we enforce a constraint on our objective function within a probabilistic framework that uses preexisting animations obtained from accurate 3D tracking systems, thus achieving more plausible results. Our system fits a Blendshape model to tracked 2D features while also handling noise in estimation of features and camera parameters. We learn separate constraints for the upper and lower regions of the face thus maintaining flexibility. We show that using this approach, we can obtain significant improvement in tracking especially along the depth dimension. Our method uses easily obtainable prior animation data. We show that our method can generate convincing animations using only a monocular video input. We quantitatively evaluate our results comparing it with an approach using a monocular input without our spatial constraints and show that our results are closer to the ground-truth geometry. Finally, we also evaluate the effect that the choice of the Blendshape set has on the results of the solver by solving for a different set of Blendshapes and quantitatively comparing it with our previous results and to the ground truth. We show that while the choice of Blendshapes does make a difference, the use of our spatial constraints generates results that are closer to the ground truth. |
| Tasks | |
| Published | 2019-01-16 |
| URL | http://arxiv.org/abs/1901.05355v1 |
| http://arxiv.org/pdf/1901.05355v1.pdf | |
| PWC | https://paperswithcode.com/paper/lightweight-markerless-monocular-face-capture |
| Repo | |
| Framework | |
A Deep Look into Neural Ranking Models for Information Retrieval
| Title | A Deep Look into Neural Ranking Models for Information Retrieval |
| Authors | Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, Xueqi Cheng |
| Abstract | Ranking models lie at the heart of research on information retrieval (IR). During the past decades, different techniques have been proposed for constructing ranking models, from traditional heuristic methods, probabilistic methods, to modern machine learning methods. Recently, with the advance of deep learning technology, we have witnessed a growing body of work in applying shallow or deep neural networks to the ranking problem in IR, referred to as neural ranking models in this paper. The power of neural ranking models lies in the ability to learn from the raw text inputs for the ranking problem to avoid many limitations of hand-crafted features. Neural networks have sufficient capacity to model complicated tasks, which is needed to handle the complexity of relevance estimation in ranking. Since there have been a large variety of neural ranking models proposed, we believe it is the right time to summarize the current status, learn from existing methodologies, and gain some insights for future development. In contrast to existing reviews, in this survey, we will take a deep look into the neural ranking models from different dimensions to analyze their underlying assumptions, major design principles, and learning strategies. We compare these models through benchmark tasks to obtain a comprehensive empirical understanding of the existing techniques. We will also discuss what is missing in the current literature and what are the promising and desired future directions. |
| Tasks | Information Retrieval |
| Published | 2019-03-16 |
| URL | https://arxiv.org/abs/1903.06902v3 |
| https://arxiv.org/pdf/1903.06902v3.pdf | |
| PWC | https://paperswithcode.com/paper/a-deep-look-into-neural-ranking-models-for |
| Repo | |
| Framework | |
A Novel Generalized Artificial Neural Network for Mining Two-Class Datasets
| Title | A Novel Generalized Artificial Neural Network for Mining Two-Class Datasets |
| Authors | Wei-Chang Yeh |
| Abstract | A novel general neural network (GNN) is proposed for two-class data mining in this study. In a GNN, each attribute in the dataset is treated as a node, with each pair of nodes being connected by an arc. The reliability is of each arc, which is similar to the weight in artificial neural network and must be solved using simplified swarm optimization (SSO), is constant. After the node reliability is made the transformed value of the related attribute, the approximate reliability of each GNN instance is calculated based on the proposed intelligent Monte Carlo simulation (iMCS). This approximate GNN reliability is then compared with a given threshold to predict each instance. The proposed iMCS-SSO is used to repeat the procedure and train the GNN, such that the predicted class values match the actual class values as much as possible. To evaluate the classification performance of the proposed GNN, experiments were performed on five well-known benchmark datasets. The computational results compared favorably with those obtained using support vector machines. |
| Tasks | |
| Published | 2019-10-23 |
| URL | https://arxiv.org/abs/1910.10461v1 |
| https://arxiv.org/pdf/1910.10461v1.pdf | |
| PWC | https://paperswithcode.com/paper/a-novel-generalized-artificial-neural-network |
| Repo | |
| Framework | |
Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
| Title | Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization |
| Authors | Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, Diana Marculescu |
| Abstract | Can we reduce the search cost of Neural Architecture Search (NAS) from days down to only few hours? NAS methods automate the design of Convolutional Networks (ConvNets) under hardware constraints and they have emerged as key components of AutoML frameworks. However, the NAS problem remains challenging due to the combinatorially large design space and the significant search time (at least 200 GPU-hours). In this work, we alleviate the NAS search cost down to less than 3 hours, while achieving state-of-the-art image classification results under mobile latency constraints. We propose a novel differentiable NAS formulation, namely Single-Path NAS, that uses one single-path over-parameterized ConvNet to encode all architectural decisions based on shared convolutional kernel parameters, hence drastically decreasing the search overhead. Single-Path NAS achieves state-of-the-art top-1 ImageNet accuracy (75.62%), hence outperforming existing mobile NAS methods in similar latency settings (~80ms). In particular, we enhance the accuracy-runtime trade-off in differentiable NAS by treating the Squeeze-and-Excitation path as a fully searchable operation with our novel single-path encoding. Our method has an overall cost of only 8 epochs (24 TPU-hours), which is up to 5,000x faster compared to prior work. Moreover, we study how different NAS formulation choices affect the performance of the designed ConvNets. Furthermore, we exploit the efficiency of our method to answer an interesting question: instead of empirically tuning the hyperparameters of the NAS solver (as in prior work), can we automatically find the hyperparameter values that yield the desired accuracy-runtime trade-off? We open-source our entire codebase at: https://github.com/dstamoulis/single-path-nas. |
| Tasks | AutoML, Hyperparameter Optimization, Image Classification, Neural Architecture Search |
| Published | 2019-07-01 |
| URL | https://arxiv.org/abs/1907.00959v1 |
| https://arxiv.org/pdf/1907.00959v1.pdf | |
| PWC | https://paperswithcode.com/paper/single-path-mobile-automl-efficient-convnet |
| Repo | |
| Framework | |
Tensor Q-Rank: A New Data Dependent Tensor Rank
| Title | Tensor Q-Rank: A New Data Dependent Tensor Rank |
| Authors | Hao Kong, Zhouchen Lin |
| Abstract | Recently, the \textit{Tensor Nuclear Norm~(TNN)} regularization based on t-SVD has been widely used in various low tubal-rank tensor recovery tasks. However, these models usually require smooth change of data along the third dimension to ensure their low rank structures. In this paper, we propose a new definition of tensor rank named \textit{tensor Q-rank} by a column orthonormal matrix $\mathbf{Q}$, and further make $\mathbf{Q}$ data-dependent. We introduce an explainable selection method of $\mathbf{Q}$, under which the data tensor may have a more significant low tensor Q-rank structure than that of low tubal-rank structure. We also provide a corresponding envelope of our rank function and apply it to the low rank tensor completion problem. Then we give an effective algorithm and briefly analyze why our method works better than TNN based methods in the case of complex data with low sampling rate. Finally, experimental results on real-world datasets demonstrate the superiority of our proposed model in the tensor completion problem. |
| Tasks | |
| Published | 2019-10-26 |
| URL | https://arxiv.org/abs/1910.12016v3 |
| https://arxiv.org/pdf/1910.12016v3.pdf | |
| PWC | https://paperswithcode.com/paper/tensor-q-rank-a-new-data-dependent-tensor |
| Repo | |
| Framework | |
Shaping representations through communication: community size effect in artificial learning systems
| Title | Shaping representations through communication: community size effect in artificial learning systems |
| Authors | Olivier Tieleman, Angeliki Lazaridou, Shibl Mourad, Charles Blundell, Doina Precup |
| Abstract | Motivated by theories of language and communication that explain why communities with large numbers of speakers have, on average, simpler languages with more regularity, we cast the representation learning problem in terms of learning to communicate. Our starting point sees the traditional autoencoder setup as a single encoder with a fixed decoder partner that must learn to communicate. Generalizing from there, we introduce community-based autoencoders in which multiple encoders and decoders collectively learn representations by being randomly paired up on successive training iterations. We find that increasing community sizes reduce idiosyncrasies in the learned codes, resulting in representations that better encode concept categories and correlate with human feature norms. |
| Tasks | Representation Learning |
| Published | 2019-12-12 |
| URL | https://arxiv.org/abs/1912.06208v1 |
| https://arxiv.org/pdf/1912.06208v1.pdf | |
| PWC | https://paperswithcode.com/paper/shaping-representations-through-communication |
| Repo | |
| Framework | |
Integrating Artificial Intelligence into Weapon Systems
| Title | Integrating Artificial Intelligence into Weapon Systems |
| Authors | Philip Feldman, Aaron Dant, Aaron Massey |
| Abstract | The integration of Artificial Intelligence (AI) into weapon systems is one of the most consequential tactical and strategic decisions in the history of warfare. Current AI development is a remarkable combination of accelerating capability, hidden decision mechanisms, and decreasing costs. Implementation of these systems is in its infancy and exists on a spectrum from resilient and flexible to simplistic and brittle. Resilient systems should be able to effectively handle the complexities of a high-dimensional battlespace. Simplistic AI implementations could be manipulated by an adversarial AI that identifies and exploits their weaknesses. In this paper, we present a framework for understanding the development of dynamic AI/ML systems that interactively and continuously adapt to their user’s needs. We explore the implications of increasingly capable AI in the kill chain and how this will lead inevitably to a fully automated, always on system, barring regulation by treaty. We examine the potential of total integration of cyber and physical security and how this likelihood must inform the development of AI-enabled systems with respect to the “fog of war”, human morals, and ethics. |
| Tasks | |
| Published | 2019-05-10 |
| URL | https://arxiv.org/abs/1905.03899v1 |
| https://arxiv.org/pdf/1905.03899v1.pdf | |
| PWC | https://paperswithcode.com/paper/integrating-artificial-intelligence-into |
| Repo | |
| Framework | |
SGD without Replacement: Sharper Rates for General Smooth Convex Functions
| Title | SGD without Replacement: Sharper Rates for General Smooth Convex Functions |
| Authors | Prateek Jain, Dheeraj Nagaraj, Praneeth Netrapalli |
| Abstract | We study stochastic gradient descent {\em without replacement} (\sgdwor) for smooth convex functions. \sgdwor is widely observed to converge faster than true \sgd where each sample is drawn independently {\em with replacement} \cite{bottou2009curiously} and hence, is more popular in practice. But it’s convergence properties are not well understood as sampling without replacement leads to coupling between iterates and gradients. By using method of exchangeable pairs to bound Wasserstein distance, we provide the first non-asymptotic results for \sgdwor when applied to {\em general smooth, strongly-convex} functions. In particular, we show that \sgdwor converges at a rate of $O(1/K^2)$ while \sgd is known to converge at $O(1/K)$ rate, where $K$ denotes the number of passes over data and is required to be {\em large enough}. Existing results for \sgdwor in this setting require additional {\em Hessian Lipschitz assumption} \cite{gurbuzbalaban2015random,haochen2018random}. For {\em small} $K$, we show \sgdwor can achieve same convergence rate as \sgd for {\em general smooth strongly-convex} functions. Existing results in this setting require $K=1$ and hold only for generalized linear models \cite{shamir2016without}. In addition, by careful analysis of the coupling, for both large and small $K$, we obtain better dependence on problem dependent parameters like condition number. |
| Tasks | |
| Published | 2019-03-04 |
| URL | https://arxiv.org/abs/1903.01463v2 |
| https://arxiv.org/pdf/1903.01463v2.pdf | |
| PWC | https://paperswithcode.com/paper/sgd-without-replacement-sharper-rates-for |
| Repo | |
| Framework | |
Compatible Natural Gradient Policy Search
| Title | Compatible Natural Gradient Policy Search |
| Authors | Joni Pajarinen, Hong Linh Thai, Riad Akrour, Jan Peters, Gerhard Neumann |
| Abstract | Trust-region methods have yielded state-of-the-art results in policy search. A common approach is to use KL-divergence to bound the region of trust resulting in a natural gradient policy update. We show that the natural gradient and trust region optimization are equivalent if we use the natural parameterization of a standard exponential policy distribution in combination with compatible value function approximation. Moreover, we show that standard natural gradient updates may reduce the entropy of the policy according to a wrong schedule leading to premature convergence. To control entropy reduction we introduce a new policy search method called compatible policy search (COPOS) which bounds entropy loss. The experimental results show that COPOS yields state-of-the-art results in challenging continuous control tasks and in discrete partially observable tasks. |
| Tasks | Continuous Control |
| Published | 2019-02-07 |
| URL | http://arxiv.org/abs/1902.02823v1 |
| http://arxiv.org/pdf/1902.02823v1.pdf | |
| PWC | https://paperswithcode.com/paper/compatible-natural-gradient-policy-search |
| Repo | |
| Framework | |
ATIS + SpiNNaker: a Fully Event-based Visual Tracking Demonstration
| Title | ATIS + SpiNNaker: a Fully Event-based Visual Tracking Demonstration |
| Authors | Arren Glover, Alan B. Stokes, Steve Furber, Chiara Bartolozzi |
| Abstract | The Asynchronous Time-based Image Sensor (ATIS) and the Spiking Neural Network Architecture (SpiNNaker) are both neuromorphic technologies that “unconventionally” use binary spikes to represent information. The ATIS produces spikes to represent the change in light falling on the sensor, and the SpiNNaker is a massively parallel computing platform that asynchronously sends spikes between cores for processing. In this demonstration we show these two hardware used together to perform a visual tracking task. We aim to show the hardware and software architecture that integrates the ATIS and SpiNNaker together in a robot middle-ware that makes processing agnostic to the platform (CPU or SpiNNaker). We also aim to describe the algorithm, why it is suitable for the “unconventional” sensor and processing platform including the advantages as well as challenges faced. |
| Tasks | Visual Tracking |
| Published | 2019-12-03 |
| URL | https://arxiv.org/abs/1912.01320v1 |
| https://arxiv.org/pdf/1912.01320v1.pdf | |
| PWC | https://paperswithcode.com/paper/atis-spinnaker-a-fully-event-based-visual |
| Repo | |
| Framework | |
Fine-grained Synthesis of Unrestricted Adversarial Examples
| Title | Fine-grained Synthesis of Unrestricted Adversarial Examples |
| Authors | Omid Poursaeed, Tianxing Jiang, Harry Yang, Serge Belongie, Ser-Nam Lim |
| Abstract | We propose a novel approach for generating unrestricted adversarial examples by manipulating fine-grained aspects of image generation. Unlike existing unrestricted attacks that typically hand-craft geometric transformations, we learn stylistic and stochastic modifications leveraging state-of-the-art generative models. This allows us to manipulate an image in a controlled, fine-grained manner without being bounded by a norm threshold. Our model can be used for both targeted and non-targeted unrestricted attacks. We demonstrate that our attacks can bypass certified defenses, yet our adversarial images look indistinguishable from natural images as verified by human evaluation. Adversarial training can be used as an effective defense without degrading performance of the model on clean images. We perform experiments on LSUN and CelebA-HQ as high resolution datasets to validate efficacy of our proposed approach. |
| Tasks | Image Generation |
| Published | 2019-11-20 |
| URL | https://arxiv.org/abs/1911.09058v1 |
| https://arxiv.org/pdf/1911.09058v1.pdf | |
| PWC | https://paperswithcode.com/paper/fine-grained-synthesis-of-unrestricted |
| Repo | |
| Framework | |
Search to Distill: Pearls are Everywhere but not the Eyes
| Title | Search to Distill: Pearls are Everywhere but not the Eyes |
| Authors | Yu Liu, Xuhui Jia, Mingxing Tan, Raviteja Vemulapalli, Yukun Zhu, Bradley Green, Xiaogang Wang |
| Abstract | Standard Knowledge Distillation (KD) approaches distill the knowledge of a cumbersome teacher model into the parameters of a student model with a pre-defined architecture. However, the knowledge of a neural network, which is represented by the network’s output distribution conditioned on its input, depends not only on its parameters but also on its architecture. Hence, a more generalized approach for KD is to distill the teacher’s knowledge into both the parameters and architecture of the student. To achieve this, we present a new Architecture-aware Knowledge Distillation (AKD) approach that finds student models (pearls for the teacher) that are best for distilling the given teacher model. In particular, we leverage Neural Architecture Search (NAS), equipped with our KD-guided reward, to search for the best student architectures for a given teacher. Experimental results show our proposed AKD consistently outperforms the conventional NAS plus KD approach, and achieves state-of-the-art results on the ImageNet classification task under various latency settings. Furthermore, the best AKD student architecture for the ImageNet classification task also transfers well to other tasks such as million level face recognition and ensemble learning. |
| Tasks | Face Recognition, Neural Architecture Search |
| Published | 2019-11-20 |
| URL | https://arxiv.org/abs/1911.09074v2 |
| https://arxiv.org/pdf/1911.09074v2.pdf | |
| PWC | https://paperswithcode.com/paper/search-to-distill-pearls-are-everywhere-but |
| Repo | |
| Framework | |
Stem-driven Language Models for Morphologically Rich Languages
| Title | Stem-driven Language Models for Morphologically Rich Languages |
| Authors | Yash Shah, Ishan Tarunesh, Harsh Deshpande, Preethi Jyothi |
| Abstract | Neural language models (LMs) have shown to benefit significantly from enhancing word vectors with subword-level information, especially for morphologically rich languages. This has been mainly tackled by providing subword-level information as an input; using subword units in the output layer has been far less explored. In this work, we propose LMs that are cognizant of the underlying stems in each word. We derive stems for words using a simple unsupervised technique for stem identification. We experiment with different architectures involving multi-task learning and mixture models over words and stems. We focus on four morphologically complex languages – Hindi, Tamil, Kannada and Finnish – and observe significant perplexity gains with using our stem-driven LMs when compared with other competitive baseline models. |
| Tasks | Multi-Task Learning |
| Published | 2019-10-25 |
| URL | https://arxiv.org/abs/1910.11536v1 |
| https://arxiv.org/pdf/1910.11536v1.pdf | |
| PWC | https://paperswithcode.com/paper/stem-driven-language-models-for |
| Repo | |
| Framework | |
S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search
| Title | S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search |
| Authors | Zhihang Yuan, Bingzhe Wu, Zheng Liang, Shiwan Zhao, Weichen Bi, Guangyu Sun |
| Abstract | Recently, dynamic inference has emerged as a promising way to reduce the computational cost of deep convolutional neural network (CNN). In contrast to static methods (e.g. weight pruning), dynamic inference adaptively adjusts the inference process according to each input sample, which can considerably reduce the computational cost on “easy” samples while maintaining the overall model performance. In this paper, we introduce a general framework, S2DNAS, which can transform various static CNN models to support dynamic inference via neural architecture search. To this end, based on a given CNN model, we first generate a CNN architecture space in which each architecture is a multi-stage CNN generated from the given model using some predefined transformations. Then, we propose a reinforcement learning based approach to automatically search for the optimal CNN architecture in the generated space. At last, with the searched multi-stage network, we can perform dynamic inference by adaptively choosing a stage to evaluate for each sample. Unlike previous works that introduce irregular computations or complex controllers in the inference or re-design a CNN model from scratch, our method can generalize to most of the popular CNN architectures and the searched dynamic network can be directly deployed using existing deep learning frameworks in various hardware devices. |
| Tasks | Neural Architecture Search |
| Published | 2019-11-16 |
| URL | https://arxiv.org/abs/1911.07033v2 |
| https://arxiv.org/pdf/1911.07033v2.pdf | |
| PWC | https://paperswithcode.com/paper/s2dnas-transforming-static-cnn-model-for |
| Repo | |
| Framework | |
