February 1, 2020

3153 words 15 mins read

Paper Group AWR 290

Effective parameter estimation methods for an ExcitNet model in generative text-to-speech systems. PySS3: A Python package implementing a novel text classifier with visualization tools for Explainable AI. Variational Pretraining for Semi-supervised Text Classification. Towards Verified Stochastic Variational Inference for Probabilistic Programs. Po …

Effective parameter estimation methods for an ExcitNet model in generative text-to-speech systems


Title	Effective parameter estimation methods for an ExcitNet model in generative text-to-speech systems
Authors	Ohsung Kwon, Eunwoo Song, Jae-Min Kim, Hong-Goo Kang
Abstract	In this paper, we propose a high-quality generative text-to-speech (TTS) system using an effective spectrum and excitation estimation method. Our previous research verified the effectiveness of the ExcitNet-based speech generation model in a parametric TTS framework. However, the challenge remains to build a high-quality speech synthesis system because auxiliary conditional features estimated by a simple deep neural network often contain large prediction errors, and the errors are inevitably propagated throughout the autoregressive generation process of the ExcitNet vocoder. To generate more natural speech signals, we exploited a sequence-to-sequence (seq2seq) acoustic model with an attention-based generative network (e.g., Tacotron 2) to estimate the condition parameters of the ExcitNet vocoder. Because the seq2seq acoustic model accurately estimates spectral parameters, and because the ExcitNet model effectively generates the corresponding time-domain excitation signals, combining these two models can synthesize natural speech signals. Furthermore, we verified the merit of the proposed method in producing expressive speech segments by adopting a global style token-based emotion embedding method. The experimental results confirmed that the proposed system significantly outperforms the systems with a similarly configured conventional WaveNet vocoder and our best prior parametric TTS counterpart.
Tasks	Speech Synthesis
Published	2019-05-21
URL	https://arxiv.org/abs/1905.08486v1
PDF	https://arxiv.org/pdf/1905.08486v1.pdf
PWC	https://paperswithcode.com/paper/effective-parameter-estimation-methods-for-an
Repo	https://github.com/sewplay/demos
Framework	none

PySS3: A Python package implementing a novel text classifier with visualization tools for Explainable AI


Title	PySS3: A Python package implementing a novel text classifier with visualization tools for Explainable AI
Authors	Sergio G. Burdisso, Marcelo Errecalde, Manuel Montes-y-Gómez
Abstract	A recently introduced text classifier, called SS3, has obtained state-of-the-art performance on the CLEF’s eRisk tasks. SS3 was created to deal with risk detection over text streams and therefore not only supports incremental training and classification but also can visually explain its rationale. However, little attention has been paid to the potential use of SS3 as a general classifier. We believe this could be due to the unavailability of an open-source implementation of SS3. In this work, we introduce PySS3, a package that not only implements SS3 but also comes with visualization tools that allow researchers deploying robust, explainable and trusty machine learning models for text classification.
Tasks	Document Classification, Multi-Label Text Classification, Sentence Classification, Text Categorization, Text Classification
Published	2019-12-19
URL	https://arxiv.org/abs/1912.09322v1
PDF	https://arxiv.org/pdf/1912.09322v1.pdf
PWC	https://paperswithcode.com/paper/pyss3-a-python-package-implementing-a-novel
Repo	https://github.com/sergioburdisso/pyss3
Framework	none

Variational Pretraining for Semi-supervised Text Classification


Title	Variational Pretraining for Semi-supervised Text Classification
Authors	Suchin Gururangan, Tam Dang, Dallas Card, Noah A. Smith
Abstract	We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find that fine-tuning to in-domain data is crucial to achieving decent performance from contextual embeddings when working with limited supervision. We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.
Tasks	Text Classification
Published	2019-06-05
URL	https://arxiv.org/abs/1906.02242v1
PDF	https://arxiv.org/pdf/1906.02242v1.pdf
PWC	https://paperswithcode.com/paper/variational-pretraining-for-semi-supervised
Repo	https://github.com/allenai/vampire
Framework	none

Towards Verified Stochastic Variational Inference for Probabilistic Programs


Title	Towards Verified Stochastic Variational Inference for Probabilistic Programs
Authors	Wonyeol Lee, Hangyeol Yu, Xavier Rival, Hongseok Yang
Abstract	Probabilistic programming is the idea of writing models from statistics and machine learning using program notations and reasoning about these models using generic inference engines. Recently its combination with deep learning has been explored intensely, leading to the development of deep probabilistic programming languages such as Pyro. At the core of this development lie inference engines based on stochastic variational inference algorithms. When asked to find information about the posterior distribution of a model written in such a language, these algorithms convert this posterior-inference query into an optimisation problem and solve it approximately by gradient ascent. In this paper, we analyse one of the most fundamental and versatile variational inference algorithms, called score estimator, using tools from denotational semantics and program analysis. We formally express what this algorithm does on models denoted by programs, and expose implicit assumptions made by the algorithm. The violation of these assumptions may lead to an undefined optimisation objective or the loss of convergence guarantee of the optimisation process. We then describe rules for proving these assumptions, which can be automated by static program analyses. Some of our rules use nontrivial facts from continuous mathematics, and let us replace requirements about integrals in the assumptions, by conditions involving differentiation or boundedness, which are much easier to prove automatically. Following our general methodology, we have developed a static program analysis for Pyro that aims at discharging the assumption about what we call model-guide support match. Applied to the eight representative model-guide pairs from the Pyro webpage, our analysis finds a bug in one of these cases, reveals a non-standard use of an inference engine in another, and shows the assumptions are met in the remaining cases.
Tasks	Probabilistic Programming
Published	2019-07-20
URL	https://arxiv.org/abs/1907.08827v4
PDF	https://arxiv.org/pdf/1907.08827v4.pdf
PWC	https://paperswithcode.com/paper/towards-verified-stochastic-variational
Repo	https://github.com/wonyeol/static-analysis-for-support-match
Framework	none

Post-Data Augmentation to Improve Deep Pose Estimation of Extreme and Wild Motions


Title	Post-Data Augmentation to Improve Deep Pose Estimation of Extreme and Wild Motions
Authors	Kohei Toyoda, Michinari Kono, Jun Rekimoto
Abstract	Contributions of recent deep-neural-network (DNN) based techniques have been playing a significant role in human-computer interaction (HCI) and user interface (UI) domains. One of the commonly used DNNs is human pose estimation. This kind of technique is widely used for motion capturing of humans, and to generate or modify virtual avatars. However, in order to gain accuracy and to use such systems, large and precise datasets are required for the machine learning (ML) procedure. This can be especially difficult for extreme/wild motions such as acrobatic movements or motions in specific sports, which are difficult to estimate in typically provided training models. In addition, training may take a long duration, and will require a high-grade GPU for sufficient speed. To address these issues, we propose a method to improve the pose estimation accuracy for extreme/wild motions by using pre-trained models, i.e., without performing the training procedure by yourselves. We assume our method to encourage usage of these DNN techniques for users in application areas that are out of the ML field, and to help users without high-end computers to apply them for personal and end use cases.
Tasks	Data Augmentation, Pose Estimation
Published	2019-02-12
URL	http://arxiv.org/abs/1902.04250v1
PDF	http://arxiv.org/pdf/1902.04250v1.pdf
PWC	https://paperswithcode.com/paper/post-data-augmentation-to-improve-deep-pose
Repo	https://github.com/ktoyod/rotatedpose
Framework	none

A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfer


Title	A Hierarchical Reinforced Sequence Operation Method for Unsupervised Text Style Transfer
Authors	Chen Wu, Xuancheng Ren, Fuli Luo, Xu Sun
Abstract	Unsupervised text style transfer aims to alter text styles while preserving the content, without aligned data for supervision. Existing seq2seq methods face three challenges: 1) the transfer is weakly interpretable, 2) generated outputs struggle in content preservation, and 3) the trade-off between content and style is intractable. To address these challenges, we propose a hierarchical reinforced sequence operation method, named Point-Then-Operate (PTO), which consists of a high-level agent that proposes operation positions and a low-level agent that alters the sentence. We provide comprehensive training objectives to control the fluency, style, and content of the outputs and a mask-based inference algorithm that allows for multi-step revision based on the single-step trained agents. Experimental results on two text style transfer datasets show that our method significantly outperforms recent methods and effectively addresses the aforementioned challenges.
Tasks	Style Transfer, Text Style Transfer
Published	2019-06-05
URL	https://arxiv.org/abs/1906.01833v1
PDF	https://arxiv.org/pdf/1906.01833v1.pdf
PWC	https://paperswithcode.com/paper/a-hierarchical-reinforced-sequence-operation
Repo	https://github.com/lancopku/Point-Then-Operate
Framework	pytorch

A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer


Title	A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer
Authors	Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, Xu Sun
Abstract	Unsupervised text style transfer aims to transfer the underlying style of text but keep its main content unchanged without parallel data. Most existing methods typically follow two steps: first separating the content from the original style, and then fusing the content with the desired style. However, the separation in the first step is challenging because the content and style interact in subtle ways in natural language. Therefore, in this paper, we propose a dual reinforcement learning framework to directly transfer the style of the text via a one-step mapping model, without any separation of content and style. Specifically, we consider the learning of the source-to-target and target-to-source mappings as a dual task, and two rewards are designed based on such a dual structure to reflect the style accuracy and content preservation, respectively. In this way, the two one-step mapping models can be trained via reinforcement learning, without any use of parallel data. Automatic evaluations show that our model outperforms the state-of-the-art systems by a large margin, especially with more than 8 BLEU points improvement averaged on two benchmark datasets. Human evaluations also validate the effectiveness of our model in terms of style accuracy, content preservation and fluency. Our code and data, including outputs of all baselines and our model are available at https://github.com/luofuli/DualLanST.
Tasks	Text Style Transfer
Published	2019-05-24
URL	https://arxiv.org/abs/1905.10060v1
PDF	https://arxiv.org/pdf/1905.10060v1.pdf
PWC	https://paperswithcode.com/paper/a-dual-reinforcement-learning-framework-for
Repo	https://github.com/luofuli/DualLanST
Framework	tf

Matching the Blanks: Distributional Similarity for Relation Learning


Title	Matching the Blanks: Distributional Similarity for Relation Learning
Authors	Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, Tom Kwiatkowski
Abstract	General purpose relation extractors, which can model arbitrary relations, are a core aspiration in information extraction. Efforts have been made to build general purpose extractors that represent relations with their surface forms, or which jointly embed surface forms with relations from an existing knowledge graph. However, both of these approaches are limited in their ability to generalize. In this paper, we build on extensions of Harris’ distributional hypothesis to relations, as well as recent advances in learning text representations (specifically, BERT), to build task agnostic relation representations solely from entity-linked text. We show that these representations significantly outperform previous work on exemplar based relation extraction (FewRel) even without using any of that task’s training data. We also show that models initialized with our task agnostic representations, and then tuned on supervised relation extraction datasets, significantly outperform the previous methods on SemEval 2010 Task 8, KBP37, and TACRED.
Tasks	Relation Extraction
Published	2019-06-07
URL	https://arxiv.org/abs/1906.03158v1
PDF	https://arxiv.org/pdf/1906.03158v1.pdf
PWC	https://paperswithcode.com/paper/matching-the-blanks-distributional-similarity
Repo	https://github.com/plkmo/BERT-Relation-Extraction
Framework	pytorch

Provable Guarantees for Gradient-Based Meta-Learning


Title	Provable Guarantees for Gradient-Based Meta-Learning
Authors	Mikhail Khodak, Maria-Florina Balcan, Ameet Talwalkar
Abstract	We study the problem of meta-learning through the lens of online convex optimization, developing a meta-algorithm bridging the gap between popular gradient-based meta-learning and classical regularization-based multi-task transfer methods. Our method is the first to simultaneously satisfy good sample efficiency guarantees in the convex setting, with generalization bounds that improve with task-similarity, while also being computationally scalable to modern deep learning architectures and the many-task setting. Despite its simplicity, the algorithm matches, up to a constant factor, a lower bound on the performance of any such parameter-transfer method under natural task similarity assumptions. We use experiments in both convex and deep learning settings to verify and demonstrate the applicability of our theory.
Tasks	Meta-Learning
Published	2019-02-27
URL	https://arxiv.org/abs/1902.10644v2
PDF	https://arxiv.org/pdf/1902.10644v2.pdf
PWC	https://paperswithcode.com/paper/provable-guarantees-for-gradient-based-meta
Repo	https://github.com/mkhodak/FMRL
Framework	tf

A New Three-stage Curriculum Learning Approach to Deep Network Based Liver Tumor Segmentation


Title	A New Three-stage Curriculum Learning Approach to Deep Network Based Liver Tumor Segmentation
Authors	Huiyu Li, Xiabi Liu, Said Boumaraf, Weihua Liu, Xiaopeng Gong, Xiaohong Ma
Abstract	Automatic segmentation of liver tumors in medical images is crucial for the computer-aided diagnosis and therapy. It is a challenging task, since the tumors are notoriously small against the background voxels. This paper proposes a new three-stage curriculum learning approach for training deep networks to tackle this small object segmentation problem. The learning in the first stage is performed on the whole input to obtain an initial deep network for tumor segmenta-tion. Then the second stage of learning focuses the strength-ening of tumor specific features by continuing training the network on the tumor patches. Finally, we retrain the net-work on the whole input in the third stage, in order that the tumor specific features and the global context can be inte-grated ideally under the segmentation objective. Benefitting from the proposed learning approach, we only need to em-ploy one single network to segment the tumors directly. We evaluated our approach on the 2017 MICCAI Liver Tumor Segmentation challenge dataset. In the experiments, our approach exhibits significant improvement compared with the commonly used cascaded counterpart.
Tasks	Semantic Segmentation
Published	2019-10-17
URL	https://arxiv.org/abs/1910.07895v1
PDF	https://arxiv.org/pdf/1910.07895v1.pdf
PWC	https://paperswithcode.com/paper/a-new-three-stage-curriculum-learning
Repo	https://github.com/Huiyu-Li/Three-stage-Curriculum-Learning
Framework	pytorch

Learning Bregman Divergences


Title	Learning Bregman Divergences
Authors	Ali Siahkamari, Venkatesh Saligrama, David Castanon, Brian Kulis
Abstract	Metric learning is the problem of learning a task-specific distance function given supervision. Classical linear methods for this problem (known as Mahalanobis metric learning approaches) are well-studied both theoretically and empirically, but are limited to Euclidean distances after learned linear transformations of the input space. In this paper, we consider learning a Bregman divergence, a rich and important class of divergences that includes Mahalanobis metrics as a special case but also includes the KL-divergence and others. We develop a formulation and algorithm for learning arbitrary Bregman divergences based on approximating their underlying convex generating function via a piecewise linear function. We show several theoretical results of our resulting model, including a PAC guarantee that the learned Bregman divergence approximates an arbitrary Bregman divergence with error O_p (m^(-1/(d+2))), where m is the number of training points and d is the dimension of the data. We provide empirical results on using the learned divergences for classification, semi-supervised clustering, and ranking problems.
Tasks	Metric Learning
Published	2019-05-28
URL	https://arxiv.org/abs/1905.11545v3
PDF	https://arxiv.org/pdf/1905.11545v3.pdf
PWC	https://paperswithcode.com/paper/learning-bregman-divergences
Repo	https://github.com/Siahkamari/Learning-Bregman-Divergences
Framework	none

Unsupervised Learning of Object Keypoints for Perception and Control


Title	Unsupervised Learning of Object Keypoints for Perception and Control
Authors	Tejas Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, Volodymyr Mnih
Abstract	The study of object representations in computer vision has primarily focused on developing representations that are useful for image classification, object detection, or semantic segmentation as downstream tasks. In this work we aim to learn object representations that are useful for control and reinforcement learning (RL). To this end, we introduce Transporter, a neural network architecture for discovering concise geometric object representations in terms of keypoints or image-space coordinates. Our method learns from raw video frames in a fully unsupervised manner, by transporting learnt image features between video frames using a keypoint bottleneck. The discovered keypoints track objects and object parts across long time-horizons more accurately than recent similar methods. Furthermore, consistent long-term tracking enables two notable results in control domains – (1) using the keypoint co-ordinates and corresponding image features as inputs enables highly sample-efficient reinforcement learning; (2) learning to explore by controlling keypoint locations drastically reduces the search space, enabling deep exploration (leading to states unreachable through random action exploration) without any extrinsic rewards.
Tasks	Image Classification, Object Detection, Semantic Segmentation
Published	2019-06-19
URL	https://arxiv.org/abs/1906.11883v2
PDF	https://arxiv.org/pdf/1906.11883v2.pdf
PWC	https://paperswithcode.com/paper/unsupervised-learning-of-object-keypoints-for
Repo	https://github.com/deepmind/deepmind-research/tree/master/transporter
Framework	tf

A Simple Algorithm for Scalable Monte Carlo Inference


Title	A Simple Algorithm for Scalable Monte Carlo Inference
Authors	Alexander Borisenko, Maksym Byshkin, Alessandro Lomi
Abstract	The methods of statistical physics are widely used for modelling complex networks. Building on the recently proposed Equilibrium Expectation approach, we derive a simple and efficient algorithm for maximum likelihood estimation (MLE) of parameters of exponential family distributions - a family of statistical models, that includes Ising model, Markov Random Field and Exponential Random Graph models. Computational experiments and analysis of empirical data demonstrate that the algorithm increases by orders of magnitude the size of network data amenable to Monte Carlo based inference. We report results suggesting that the applicability of the algorithm may readily be extended to the analysis of large samples of dependent observations commonly found in biology, sociology, astrophysics, and ecology.
Tasks
Published	2019-01-02
URL	https://arxiv.org/abs/1901.00533v4
PDF	https://arxiv.org/pdf/1901.00533v4.pdf
PWC	https://paperswithcode.com/paper/a-simple-algorithm-for-scalable-monte-carlo
Repo	https://github.com/Byshkin/EquilibriumExpectation
Framework	none

The Tradeoff Between Privacy and Accuracy in Anomaly Detection Using Federated XGBoost


Title	The Tradeoff Between Privacy and Accuracy in Anomaly Detection Using Federated XGBoost
Authors	Mengwei Yang, Linqi Song, Jie Xu, Congduan Li, Guozhen Tan
Abstract	Privacy has raised considerable concerns recently, especially with the advent of information explosion and numerous data mining techniques to explore the information inside large volumes of data. In this context, a new distributed learning paradigm termed federated learning becomes prominent recently to tackle the privacy issues in distributed learning, where only learning models will be transmitted from the distributed nodes to servers without revealing users’ own data and hence protecting the privacy of users. In this paper, we propose a horizontal federated XGBoost algorithm to solve the federated anomaly detection problem, where the anomaly detection aims to identify abnormalities from extremely unbalanced datasets and can be considered as a special classification problem. Our proposed federated XGBoost algorithm incorporates data aggregation and sparse federated update processes to balance the tradeoff between privacy and learning performance. In particular, we introduce the virtual data sample by aggregating a group of users’ data together at a single distributed node. We compute parameters based on these virtual data samples in the local nodes and aggregate the learning model in the central server. In the learning model upgrading process, we focus more on the wrongly classified data before in the virtual sample and hence to generate sparse learning model parameters. By carefully controlling the size of these groups of samples, we can achieve a tradeoff between privacy and learning performance. Our experimental results show the effectiveness of our proposed scheme by comparing with existing state-of-the-arts.
Tasks	Anomaly Detection, Sparse Learning
Published	2019-07-16
URL	https://arxiv.org/abs/1907.07157v2
PDF	https://arxiv.org/pdf/1907.07157v2.pdf
PWC	https://paperswithcode.com/paper/the-tradeoff-between-privacy-and-accuracy-in
Repo	https://github.com/Raymw/Federated-XGBoost
Framework	none

Bag of Freebies for Training Object Detection Neural Networks


Title	Bag of Freebies for Training Object Detection Neural Networks
Authors	Zhi Zhang, Tong He, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Mu Li
Abstract	Training heuristics greatly improve various image classification model accuracies~\cite{he2018bag}. Object detection models, however, have more complex neural network structures and optimization targets. The training strategies and pipelines dramatically vary among different models. In this works, we explore training tweaks that apply to various models including Faster R-CNN and YOLOv3. These tweaks do not change the model architectures, therefore, the inference costs remain the same. Our empirical results demonstrate that, however, these freebies can improve up to 5% absolute precision compared to state-of-the-art baselines.
Tasks	Image Classification, Object Detection
Published	2019-02-11
URL	http://arxiv.org/abs/1902.04103v3
PDF	http://arxiv.org/pdf/1902.04103v3.pdf
PWC	https://paperswithcode.com/paper/bag-of-freebies-for-training-object-detection
Repo	https://github.com/lingtengqiu/Yolo_Nano
Framework	pytorch