Paper Group ANR 539
How Good is SGD with Random Shuffling?. Connecting exciton diffusion with surface roughness via deep learning. On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment. Bayesian sparse convex clustering via global-local shrinkage priors. NASIB: Neural Archit …
How Good is SGD with Random Shuffling?
| Title | How Good is SGD with Random Shuffling? |
| Authors | Itay Safran, Ohad Shamir |
| Abstract | We study the performance of stochastic gradient descent (SGD) on smooth and strongly-convex finite-sum optimization problems. In contrast to the majority of existing theoretical works, which assume that individual functions are sampled with replacement, we focus here on popular but poorly-understood heuristics, which involve going over random permutations of the individual functions. This setting has been investigated in several recent works, but the optimal error rates remain unclear. In this paper, we provide lower bounds on the expected optimization error with these heuristics (using SGD with any constant step size), which elucidate their advantages and disadvantages. In particular, we prove that after $k$ passes over $n$ individual functions, if the functions are re-shuffled after every pass, the best possible optimization error for SGD is at least $\Omega\left(1/(nk)^2+1/nk^3\right)$, which partially corresponds to recently derived upper bounds. Moreover, if the functions are only shuffled once, then the lower bound increases to $\Omega(1/nk^2)$. Since there are strictly smaller upper bounds for repeated reshuffling, this proves an inherent performance gap between SGD with single shuffling and repeated shuffling. As a more minor contribution, we also provide a non-asymptotic $\Omega(1/k^2)$ lower bound (independent of $n$) for the incremental gradient method, when no random shuffling takes place. Finally, we provide an indication that our lower bounds are tight, by proving matching upper bounds for univariate quadratic functions. |
| Tasks | |
| Published | 2019-07-31 |
| URL | https://arxiv.org/abs/1908.00045v2 |
| https://arxiv.org/pdf/1908.00045v2.pdf | |
| PWC | https://paperswithcode.com/paper/how-good-is-sgd-with-random-shuffling |
| Repo | |
| Framework | |
Connecting exciton diffusion with surface roughness via deep learning
| Title | Connecting exciton diffusion with surface roughness via deep learning |
| Authors | Liyao Lyu, Zhiwen Zhang, Jingrun Chen |
| Abstract | Exciton diffusion plays a vital role in the function of many organic semiconducting opto-electronic devices, where an accurate description requires precise control of heterojunctions. This poses a challenging problem because the parameterization of heterojunctions in high-dimensional random space is far beyond the capability of classical simulation tools. Here, we develop a novel method based on deep neural network to extract a function for exciton diffusion length on surface roughness with high accuracy and unprecedented efficiency, yielding an abundance of information over the entire parameter space. Our method provides a new strategy to analyze the impact of interfacial ordering on exciton diffusion and is expected to assist experimental design with tailored opto-electronic functionalities. |
| Tasks | |
| Published | 2019-10-31 |
| URL | https://arxiv.org/abs/1910.14209v1 |
| https://arxiv.org/pdf/1910.14209v1.pdf | |
| PWC | https://paperswithcode.com/paper/connecting-exciton-diffusion-with-surface |
| Repo | |
| Framework | |
On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment
| Title | On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra-observer Variability in 2D Echocardiography Quality Assessment |
| Authors | Zhibin Liao, Hany Girgis, Amir Abdi, Hooman Vaseli, Jorden Hetherington, Robert Rohling, Ken Gin, Teresa Tsang, Purang Abolmaesumi |
| Abstract | Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecting cardiovascular disease at point-of-care. Echo imaging quality and acquisition time is highly dependent on the operator’s experience level. Recent developments have shown the possibility of automating echo image quality quantification by mapping an expert’s assessment of quality to the echo image via deep learning techniques. Nevertheless, the observer variability in the expert’s assessment can impact the quality quantification accuracy. Here, we aim to model the intra-observer variability in echo quality assessment as an aleatoric uncertainty modelling regression problem with the introduction of a novel method that handles the regression problem with categorical labels. A key feature of our design is that only a single forward pass is sufficient to estimate the level of uncertainty for the network output. Compared to the $0.11 \pm 0.09$ absolute error (in a scale from 0 to 1) archived by the conventional regression method, the proposed method brings the error down to $0.09 \pm 0.08$, where the improvement is statistically significant and equivalents to $5.7%$ test accuracy improvement. The simplicity of the proposed approach means that it could be generalized to other applications of deep learning in medical imaging, where there is often uncertainty in clinical labels. |
| Tasks | |
| Published | 2019-11-02 |
| URL | https://arxiv.org/abs/1911.00674v1 |
| https://arxiv.org/pdf/1911.00674v1.pdf | |
| PWC | https://paperswithcode.com/paper/on-modelling-label-uncertainty-in-deep-neural |
| Repo | |
| Framework | |
Bayesian sparse convex clustering via global-local shrinkage priors
| Title | Bayesian sparse convex clustering via global-local shrinkage priors |
| Authors | Kaito Shimamura, Shuichi Kawano |
| Abstract | Sparse convex clustering is to cluster observations and conduct variable selection simultaneously in the framework of convex clustering. Although the weighted $L_1$ norm as the regularization term is usually employed in the sparse convex clustering, this increases the dependence on the data and reduces the estimation accuracy if the sample size is not sufficient. To tackle these problems, this paper proposes a Bayesian sparse convex clustering via the idea of Bayesian lasso and global-local shrinkage priors. We introduce Gibbs sampling algorithms for our method using scale mixtures of normals. The effectiveness of the proposed methods is shown in simulation studies and a real data analysis. |
| Tasks | |
| Published | 2019-11-20 |
| URL | https://arxiv.org/abs/1911.08703v1 |
| https://arxiv.org/pdf/1911.08703v1.pdf | |
| PWC | https://paperswithcode.com/paper/bayesian-sparse-convex-clustering-via-global |
| Repo | |
| Framework | |
NASIB: Neural Architecture Search withIn Budget
| Title | NASIB: Neural Architecture Search withIn Budget |
| Authors | Abhishek Singh, Anubhav Garg, Jinan Zhou, Shiv Ram Dubey, Debo Dutta |
| Abstract | Neural Architecture Search (NAS) represents a class of methods to generate the optimal neural network architecture and typically iterate over candidate architectures till convergence over some particular metric like validation loss. They are constrained by the available computation resources, especially in enterprise environments. In this paper, we propose a new approach for NAS, called NASIB, which adapts and attunes to the computation resources (budget) available by varying the exploration vs. exploitation trade-off. We reduce the expert bias by searching over an augmented search space induced by Superkernels. The proposed method can provide the architecture search useful for different computation resources and different domains beyond image classification of natural images where we lack bespoke architecture motifs and domain expertise. We show, on CIFAR10, that itis possible to search over a space that comprises of 12x more candidate operations than the traditional prior art in just 1.5 GPU days, while reaching close to state of the art accuracy. While our method searches over an exponentially larger search space, it could lead to novel architectures that require lesser domain expertise, compared to the majority of the existing methods. |
| Tasks | Image Classification, Neural Architecture Search |
| Published | 2019-10-19 |
| URL | https://arxiv.org/abs/1910.08665v1 |
| https://arxiv.org/pdf/1910.08665v1.pdf | |
| PWC | https://paperswithcode.com/paper/nasib-neural-architecture-search-within |
| Repo | |
| Framework | |
A Characterization of Mean Squared Error for Estimator with Bagging
| Title | A Characterization of Mean Squared Error for Estimator with Bagging |
| Authors | Martin Mihelich, Charles Dognin, Yan Shu, Michael Blot |
| Abstract | Bagging can significantly improve the generalization performance of unstable machine learning algorithms such as trees or neural networks. Though bagging is now widely used in practice and many empirical studies have explored its behavior, we still know little about the theoretical properties of bagged predictions. In this paper, we theoretically investigate how the bagging method can reduce the Mean Squared Error (MSE) when applied on a statistical estimator. First, we prove that for any estimator, increasing the number of bagged estimators $N$ in the average can only reduce the MSE. This intuitive result, observed empirically and discussed in the literature, has not yet been rigorously proved. Second, we focus on the standard estimator of variance called unbiased sample variance and we develop an exact analytical expression of the MSE for this estimator with bagging. This allows us to rigorously discuss the number of iterations $N$ and the batch size $m$ of the bagging method. From this expression, we state that only if the kurtosis of the distribution is greater than $\frac{3}{2}$, the MSE of the variance estimator can be reduced with bagging. This result is important because it demonstrates that for distribution with low kurtosis, bagging can only deteriorate the performance of a statistical prediction. Finally, we propose a novel general-purpose algorithm to estimate with high precision the variance of a sample. |
| Tasks | |
| Published | 2019-08-07 |
| URL | https://arxiv.org/abs/1908.02718v1 |
| https://arxiv.org/pdf/1908.02718v1.pdf | |
| PWC | https://paperswithcode.com/paper/a-characterization-of-mean-squared-error-for |
| Repo | |
| Framework | |
Measuring Human Perception to Improve Handwritten Document Transcription
| Title | Measuring Human Perception to Improve Handwritten Document Transcription |
| Authors | Samuel Grieggs, Bingyu Shen, Pei Li, Cana Short, Jiaqi Ma, Mihow McKenny, Melody Wauke, Brian Price, Walter Scheirer |
| Abstract | The subtleties of human perception, as measured by vision scientists through the use of psychophysics, are important clues to the internal workings of visual recognition. For instance, measured reaction time can indicate whether a visual stimulus is easy for a subject to recognize, or whether it is hard. In this paper, we consider how to incorporate psychophysical measurements of visual perception into the loss function of a deep neural network being trained for a recognition task, under the assumption that such information can enforce consistency with human behavior. As a case study to assess the viability of this approach, we look at the problem of handwritten document transcription. While good progress has been made towards automatically transcribing modern handwriting, significant challenges remain in transcribing historical documents. Here we work towards a comprehensive transcription solution for Medieval manuscripts that combines networks trained using our novel loss formulation with natural language processing elements. In a baseline assessment, reliable performance is demonstrated for the standard IAM and RIMES datasets. Further, we go on to show feasibility for our approach on a previously published dataset and a new dataset of digitized Latin manuscripts, originally produced by scribes in the Cloister of St. Gall around the middle of the 9th century. |
| Tasks | |
| Published | 2019-04-07 |
| URL | http://arxiv.org/abs/1904.03734v3 |
| http://arxiv.org/pdf/1904.03734v3.pdf | |
| PWC | https://paperswithcode.com/paper/measuring-human-perception-to-improve |
| Repo | |
| Framework | |
Robust Anomaly Detection in Images using Adversarial Autoencoders
| Title | Robust Anomaly Detection in Images using Adversarial Autoencoders |
| Authors | Laura Beggel, Michael Pfeiffer, Bernd Bischl |
| Abstract | Reliably detecting anomalies in a given set of images is a task of high practical relevance for visual quality inspection, surveillance, or medical image analysis. Autoencoder neural networks learn to reconstruct normal images, and hence can classify those images as anomalies, where the reconstruction error exceeds some threshold. Here we analyze a fundamental problem of this approach when the training set is contaminated with a small fraction of outliers. We find that continued training of autoencoders inevitably reduces the reconstruction error of outliers, and hence degrades the anomaly detection performance. In order to counteract this effect, an adversarial autoencoder architecture is adapted, which imposes a prior distribution on the latent representation, typically placing anomalies into low likelihood-regions. Utilizing the likelihood model, potential anomalies can be identified and rejected already during training, which results in an anomaly detector that is significantly more robust to the presence of outliers during training. |
| Tasks | Anomaly Detection |
| Published | 2019-01-18 |
| URL | http://arxiv.org/abs/1901.06355v1 |
| http://arxiv.org/pdf/1901.06355v1.pdf | |
| PWC | https://paperswithcode.com/paper/robust-anomaly-detection-in-images-using |
| Repo | |
| Framework | |
Neural Belief Reasoner
| Title | Neural Belief Reasoner |
| Authors | Haifeng Qian |
| Abstract | This paper proposes a new generative model called neural belief reasoner (NBR). It differs from previous models in that it specifies a belief function rather than a probability distribution. Its implementation consists of neural networks, fuzzy-set operations and belief-function operations, and query-answering, sample-generation and training algorithms are presented. This paper studies NBR in two tasks. The first is a synthetic unsupervised-learning task, which demonstrates NBR’s ability to perform multi-hop reasoning, reasoning with uncertainty and reasoning about conflicting information. The second is supervised learning: a robust MNIST classifier. Without any adversarial training, this classifier exceeds the state of the art in adversarial robustness as measured by the L2 metric, and at the same time maintains 99% accuracy on natural images. A proof is presented that, as capacity increases, NBR classifiers can asymptotically approach the best possible robustness. |
| Tasks | |
| Published | 2019-09-10 |
| URL | https://arxiv.org/abs/1909.04719v1 |
| https://arxiv.org/pdf/1909.04719v1.pdf | |
| PWC | https://paperswithcode.com/paper/neural-belief-reasoner |
| Repo | |
| Framework | |
Sampling of Bayesian posteriors with a non-Gaussian probabilistic learning on manifolds from a small dataset
| Title | Sampling of Bayesian posteriors with a non-Gaussian probabilistic learning on manifolds from a small dataset |
| Authors | Christian Soize, Roger Ghanem |
| Abstract | This paper tackles the challenge presented by small-data to the task of Bayesian inference. A novel methodology, based on manifold learning and manifold sampling, is proposed for solving this computational statistics problem under the following assumptions: 1) neither the prior model nor the likelihood function are Gaussian and neither can be approximated by a Gaussian measure; 2) the number of functional input (system parameters) and functional output (quantity of interest) can be large; 3) the number of available realizations of the prior model is small, leading to the small-data challenge typically associated with expensive numerical simulations; the number of experimental realizations is also small; 4) the number of the posterior realizations required for decision is much larger than the available initial dataset. The method and its mathematical aspects are detailed. Three applications are presented for validation: The first two involve mathematical constructions aimed to develop intuition around the method and to explore its performance. The third example aims to demonstrate the operational value of the method using a more complex application related to the statistical inverse identification of the non-Gaussian matrix-valued random elasticity field of a damaged biological tissue (osteoporosis in a cortical bone) using ultrasonic waves. |
| Tasks | Bayesian Inference |
| Published | 2019-10-28 |
| URL | https://arxiv.org/abs/1910.12717v1 |
| https://arxiv.org/pdf/1910.12717v1.pdf | |
| PWC | https://paperswithcode.com/paper/sampling-of-bayesian-posteriors-with-a-non |
| Repo | |
| Framework | |
Sparse Representation Classification via Screening for Graphs
| Title | Sparse Representation Classification via Screening for Graphs |
| Authors | Cencheng Shen, Li Chen, Yuexiao Dong, Carey Priebe |
| Abstract | The sparse representation classifier (SRC) is shown to work well for image recognition problems that satisfy a subspace assumption. In this paper we propose a new implementation of SRC via screening, establish its equivalence to the original SRC under regularity conditions, and prove its classification consistency for random graphs drawn from stochastic blockmodels. The results are demonstrated via simulations and real data experiments, where the new algorithm achieves comparable numerical performance but significantly faster. |
| Tasks | Classification Consistency |
| Published | 2019-06-04 |
| URL | https://arxiv.org/abs/1906.01601v1 |
| https://arxiv.org/pdf/1906.01601v1.pdf | |
| PWC | https://paperswithcode.com/paper/sparse-representation-classification-via |
| Repo | |
| Framework | |
Manipulating the Difficulty of C-Tests
| Title | Manipulating the Difficulty of C-Tests |
| Authors | Ji-Ung Lee, Erik Schwan, Christian M. Meyer |
| Abstract | We propose two novel manipulation strategies for increasing and decreasing the difficulty of C-tests automatically. This is a crucial step towards generating learner-adaptive exercises for self-directed language learning and preparing language assessment tests. To reach the desired difficulty level, we manipulate the size and the distribution of gaps based on absolute and relative gap difficulty predictions. We evaluate our approach in corpus-based experiments and in a user study with 60 participants. We find that both strategies are able to generate C-tests with the desired difficulty level. |
| Tasks | |
| Published | 2019-06-17 |
| URL | https://arxiv.org/abs/1906.06905v2 |
| https://arxiv.org/pdf/1906.06905v2.pdf | |
| PWC | https://paperswithcode.com/paper/manipulating-the-difficulty-of-c-tests |
| Repo | |
| Framework | |
Monocular 3D Object Detection via Geometric Reasoning on Keypoints
| Title | Monocular 3D Object Detection via Geometric Reasoning on Keypoints |
| Authors | Ivan Barabanau, Alexey Artemov, Evgeny Burnaev, Vyacheslav Murashkin |
| Abstract | Monocular 3D object detection is well-known to be a challenging vision task due to the loss of depth information; attempts to recover depth using separate image-only approaches lead to unstable and noisy depth estimates, harming 3D detections. In this paper, we propose a novel keypoint-based approach for 3D object detection and localization from a single RGB image. We build our multi-branch model around 2D keypoint detection in images and complement it with a conceptually simple geometric reasoning method. Our network performs in an end-to-end manner, simultaneously and interdependently estimating 2D characteristics, such as 2D bounding boxes, keypoints, and orientation, along with full 3D pose in the scene. We fuse the outputs of distinct branches, applying a reprojection consistency loss during training. The experimental evaluation on the challenging KITTI dataset benchmark demonstrates that our network achieves state-of-the-art results among other monocular 3D detectors. |
| Tasks | 3D Object Detection, Keypoint Detection, Object Detection |
| Published | 2019-05-14 |
| URL | https://arxiv.org/abs/1905.05618v1 |
| https://arxiv.org/pdf/1905.05618v1.pdf | |
| PWC | https://paperswithcode.com/paper/monocular-3d-object-detection-via-geometric |
| Repo | |
| Framework | |
Synthesizing a 4D Spatio-Angular Consistent Light Field from a Single Image
| Title | Synthesizing a 4D Spatio-Angular Consistent Light Field from a Single Image |
| Authors | Andre Ivan, Williem, In Kyu Park |
| Abstract | Synthesizing a densely sampled light field from a single image is highly beneficial for many applications. The conventional method reconstructs a depth map and relies on physical-based rendering and a secondary network to improve the synthesized novel views. Simple pixel-based loss also limits the network by making it rely on pixel intensity cue rather than geometric reasoning. In this study, we show that a different geometric representation, namely, appearance flow, can be used to synthesize a light field from a single image robustly and directly. A single end-to-end deep neural network that does not require a physical-based approach nor a post-processing subnetwork is proposed. Two novel loss functions based on known light field domain knowledge are presented to enable the network to preserve the spatio-angular consistency between sub-aperture images effectively. Experimental results show that the proposed model successfully synthesizes dense light fields and qualitatively and quantitatively outperforms the previous model . The method can be generalized to arbitrary scenes, rather than focusing on a particular class of object. The synthesized light field can be used for various applications, such as depth estimation and refocusing. |
| Tasks | Depth Estimation |
| Published | 2019-03-29 |
| URL | http://arxiv.org/abs/1903.12364v1 |
| http://arxiv.org/pdf/1903.12364v1.pdf | |
| PWC | https://paperswithcode.com/paper/synthesizing-a-4d-spatio-angular-consistent |
| Repo | |
| Framework | |
TensorMap: Lidar-Based Topological Mapping and Localization via Tensor Decompositions
| Title | TensorMap: Lidar-Based Topological Mapping and Localization via Tensor Decompositions |
| Authors | Sirisha Rambhatla, Nikos D. Sidiropoulos, Jarvis Haupt |
| Abstract | We propose a technique to develop (and localize in) topological maps from light detection and ranging (Lidar) data. Localizing an autonomous vehicle with respect to a reference map in real-time is crucial for its safe operation. Owing to the rich information provided by Lidar sensors, these are emerging as a promising choice for this task. However, since a Lidar outputs a large amount of data every fraction of a second, it is progressively harder to process the information in real-time. Consequently, current systems have migrated towards faster alternatives at the expense of accuracy. To overcome this inherent trade-off between latency and accuracy, we propose a technique to develop topological maps from Lidar data using the orthogonal Tucker3 tensor decomposition. Our experimental evaluations demonstrate that in addition to achieving a high compression ratio as compared to full data, the proposed technique, $\textit{TensorMap}$, also accurately detects the position of the vehicle in a graph-based representation of a map. We also analyze the robustness of the proposed technique to Gaussian and translational noise, thus initiating explorations into potential applications of tensor decompositions in Lidar data analysis. |
| Tasks | |
| Published | 2019-02-26 |
| URL | http://arxiv.org/abs/1902.10226v1 |
| http://arxiv.org/pdf/1902.10226v1.pdf | |
| PWC | https://paperswithcode.com/paper/tensormap-lidar-based-topological-mapping-and |
| Repo | |
| Framework | |
