Paper Group ANR 440
Optimizing Privacy-Preserving Outsourced Convolutional Neural Network Predictions. Bone Suppression on Chest Radiographs With Adversarial Learning. Cross-modal Deep Face Normals with Deactivable Skip Connections. Semi-Supervised Semantic Segmentation with Cross-Consistency Training. On Information Plane Analyses of Neural Network Classifiers – A R …
Optimizing Privacy-Preserving Outsourced Convolutional Neural Network Predictions
| Title | Optimizing Privacy-Preserving Outsourced Convolutional Neural Network Predictions |
| Authors | Minghui Li, Sherman S. M. Chow, Shengshan Hu, Yuejing Yan, Minxin Du, Zhibo Wang |
| Abstract | Neural networks provide better prediction performance than previous techniques. Prediction-as-a-service thus becomes popular, especially in the outsourced setting since it involves extensive computation. Recent researches focus on the privacy of the query and results, but they do not provide model privacy against the model-hosting server and may leak partial information about the results. Some of them further require frequent interactions with the querier or heavy computation overheads. This paper proposes a new scheme for privacy-preserving neural network prediction in the outsourced setting, i.e., the server cannot learn the query, (intermediate) results, and the model. Similar to SecureML (S&P’17), a representative work which provides model privacy, we leverage two non-colluding servers with secret sharing and triplet generation to minimize the usage of heavyweight cryptography. Further, we adopt asynchronous computation to improve the throughput, and design garbled circuits for the non-polynomial activation function to keep the same accuracy as the underlying network (instead of approximating it). Our experiments on four neural network architectures show that our scheme achieves an average of 282 improvements in reducing latency compared to SecureML. Compared to MiniONN (CCS’17) and EzPC (EuroS&P’19), both without model privacy, our scheme achieves a lower latency by a factor of 18 and 10, respectively. For the communication costs, our scheme outperforms SecureML by 122, MiniONN by 49, and EzPC by 38 times. |
| Tasks | |
| Published | 2020-02-22 |
| URL | https://arxiv.org/abs/2002.10944v2 |
| https://arxiv.org/pdf/2002.10944v2.pdf | |
| PWC | https://paperswithcode.com/paper/optimizing-privacy-preserving-outsourced |
| Repo | |
| Framework | |
Bone Suppression on Chest Radiographs With Adversarial Learning
| Title | Bone Suppression on Chest Radiographs With Adversarial Learning |
| Authors | Jia Liang, Yuxing Tang, Youbao Tang, Jing Xiao, Ronald M. Summers |
| Abstract | Dual-energy (DE) chest radiography provides the capability of selectively imaging two clinically relevant materials, namely soft tissues, and osseous structures, to better characterize a wide variety of thoracic pathology and potentially improve diagnosis in posteroanterior (PA) chest radiographs. However, DE imaging requires specialized hardware and a higher radiation dose than conventional radiography, and motion artifacts sometimes happen due to involuntary patient motion. In this work, we learn the mapping between conventional radiographs and bone suppressed radiographs. Specifically, we propose to utilize two variations of generative adversarial networks (GANs) for image-to-image translation between conventional and bone suppressed radiographs obtained by DE imaging technique. We compare the effectiveness of training with patient-wisely paired and unpaired radiographs. Experiments show both training strategies yield “radio-realistic’’ radiographs with suppressed bony structures and few motion artifacts on a hold-out test set. While training with paired images yields slightly better performance than that of unpaired images when measuring with two objective image quality metrics, namely Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), training with unpaired images demonstrates better generalization ability on unseen anteroposterior (AP) radiographs than paired training. |
| Tasks | Image-to-Image Translation |
| Published | 2020-02-08 |
| URL | https://arxiv.org/abs/2002.03073v1 |
| https://arxiv.org/pdf/2002.03073v1.pdf | |
| PWC | https://paperswithcode.com/paper/bone-suppression-on-chest-radiographs-with |
| Repo | |
| Framework | |
Cross-modal Deep Face Normals with Deactivable Skip Connections
| Title | Cross-modal Deep Face Normals with Deactivable Skip Connections |
| Authors | Victoria Fernandez Abrevaya, Adnane Boukhayma, Philip H. S. Torr, Edmond Boyer |
| Abstract | We present an approach for estimating surface normals from in-the-wild color images of faces. While data-driven strategies have been proposed for single face images, limited available ground truth data makes this problem difficult. To alleviate this issue, we propose a method that can leverage all available image and normal data, whether paired or not, thanks to a novel cross-modal learning architecture. In particular, we enable additional training with single modality data, either color or normal, by using two encoder-decoder networks with a shared latent space. The proposed architecture also enables face details to be transferred between the image and normal domains, given paired data, through skip connections between the image encoder and normal decoder. Core to our approach is a novel module that we call deactivable skip connections, which allows integrating both the auto-encoded and image-to-normal branches within the same architecture that can be trained end-to-end. This allows learning of a rich latent space that can accurately capture the normal information. We compare against state-of-the-art methods and show that our approach can achieve significant improvements, both quantitative and qualitative, with natural face images. |
| Tasks | |
| Published | 2020-03-21 |
| URL | https://arxiv.org/abs/2003.09691v2 |
| https://arxiv.org/pdf/2003.09691v2.pdf | |
| PWC | https://paperswithcode.com/paper/cross-modal-deep-face-normals-with |
| Repo | |
| Framework | |
Semi-Supervised Semantic Segmentation with Cross-Consistency Training
| Title | Semi-Supervised Semantic Segmentation with Cross-Consistency Training |
| Authors | Yassine Ouali, Céline Hudelot, Myriam Tami |
| Abstract | In this paper, we present a novel cross-consistency based semi-supervised approach for semantic segmentation. Consistency training has proven to be a powerful semi-supervised learning framework for leveraging unlabeled data under the cluster assumption, in which the decision boundary should lie in low-density regions. In this work, we first observe that for semantic segmentation, the low-density regions are more apparent within the hidden representations than within the inputs. We thus propose cross-consistency training, where an invariance of the predictions is enforced over different perturbations applied to the outputs of the encoder. Concretely, a shared encoder and a main decoder are trained in a supervised manner using the available labeled examples. To leverage the unlabeled examples, we enforce a consistency between the main decoder predictions and those of the auxiliary decoders, taking as inputs different perturbed versions of the encoder’s output, and consequently, improving the encoder’s representations. The proposed method is simple and can easily be extended to use additional training signal, such as image-level labels or pixel-level labels across different domains. We perform an ablation study to tease apart the effectiveness of each component, and conduct extensive experiments to demonstrate that our method achieves state-of-the-art results in several datasets. |
| Tasks | Semantic Segmentation, Semi-Supervised Semantic Segmentation |
| Published | 2020-03-19 |
| URL | https://arxiv.org/abs/2003.09005v1 |
| https://arxiv.org/pdf/2003.09005v1.pdf | |
| PWC | https://paperswithcode.com/paper/semi-supervised-semantic-segmentation-with-1 |
| Repo | |
| Framework | |
On Information Plane Analyses of Neural Network Classifiers – A Review
| Title | On Information Plane Analyses of Neural Network Classifiers – A Review |
| Authors | Bernhard C. Geiger |
| Abstract | We review the current literature concerned with information plane analyses of neural network classifiers. While the underlying information bottleneck theory and the claim that information-theoretic compression is causally linked to generalization are plausible, empirical evidence was found to be both supporting and conflicting. We review this evidence together with a detailed analysis how the respective information quantities were estimated. Our analysis suggests that compression visualized in information planes is not information-theoretic, but is rather compatible with geometric compression of the activations. |
| Tasks | |
| Published | 2020-03-21 |
| URL | https://arxiv.org/abs/2003.09671v1 |
| https://arxiv.org/pdf/2003.09671v1.pdf | |
| PWC | https://paperswithcode.com/paper/on-information-plane-analyses-of-neural |
| Repo | |
| Framework | |
Privacy-Preserving Deep Learning Computation for Geo-Distributed Medical Big-Data Platforms
| Title | Privacy-Preserving Deep Learning Computation for Geo-Distributed Medical Big-Data Platforms |
| Authors | Joohyung Jeon, Junhui Kim, Joongheon Kim, Kwangsoo Kim, Aziz Mohaisen, Jong-Kook Kim |
| Abstract | This paper proposes a distributed deep learning framework for privacy-preserving medical data training. In order to avoid patients’ data leakage in medical platforms, the hidden layers in the deep learning framework are separated and where the first layer is kept in platform and others layers are kept in a centralized server. Whereas keeping the original patients’ data in local platforms maintain their privacy, utilizing the server for subsequent layers improves learning performance by using all data from each platform during training. |
| Tasks | Privacy Preserving Deep Learning |
| Published | 2020-01-09 |
| URL | https://arxiv.org/abs/2001.02932v1 |
| https://arxiv.org/pdf/2001.02932v1.pdf | |
| PWC | https://paperswithcode.com/paper/privacy-preserving-deep-learning-computation |
| Repo | |
| Framework | |
Hierarchical Severity Staging of Anterior Cruciate Ligament Injuries using Deep Learning
| Title | Hierarchical Severity Staging of Anterior Cruciate Ligament Injuries using Deep Learning |
| Authors | Nikan K. Namiri, Io Flament, Bruno Astuto, Rutwik Shah, Radhika Tibrewala, Francesco Caliva, Thomas M. Link, Valentina Pedoia, Sharmila Majumdar |
| Abstract | Purpose: To evaluate diagnostic utility of two convolutional neural networks (CNNs) for severity staging anterior cruciate ligament (ACL) injuries. Materials and Methods: This retrospective analysis was conducted on 1243 knee MR images (1008 intact, 18 partially torn, 77 fully torn, 140 reconstructed ACLs) from 224 subjects collected between 2011 and 2014 (age=46.50+-13.55 years, body mass index=24.58+-3.60 kg/m2, 46% women (mean+-standard deviation). Images were acquired with a 3.0T MR scanner using 3D fast spin echo CUBE-sequences. The radiologists used a modified scoring metric analagous to the ACLOAS and WORMS for grading standard. To classify ACL injuries with deep learning, two types of CNNs were used, one with three-dimensional (3D) and the other with two-dimensional (2D) convolutional kernels. Performance metrics included sensitivity, specificity, weighted Cohen’s kappa, and overall accuracy, followed by two-sample t-tests to compare CNN performance. Results: The overall accuracy (84%) and weighted Cohen’s kappa (.92) reported for ACL injury classification were higher using the 2D CNN than the 3D CNN. The 2D CNN and 3D CNN performed similarly in assessing intact ACLs (2D CNN: 93% sensitivity and 90% specificity, 3D CNN: 89% sensitivity and 88% specificity). Classification of full tears by both networks were also comparable (2D CNN: 83% sensitivity and 94% specificity, 3D CNN: 77% sensitivity and 100% sensitivity). The 2D CNN classified all reconstructed ACLs correctly. Conclusion: CNNs applied to ACL lesion classification results in high sensitivity and specificity, leading to potential use in helping grade ACL injuries by non-experts. |
| Tasks | |
| Published | 2020-03-20 |
| URL | https://arxiv.org/abs/2003.09089v1 |
| https://arxiv.org/pdf/2003.09089v1.pdf | |
| PWC | https://paperswithcode.com/paper/hierarchical-severity-staging-of-anterior |
| Repo | |
| Framework | |
A mean-field analysis of two-player zero-sum games
| Title | A mean-field analysis of two-player zero-sum games |
| Authors | Carles Domingo-Enrich, Samy Jelassi, Arthur Mensch, Grant Rotskoff, Joan Bruna |
| Abstract | Finding Nash equilibria in two-player zero-sum continuous games is a central problem in machine learning, e.g. for training both GANs and robust models. The existence of pure Nash equilibria requires strong conditions which are not typically met in practice. Mixed Nash equilibria exist in greater generality and may be found using mirror descent. Yet this approach does not scale to high dimensions. To address this limitation, we parametrize mixed strategies as mixtures of particles, whose positions and weights are updated using gradient descent-ascent. We study this dynamics as an interacting gradient flow over measure spaces endowed with the Wasserstein-Fisher-Rao metric. We establish global convergence to an approximate equilibrium for the related Langevin gradient-ascent dynamic. We prove a law of large numbers that relates particle dynamics to mean-field dynamics. Our method identifies mixed equilibria in high dimensions and is demonstrably effective for training mixtures of GANs. |
| Tasks | |
| Published | 2020-02-14 |
| URL | https://arxiv.org/abs/2002.06277v2 |
| https://arxiv.org/pdf/2002.06277v2.pdf | |
| PWC | https://paperswithcode.com/paper/a-mean-field-analysis-of-two-player-zero-sum |
| Repo | |
| Framework | |
Applications of Deep Learning for Ill-Posed Inverse Problems Within Optical Tomography
| Title | Applications of Deep Learning for Ill-Posed Inverse Problems Within Optical Tomography |
| Authors | Adam Peace |
| Abstract | Increasingly in medical imaging has emerged an issue surrounding the reconstruction of noisy images from raw measurement data. Where the forward problem is the generation of raw measurement data from a ground truth image, the inverse problem is the reconstruction of those images from the measurement data. In most cases with medical imaging, classical inverse Radon transforms, such as an inverse Fourier transform for MRI, work well for recovering clean images from the measured data. Unfortunately in the case of X-Ray CT, where undersampled data is very common, more than this is needed to resolve faithful and usable images. In this paper, we explore the history of classical methods for solving the inverse problem for X-Ray CT, followed by an analysis of the state of the art methods that utilize supervised deep learning. Finally, we will provide some possible avenues for research in the future. |
| Tasks | |
| Published | 2020-03-21 |
| URL | https://arxiv.org/abs/2003.09647v1 |
| https://arxiv.org/pdf/2003.09647v1.pdf | |
| PWC | https://paperswithcode.com/paper/applications-of-deep-learning-for-ill-posed |
| Repo | |
| Framework | |
The Early Phase of Neural Network Training
| Title | The Early Phase of Neural Network Training |
| Authors | Jonathan Frankle, David J. Schwab, Ari S. Morcos |
| Abstract | Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here, we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of the network state during these early iterations of training and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only a few hundred iterations. Despite this behavior, pre-training with blurred inputs or an auxiliary self-supervised task can approximate the changes in supervised networks, suggesting that these changes are not inherently label-dependent, though labels significantly accelerate this process. Together, these results help to elucidate the network changes occurring during this pivotal initial period of learning. |
| Tasks | |
| Published | 2020-02-24 |
| URL | https://arxiv.org/abs/2002.10365v1 |
| https://arxiv.org/pdf/2002.10365v1.pdf | |
| PWC | https://paperswithcode.com/paper/the-early-phase-of-neural-network-training-1 |
| Repo | |
| Framework | |
Pre-training Tasks for Embedding-based Large-scale Retrieval
| Title | Pre-training Tasks for Embedding-based Large-scale Retrieval |
| Authors | Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, Sanjiv Kumar |
| Abstract | We consider the large-scale query-document retrieval problem: given a query (e.g., a question), return the set of relevant documents (e.g., paragraphs containing the answer) from a large document corpus. This problem is often solved in two steps. The retrieval phase first reduces the solution space, returning a subset of candidate documents. The scoring phase then re-ranks the documents. Critically, the retrieval algorithm not only desires high recall but also requires to be highly efficient, returning candidates in time sublinear to the number of documents. Unlike the scoring phase witnessing significant advances recently due to the BERT-style pre-training tasks on cross-attention models, the retrieval phase remains less well studied. Most previous works rely on classic Information Retrieval (IR) methods such as BM-25 (token matching + TF-IDF weights). These models only accept sparse handcrafted features and can not be optimized for different downstream tasks of interest. In this paper, we conduct a comprehensive study on the embedding-based retrieval models. We show that the key ingredient of learning a strong embedding-based Transformer model is the set of pre-training tasks. With adequately designed paragraph-level pre-training tasks, the Transformer models can remarkably improve over the widely-used BM-25 as well as embedding models without Transformers. The paragraph-level pre-training tasks we studied are Inverse Cloze Task (ICT), Body First Selection (BFS), Wiki Link Prediction (WLP), and the combination of all three. |
| Tasks | Information Retrieval, Link Prediction |
| Published | 2020-02-10 |
| URL | https://arxiv.org/abs/2002.03932v1 |
| https://arxiv.org/pdf/2002.03932v1.pdf | |
| PWC | https://paperswithcode.com/paper/pre-training-tasks-for-embedding-based-large |
| Repo | |
| Framework | |
Regression with Deep Learning for Sensor Performance Optimization
| Title | Regression with Deep Learning for Sensor Performance Optimization |
| Authors | Ruthvik Vaila, Denver Lloyd, Kevin Tetz |
| Abstract | Neural networks with at least two hidden layers are called deep networks. Recent developments in AI and computer programming in general has led to development of tools such as Tensorflow, Keras, NumPy etc. making it easier to model and draw conclusions from data. In this work we re-approach non-linear regression with deep learning enabled by Keras and Tensorflow. In particular, we use deep learning to parametrize a non-linear multivariate relationship between inputs and outputs of an industrial sensor with an intent to optimize the sensor performance based on selected key metrics. |
| Tasks | |
| Published | 2020-02-22 |
| URL | https://arxiv.org/abs/2002.11044v1 |
| https://arxiv.org/pdf/2002.11044v1.pdf | |
| PWC | https://paperswithcode.com/paper/regression-with-deep-learning-for-sensor |
| Repo | |
| Framework | |
Robust Generative Restricted Kernel Machines using Weighted Conjugate Feature Duality
| Title | Robust Generative Restricted Kernel Machines using Weighted Conjugate Feature Duality |
| Authors | Arun Pandey, Joachim Schreurs, Johan A. K. Suykens |
| Abstract | Interest in generative models has grown tremendously in the past decade. However, their training performance can be adversely affected by contamination, where outliers are encoded in the representation of the model. This results in the generation of noisy data. In this paper, we introduce weighted conjugate feature duality in the framework of Restricted Kernel Machines (RKMs). The RKM formulation allows for an easy integration of methods from classical robust statistics. This formulation is used to fine-tune the latent space of generative RKMs using a weighting function based on the Minimum Covariance Determinant, which is a highly robust estimator of multivariate location and scatter. Experiments show that the weighted RKM is capable of generating clean images when contamination is present in the training data. We further show that the robust method also preserves uncorrelated feature learning through qualitative and quantitative experiments on standard datasets. |
| Tasks | |
| Published | 2020-02-04 |
| URL | https://arxiv.org/abs/2002.01180v2 |
| https://arxiv.org/pdf/2002.01180v2.pdf | |
| PWC | https://paperswithcode.com/paper/robust-generative-restricted-kernel-machines |
| Repo | |
| Framework | |
Automated Deep Abstractions for Stochastic Chemical Reaction Networks
| Title | Automated Deep Abstractions for Stochastic Chemical Reaction Networks |
| Authors | Tatjana Petrov, Denis Repin |
| Abstract | Predicting stochastic cellular dynamics as emerging from the mechanistic models of molecular interactions is a long-standing challenge in systems biology: low-level chemical reaction network (CRN) models give raise to a highly-dimensional continuous-time Markov chain (CTMC) which is computationally demanding and often prohibitive to analyse in practice. A recently proposed abstraction method uses deep learning to replace this CTMC with a discrete-time continuous-space process, by training a mixture density deep neural network with traces sampled at regular time intervals (which can obtained either by simulating a given CRN or as time-series data from experiment). The major advantage of such abstraction is that it produces a computational model that is dramatically cheaper to execute, while preserving the statistical features of the training data. In general, the abstraction accuracy improves with the amount of training data. However, depending on a CRN, the overall quality of the method – the efficiency gain and abstraction accuracy – will also depend on the choice of neural network architecture given by hyper-parameters such as the layer types and connections between them. As a consequence, in practice, the modeller would have to take care of finding the suitable architecture manually, for each given CRN, through a tedious and time-consuming trial-and-error cycle. In this paper, we propose to further automatise deep abstractions for stochastic CRNs, through learning the optimal neural network architecture along with learning the transition kernel of the abstract process. Automated search of the architecture makes the method applicable directly to any given CRN, which is time-saving for deep learning experts and crucial for non-specialists. We implement the method and demonstrate its performance on a number of representative CRNs with multi-modal emergent phenotypes. |
| Tasks | Time Series |
| Published | 2020-01-30 |
| URL | https://arxiv.org/abs/2002.01889v1 |
| https://arxiv.org/pdf/2002.01889v1.pdf | |
| PWC | https://paperswithcode.com/paper/automated-deep-abstractions-for-stochastic |
| Repo | |
| Framework | |
Implicit Geometric Regularization for Learning Shapes
| Title | Implicit Geometric Regularization for Learning Shapes |
| Authors | Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, Yaron Lipman |
| Abstract | Representing shapes as level sets of neural networks has been recently proved to be useful for different shape analysis and reconstruction tasks So far, such representations were computed using either: (i) pre-computed implicit shape representations; or (ii) loss functions explicitly defined over the neural level sets. In this paper we offer a new paradigm for computing high fidelity implicit neural representations directly from raw data (i.e., point clouds, with or without normal information). We observe that a rather simple loss function, encouraging the neural network to vanish on the input point cloud and to have a unit norm gradient, possesses an implicit geometric regularization property that favors smooth and natural zero level set surfaces, avoiding bad zero-loss solutions. We provide a theoretical analysis of this property for the linear case, and show that, in practice, our method leads to state of the art implicit neural representations with higher level-of-details and fidelity compared to previous methods. |
| Tasks | |
| Published | 2020-02-24 |
| URL | https://arxiv.org/abs/2002.10099v1 |
| https://arxiv.org/pdf/2002.10099v1.pdf | |
| PWC | https://paperswithcode.com/paper/implicit-geometric-regularization-for |
| Repo | |
| Framework | |
