Incorporating Nesterov Momentum into Adam

Abstract:
This work aims to improve upon the recently proposed and rapidly popular-
ized optimization algorithm Adam (Kingma & Ba, 2014). Adam has two main
components—a momentum component and an adaptive learning rate component.
However, regular momentum can be shown conceptually and empirically to be in-
ferior to a similar algorithm known as Nesterov’s accelerated gradient (NAG). We
show how to modify Adam’s momentum component to take advantage of insights
from NAG, and then we present preliminary evidence suggesting that making this
substitution improves the speed of convergence and the quality of the learned mod-
els.

Title of paper.

Abstract of paper.


Neural Variational Inference for Text Processing

Abstract: Recent advances in neural variational inference have spawned a renaissance in deep latent variable models. In this paper we introduce a generic variational inference framework for generative and conditional models of text. While traditional variational methods derive an analytic approximation for the intractable distributions over latent variables, here we construct an inference network conditioned on the discrete text input to provide the variational distribution. We validate this framework on two very different text modelling applications, generative document modelling and supervised question answering. Our neural variational document model combines a continuous stochastic document representation with a bag-of-words generative model and achieves the lowest reported perplexities on two standard test corpora. The neural answer selection model employs a stochastic representation layer within an attention mechanism to extract the semantics between a question and answer pair. On two question answering benchmarks this model exceeds all previous published benchmarks.

Your displayed identity associated with the above content.

The users who will be allowed to read the above content.

Either upload a PDF file or provide a direct link to your PDF on ArXiv.

Title of paper.

Abstract of paper.

Comma separated list of author names, as they appear in the paper.

Comma separated list of author email addresses, in the same order as above.

Semi-colon separated list of email domains of people who would have a conflict of interest in reviewing this paper, (e.g., cs.umass.edu;google.com, etc.).

If the paper is a resubmission from the ICLR 2016 Conference Track, enter its CMT ID; otherwise, leave blank.


ADDING GRADIENT NOISE IMPROVES LEARNING FOR VERY DEEP NETWORKS

Abstract: Deep feedforward and recurrent networks have achieved impressive results in many perception and language processing applications. This success is partially attributed to architectural innovations such as convolutional and long short-term memory networks. A major reason for these architectural innovations is that they capture better domain knowledge, and importantly are easier to optimize than more basic architectures. Recently, more complex architectures such as Neural Turing Machines and Memory Networks have been proposed for tasks including question answering and general computation, creating a new set of optimization challenges. In this paper, we discuss a low-overhead and easy-to-implement technique of adding gradient noise which we find to be surprisingly effective when training these very deep architectures. The technique not only helps to avoid overfitting, but also can result in lower training loss. This method alone allows a fully-connected 20-layer deep network to be trained with standard gradient descent, even starting from a poor initialization. We see consistent improvements for many complex models, including a 72% relative reduction in error rate over a carefully-tuned baseline on a challenging question-answering task, and a doubling of the number of accurate binary multiplication models learned across 7,000 random restarts. We encourage further application of this technique to additional complex modern architectures.

Arvind Neelakantan
Luke Vilnis
Quoc V. Le
Ilya Sutskever
Lukasz Kaiser
Karol Kurach
James Martens

Title of paper.

Abstract of paper.


Deep Motif: Visualizing Genomic Sequence Classifications

Abstract:
This paper applies a deep convolutional/highway MLP framework to classify genomic
sequences on the transcription factor binding site task. To make the model
understandable, we propose an optimization driven strategy to extract “motifs”, or
symbolic patterns which visualize the positive class learned by the network. We
show that our system, Deep Motif (DeMo), extracts motifs that are similar to, and
in some cases outperform the current well known motifs. In addition, we find that
a deeper model consisting of multiple convolutional and highway layers can outperform
a single convolutional and fully connected layer in the previous state-of-the-art.

Jack Lanchantin
Ritambhara Singh
Zeming Lin
& Yanjun Qi

Title of paper.

Abstract of paper.


Lessons from the Rademacher Complexity for Deep Learning

Abstract: Understanding the generalization properties of deep learning models is critical for successful applications, especially in the regimes where the number of training samples is limited. We study the generalization properties of deep neural networks via the empirical Rademacher complexity and show that it is easier to control the complexity of  convolutional networks compared to general fully connected networks. In particular, we justify the usage of small convolutional kernels in deep networks as they lead to a better generalization error. Moreover, we propose a representation based regularization method that allows to decrease the generalization error  by controlling the coherence of the representation. Experiments on the MNIST dataset support these foundations.

Jure Sokolic
Raja Giryes
Guillermo Sapiro
Miguel R. D. Rodrigues

Title of paper.

Abstract of paper.


Temporal Convolutional Neural Networks for Diagnosis from Lab Tests

Abstract: Early diagnosis of treatable diseases is essential for improving healthcare, and many diseases' onsets are predictable from annual lab tests and their temporal trends. We introduce a multi-resolution convolutional neural network for early detection of multiple diseases from irregularly measured sparse lab values. Our novel architecture takes as input both an imputed version of the data and a binary observation matrix. For imputing the temporal sparse observations, we develop a flexible, fast to train method for differentiable multivariate kernel regression. Our experiments on data from 298K individuals over 8 years, 18 common lab measurements, and 171 diseases show that the temporal signatures learned via convolution are significantly more predictive than baselines commonly used for early disease diagnosis.

Title of paper.

Abstract of paper.


Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems

Abstract: We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.

Title of paper.

Abstract of paper.


Neural Enquirer: Learning to Query Tables in Natural Language

Abstract: We propose Neural Enquirer — a neural network architecture for answering natural language (NL) questions given a knowledge base (KB) table. Unlike previous work on end-to-end training of semantic parsers, Neural Enquirer is fully “neuralized”: it gives distributed representations of queries and KB tables, and executes queries through a series of differentiable operations. The model can be trained with gradient descent using both end-to-end and step-by-step supervision. During training the representations of queries and the KB table are jointly optimized with the query execution logic. Our experiments show that the model can learn to execute complex NL queries on KB tables with rich structures.

Pengcheng Yin
Zhengdong Lu
Hang Li
Ben Kao

Title of paper.

Abstract of paper.


Document Context Language Models

Abstract: Text documents are structured on multiple levels of detail: individual words are related by syntax, and larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and generate coherent texts. We present and empirically evaluate a set of multi-level recurrent neural network language models, called Document-Context Language Models (DCLMs), which incorporate contextual information both within and beyond the sentence. In comparison with sentence-level recurrent neural network language models, the DCLMs obtain slightly better predictive likelihoods, and considerably better assessments of document coherence.

Yangfeng Ji
Trevor Cohn
Lingpeng Kong
Chris Dyer
Jacob Eisenstein

Title of paper.

Abstract of paper.


Convolutional Monte Carlo Rollouts for the Game of Go

Abstract: In this work, we present a Monte Carlo tree search-based program for playing Go which uses convolutional rollouts. Our method performs MCTS in batches, explores the Monte Carlo tree using Thompson sampling and a convolutional policy network, and evaluates convnet-based rollouts on the GPU. We achieve strong win rates against an open source Go program and attain competitive results against state of the art convolutional net-based Go-playing programs.

Title of paper.

Abstract of paper.


Neural Generative Question Answering

Abstract: This paper presents an end-to-end neural network model, named Neural Generative Question Answering (genQA), that can generate answers to simple factoid questions, both in natural language. More specifically, the model is built on the encoder-decoder framework for sequence-to-sequence learning, while equipped with the ability to access an embedded knowledge-base through an attention-like mechanism. The model is trained on a corpus of question-answer pairs, with their associated triples in the given knowledge-base. Empirical study shows the proposed model can effectively deal with the language variation of the question and generate a right answer by referring to the facts in the knowledge-base. The experiment on question answering demonstrates that the proposed model can outperform the embedding-based QA model as well as the neural dialogue models trained on the same data.

Jun Yin
Xin Jiang
Zhengdong Lu
Lifeng Shang
Hang Li
Xiaoming Li

Title of paper.

Abstract of paper.


Deep Bayesian Neural Nets as Deep Matrix Gaussian Processes

Abstract: We show that by employing a distribution over random matrices, the matrix variate Gaussian~\cite{gupta1999matrix}, for the neural network parameters we can obtain a non-parametric interpretation for the hidden units after the application of the ``local reprarametrization trick"~\citep{kingma2015variational}. This provides a nice duality between Bayesian neural networks and deep Gaussian Processes~\cite{damianou2012deep}, a property that was also shown by~\cite{gal2015dropout}. We show that we can borrow ideas from the Gaussian Process literature so as to exploit the non-parametric properties of such a model. We empirically verified this model on a regression task.

Title of paper.

Abstract of paper.


Mixtures of Sparse Autoregressive Networks

Abstract: We consider high-dimensional distribution estimation through autoregressive networks. By combining the concepts of sparsity, mixtures and parameter sharing we obtain a simple model which is fast to train and which achieves state-of-the-art or better results on several standard benchmark datasets. Specifically, we use an L1-penalty to regularize the conditional distributions and introduce a procedure for automatic parameter sharing between mixture components. Moreover, we propose a simple distributed representation which permits exact likelihood evaluations since the latent variables are interleaved with the observable variables and can be easily integrated out. Our model achieves excellent generalization performance and scales well to extremely high dimensions.

Title of paper.

Abstract of paper.


ParseNet: Looking Wider to See Better

Abstract: We present a technique for adding global context to fully convolutional networks for semantic segmentation. The approach is simple, using the average feature for a layer to augment the features at each location. In addition, we study several idiosyncrasies of training, significantly increasing the performance of baseline networks (e.g. from FCN~\cite{long2014fully}). When we add our proposed global feature, and a technique for learning normalization parameters, accuracy increases consistently even over our improved versions of the baselines. Our proposed approach, ParseNet, achieves state-of-the-art performance on SiftFlow and PASCAL-Context with small additional computational cost over baselines, and near state-of-the-art performance on PASCAL VOC 2012 semantic segmentation with a simple approach. Code is available at \url{https://github.com/weiliu89/caffe/tree/fcn} .

Wei Liu
Andrew Rabinovich
Alexander C. Berg

Title of paper.

Abstract of paper.


Unitary Evolution Recurrent Neural Networks

Abstract: Recurrent neural networks (RNNs) are notoriously difficult to train. When the eigenvalues of the hidden to hidden weight matrix deviate from absolute value 1, optimization becomes difficult due to the well studied issue of vanishing and exploding gradients, especially when trying to learn long-term dependencies. To circumvent this problem, we propose a new architecture that learns a unitary weight matrix, with eigenvalues of absolute value exactly 1. The challenge we address is that of parametrizing unitary matrices in a way that does not require expensive computations (such as eigendecomposition) after each weight update. We construct an expressive unitary weight matrix by composing several structured matrices that act as building blocks with parameters to be learned. Optimization with this parameterization becomes feasible only when considering hidden states in the complex domain. We demonstrate the potential of this architecture by achieving state of the art results in several hard tasks involving very long-term dependencies.

Title of paper.

Abstract of paper.


Variance Reduction in SGD by Distributed Importance Sampling

Abstract:
Humans are able to accelerate their learning by selecting training
materials that are the most informative and at the appropriate level of
difficulty.  We propose a framework for distributing deep learning in which
one set of workers search for the most informative examples in parallel
while a single worker updates the model on examples selected by importance
sampling.  This leads the model to update using an unbiased estimate of the
gradient which also has minimum variance when the sampling proposal is
proportional to the L2-norm of the gradient. We show experimentally that
this method reduces gradient variance even in a context where the cost of
synchronization across machines cannot be ignored, and where the factors
for importance sampling are not updated instantly across the training set.

Guillaume Alain
Alex Lamb
Chinnadhurai Sankar
Aaron Courville
Yoshua Bengio

Title of paper.

Abstract of paper.


Adaptive Natural Gradient Learning Based on Riemannian Metric of Score Matching

Abstract: The natural gradient is a powerful method to improve the transient dynamics of learning by considering the geometric structure of the parameter space. Many natural gradient methods have been developed with regards to Kullback-Leibler (KL) divergence and its Fisher metric, but the framework of natural gradient can be essentially extended to other divergences. In this study, we focus on score matching, which is an alternative to maximum likelihood learning for unnormalized statistical models, and introduce its Riemannian metric. By using the score matching metric, we derive an adaptive natural gradient algorithm that does not require computationally demanding inversion of the metric. Experimental results in a multi-layer neural network model demonstrate that the proposed method avoids the plateau phenomenon and accelerates the convergence of learning compared to the conventional stochastic gradient descent method.

Ryo Karakida
Masato Okada
Shun-ichi Amari

Title of paper.

Abstract of paper.


A Differentiable Transition Between Additive and Multiplicative Neurons

Abstract:
Existing approaches to combine both additive and multiplicative neural units either use a fixed assignment of operations or require discrete optimization to determine what function a neuron should perform. However, this leads to an extensive increase in the computational complexity of the training procedure.

We present a novel, parameterizable transfer function based on the mathematical concept of non-integer functional iteration that allows the operation each neuron performs to be smoothly and, most importantly, differentiablely adjusted between addition and multiplication. This allows the decision between addition and multiplication to be integrated into the standard backpropagation training procedure.

Wiebke Koepp
Patrick van der Smagt
Sebastian Urban

Title of paper.

Abstract of paper.


Hardware-Oriented Approximation of Convolutional Neural Networks

Abstract: High computational complexity hinders the widespread usage of Convolutional Neural Networks (CNNs), especially in mobile devices. Hardware accelerators are arguably the most promising approach for reducing both execution time and power consumption. One of the most important steps in accelerator development is hardware-oriented model approximation. In this paper we present Ristretto, a model approximation framework that analyzes a given CNN with respect to numerical resolution used in representing weights and outputs of convolutional and fully connected layers. Ristretto can condense models by using fixed point arithmetic and representation instead of floating point. Moreover, Ristretto fine-tunes the resulting fixed point network. Given a maximum error tolerance of 1%, Ristretto can successfully condense CaffeNet and SqueezeNet to 8-bit. The code for Ristretto is available.

Philipp Gysel
Mohammad Motamedi
Soheil Ghiasi

Title of paper.

Abstract of paper.


Understanding Very Deep Networks via Volume Conservation

Abstract:
Recently, very deep neural networks set new records across many application domains, like Residual Networks at the ImageNet challenge and Highway Networks at language processing tasks. We expect further excellent performance improvements in different fields from these very deep networks. However these networks are still poorly understood, especially since they rely on non-standard architectures.

In this contribution we analyze the learning dynamics which are required for successfully training very deep neural networks. For the analysis we use a symplectic network architecture which inherently conserves volume when mapping a representation from one to the next layer. Therefore it avoids the vanishing gradient problem, which in turn allows to effectively train thousands of layers. We consider highway and residual networks as well as the LSTM model, all of which have approximately volume conserving mappings.

We identified two important factors for making deep architectures working:
(1) (near) volume conserving mappings through $x = x + f(x)$ or similar (cf.\ avoiding the vanishing gradient);
(2) Controlling the drift effect, which increases/decreases $x$ during propagation toward the output (cf.\ avoiding bias shifts);

Title of paper.

Abstract of paper.


Learning to Represent Words in Context with Multilingual Supervision

Abstract:
We present a neural network architecture based on bidirectional LSTMs to compute
representations of words in the sentential contexts. These context-sensitive
word representations are suitable for, e.g., distinguishing different word senses
and other context-modulated variations in meaning. To learn the parameters of
our model, we use cross-lingual supervision, hypothesizing that a good representation
of a word in context will be one that is sufficient for selecting the correct
translation into a second language. We evaluate the quality of our representations
as features in three downstream tasks: prediction of semantic supersenses (which
assign nouns and verbs into a few dozen semantic classes), low resource machine
translation, and a lexical substitution task, and obtain state-of-the-art results on all
of these.

Title of paper.

Abstract of paper.


Guided Sequence-to-Sequence Learning with External Rule Memory

Abstract: External memory has been proven to be essential for the success of neural network-based systems on many tasks, including Question-Answering, classification, machine translation and reasoning. In all those models the memory is used to store instance representations of multiple levels, analogous to “data” in the Von Neumann architecture of a computer, while the “instructions” are stored in the weights. In this paper, we however propose to use the memory for storing part of the instructions, and more specifically, the transformation rules in sequence-to-sequence learning tasks, in an external memory attached to a neural system. This memory can be accessed both by the neural network and by the human experts, hence serving as an interface for a novel learning paradigm where not only the instances but also the rule can be taught to the neural network. Our empirical study on a synthetic but challenging dataset verifies that our model is effective.

Jiatao Gu
Baotian Hu
Zhengdong Lu
Hang Li
Victor O.K. Li

Title of paper.

Abstract of paper.


Convolutional Clustering for Unsupervised Learning

Abstract: The task of labeling data for training deep neural networks is daunting and tedious, requiring millions of labels to achieve the current state-of-the-art results. Such reliance on large amounts of labeled data can be relaxed by exploiting hierarchical features via unsupervised learning techniques. In this work, we propose to train a deep convolutional network based on an enhanced version of the k-means clustering algorithm, which reduces the number of correlated parameters in the form of similar filters, and thus increases test categorization accuracy. We call our algorithm convolutional k-means clustering. We further show that learning the connection between the layers of a deep convolutional neural network improves its ability to be trained on a smaller amount of labeled data. Our experiments show that the proposed algorithm outperforms other techniques that learn filters unsupervised. Specifically, we obtained a test accuracy of 74.1% on STL-10 and a test error of 0.5% on MNIST.

Aysegul Dundar
Jonghoon Jin
and Eugenio Culurciello

Title of paper.

Abstract of paper.


Improving performance of recurrent neural network with relu nonlinearity

Abstract: In recent years significant progress has been made in successfully training recurrent neural networks (RNNs) on sequence learning problems involving long range temporal dependencies. The progress has been made on three fronts: (a) Algorithmic improvements involving sophisticated optimization techniques, (b) network design involving complex hidden layer nodes and specialized recurrent layer connections and (c) weight initialization methods. In this paper, we focus on recently proposed weight initialization with identity matrix for the recurrent weights in a RNN. This initialization is specifically proposed for hidden nodes with Rectified Linear Unit (ReLU) non linearity. We offer a simple dynamical systems perspective on weight initialization process, which allows us to propose a modified weight initialization strategy. We show that this initialization technique leads to successfully training RNNs composed of ReLUs. We demonstrate that our proposal produces comparable or better solution for three toy problems involving long range temporal structure: the addition problem, the multiplication problem and the MNIST classification problem using sequence of pixels. In addition, we present results for a benchmark action recognition problem.

Title of paper.

Abstract of paper.


Why are deep nets reversible: A simple theory, with implications for training

Abstract:
Generative models for  deep learning are promising  both to improve understanding of the model, and yield training methods requiring fewer labeled samples. Recent works use generative model approaches to produce the deep net's input given the value of a hidden layer several levels above.
However, there is no accompanying "proof of correctness" for the generative model, showing that the feedforward deep net is the correct inference method for recovering the hidden layer given the input.  Furthermore, these models are complicated.

The current paper takes a more theoretical tack. It presents a very simple generative model for ReLU deep nets, with the following characteristics:
 (i) The generative model is just the reverse of the feedforward net: if the forward transformation at a layer is $A$ then the reverse transformation is $A^T$. (This can be seen as an explanation of the old weight tying idea for denoising autoencoders.)
  (ii) Its correctness can be proven under a clean theoretical assumption: the edge weights in real-life deep nets behave like random numbers. Under this assumption ---which is experimentally tested on real-life nets like AlexNet--- it is formally proved that feed forward net is a correct inference method for  recovering the hidden layer.

The generative model suggests a simple modification for training: use the generative model to produce synthetic data with labels and  include it in the training set. Experiments are shown to support this theory of random-like deep nets; and that it helps the training.

This extended abstract provides a succinct description of our results while the full paper is available on arXiv.

Title of paper.

Abstract of paper.


Efficient Inference in Occlusion-Aware Generative Models of Images

Abstract:
We present a generative model of images based on layering, in which image layers are individually generated, then composited from front to back. We are thus
able to factor the appearance of an image into the appearance of individual objects within the image — and additionally for each individual object, we can factor content from pose. Unlike prior work on layered models, we learn a shape prior for each object/layer, allowing the model to tease out which object is in front by looking for a consistent shape, without needing access to motion cues or any labeled data. We show that ordinary stochastic gradient variational bayes (SGVB), which optimizes our fully differentiable lower-bound on the log-likelihood, is sufficient to learn an interpretable representation of images. Finally we present experiments demonstrating the effectiveness of the model for inferring foreground and background objects in images.

Title of paper.

Abstract of paper.


Robust Convolutional Neural Networks under Adversarial Noise

Abstract: Recent studies have shown that Convolutional Neural Networks (CNNs) are vulnerable to a small perturbation of input called "adversarial examples". In this work, we propose a new feedforward CNN that improves robustness in the presence of adversarial noise. Our model uses stochastic additive noise added to the input image and to the CNN models. The proposed model operates in conjunction with a CNN trained with either standard or adversarial objective function. In particular, convolution, max-pooling, and ReLU layers are modified to benefit from the noise model. Our feedforward model is parameterized by only a mean and variance per pixel which simplifies computations and makes our method scalable to a deep architecture. From CIFAR-10 and ImageNet test, the proposed model outperforms other methods and the improvement is more evident for difficult classification tasks or stronger adversarial noise.

Jonghoon Jin
Aysegul Dundar
Eugenio Culurciello

Title of paper.

Abstract of paper.


Visualizing and Understanding Recurrent Networks

Abstract: Recurrent Neural Networks (RNNs), and specifically a variant with Long Short-Term Memory (LSTM), are enjoying renewed interest as a result of successful applications in a wide range of machine learning problems that involve sequential data. However, while LSTMs provide exceptional results in practice, the source of their performance and their limitations remain rather poorly understood. Using character-level language models as an interpretable testbed, we aim to bridge this gap by providing an analysis of their representations, predictions and error types. In particular, our experiments reveal the existence of interpretable cells that keep track of long-range dependencies such as line lengths, quotes and brackets. Moreover, our comparative analysis with finite horizon n-gram models traces the source of the LSTM improvements to long-range structural dependencies. Finally, we provide analysis of the remaining errors and suggests areas for further study.

Andrej Karpathy
Justin Johnson
Li Fei-Fei

Title of paper.

Abstract of paper.


Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference

Abstract:
Convolutional neural networks (CNNs) work well on large datasets. But labelled
data is hard to collect, and in some applications larger amounts of data are not
available. The problem then is how to use CNNs with small data – as CNNs
overfit quickly. We present an efficient Bayesian CNN, offering better robustness
to over-fitting on small data than traditional approaches. This is by placing a
probability distribution over the CNN’s kernels. We approximate our model’s intractable
posterior with Bernoulli variational distributions, requiring no additional
model parameters.
On the theoretical side, we cast dropout network training as approximate inference
in Bayesian neural networks. This allows us to implement our model using existing
tools in deep learning with no increase in time complexity, while highlighting a
negative result in the field. We show a considerable improvement in classification
accuracy compared to standard techniques and improve on published state-of-the-art
results for CIFAR-10.

Title of paper.

Abstract of paper.


Doctor AI: Predicting Clinical Events via Recurrent Neural Networks

Abstract:
Large amount of  Electronic Health Record (EHR) data have been collected over millions of patients over multiple years. The rich longitudinal EHR data documented the collective experiences of physicians including diagnosis, medication prescription and procedures. We argue it is possible now to leverage the EHR data to model how physicians behave, and we call our model Doctor AI.
Towards this direction of modeling clinical behavior of physicians,  we develop a successful application of Recurrent Neural Networks (RNN) to jointly forecast the future disease diagnosis and medication prescription along with their timing. Unlike traditional classification models where a single target is of interest, our model can assess the entire history of patients and make continuous and multilabel predictions based on patients' historical data. We evaluate the performance of the proposed method on a large real-world EHR data over 260K patients over 8 years.
We observed Doctor AI can perform differential diagnosis with similar accuracy to physicians. In particular, Doctor AI achieves up to 79% recall@30, significantly higher than several baselines. Moreover, we demonstrate great generalizability of Doctor AI by applying the resulting models on data from a completely different medication institution achieving comparable performance.

Edward Choi
Mohammad Taha Bahadori
Andy Schuetz
Walter F. Stewart
Joshua C. Denny
Bradley A. Malin
Jimeng Sun

Title of paper.

Abstract of paper.


A constrained l1 minimization approach for estimating multiple Sparse Gaussian or Nonparanormal Graphical Models

Abstract: The flood of multi-context measurement data from many scientific domains has created an urgent need to reconstruct context-specific variable networks, that could significantly simplify network-driven studies. Computationally, this problem can be formulated  as jointly estimating multiple different, but related, sparse Undirected Graphical Models (UGM) from  samples aggregated across several contexts. Previous joint-UGM studies could not address this challenge since they mostly focus on Gaussian Graphical Models (GGM) and have used likelihood-based formulation to infer multiple graphs toward a common pattern. Differently, we propose a novel approach, SIMULE (learning Shared and Individual parts of MULtiple graphs Explicitly) to solve multi-task UGM using a $\ell$1 constrained optimization. SIMULE is cast as independent subproblems of linear programming that can be solved  efficiently. It automatically infers specific dependencies that are unique to each context as well as shared substructures preserved among all the contexts. SIMULE can handle both multivariate Gaussian and multivariate Nonparanormal data that greatly relax the normality assumption. Theoretically we prove that SIMULE achieves a consistent result at rate $O(\log(Kp)/n_{tot})$ (not been proved before). On four synthetic datasets, SIMULE shows significant improvements over state-of-the-art multi-sGGM and single-UGM baselines.

Title of paper.

Abstract of paper.


Sequence Modeling with Recurrent Tensor Networks

Abstract: We introduce the recurrent tensor network, a recurrent neural network model that replaces the matrix-vector multiplications of a standard recurrent neural network with bilinear tensor products. We compare its performance against networks that employ long short-term memory (LSTM) networks. Our results demonstrate that using tensors to capture the interactions between network inputs and history can lead to substantial improvement in predictive performance on the language modeling task.

Title of paper.

Abstract of paper.


Distinct Class Saliency Maps for Multiple Object Images

Abstract:
This paper proposes a method to obtain more distinct class saliency maps than
Simonyan et al. (2014). We made three improvements over their method: (1) using
CNN derivatives with respect to feature maps of the intermediate convolutional
layers with up-sampling instead of an input image; (2) subtracting saliency
maps of the other classes from saliency maps of the target class to differentiate
target objects from other objects; (3) aggregating multi-scale class saliency maps
to compensate lower resolution of the feature maps.

Title of paper.

Abstract of paper.


A Deep Memory-based Architecture for Sequence-to-Sequence Learning

Abstract: We propose DEEPMEMORY, a novel deep architecture for sequence-to-sequence learning, which performs the task through a series of nonlinear transformations from the representation of the input sequence (e.g., a Chinese sentence) to the final output sequence (e.g., translation to English). Inspired by the recently proposed Neural Turing Machine (Graves et al., 2014), we store the intermediate representations in stacked layers of memories, and use read-write operations on the memories to realize the nonlinear transformations between the representations. The types of transformations are designed in advance but the parameters are learned from data. Through layer-by-layer transformations, DEEPMEMORY can model complicated relations between sequences necessary for applications such as machine translation between distant languages. The architecture can be trained with normal back-propagation on sequence-to-sequence data, and the learning can be easily scaled up to a large corpus. DEEPMEMORY is broad enough to subsume the state-of-the-art neural translation model in (Bahdanau et al., 2015) as its special case, while significantly improving upon the model with its deeper architecture. Remarkably, DEEPMEMORY, being purely neural network-based, can achieve performance comparable to the traditional phrase-based machine translation system Moses with a small vocabulary and a modest parameter size.

Fandong Meng
Zhengdong Lu
Zhaopeng Tu
Hang Li
Qun Liu

Title of paper.

Abstract of paper.