Fully Convolutional Nerual Network for Body Part Segmentation

Abstract: This paper presents the foundation of a new system for human body segmentation. It is based on a Fully Convolutional Neural Network that uses depth images as input and produces a per-pixel labeling of the image where each pixel has been labeled as a body segment of interest or as non-person. The training data are fully synthetic which allow for large amounts of data to be generated in a relatively short period of time. By using a GPU accelerated implementation of the convolutional neural network, the system is capable of segmenting an image in 8.5 milliseconds. This work will form the basis for more robust system in the future that will be suitable for finding pose skeletons in more cluttered environments.

David Frank
Richard Kelley
David Feil-Seifer

Your displayed identity associated with the above content.

The users who will be allowed to read the above content.

Either upload a PDF file or provide a direct link to your PDF on ArXiv.

Title of paper.

Abstract of paper.

Comma separated list of author names, as they appear in the paper.

Comma separated list of author email addresses, in the same order as above.

Semi-colon separated list of email domains of people who would have a conflict of interest in reviewing this paper, (e.g., cs.umass.edu;google.com, etc.).

If the paper is a resubmission from the ICLR 2016 Conference Track, enter its CMT ID; otherwise, leave blank.


A metric learning approach for graph-based label propagation

Abstract: The efficiency of graph-based semi-supervised algorithms depends on the graph of instances on which they are applied. The instances are often in a vectorial form before a graph linking them is built. The construction of the graph relies on a metric over the vectorial space that help define the weight of the connection between entities. The classic choice for this metric is usually a distance measure or a similarity measure based on the euclidean norm. We claim that in some cases the euclidean norm on the initial vectorial space might not be the more appropriate to solve the task efficiently. We propose an algorithm that aims at learning the most appropriate vectorial representation for building a graph on which the task at hand is solved efficiently.

Title of paper.

Abstract of paper.


Learning to SMILE(S)

Abstract:
This paper shows how one can directly apply natural language processing (NLP) methods to classification problems in cheminformatics.
Connection between these seemingly separate fields is shown by considering standard textual representation of compound, SMILES.
The problem of activity prediction against a target protein is considered, which is a crucial part of computer aided drug design process.
Conducted experiments show that this way one can not only outrank state of the art results of hand crafted representations but also
gets direct structural insights into the way decisions are made.

Stanisław Jastrzębski
Damian Leśniak
Wojciech Marian Czarnecki

Title of paper.

Abstract of paper.


Deep Reinforcement Learning with an Action Space Defined by Natural Language

Abstract: In this paper, we propose the deep reinforcement relevance network (DRRN), a novel deep architecture, to design a model for handling an action space characterized using natural language with applications to text-based games. For a particular class of games, a user must choose among a number of actions described by text, with the goal of maximizing long-term reward. In these games, the best action is typically what fits the current situation best (modeled as a state in the DRRN), also described by text. Because of the exponential complexity of natural language with respect to sentence length, there is typically an unbounded set of unique actions. Even with a constrained vocabulary, the action space is very large and sparse, posing challenges for learning. To address this challenge, the DRRN extracts separate high-level embedding vectors from the texts that describe states and actions, respectively, using a general interaction function, such as inner product, bilinear, and DNN interaction, between these embedding vectors to approximate the Q-function. We evaluate the DRRN on two popular text games, showing superior performance over other deep Q-learning architectures.

Ji He
Jianshu Chen
Xiaodong He
Jianfeng Gao
Lihong Li
Li Deng
Mari Ostendorf

Title of paper.

Abstract of paper.


Contextual convolutional neural network filtering improves EM image segmentation

Abstract: We designed a contextual filtering algorithm for improving the quality of image segmentation. The algorithm was applied on the task of building the Membrane Detection Probability Maps (MDPM) for segmenting electron microscopy (EM) images of brain tissues. To achieve this, we executed supervised training of a convolutional neural network to recover the ground-truth label of the masked-out center pixel from patches sampled from an un-refined MDPM. Through this training process the model learns the distribution of the segmentation ground-truth map . By applying this trained network over MDPMs we are able to integrate contextual information to obtain with better spatial consistency in the high-level representation space. By iteratively applying this network over the MDPMs for multiple rounds, we were able to significantly improve the EM image segmentation results.

Xundong Wu
Yong Wu
Ligia Toro
Enrico Stefani

Title of paper.

Abstract of paper.


HARDWARE-FRIENDLY CONVOLUTIONAL NEURAL NETWORK WITH EVEN-NUMBER FILTER SIZE

Abstract: Convolutional Neural Network (CNN) has led to great advances in computer vision. Various customized CNN accelerators on embedded FPGA or ASIC platforms have been designed to accelerate CNN and improve energy efficiency. However, the odd-number filter size in existing CNN models prevents hardware accelerators from having optimal efficiency. In this paper, we analyze the influences of filter size on CNN accelerator performance and show that even-number filter size is much more hardware-friendly that can ensure high bandwidth and resource utilization. Experimental results on MNIST and CIFAR-10 demonstrate that hardware-friendly even kernel CNNs can reduce the FLOPs by 1.4x to 2x with comparable accuracy; With same FLOPs, even kernel can have even higher accuracy than odd size kernel.

Song Yao
Song Han
Kaiyuan Guo
Jianqiao Wangni
Yu Wang

Title of paper.

Abstract of paper.


RECURRENT MODELS FOR AUDITORY ATTENTION IN MULTI-MICROPHONE DISTANCE SPEECH RECOGNITION

Abstract: Integration of multiple microphone data is one of the key ways to achieve robust speech recognition in noisy environments or when the speaker is located at some distance from the input device. Signal processing techniques such as beamforming are widely used to extract a speech signal of interest from background noise. These techniques, however, are highly dependent on prior spatial information about the microphones and the environment in which the system is being used. In this work, we present a neural attention network that directly combines multichannel audio to generate phonetic states without requiring any prior knowledge of the microphone layout or any explicit signal preprocessing for speech enhancement. We embed an attention mechanism within a Recurrent Neural Network (RNN) based acoustic model to automatically tune its attention to a more reliable input source. Unlike traditional multi-channel preprocessing, our system can be optimized towards the desired output in one step. Although attention-based models have recently achieved impressive results on sequence-to-sequence learning, no attention mechanisms have previously been applied to learn potentially asynchronous and non-stationary multiple inputs. We evaluate our neural attention model on the CHiME-3 challenge task, and show that the model achieves comparable performance to beamforming using a purely data-driven method.

Title of paper.

Abstract of paper.


Conditional computation in neural networks for faster models

Abstract: Deep learning has become the state-of-art tool in many applications, but the evaluation of expressive deep models can be unfeasible on resource-constrained devices. The conditional computation approach has been proposed to tackle this problem (Bengio et al., 2013; Davis & Arel, 2013). It operates by selectively activating only parts of the network at a time. We propose to use reinforcement learning as a tool to optimize conditional computation policies. More specifically, we cast the problem of learning activation-dependent policies for dropping out blocks of units as reinforcement learning. We propose a learning scheme motivated by computation speed, capturing the idea of wanting to have parsimonious activations while maintaining prediction accuracy. We apply a policy gradient algorithm for learning policies that optimize this loss function and propose a regularization mechanism that encourages diversification of the dropout policy. We present encouraging empirical results showing that this approach improves the speed of computation without impacting the quality of the approximation.

Emmanuel Bengio
Pierre-Luc Bacon
Joelle Pineau
Doina Precup

Title of paper.

Abstract of paper.


Generating Sentences from a Continuous Space

Abstract: The standard unsupervised recurrent neural network language model (RNNLM) generates sentences one word at a time and does not work from an explicit global distributed sentence representation. In this work, we present an RNN-based variational autoencoder language model that incorporates distributed latent representations of entire sentences. This factorization allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features. Samples from the prior over these sentence representations remarkably produce diverse and well-formed sentences through simple deterministic decoding. By examining paths through this latent space, we are able to generate coherent novel sentences that interpolate between known sentences. We present techniques for solving the difficult learning problem presented by this model, demonstrate strong performance in the imputation of missing tokens, and explore many interesting properties of the latent sentence space.

Samuel R. Bowman
Luke Vilnis
Oriol Vinyals
Andrew M. Dai
Rafal Jozefowicz
Samy Bengio

Title of paper.

Abstract of paper.


Stacked What-Where Auto-encoders

Abstract: We present a novel architecture, the "stacked what-where auto-encoders" (SWWAE), which integrates discriminative and generative pathways and provides a unified approach to supervised, semi-supervised and unsupervised learning without relying on sampling during training. An instantiation of SWWAE uses a convolutional net (Convnet) (LeCun et al. (1998)) to encode the input, and employs a deconvolutional net (Deconvnet) (Zeiler et al. (2010)) to produce the reconstruction. The objective function includes reconstruction terms that induce the hidden states in the Deconvnet to be similar to those of the Convnet. Each pooling layer produces two sets of variables: the "what" which are fed to the next layer, and its complementary variable "where" that are fed to the corresponding layer in the generative decoder.

Junbo Zhao
Michael Mathieu
Ross Goroshin
Yann LeCun

Title of paper.

Abstract of paper.


Unsupervised Learning of Visual Structure using Predictive Generative Networks

Abstract: The ability to predict future states of the environment is a central pillar of intelligence. At its core, effective prediction requires an internal model of the world and an understanding of the rules by which the world changes. Here, we explore the internal models developed by deep neural networks trained using a loss based on predicting future frames in synthetic video sequences, using a CNN-LSTM-deCNN framework. We first show that this architecture can achieve excellent performance in visual sequence prediction tasks, including state-of-the-art performance in a standard 'bouncing balls' dataset (Sutskever et al., 2009). Using a weighted mean-squared error and adversarial loss (Goodfellow et al., 2014), the same architecture successfully extrapolates out-of-the-plane rotations of computer-generated faces. Furthermore, despite being trained end-to-end to predict only pixel-level information, our Predictive Generative Networks learn a representation of the latent structure of the underlying three-dimensional objects themselves. Importantly, we find that this representation is naturally tolerant to object transformations, and generalizes well to new tasks, such as classification of static images. Similar models trained solely with a reconstruction loss fail to generalize as effectively. We argue that prediction can serve as a powerful unsupervised loss for learning rich internal representations of high-level object features.

William Lotter
Gabriel Kreiman
David Cox

Title of paper.

Abstract of paper.


Comparative Study of Caffe, Neon, Theano, and Torch for Deep Learning

Abstract: Deep learning methods have resulted in significant performance improvements in several application domains and as such several software frameworks have been developed to facilitate their implementation. This paper presents a comparative study of four deep learning frameworks, namely Caffe, Neon, Theano, and Torch, on three aspects: extensibility, hardware utilization, and speed. The study is performed on several types of deep learning architectures and we evaluate the performance of the above frameworks when employed on a single machine for both (multi-threaded) CPU and GPU (Nvidia Titan X) settings. The speed performance metrics used here include the gradient computation time, which is important during the training phase of deep networks, and the forward time, which is important from the deployment perspective of trained networks. For convolutional networks, we also report how each of these frameworks support various convolutional algorithms and their corresponding performance. From our experiments, we observe that Theano and Torch are the most easily extensible frameworks. We observe that Torch is best suited for any deep architecture on CPU, followed by Theano. It also achieves the best performance on the GPU for large convolutional and fully connected networks, followed closely by Neon. Theano achieves the best performance on GPU for training and deployment of LSTM networks. Finally Caffe is the easiest for evaluating the performance of standard deep architectures.

Soheil Bahrampour
Naveen Ramakrishnan
Lukas Schott
Mohak Shah

Title of paper.

Abstract of paper.


Convolutional Models for Joint Object Categorization and Pose Estimation

Abstract: In the task of Object Recognition, there exists a dichotomy between the categorization of objects and estimating object pose, where the former necessitates a view-invariant representation, while the latter requires a representation capable of capturing pose information over different categories of objects. With the rise of deep architectures, the prime focus has been on object category recognition. Deep learning methods have achieved wide success in this task. In contrast, object pose regression using these approaches has received relatively much less attention. In this paper we show how deep architectures, specifically Convolutional Neural Networks (CNN), can be adapted to the task of simultaneous categorization and pose estimation of objects. We investigate and analyze the layers of various CNN models and extensively compare between them with the goal of discovering how the layers of distributed representations of CNNs represent object pose information and how this contradicts with object category representations. We extensively experiment on two recent large and challenging multi-view datasets. Our models achieve better than state-of-the-art performance on both datasets.

Mohamed Elhoseiny
Tarek El-Gaaly
Amr Bakry
Ahmed Elgammal

Title of paper.

Abstract of paper.


Lookahead Convolution Layer for Unidirectional Recurrent Neural Networks

Abstract:
Recurrent neural networks (RNNs) have been shown to be very effective for many
sequential prediction problems such as speech recognition, machine translation, part-of-speech tagging, and others.
The best variant is typically a bidirectional RNN that learns
representation for a sequence by performing a forward and a backward pass through the entire sequence.
However, unlike unidirectional RNNs, bidirectional RNNs
are challenging to deploy in an online and low-latency setting (e.g., in a speech recognition system),
because they need to see an entire sequence before making a prediction.
We introduce a lookahead convolution layer that incorporates information from future subsequences
in a computationally efficient manner to improve unidirectional recurrent neural networks.
We evaluate our method on speech recognition tasks for two languages---English and Chinese.
Our experiments show that the proposed method outperforms vanilla unidirectional
RNNs and is competitive with bidirectional RNNs in terms of character and word error rates.

Chong Wang
Dani Yogatama
Adam Coates
Tony Han
Awni Hannun
Bo Xiao

Title of paper.

Abstract of paper.


Coverage-based Neural Machine Translation

Abstract: Attention mechanism advanced state-of-the-art neural machine translation (NMT) by jointly learning to align and translate. However, attentional NMT ignores past alignment information, which leads to over-translation and under-translation problems. In response to this problem, we maintain a coverage vector to keep track of the attention history. The coverage vector is fed to the attention model to help adjust the future attention, which guides NMT to pay more attention to the untranslated source words. Experiments show that coverage-based NMT significantly improves both translation and alignment qualities over NMT without coverage.

Zhaopeng Tu
Zhengdong Lu
Yang Liu
Xiaohua Liu
Hang Li

Title of paper.

Abstract of paper.


Fine-grained pose prediction, normalization, and recognition

Abstract:
Pose variation and subtle differences in appearance are key challenges to fine-
grained classification. While deep networks have markedly improved general
recognition, many approaches to fine-grained recognition rely on anchoring net-
works to parts for better accuracy. Identifying parts to find correspondence dis-
counts pose variation so that features can be tuned to appearance. To this end
previous methods have examined how to find parts and extract pose-normalized
features. These methods have generally separated fine-grained recognition into
stages which first localize parts using hand-engineered and coarsely-localized pro-
posal features, and then separately learn deep descriptors centered on inferred part
positions. We unify these steps in an end-to-end trainable network supervised by
keypoint locations and class labels that localizes parts by a fully convolutional
network to focus the learning of feature representations for the fine-grained clas-
sification task. Experiments on the popular CUB200 dataset show that our method
is state-of-the-art and suggest a continuing role for strong supervision.

Ning Zhang
Evan Shelhamer
Yang Gao
Trevor Darrell

Title of paper.

Abstract of paper.


Bidirectional Helmholtz Machines

Abstract:
Efficient unsupervised training and inference in deep generative models remains
a challenging problem. One basic approach, called Helmholtz machine, involves
training a top-down directed generative model together with a bottom-up
auxiliary model that is trained to help perform approximate inference.
Recent results indicate that better results can be obtained with better approximate
inference procedures. Instead of employing more powerful procedures, we here
propose to regularize the generative model to stay close to the class of
distributions that can be efficiently inverted by the approximate inference model.
We achieve this by interpreting both the top-down and the bottom-up
directed models as approximate inference distributions and by defining the model
distribution to be the geometric mean of these two. We present a lower-bound for
the likelihood of this model and we show that optimizing this bound
regularizes the model so that the Bhattacharyya distance between the bottom-up
and top-down approximate distributions is minimized. We demonstrate that we can
use this approach to fit generative models with many layers of hidden binary stochastic variables
to complex training distributions and that this method prefers significantly
deeper architectures while it supports orders of magnitude more efficient
approximate inference than other approaches.

Jorg Bornschein
Samira Shabanian
Asja Fischer
Yoshua Bengio

Title of paper.

Abstract of paper.


CMA-ES for Hyperparameter Optimization of Deep Neural Networks

Abstract:
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization.
As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy usage example using CMA-ES to tune hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.

Title of paper.

Abstract of paper.


VARIATIONAL STOCHASTIC GRADIENT DESCENT

Abstract: In Bayesian approach to probabilistic modeling of data we select a model for probabilities of data that depends on a continuous vector of parameters. For a given data set Bayesian theorem gives a probability distribution of the model parameters. Then the inference of outcomes and probabilities of new data could be found by averaging over the parameter distribution of the model, which is an intractable problem. In this paper we propose to use Variational Bayes (VB) to estimate Gaussian posterior of model parameters for a given Gaussian prior and Bayesian updates in a form that resembles SGD rules. It is shown that with incremental updates of posteriors for a selected sequence of data points and a given number of iterations the variational approximations are defined by a trajectory in space of Gaussian parameters, which depends on a starting point defined by priors of the parameter distribution, which are true hyper-parameters. The same priors are providing a weight decay or L2 regularization for the training. Then a selection of L2 regularization parameters and a number of iterations is completely defining a learning rule for VB SGD optimization, unlike other methods with momentum (Duchi et al., 2011; Kingma & Ba, 2014; Zeiler, 2012) that need selecting learning, regularization rates, etc., separately. We consider application of VB SGD for important practical case of fast training neural networks with very large data. While the speedup is achieved by partitioning data and training in parallel the resulting set of solutions obtained with VB SGD forms a Gaussian mixture. By applying VB SGD optimization to the Gaussian mixture we can merge multiple neural networks of same dimensions into a new single neural network that has almost the same performance as an original Gaussian mixture.

Title of paper.

Abstract of paper.


Manifold traversal using density ridges

Abstract:
In this work we present two examples of how a manifold learning model can represent the complexity of shape variation in images.
Manifold learning techniques for image manifolds can be used to model data in sparse manifold regions.
Additionally, they can be used as generative models as they can often better represent or learn structure in the data.
We propose a method of estimating the underlying manifold using the ridges of a kernel density estimate as well as tangent space operations that allows interpolation between images along the manifold and offers a novel approach to analyzing the image manifold.

Jonas Nordhaug Myhre
Michael Kampffmeyer
Robert Jenssen

Title of paper.

Abstract of paper.


On-the-fly Network Pruning for Object Detection

Abstract: Object detection with deep neural networks is often performed by passing a few thousand candidate bounding boxes through a deep neural network for each image. These bounding boxes are highly correlated since they originate from the same image. In this paper we investigate how to exploit feature  occurrence at the image scale to prune the neural network which is subsequently applied to all bounding boxes. We show that removing units which have near-zero activation in the image allows us to significantly reduce the number of parameters in the network. Results on the PASCAL 2007 Object Detection Challenge demonstrate that up to 40% of units in some fully-connected layers can be entirely eliminated with little change in the detection result.

Marc Masana
Joost van de Weijer
Andrew D. Bagdanov

Title of paper.

Abstract of paper.


Data Cleaning by Deep Dictionary Learning

Abstract: The soundness of training data is important to the performance of a learning model. However in recommender systems, the training data are usually noisy, because of the randomness nature of users' behaviors and the sparseness of the users' feedback towards the recommendations. In this work, we would like to propose a noise elimination model to preprocess the training data in recommender systems. We define the noise as the abnormal patterns in the users' feedback. The proposed deep dictionary learning model tries to find the common patterns through dictionary learning. We define a dictionary through the output layer of a stacked autoencoder, so that the dictionary is represented by a deep structure and the noise in the dictionary is further filtered-out.

Title of paper.

Abstract of paper.


Binding via Reconstruction Clustering

Abstract:
Disentangled distributed representations of data are desirable for machine learning, since they are more expressive and can generalize from fewer examples.
However, for complex data, the distributed representations of multiple objects present in the same input can interfere and lead to ambiguities, which is commonly referred to as the binding problem.
We argue for the importance of the binding problem to the field of representation learning, and develop a probabilistic framework that explicitly models inputs as a composition of multiple objects.
We propose an algorithm that uses a denoising autoencoder to dynamically bind features together in multi-object inputs through an Expectation-Maximization-like clustering process.
The effectiveness of this method is demonstrated on artificially generated datasets of binary images, showing that it can even generalize to bind together new objects never seen by the autoencoder during training.

Klaus Greff
Rupesh Srivastava
Jürgen Schmidhuber

Title of paper.

Abstract of paper.


Deep Directed Generative Models with Energy-Based Probability Estimation

Abstract: Energy-based probabilistic models have been confronted with intractable computations during the learning that requires to have appropriate samples drawn from the estimated probability distribution. It can be approximately achieved by a Monte Carlo Markov Chain sampling process, but still has mixing problems especially with deep models that slow the learning. We introduce an auxiliary deep model that deterministically generates samples based on the estimated distribution, and this makes the learning easier without any high cost sampling process. As a result, we propose a new framework to train the energy-based probabilistic models with two separate deep feed-forward models. The one is only to estimate the energy function, and the another is to deterministically generate samples based on it. Consequently, we can estimate the probability distribution and its corresponding deterministic generator with deep models.

Title of paper.

Abstract of paper.


GradNets: Dynamic Interpolation Between Neural Architectures

Abstract: In machine learning, there is a fundamental trade-off between ease of optimization and expressive power. Neural Networks, in particular, have enormous expressive power and yet are notoriously challenging to train. The nature of that optimization challenge changes over the course of learning. Traditionally in deep learning, one makes a static trade-off between the needs of early and late optimization. In this paper, we investigate a novel framework, GradNets, for dynamically adapting architectures during training to get the benefits of both. For example, we can gradually transition from linear to non-linear networks, deterministic to stochastic computation, shallow to deep architectures, or even simple downsampling to fully differentiable attention mechanisms. Benefits include increased accuracy, easier convergence with more complex architectures, solutions to test-time execution of batch normalization, and the ability to train networks of up to 200 layers.

Title of paper.

Abstract of paper.


Input-Convex Deep Networks

Abstract:
This paper introduces a new class of neural networks that we
refer to as input-convex neural networks, networks that are convex in
their inputs (as opposed to their parameters).  We discuss the nature and
representational power of these networks, illustrate how the prediction
(inference) problem can be solved via convex optimization, and discuss their
application to structured prediction problems.  We highlight a few simple
examples of these networks applied to classification tasks, where we illustrate
that the networks perform substantially better than any other approximator we
are aware of that is convex in its inputs.

Title of paper.

Abstract of paper.


Do Deep Convolutional Nets Really Need to be Deep (Or Even Convolutional)?

Abstract:
Yes, apparently they do.

Previous research by Ba and Caruana (2014)
demonstrated that shallow feed-forward nets
sometimes can learn the complex functions pre-
viously learned by deep nets while using a simi-
lar number of parameters as the deep models they
mimic. In this paper we investigate if shallow
models can learn to mimic the functions learned
by deep convolutional models. We experiment
with shallow models and models with a vary-
ing number of convolutional layers, all trained to
mimic a state-of-the-art ensemble of CIFAR-10
models. We demonstrate that we are unable to
train shallow models to be of comparable accu-
racy to deep convolutional models. Although the
student models do not have to be as deep as the
teacher models they mimic, the student models
apparently need multiple convolutional layers to
learn functions of comparable accuracy.

Gregor Urban
Krzysztof J. Geras
Samira Ebrahimi Kahou
Ozlem Aslan Shengjie Wang
Rich Caruana
Abdelrahman Mohamed
Matthai Philipose
Matt Richardson

Title of paper.

Abstract of paper.


Deconstructing the Ladder Network Architecture

Abstract: The Ladder Network is a recent new approach to semi-supervised learning that turned out to be very successful. While showing impressive performance, the Ladder Network has many components intertwined, whose contributions are not obvious in such a complex architecture. This paper presents an extensive experimental investigation of variants of the Ladder Network in which we replaced or removed individual components to learn about their relative importance. For semi-supervised tasks, we conclude that  the most important contribution is made by the lateral connections, followed by the application of noise, and the choice of what we refer to as the `combinator function'. As the number of labeled training examples increases, the lateral connections and the reconstruction criterion become less important, with most of the generalization improvement coming from the injection of noise in each layer. Finally, we introduce a combinator function that reduces test error rates on Permutation-Invariant MNIST to 0.57\% for the supervised setting, and to 0.97 % and 1.0 % for semi-supervised settings with 1000 and 100 labeled examples, respectively.

Mohammad Pezeshki
Linxi Fan
Philemon Brakel
Aaron Courville
Yoshua Bengio

Title of paper.

Abstract of paper.


Fixed Point Quantization of Deep Convolutional Networks

Abstract: In recent years increasingly complex architectures for deep convolution networks (DCNs) have been proposed to boost the performance on image recognition tasks. However, the gains in performance have come at a cost of substantial increase in computation and model storage resources. Fixed point implementation of DCNs has the potential to alleviate some of these complexities and facilitate potential deployment on embedded hardware. In this paper, we formulate and solve an optimization problem to identify the optimal fixed point bit-width allocation across layers to enable efficient fixed point implementation of DCNs. Our experiments show that in comparison to equal bit-width settings, optimized bit-width allocation offers >20% reduction in model size without any loss in accuracy on CIFAR-10 benchmark. We also demonstrate that fine-tuning can further enhance the accuracy of fixed point DCNs beyond that of the original floating point model. In doing so, we report a new state-of-the-art fixed point performance of 6.78% error-rate on CIFAR-10 benchmark.

Darryl D. Lin
Sachin S. Talathi
V. Sreekanth Annapureddy

Title of paper.

Abstract of paper.


Spatio-Temporal Video Autoencoder with Differentiable Memory

Abstract: We describe a new spatio-temporal video autoencoder, based on a classic spatial image autoencoder and a novel nested temporal autoencoder. The temporal encoder is represented by a differentiable visual memory composed of convolutional long short-term memory (LSTM) cells that integrate changes over time. Here we target motion changes and use as temporal decoder a robust optical flow prediction module together with an image sampler serving as built-in feedback loop. The architecture is end-to-end differentiable. At each time step, the system receives as input a video frame, predicts the optical flow based on the current observation and the LSTM memory state as a dense transformation map, and applies it to the current frame to generate the next frame. By minimising the reconstruction error between the predicted next frame and the corresponding ground truth next frame, we train the whole system to extract features useful for motion estimation without any supervision effort. We believe these features can in turn facilitate learning high-level tasks such as path planning, semantic segmentation, or action recognition, reducing the overall supervision effort.

Viorica Patraucean
Ankur Handa
Roberto Cipolla

Title of paper.

Abstract of paper.


Learning Representations of Affect from Speech

Abstract: There has been a lot of prior work on representation learning for speech recognition applications, but not much emphasis has been given to an investigation of effective representations of affect from speech, where the paralinguistic elements of speech are separated out from the verbal content. In this paper, we explore denoising autoencoders for learning paralinguistic attributes, i.e. categorical and dimensional affective traits from speech. We show that the representations learnt by the bottleneck layer of the autoencoder are highly discriminative of activation intensity and at separating out negative valence (sadness and anger) from positive valence (happiness). We experiment with different input speech features (such as FFT and log-mel spectrograms with temporal context windows), and different autoencoder architectures (such as stacked and deep autoencoders). We also learn utterance specific representations by a combination of denoising autoencoders and BLSTM based recurrent autoencoders. Emotion classification is performed with the learnt temporal/dynamic representations to evaluate the quality of the representations. Experiments on a well-established real-life speech dataset (IEMOCAP) show that the learnt representations are comparable to state of the art feature extractors (such as voice quality features and MFCCs) and are competitive with state-of-the-art approaches at emotion and dimensional affect recognition.

Sayan Ghosh
Eugene Laksana
Louis-Philippe Morency
Stefan Scherer

Title of paper.

Abstract of paper.


Applying Representation Learning for Educational Data Mining

Abstract:
Educational Data Mining is an area of growing interest,
given the increase of available data and generalization of
online learning environments. In this paper we present a
first approach to integrating Representation Learning techniques
in Educational Data Mining by adding autoencoders
as a preprocessing step in a standard performance prediction
problem.
Preliminary results do not show an improvement in performance
by using autoencoders, but we expect that a fine
tuning of parameters will provide an improvement. Also, we
expect that autoencoders will be more useful combined with
different kinds of classifiers, like multilayer perceptrons.

Title of paper.

Abstract of paper.


PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions

Abstract: We propose a novel approach to reduce the computational cost of evaluation of convolutional neural networks, a factor that has hindered their deployment in low-power devices such as mobile phones. Inspired by the loop perforation technique from source code optimization, we speed up the bottleneck convolutional layers by skipping their evaluation in some of the spatial positions. We propose and analyze several strategies of choosing these positions. Our method allows to reduce the evaluation time of modern convolutional neural networks by 50% with a small decrease in accuracy.

Michael Figurnov
Dmitry Vetrov
Pushmeet Kohli

Title of paper.

Abstract of paper.


Learning Document Embeddings by Predicting N-grams for Sentiment Classification of Long Movie Reviews

Abstract: Bag-of-ngram based methods still achieve state-of-the-art results for tasks such as sentiment classification of long movie reviews, though semantic information is partially lost for these methods. Many document embeddings methods have been proposed to capture semantics, but they still can't outperform bag-of-ngram based methods on this task. In this paper, we modify the architecture of the recently proposed Paragraph Vector, allowing it to learn document vectors by predicting not only words, but n-gram features as well. Our model is able to capture both semantics and word order in documents while keeping the expressive power of learned vectors. Experimental results on IMDB movie review dataset show that our model outperforms previous deep learning models and bag-of-ngram based models due to the above advantages.

Bofang Li
Tao Liu
Xiaoyong Du
Deyuan Zhang
Zhe Zhao

Title of paper.

Abstract of paper.


Neural Network Training Variations in Speech and Subsequent Performance Evaluation

Abstract:
In this work we study variance in the results of neural network training on a wide
variety of configurations in automatic speech recognition. Although this variance
itself is well known, this is, to the best of our knowledge, the first paper that
performs an extensive empirical study on its effects in speech recognition. We
view training as sampling from a distribution and show that these distributions
can have a substantial variance. These observations have important implications
on way results in the literature are reported and interpreted.

Ewout van den Berg
Bhuvana Ramabhadran
Michael Picheny

Title of paper.

Abstract of paper.


Variational Inference for On-line Anomaly Detection in High-Dimensional Time Series

Abstract: Approximate variational inference has shown to be a powerful tool for modeling unknown, complex probability distributions. Recent advances in the field allow us to learn probabilistic sequence models. We apply a Stochastic Recurrent Network (STORN) to learn robot time series data. Our evaluation demonstrates that we can robustly detect anomalies both off- and on-line.

Maximilian Sölch
Justin Bayer
Marvin Ludersdorfer
Patrick van der Smagt

Title of paper.

Abstract of paper.


How far can we go without convolution: Improving fully-connected networks

Abstract: We propose ways to improve the performance of fully connected networks. We found that two approaches in particular have a strong effect on performance: linear bottleneck layers and unsupervised pre-training using autoencoders without hidden unit biases. We show how both approaches can be related to improving gradient flow and reducing sparsity in the network. We show that a fully connected network can yield approximately 70% classification accuracy on the permutation-invariant CIFAR-10 task, which is much higher than the current state-of-the-art. By adding deformations to the training data, the fully connected network achieves 78% accuracy, which is close to the performance of a decent convolutional network.

Zhouhan Lin
Roland Memisevic
Kishore Konda

Title of paper.

Abstract of paper.


Nonparametric Canonical Correlation Analysis

Abstract: Canonical correlation analysis (CCA) is a fundamental technique in multi-view data analysis and representation learning. Several nonlinear extensions of the classical linear CCA method have been proposed, including kernel and deep neural network methods. These approaches restrict attention to certain families of nonlinear projections, which the user must specify (by choosing a kernel or a neural network architecture), and are computationally demanding. Interestingly, the theory of nonlinear CCA without any functional restrictions, has been studied in the population setting by Lancaster already in the 50’s. However, these results, have not inspired practical algorithms. In this paper, we revisit Lancaster’s theory, and use it to devise a practical algorithm for nonparametric CCA (NCCA). Specifically, we show that the most correlated nonlinear projections of two random vectors can be expressed in terms of the singular value decomposition of a certain operator associated with their joint density. Thus, by estimating the population density from data, NCCA reduces to solving an eigenvalue system, superficially like kernel CCA but, importantly, without having to compute the inverse of any kernel matrix. We also derive a partially linear CCA (PLCCA) variant in which one of the views undergoes a linear projection while the other is nonparametric. PLCCA turns out to have a similar form to the classical linear CCA, but with a nonparametric regression term replacing the linear regression in CCA. Using a kernel density estimate based on a small number of nearest neighbors, our NCCA and PLCCA algorithms are memory-efficient, often run much faster, and achieve better performance than kernel CCA and comparable performance to deep CCA.

Tomer Michaeli
Weiran Wang
Karen Livescu

Title of paper.

Abstract of paper.


Improving Variational Inference with Inverse Autoregressive Flow

Abstract:
We propose a simple and practical method for improving the flexibility of the approximate posterior in variational auto-encoders (VAEs) through a transformation with autoregressive networks.

Autoregressive networks, such as RNNs and RNADE networks, are very powerful models. However, their sequential nature makes them impractical for direct use with VAEs, as sequentially sampling the latent variables is slow when implemented on a GPU. Fortunately, we find that by inverting autoregressive networks we can obtain equally powerful data transformations that can be computed in parallel. We call these data transformations inverse autoregressive flows (IAF), and we show that they can be used to transform a simple distribution over the latent variables into a much more flexible distribution, while still allowing us to compute the resulting variables' probability density function. The method is computationally cheap, can be made arbitrarily flexible, and (in contrast with previous work) is naturally applicable to latent variables that are organized in multidimensional tensors, such as 2D grids or time series.

The method is applied to a novel deep architecture of variational auto-encoders. In experiments we demonstrate that autoregressive flow leads to significant performance gains when applied to variational autoencoders for natural images.

Diederik P. Kingma
Tim Salimans
Max Welling

Title of paper.

Abstract of paper.


Unsupervised Learning with Imbalanced Data via Structure Consolidation Latent Variable Model

Abstract: Unsupervised learning on imbalanced data is challenging because, when given imbalanced data, current model is often dominated by the major category and ignores the categories with small amount of data. We develop a latent variable model that can cope with imbalanced data by dividing the latent space into a shared space and a private space. Based on Gaussian Process Latent Variable Models, we propose a new kernel formulation that enables the separation of latent space and derive an efficient variational inference method. The performance of our model is demonstrated with an imbalanced medical image dataset.

Fariba Yousefi
Zhenwen Dai
Carl Henrik Ek
Neil Lawrence

Title of paper.

Abstract of paper.


A CONTROLLER-RECOGNIZER FRAMEWORK: HOW NECESSARY IS RECOGNITION FOR CONTROL?

Abstract: Recently there has been growing interest in building ``active'' visual object recognizers, as opposed to ``passive'' recognizers which classifies a given static image into a predefined set of object categories. In this paper we propose to generalize recent end-to-end active visual recognizers into a controller-recognizer framework. In this framework, the interfaces with an external manipulator, while the recognizer classifies the visual input adjusted by the manipulator. We describe two recently proposed controller-recognizer models-- the recurrent attention model (Mnih et al., 2014) and spatial transformer network (Jaderberg et al., 2015)-- as representative examples of controller-recognizer models. Based on this description we observe that most existing end-to-end controller-recognizers tightly couple the controller and recognizer. We consider whether this tight coupling is necessary, and try to answer this empirically by investigating a decoupled controller and recognizer.  Our experiments revealed that it is not always necessary to tightly couple them, and that by decoupling the controller and recognizer, there is a possibility to build a generic controller that is pretrained and works together with any subsequent recognizer.

Marcin Moczulski
Kelvin Xu
Aaron Courville
Kyunghyun Cho

Title of paper.

Abstract of paper.


Close-to-clean regularization relates virtual adversarial training, ladder networks and others

Abstract: We propose a regularization framework where we feed an original clean data point and a nearby point through a mapping, which is then penalized by the Euclidian distance between the corresponding outputs. The nearby point may be chosen randomly or adversarially. We relate this framework to many existing regularization methods: It is a stochastic estimate of penalizing the Frobenius norm of the Jacobian of the mapping as in Poggio & Girosi (1990), it generalizes noise regularization (Sietsma & Dow, 1991), and it is a simplification of the canonical regularization term by the ladder networks in Rasmus et al. (2015). We also study the connection to virtual adversarial training (VAT) (Miyato et al., 2016) and show how VAT can be interpreted as penalizing the largest eigenvalue of a Fisher information matrix. Our main contribution is discovering connections between the proposed and existing regularization methods.

Mudassar Abbas
Jyri Kivinen
Tapani Raiko

Title of paper.

Abstract of paper.


JOINT STOCHASTIC APPROXIMATION LEARNING OF HELMHOLTZ MACHINES

Abstract:
Though with progress, model learning and performing posterior inference still re-
mains a common challenge for using deep generative models, especially for han-
dling discrete hidden variables. This paper is mainly concerned with algorithms
for learning Helmholz machines, which is characterized by pairing the genera-
tive model with an auxiliary inference model. A common drawback of previous
learning algorithms is that they indirectly optimize some bounds of the targeted
marginal log-likelihood. In contrast, we successfully develop a new class of al-
gorithms, based on stochastic approximation (SA) theory of the Robbins-Monro
type, to directly optimize the marginal log-likelihood and simultaneously mini-
mize the inclusive KL-divergence. The resulting learning algorithm is thus called
joint SA (JSA). Moreover, we construct an effective MCMC operator for JSA. Our
results on the MNIST datasets demonstrate that the JSA’s performance is consis-
tently superior to that of competing algorithms like RWS, for learning a range of
difficult models.

Title of paper.

Abstract of paper.


Empirical performance upper bounds for image and video captioning

Abstract: The task of associating images and videos with a natural language description has attracted a great amount of attention recently. Rapid progress has been made in terms of both developing novel algorithms and releasing new datasets. Indeed, the state-of-the-art results on some of the standard datasets have been pushed into the regime where it has become more and more difficult to make significant improvements. Instead of proposing new models, this work investigates the possibility of empirically establishing performance upper bounds on various visual captioning datasets without extra data labelling effort or human evaluation. In particular, it is assumed that visual captioning is decomposed into two steps: from visual inputs to visual concepts, and from visual concepts to natural language descriptions. One would be able to obtain an upper bound when assuming the first step is perfect and only requiring training a conditional language model for the second step. We demonstrate the construction of such bounds on MS-COCO, YouTube2Text and LSMDC (a combination of M-VAD and MPII-MD). Surprisingly, despite of the imperfect process we used for visual concept extraction in the first step and the simplicity of the language model for the second step, we show that current state-of-the-art models fall short when being compared with the learned upper bounds. Furthermore, with such a bound, we quantify several important factors concerning image and video captioning: the number of visual concepts captured by different models, the trade-off between the amount of visual elements captured and their accuracy, and the intrinsic difficulty and blessing of different datasets.

Li Yao
Nicolas Ballas
Kyunghyun Cho
John R. Smith
Yoshua Bengio

Title of paper.

Abstract of paper.


Task Loss Estimation for Structured Prediction

Abstract:
Often, the performance on a supervised machine learning task is evaluated with a
\emph{task loss} function that cannot be optimized directly. Examples of such loss functions
include the classification error, the edit distance and the BLEU
score. A common workaround for this
problem is to instead optimize a \emph{surrogate loss}
function, such as for instance cross-entropy or hinge loss. In
order for this remedy to be effective, it is
important to ensure that minimization of the surrogate loss
results in minimization of the task loss, a condition
that we call \emph{consistency with the task loss}.
In this work, we propose another method for deriving
differentiable surrogate losses that provably meet this requirement.
We focus on the broad class
of models that define a score for every input-output pair.
Our idea is that this score can be interpreted as an
estimate of the task loss, and that the estimation error may
be used as a consistent surrogate loss. A distinct feature of
such an approach is that it defines the desirable value of the
score for every input-output pair. We use this property
to design specialized surrogate losses for  Encoder-Decoder
models often used for sequence prediction tasks.
In our experiment, we benchmark on the task of speech
recognition. Using a new surrogate loss instead of
cross-entropy to train an Encoder-Decoder speech recognizer
brings a significant ~9\% relative improvement in
terms of Character Error Rate (CER) in the case when no
extra corpora are used for language modeling.

Dzmitry Bahdanau
Dmiriy Serdyuk
Philémon Brakel
Nan Rosemary Ke
Jan Chorowski
Aaron Courville
Yoshua Bengio

Title of paper.

Abstract of paper.


Learning visual groups from co-occurrences in space and time

Abstract: We propose a self-supervised framework that learns to group visual entities based on their rate of co-occurrence in space and time. To model statistical dependencies between the entities, we set up a simple binary classification problem in which the goal is to predict if two visual primitives occur in the same spatial or temporal context. We apply this framework to three domains: learning patch affinities from spatial adjacency in images, learning frame affinities from temporal adjacency in videos, and learning photo affinities from geospatial proximity in image collections. We demonstrate that in each case the learned affinities uncover meaningful semantic groupings. From patch affinities we generate object proposals that are competitive with supervised methods. From frame affinities we generate movie scene segmentations that correlate well with DVD chapter structure. Finally, from geospatial affinities we learn groups that relate well to semantic place categories.

Phillip Isola
Daniel Zoran
Dilip Krishnan
Edward H. Adelson

Title of paper.

Abstract of paper.


Basic Level Categorization Facilitates Visual Object Recognition

Abstract:
Recent advances in deep learning have led to significant progress in the computer vision field, especially for visual object recognition tasks. The features useful for object classification are learned by feed-forward deep convolutional neural networks (CNNs) automatically, and they are shown to be able to predict and decode neural representations in the ventral visual pathway of humans and monkeys. However, despite the huge amount of work on optimizing CNNs, there has not been much research focused on linking CNNs with guiding principles from the human visual cortex. In this work, we propose a network optimization strategy inspired by both of the developmental trajectory of children’s visual object recognition capabilities, and Bar (2003), who hypothesized that basic level information is carried in the fast magnocellular pathway through the prefrontal cortex (PFC) and then projected back to inferior temporal cortex (IT), where subordinate level categorization is achieved. We instantiate this idea by training a deep CNN
to perform basic level object categorization first, and then train it on subordinate level categorization. We apply this idea to training AlexNet (Krizhevsky et al., 2012) on the ILSVRC 2012 dataset and show that the top-5 accuracy increases from 80.13% to 82.14%, demonstrating the effectiveness of the method. We also show that subsequent transfer learning on smaller datasets gives superior results.

Title of paper.

Abstract of paper.


Neural Variational Random Field Learning

Abstract: We propose variational bounds on the log-likelihood of an undirected probabilistic graphical model p that are parametrized by flexible approximating distributions q. These bounds are tight when q = p, are convex in the parameters of q for interesting classes of q, and may be further parametrized by an arbitrarily complex neural network. When optimized jointly over q and p, our bounds enable us to accurately track the partition function during learning.

Title of paper.

Abstract of paper.


Neural Text Understanding with Attention Sum Reader

Abstract:
Two large-scale cloze-style context-question-answer datasets have been introduced recently: i) the CNN and Daily Mail news data and ii) the Children's Book Test.
Thanks to the size of these datasets, the associated task is well suited for deep-learning techniques that seem to outperform all alternative approaches.
We present a new, simple model that is tailor made for such question-answering problems.
Our model directly sums attention over candidate answer words in the document instead of using it to compute weighted sum of word embeddings.
Our model outperforms models previously proposed for these tasks by a large margin.

Rudolf Kadlec
Martin Schmid
Ondřej Bajgar
Jan Kleindienst

Title of paper.

Abstract of paper.


Revisiting Distributed Synchronous SGD

Abstract:
The recent success of deep learning approaches for domains like speech recognition (Hinton et al., 2012) and computer vision (Ioffe & Szegedy, 2015) stems from many algorithmic improvements but also from the fact that the size of available training data has grown significantly over the years, together with the computing power, in terms of both CPUs and GPUs. While a single GPU often provides algorithmic simplicity and speed up to a given scale of data and model, there exist an operating point where a distributed implementation of training algorithms for deep architectures becomes necessary.

Previous works have been focusing on asynchronous SGD training, which works well up to a few dozens of workers for some models. In this work, we show that synchronous SGD training, with the help of backup workers, can not only achieve better accuracy, but also reach convergence faster with respect to wall time, i.e. use more workers more efficiently.

Jianmin Chen
Rajat Monga
Samy Bengio
Rafal Jozefowicz

Title of paper.

Abstract of paper.


Resnet in Resnet: Generalizing Residual Architectures

Abstract: ResNets have recently achieved state-of-the-art results on challenging computer vision tasks. In this paper, we create a novel architecture that improves ResNets by adding the ability to forget and by making the residuals more expressive, yielding excellent results. ResNet in ResNet outperforms architectures with similar amounts of augmentation on CIFAR-10 and establishes a new state-of-the-art on CIFAR-100.

Title of paper.

Abstract of paper.


Data Cleaning by Deep Dictionary Learning

Abstract: The soundness of training data is important to the performance of a learning model. However in recommender systems, the training data are usually noisy. Because the users' behaviors have some randomness and their feedback towards the recommendations are sparse. In this work, we would like to propose a noise elimination model to preprocess the training data in the recommender systems. We define the noise as the abnormal patterns in the users' feedback. The proposed deep dictionary learning model tries to find the common patterns through dictionary learning. Meanwhile, we define the dictionary through the output layer of a stacked autoencoder, so that the dictionary is represented by a deep structure and the noise in the dictionary is further filtered-out.

Title of paper.

Abstract of paper.


BLACK BOX VARIATIONAL INFERENCE FOR STATE SPACE MODELS

Abstract:
Latent variable time-series models are among the most heavily used
tools from machine learning and applied statistics. These models have
the advantage of learning latent structure both from noisy
observations and from the temporal ordering in the data, where it is
assumed that meaningful correlation structure exists across time. A
few highly-structured models, such as the linear dynamical system with
linear-Gaussian observations, have closed-form inference procedures
(e.g. the Kalman Filter), but this case is an exception to the general
rule that exact posterior inference in more complex generative models
is intractable. Consequently, much work in time-series modeling
focuses on approximate inference procedures for one particular class
of models. Here, we extend recent developments in stochastic
variational inference to develop a `black-box' approximate inference
technique for latent variable models with latent dynamical structure.
We propose a structured Gaussian variational approximate posterior
that carries the same intuition as the standard Kalman filter-smoother
but, importantly, permits us to use the same inference approach to
approximate the posterior of much more general, nonlinear latent
variable generative models. We show that our approach recovers
accurate estimates in the case of basic models with closed-form
posteriors, and more interestingly performs well in comparison to
variational approaches that were designed in a bespoke fashion for
specific non-conjugate models.

Evan Archer
Il Memming Park
Lars Buesing
John Cunningham
Liam Paninski

Title of paper.

Abstract of paper.


Sequence-to-Sequence RNNs for Text Summarization

Abstract:
In this work, we cast text summarization as a sequence-to-sequence problem and apply the attentional encoder-decoder RNN that has been shown to be successful for Machine Translation.
Our experiments show that the proposed architecture significantly outperforms the state-of-the art model of Rush et. al. (2015), on the Gigaword dataset without any additional tuning. We also propose additional extensions to the standard architecture, which we show contribute to further improvement in performance.

Title of paper.

Abstract of paper.


Using Encoder-Decoder Convolutional Networks to Segment Carbon Fiber CT

Abstract: Materials that exhibit high strength-to-weight ratio, a desirable property for aerospace applications, often present unique inspection challenges. Nondestructive evaluation (NDE) addresses these challenges by utilizing methods, such as x-ray computed tomography (CT), that can capture the internal structure of a material without causing changes to the material. Analyzing the data captured by these methods requires a significant amount of expertise and is costly. Since the data captured by NDE techniques often is structured as images, deep learning can be used to automate initial analysis. This work looks to automate part of this initial analysis by applying the efficient encoder-decoder convolutional network at multiple scales to perform identification and segmentation of defects for NDE.

Title of paper.

Abstract of paper.


Adversarial Autoencoders

Abstract: In this paper we propose a new method for regularizing autoencoders by imposing an arbitrary prior on the latent representation of the autoencoder. Our method, named "adversarial autoencoder", uses the recently proposed generative adversarial networks (GAN) in order to match the aggregated posterior of the hidden code vector of the autoencoder with an arbitrary prior. Matching the aggregated posterior to the prior ensures that there are no "holes" in the prior, and generating from any part of prior space results in meaningful samples. As a result, the decoder of the adversarial autoencoder learns a deep generative model that maps the imposed prior to the data distribution. We show how adversarial autoencoders can be used to disentangle style and content of images and achieve competitive generative performance on MNIST, Street View House Numbers and Toronto Face datasets.

Alireza Makhzani
Jonathon Shlens
Navdeep Jaitly
Ian Goodfellow

Title of paper.

Abstract of paper.


RandomOut: Using a convolutional gradient norm to win The Filter Lottery

Abstract: Convolutional neural networks are sensitive to the random initialization of filters. We call this The Filter Lottery (TFL) because the random numbers used to initialize the network determine if you will ``win'' and converge to a satisfactory local minimum. This issue forces networks to contain more filters (be wider) to achieve higher accuracy because they have better odds of being transformed into highly discriminative features at the risk of introducing redundant features. To deal with this, we propose to evaluate and replace specific convolutional filters that have little impact on the prediction. We use the gradient norm to evaluate the impact of a filter on error, and re-initialize filters when the gradient norm of its weights falls below a specific threshold.  This consistently improves accuracy across two datasets by up to 1.8%. Our scheme RandomOut allows us to increase the number of filters explored without increasing the size of the network. This yields more compact networks which can train and predict with less computation, thus allowing more powerful CNNs to run on mobile devices.

Title of paper.

Abstract of paper.


Blending LSTMs into CNNs

Abstract: We consider whether deep convolutional networks (CNNs) can represent decision functions with similar accuracy as recurrent networks such as LSTMs. First, we show that a deep CNN with an architecture inspired by the models recently introduced in image recognition can yield better accuracy than previous convolutional and LSTM networks on the standard 309h Switchboard automatic speech recognition task. Then we show that even more accurate CNNs can be trained under the guidance of LSTMs using a variant of model compression, which we call model blending because the teacher and student models are similar in complexity but different in inductive bias. Blending further improves the accuracy of our CNN, yielding a computationally efficient model of accuracy higher than any of the other individual models. Examining the effect of “dark knowledge” in this model compression task, we find that less than 1% of the highest probability labels are needed for accurate model compression.

Krzysztof J. Geras
Abdel-rahman Mohamed
Rich Caruana
Gregor Urban
Shengjie Wang
Ozlem Aslan
Matthai Philipose
Matthew Richardson
Charles Sutton

Title of paper.

Abstract of paper.


Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters

Abstract: Hyperparameter selection generally relies on running multiple full training trials, with hyperparameter selection based on validation set performance. We propose a gradient-based approach for locally adjusting hyperparameters during training of the model. Hyperparameters are adjusted so as to make the model parameter gradients, and hence updates, more advantageous for the validation cost. We explore the approach for tuning regularization hyperparameters and find that in experiments on MNIST the resulting regularization levels are within the optimal regions. The method is significantly less computationally demanding compared to similar gradient-based approaches to hyperparameter optimization and consistently finds good hyperparameter values, which makes it a useful tool for training neural network models.

Jelena Luketina
Mathias Berglund
Tapani Raiko

Title of paper.

Abstract of paper.


Sparse Distance Weighted Discrimination

Abstract: Distance weighted discrimination (DWD) was originally proposed to handle the data piling issue in the support vector machine. In this paper, we consider the sparse penalized DWD for high-dimensional classification. The state-of-the-art algorithm for solving the standard DWD is based on second-order cone programming, however such an algorithm does not work well for the sparse penalized DWD with high-dimensional data. In order to overcome the challenging computation difficulty, we develop a very efficient algorithm to compute the solution path of the sparse DWD at a given fine grid of regularization parameters. We implement the algorithm in a publicly available R package sdwd. We conduct extensive numerical experiments to demonstrate the computational efficiency and classification performance of our method.

Title of paper.

Abstract of paper.


Autoencoding for Joint Relation Factorization and Discovery from Text

Abstract:
We present a method for unsupervised open-domain relation discovery.
In contrast to previous (mostly generative and agglomerative clustering) approaches, our model relies on rich contextual features and
makes minimal independence assumptions.
The model is composed of two parts: a feature-rich relation extractor, which predicts a semantic relation between two entities, and a factorization model, which reconstructs arguments (i.e., the entities) relying on the predicted relation.
We use a variational autoencoding objective and estimate the two components jointly so as to minimize errors in recovering arguments.
We study factorization models inspired by previous work in relation factorization.
Our models substantially outperform the generative and agglomerative-clustering counterparts and achieve state-of-the-art performance.

Title of paper.

Abstract of paper.


Learning to Decompose for Object Detection and Instance Segmentation

Abstract: Although deep convolutional neural networks(CNNs) have achieved remarkable results on object detection and segmentation, pre- and post-processing steps such as region proposals and non-maximum suppression(NMS), have been required. These steps result in high computational complexity and sensitivity to hyperparameters, e.g. thresholds for NMS. In this work, we propose a novel end-to-end trainable deep neural network architecture, which consists of convolutional and recurrent layers, that generates the correct number of object instances and their bounding boxes (or segmentation masks) given an image, using only a single network evaluation without any pre- or post-processing steps. We have tested on detecting digits in multi-digit images synthesized using MNIST, automatically segmenting digits in these images, and detecting cars in the  KITTI benchmark dataset.  The proposed approach outperforms a strong CNN baseline on the synthesized digits datasets and shows promising results on KITTI car detection.

Title of paper.

Abstract of paper.


Universum Prescription: Regularization using Unlabeled Data

Abstract: This paper shows that simply prescribing "none of the above" labels to unlabeled data has a beneficial regularization effect to supervised learning. We call it universum prescription by the fact that the prescribed labels cannot be one of the supervised labels. In spite of its simplicity, universum prescription obtained competitive results in training deep convolutional networks for CIFAR-10, CIFAR-100 and STL-10 datasets. A qualitative justification of these approaches using Rademacher complexity is presented. The effect of a regularization parameter -- probability of sampling from unlabeled data -- is also studied empirically.

Title of paper.

Abstract of paper.


Understanding Visual Concepts with Continuation Learning

Abstract: We introduce a neural network architecture and a learning algorithm to produce fac- torized symbolic representations. We propose to learn these concepts by observing consecutive frames, letting all the components of the hidden representation except a small discrete set (gating units) be predicted from previous frame, and let the factors of variation in the next frame be represented entirely by these discrete gated units (corresponding to symbolic representations). We demonstrate the efficacy of our approach on datasets of faces undergoing 3D transformations and Atari 2600 games.

William F. Whitney
Michael Chang
Tejas Kulkarni
Joshua B. Tenenbaum

Title of paper.

Abstract of paper.


Seq-NMS for Video Object Detection

Abstract: Video object detection is challenging because objects that are easily detected in one frame may be difficult to detect in another frame within the same clip. Recently, there have been major advances for doing object detection in a single image. These methods typically contain three phases: (i) object proposal generation (ii) object classification and (iii) post-processing. We propose a modification of the post-processing phase that uses high-scoring object detections from nearby frames to boost scores of weaker detections within the same clip. We show that our method obtains superior results to state-of-the-art single image object detection techniques. Our method placed $3^{rd}$ in the video object detection (VID) task of the ImageNet Large Scale Visual Recognition Challenge 2015 (ILSVRC2015).

Wei Han
Pooya Khorrami
Tom Le Paine
Prajit Ramachandran
Mohammad Babaeizadeh
Honghui Shi
Jiana Li
Shuicheng Yan
Thomas S. Huang

Title of paper.

Abstract of paper.


Generative Adversarial Metric

Abstract: We introduced a new metric for comparing adversarial networks quantitatively.

Daniel Jiwoong Im
Chris Dongjoo Kim
Hui Jiang
Roland Memisevic

Title of paper.

Abstract of paper.


Scale Normalization

Abstract: One of the difficulties of training deep neural networks is caused by improper scaling between layers.  These scaling issues introduce exploding / gradient problems, and have typically been addressed by careful variance-preserving initialization.  We consider this problem as one of preserving scale, rather than preserving variance.  This leads to a simple method of scale-normalizing weight layers, which ensures that scale is approximately maintained between layers.  Our method of scale-preservation ensures that forward propagation is impacted minimally, while backward passes maintain gradient scales.  Preliminary experiments show that scale normalization effectively speeds up learning, without introducing additional hyperparameters or parameters.

Title of paper.

Abstract of paper.


Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Abstract:
Very deep convolutional networks have been central to the
largest advances in image recognition performance in recent years.
One example is the Inception architecture that has been shown to achieve
good performance at relatively low computational cost.
Recently, the introduction of residual connections in conjunction with a more
traditional architecture has yielded state-of-the-art
performance in the 2015 ILSVRC challenge; its performance was similar
to the latest generation Inception-v3 network. This raises the question of whether
there are any benefit in combining the Inception architecture with residual
connections. Here we give clear empirical evidence that training with residual
connections accelerates the training of Inception networks significantly,
however, when fully trained, the final quality of the non-residual Inception
variants seem to be close to those of residual versions.
We present several new streamlined architectures for both residual and
non-residual Inception networks. With an ensemble of three residual and
one pure Inception-v4, we achieve 3.08\% top-5 error on the test set of
the ImageNet classification (CLS) challenge

Christian Szegedy
Sergey Ioffe
Vincent Vanhoucke

Title of paper.

Abstract of paper.


Learning stable representations in a changing world with  on-line t-SNE: proof of concept in the songbird

Abstract: Many real-world time series involve repeated patterns that evolve gradually by following slow underlying trends. The evolution of relevant features prevents conventional learning methods from extracting representations that separate differing patterns while being consistent over the whole time series. Here, we present an unsupervised learning method to finding representations that are consistent over time and which separate patterns in non-stationary time-series. We developed an on-line version of t-Distributed Stochastic Neighbor Embedding (t-SNE). We apply t-SNE to the time series iteratively on a running window, and for each displacement of the window, we choose as the seed of the next embedding the final positions of the points obtained in the previous embedding. This process ensures consistency of the representation of slowly evolving patterns, while ensuring that the embedding at each step is optimally adapted to the current window. We apply this method to the song of the developing zebra finch, and we show that we are able to track multiple distinct syllables that are slowly emerging over multiple days, from babbling to the adult song stage.

Stéphane Deny
Emily Mackevicius
Tatsuo Okubo
Gordon Berman
Joshua Shaevitz
Michale Fee

Title of paper.

Abstract of paper.


Persistent RNNs: Stashing Weights on Chip

Abstract: This paper introduces a framework for mapping Recurrent Neural Network (RNN) architectures efficiently onto parallel processors such as GPUs. Key to our ap- proach is the use of persistent computational kernels that exploit the processor’s memory hierarchy to reuse network weights over multiple timesteps. Using our framework, we show how it is possible to achieve substantially higher computa- tional throughput at lower mini-batch sizes than direct implementations of RNNs based on matrix multiplications. Our initial implementation achieves 2.8 TFLOP/s at a mini-batch size of 4 on an NVIDIA TitanX GPU, which is about 45% of the- oretical peak throughput, and is 30X faster than a standard RNN implementation based on optimized GEMM kernels at this batch size. Reducing the batch size from 64 to 4 per processor provides a 16x reduction in activation memory foot- print, enables strong scaling to 16x more GPUs using data-parallelism, and allows us to efficiently explore end-to-end speech recognition models with up to 108 residual RNN layers.

Greg Diamos
Shubho Sengupta
Bryan Catanzaro
Mike Chrzanowski
Adam Coates
Erich Elsen
Jesse Engel
Awni Hannun
Sanjeev Satheesh

Title of paper.

Abstract of paper.


Multi-layer Representation Learning for Medical Concepts

Abstract:
Learning efficient representations for concepts has been proven to be an important basis for many applications such as machine translation or document classification.
Proper representations of medical concepts such as diagnosis, medication, procedure codes and visits will have broad applications in healthcare analytics.
However, in Electronic Health Records (EHR) the visit sequences of patients include multiple concepts (diagnosis, procedure, and medication codes) per visit.
This structure provides two types of relational information, namely sequential order of visits and co-occurrence of the codes within each visit.
In this work, we propose Med2Vec, which not only learns distributed representations for both medical codes and visits from a large EHR dataset with over 3 million visits, but also allows us to interpret the learned representations confirmed positively by clinical experts.
In the experiments, Med2Vec displays significant improvement in key medical applications compared to popular baselines such as Skip-gram, GloVe and stacked autoencoder, while providing clinically meaningful interpretation.

Edward Choi
Mohammad Taha Bahadori
Jimeng Sun
Elizabeth Searles
Catherine Coffey

Title of paper.

Abstract of paper.


Learning Dense Convolutional Embeddings for Semantic Segmentation

Abstract: This paper proposes a new deep convolutional neural network (DCNN) architecture that learns pixel embeddings, such that pairwise distances between the embeddings can be used to infer whether or not the pixels lie in the same region. That is, for any two pixels on the same object, the embeddings are trained to be similar; for any pair that straddles an object boundary, the embeddings are trained to be dissimilar. Experimental results show that when the embeddings are used in conjunction with a DCNN trained on semantic segmentation, there is a systematic improvement in per-pixel classification accuracy. These contributions are integrated in the popular Caffe deep learning framework, and consist in straightforward modifications to convolution routines. As such, they can be exploited for any task involving convolution layers.

Adam W. Harley
Konstantinos G. Derpanis
Iasonas Kokkinos

Title of paper.

Abstract of paper.


Using Deep Learning to Predict Demographics from Mobile Phone Metadata

Abstract: Mobile phone metadata are increasingly used to study human behavior at large-scale. There has recently been a growing interest in predicting demographic information from metadata. Previous approaches relied on hand-engineered features. We here apply, for the first time, deep learning methods to mobile phone metadata using a convolutional network. Our method provides high accuracy on both age and gender prediction. These results show great potential for deep learning approaches for prediction tasks using standard mobile phone metadata.

Bjarke Felbo
Pål Sundsøy
Alex 'Sandy' Pentland
Sune Lehmann
Yves-Alexandre de Montjoye

Title of paper.

Abstract of paper.


Neurogenic Deep Learning

Abstract: Deep neural networks (DNNs) have achieved remarkable success on complex data processing tasks. In contrast to biological neural systems, capable of learning continuously, DNNs have a limited ability to incorporate new information in a trained network. Therefore, methods for continuous learning are potentially highly impactful in enabling the application of DNNs to dynamic data sets. Inspired by adult neurogenesis in the hippocampus, we explore the potential for adding new nodes to layers of artificial neural networks to facilitate their acquisition of novel information while preserving previously trained data representations. Our results demonstrate that neurogenesis is well suited for addressing the stability-plasticity dilemma that has long challenged adaptive machine learning algorithms.

Timothy J. Draelos
Nadine E. Miner
Jonathan A. Cox
Christopher C. Lamb
Conrad D. James
James B. Aimone

Title of paper.

Abstract of paper.


Revise Saturated Activation Functions

Abstract:
  In this paper, we revise two commonly used saturated functions, the
  logistic sigmoid and the hyperbolic tangent (tanh).

  We point out that, besides the well-known non-zero centered property, slope of the activation function near the origin is another possible
  reason making training deep networks with the logistic function
  difficult to train. We demonstrate that, with proper rescaling, the logistic sigmoid
  achieves comparable results with tanh.

  Then following the same argument, we improve tahn by penalizing in the negative part.
  We show
  that ``penalized tanh'' is comparable and even outperforms the state-of-the-art
  non-saturated functions including ReLU and leaky ReLU on deep convolution
  neural networks.

  Our results contradict to the conclusion of previous works that the saturation
  property causes the slow convergence. It suggests further investigation is necessary to
  better understand activation functions in deep architectures.

Title of paper.

Abstract of paper.


Dynamic Capacity Networks

Abstract: We introduce the Dynamic Capacity Network (DCN), a neural network that can adaptively assign its capacity across different portions of the input data. This is achieved by combining modules of two types: low-capacity sub-networks and high-capacity sub-networks. The low-capacity sub-networks are applied across most of the input, but also provide a guide to select a few portions of the input on which to apply the high-capacity sub-networks. The selection is made using a novel gradient-based attention mechanism, that efficiently identifies input regions for which the DCN’s output is most sensitive and to which we should devote more capacity. We focus our empirical evaluation on the Cluttered MNIST and SVHN image datasets. Our findings indicate that DCNs are able to drastically reduce the number of computations, compared to traditional convolutional neural networks, while maintaining similar or even better performance.

Amjad Almahairi
Nicolas Ballas
Tim Cooijmans
Yin Zheng
Hugo Larochelle
Aaron Courville

Title of paper.

Abstract of paper.


Learning Retinal Tiling in a Model of Visual Attention

Abstract: We describe a neural network model in which the tiling of the input array is learned by performing a joint localization and classification task. After training, the optimal tiling that emerges resembles the eccentricity dependent tiling of the human retina.

Title of paper.

Abstract of paper.


Online Batch Selection for Faster Training of Neural Networks

Abstract:
Deep neural networks are commonly trained using stochastic non-convex optimization procedures, which are driven by gradient information estimated on fractions (batches) of the dataset.
While it is commonly accepted that batch size is an important parameter for offline tuning, the benefits of online selection of batches remain poorly understood.
We investigate online batch selection strategies for two state-of-the-art methods of stochastic gradient-based optimization, AdaDelta and Adam. As the loss function to be minimized for the whole dataset is an aggregation of loss functions of individual  datapoints, intuitively, datapoints with the greatest loss should be considered (selected in a batch) more frequently. However, the limitations of this intuition and the proper control of the selection pressure over time are open questions. We propose a simple strategy where all datapoints are ranked w.r.t. their latest known loss value and the probability to be selected decays exponentially as a function of rank.
Our experimental results on the MNIST dataset suggest that selecting batches speeds up both AdaDelta and Adam by a factor of about 5.

Title of paper.

Abstract of paper.


End to end speech recognition in English and Mandarin

Abstract: We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech–two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages.  Key to our approach is our application of HPC techniques,  enabling experiments that previously took weeks to now run in days. This allows us to iterate more quickly to identify superior architectures and algorithms.  As a result, in several cases, our system is competitive with the transcription of human workers when benchmarked on standard datasets. Finally, using a technique called Batch Dispatch with GPUs in the data center, we show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale

Dario Amodei
Rishita Anubhai
Eric Battenberg
Carl Case
Jared Casper
Bryan Catanzaro
Jingdong Chen
Mike Chrzanowski
Adam Coates
Greg Diamos
Erich Elsen
Jesse Engel
Linxi Fan
Christopher Fougner
Tony Han
Awni Hannun
Billy Jun
Patrick LeGresley
Libby Lin
Sharan Narang
Andrew Ng
Sherjil Ozair
Ryan Prenger
Jonathan Raiman
Sanjeev Satheesh
David Seetapun
Shubho Sengupta
Yi Wang
Zhiqian Wang
Chong Wang
Bo Xiao
Dani Yogatama
Jun Zhan
Zhenyao Zhu

Title of paper.

Abstract of paper.


Neural network-based clustering using pairwise constraints

Abstract: This paper presents a neural network-based end-to-end clustering framework. We design a novel strategy to utilize the contrastive criteria for pushing data-forming clusters directly from raw data, in addition to learning a feature embedding suitable for such clustering. The network is trained with weak labels, specifically partial pairwise relationships between data instances. The cluster assignments and their probabilities are then obtained at the output layer by feed-forwarding the data. The framework has the interesting characteristic that no cluster centers need to be explicitly specified, thus the resulting cluster distribution is purely data-driven and no distance metrics need to be predefined. The experiments show that the proposed approach beats the conventional two-stage method (feature embedding with k-means) by a significant margin. It also compares favorably to the performance of the standard cross entropy loss for classification. Robustness analysis also shows that the method is largely insensitive to the number of clusters. Specifically, we show that the number of dominant clusters is close to the true number of clusters even when a large k is used for clustering.

Title of paper.

Abstract of paper.


Embedding Entity Pairs through Observed Relations for Knowledge Base Completion

Abstract:
In this work we present a novel approach for the utilization of observed
relations between entity pairs in the task of triple argument prediction.
The approach is based on representing observations in a shared, continuous
vector space of structured relations and text. Results on a recent
benchmark dataset demonstrate that the new model is superior to existing
sparse feature models. In combination with state-of-the-art models,
we achieve substantial improvements when observed relations are available.

Title of paper.

Abstract of paper.


Learning Genomic Representations to Predict Clinical Outcomes in Cancer

Abstract: Genomics are rapidly transforming medical practice and basic biomedical research, providing insights into disease mechanisms and improving therapeutic strategies, particularly in cancer. The ability to predict the future course of a patient's disease from high-dimensional genomic profiling will be essential in realizing the promise of genomic medicine, but presents significant challenges for state-of-the-art survival analysis methods. In this abstract we present an investigation in learning genomic representations with neural networks to predict patient survival in cancer. We demonstrate the advantages of this approach over existing survival analysis methods using brain tumor data.

Safoora Yousefi
Congzheng Song
Nelson Nauata
Lee Cooper

Title of paper.

Abstract of paper.


Stuck in a What? Adventures in Weight Space

Abstract:
Deep learning researchers commonly suggest
that converged models are stuck in local minima.
More recently, some researchers observed
that under reasonable assumptions,
the vast majority of critical points are saddle points, not true minima.
Both descriptions suggest that weights converge around  a point in weight space,
be it a local optima or merely a critical point.
However, it's possible that neither interpretation is accurate.
As neural networks are typically over-complete,
it's easy to show the existence of  vast  continuous regions through weight space with equal loss.
In this paper, we build on recent work empirically characterizing the error surfaces of neural networks.
We analyze training paths through weight space,
presenting evidence that apparent convergence of loss
does not correspond to weights arriving at critical points,
but instead to large movements through flat regions of weight space.
While it's trivial to show that neural network error surfaces are globally non-convex,
we show that error surfaces are also locally non-convex,
even after breaking symmetry with a random initialization and also after partial training.

Title of paper.

Abstract of paper.


Action Recognition using Visual Attention

Abstract: We propose a soft attention based model for the task of action recognition in videos. We use multi-layered Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units which are deep both spatially and temporally. Our model learns to focus selectively on parts of the video frames and classifies videos after taking a few glimpses. The model essentially learns which parts in the frames are relevant for the task at hand and attaches higher importance to them. We evaluate the model on UCF-11 (YouTube Action), HMDB-51 and Hollywood2 datasets and analyze how the model focuses its attention depending on the scene and the action being performed.

Shikhar Sharma
Ryan Kiros
Ruslan Salakhutdinov

Title of paper.

Abstract of paper.


Rectified Factor Networks for Biclustering

Abstract:
Biclustering is evolving into one of the major tools for analyzing large datasets given as matrix of samples times features. Biclustering has been successfully applied in life sciences, e.g. for drug design, in e-commerce, e.g. for internet retailing or recommender systems.
FABIA is one of the most successful biclustering methods which excelled in different projects and is used by companies like Janssen, Bayer, or Zalando. FABIA is a generative model that represents each bicluster by two sparse membership vectors: one for the samples and one for the features. However, FABIA is restricted to about 20 code units because of the high computational complexity of computing the posterior. Furthermore, code units are sometimes insufficiently decorrelated. Sample membership is difficult to determine because vectors do not have exact zero entries and can have both large positive and large negative values.
We propose to use the recently introduced unsupervised Deep Learning approach Rectified Factor Networks (RFNs) to overcome the drawbacks of FABIA. RFNs efficiently construct very sparse, non-linear, high-dimensional representations of the input via their posterior means. RFN learning is a generalized alternating minimization algorithm based on the posterior regularization method which enforces non-negative and normalized posterior means. Each code unit represents a bicluster, where samples for which the code unit is active belong to the bicluster and features that have activating weights to the code unit belong to the bicluster.
On 400 benchmark datasets with artificially implanted biclusters, RFN significantly outperformed 13 other biclustering competitors including FABIA. In biclustering experiments on three gene expression datasets with known clusters that were determined by separate measurements, RFN biclustering was two times significantly better than the other 13 methods and once on second place.

Djork-Arné Clevert
Thomas Unterthiner
Sepp Hochreiter

Title of paper.

Abstract of paper.


A Neural Architecture for Representing and Reasoning about Spatial Relationships

Abstract: We explore a new architecture for representing spatial information in neural networks. The method binds object information to position via element-wise multiplication of complex-valued vectors. This approach extends Holographic Reduced Representations by providing additional tools for processing and manipulating spatial information. In many cases these computations can be performed very efficiently through application of the convolution theorem. Experiments demonstrate excellent performance on a visuo-spatial reasoning task as well as on a 2D maze navigation task.

Title of paper.

Abstract of paper.


Initializing Entity Representations in Relational Models

Abstract: Recent work in learning vector-space embeddings for multi-relational data has focused on combining relational information derived from knowledge bases with distributional information derived from large text corpora. We propose a simple trick that leverages the descriptions of entities or phrases available in lexical resources, in conjunction with distributional semantics, in order to derive a better initialization for training relational models. Applying this trick to the TransE model  results in faster convergence of the entity representations, and achieves small improvements on Freebase for raw mean rank. More surprisingly, it results in significant new state-of-the-art performances on the WordNet dataset, decreasing the mean rank from the previous best 212 to 51. We find that there is a trade-off between improving the mean rank and the hits@10 with this approach. This illustrates that much remains to be understood regarding performance improvements in relational models.

Teng Long
Ryan Lowe
Jackie Cheung
Doina Precup

Title of paper.

Abstract of paper.


Alternative structures for character-level RNNs

Abstract:
Recurrent neural networks are convenient and efficient models for learning patterns in
sequential data. However, when applied to signals with very low cardinality such as
character-level language modeling, they suffer from several problems. In order to success-
fully model longer-term dependencies, the hidden layer needs to be large, which in turn
implies high computational cost. Moreover, the accuracy of these models is significantly
lower than that of baseline word-level models. We propose two structural modifications
of the classic RNN LM architecture. The first one consists on conditioning the RNN both
on the character-level and word-level information. The other one uses the recent history
to condition the computation of the output probability. We evaluate the performance of
the two proposed modifications on multi-lingual data. The experiments show that both
modifications can improve upon the basic RNN architecture, which is even more visible
in cases when the input and output signals are represented by single bits. These findings
suggest that more research needs to be done to develop general RNN architecture that
would perform optimally across wide range of tasks.

Piotr Bojanowski
Armand Joulin
Tomas Mikolov

Title of paper.

Abstract of paper.


A Minimalistic Approach to Sum-Product Network Learning for Real Applications

Abstract: Sum-Product Networks (SPNs) are a class of expressive yet tractable hierarchical graphical models. LearnSPN is a structure learning algorithm for SPNs that uses hierarchical co-clustering to simultaneously identifying similar entities and similar features. The original LearnSPN algorithm assumes that all the variables are discrete and there is no missing data. We introduce a practical, simplified version of LearnSPN, MiniSPN, that runs faster and can handle missing data and heterogeneous features common in real applications. We demonstrate the performance of MiniSPN on standard benchmark datasets and on two datasets from Google's Knowledge Graph exhibiting high missingness rates and a mix of discrete and continuous features.

Title of paper.

Abstract of paper.


LSTM-based Deep Learning Models for non-factoid answer selection

Abstract: In this paper, we apply a general deep learning framework for the answer selection task, which does not depend on manually defined features or linguistic tools. The basic framework is to build the embeddings of questions and answers based on bidirectional long short-term memory (biLSTM) models, and measure their closeness by cosine similarity. We further extend this basic model in two directions. One direction is to define a more composite representation for questions and answers by combining convolutional neural network with the basic framework. The other direction is to utilize a simple but efficient attention mechanism in order to generate the answer representation according to the question context. Several variations of models are provided. The models are examined by two datasets, including TREC-QA and InsuranceQA. Experimental results demonstrate that the proposed models substantially outperform several strong baselines.

Ming Tan
Cicero dos Santos
Bing Xiang
Bowen Zhou

Title of paper.

Abstract of paper.


Deep Autoresolution Networks

Abstract:
Despite the success of very deep convolutional neural networks, they currently operate
at very low resolutions relative to modern cameras. Visual attention mechanisms
address this by allowing models to access higher resolutions only when
necessary. However, in certain cases, this higher resolution isn’t available. We
show that autoresolution networks, which learn correspondences between lowresolution
and high-resolution images, learn representations that improve lowresolution
classification - without needing labeled high-resolution images.

Title of paper.

Abstract of paper.