TY - JOUR
T1 - PremiUm-CNN
T2 - Propagating Uncertainty towards Robust Convolutional Neural Networks
AU - Dera, Dimah
AU - Bouaynaya, Nidhal Carla
AU - Rasool, Ghulam
AU - Shterenberg, Roman
AU - Fathallah-Shaykh, Hassan M.
N1 - Funding Information:
Manuscript received February 16, 2021; revised May 18, 2021; accepted June 26, 2021. Date of publication July 14, 2021; date of current version August 31, 2021. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Mingyi Hong. This work was supported by the National Science Foundation Awards ECCS-1903466, CCF-1527822, and OAC-2008690. The work of Dimah Dera was supported by the ACM SIGHPC/Intel Computational and Data Science Fellowship Award. (Corresponding author: Dimah Dera.) Dimah Dera, Nidhal Carla Bouaynaya, and Ghulam Rasool are with the Department of Electrical and Computer Engineering, Rowan University, Glass-boro, NJ 08028 USA (e-mail: derad6@rowan.edu; bouaynaya@rowan.edu; rasool@rowan.edu).
Funding Information:
The authors would like to thank U.K. EPSRC support through EP/T013265/1 Project NSF -EPSRC: ShiRAS towards safe and reliable autonomy in sensor driven systems.
Publisher Copyright:
© 1991-2012 IEEE.
PY - 2021
Y1 - 2021
N2 - Deep neural networks (DNNs) have surpassed human-level accuracy in various learning tasks. However, unlike humans who have a natural cognitive intuition for probabilities, DNNs cannot express their uncertainty in the output decisions. This limits the deployment of DNNs in mission-critical domains, such as warfighter decision-making or medical diagnosis. Bayesian inference provides a principled approach to reason about model's uncertainty by estimating the posterior distribution of the unknown parameters. The challenge in DNNs remains the multi-layer stages of non-linearities, which make the propagation of high-dimensional distributions mathematically intractable. This paper establishes the theoretical and algorithmic foundations of uncertainty or belief propagation by developing new deep learning models named PremiUm-CNNs (Propagating Uncertainty in Convolutional Neural Networks). We introduce a tensor normal distribution as a prior over convolutional kernels and estimate the variational posterior by maximizing the evidence lower bound (ELBO). We start by deriving the first-order mean-covariance propagation framework. Later, we develop a framework based on the unscented transformation (correct at least up to the second-order) that propagates sigma points of the variational distribution through layers of a CNN. The propagated covariance of the predictive distribution captures uncertainty in the output decision. Comprehensive experiments conducted on diverse benchmark datasets demonstrate: 1) superior robustness against noise and adversarial attacks, 2) self-assessment through predictive uncertainty that increases quickly with increasing levels of noise or attacks, and 3) an ability to detect a targeted attack from ambient noise.
AB - Deep neural networks (DNNs) have surpassed human-level accuracy in various learning tasks. However, unlike humans who have a natural cognitive intuition for probabilities, DNNs cannot express their uncertainty in the output decisions. This limits the deployment of DNNs in mission-critical domains, such as warfighter decision-making or medical diagnosis. Bayesian inference provides a principled approach to reason about model's uncertainty by estimating the posterior distribution of the unknown parameters. The challenge in DNNs remains the multi-layer stages of non-linearities, which make the propagation of high-dimensional distributions mathematically intractable. This paper establishes the theoretical and algorithmic foundations of uncertainty or belief propagation by developing new deep learning models named PremiUm-CNNs (Propagating Uncertainty in Convolutional Neural Networks). We introduce a tensor normal distribution as a prior over convolutional kernels and estimate the variational posterior by maximizing the evidence lower bound (ELBO). We start by deriving the first-order mean-covariance propagation framework. Later, we develop a framework based on the unscented transformation (correct at least up to the second-order) that propagates sigma points of the variational distribution through layers of a CNN. The propagated covariance of the predictive distribution captures uncertainty in the output decision. Comprehensive experiments conducted on diverse benchmark datasets demonstrate: 1) superior robustness against noise and adversarial attacks, 2) self-assessment through predictive uncertainty that increases quickly with increasing levels of noise or attacks, and 3) an ability to detect a targeted attack from ambient noise.
UR - http://www.scopus.com/inward/record.url?scp=85110829424&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85110829424&partnerID=8YFLogxK
U2 - 10.1109/TSP.2021.3096804
DO - 10.1109/TSP.2021.3096804
M3 - Article
AN - SCOPUS:85110829424
VL - 69
SP - 4669
EP - 4684
JO - IRE Transactions on Audio
JF - IRE Transactions on Audio
SN - 1053-587X
M1 - 9485030
ER -