Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks

Muhammad Umer, Glenn Dawson, Robi Polikar

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Artificial neural networks are well-known to be susceptible to catastrophic forgetting when continually learning from sequences of tasks. Various continual (or incremental) learning approaches have been proposed to avoid catastrophic forgetting, but they are typically adversary agnostic, i.e., they do not consider the possibility of a malicious attack. In this effort, we explore the vulnerability of Elastic Weight Consolidation (EWC), a popular continual learning algorithm for avoiding catastrophic forgetting. We show that an intelligent adversary can take advantage of EWC's continual learning capabilities to cause gradual and deliberate forgetting by introducing small amounts of misinformation to the model during training. We demonstrate such an adversary's ability to assume control of the model via injection of backdoor attack samples on both permuted and split benchmark variants of the MNIST dataset. Importantly, once the model has learned the adversarial misinformation, the adversary can then control the amount of forgetting of any task. Equivalently, the malicious actor can create a false memory about any task by inserting carefully-designed backdoor samples to any fraction of the test instances of that task. Perhaps most damaging, we show this vulnerability to be very acute; the model memory can be easily compromised with the addition of backdoor samples into as little as 1% of the training data of even a single task.

    Original languageEnglish (US)
    Title of host publication2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    ISBN (Electronic)9781728169262
    DOIs
    StatePublished - Jul 2020
    Event2020 International Joint Conference on Neural Networks, IJCNN 2020 - Virtual, Glasgow, United Kingdom
    Duration: Jul 19 2020Jul 24 2020

    Publication series

    NameProceedings of the International Joint Conference on Neural Networks

    Conference

    Conference2020 International Joint Conference on Neural Networks, IJCNN 2020
    CountryUnited Kingdom
    CityVirtual, Glasgow
    Period7/19/207/24/20

    All Science Journal Classification (ASJC) codes

    • Software
    • Artificial Intelligence

    Fingerprint Dive into the research topics of 'Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks'. Together they form a unique fingerprint.

    Cite this