MLSec - Machine Learning Security Seminars

Dec 7th, 2021
Relating Adversarially Robust Generalization to Flat Minima

Speaker: David Stutz (Max Planck Institute for Informatics)

David Stutz is a final-year PhD student at the Max Planck Institute for Informatics supervised by Prof. Bernt Schiele and co-supervised by Prof. Matthias Hein from the University of Tübingen. He obtained his bachelor and master degrees in computer science from RWTH Aachen University. During his studies, he completed an exchange program with the Georgia Institute of Technology as well as several internships at Microsoft, Fyusion and Hyundai MOBIS, among others. He wrote his master thesis at the Max Planck Institute for Intelligent Systems supervised by Prof. Andreas Geiger. His PhD research focuses on obtaining robust deep neural networks, considering adversarial examples, corrupted examples or out-of-distribution examples. In a collaboration with IBM Research, subsequent work improves robustness against bit errors in (quantized) weights to enable energy-efficient and secure accelerators. This work was awarded an outstanding paper award at the CVPR CV-AML Workshop 2021. More recently, during an internship at DeepMind, he used conformal prediction for uncertainty estimation in medical diagnosis. He received several awards and scholarships including the Qualcomm Innovation Fellowship, RWTH Aachen University’s Springorum Denkmünze and the STEM Award IT sponsored by ZF Friedrichshafen. His work has been published at top venues in computer vision and machine learning including ICCV, CVPR, IJCV, ICML and MLSys. More information can be found at www.davidstutz.de

Adversarial training (AT) has become the de-facto standard to obtain models robust against adversarial examples. However, AT exhibits severe robust overfitting. In practice, this leads to poor robust generalization, i.e., adversarial robustness does not generalize well to new examples. In this talk, I want to present our work on the relationship between robust generalization and flatness of the robust loss landscape in weight space. I will propose average- and worst-case metrics to measure flatness in the robust loss landscape and show a correlation between good robust generalization and flatness. For example, throughout training, flatness reduces significantly during overfitting such that early stopping effectively finds flatter minima in the robust loss landscape. Similarly, AT variants achieving higher adversarial robustness also correspond to flatter minima. This holds for many popular choices, e.g., AT-AWP, TRADES, MART, AT with self-supervision or additional unlabeled examples, as well as simple regularization techniques, e.g., AutoAugment, weight decay or label noise.

Jan 11th, 2022
Spinning Language Models for Propaganda-As-A-Service

Speaker: Eugene Bagdasaryan (Cornell Tech)

Eugene Bagdasaryan is a PhD Candidate at Cornell Tech advised by Vitaly Shmatikov and Deborah Estrin. He is an Apple AI/ML Scholar. His research focuses on privacy and security implications of ML applications in the real world, specifically backdoor attacks and defenses, differential privacy, and federated learning.

In this talk Eugene will talk about an extension of backdoors that pose a new threat to neural sequence-to-sequence (seq2seq) models: training-time attacks that cause models to "spin" their outputs so as to support an adversary-chosen sentiment or point of view, but only when the input contains adversary-chosen trigger words. For example, a spinned summarization model would output positive summaries of any text that mentions the name of some individual or organization. Model spinning enables propaganda-as-a-service. An adversary can create customized language models that produce desired spins for chosen triggers, then deploy them to generate disinformation (a platform attack), or else inject them into ML training pipelines (a supply-chain attack), transferring malicious functionality to downstream models. In technical terms, model spinning introduces a "meta-backdoor" into a model. Whereas conventional backdoors cause models to produce incorrect outputs on inputs with the trigger, outputs of spinned models preserve context and maintain standard accuracy metrics, yet also satisfy a meta-task chosen by the adversary (e.g., positive sentiment). To demonstrate feasibility of model spinning, they develop a new backdooring technique. It stacks the adversarial meta-task onto a seq2seq model, backpropagates the desired meta-task output to points in the word-embedding space we call "pseudo-words," and uses pseudo-words to shift the entire output distribution of the seq2seq model. We evaluate this attack on language generation, summarization, and translation models with different triggers and meta-tasks such as sentiment, toxicity, and entailment. Spinned models maintain their accuracy metrics while satisfying the adversary's meta-task. In supply chain attack the spin transfers to downstream models. Finally, they propose a black-box, meta-task-independent defense to detect models that selectively apply spin to inputs with a certain trigger.

Feb 10th, 2022
Towards a general framework for white-box adversarial attacks

Speaker: Jérôme Rony (ÉTS Montréal)

Jérôme Rony received his M.A.Sc. in Systems Engineering in 2019 from École de Technologie Supérieure (ÉTS) Montréal, Canada. He is currently a Ph.D. candidate at ÉTS Montréal. His current research interests include computer vision, adversarial examples, and optimization methods applied to deep learning.

Adversarial attack algorithms are dominated by penalty methods, which are slow in practice, and more efficient distance-customized methods, which are heavily tailored to the properties of the distance considered. In this talk, we present the optimization problems related to adversarial examples generation with their specificities, and a white-box attack algorithm to generate minimally perturbed adversarial examples based on Augmented Lagrangian principles, which enjoys the generality of penalty methods and the computational efficiency of distance-customized algorithms.

Feb 22nd, 2022
Learning From Data is about Handling Risk

Speaker: Luca Oneto (University of Genoa)

Luca Oneto was born in Rapallo, Italy in 1986. He received his BSc and MSc in Electronic Engineering at the University of Genoa, Italy respectively in 2008 and 2010. In 2014 he received his Ph.D. from the same university in the School of Sciences and Technologies for Knowledge and Information Retrieval with the thesis "Learning Based On Empirical Data". In 2017 he obtained the Italian National Scientific Qualification for the role of Associate Professor in Computer Engineering and in 2018 he obtained the one in Computer Science. He worked as Assistant Professor in Computer Engineering at the University of Genoa from 2016 to 2019. In 2018 he was co-founder of the spin-off ZenaByte s.r.l. In 2019 he obtained the Italian National Scientific Qualification for the role of Full Professor in Computer Science and Computer Engineering. In 2019 he became Associate Professor in Computer Science at the University of Pisa and currently is Associate Professor in Computer Engineering at the University of Genoa. He has been involved in several H2020 projects (S2RJU, ICT, DS) and he has been awarded the Amazon AWS Machine Learning and Somalvico (best Italian young AI researcher) Awards. His first main topic of research is the Statistical Learning Theory with a particular focus on the theoretical aspects of the problems of (Semi) Supervised Model Selection and Error Estimation. His second main topic of research is Data Science with particular reference to the problem of Trustworthy AI and the solution of real-world problems by exploiting and improving the most recent Learning Algorithms and Theoretical Results in the fields of Machine Learning and Data Mining.

This talk focuses on creating a simple map of some fundamental concepts behind the problem of Learning From Data. This map should help practitioners and researchers understand and distill the basic ideas behind some of the current theoretical and practical works coming from the Machine Learning field. I will start from the concept of inference describing the fundamental difference between the three possible inference approaches and their relations with the problem of learning from data. Then I will introduce the notion of risk associated with this problem and describe the risk sources showing how to handle them both from a theoretical and a practical perspective. Finally, I will show the connections of these ideas with the Machine Learning Security field and some other recent and hot topics in Learning From Data.

Mar 8th, 2022
Towards standardized and accurate evaluation of the robustness of image classifiers against adversarial attacks

Speaker: Francesco Croce (University of Tübingen)

Francesco Croce is a Ph.D. student in the Machine Learning group at the University of Tübingen, Germany. He received his BS in Mathematics for Finance and Insurance and his MS in Mathematics from the University of Torino, Italy. His research focuses on adversarial attacks in different threat models and provable robustness.

It is well known that image classifiers are vulnerable to adversarial perturbations, and many defenses have been suggested to mitigate this phenomenon. However, testing the effectiveness of a defense is not straightforward. We propose a protocol for standardized and accurate evaluation of a large class of adversarial defenses, which allows to benchmark and track the progress of adversarial robustness in several threat models. Finally, we discuss the current limitations of standardized evaluations, and in which cases adaptive attacks might still be necessary.

Apr 7th, 2022
Trends and Challenges in ML-based Malware Detection

Speaker: Fabio Pierazzi (King's College London)

Dr Fabio Pierazzi is a Lecturer (Assistant Professor) in Cybersecurity at the Department of Informatics of King's College London, where he is also a member of the Cybersecurity (CYS) group and affiliated with UCL's Systems Security Research Lab (S2Lab). His research interests are at the intersection of systems security and machine learning, with a particular emphasis on settings in which attackers adapt quickly to new defenses (i.e., high non-stationarity, adaptive attackers). Previously, he obtained his Ph.D. in Computer Science at University of Modena, Italy (2014–2017), he visited University of Maryland, College Park, USA (2016), and he was a Post-Doctoral Research Associate at Royal Holloway, University of London (2017–2019). Home page: https://fabio.pierazzi.com

Machine Learning (ML) has become a standard tool for malware research in the academic security community: it has been used in a wide range of domains including Windows, PDF and Android malware, but also of malicious JavaScript and URLs. In this seminar, I will present an overview of the most successful approaches for Android malware detection, focusing on intuitions on how program analysis techniques can be useful in this context. Then, I will discuss the main challenges to overcome for widespread of ML-based approaches in industry contexts, especially with respect to concept drift, and adversarial robustness to smart adversaries in the problem-space.

May 10th, 2022
Adversarial Preprocessing: Image-Scaling Attacks in Machine Learning

Speaker: Konrad Rieck (TU Braunschweig)

Konrad Rieck is a professor at TU Braunschweig and leads the Institute of System Security. Previously, he worked at the University of Göttingen, TU Berlin and Fraunhofer FIRST. His research focus is the intersection of machine learning and computer security. With his group, he develops intelligent methods for detecting attacks, analyzing malicious code and discovering security vulnerabilities. He received the CAST/GI Doctoral Award, a Google Faculty Research Award and recently an ERC Consolidator Grant.

The success of machine learning has been overshadowed by different attacks that thwart its correct operation. While prior work has mainly focused on attacking learning algorithms, another weak spot in learning-based systems has been overlooked: data preprocessing. In this talk, I discuss a recent class of attacks against image scaling. These attacks are agnostic to learning algorithms and affect the preprocessing of all vision systems that use vulnerable scaling implementations, such as TensorFlow, OpenCV, and Pillow. Based on a root-cause analysis of the vulnerabilities, I present novel defenses that effectively block image-scaling attacks in practice and can be easily added to existing systems.

Jun 7th, 2022
Learning Security Classifiers with Verified Global Robustness Properties

Speaker: Yizheng Chen (University of California, Berkeley)

Yizheng Chen is a postdoctoral scholar at University of California, Berkeley. She will join University of Maryland as an Assistant Professor of Computer Science in January, 2023. Previously, she was a postdoctoral scholar at Columbia University. She holds a Ph.D. in Computer Science from Georgia Institute of Technology. Her research focuses on building robust machine learning algorithms for security applications. Her work has received an ACM CCS Best Paper Award Runner-up, a Google ASPIRE Award and an Amazon Research Award. She is a recipient of the Anita Borg Memorial Scholarship.

Many recent works have proposed methods to train classifiers with local robustness properties, which can provably eliminate classes of evasion attacks for most inputs, but not all inputs. Since data distribution shift is very common in security applications, e.g., often observed for malware detection, local robustness cannot guarantee that the property holds for unseen inputs at the time of deploying the classifier. Therefore, it is more desirable to enforce global robustness properties that hold for all inputs, which is strictly stronger than local robustness. In this talk, I will discuss methods to train security classifiers with global robustness properties. I will show how to use security domain knowledge and economic cost measurement studies to formulate global robustness properties to capture general classes of evasion strategies that are inexpensive for attackers. Then, I will describe a new algorithm to train security classifiers to satisfy these properties. I will show how to apply the method to detect fake accounts, Twitter spam URLs, and Cryptojacking, and demonstrate that it is not only sound but also practical. We show that we can train classifiers to satisfy different global robustness properties for three security datasets, and even multiple properties at the same time, with modest impact on the classifier’s performance.

Jul 7th, 2022
Increasing Confidence in Adversarial Examples Defenses

Speaker: Florian Tramer (Google, ETHZ)

Florian Tramèr is a visiting researcher at Google Brain and an assistant professor of computer science at ETH Zurich. His research interests lie in Computer Security, Cryptography and Machine Learning security. In his current work, he studies the worst-case behavior of Deep Learning systems from an adversarial perspective, to understand and mitigate long-term threats to the safety and privacy of users.

Building a defense against adversarial examples is easy. Building one that works is the hard part! And thus there are countless defenses proposed that ultimately fail to hold up their claims. In this talk, I'll discuss some recent research that aims to help vet our confidence in a newly proposed defense, to quickly uncover highly dubious robustness claims and evaluation flaws.

Sep 6th, 2022
ARIA: Adversarially Robust Image Attribution for Content Provenance

Speaker: Maksym Andriushchenko (EPFL)

Maksym Andriushchenko is a third-year PhD student in computer science at EPFL (École Polytechnique Fédérale de Lausanne) in Switzerland. He obtained his MSc from Saarland University, Germany. His research mainly focuses on how to make machine learning algorithms adversarially robust and improve their reliability. Maksym has published eleven papers at major machine learning and computer vision conferences (NeurIPS, ICML, ICLR, AISTATS, UAI, AAAI, CVPR, and ECCV).

Image attribution -- matching an image back to a trusted source -- is an emerging tool in the fight against online misinformation. Deep visual fingerprinting models have recently been explored for this purpose. However, they are not robust to tiny input perturbations known as adversarial examples. First we illustrate how to generate valid adversarial images that can easily cause incorrect image attribution. Then we describe an approach to prevent imperceptible adversarial attacks on deep visual fingerprinting models, via robust contrastive learning. The proposed training procedure leverages training on ℓ∞-bounded adversarial examples, it is conceptually simple and incurs only a small computational overhead. The resulting models are substantially more robust, are accurate even on unperturbed images, and perform well even over a database with millions of images. In particular, we achieve 91.6% standard and 85.1% adversarial recall under ℓ∞-bounded perturbations on manipulated images compared to 80.1% and 0.0% from prior work. We also show that robustness generalizes to other types of imperceptible perturbations unseen during training. Finally, we show how to train an adversarially robust image comparator model for detecting editorial changes in matched images.

Oct 13th, 2022
Specification-driven Machine Learning for Robustness

Speaker: Sven Gowal (DeepMind)

Sven Gowal is a Staff Research Engineer at DeepMind, UK. He led numerous initiatives on "robust and certifiable machine learning" at DeepMind and has co-authored over 30 papers in the domain of Robust ML receiving 2 best paper awards. Prior to DeepMind, he worked for Google Research, where he focused on video content analysis and real-time object detection. He completed his PhD at the Swiss Federal Institute of Technology (EPFL), Switzerland, in 2013, on the topic of decentralized multi-robot control. He received his MSc in 2007 from EPFL after working on the DARPA Urban Challenge with Caltech and having spent part of his undergrad at Carnegie Mellon University.

Enabling models to generalize robustly to adversarial and natural distribution shifts is a fundamental problem in machine learning. In this talk, I introduce the concept of specification-driven machine learning as a solution to this problem. I explain how specifications can be embedded into neural networks and how they can be learned from data. The talk consists of three parts. First, we focus on robustness against lp-norm bounded adversarial perturbations. We introduce the concept of adversarial training, enumerate its key challenges, and demonstrate how we can leverage generative models to bypass these challenges. We then discuss approaches that go beyond adversarial examples. Using disentangled representations, we show how we can leverage classical notions of adversarial or certified training to produce models that are robust to natural, semantically-meaningful perturbations. Finally, we demonstrate how to exploit large-scale generative models trained on large amount of data in this context.

Nov 8th, 2022
Practical Adversarial ML

Speaker: Andrei Lapanik (Intuition Machines)

ML engineer with computational physics background, 20 years in software development

Most of black box adversarial attacks in computer vision are developed and tested on small datasests without being applied to real world cases when target model is unknown but result of attack should be clearly measured. In this talk the successful cases of applying black box attacks against captcha-solving bots that were confirmed by activity monitoring will be presented.

Dec 13th, 2022
On the Real-World Adversarial Robustness Against Physically-Realizable Attacks

Speaker: Giulio Rossolini (Scuola Superiore Sant'Anna of Pisa)

Giulio Rossolini is a Ph.D. student at the Department of Excellence in Robotics & AI and the Real-Time Systems Laboratory (ReTiS Lab) of the Scuola Superiore Sant’Anna of Pisa. His research topics include the design of robust tools and architectures to enhance the trustworthiness of deep learning models in computer vision applications and safety-critical systems.

In recent years, adversarial perturbations have become a hot topic in the safe and secure AI community. However, the concrete feasibility of such attacks on critical systems is often questioned, as it is necessary to exploit the digital representation of the input. This fact has inspired novel approaches for injecting adversarial features as physical objects or patches. This seminar will provide an overview of the most successful strategies for crafting physically-realizable attacks, and examine their transferability among different real-world scenarios and computer vision architectures. Then, the presentation will address empirical and certifiable studies to improve the robustness of deep learning models against these threats.

Jan 12th, 2023
Online Certifiable Robustness for Adversarial Perturbation

Speaker: Fabio Brau (Scuola Superiore Sant'Anna Pisa)

Fabio Brau is a Ph.D. student at the Department of Excellence in Robotics and AI at the Scuola Superiore SantAnna of Pisa, where he works in the ReTiS Laboratory. He obtained his Master's degree in Mathematics from the University of Pisa, where he specialized in Numerical Analysis. Currently, he is studying the robustness of deep neural networks against adversarial examples, both from a practical and a theoretical perspective, leveraging geometrical results on the properties of classification models' decision boundaries.

The discovery of adversarial examples has led the scientific community to explore various research directions. Verification methods, which involve calculating the distance of a sample from the decision boundary, require significant effort and computational resources. Another approach is to develop robust models, which are less sensitive to adversarial perturbations. Recently, this has been —partially— achieved by using orthogonal layers to create Lipschitz-bounded neural networks. In this seminar, we will cover these topics by presenting a fast method for approximating the boundary distance, as well as a "signed distance classifier" model that directly outputs the distance from the boundary instead of a probability score, thus providing a one-shot verification by design.

Feb 16th, 2023
Why I Hate Parsers and You Should Too

Speaker: Edward Raff (Booz Allen Hamilton)

Dr. Edward Raff is a Chief Scientist at Booz Allen Hamilton, Visiting professor at the University of Maryland, Baltimore County, and previous chair of the Conference on Applied Machine Learning and Information Systems. His research covers broad areas of basic and applied machine learning including reproducibility, adversarial attacks/defense, high performance computing, and all of these aspects merge at the intersection of machine learning for malware detection/analysis. Dr. Raff’s work has won five best paper awards, working at the intersection of academia, government, and industry.

Cyber security at large, especially malware analysis, often spans multiple file formats and technical details. Extraordinary effort by the research community in dynamic analysis, sandboxing, kernel hooks, and other technologies delve deep into parsing all these formats to obtain the most accurate system possible. However, the research community often fails to neglect the runtime constraints of real-world solutions. This talk will go over why absconding parsing is good for AI research, better for practical cyber security, and how we’ve pushed forward in this direction.

Mar 7th, 2023
Evaluating Heuristic Defenses in Machine Learning Privacy

Speaker: Matthew Jagielski (Google Research)

Matthew is a research scientist at Google, Cambridge, working in the intersection between machine learning, security, and privacy. He received his PhD from Northeastern University, where he was advised by Cristina Nita-Rotaru and Alina Oprea.

In this talk, I will discuss four recent papers which evaluate potential heuristic defenses for membership inference and training data extraction. Starting with training data extraction, I will talk about the pitfalls in attempting to ``censor'' memorized training data, with some analysis on Github Copilot. Next, I will talk about membership inference defenses. I will discuss what happens after removing vulnerable examples from a training set. Next, I'll talk about how membership inference and extraction appear to have a ``recency bias'': recently seen examples tend to be more vulnerable. Finally, I will discuss the possibility of using model distillation as a defense for membership inference.

Apr 4th, 2023
My personal tour of Machine Learning Security

Speaker: Teddy Furon (Inria centre at Rennes University)

Teddy Furon (Ph.D. Telecom ParisTech (2002), Habilitation (2018), IEEE Senior Member) is Directeur de Recherche at Inria centre at Rennes University, France. His research interests include the security related to multimedia, signal processing, and more recently machine learning. He has worked in industry (Thomson, Technicolor) and academia. He co-founded the company Imatag protecting rights of photo agencies. He has been an Associate Editor for four journals, including IEEE Transactions on Information Forensics and Security. He has been named the French AID chair in Security of Artificial Intelligence.

Adversarial examples is the tip of the "machine learning security” iceberg because its literature is impressively large. The first part of the talk briefly presents our humble contributions on white box, black box, and transferable attacks. Yet, there are many more threats putting machine learning algorithms into danger. The second part mentions some of these scenarios like membership inference attack, fingerprinting, and watermarking of DNN. This leaves the impression that ML security is about studying attacks and defenses for an endless list of scenarios. The conclusion of the talk tries to give a more concise definition of what machine learning security is.

May 16th, 2023
Efficient Malware Analysis Using Metric Embeddings

Speaker: Scott Coull (Google)

Dr. Scott Coull is the Head of Data Science Research at Google Cloud Security, where he leads a team of talented data scientists in applying cutting-edge machine learning techniques to solve complex computer and network security problems. Over the course of his 20-year career, his research interests have spanned a range of topics, including malware classification, censorship circumvention, network traffic analysis, data privacy, and applied cryptography. Most recently, he and his team have begun exploring ways to more tightly couple cybersecurity subject matter expertise with machine learning technologies through the application of weakly supervised learning and large language models. Dr. Coull received his PhD in Computer Science from Johns Hopkins University and completed a NSF/CRA Computing Innovation Fellowship at the University of North Carolina at Chapel Hill. Home page: https://scottcoull.com

When most researchers think of machine learning applied to malware analysis their minds usually turn toward malware detection. However, real-world malware analysis consists of a complex pipeline of classifiers and data analysis – from dataset sampling techniques to classification of capabilities to retrieval of new training samples from user systems. In this talk, we explore the possibility of streamlining this malware analysis pipeline and reducing tech debt through a single, efficient malware embedding that can be leveraged for multiple downstream tasks. Our evaluation on multiple malware classification datasets, consisting of millions of binaries, shows that our efficient embeddings are capable of successfully transferring to tasks across the entire analysis pipeline. Despite this success, the performance on some tasks lags behind that of purpose-built models, indicating an interesting trade-off between maintenance cost and performance.

Jun 6th, 2023
Realistic Neural Networks with Guarantees

Speaker: Mark Niklas Müller (ETH Zurich)

Mark Niklas Müller is a Ph.D. student at the Secure, Reliable, and Intelligent Systems Lab at ETH Zurich and advised by Prof. Martin Vechev. Mark’s research focuses on provable guarantees for machine learning models, including both certified training methods as well as deterministic and probabilistic certification methods for a diverse range of neural architectures.

Following the discovery of adversarial examples, provable robustness guarantees for neural networks have received increasing attention from the research community. While relatively small or heavily regularized models with limited standard accuracy can now be efficiently analyzed, obtaining guarantees for more accurate models remains an open problem. Recently, a new verification paradigm has emerged that tackles this challenge by combining a Branch-and-Bound approach with precise multi-neuron constraints. The resulting, more precise verifiers have in turn enabled novel certified training methods which reduce (over-)regularization to obtain more precise yet certifiable networks. In this talk, we discuss these certification and training methods.

Jul 27th, 2023
Trusted Computing Base of Machine Learning

Speaker: Ilia Shumailov (University of Oxford)

Ilia Shumailov holds a PhD in Computer Science from University of Cambridge, specialising in Machine Learning and Computer Security. During the PhD under the supervision of Prof Ross Anderson, Ilia worked on a number of projects spanning the fields of machine learning security, cybercrime analysis and signal processing. Following the PhD, Ilia joined Vector Institute in Canada as a Postdoctoral Fellow, where he worked under the supervision of Prof Nicolas Papernot and Prof Kassem Fawaz. Ilia is currently a Junior Research Fellow at Christ Church, University of Oxford, and a member of the Oxford Applied and Theoretical Machine Learning Group with Prof Yarin Gal.

Machine learning (ML) has proven to be more fragile than previously thought, especially in adversarial settings. A capable adversary can cause ML systems to break at the training, inference, and deployment stages. While most of the current literature focuses on the security of the machine learning components, real-world vulnerability often comes from the underlying infrastructure. In this talk, I will identify the trusted computing base of modern machine learning and discuss where to look for vulnerabilities in the future.

Dec 7th, 2023
Can individuals trust privacy mechanisms for machine learning? A case study of federated learning

Speaker: Franziska Boenisch (CISPA)

Franziska is a tenure-track faculty at the CISPA Helmholtz Center for Information Security where she co-leads the SprintML lab. Before, she was a Postdoctoral Fellow at the University of Toronto and Vector Institute in Toronto advised by Prof. Nicolas Papernot. Her current research centers around private and trustworthy machine learning with a focus on decentralized applications. Franziska obtained her Ph.D. at the Computer Science Department at Freie University Berlin, where she pioneered the notion of individualized privacy in machine learning. During her Ph.D., Franziska was a research associate at the Fraunhofer Institute for Applied and Integrated Security (AISEC), Germany. She received a Fraunhofer TALENTA grant for outstanding female early career researchers and the German Industrial Research Foundation prize for her research on machine learning privacy.

What is the trusted computing base for privacy? This talk will answer this question from the perspective of individual users. I will first focus on a case study of federated learning (FL). My work shows that vanilla FL currently does not provide meaningful privacy for individual users who cannot trust the central server orchestrating the FL protocol. This is because gradients of the shared model directly leak individual training data points.The resulting leakage can be amplified by a malicious attacker through small, targeted manipulations of the model weights. My work thus shows that the protection that vanilla FL claims to offer is but a thin facade: data may never "leave'' personal devices explicitly but it certainly does so implicitly through gradients. Then, I will show that the leakage is still exploitable for what is considered the most private instantiation of FL: a protocol that combines secure aggregation with differential privacy. This highlights that individuals unable to trust the central server should instead rely on verifiable mechanisms to obtain privacy. I will conclude my talk with an outlook on how such verifiable mechanisms can be designed in the future, as well as how my work generally advances the ability to audit privacy mechanisms.

Mar 26th, 2024
A Deep Dive into the Privacy of Machine Learning

Speaker: Giovanni Cherubin (Microsoft)

Giovanni Cherubin is a Senior Researcher at Microsoft (Cambridge) working with the Microsoft Response Centre (MSRC). Before joining Microsoft, he held research positions at the Alan Turing Institute and EPFL, and he obtained a PhD in Machine Learning and Cyber Security from Royal Holloway University of London. His research focuses on privacy and security properties of machine learning models, and on the theoretical/empirical study of their information leakage. He also works on reliable machine learning tools, such as distribution-free uncertainty estimation for machine learning (e.g., Conformal Prediction). Some of his work on security and machine learning has been recognised with best student paper awards (SLDS15, PETS17), distinguished paper (USENIX22), and with a USENIX Internet Defense Prize (2022).

The hope to train machine learning models whilst ensuring the privacy of their training data is well within reach, but it requires good care. To succeed, one needs to carefully analyse how and where they plan to deploy the model, and decide which threats are worrisome for the particular application (threat modelling). Luckily, >20 years of research in the area can help a lot in this endeavour. This talk gives an introduction to privacy preserving machine learning (PPML). We will look at the basic threats against the private training data of a machine learning model, at what defence mechanisms researchers devised to counter them, and what are the research opportunities for the future.

May 2nd, 2024
On New Security and Safety Challenges Posed by LLMs and How to Evaluate Them

Speaker: Sahar Abdelnabi (Microsoft)

Sahar Abdelnabi is an AI security researcher at Microsoft Security Response Center (Cambridge). Previously, she was a PhD candidate at CISPA Helmholtz Center for Information Security, advised by Prof. Dr. Mario Fritz and she obtained her MSc degree at Saarland University. Her research interests lie in the broad intersection of machine learning with security, safety, and sociopolitical aspects. This includes the following areas: 1) Understanding and mitigating the failure modes of machine learning models, their biases, and their misuse scenarios. 2) How machine learning models could amplify or help counter existing societal and safety problems (e.g., misinformation, biases, stereotypes, cybersecurity risks, etc.). 3) Emergent challenges posed by new foundation and large language models.

Large Language Models (LLMs) are integrated into many widely used and real-world applications and use-case scenarios. With their capabilities and agentic-like adoption, they open new frontiers to assist in various tasks. However, they also bring new security and safety risks. Unlike previous models with static generation, LLMs’ nature of dynamic, multi-turn, and flexible functionality makes them notoriously hard to robustly evaluate and control. This talk will cover some of these new potential risks imposed by LLMs, how to evaluate them, and the challenges of mitigations.

May 27th, 2024
Private Prompt Learning for Large Language Models

Speaker: Adam Dziedzic (CISPA)

Adam is a Tenure Track Faculty Member at CISPA Helmholtz Center for Information Security, co-leading the SprintML group. His research is focused on secure and trustworthy Machine Learning as a Service (MLaaS). Adam designs robust and reliable machine learning methods for training and inference of ML models while preserving data privacy and model confidentiality. Adam was a Postdoctoral Fellow at the Vector Institute and the University of Toronto, and a member of the CleverHans Lab, advised by Prof. Nicolas Papernot. He earned his PhD at the University of Chicago, where he was advised by Prof. Sanjay Krishnan and worked on input and model compression for adaptive and robust neural networks. Adam obtained his Bachelor's and Master's degrees from the Warsaw University of Technology in Poland. He was also studying at DTU (Technical University of Denmark) and carried out research at EPFL, Switzerland. Adam also worked at CERN (Geneva, Switzerland), Barclays Investment Bank in London (UK), Microsoft Research (Redmond, USA), and Google (Madison, USA).

Large language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attack against the data used to prompt LLMs. To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent. However, this comes at the expense of the practicality and efficiency offered by prompting. Therefore, we propose to privately learn to prompt. We first show that soft prompts can be obtained privately through gradient descent on downstream data. However, this is not the case for discrete prompts. Thus, we orchestrate a noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock’s knowledge into a single public prompt. We show that LLMs prompted with our private algorithms closely match the non-private baselines. Paper: NeurIPS2023: https://openreview.net/forum?id=u6Xv3FuF8N

Jun 17th, 2024
Cross-Context Backdoor Attacks against Graph Prompt Learning

Speaker: Yufei Han (INRIA)

Yufei Han is currently a senior researcher at INRIA, PIRAT project team. He is focusing on two topics: 1) adversarial attack and defense of AI techniques and 2) AI-boosted cyber security applications, such as AI-based malware classification. Yufei has over 30 peer-reviewed research publications on top venues and journals of AI and security research, such as ICML, ICLR, AAAI, KDD, ACM CCS, IEEE SP Oakland, Usenix Security and IEEE TDSC. Besides, Yufei also has 15 US patents granted.

Graph Prompt Learning (GPL) bridges significant disparities between pretraining and downstream applications to alleviate the knowledge transfer bottleneck in real-world graph learning. While GPL offers superior effectiveness in graph knowledge transfer and computational efficiency, the security risks posed by backdoor poisoning effects embedded in pretrained models remain largely unexplored. Our study provides a comprehensive analysis of GPL's vulnerability to backdoor attacks. We introduce CrossBA, the first cross-context backdoor attack against GPL, which manipulates only the pretraining phase without requiring knowledge of downstream applications. Our investigation reveals both theoretically and empirically that tuning trigger graphs, combined with prompt transformations, can seamlessly transfer the backdoor threat from pretrained encoders to downstream applications. Through extensive experiments involving 3 representative GPL methods across 5 distinct cross-context scenarios and 5 benchmark datasets of node and graph classification tasks, we demonstrate that CrossBA consistently achieves high attack success rates while preserving the functionality of downstream applications over clean input. We also explore potential countermeasures against CrossBA and conclude that current defenses are insufficient to mitigate CrossBA. Our study highlights the persistent backdoor threats to GPL systems, raising trustworthiness concerns in the practices of GPL techniques.