Total events: 34

• 

  Jun 24th, 2026
  What Should an Agent Remember, Reveal, and Redact? The New Data Sinks and Leaks of Agentic AI
  

  Speaker: Niloofar Mireshghallah (Carnegie Mellon)

  Niloofar Mireshghallah is an assistant professor at Carnegie Mellon University, jointly appointed in the Language Technologies Institute and the Department of Engineering and Public Policy, and a core member of CyLab. Her research lies at the intersection of privacy, natural language processing, and the societal implications of machine learning, with a focus on contextual integrity and information-flow norms in LLMs and agentic systems. Previously, she was a research scientist at Meta's FAIR lab and a postdoctoral scholar at the University of Washington's Paul G. Allen School. She received her Ph.D. from UC San Diego in 2023. Her work has been covered by The Washington Post and WIRED, and she is a recipient of the NCWIT Collegiate Award and the Rising Star in Adversarial ML Award.

  
  GenAI is no longer a chat box. It is agents moving our data through tools, APIs, memories, and modalities. This shift creates new ``data sinks'' where sensitive information quietly accumulates, and new leak paths where it slips across contexts: persistent memories surfacing in the wrong conversation, users oversharing because they don't know what a model actually needs. Through CIMemories [ICLR 2026], we introduce a compositional benchmark for whether LLMs respect contextual integrity when drawing on persistent memory, and find that frontier models leak sensitive attributes at high rates, that violations compound with use, and that privacy-conscious prompting does not fix the underlying instability. Beyond these inference-time risks, the second half of the talk turns to memorization. We study it through a transitive lens: not whether a model reproduces a single sequence it saw, but how co-occurrences and similar-appearing data chain together during training. In Privacy Ripple Effects [ACL 2025], we characterize assisted memorization, where PII that was not extractable earlier becomes extractable later, because data seen downstream acts as a key to earlier sequences. Fine-tuning forges connections that did not exist before, exposing memorized content that no prior probing could surface. In Alignment Whack-a-Mole [preprint 2026], we push this onto commercial models: fine-tuning them to expand plot summaries into prose, an ordinary writing-assistant task, bypasses safety alignment and drives GPT-4o, Gemini-2.5-Pro, and DeepSeek-V3.1 to reproduce held-out copyrighted books from parametric memory, with recall generalizing across authors the model was never tuned on. Taken together, these results reframe emergent misalignment not as a sudden leap in capability but as a memorization phenomenon: fine-tuning surfaces latent structure the model already held. Memorization is a dynamic property of the whole training pipeline, and emergent misalignment is one of its downstream impacts.
  
• 

  Apr 29th, 2026
  The Way Forward: Towards Trustworthy AI Agents
  

  Speaker: Alina Oprea (Northeastern University)

  Alina Oprea is a Professor at Northeastern University in the Khoury College of Computer Sciences, where she co-directs the Network and Distributed Systems Security (NDS2) Lab. She joined Northeastern University in Fall 2016 after spending 9 years as a Research Scientist at RSA Laboratories. She is the recipient of the Technology Review TR35 award for research in cloud security in 2011, the Google Security and Privacy Award in 2019, the Ruth and Joel Spira Award for Excellence in Teaching in 2020, and the CMU Cylab Distinguished Alumni Award in 2024. Her research interests are trustworthy AI, security of agentic AI systems, AI for cybersecurity, machine learning privacy, and cloud security. She is currently the General Chair for the IEEE Symposium on Security and Privacy 2026.

  
  AI agentic systems have emerged as autonomous software powered by large language models, increasingly deployed in critical domains. This talk offers a perspective on how adversarial machine learning research has evolved over the past decade and examines the novel threats that agentic systems introduce. We will discuss two recent research projects: building trusted infrastructure for AI agents, and mitigating risks arising from agent interaction with untrusted tools and data. The talk will conclude with a discussion of open problems in trustworthy AI agents and directions for future research.
  
• 

  Mar 11th, 2026
  Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
  

  Speaker: Mantas Mazeika (Center for AI Safety)

  Mantas Mazeika is a Research Scientist at the Center for AI Safety. His recent work focuses on AI alignment, tamper-resistance for open-weight models, and AI safety benchmarks. He received his Ph.D. in Computer Science from the University of Illinois Urbana-Champaign in 2024.

  
  Large language models don’t just get smarter as they scale—they also begin to acquire surprisingly coherent value systems. In this talk, I’ll introduce utility engineering, a new research agenda for analyzing and controlling those emergent values. Using the framework of utility functions, we identify structural patterns in LLM values, such as expected utility maximization, as well as concerning tendencies. I will outline our experimental findings and discuss why utilities can be a useful tool for further work on AI alignment.
  
• 

  Feb 4th, 2026
  Improving Capabilities of Efficient Foundation Models beyond Language
  

  Speaker: Antonio Orvieto (ELLIS Institute Tübingen - MPI)

  Antonio studied Control Engineering in Italy and Switzerland. He holds a PhD in Computer Science from ETH Zürich and spent time at Google Deepmind (UK), Meta (US), MILA (CA), INRIA (FR), and HILTI (LI). He is currently a Hector Endowed Fellow and Principal Investigator (PI) at the ELLIS Institute Tübingen and Independent Group Leader of the MPI for Intelligent Systems, where he leads the Deep Models and Optimization group. He received the ETH medal for outstanding doctoral theses and the Schmidt Sciences AI2050 Early Career Fellowship. In his research, Antonio strives to improve the efficiency of deep learning technologies by pioneering new architectures and training techniques grounded in theoretical knowledge. His work encompasses two main areas, i.e., understanding the intricacies of large-scale optimization dynamics and designing innovative architectures and powerful optimizers capable of handling complex data.

  
  In this talk, we discuss recent alternatives to transformers based on linear RNNs and linear attention. While these new models (Mamba, Hyena, DeltaNet, RWKV) offer improved throughput and extremely long context processing abilities (e.g., all human genome), they often suffer from poor trainability and compute/expressivity tradeoffs. After providing an introduction to these architectures, we present central results on universality and complexity in the literature (e.g., https://arxiv.org/abs/2402.19047). Building on the presented insights and motivated by the need to reason beyond next-token prediction, we discuss new architectures, including Fixed-Point Mamba (spotlight at NeurIPS this year; https://arxiv.org/abs/2503.10799), that can automatically enhance expressivity through an adaptive compute mechanism.
  
• 

  Jan 14th, 2026
  ASIDE: Architectural Separation of Instructions and Data in Language Models
  

  Speaker: Egor Zverev (ISTA)

  Egor is a 3rd year PhD student at ISTA, Austria, working under the supervision of Christoph Lampert. He is also an ELLIS PhD Student co-supervised by Florian Tramer. He is broadly interested in AI Safety and Security, with a particular focus on improving LLM Security through architectures. In his previous work he has formalized the problem of instruction-data separation (i.e., what it means for the model to separate executable instructions from non-executable data) and proposed a method to increase such separation through architectural changes. Before coming to ISTA, he got his B.S.(Hons) in Applied Math and CS from the Yandex Department of Data Analysis at the Moscow Institute of Physics and Technology, where he also taught python and stats.

  
  Despite their remarkable performance, large language models lack elementary safety features, making them susceptible to numerous malicious attacks. In particular, previous work has identified the absence of an intrinsic separation between instructions and data as a root cause of the success of prompt injection attacks. In this work, we propose a new architectural element, ASIDE, that allows language models to clearly separate instructions and data at the level of embeddings. ASIDE applies an orthogonal rotation to the embeddings of data tokens, thus creating clearly distinct representations of instructions and data tokens without introducing any additional parameters. As we demonstrate experimentally across a range of models, instruction-tuning LLMs with ASIDE (1) leads to highly increased instruction-data separation without a loss in model utility and (2) makes the models more robust to prompt injection benchmarks, even without dedicated safety training. Additionally, we provide insights into the mechanism underlying our method through an analysis of the model representations.
  
• 

  Dec 17th, 2025
  A mechanistic view of refusal in language models
  

  Speaker: Andy Arditi (Northeastern University)

  Andy is a PhD student at Northeastern University advised by David Bau, where he researches mechanistic interpretability and the internal structure of large language models. His recent work focuses on applying mechanistic interpretability to practical problems, such as understanding model refusals, detecting hallucinations, and uncovering linear representations of persona in chat assistants. Before his PhD, he earned bachelor’s and master’s degrees from Columbia University and worked as a software engineer at Microsoft."

  
  Refusal is currently the primary mechanism by which LLM developers prevent misuse: models are trained to decline harmful or inappropriate requests. We study this critical behavior through a mechanistic lens, asking how refusal is implemented internally, and we show that it is largely mediated by a single direction in activation space. This simple observation enables several applications, including “weight orthogonalization,” a cheap and effective method for disabling refusal guardrails in open-source models. It has also proven to be useful for implementing more efficient red teaming and adversarial training pipelines. In this talk, I will present these findings, discuss follow-up work by other groups, and outline the greater implications for open-source model development.
  
• 

  Nov 26th, 2025
  Explanation-Aware Attacks and the Limits of using XAI in Computer Security
  

  Speaker: Christian Wressnegger (KIT)

  Christian Wressnegger is a professor of computer science at the Karlsruhe Institute of Technology (KIT), heading the chair of Artificial Intelligence & Security. Additionally, he serves as the spokesperson for the KIT Graduate School Cyber Security, is co-director of the KASTEL Security Research Labs, one of three competence centers for cybersecurity in Germany, and has been appointed Junior Fellow of the Berlin Institute for the Foundations of Learning and Data (BIFOLD). He holds a Ph.D. from TU Braunschweig and has graduated from Graz University of Technology, where he majored in computer science. His research revolves around combining machine learning (or AI in the broader scope) and computer security. He develop methods in the area of system security but also research the robustness of machine learning against attacks striving for secure AI. He is particularly visible for his work on the security of XAI.

  
  Learning-based systems effectively assist in various computer security challenges, such as preventing network intrusions, reverse engineering, vulnerability discovery, or detecting malware. However, modern (deep) learning methods often lack understandable reasoning in their decision process, making crucial decisions less trustworthy. Recent advances in Explainable AI (XAI) have turned the tables, enabling precise relevance attribution of input features for otherwise opaque models. This progression has raised expectations that these techniques can also benefit defense against attacks on computer systems and even machine learning models themselves. This talk disucsses explanation-aware attacks against neural networks and explores limits of XAI in computer security, demonstrating where it can and cannot (yet) be used reliably.
  
• 

  Oct 27th, 2025
  Learning Safety Constraints for Large Language Models
  

  Speaker: Xin Chen (ETH)

  Xin Chen, Cynthia is a PhD student at ETH Zurich, supervised by Profs. Andreas Krause and Florian Tramer. Her research focuses on Large Language Model safety and alignment, combining principled methods with empirical findings to make LLM safety mechanisms more trustworthy. Cynthia is a fellow at the Open Philanthropy AI Fellowship and the Vitalik Buterin PhD Fellowship.

  
  Large language models (LLMs) have emerged as powerful tools but pose significant safety risks through harmful outputs and vulnerability to adversarial attacks. We propose SaP, short for Safety Polytope, a geometric approach to LLM safety that learns and enforces multiple safety constraints directly in the model's representation space. We develop a framework that identifies safe and unsafe regions via the polytope's facets, enabling both detection and correction of unsafe outputs through geometric steering. Unlike existing approaches that modify model weights, SaP operates post-hoc in the representation space, preserving model capabilities while enforcing safety constraints. Experiments across multiple LLMs demonstrate that our method can effectively detect unethical inputs, reduce adversarial attack success rates while maintaining performance on standard tasks, thus highlighting the importance of having an explicit geometric model for safety. Analysis of the learned polytope facets reveals emergence of specialization in detecting different semantic notions of safety, providing interpretable insights into how safety is captured in LLMs' representation space.
  
• 

  Oct 9th, 2025
  On Robust Conformal Prediction
  

  Speaker: Soroush H. Zargarbashi (CISPA)

  Soroush is a research assistant (PhD candidate) focusing on Trustworthy AI — robustness and uncertainty quantification. He works at CISPA Helmholtz Center for information security, as a PhD researcher. Previously he worked as intern at Apple (machine translation team).

  
  The softmax outputs are not a trustworthy indicator of uncertainty; models return many confident but wrong predictions. Conformal prediction offers prediction sets with coverage guarantee. Despite its versatility, it is fragile under adversarial and natural noises. We develop a progression of techniques that bridge this robustness gap while preserving efficiency and practicality.
  
• 

  Jun 17th, 2024
  Cross-Context Backdoor Attacks against Graph Prompt Learning
  

  Speaker: Yufei Han (INRIA)

  Yufei Han is currently a senior researcher at INRIA, PIRAT project team. He is focusing on two topics: 1) adversarial attack and defense of AI techniques and 2) AI-boosted cyber security applications, such as AI-based malware classification. Yufei has over 30 peer-reviewed research publications on top venues and journals of AI and security research, such as ICML, ICLR, AAAI, KDD, ACM CCS, IEEE SP Oakland, Usenix Security and IEEE TDSC. Besides, Yufei also has 15 US patents granted.

  
  Graph Prompt Learning (GPL) bridges significant disparities between pretraining and downstream applications to alleviate the knowledge transfer bottleneck in real-world graph learning. While GPL offers superior effectiveness in graph knowledge transfer and computational efficiency, the security risks posed by backdoor poisoning effects embedded in pretrained models remain largely unexplored. Our study provides a comprehensive analysis of GPL's vulnerability to backdoor attacks. We introduce CrossBA, the first cross-context backdoor attack against GPL, which manipulates only the pretraining phase without requiring knowledge of downstream applications. Our investigation reveals both theoretically and empirically that tuning trigger graphs, combined with prompt transformations, can seamlessly transfer the backdoor threat from pretrained encoders to downstream applications. Through extensive experiments involving 3 representative GPL methods across 5 distinct cross-context scenarios and 5 benchmark datasets of node and graph classification tasks, we demonstrate that CrossBA consistently achieves high attack success rates while preserving the functionality of downstream applications over clean input. We also explore potential countermeasures against CrossBA and conclude that current defenses are insufficient to mitigate CrossBA. Our study highlights the persistent backdoor threats to GPL systems, raising trustworthiness concerns in the practices of GPL techniques.
  
• 

  May 27th, 2024
  Private Prompt Learning for Large Language Models
  

  Speaker: Adam Dziedzic (CISPA)

  Adam is a Tenure Track Faculty Member at CISPA Helmholtz Center for Information Security, co-leading the SprintML group. His research is focused on secure and trustworthy Machine Learning as a Service (MLaaS). Adam designs robust and reliable machine learning methods for training and inference of ML models while preserving data privacy and model confidentiality. Adam was a Postdoctoral Fellow at the Vector Institute and the University of Toronto, and a member of the CleverHans Lab, advised by Prof. Nicolas Papernot. He earned his PhD at the University of Chicago, where he was advised by Prof. Sanjay Krishnan and worked on input and model compression for adaptive and robust neural networks. Adam obtained his Bachelor's and Master's degrees from the Warsaw University of Technology in Poland. He was also studying at DTU (Technical University of Denmark) and carried out research at EPFL, Switzerland. Adam also worked at CERN (Geneva, Switzerland), Barclays Investment Bank in London (UK), Microsoft Research (Redmond, USA), and Google (Madison, USA).

  
  Large language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attack against the data used to prompt LLMs. To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent. However, this comes at the expense of the practicality and efficiency offered by prompting. Therefore, we propose to privately learn to prompt. We first show that soft prompts can be obtained privately through gradient descent on downstream data. However, this is not the case for discrete prompts. Thus, we orchestrate a noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock’s knowledge into a single public prompt. We show that LLMs prompted with our private algorithms closely match the non-private baselines. Paper: NeurIPS2023: https://openreview.net/forum?id=u6Xv3FuF8N
  
• 

  May 2nd, 2024
  On New Security and Safety Challenges Posed by LLMs and How to Evaluate Them
  

  Speaker: Sahar Abdelnabi (Microsoft)

  Sahar Abdelnabi is an AI security researcher at Microsoft Security Response Center (Cambridge). Previously, she was a PhD candidate at CISPA Helmholtz Center for Information Security, advised by Prof. Dr. Mario Fritz and she obtained her MSc degree at Saarland University. Her research interests lie in the broad intersection of machine learning with security, safety, and sociopolitical aspects. This includes the following areas: 1) Understanding and mitigating the failure modes of machine learning models, their biases, and their misuse scenarios. 2) How machine learning models could amplify or help counter existing societal and safety problems (e.g., misinformation, biases, stereotypes, cybersecurity risks, etc.). 3) Emergent challenges posed by new foundation and large language models.

  
  Large Language Models (LLMs) are integrated into many widely used and real-world applications and use-case scenarios. With their capabilities and agentic-like adoption, they open new frontiers to assist in various tasks. However, they also bring new security and safety risks. Unlike previous models with static generation, LLMs’ nature of dynamic, multi-turn, and flexible functionality makes them notoriously hard to robustly evaluate and control. This talk will cover some of these new potential risks imposed by LLMs, how to evaluate them, and the challenges of mitigations.
  
• 

  Mar 26th, 2024
  A Deep Dive into the Privacy of Machine Learning
  

  Speaker: Giovanni Cherubin (Microsoft)

  Giovanni Cherubin is a Senior Researcher at Microsoft (Cambridge) working with the Microsoft Response Centre (MSRC). Before joining Microsoft, he held research positions at the Alan Turing Institute and EPFL, and he obtained a PhD in Machine Learning and Cyber Security from Royal Holloway University of London. His research focuses on privacy and security properties of machine learning models, and on the theoretical/empirical study of their information leakage. He also works on reliable machine learning tools, such as distribution-free uncertainty estimation for machine learning (e.g., Conformal Prediction). Some of his work on security and machine learning has been recognised with best student paper awards (SLDS15, PETS17), distinguished paper (USENIX22), and with a USENIX Internet Defense Prize (2022).

  
  The hope to train machine learning models whilst ensuring the privacy of their training data is well within reach, but it requires good care. To succeed, one needs to carefully analyse how and where they plan to deploy the model, and decide which threats are worrisome for the particular application (threat modelling). Luckily, >20 years of research in the area can help a lot in this endeavour. This talk gives an introduction to privacy preserving machine learning (PPML). We will look at the basic threats against the private training data of a machine learning model, at what defence mechanisms researchers devised to counter them, and what are the research opportunities for the future.
  
• 

  Dec 7th, 2023
  Can individuals trust privacy mechanisms for machine learning? A case study of federated learning
  

  Speaker: Franziska Boenisch (CISPA)

  Franziska is a tenure-track faculty at the CISPA Helmholtz Center for Information Security where she co-leads the SprintML lab. Before, she was a Postdoctoral Fellow at the University of Toronto and Vector Institute in Toronto advised by Prof. Nicolas Papernot. Her current research centers around private and trustworthy machine learning with a focus on decentralized applications. Franziska obtained her Ph.D. at the Computer Science Department at Freie University Berlin, where she pioneered the notion of individualized privacy in machine learning. During her Ph.D., Franziska was a research associate at the Fraunhofer Institute for Applied and Integrated Security (AISEC), Germany. She received a Fraunhofer TALENTA grant for outstanding female early career researchers and the German Industrial Research Foundation prize for her research on machine learning privacy.

  
  What is the trusted computing base for privacy? This talk will answer this question from the perspective of individual users. I will first focus on a case study of federated learning (FL). My work shows that vanilla FL currently does not provide meaningful privacy for individual users who cannot trust the central server orchestrating the FL protocol. This is because gradients of the shared model directly leak individual training data points.The resulting leakage can be amplified by a malicious attacker through small, targeted manipulations of the model weights. My work thus shows that the protection that vanilla FL claims to offer is but a thin facade: data may never "leave'' personal devices explicitly but it certainly does so implicitly through gradients. Then, I will show that the leakage is still exploitable for what is considered the most private instantiation of FL: a protocol that combines secure aggregation with differential privacy. This highlights that individuals unable to trust the central server should instead rely on verifiable mechanisms to obtain privacy. I will conclude my talk with an outlook on how such verifiable mechanisms can be designed in the future, as well as how my work generally advances the ability to audit privacy mechanisms.
  
• 

  Jul 27th, 2023
  Trusted Computing Base of Machine Learning
  

  Speaker: Ilia Shumailov (University of Oxford)

  Ilia Shumailov holds a PhD in Computer Science from University of Cambridge, specialising in Machine Learning and Computer Security. During the PhD under the supervision of Prof Ross Anderson, Ilia worked on a number of projects spanning the fields of machine learning security, cybercrime analysis and signal processing. Following the PhD, Ilia joined Vector Institute in Canada as a Postdoctoral Fellow, where he worked under the supervision of Prof Nicolas Papernot and Prof Kassem Fawaz. Ilia is currently a Junior Research Fellow at Christ Church, University of Oxford, and a member of the Oxford Applied and Theoretical Machine Learning Group with Prof Yarin Gal.

  
  Machine learning (ML) has proven to be more fragile than previously thought, especially in adversarial settings. A capable adversary can cause ML systems to break at the training, inference, and deployment stages. While most of the current literature focuses on the security of the machine learning components, real-world vulnerability often comes from the underlying infrastructure. In this talk, I will identify the trusted computing base of modern machine learning and discuss where to look for vulnerabilities in the future.
  
• 

  Jun 6th, 2023
  Realistic Neural Networks with Guarantees
  

  Speaker: Mark Niklas Müller (ETH Zurich)

  Mark Niklas Müller is a Ph.D. student at the Secure, Reliable, and Intelligent Systems Lab at ETH Zurich and advised by Prof. Martin Vechev. Mark’s research focuses on provable guarantees for machine learning models, including both certified training methods as well as deterministic and probabilistic certification methods for a diverse range of neural architectures.

  
  Following the discovery of adversarial examples, provable robustness guarantees for neural networks have received increasing attention from the research community. While relatively small or heavily regularized models with limited standard accuracy can now be efficiently analyzed, obtaining guarantees for more accurate models remains an open problem. Recently, a new verification paradigm has emerged that tackles this challenge by combining a Branch-and-Bound approach with precise multi-neuron constraints. The resulting, more precise verifiers have in turn enabled novel certified training methods which reduce (over-)regularization to obtain more precise yet certifiable networks. In this talk, we discuss these certification and training methods.
  
• 

  May 16th, 2023
  Efficient Malware Analysis Using Metric Embeddings
  

  Speaker: Scott Coull (Google)

  Dr. Scott Coull is the Head of Data Science Research at Google Cloud Security, where he leads a team of talented data scientists in applying cutting-edge machine learning techniques to solve complex computer and network security problems. Over the course of his 20-year career, his research interests have spanned a range of topics, including malware classification, censorship circumvention, network traffic analysis, data privacy, and applied cryptography. Most recently, he and his team have begun exploring ways to more tightly couple cybersecurity subject matter expertise with machine learning technologies through the application of weakly supervised learning and large language models. Dr. Coull received his PhD in Computer Science from Johns Hopkins University and completed a NSF/CRA Computing Innovation Fellowship at the University of North Carolina at Chapel Hill.

  
  When most researchers think of machine learning applied to malware analysis their minds usually turn toward malware detection. However, real-world malware analysis consists of a complex pipeline of classifiers and data analysis – from dataset sampling techniques to classification of capabilities to retrieval of new training samples from user systems. In this talk, we explore the possibility of streamlining this malware analysis pipeline and reducing tech debt through a single, efficient malware embedding that can be leveraged for multiple downstream tasks. Our evaluation on multiple malware classification datasets, consisting of millions of binaries, shows that our efficient embeddings are capable of successfully transferring to tasks across the entire analysis pipeline. Despite this success, the performance on some tasks lags behind that of purpose-built models, indicating an interesting trade-off between maintenance cost and performance.
  
• 

  Apr 4th, 2023
  My personal tour of Machine Learning Security
  

  Speaker: Teddy Furon (Inria centre at Rennes University)

  Teddy Furon (Ph.D. Telecom ParisTech (2002), Habilitation (2018), IEEE Senior Member) is Directeur de Recherche at Inria centre at Rennes University, France. His research interests include the security related to multimedia, signal processing, and more recently machine learning. He has worked in industry (Thomson, Technicolor) and academia. He co-founded the company Imatag protecting rights of photo agencies. He has been an Associate Editor for four journals, including IEEE Transactions on Information Forensics and Security. He has been named the French AID chair in Security of Artificial Intelligence.

  
  Adversarial examples is the tip of the "machine learning security” iceberg because its literature is impressively large. The first part of the talk briefly presents our humble contributions on white box, black box, and transferable attacks. Yet, there are many more threats putting machine learning algorithms into danger. The second part mentions some of these scenarios like membership inference attack, fingerprinting, and watermarking of DNN. This leaves the impression that ML security is about studying attacks and defenses for an endless list of scenarios. The conclusion of the talk tries to give a more concise definition of what machine learning security is.
  
• 

  Mar 7th, 2023
  Evaluating Heuristic Defenses in Machine Learning Privacy
  

  Speaker: Matthew Jagielski (Google Research)

  Matthew is a research scientist at Google, Cambridge, working in the intersection between machine learning, security, and privacy. He received his PhD from Northeastern University, where he was advised by Cristina Nita-Rotaru and Alina Oprea.

  
  In this talk, I will discuss four recent papers which evaluate potential heuristic defenses for membership inference and training data extraction. Starting with training data extraction, I will talk about the pitfalls in attempting to censor memorized training data, with some analysis on Github Copilot. Next, I will talk about membership inference defenses. I will discuss what happens after removing vulnerable examples from a training set. Next, I'll talk about how membership inference and extraction appear to have a recency bias: recently seen examples tend to be more vulnerable. Finally, I will discuss the possibility of using model distillation as a defense for membership inference.
  
• 

  Feb 16th, 2023
  Why I Hate Parsers and You Should Too
  

  Speaker: Edward Raff (Booz Allen Hamilton)

  Dr. Edward Raff is a Chief Scientist at Booz Allen Hamilton, Visiting professor at the University of Maryland, Baltimore County, and previous chair of the Conference on Applied Machine Learning and Information Systems. His research covers broad areas of basic and applied machine learning including reproducibility, adversarial attacks/defense, high performance computing, and all of these aspects merge at the intersection of machine learning for malware detection/analysis. Dr. Raff’s work has won five best paper awards, working at the intersection of academia, government, and industry.

  
  Cyber security at large, especially malware analysis, often spans multiple file formats and technical details. Extraordinary effort by the research community in dynamic analysis, sandboxing, kernel hooks, and other technologies delve deep into parsing all these formats to obtain the most accurate system possible. However, the research community often fails to neglect the runtime constraints of real-world solutions. This talk will go over why absconding parsing is good for AI research, better for practical cyber security, and how we’ve pushed forward in this direction.
  
• 

  Jan 12th, 2023
  Online Certifiable Robustness for Adversarial Perturbation
  

  Speaker: Fabio Brau (Scuola Superiore Sant'Anna Pisa)

  Fabio Brau is a Ph.D. student at the Department of Excellence in Robotics and AI at the Scuola Superiore SantAnna of Pisa, where he works in the ReTiS Laboratory. He obtained his Master's degree in Mathematics from the University of Pisa, where he specialized in Numerical Analysis. Currently, he is studying the robustness of deep neural networks against adversarial examples, both from a practical and a theoretical perspective, leveraging geometrical results on the properties of classification models' decision boundaries.

  
  The discovery of adversarial examples has led the scientific community to explore various research directions. Verification methods, which involve calculating the distance of a sample from the decision boundary, require significant effort and computational resources. Another approach is to develop robust models, which are less sensitive to adversarial perturbations. Recently, this has been —partially— achieved by using orthogonal layers to create Lipschitz-bounded neural networks. In this seminar, we will cover these topics by presenting a fast method for approximating the boundary distance, as well as a "signed distance classifier" model that directly outputs the distance from the boundary instead of a probability score, thus providing a one-shot verification by design.
  
• 

  Dec 13th, 2022
  On the Real-World Adversarial Robustness Against Physically-Realizable Attacks
  

  Speaker: Giulio Rossolini (Scuola Superiore Sant'Anna of Pisa)

  Giulio Rossolini is a Ph.D. student at the Department of Excellence in Robotics & AI and the Real-Time Systems Laboratory (ReTiS Lab) of the Scuola Superiore Sant’Anna of Pisa. His research topics include the design of robust tools and architectures to enhance the trustworthiness of deep learning models in computer vision applications and safety-critical systems.

  
  In recent years, adversarial perturbations have become a hot topic in the safe and secure AI community. However, the concrete feasibility of such attacks on critical systems is often questioned, as it is necessary to exploit the digital representation of the input. This fact has inspired novel approaches for injecting adversarial features as physical objects or patches. This seminar will provide an overview of the most successful strategies for crafting physically-realizable attacks, and examine their transferability among different real-world scenarios and computer vision architectures. Then, the presentation will address empirical and certifiable studies to improve the robustness of deep learning models against these threats.
  
• 

  Nov 8th, 2022
  Practical Adversarial ML
  

  Speaker: Andrei Lapanik (Intuition Machines)

  ML engineer with computational physics background, 20 years in software development

  
  Most of black box adversarial attacks in computer vision are developed and tested on small datasests without being applied to real world cases when target model is unknown but result of attack should be clearly measured. In this talk the successful cases of applying black box attacks against captcha-solving bots that were confirmed by activity monitoring will be presented.
  
• 

  Oct 13th, 2022
  Specification-driven Machine Learning for Robustness
  

  Speaker: Sven Gowal (DeepMind)

  Sven Gowal is a Staff Research Engineer at DeepMind, UK. He led numerous initiatives on "robust and certifiable machine learning" at DeepMind and has co-authored over 30 papers in the domain of Robust ML receiving 2 best paper awards. Prior to DeepMind, he worked for Google Research, where he focused on video content analysis and real-time object detection. He completed his PhD at the Swiss Federal Institute of Technology (EPFL), Switzerland, in 2013, on the topic of decentralized multi-robot control. He received his MSc in 2007 from EPFL after working on the DARPA Urban Challenge with Caltech and having spent part of his undergrad at Carnegie Mellon University.

  
  Enabling models to generalize robustly to adversarial and natural distribution shifts is a fundamental problem in machine learning. In this talk, I introduce the concept of specification-driven machine learning as a solution to this problem. I explain how specifications can be embedded into neural networks and how they can be learned from data. The talk consists of three parts. First, we focus on robustness against lp-norm bounded adversarial perturbations. We introduce the concept of adversarial training, enumerate its key challenges, and demonstrate how we can leverage generative models to bypass these challenges. We then discuss approaches that go beyond adversarial examples. Using disentangled representations, we show how we can leverage classical notions of adversarial or certified training to produce models that are robust to natural, semantically-meaningful perturbations. Finally, we demonstrate how to exploit large-scale generative models trained on large amount of data in this context.
  
• 

  Sep 6th, 2022
  ARIA: Adversarially Robust Image Attribution for Content Provenance
  

  Speaker: Maksym Andriushchenko (EPFL)

  Maksym Andriushchenko is a third-year PhD student in computer science at EPFL (École Polytechnique Fédérale de Lausanne) in Switzerland. He obtained his MSc from Saarland University, Germany. His research mainly focuses on how to make machine learning algorithms adversarially robust and improve their reliability. Maksym has published eleven papers at major machine learning and computer vision conferences (NeurIPS, ICML, ICLR, AISTATS, UAI, AAAI, CVPR, and ECCV).

  
  Image attribution -- matching an image back to a trusted source -- is an emerging tool in the fight against online misinformation. Deep visual fingerprinting models have recently been explored for this purpose. However, they are not robust to tiny input perturbations known as adversarial examples. First we illustrate how to generate valid adversarial images that can easily cause incorrect image attribution. Then we describe an approach to prevent imperceptible adversarial attacks on deep visual fingerprinting models, via robust contrastive learning. The proposed training procedure leverages training on ℓ∞-bounded adversarial examples, it is conceptually simple and incurs only a small computational overhead. The resulting models are substantially more robust, are accurate even on unperturbed images, and perform well even over a database with millions of images. In particular, we achieve 91.6% standard and 85.1% adversarial recall under ℓ∞-bounded perturbations on manipulated images compared to 80.1% and 0.0% from prior work. We also show that robustness generalizes to other types of imperceptible perturbations unseen during training. Finally, we show how to train an adversarially robust image comparator model for detecting editorial changes in matched images.
  
• 

  Jul 7th, 2022
  Increasing Confidence in Adversarial Examples Defenses
  

  Speaker: Florian Tramer (Google, ETHZ)

  Florian Tramèr is a visiting researcher at Google Brain and an assistant professor of computer science at ETH Zurich. His research interests lie in Computer Security, Cryptography and Machine Learning security. In his current work, he studies the worst-case behavior of Deep Learning systems from an adversarial perspective, to understand and mitigate long-term threats to the safety and privacy of users.

  
  Building a defense against adversarial examples is easy. Building one that works is the hard part! And thus there are countless defenses proposed that ultimately fail to hold up their claims. In this talk, I'll discuss some recent research that aims to help vet our confidence in a newly proposed defense, to quickly uncover highly dubious robustness claims and evaluation flaws.
  
• 

  Jun 7th, 2022
  Learning Security Classifiers with Verified Global Robustness Properties
  

  Speaker: Yizheng Chen (University of California, Berkeley)

  Yizheng Chen is a postdoctoral scholar at University of California, Berkeley. She will join University of Maryland as an Assistant Professor of Computer Science in January, 2023. Previously, she was a postdoctoral scholar at Columbia University. She holds a Ph.D. in Computer Science from Georgia Institute of Technology. Her research focuses on building robust machine learning algorithms for security applications. Her work has received an ACM CCS Best Paper Award Runner-up, a Google ASPIRE Award and an Amazon Research Award. She is a recipient of the Anita Borg Memorial Scholarship.

  
  Many recent works have proposed methods to train classifiers with local robustness properties, which can provably eliminate classes of evasion attacks for most inputs, but not all inputs. Since data distribution shift is very common in security applications, e.g., often observed for malware detection, local robustness cannot guarantee that the property holds for unseen inputs at the time of deploying the classifier. Therefore, it is more desirable to enforce global robustness properties that hold for all inputs, which is strictly stronger than local robustness. In this talk, I will discuss methods to train security classifiers with global robustness properties. I will show how to use security domain knowledge and economic cost measurement studies to formulate global robustness properties to capture general classes of evasion strategies that are inexpensive for attackers. Then, I will describe a new algorithm to train security classifiers to satisfy these properties. I will show how to apply the method to detect fake accounts, Twitter spam URLs, and Cryptojacking, and demonstrate that it is not only sound but also practical. We show that we can train classifiers to satisfy different global robustness properties for three security datasets, and even multiple properties at the same time, with modest impact on the classifier’s performance.
  
• 

  May 10th, 2022
  Adversarial Preprocessing: Image-Scaling Attacks in Machine Learning
  

  Speaker: Konrad Rieck (TU Braunschweig)

  Konrad Rieck is a professor at TU Braunschweig and leads the Institute of System Security. Previously, he worked at the University of Göttingen, TU Berlin and Fraunhofer FIRST. His research focus is the intersection of machine learning and computer security. With his group, he develops intelligent methods for detecting attacks, analyzing malicious code and discovering security vulnerabilities. He received the CAST/GI Doctoral Award, a Google Faculty Research Award and recently an ERC Consolidator Grant.

  
  The success of machine learning has been overshadowed by different attacks that thwart its correct operation. While prior work has mainly focused on attacking learning algorithms, another weak spot in learning-based systems has been overlooked: data preprocessing. In this talk, I discuss a recent class of attacks against image scaling. These attacks are agnostic to learning algorithms and affect the preprocessing of all vision systems that use vulnerable scaling implementations, such as TensorFlow, OpenCV, and Pillow. Based on a root-cause analysis of the vulnerabilities, I present novel defenses that effectively block image-scaling attacks in practice and can be easily added to existing systems.
  
• 

  Apr 7th, 2022
  Trends and Challenges in ML-based Malware Detection
  

  Speaker: Fabio Pierazzi (King's College London)

  Dr Fabio Pierazzi is a Lecturer (Assistant Professor) in Cybersecurity at the Department of Informatics of King's College London, where he is also a member of the Cybersecurity (CYS) group and affiliated with UCL's Systems Security Research Lab (S2Lab). His research interests are at the intersection of systems security and machine learning, with a particular emphasis on settings in which attackers adapt quickly to new defenses (i.e., high non-stationarity, adaptive attackers). Previously, he obtained his Ph.D. in Computer Science at University of Modena, Italy (2014–2017), he visited University of Maryland, College Park, USA (2016), and he was a Post-Doctoral Research Associate at Royal Holloway, University of London (2017–2019). Home page: https://fabio.pierazzi.com

  
  Machine Learning (ML) has become a standard tool for malware research in the academic security community: it has been used in a wide range of domains including Windows, PDF and Android malware, but also of malicious JavaScript and URLs. In this seminar, I will present an overview of the most successful approaches for Android malware detection, focusing on intuitions on how program analysis techniques can be useful in this context. Then, I will discuss the main challenges to overcome for widespread of ML-based approaches in industry contexts, especially with respect to concept drift, and adversarial robustness to smart adversaries in the problem-space.
  
• 

  Mar 8th, 2022
  Towards standardized and accurate evaluation of the robustness of image classifiers against adversarial attacks
  

  Speaker: Francesco Croce (University of Tübingen)

  Francesco Croce is a Ph.D. student in the Machine Learning group at the University of Tübingen, Germany. He received his BS in Mathematics for Finance and Insurance and his MS in Mathematics from the University of Torino, Italy. His research focuses on adversarial attacks in different threat models and provable robustness.

  
  It is well known that image classifiers are vulnerable to adversarial perturbations, and many defenses have been suggested to mitigate this phenomenon. However, testing the effectiveness of a defense is not straightforward. We propose a protocol for standardized and accurate evaluation of a large class of adversarial defenses, which allows to benchmark and track the progress of adversarial robustness in several threat models. Finally, we discuss the current limitations of standardized evaluations, and in which cases adaptive attacks might still be necessary.
  
• 

  Feb 22nd, 2022
  Learning From Data is about Handling Risk
  

  Speaker: Luca Oneto (University of Genoa)

  Luca Oneto was born in Rapallo, Italy in 1986. He received his BSc and MSc in Electronic Engineering at the University of Genoa, Italy respectively in 2008 and 2010. In 2014 he received his Ph.D. from the same university in the School of Sciences and Technologies for Knowledge and Information Retrieval with the thesis "Learning Based On Empirical Data". In 2017 he obtained the Italian National Scientific Qualification for the role of Associate Professor in Computer Engineering and in 2018 he obtained the one in Computer Science. He worked as Assistant Professor in Computer Engineering at the University of Genoa from 2016 to 2019. In 2018 he was co-founder of the spin-off ZenaByte s.r.l. In 2019 he obtained the Italian National Scientific Qualification for the role of Full Professor in Computer Science and Computer Engineering. In 2019 he became Associate Professor in Computer Science at the University of Pisa and currently is Associate Professor in Computer Engineering at the University of Genoa. He has been involved in several H2020 projects (S2RJU, ICT, DS) and he has been awarded the Amazon AWS Machine Learning and Somalvico (best Italian young AI researcher) Awards. His first main topic of research is the Statistical Learning Theory with a particular focus on the theoretical aspects of the problems of (Semi) Supervised Model Selection and Error Estimation. His second main topic of research is Data Science with particular reference to the problem of Trustworthy AI and the solution of real-world problems by exploiting and improving the most recent Learning Algorithms and Theoretical Results in the fields of Machine Learning and Data Mining.

  
  This talk focuses on creating a simple map of some fundamental concepts behind the problem of Learning From Data. This map should help practitioners and researchers understand and distill the basic ideas behind some of the current theoretical and practical works coming from the Machine Learning field. I will start from the concept of inference describing the fundamental difference between the three possible inference approaches and their relations with the problem of learning from data. Then I will introduce the notion of risk associated with this problem and describe the risk sources showing how to handle them both from a theoretical and a practical perspective. Finally, I will show the connections of these ideas with the Machine Learning Security field and some other recent and hot topics in Learning From Data.
  
• 

  Feb 10th, 2022
  Towards a general framework for white-box adversarial attacks
  

  Speaker: Jérôme Rony (ÉTS Montréal)

  Jérôme Rony received his M.A.Sc. in Systems Engineering in 2019 from École de Technologie Supérieure (ÉTS) Montréal, Canada. He is currently a Ph.D. candidate at ÉTS Montréal. His current research interests include computer vision, adversarial examples, and optimization methods applied to deep learning.

  
  Adversarial attack algorithms are dominated by penalty methods, which are slow in practice, and more efficient distance-customized methods, which are heavily tailored to the properties of the distance considered. In this talk, we present the optimization problems related to adversarial examples generation with their specificities, and a white-box attack algorithm to generate minimally perturbed adversarial examples based on Augmented Lagrangian principles, which enjoys the generality of penalty methods and the computational efficiency of distance-customized algorithms.
  
• 

  Jan 11th, 2022
  Spinning Language Models for Propaganda-As-A-Service
  

  Speaker: Eugene Bagdasaryan (Cornell Tech)

  Eugene Bagdasaryan is a PhD Candidate at Cornell Tech advised by Vitaly Shmatikov and Deborah Estrin. He is an Apple AI/ML Scholar. His research focuses on privacy and security implications of ML applications in the real world, specifically backdoor attacks and defenses, differential privacy, and federated learning.

  
  In this talk Eugene will talk about an extension of backdoors that pose a new threat to neural sequence-to-sequence (seq2seq) models: training-time attacks that cause models to "spin" their outputs so as to support an adversary-chosen sentiment or point of view, but only when the input contains adversary-chosen trigger words. For example, a spinned summarization model would output positive summaries of any text that mentions the name of some individual or organization. Model spinning enables propaganda-as-a-service. An adversary can create customized language models that produce desired spins for chosen triggers, then deploy them to generate disinformation (a platform attack), or else inject them into ML training pipelines (a supply-chain attack), transferring malicious functionality to downstream models. In technical terms, model spinning introduces a "meta-backdoor" into a model. Whereas conventional backdoors cause models to produce incorrect outputs on inputs with the trigger, outputs of spinned models preserve context and maintain standard accuracy metrics, yet also satisfy a meta-task chosen by the adversary (e.g., positive sentiment). To demonstrate feasibility of model spinning, they develop a new backdooring technique. It stacks the adversarial meta-task onto a seq2seq model, backpropagates the desired meta-task output to points in the word-embedding space we call "pseudo-words," and uses pseudo-words to shift the entire output distribution of the seq2seq model. We evaluate this attack on language generation, summarization, and translation models with different triggers and meta-tasks such as sentiment, toxicity, and entailment. Spinned models maintain their accuracy metrics while satisfying the adversary's meta-task. In supply chain attack the spin transfers to downstream models. Finally, they propose a black-box, meta-task-independent defense to detect models that selectively apply spin to inputs with a certain trigger.
  
• 

  Dec 7th, 2021
  Relating Adversarially Robust Generalization to Flat Minima
  

  Speaker: David Stutz (Max Planck Institute for Informatics)

  David Stutz is a final-year PhD student at the Max Planck Institute for Informatics supervised by Prof. Bernt Schiele and co-supervised by Prof. Matthias Hein from the University of Tübingen. He obtained his bachelor and master degrees in computer science from RWTH Aachen University. During his studies, he completed an exchange program with the Georgia Institute of Technology as well as several internships at Microsoft, Fyusion and Hyundai MOBIS, among others. He wrote his master thesis at the Max Planck Institute for Intelligent Systems supervised by Prof. Andreas Geiger. His PhD research focuses on obtaining robust deep neural networks, considering adversarial examples, corrupted examples or out-of-distribution examples. In a collaboration with IBM Research, subsequent work improves robustness against bit errors in (quantized) weights to enable energy-efficient and secure accelerators. This work was awarded an outstanding paper award at the CVPR CV-AML Workshop 2021. More recently, during an internship at DeepMind, he used conformal prediction for uncertainty estimation in medical diagnosis. He received several awards and scholarships including the Qualcomm Innovation Fellowship, RWTH Aachen University’s Springorum Denkmünze and the STEM Award IT sponsored by ZF Friedrichshafen. His work has been published at top venues in computer vision and machine learning including ICCV, CVPR, IJCV, ICML and MLSys. More information can be found at www.davidstutz.de

  
  Adversarial training (AT) has become the de-facto standard to obtain models robust against adversarial examples. However, AT exhibits severe robust overfitting. In practice, this leads to poor robust generalization, i.e., adversarial robustness does not generalize well to new examples. In this talk, I want to present our work on the relationship between robust generalization and flatness of the robust loss landscape in weight space. I will propose average- and worst-case metrics to measure flatness in the robust loss landscape and show a correlation between good robust generalization and flatness. For example, throughout training, flatness reduces significantly during overfitting such that early stopping effectively finds flatter minima in the robust loss landscape. Similarly, AT variants achieving higher adversarial robustness also correspond to flatter minima. This holds for many popular choices, e.g., AT-AWP, TRADES, MART, AT with self-supervision or additional unlabeled examples, as well as simple regularization techniques, e.g., AutoAugment, weight decay or label noise.