How Security Will Evolve in the Age of AI
With new large AI models emerging every day, how do we proactively tackle the challenges of security and safeguard our online experiences? I hope this essay sparks more questions that stay with you.
Security is a complex field that encompasses everything from guaranteeing the Confidentiality, Integrity, and Availability (CIA) properties to thwarting malicious actors from exploiting systems. Proving the security of a system is as challenging as proving the non-existence of something in the world. My essay is in no way comprehensive, but rather an attempt to capture key ideas that linger in my mind. It is meant to be a general exploration, not restricted to a specific use case. I also narrow the scope of AI to Machine Learning domains, since the new AI wave primarily stems from ML techniques.
Now, let's break this down from the bottom up.
Infrastructure runs AI-powered products, which may not necessarily be aware of the higher stack of applications or ML models. In this context, we often employ the DevOps framework to ensure the model's functionality aligns with the product definition.
Traditionally, DevOps left out security entirely, but it has evolved into DevSecOps, where security engineering plays a pivotal role through security reviews, audits, policy enforcement, and monitoring. From the org structure point of view, that means a separate security team is responsible for security-related testing and review. In practice, due to the release pressure, many teams naturally deprioritize security.
The next iteration of the framework is SecDevOps, where teams lay down best practices and embed security early in the development lifecycle. It consists of two components: security as code (SaC) and infrastructure as code (IaC). SaC involves integrating security into DevOps tools and practices, while DevOps professionals use IaC to quickly establish and maintain infrastructure.
How does SecDevOps apply to MLOps? Mainly, the challenge of enhancing ML platforms to make security a first-class citizen. We need to develop and streamline the workflow for ML-native paradigms. That means, we will develop ML meta-measurement, adopt security practices through the ML model development lifecycle, conduct threat modeling with ML in mind, secure deployed models, and perform incident response.
One stack up, we have to think differently about data security and privacy when running ML natively in the organization’s infrastructure versus leveraging ML as a service (MLaaS). In both cases, organizations need to manage how data is used and how it impacts their employees and customers, with particular attention paid to data leakage from models.
For MLaaS, when data leaves the premises for third parties, simply anonymizing identities is no longer sufficient, as countless research has shown. No one wants to be in Samsung’s position, where their engineers accidentally leaked internal source code by uploading it to ChatGPT, who retains the data for re-training, making the model available to public use and susceptible to data exfiltration.
Organizations are required by law to invest more in reaching compliance; otherwise, they will face significant financial penalties. With GDPR compliances and similar regulations, users are not just silent anymore; they demand more transparency and control. Law and policy are catching up and will apply to generative AI and other unknown technologies in the future.
The industry generally agrees that having security and privacy built in early will enable organizational operation and growth down the road. Whether the investment becomes fruitful depends on how fast Privacy Enhancing Technology (PET) evolves to be production-ready and how well organizations adapt them (I’m talking about privacy engineering here).
All of the above S&P problems and frameworks are not new, but they present unique complications when we look at the security problems arising from deploying AI models.
First, I cannot say enough that the model becomes the code and sensitive data lives in the model; it’s no longer about what you write that dictates how the program will run, but the latent space or the hidden states of the model. We have landed in unexplored territory, which requires continuous monitoring and evaluation.
In critical decisions, we must go beyond a super large vector that speaks nothing more than numbers and figure out a way to enhance interpretability so that humans are part of the loop. Ultimately, humans are the subject of accountability, and holding them responsible will be the key to proactively address AI Trust & Safety issues. Measurability and scalability of the safe model requires some level of interpretability. The tooling around assessment, monitoring, and detection also needs to evolve to be model-conscious, enabling us to quickly recover from mistakes and adapt to new information.
Another concerning domain is applying ML models to make security decisions. Typical applications include fraud and abuse detection, spam filtering, and anomaly detection. We must carefully design production systems to prioritize both accuracy and reliability.
Simple models with well-crafted features deliver reliable decision-making, but might be easily gamed by curious or malicious entities in the system. On the other side of the spectrum, deep models may exhibit bias and misclassification issues that are unpredictable and challenging to control. We are likely to find answers in leveraging deep models to discover key invariants about malicious activities and design systems in a manner that adversaries cannot alter their behaviors without diminishing their offensive capabilities. This domain remains open for research, and many questions remain unanswered.
The development of large models is way faster than our ability to interpret the model itself. It almost seems like a losing game for the entire research field of eXplainable AI. If you have been to any museums that recount big turbulent eras caused by technological advancements, you would understand the constant battle between utility and security. The Steven F. Udvar-Hazy Center in DC showed me the actual human life cost that brought us to today's impressive safety systems in current airplanes. In less life-critical systems, we may see user churn due to false positives and significant losses due to false negatives.
What will happen, I predict, is a back-and-forth dance between speed and reliability. Most companies will continue deploying more powerful models to detect threats but sometimes make catastrophic mistakes. Then, great companies will slow down a bit and reason, so they can add necessary security measures and pave the secure road for a more robust and scalable operation.
On the other side of the equation, attackers are already using these "cheap" models to scale their operations. Generally speaking, destruction is so much easier than construction. We cannot ignore the overwhelming prevalence of misinformation, deepfake, spamware, and malware since the dawn of the internet, and the game has escalated further with the advent of generative models. No one can confidently say we are able to identify them just yet. We have to develop more and better counter models that can identify adversarial behavior with high throughput and accuracy, and equip them with adaptive platform policies to tackle these issues.
However, can we improve security capabilities by leveraging bigger models? I don’t think there is a one-size-fits-all answer. The threat landscape changes moment by moment, so updating our learning effectively is critical, and we should not rule out models of simple rules that offer a high target rate.
We need to pay attention to our hypotheses because they do not always match 100% when we use off-the-shelf models trained for general-purpose objectives. We might see no significant improvement or even worse performance in some scenarios. For example, detecting a visiting pattern anomaly does not naturally fit into NLP or Graphic models.
That being said, I see substantial potential in investing in fine-tuning pretrained models, which have the best-in-class world knowledge, for a well-defined use case. Businesses stand out by offering unique products, which means unique threats. Having the patience to curate unique data for specific business threats will help build moats to address business pains better than their competitors. Don’t forget that it might be worthwhile to buy good solutions first, while building great ones later.
Then how do we build great models? We start from simple models built on domain expertise. Many people overlook the value of interdisciplinary experience because it is not required to work in tech. They look for parts for the API centric world and they are successful. The side effect is that engineers are shielded from understanding the under-the-hood magic that makes things work.
Fortunately, engineers are not stagnant; often, they are curious and are willing to maintain a beginner's mind while working in different domains like engineering, data, and security. When threats come from all sides, their hands-on experience and end-to-end comprehension of systems allow them to connect dots and discover gaps.
Another captivating quality is the ability to encode such knowledge into black-box ML models in an applicable manner. It's like teaching a child a task in their language; the ML model is the child, and how we teach it matters. If the child recognizes the problem from first principles, they are likely to make decisions that ultimately align with organization core values.
The last and most important piece of building great models for competitive advantage is high-quality proprietary data that are customized for your business needs. When organizations amalgamate deep talents and unique dataset, they lay a secure foundation for rapid growth.
All that is to say, security will become even more complicated in the era of prevalent AI, and we should start creating security practices and capabilities that are ready for this next battleground.
Disclaimer: The essay is rooted in my own experience and viewpoint and does not acknowledge or deny my employer’s business plan or strategy in any way.