ai – IOActive

Artificial Intelligence (AI) is transforming industries and everyday life, driving innovations once relegated to the realm of science fiction into modern reality. As AI technologies grow more integral to complex systems like autonomous vehicles, healthcare diagnostics, and automated financial trading platforms, the imperative for robust security measures increases exponentially.

Securing AI is not only about safeguarding data but also about ensuring the core systems — in particular, the trained models that really put the “intelligence” in AI — function as intended without malicious interference. Historical lessons from earlier technologies offer some guidance and can be used to inform today’s strategies for securing AI systems. Here, we’ll explore the evolution, current state, and future direction of AI security, with a focus on why it’s essential to learn from the past, secure the present, and plan for a resilient future.

AI: The Newest Crown Jewel

Security in the context of AI is paramount precisely because AI systems increasingly handle sensitive data, make important, autonomous decisions, and operate with limited supervision in critical environments where safety and confidentiality are key. As AI technologies burrow further into sectors like healthcare, finance, and national security, the potential for misuse or harmful consequences due to security shortcomings rises to concerning levels. Several factors drive the criticality of AI security:

Data Sensitivity: AI systems process and learn from large volumes of data, including personally identifiable information, proprietary business information, and other sensitive data types. Ensuring the security of enterprise training data as it passes to and through AI models is crucial to maintaining privacy, regulatory compliance, and the integrity of intellectual property.
System Integrity: The integrity of AI systems themselves must be well defended in order to prevent malicious alterations or tampering that could lead to bogus outputs and incorrect decisions. In autonomous vehicles or medical diagnosis systems, for example, instructions issued by compromised AI platforms could have life-threatening consequences.
Operational Reliability: AI is increasingly finding its way into critical infrastructure and essential services. Therefore, ensuring these systems are secure from attacks is vital for maintaining their reliability and functionality in critical operations.
Matters of Trust: For AI to be widely adopted, users and stakeholders must trust that the systems are secure and will function as intended without causing unintended harm. Security breaches or failures can undermine public confidence and hinder the broader adoption of emerging AI technologies over the long haul.
Adversarial Activity: AI systems are uniquely susceptible to certain attacks, whereby slight manipulations in inputs — sometimes called prompt hacking — can deceive an AI system into making incorrect decisions or spewing malicious output. Understanding the capabilities of malicious actors and building robust defenses against such prompt-based attacks is crucial for the secure deployment of AI technologies.

In short, security in AI isn’t just about protecting data. It’s also about ensuring safe, reliable, and ethical use of AI technologies across all applications. These inexorably nested requirements continue to drive research and ongoing development of advanced security measures tailored to the unique challenges posed by AI.

Looking Back: Historical Security Pitfalls

We don’t have to turn the clock back very far to witness new, vigorously hyped technology solutions wreaking havoc on the global cybersecurity risk register. Consider the peer-to-peer recordkeeping database mechanism known as blockchain. When blockchain exploded into the zeitgeist circa 2008 — alongside the equally disruptive concept of cryptocurrency — its introduction brought great excitement thanks to its potential for both decentralization of data management and the promise of enhanced data security. In short order, however, events such as the DAO hack —an exploitation of smart contract vulnerabilities that led to substantial, if temporary, financial losses — demonstrated the risk of adopting new technologies without diligent security vetting.

As a teaching moment, the DAO incident highlights several issues: the complex interplay of software immutability and coding mistakes; and the disastrous consequences of security oversights in decentralized systems. The case study teaches us that with every innovative leap, a thorough understanding of the new security landscape is crucial, especially as we integrate similar technologies into AI-enabled systems.

Historical analysis of other emerging technology failures over the years reveals other common themes, such as overreliance on untested technologies, misjudgment of the security landscape, and underestimation of cyber threats. These pitfalls are exacerbated by hype-cycle-powered rapid adoption that often outstrips current security capacity and capabilities. For AI, these themes underscore the need for a security-first approach in development phases, continuous vulnerability assessments, and the integration of robust security frameworks from the outset.

Current State of AI Security

With AI solutions now pervasive, each use case introduces unique security challenges. Be it predictive analytics in finance, real-time decision-making systems in manufacturing systems, or something else entirely, each application requires a tailored security approach that takes into account the specific data types and operational environments involved. It’s a complex landscape where rapid technological advancements run headlong into evolving security concerns. Key features of this challenging infosec environment include:

Advanced Threats: AI systems face a range of sophisticated threats, including data poisoning, which can skew an AI’s learning and reinforcement processes, leading to flawed outputs; model theft, in which proprietary intellectual property is exposed; and other adversarial actions that can manipulate AI perceptions and decisions in unexpected and harmful ways. These threats are unique to AI and demand specialized security responses that go beyond traditional cybersecurity controls.
Regulatory and Compliance Issues: With statutes such as GDPR in Europe, CCPA in the U.S., and similar data security and privacy mandates worldwide, technology purveyors and end users alike are under increased pressure to prioritize safe data handling and processing. On top of existing privacy rules, the Biden administration in the U.S. issued a comprehensive executive order last October establishing new standards for AI safety and security. In Europe, meanwhile, the EU’s newly adopted Artificial Intelligence Act provides granular guidelines for dealing with AI-related risk. This spate of new rules can often clash with AI-enabled applications that demand more and more access to data without much regard for its origin or sensitivity.
Integration Challenges: As AI becomes more integrated into critical systems across a wide swath of vertical industries, ensuring security coherence across different platforms and blended technologies remains a significant challenge. Rapid adoption and integration expose modern AI systems to traditional threats and legacy network vulnerabilities, compounding the risk landscape.
Explainability: As adoption grows, the matter of AI explainability — or the ability to understand and interpret the decisions made by AI systems — becomes increasingly important. This concept is crucial in building trust, particularly in sensitive fields like healthcare where decisions can have profound impacts on human lives.Consider an AI system used to diagnose disease from medical imaging. If such a system identifies potential tumors in a scan, clinicians and patients must be able to understand the basis of these conclusions to trust in their reliability and accuracy. Without clear explanations, hesitation to accept the AI’s recommendations ensues, leading to delays in treatment or disregard of useful AI-driven insights. Explainability not only enhances trust, it also ensures AI tools can be effectively integrated into clinical workflows, providing clear guidance that healthcare professionals can evaluate alongside their own expertise.

Addressing such risks requires a deep understanding of AI operations and the development of specialized security techniques such as differential privacy, federated learning, and robust adversarial training methods. The good news here: In response to AI’s risk profile, the field of AI security research and development is on a steady growth trajectory. Over the past 18 months the industry has witnessed increased investment aimed at developing new methods to secure AI systems, such as encryption of AI models, robustness testing, and intrusion detection tailored to AI-specific operations.

At the same time, there’s also rising awareness of AI security needs beyond the boundaries of cybersecurity organizations and infosec teams. That’s led to better education and training for application developers and users, for example, on the potential risks and best practices for securing A-powered systems.

Overall, enterprises at large have made substantial progress in identifying and addressing AI-specific risk, but significant challenges remain, requiring ongoing vigilance, innovation, and adaptation in AI defensive strategies.

Data Classification and AI Security

One area getting a fair bit of attention in the context of safeguarding AI-capable environments is effective data classification. The ability to earmark data (public, proprietary, confidential, etc.) is essential for good AI security practice. Data classification ensures that sensitive information is handled appropriately within AI systems. Proper classification aids in compliance with regulations and prevents sensitive data from being used — intentionally or unintentionally — in training datasets that can be targets for attack and compromise.

The inadvertent inclusion of personally identifiable information (PII) in model training data, for example, is a hallmark of poor data management in an AI environment. A breach in such systems not only compromises privacy but exposes organizations to profound legal and reputational damage as well. Organizations in the business of adopting AI to further their business strategies must be ever aware of the need for stringent data management protocols and advanced data anonymization techniques before data enters the AI processing pipeline.

The Future of AI Security: Navigating New Horizons

As AI continues to evolve and tunnel its way further into every facet of human existence, securing these systems from potential threats, both current and future, becomes increasingly critical. Peering into AI’s future, it’s clear that any promising new developments in AI capabilities must be accompanied by robust strategies to safeguard systems and data against the sophisticated threats of tomorrow.

The future of AI security will depend heavily on our ability to anticipate potential security issues and tackle them proactively before they escalate. Here are some ways security practitioners can prevent future AI-related security shortcomings:

Continuous Learning and Adaptation: AI systems can be designed to learn from past attacks and adapt to prevent similar vulnerabilities in the future. This involves using machine learning algorithms that evolve continuously, enhancing their detection capabilities over time.
Enhanced Data Privacy Techniques: As data is the lifeblood of AI, employing advanced and emerging data privacy technologies such as differential privacy and homomorphic encryption will ensure that data can be used for training without exposing sensitive information.
Robust Security Protocols: Establishing rigorous security standards and protocols from the initial phases of AI development will be crucial. This includes implementing secure coding practices, regular security audits, and vulnerability assessments throughout the AI lifecycle.
Cross-Domain Collaboration: Sharing knowledge and strategies across industries and domains can lead to a more robust understanding of AI threats and mitigation strategies, fostering a community approach to AI security.

Looking Further Ahead

Beyond the immediate horizon, the field of AI security is set to witness several meaningful advancements:

Autonomous Security: AI systems capable of self-monitoring and self-defending against potential threats will soon become a reality. These systems will autonomously detect, analyze, and respond to threats in real time, greatly reducing the window for attacks.
Predictive Security Models: Leveraging big data and predictive analytics, AI can forecast potential security threats before they manifest. This proactive approach will allow organizations to implement defensive measures in advance.
AI in Cybersecurity Operations: AI will increasingly become both weapon and shield. AI is already being used to enhance cybersecurity operations, providing the ability to sift through massive amounts of data for threat detection and response at a speed and accuracy unmatchable by humans. The technology and its underlying methodologies will only get better with time. This ability for AI to remove the so-called “human speed bump” in incident detection and response will take on greater importance as the adversaries themselves increasingly leverage AI to generate malicious attacks that are at once faster, deeper, and potentially more damaging than ever before.
Decentralized AI Security Frameworks: With the rise of blockchain technology, decentralized approaches to AI security will likely develop. These frameworks can provide transparent and tamper-proof systems for managing AI operations securely.
Ethical AI Development: As part of securing AI, strong initiatives are gaining momentum to ensure that AI systems are developed with ethical considerations in mind will prevent biases and ensure fairness, thus enhancing security by aligning AI operations with human values.

As with any rapidly evolving technology, the journey toward a secure AI-driven future is complex and fraught with challenges. But with concerted effort and prudent innovation, it’s entirely within our grasp to anticipate and mitigate these risks effectively. As we advance, the integration of sophisticated AI security controls will not only protect against potential threats, it will foster trust and promote broader adoption of this transformative technology. The future of AI security is not just about defense but about creating a resilient, reliable foundation for the growth of AI across all sectors.

Charting a Path Forward in AI Security

Few technologies in the past generation have held the promise for world-altering innovation in the way AI has. Few would quibble with AI’s immense potential to disrupt and benefit human pursuits from healthcare to finance, from manufacturing to national security and beyond. Yes, Artificial Intelligence is revolutionary. But it’s not without cost. AI comes with its own inherent collection of vulnerabilities that require vigilant, innovative defenses tailored to their unique operational contexts.

As we’ve discussed, embracing sophisticated, proactive, ethical, collaborative AI security and privacy measures is the only way to ensure we’re not only safeguarding against potential threats but also fostering trust to promote the broader adoption of what most believe is a brilliantly transformative technology.

The journey towards a secure AI-driven future is indeed complex and fraught with obstacles. However, with concerted effort, continuous innovation, and a commitment to ethical practices, successfully navigating these impediments is well within our grasp. As AI continues to evolve, so too must our strategies for defending it.

It’s tough to be a secure code reviewer. There are already over 700 programming languages according to Wikipedia, and seemingly more languages materializing every year. Expectations are high that rapid developments in Artificial Generative Intelligence (AGI) will bring a new suite of languages and security issues that’ll have an oversized impact on software development. Consequently, secure software development lifecycle (SDL) processes and security code review are having to evolve rapidly.

I’m both excited and nervous about AGI advancements in the world of software development and secure application design. It’s exciting to see how prompt engineering of Large Language Models (LLM) and adoption of AI augmentation in the form of copilots and chatbots are increasing the pace of ideation into new products. I’m nervous about the hallucinations and code quality being generated in response though.

English as a Programming Language

2023 was the breakthrough year for AI, with LLM and AGI permeating every industry, technology, and product. Today, the most in-demand languages currently are Python, C, and C++ but, controversially, the future star programming language may in fact be English; something that’ll take some time to adjust to.

For over a decade we’ve been told that the supply of experienced cybersecurity professionals has trailed the market’s requirements, with a deficit growing year-on-year, and a casual scan across office desks and cubicles will highlight a more significant gender gap across the cybersecurity (and software development) industry. I think AGI and emergence of English as a critical programming language are fundamental to correcting both industry problems.

AGI, particularly those based upon LLM advancements, are increasingly sophisticated language machines – and women may have an advantage over men in maximizing utility and productivity from them.

Multiple studies over the last 30 years have constantly highlighted that women are better communicators than men. “Better” is obviously an explosive and controversial term even amongst the academics who published the studies, but in general women have more expansive vocabularies and stronger interpretative communication skills. Modern neuroscience and studies in children and adolescents identify girls as more garrulous than boys, with greater complexity and sophistication of language, and tend to develop more in the realm of listening with greater focus and concentration as they age. This historically translates into women being better coders than men (once you remove the bias in the system).

As I look to AGI and the expanding world of prompt engineering, I anticipate that women will have an advantage over their male developer counterparts. Strong and well-developed communication skills (and the reasoning and understanding that underlays those polished skills) are prerequisites for maximizing efficiency of AGI-returned results and tuning responses – both now and for the immediate future.

Starter-job Experience

But what about experience? The “experience gap” is often called out as a chasm for newly minted degree-holding graduates and landing a starter-job in cybersecurity.

It’s rare to find an entry-level job in our industry that doesn’t require multiple years of hands-on security experience nowadays as many of those traditional starter roles – network scanning, alert triage, playbook maintenance, patch management – have been automated away, with many more projected to disappear as AI adoption increases.

Most successful new entrants into the cybersecurity profession come from adjacent technical industries making a career jump rather than direct from a college or university. Armed with transferable skills and technical experience, they’re capable of crossing the chasm left in the wake of cyber automation. However, the security knowledge gap between a cybersecurity veteran and a recent transfer remains large and a growing concern for the industry.

I’m excited to think AI augmentation and copilot technologies will have one of the largest impacts on our industry – removing much of the security knowledge gap and reducing the overall impact of the experience gap – like what is happening in other industries, such as the medical field. For example, AI use in patient triage, predictive analytics, and virtual assistants are augmenting generalist regional nurses (two-year qualification) and Bachelor of Science in Nursing (four-year qualification) graduates, and allowing them to perform many of the roles and responsibilities traditionally associated with a completed medical doctor degree (10 to 12 years).

Secure Code Reviews

It’s tough to be a secure code reviewer. There aren’t enough of them. The job requires tremendous amounts of experience and advanced security knowledge, and it’s tiring and hard work.

AGI is going to have a huge impact on their job.

On the positive side, English as a programming language and AI augmentation and copilots is going to help increase both the breadth and depth of the cybersecurity talent pool available to perform this critical job. The tools available to code reviewers to automatically review and assess code security are advancing quickly and, while still in their first generation of AI adoption, are anticipated to mature rapidly and identify vulnerabilities and logic flaws with higher fidelity and trust. I’m sure there’ll be a gap between the best that a tool can achieve versus the best-of-the-best human expert though – especially when that expert is augmented and using similar tools themselves.

Meanwhile, AGI is spearheading prompt engineering of new software applications. A new generation of product developers may have little to no influence over the code that powers the application. Indeed, I’ve previously argued that the role of product manager will change greatly in the coming years as their skills in product design and requirement setting pivot from being directed to engineering teams and into AGI prompts instead.

What does an AGI-generated application look like under the covers? Time will tell. We anticipate it’ll increasingly become more secure – using best security practices and recycling pretested code behind the scenes – and that it’ll constantly learn, optimize, and apply best and better security practices – but we’ll still need those human secure code reviewers for some time to come, specially when it comes to high-impact applications and certification.

A concern though as AGI does more of the application development software developers will have less direct influence over the underlying code is that powering the application and business logic. It would be a nightmare if the AGI produced entirely different code throughout the application each time it received new design criteria and re-optimized – making vulnerability triage, reporting, reconciliation, and tracking near impossible, and code reviews and code certifications (human or tool led) largely nonsensical.

Watch this space!

I think we’re still some years away from having to worry about continuously reimagined code generated by AGI without human software developers tweaking and refining the underlaying application code, but it is tremendously exciting to see the rapid advances in prompt engineering and how LLM’s are being incorporated into both old and new products.

Our industry has consistently struggled to attract and retain women. AGI has the potential to not only level the field and make it easier to join the game, but to also leverage previously poorly-tapped communication skills for the betterment of both application development and security. There’s a lot of work ahead. There’s a lot of research to be done. There’s a lot of opportunities to make code more secure!

Tag: ai

The Security Imperative in Artificial Intelligence